diff --git a/README.md b/README.md index 955e6dd..cc33661 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,7 @@ Access the application locally at `http://localhost:3000`. ### The Actor Agent -Each entity takes turns through an **Actor Agent** that receives a strictly epistemically bounded prompt: its own attributes (public, plus private ones explicitly granted to itself), its subjective memory buffer, the entities co-present at its location, and the current moment. Nothing else. The actor responds with free narrative prose. +Each entity takes turns through an **Actor Agent** that receives a strictly epistemically bounded prompt: its own attributes (public, plus private ones explicitly granted to itself), its **Cognitive Buffer**, the entities co-present at its location, and the current moment. Nothing else. The actor responds with free narrative prose. Prose is decoded into typed intents: @@ -121,8 +121,8 @@ Space is a graph: `world → region → location → point of interest`, connect ### Memory Tiers -- **Verbatim Buffer (implemented):** Per-character subjective event log. Every entry is stored from the owner's perspective actors resolved through the owner's alias map, outcomes attached — and recalled with naturalized time phrasing. -- **Vector Archive (implemented):** Summarized, embedded memory entries for semantic retrieval, keeping verbatim quotes only for high-salience lines. +- **Cognitive Buffer (implemented):** Per-character subjective event log. Every entry is stored from the owner's perspective actors resolved through the owner's alias map, outcomes attached — and recalled with naturalized time phrasing. +- **Memory Ledger (implemented):** Summarized, embedded memory entries for semantic retrieval, keeping verbatim quotes only for high-salience lines. - **Dossier (planned):** Each observer's subjective beliefs about another character. Memory is per-character on purpose: recall is testimony from a vantage point, which is what makes interrogating two witnesses interesting. @@ -156,14 +156,14 @@ The finish line for the first milestone is small on purpose. `v0` is almost on t - [x] Typed intent pipeline: `dialogue` / `action` / `monologue`, decoded from free prose. - [x] World Architect: LLM validation plus time-delta generation, end-to-end for single actions. - [x] Actor Agent with epistemically-bounded prompts (self, memory, co-located entities, subjective time). -- [x] Verbatim memory buffer with per-observer subjective serialization and alias resolution. +- [x] Verbatim Cognitive Buffer with per-observer subjective serialization and alias resolution. - [x] Spatial location graph (data model; perception is co-location only). - [x] Scenario loader (JSON → SQLite) and a playable CLI loop with human or LLM actors. **[The `v0` Milestone:](https://github.com/sortedcord/omnia-consolidated/milestone/1)** - [x] Two hand-authored NPCs live in one location, playable via CLI. -- [x] Each has buffer and vector-archive memory and recalls something said a few turns earlier. +- [x] Each has Cognitive Buffer and Memory Ledger memory and recalls something said a few turns earlier. - [x] One NPC knows a fact the other does not and, provably by testing, will not leak it. - [x] The Architect processes at least one non-trivial action per exchange with a visible state change. - [x] The whole thing persists to a SQLite file and reloads identically. @@ -183,7 +183,7 @@ omnia/ intent/ intent types (dialogue/action/monologue) and the prose decoder architect/ World Architect: LLM validation plus time-delta generation actor/ actor agent: epistemically-bounded prompts, pluggable prose generators - memory/ verbatim buffer; later the vector archive, dossier, and affect vectors + memory/ Cognitive Buffer; Memory Ledger (vector archive), dossier, and affect vectors spatial/ location and POI graph, portal-based perception llm/ ILLMProvider interface plus Gemini and deterministic mock implementations scenario/ scenario JSON schema and loader (JSON → SQLite) diff --git a/apps/gui/next-env.d.ts b/apps/gui/next-env.d.ts index 9edff1c..c4b7818 100644 --- a/apps/gui/next-env.d.ts +++ b/apps/gui/next-env.d.ts @@ -1,6 +1,6 @@ /// /// -import "./.next/types/routes.d.ts"; +import "./.next/dev/types/routes.d.ts"; // NOTE: This file should not be edited // see https://nextjs.org/docs/app/api-reference/config/typescript for more information. diff --git a/apps/gui/package.json b/apps/gui/package.json index bfe39ce..69f8a44 100644 --- a/apps/gui/package.json +++ b/apps/gui/package.json @@ -19,6 +19,7 @@ "@omnia/memory": "workspace:*", "@omnia/scenario": "workspace:*", "@omnia/spatial": "workspace:*", + "@omnia/voice": "workspace:*", "@radix-ui/react-dialog": "^1.1.19", "@radix-ui/react-separator": "^1.1.11", "@radix-ui/react-slot": "^1.3.0", diff --git a/apps/gui/src/components/config/ConfigView.tsx b/apps/gui/src/components/config/ConfigView.tsx index a713b5d..ab80062 100644 --- a/apps/gui/src/components/config/ConfigView.tsx +++ b/apps/gui/src/components/config/ConfigView.tsx @@ -177,13 +177,13 @@ export function ConfigView() { { key: "handoff", label: "Memory Handoff Engine", - desc: "Promotes entities' working memories to the long-term Ledger via LLM summarization and pruning.", + desc: "Promotes entities' Cognitive Buffer entries to the Memory Ledger via LLM summarization and pruning.", type: "generative", }, { key: "embeddings", label: "Text Embeddings Generator", - desc: "Generates vector embeddings for long-term memory retrieval.", + desc: "Generates vector embeddings for Memory Ledger retrieval.", type: "embedding", }, ].map((task) => ( diff --git a/apps/gui/src/components/config/ProviderInstancesConfig.tsx b/apps/gui/src/components/config/ProviderInstancesConfig.tsx index f405581..ad84f2b 100644 --- a/apps/gui/src/components/config/ProviderInstancesConfig.tsx +++ b/apps/gui/src/components/config/ProviderInstancesConfig.tsx @@ -43,13 +43,7 @@ import { CardTitle, CardAction, } from "@/components/ui/card"; -import { - Item, - ItemContent, - ItemGroup, - ItemTitle, - ItemDescription, -} from "@/components/ui/item"; +import { Item, ItemContent, ItemGroup, ItemTitle } from "@/components/ui/item"; import { Empty, EmptyTitle, EmptyDescription } from "@/components/ui/empty"; import { cn } from "@/lib/utils"; import { RefreshCwIcon } from "lucide-react"; @@ -200,14 +194,12 @@ export function ProviderInstancesConfig({ fetchModelsForExistingInstance(selectedInstanceId); } } - // eslint-disable-next-line react-hooks/exhaustive-deps }, [selectedInstanceId, instances, availableProviders]); // Re-fetch models when provider/key/endpoint changes on new instance form useEffect(() => { if (selectedInstanceId !== "new") return; fetchModelsForNewInstance(editProvider, editKey, editEndpointUrl); - // eslint-disable-next-line react-hooks/exhaustive-deps }, [editProvider, editKey, editEndpointUrl, selectedInstanceId]); const handleProviderChange = (providerId: string | null) => { diff --git a/apps/gui/src/components/play/HandoffModal.tsx b/apps/gui/src/components/play/HandoffModal.tsx new file mode 100644 index 0000000..55af187 --- /dev/null +++ b/apps/gui/src/components/play/HandoffModal.tsx @@ -0,0 +1,254 @@ +"use client"; + +import { useState } from "react"; +import type { SimSnapshot } from "@/lib/simulation-types"; +import { + Dialog, + DialogContent, + DialogHeader, + DialogTitle, +} from "@/components/ui/dialog"; +import { Badge } from "@/components/ui/badge"; +import { PromptAnalyzer } from "@/components/play/PromptAnalyzer"; + +interface HandoffModalProps { + entry: SimSnapshot["log"][number]; + onClose: () => void; +} + +export function HandoffModal({ entry, onClose }: HandoffModalProps) { + const [activeTab, setActiveTab] = useState<"chunks" | "prompt" | "output">( + "chunks", + ); + + const handoffResult = entry.handoffResult; + const chunks = handoffResult?.chunks || []; + + const getImportanceColor = (score: number) => { + if (score >= 8) + return "bg-destructive/10 text-destructive border-destructive/30"; + if (score >= 5) return "bg-amber-500/10 text-amber-500 border-amber-500/30"; + return "bg-emerald-500/10 text-emerald-500 border-emerald-500/30"; + }; + + return ( + !open && onClose()}> + + + + Memory Handoff Details — {entry.entityName} + {entry.usage && ( + + {entry.usage.modelName || "Handoff Model"} + + )} + + + + {/* Custom Tab Switcher */} +
+ + + +
+ +
+ {activeTab === "chunks" && ( +
+ {entry.usage && ( +
+
+ + Input Tokens + + + {entry.usage.inputTokens} + +
+
+ + Output Tokens + + + {entry.usage.outputTokens} + +
+
+ + Total Tokens + + + {entry.usage.totalTokens} + +
+
+ )} + + {chunks.length === 0 ? ( +
+

+ No memories were promoted to the Memory Ledger during this + turn. +

+

+ All Cognitive Buffer entries were summarized or forgotten. +

+
+ ) : ( +
+

+ Memory Ledger Additions +

+ {chunks.map( + ( + chunk: { content: string; importance: number }, + index: number, + ) => ( +
+
+
+ {chunk.content} +
+ + Importance: {chunk.importance} + +
+ + {chunk.quotes && chunk.quotes.length > 0 && ( +
+ {chunk.quotes.map((quote: string, qIdx: number) => ( +
“{quote}”
+ ))} +
+ )} + +
+ {chunk.retainInBuffer ? ( + + Pinned in Buffer + + ) : ( + + Pruned from Buffer + + )} + + {chunk.involvedEntityIds && + chunk.involvedEntityIds.length > 0 && ( +
+ Entities: + {chunk.involvedEntityIds.map( + (entId: string) => ( + + {entId} + + ), + )} +
+ )} +
+
+ ), + )} +
+ )} +
+ )} + + {activeTab === "prompt" && entry.rawPrompt && ( + 0 + ? entry.rawPrompt.components + : [ + { + label: "System Prompt", + type: "system", + content: entry.rawPrompt.systemPrompt || "", + }, + { + label: "User Context", + type: "world", + content: entry.rawPrompt.userContext || "", + }, + ] + } + inputTokens={entry.usage?.inputTokens || 0} + maxContext={ + entry.usage?.maxContext !== undefined + ? entry.usage.maxContext + : 32768 + } + modelName={entry.usage?.modelName} + providerInstanceName={entry.usage?.providerInstanceName} + outputLabel="LLM Output (Promoted Memory Chunks)" + outputText={ + handoffResult + ? JSON.stringify(handoffResult, null, 2) + : undefined + } + outputTokens={entry.usage?.outputTokens} + /> + )} + + {activeTab === "output" && ( +
+

+ Raw JSON Output +

+
+                {handoffResult
+                  ? JSON.stringify(handoffResult, null, 2)
+                  : "No JSON Output recorded."}
+              
+
+ )} +
+
+
+ ); +} diff --git a/apps/gui/src/components/play/InteractView.tsx b/apps/gui/src/components/play/InteractView.tsx index 92ddb04..503b484 100644 --- a/apps/gui/src/components/play/InteractView.tsx +++ b/apps/gui/src/components/play/InteractView.tsx @@ -7,16 +7,28 @@ import { Button } from "@/components/ui/button"; import { Textarea } from "@/components/ui/textarea"; import { Spinner } from "@/components/ui/spinner"; import { cn } from "@/lib/utils"; +import { hydrate } from "@omnia/voice"; +import { + Alert, + AlertAction, + AlertDescription, + AlertTitle, +} from "@/components/ui/alert"; function IntentTag({ intent, - isSelf, + playerAliases, + playerId, + entities, }: { intent: SimSnapshot["log"][number]["intents"][number]; - isSelf?: boolean; + playerAliases: Record; + playerId: string; + entities: SimSnapshot["entities"]; }) { const labels: Record = { monologue: "thought", + thought: "thought", dialogue: "dialogue", action: "action", }; @@ -28,10 +40,27 @@ function IntentTag({ outcome = intent.isValid ? " ✅" : ` ❌ (${intent.reason})`; } - const textToDisplay = - isSelf && intent.selfDescription - ? intent.selfDescription - : intent.description; + const viewerAliasesMap = new Map(); + if (entities) { + for (const ent of entities) { + viewerAliasesMap.set(ent.id, ent.name || ent.id); + } + } + if (playerAliases) { + for (const [targetId, alias] of Object.entries(playerAliases)) { + viewerAliasesMap.set(targetId, alias); + } + } + + const viewerEntityMock = { + id: playerId || "", + aliases: viewerAliasesMap, + }; + + const textToDisplay = hydrate( + intent.content, + viewerEntityMock as unknown as Parameters[1], + ); const modifiersStr = intent.modifiers && intent.modifiers.length > 0 ? ( @@ -69,10 +98,16 @@ function LogEntryCard({ entry, onShowPrompt, isPlayerCard, + playerAliases, + playerId, + entities, }: { entry: SimSnapshot["log"][number]; onShowPrompt: (entry: SimSnapshot["log"][number]) => void; isPlayerCard: boolean; + playerAliases: Record; + playerId: string; + entities: SimSnapshot["entities"]; }) { const showMenu = !!(entry.rawPrompt || entry.decoderPrompt); @@ -110,7 +145,13 @@ function LogEntryCard({
{entry.intents.map((intent, i) => ( - + ))}
@@ -125,6 +166,7 @@ interface InteractViewProps { setPlayerInput: (value: string) => void; onSubmitAction: (e: React.FormEvent) => void; onShowPrompt: (entry: SimSnapshot["log"][number]) => void; + onShowHandoff: (entry: SimSnapshot["log"][number]) => void; logEndRef: React.RefObject; } @@ -136,6 +178,7 @@ export function InteractView({ setPlayerInput, onSubmitAction, onShowPrompt, + onShowHandoff, logEndRef, }: InteractViewProps) { const router = useRouter(); @@ -147,14 +190,49 @@ export function InteractView({ {/* Scrollable Center Viewport */}
- {snapshot.log.map((entry, i) => ( - - ))} + {snapshot.log.map((entry, i) => { + if (entry.isHandoff) { + return ( + +
+ + Handoff triggered for {entry.entityName} + + + Memories were transferred from Cognitive Buffer to Memory + Ledger + +
+ + + +
+ ); + } + const playerAliases = playerEntity?.aliases || {}; + const playerId = playerEntity?.id || ""; + + return ( + + ); + })} {loading && (
diff --git a/apps/gui/src/components/play/PlayView.tsx b/apps/gui/src/components/play/PlayView.tsx index 322d4f8..ed1b284 100644 --- a/apps/gui/src/components/play/PlayView.tsx +++ b/apps/gui/src/components/play/PlayView.tsx @@ -11,6 +11,7 @@ import type { SimSnapshot } from "@/lib/simulation-types"; import { Button } from "@/components/ui/button"; import { Spinner } from "@/components/ui/spinner"; import { PromptModal } from "./PromptModal"; +import { HandoffModal } from "./HandoffModal"; import { InteractView } from "./InteractView"; import { ManageView } from "./ManageView"; import { cn } from "@/lib/utils"; @@ -56,6 +57,9 @@ export function PlayView() { const [selectedEntryForModal, setSelectedEntryForModal] = useState< SimSnapshot["log"][number] | null >(null); + const [selectedHandoffForModal, setSelectedHandoffForModal] = useState< + SimSnapshot["log"][number] | null + >(null); const logEndRef = useRef(null); const steppingRef = useRef(false); @@ -384,6 +388,7 @@ export function PlayView() { setPlayerInput={setPlayerInput} onSubmitAction={handleSubmitAction} onShowPrompt={setSelectedEntryForModal} + onShowHandoff={setSelectedHandoffForModal} logEndRef={logEndRef} /> ) : ( @@ -403,6 +408,13 @@ export function PlayView() { onClose={() => setSelectedEntryForModal(null)} /> )} + + {selectedHandoffForModal && ( + setSelectedHandoffForModal(null)} + /> + )}
); diff --git a/apps/gui/src/components/play/PromptAnalyzer.tsx b/apps/gui/src/components/play/PromptAnalyzer.tsx new file mode 100644 index 0000000..8a064a7 --- /dev/null +++ b/apps/gui/src/components/play/PromptAnalyzer.tsx @@ -0,0 +1,172 @@ +"use client"; + +import { + Accordion, + AccordionContent, + AccordionItem, + AccordionTrigger, +} from "@/components/ui/accordion"; +import type { PromptComponent } from "@/lib/simulation-types"; + +interface PromptAnalyzerProps { + components: PromptComponent[]; + inputTokens: number; + maxContext?: number; + modelName?: string; + providerInstanceName?: string; + outputLabel?: string; + outputText?: string; + outputTokens?: number; +} + +export function PromptAnalyzer({ + components, + inputTokens, + maxContext = 32768, + modelName, + providerInstanceName, + outputLabel = "LLM Output", + outputText, + outputTokens, +}: PromptAnalyzerProps) { + const totalLen = components.reduce((sum, s) => sum + s.content.length, 0); + + if (totalLen === 0) { + return ( +
+ No prompt context recorded. +
+ ); + } + + const sections = components.map((s) => { + const pct = totalLen > 0 ? (s.content.length / totalLen) * 100 : 0; + return { + ...s, + pct, + tokens: Math.round((s.content.length / totalLen) * inputTokens), + }; + }); + + const usagePctOfContext = + maxContext > 0 ? (inputTokens / maxContext) * 100 : 0; + const isAbsolute = maxContext > 0 && usagePctOfContext >= 20; + + const getColorClass = (type: string) => { + switch (type) { + case "system": + return "bg-blue-500"; + case "world": + return "bg-emerald-500"; + case "events": + return "bg-purple-500"; + case "memories": + return "bg-pink-500"; + case "input": + return "bg-amber-500"; + default: + return "bg-slate-500"; + } + }; + + return ( +
+ {/* Provider Details */} + {(providerInstanceName || modelName) && ( +
+ LLM Instance:{" "} + {providerInstanceName || "Default"} + {modelName && ({modelName})} +
+ )} + + {/* Progress Bar & Breakdown */} +
+
+ Input Prompt Breakdown + + Total Input Tokens: {inputTokens} + {maxContext > 0 ? ( + + {" "} + / {maxContext} ({usagePctOfContext.toFixed(1)}% used) + + ) : ( + (infinite context) + )} + +
+ + {/* Token Bar */} +
+ {sections.map((item, idx) => { + const widthPct = isAbsolute + ? item.pct * (inputTokens / maxContext) + : item.pct; + return ( +
+ ); + })} + {isAbsolute && ( +
+ )} +
+ + {/* Accordion Components */} + + {sections.map((item, idx) => { + return ( + + +
+ + {item.label}: + + {item.tokens} tokens ( + {item.pct.toFixed(0)}%) + +
+
+ +
+                    {item.content}
+                  
+
+
+ ); + })} +
+
+ + {/* Output Section */} + {outputText && ( +
+
+ {outputLabel} + {outputTokens !== undefined && ( + + Total Output Tokens: {outputTokens} + + )} +
+
+
+              {outputText}
+            
+
+
+ )} +
+ ); +} diff --git a/apps/gui/src/components/play/PromptModal.tsx b/apps/gui/src/components/play/PromptModal.tsx index 36c940f..20f05b4 100644 --- a/apps/gui/src/components/play/PromptModal.tsx +++ b/apps/gui/src/components/play/PromptModal.tsx @@ -1,7 +1,7 @@ "use client"; import { useEffect, useState } from "react"; -import type { SimSnapshot } from "@/lib/simulation-types"; +import type { SimSnapshot, PromptBreakdown } from "@/lib/simulation-types"; import { Dialog, DialogContent, @@ -9,13 +9,8 @@ import { DialogTitle, } from "@/components/ui/dialog"; -import { - Accordion, - AccordionItem, - AccordionTrigger, - AccordionContent, -} from "@/components/ui/accordion"; import { PromptSwitcher } from "@/components/play/PromptSwitcher"; +import { PromptAnalyzer } from "@/components/play/PromptAnalyzer"; interface PromptModalProps { entry: SimSnapshot["log"][number]; @@ -23,172 +18,7 @@ interface PromptModalProps { } export function PromptModal({ entry, onClose }: PromptModalProps) { - const [activeTab, setActiveTab] = useState<"actor" | "decoder">("actor"); - - const parseActorPrompt = ( - systemPrompt: string, - userContext: string, - inputTokens: number, - ) => { - const recentHeader = "=== RECENT EVENTS ==="; - const ledgerHeader = "=== YOUR MEMORIES ==="; - - const recentIdx = userContext.indexOf(recentHeader); - let worldStr = userContext; - let recentStr = ""; - let ledgerStr = ""; - - if (recentIdx !== -1) { - worldStr = userContext.substring(0, recentIdx).trim(); - const rest = userContext.substring(recentIdx).trim(); - const ledgerIdx = rest.indexOf(ledgerHeader); - if (ledgerIdx !== -1) { - recentStr = rest.substring(0, ledgerIdx).trim(); - ledgerStr = rest.substring(ledgerIdx).trim(); - } else { - recentStr = rest; - } - } - - const sections: { label: string; type: string; content: string }[] = [ - { label: "System Prompt", type: "system", content: systemPrompt }, - { label: "World Info", type: "world", content: worldStr }, - { - label: "Recent Events", - type: "events", - content: recentStr || "(No recent events.)", - }, - { - label: "Long-Term Memories", - type: "memories", - content: ledgerStr || "(No long-term memories.)", - }, - ]; - - const totalLen = sections.reduce((sum, s) => sum + s.content.length, 0); - if (totalLen === 0) return null; - - return sections.map((s) => { - const pct = (s.content.length / totalLen) * 100; - return { - ...s, - pct, - relativePct: pct, - tokens: Math.round((s.content.length / totalLen) * inputTokens), - }; - }); - }; - - const parseDecoderPrompt = ( - systemPrompt: string, - userContext: string, - inputTokens: number, - ) => { - const proseHeader = "=== NARRATIVE PROSE ==="; - const idx = userContext.indexOf(proseHeader); - - let worldStr = userContext; - let proseStr = ""; - - if (idx !== -1) { - worldStr = userContext.substring(0, idx).trim(); - proseStr = userContext.substring(idx).trim(); - } - - const sysLen = systemPrompt.length; - const worldLen = worldStr.length; - const proseLen = proseStr.length; - const totalLen = sysLen + worldLen + proseLen; - - if (totalLen === 0) return null; - - const sysPct = (sysLen / totalLen) * 100; - const worldPct = (worldLen / totalLen) * 100; - const prosePct = (proseLen / totalLen) * 100; - - const sysTokens = Math.round((sysLen / totalLen) * inputTokens); - const worldTokens = Math.round((worldLen / totalLen) * inputTokens); - const proseTokens = Math.max(0, inputTokens - sysTokens - worldTokens); - - return [ - { - label: "System Prompt", - pct: sysPct, - relativePct: sysPct, - tokens: sysTokens, - type: "system", - content: systemPrompt, - }, - { - label: "Decoder Context", - pct: worldPct, - relativePct: worldPct, - tokens: worldTokens, - type: "world", - content: worldStr, - }, - { - label: "Narrative Prose", - pct: prosePct, - relativePct: prosePct, - tokens: proseTokens, - type: "memories", - content: proseStr, - }, - ]; - }; - - const actorBreakdown = - entry.rawPrompt && entry.usage - ? parseActorPrompt( - entry.rawPrompt.systemPrompt, - entry.rawPrompt.userContext, - entry.usage.inputTokens, - ) - : null; - const decoderBreakdown = - entry.decoderPrompt && entry.decoderUsage - ? parseDecoderPrompt( - entry.decoderPrompt.systemPrompt, - entry.decoderPrompt.userContext, - entry.decoderUsage.inputTokens, - ) - : null; - - const actorMaxContext = - entry.usage?.maxContext !== undefined ? entry.usage.maxContext : 32768; - const actorUsedTokens = entry.usage?.inputTokens || 0; - const actorUsagePctOfContext = - actorMaxContext > 0 ? (actorUsedTokens / actorMaxContext) * 100 : 0; - const isActorAbsolute = actorMaxContext > 0 && actorUsagePctOfContext >= 20; - - const scaledActorBreakdown = actorBreakdown - ? actorBreakdown.map((item) => ({ - ...item, - pct: isActorAbsolute - ? item.relativePct * (actorUsedTokens / actorMaxContext) - : item.relativePct, - })) - : null; - - const decoderMaxContext = - entry.decoderUsage?.maxContext !== undefined - ? entry.decoderUsage.maxContext - : 32768; - const decoderUsedTokens = entry.decoderUsage?.inputTokens || 0; - const decoderUsagePctOfContext = - decoderMaxContext > 0 ? (decoderUsedTokens / decoderMaxContext) * 100 : 0; - const isDecoderAbsolute = - decoderMaxContext > 0 && decoderUsagePctOfContext >= 20; - - const scaledDecoderBreakdown = decoderBreakdown - ? decoderBreakdown.map((item) => ({ - ...item, - pct: isDecoderAbsolute - ? item.relativePct * (decoderUsedTokens / decoderMaxContext) - : item.relativePct, - })) - : null; + const [activeTab, setActiveTab] = useState("actor"); useEffect(() => { if (!entry.rawPrompt && entry.decoderPrompt) { @@ -196,9 +26,47 @@ export function PromptModal({ entry, onClose }: PromptModalProps) { } }, [entry]); + // Helper to resolve components with a fallback if none exist (for backwards-compatibility) + const getComponents = ( + promptBreakdown: PromptBreakdown | null | undefined, + defaultType: "world" | "input", + ) => { + if (!promptBreakdown) return []; + if (promptBreakdown.components && promptBreakdown.components.length > 0) { + return promptBreakdown.components; + } + // Fallback: convert flat strings into components list + return [ + { + label: "System Prompt", + type: "system" as const, + content: promptBreakdown.systemPrompt || "", + }, + { + label: "User Context", + type: defaultType, + content: promptBreakdown.userContext || "", + }, + ]; + }; + + const actorComponents = getComponents(entry.rawPrompt, "world"); + const decoderComponents = getComponents(entry.decoderPrompt, "input"); + + const isValidatorTab = activeTab.startsWith("validator-"); + const validatorIndex = isValidatorTab + ? parseInt(activeTab.substring("validator-".length), 10) + : -1; + const validatorCall = isValidatorTab + ? entry.validatorCalls?.find((c) => c.intentIndex === validatorIndex) + : null; + const validatorComponents = validatorCall + ? getComponents(validatorCall.prompt, "world") + : []; + return ( !open && onClose()}> - + Raw Prompts & Token Usage ({entry.entityName}) @@ -210,244 +78,78 @@ export function PromptModal({ entry, onClose }: PromptModalProps) { onTabChange={setActiveTab} hasActor={!!entry.rawPrompt} hasDecoder={!!entry.decoderPrompt} + validatorCalls={ + entry.validatorCalls?.map((c) => ({ + intentIndex: c.intentIndex, + intentContent: c.intentContent, + })) || [] + } />
{activeTab === "actor" && entry.rawPrompt && ( -
- {entry.usage ? ( -
- LLM Instance:{" "} - {entry.usage.providerInstanceName || "Default"} - {entry.usage.modelName && ( - ({entry.usage.modelName}) - )} -
- ) : ( -
- No LLM token usage (Player turn used fixed prose). -
- )} - - {scaledActorBreakdown && ( -
-
- - Input Prompt Breakdown - - - Total Input Tokens: {actorUsedTokens} - {actorMaxContext > 0 ? ( - - {" "} - / {actorMaxContext} ( - {actorUsagePctOfContext.toFixed(1)}% used) - - ) : ( - (infinite context) - )} - -
-
- {scaledActorBreakdown.map((item, idx) => { - const displayPct = - actorMaxContext > 0 - ? (item.tokens / actorMaxContext) * 100 - : item.relativePct; - return ( -
- ); - })} - {isActorAbsolute && ( -
- )} -
- - {scaledActorBreakdown.map((item, idx) => { - const displayPct = - actorMaxContext > 0 - ? (item.tokens / actorMaxContext) * 100 - : item.relativePct; - return ( - - - - {item.label}: {item.tokens} tokens - ({displayPct.toFixed(0)}%) - - -
-                              {item.content}
-                            
-
-
- ); - })} -
-
- )} - - {entry.usage && ( -
-
- LLM Output - - Total Output Tokens:{" "} - {entry.usage.outputTokens} - -
-
-
-                      {entry.narrativeProse}
-                    
-
-
- )} -
+ )} {activeTab === "decoder" && entry.decoderPrompt && ( -
- {entry.decoderUsage && ( -
- LLM Instance:{" "} - - {entry.decoderUsage.providerInstanceName || "Default"} - - {entry.decoderUsage.modelName && ( - ({entry.decoderUsage.modelName}) - )} -
+ + )} - {scaledDecoderBreakdown && ( -
-
- - Input Prompt Breakdown - - - Total Input Tokens: {decoderUsedTokens} - {decoderMaxContext > 0 ? ( - - {" "} - / {decoderMaxContext} ( - {decoderUsagePctOfContext.toFixed(1)}% used) - - ) : ( - (infinite context) - )} - -
-
- {scaledDecoderBreakdown.map((item, idx) => { - const displayPct = - decoderMaxContext > 0 - ? (item.tokens / decoderMaxContext) * 100 - : item.relativePct; - return ( -
- ); - })} - {isDecoderAbsolute && ( -
- )} -
- - {scaledDecoderBreakdown.map((item, idx) => { - const displayPct = - decoderMaxContext > 0 - ? (item.tokens / decoderMaxContext) * 100 - : item.relativePct; - return ( - - - - {item.label}: {item.tokens} tokens - ({displayPct.toFixed(0)}%) - - -
-                              {item.content}
-                            
-
-
- ); - })} -
-
- )} + {validatorCall && validatorCall.prompt && ( + + )} - {entry.decoderUsage && ( -
-
- LLM Output - - Total Output Tokens:{" "} - {entry.decoderUsage.outputTokens} - -
-
-
-                      {JSON.stringify(entry.intents, null, 2)}
-                    
-
-
- )} + {validatorCall && !validatorCall.prompt && ( +
+ + Bypassed LLM Validation + +

+ {validatorCall.response.reason} +

)}
diff --git a/apps/gui/src/components/play/PromptSwitcher.tsx b/apps/gui/src/components/play/PromptSwitcher.tsx index a30da29..b03677c 100644 --- a/apps/gui/src/components/play/PromptSwitcher.tsx +++ b/apps/gui/src/components/play/PromptSwitcher.tsx @@ -1,10 +1,11 @@ "use client"; interface PromptSwitcherProps { - activeTab: "actor" | "decoder"; - onTabChange: (tab: "actor" | "decoder") => void; + activeTab: string; + onTabChange: (tab: string) => void; hasActor: boolean; hasDecoder: boolean; + validatorCalls?: { intentIndex: number; intentContent: string }[]; } export function PromptSwitcher({ @@ -12,32 +13,67 @@ export function PromptSwitcher({ onTabChange, hasActor, hasDecoder, + validatorCalls = [], }: PromptSwitcherProps) { return ( -
- - - +
+ {/* Primary Pipeline (Linear flow to the left) */} +
+ + + + + +
+ + {/* Branching Validator Column to the right of Intent Decoder */} + {validatorCalls.length > 0 && ( +
+ + +
+
+ {validatorCalls.map((call) => { + const tabKey = `validator-${call.intentIndex}`; + return ( + + ); + })} +
+
+
+ )}
); } diff --git a/apps/gui/src/components/ui/alert.tsx b/apps/gui/src/components/ui/alert.tsx new file mode 100644 index 0000000..591cad1 --- /dev/null +++ b/apps/gui/src/components/ui/alert.tsx @@ -0,0 +1,62 @@ +import * as React from "react"; +import { cn } from "@/lib/utils"; + +const Alert = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)); +Alert.displayName = "Alert"; + +const AlertTitle = React.forwardRef< + HTMLParagraphElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)); +AlertTitle.displayName = "AlertTitle"; + +const AlertDescription = React.forwardRef< + HTMLParagraphElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)); +AlertDescription.displayName = "AlertDescription"; + +const AlertAction = React.forwardRef< + HTMLDivElement, + React.HTMLAttributes +>(({ className, ...props }, ref) => ( +
+)); +AlertAction.displayName = "AlertAction"; + +export { Alert, AlertTitle, AlertDescription, AlertAction }; diff --git a/apps/gui/src/components/ui/button.tsx b/apps/gui/src/components/ui/button.tsx index 6af517f..a9b62d9 100644 --- a/apps/gui/src/components/ui/button.tsx +++ b/apps/gui/src/components/ui/button.tsx @@ -22,6 +22,7 @@ const buttonVariants = cva( size: { default: "h-10 px-4 py-2", sm: "h-9 px-3 text-xs", + xs: "h-7 px-2.5 text-xs", lg: "h-11 px-8 text-base", icon: "h-10 w-10", }, diff --git a/apps/gui/src/components/ui/combobox.tsx b/apps/gui/src/components/ui/combobox.tsx index 52d5399..a4d0ebe 100644 --- a/apps/gui/src/components/ui/combobox.tsx +++ b/apps/gui/src/components/ui/combobox.tsx @@ -5,7 +5,6 @@ import { Combobox as ComboboxPrimitive } from "@base-ui/react"; import { CheckIcon, ChevronDownIcon, XIcon } from "lucide-react"; import { cn } from "@/lib/utils"; -import { Button } from "@/components/ui/button"; import { InputGroup, InputGroupAddon, diff --git a/apps/gui/src/components/ui/input-group.tsx b/apps/gui/src/components/ui/input-group.tsx index d005a09..2caa3ed 100644 --- a/apps/gui/src/components/ui/input-group.tsx +++ b/apps/gui/src/components/ui/input-group.tsx @@ -99,12 +99,13 @@ function InputGroupButton({ return React.cloneElement(render, { className: cn( inputGroupButtonVariants({ size }), - (render.props as any)?.className, + (render.props as Record)?.className as + string | undefined, className, ), type, ...props, - } as any); + } as Record as React.HTMLAttributes); } return ( diff --git a/apps/gui/src/lib/simulation-types.ts b/apps/gui/src/lib/simulation-types.ts index c4ee4bd..81037f3 100644 --- a/apps/gui/src/lib/simulation-types.ts +++ b/apps/gui/src/lib/simulation-types.ts @@ -1,7 +1,6 @@ export interface IntentInfo { type: string; - description: string; - selfDescription?: string; + content: string; modifiers: string[]; targetIds: string[]; isValid?: boolean; @@ -9,16 +8,25 @@ export interface IntentInfo { minutesToAdvance?: number; } -export interface LogEntry { - turn: number; - entityId: string; - entityName: string; - narrativeProse: string; - intents: IntentInfo[]; - timestamp: string; - rawPrompt?: { - systemPrompt: string; - userContext: string; +export interface PromptComponent { + label: string; + type: "system" | "world" | "events" | "memories" | "input" | "other"; + content: string; +} + +export interface PromptBreakdown { + systemPrompt: string; + userContext: string; + components?: PromptComponent[]; +} + +export interface ValidatorCall { + intentIndex: number; + intentContent: string; + prompt?: PromptBreakdown; + response: { + isValid: boolean; + reason: string; }; usage?: { inputTokens: number; @@ -28,10 +36,36 @@ export interface LogEntry { providerInstanceName?: string; maxContext?: number; }; - decoderPrompt?: { - systemPrompt: string; - userContext: string; +} + +export interface HandoffResult { + chunks: { + content: string; + importance: number; + }[]; +} + +export interface LogEntry { + turn: number; + entityId: string; + entityName: string; + narrativeProse: string; + intents: IntentInfo[]; + timestamp: string; + isHandoff?: boolean; + handoffResult?: HandoffResult; + decodedIntents?: IntentInfo[]; + validatorCalls?: ValidatorCall[]; + rawPrompt?: PromptBreakdown; + usage?: { + inputTokens: number; + outputTokens: number; + totalTokens: number; + modelName?: string; + providerInstanceName?: string; + maxContext?: number; }; + decoderPrompt?: PromptBreakdown; decoderUsage?: { inputTokens: number; outputTokens: number; @@ -47,6 +81,7 @@ export interface EntityInfo { name: string; isPlayer: boolean; isAgent: boolean; + aliases?: Record; } export interface WaitingContext { diff --git a/apps/gui/src/lib/simulation/alias-handoff.ts b/apps/gui/src/lib/simulation/alias-handoff.ts index dc17697..f80bc03 100644 --- a/apps/gui/src/lib/simulation/alias-handoff.ts +++ b/apps/gui/src/lib/simulation/alias-handoff.ts @@ -3,7 +3,7 @@ import type { SimSession } from "./types"; /** * Runs the HandoffEngine for every agent entity that has accumulated enough - * buffer entries to warrant a handoff (compression to long-term memory). + * buffer entries to warrant a handoff (compression to the Memory Ledger). */ export async function runHandoffResolution(session: SimSession): Promise { const worldState = session.coreRepo.loadWorldState(session.worldInstanceId); @@ -33,11 +33,39 @@ export async function runHandoffResolution(session: SimSession): Promise { maxContext, ); if (trigger !== "none") { - await handoffEngine.runHandoff( + const ran = await handoffEngine.runHandoff( entity, bufferEntries, worldState.clock.get(), ); + if (ran) { + const lastResult = handoffEngine.lastResult; + const lastCall = + session.handoffProvider.lastCalls?.[ + (session.handoffProvider.lastCalls?.length || 0) - 1 + ]; + const info = session.entities.find((e) => e.id === entity.id); + const entityName = info?.name || entity.id; + + session.log.push({ + turn: session.turn, + entityId: entity.id, + entityName, + narrativeProse: `Handoff triggered for ${entityName}: memories were transferred from Cognitive Buffer to Memory Ledger`, + intents: [], + timestamp: worldState.clock.get().toISOString(), + isHandoff: true, + rawPrompt: lastResult + ? { + systemPrompt: lastResult.systemPrompt || "", + userContext: lastResult.userContext || "", + components: lastResult.promptComponents, + } + : undefined, + usage: lastCall?.usage, + handoffResult: lastResult?.response || lastCall?.response, + }); + } } } } diff --git a/apps/gui/src/lib/simulation/simulation-manager.ts b/apps/gui/src/lib/simulation/simulation-manager.ts index a3f9c96..4400075 100644 --- a/apps/gui/src/lib/simulation/simulation-manager.ts +++ b/apps/gui/src/lib/simulation/simulation-manager.ts @@ -468,6 +468,21 @@ export class SimulationManager { // --------------------------------------------------------------------------- private snapshot(session: SimSession): SimSnapshot { + const worldState = session.coreRepo.loadWorldState(session.worldInstanceId); + const hydratedEntities = session.entities.map((e) => { + const actualEntity = worldState?.getEntity(e.id); + const aliases: Record = {}; + if (actualEntity) { + for (const [targetId, alias] of actualEntity.aliases.entries()) { + aliases[targetId] = alias; + } + } + return { + ...e, + aliases, + }; + }); + return { id: session.worldInstanceId, status: session.status, @@ -475,7 +490,7 @@ export class SimulationManager { maxTurns: session.maxTurns, scenarioName: session.scenarioName, scenarioDescription: session.scenarioDescription, - entities: session.entities, + entities: hydratedEntities, log: session.log, entityIndex: session.entityIndex, waitingEntity: session.waitingEntity, diff --git a/apps/gui/src/lib/simulation/turn-executor.ts b/apps/gui/src/lib/simulation/turn-executor.ts index 825ffc0..edaa2b9 100644 --- a/apps/gui/src/lib/simulation/turn-executor.ts +++ b/apps/gui/src/lib/simulation/turn-executor.ts @@ -49,17 +49,18 @@ async function processIntents( // eslint-disable-next-line @typescript-eslint/no-explicit-any worldState: any, session: SimSession, -): Promise { +): Promise<{ intentInfos: IntentInfo[]; validatorCalls: ValidatorCall[] }> { const intentInfos: IntentInfo[] = []; + const validatorCalls: ValidatorCall[] = []; - for (const intent of intents) { + for (let i = 0; i < intents.length; i++) { + const intent = intents[i]; const outcome = await session.architect.processIntent(worldState, intent); const ts = worldState.clock.get().toISOString(); intentInfos.push({ type: intent.type, - description: intent.description, - selfDescription: intent.selfDescription, + content: intent.content, modifiers: intent.modifiers || [], targetIds: intent.targetIds, isValid: outcome.isValid, @@ -67,6 +68,50 @@ async function processIntents( minutesToAdvance: outcome.timeDelta?.minutesToAdvance, }); + if (intent.type === "action" && session.architect.validator.lastResult) { + const lastResult = session.architect.validator.lastResult; + let usage = undefined; + if ( + session.validatorProvider.lastCalls && + session.validatorProvider.lastCalls.length > 0 + ) { + const valCall = + session.validatorProvider.lastCalls[ + session.validatorProvider.lastCalls.length - 1 + ]; + usage = valCall.usage; + } + + validatorCalls.push({ + intentIndex: i, + intentContent: intent.content, + prompt: { + systemPrompt: lastResult.systemPrompt || "", + userContext: lastResult.userContext || "", + components: lastResult.promptComponents, + }, + response: { + isValid: outcome.isValid, + reason: outcome.reason, + }, + usage, + }); + } else { + const reason = + intent.type === "dialogue" + ? "Dialogue intents represent verbal/communication actions and are automatically valid." + : "Monologue/thought intents represent internal reflections and bypass validation."; + + validatorCalls.push({ + intentIndex: i, + intentContent: intent.content, + response: { + isValid: true, + reason: outcome.reason || reason, + }, + }); + } + const actorEntry = buildBufferEntryForIntent(intent, ts, entity.locationId); if (intent.type === "action") { actorEntry.outcome = { isValid: outcome.isValid, reason: outcome.reason }; @@ -100,7 +145,7 @@ async function processIntents( } } - return intentInfos; + return { intentInfos, validatorCalls }; } // --------------------------------------------------------------------------- @@ -166,6 +211,11 @@ export async function processNpcTurn( narrativeProse: result.narrativeProse, intents: [], timestamp: worldState.clock.get().toISOString(), + rawPrompt: { + systemPrompt: result.systemPrompt || "", + userContext: result.userContext || "", + components: result.promptComponents, + }, }; if ( @@ -176,10 +226,6 @@ export async function processNpcTurn( session.actorProvider.lastCalls[ session.actorProvider.lastCalls.length - 1 ]; - entry.rawPrompt = { - systemPrompt: actorCall.systemPrompt, - userContext: actorCall.userContext, - }; entry.usage = actorCall.usage; } @@ -191,20 +237,49 @@ export async function processNpcTurn( session.decoderProvider.lastCalls[ session.decoderProvider.lastCalls.length - 1 ]; + const proseHeader = "=== NARRATIVE PROSE ==="; + const userContext = decoderCall.userContext; + const idx = userContext.indexOf(proseHeader); + + let contextStr = userContext; + let proseStr = ""; + + if (idx !== -1) { + contextStr = userContext.substring(0, idx).trim(); + proseStr = userContext.substring(idx).trim(); + } + entry.decoderPrompt = { systemPrompt: decoderCall.systemPrompt, userContext: decoderCall.userContext, + components: [ + { + label: "System Prompt", + type: "system", + content: decoderCall.systemPrompt, + }, + { label: "Decoder Context", type: "world", content: contextStr }, + { label: "Narrative Prose", type: "input", content: proseStr }, + ], }; entry.decoderUsage = decoderCall.usage; } - entry.intents = await processIntents( + const { intentInfos, validatorCalls } = await processIntents( result.intents.intents, info.id, entity, worldState, session, ); + entry.intents = intentInfos; + entry.validatorCalls = validatorCalls; + entry.decodedIntents = result.intents.intents.map((intent) => ({ + type: intent.type, + content: intent.content, + modifiers: intent.modifiers || [], + targetIds: intent.targetIds, + })); session.log.push(entry); session.coreRepo.saveWorldState(worldState); @@ -244,8 +319,9 @@ export async function executePlayerAction( intents: [], timestamp: worldState.clock.get().toISOString(), rawPrompt: { - systemPrompt: ctx.systemPrompt, - userContext: ctx.userContext, + systemPrompt: result.systemPrompt || ctx.systemPrompt, + userContext: result.userContext || ctx.userContext, + components: result.promptComponents, }, }; @@ -257,20 +333,45 @@ export async function executePlayerAction( session.decoderProvider.lastCalls[ session.decoderProvider.lastCalls.length - 1 ]; + const proseHeader = "=== NARRATIVE PROSE ==="; + const userContext = call.userContext; + const idx = userContext.indexOf(proseHeader); + + let contextStr = userContext; + let proseStr = ""; + + if (idx !== -1) { + contextStr = userContext.substring(0, idx).trim(); + proseStr = userContext.substring(idx).trim(); + } + entry.decoderPrompt = { systemPrompt: call.systemPrompt, userContext: call.userContext, + components: [ + { label: "System Prompt", type: "system", content: call.systemPrompt }, + { label: "Decoder Context", type: "world", content: contextStr }, + { label: "Narrative Prose", type: "input", content: proseStr }, + ], }; entry.decoderUsage = call.usage; } - entry.intents = await processIntents( + const { intentInfos, validatorCalls: playerValCalls } = await processIntents( result.intents.intents, ctx.entityId, entity, worldState, session, ); + entry.intents = intentInfos; + entry.validatorCalls = playerValCalls; + entry.decodedIntents = result.intents.intents.map((intent) => ({ + type: intent.type, + content: intent.content, + modifiers: intent.modifiers || [], + targetIds: intent.targetIds, + })); session.log.push(entry); session.coreRepo.saveWorldState(worldState); diff --git a/content/demo/scenarios/talking-room.json b/content/demo/scenarios/talking-room.json index bbec88d..1686dfc 100644 --- a/content/demo/scenarios/talking-room.json +++ b/content/demo/scenarios/talking-room.json @@ -72,16 +72,37 @@ ], "initialMemories": [ { - "id": "alpha-wake", - "timestamp": "2026-07-09T07:58:00.000Z", + "id": "ab3f29d2-cf11-4111-9a99-b13c126d123e", + "timestamp": "2026-07-01T07:58:00.000Z", "locationId": "white-room", "intent": { "type": "monologue", - "originalText": "I didn't have a choice. I would have been sent to jail if I hadn't agreed to do this experiement.", - "description": "", + "content": "I didn't have a choice. I would have been sent to jail if I hadn't agreed to do this experiment.", "actorId": "7c9b83b3-8cfb-4e89-8d77-626a5757d591", "targetIds": [] } + }, + { + "id": "10ak29d2-as11-9811-9a99-b13c126d123e", + "timestamp": "2026-07-09T06:00:00.000Z", + "locationId": "white-room", + "intent": { + "type": "action", + "content": "entity@7c9b83b3-8cfb-4e89-8d77-626a5757d591[I] woke up today in the room and saw entity@bf3f29d2-cf11-4b11-9a99-b13c126d400e[another man]!", + "actorId": "7c9b83b3-8cfb-4e89-8d77-626a5757d591", + "targetIds": ["bf3f29d2-cf11-4b11-9a99-b13c126d400e"] + } + }, + { + "id": "zz3f29d2-as11-9811-9a99-b13c126d123e", + "timestamp": "2026-07-09T07:58:00.000Z", + "locationId": "white-room", + "intent": { + "type": "action", + "content": "entity@bf3f29d2-cf11-4b11-9a99-b13c126d400e[I] wake up from my sleep.", + "actorId": "bf3f29d2-cf11-4b11-9a99-b13c126d400e", + "targetIds": [] + } } ] }, @@ -119,13 +140,12 @@ ], "initialMemories": [ { - "id": "beta-wake", - "timestamp": "2026-07-09T07:58:30.000Z", + "id": "zx1f29d2-cf11-4111-9a99-b13c126d123e", + "timestamp": "2026-07-09T07:58:00.000Z", "locationId": "white-room", "intent": { "type": "action", - "originalText": "Why can't I remember anything before the research agreement. It's like my memory was erased.", - "description": "", + "content": "entity@bf3f29d2-cf11-4b11-9a99-b13c126d400e[I] wake up in an unfamiliar place.", "actorId": "bf3f29d2-cf11-4b11-9a99-b13c126d400e", "targetIds": [] } diff --git a/docs b/docs index 51037cd..9f43cd9 120000 --- a/docs +++ b/docs @@ -1 +1 @@ -web/docs/src \ No newline at end of file +web/docs/src/content/docs \ No newline at end of file diff --git a/package.json b/package.json index 15ec7a7..21f3dac 100644 --- a/package.json +++ b/package.json @@ -56,6 +56,7 @@ "@langchain/openai": "^0.3.17", "@langchain/openrouter": "^0.4.3", "@types/node": "^20.19.43", + "compromise": "^14.16.0", "dotenv": "^17.4.2" } } diff --git a/packages/actor/package.json b/packages/actor/package.json index 1a8c11c..761dc1b 100644 --- a/packages/actor/package.json +++ b/packages/actor/package.json @@ -11,6 +11,7 @@ "@omnia/intent": "workspace:*", "@omnia/llm": "workspace:*", "@omnia/memory": "workspace:*", + "@omnia/voice": "workspace:*", "zod": "^4.4.3" } } diff --git a/packages/actor/src/actor-prompt-builder.ts b/packages/actor/src/actor-prompt-builder.ts index 81a93d8..ab4ff82 100644 --- a/packages/actor/src/actor-prompt-builder.ts +++ b/packages/actor/src/actor-prompt-builder.ts @@ -4,7 +4,6 @@ import { WorldState, naturalizeTime, serializeSubjectiveWorldState, - resolveAlias, } from "@omnia/core"; import { BufferEntry, @@ -13,6 +12,8 @@ import { LedgerEntry, LedgerRepository, } from "@omnia/memory"; +import { hydrate } from "@omnia/voice"; +import { PromptComponent, IPromptBuilder, PromptBreakdown } from "@omnia/llm"; /** * Zod schema for the structured response expected from the actor LLM. @@ -33,17 +34,19 @@ export type ActorResponse = z.infer; * * The prompt is strictly epistemically bounded: the entity only sees what * it is allowed to see (public attributes + private attributes explicitly - * ACL'd to it), its own recent memory buffer, and the entities co-located + * ACL'd to it), its own Cognitive Buffer, and the entities co-located * with it. System UUIDs are surfaced as subjective aliases. */ -export class ActorPromptBuilder { +export class ActorPromptBuilder implements IPromptBuilder< + [WorldState, Entity] +> { /** - * @param bufferRepo Used to fetch the actor's recent memory. Optional — + * @param bufferRepo Used to fetch the actor's Cognitive Buffer. Optional — * if absent, the memory section is omitted. - * @param ledgerRepo Used to fetch long-term memories. Optional. - * @param memoryLimit Maximum number of recent buffer entries to inject. + * @param ledgerRepo Used to fetch Memory Ledger entries. Optional. + * @param memoryLimit Maximum number of recent Cognitive Buffer entries to inject. * Defaults to 20. - * @param ledgerLimit Maximum number of long-term memories to retrieve. + * @param ledgerLimit Maximum number of Memory Ledger entries to retrieve. * Defaults to 5. */ constructor( @@ -56,13 +59,20 @@ export class ActorPromptBuilder { /** * Assembles the system prompt and user context for a given entity. */ - build( - worldState: WorldState, - entity: Entity, - ): { systemPrompt: string; userContext: string } { + /** + * Assembles the system prompt and user context for a given entity. + */ + /** + * Assembles the system prompt and user context for a given entity. + */ + build(worldState: WorldState, entity: Entity): PromptBreakdown { const systemPrompt = this.buildSystemPrompt(); - const userContext = this.buildUserContext(worldState, entity); - return { systemPrompt, userContext }; + const { userContext, components } = this.buildUserContext( + worldState, + entity, + systemPrompt, + ); + return { systemPrompt, userContext, components }; } private buildSystemPrompt(): string { @@ -76,30 +86,32 @@ Your output is a short block of narrative prose describing what your character d Guidelines: - Always write in the first person -- Only describe your character's own actions, spoken words, and internal reactions. Do NOT narrate or describe the environment or your surroundings, or other characters' actions. - Refer to other entities by the subjective names/aliases that you refer to them as. - Keep your prose vivid but concise. Write it in natural narrative order. - Not every response requires an outward action. It is perfectly valid to only think (a monologue) and do nothing perceivable. - Never speak or act on another entity's behalf. You only control your own character. - Stay strictly within what your character knows. Do not invent knowledge that doesn't exist or act on it. -- You are limited by just your memory. If your memory is limited, then that's all you can remember. If you do make stuff up then that's lying. Which is allowed, but remember that you're lying. +- You are limited by just your memory. If your memory is limited, then that's all you can remember. If you do make stuff up then that's lying. Which is allowed, but remember that you're lying +- Only describe your character's own actions, spoken words, and internal reactions. Do NOT narrate the environment or your surroundings, or other characters' actions. +- Be clear about who or what you are interacting with. ". `.trim(); } - private buildUserContext(worldState: WorldState, entity: Entity): string { - const sections: string[] = []; + private buildUserContext( + worldState: WorldState, + entity: Entity, + systemPrompt: string, + ): { + userContext: string; + components: PromptComponent[]; + } { const now = worldState.clock.get(); - // --- Subjective present time --- - sections.push( - `=== CURRENT MOMENT ===\nIt is ${now.toISOString()} right now.`, - ); - - // --- Subjective world state (self + perceived entities + co-location) --- - sections.push( - `=== THE WORLD AS YOU PERCEIVE IT ===\n${serializeSubjectiveWorldState(worldState, entity.id)}`, - ); + // --- Subjective present time & world state --- + const momentStr = `=== CURRENT MOMENT ===\nIt is ${now.toISOString()} right now.`; + const perceivedStr = `=== THE WORLD AS YOU PERCEIVE IT ===\n${serializeSubjectiveWorldState(worldState, entity.id)}`; + const worldInfo = `${momentStr}\n\n${perceivedStr}`; // Fetch recent buffer entries once let recentEntries: BufferEntry[] = []; @@ -111,27 +123,55 @@ Guidelines: } } - // --- Recent memory --- - const memorySection = this.buildMemorySection(entity, recentEntries, now); - if (memorySection) { - sections.push(memorySection); - } - - // --- Recalled Long-Term memory --- + // --- Recalled Memory Ledger --- const ledgerSection = this.buildLedgerSection( worldState, entity, recentEntries, now, ); - if (ledgerSection) { - sections.push(ledgerSection); + const memoryLedger = ledgerSection || ""; + + // --- Cognitive Buffer --- + const memorySection = this.buildCognitiveBufferSection( + entity, + recentEntries, + now, + ); + const cognitiveBuffer = memorySection || ""; + + // Assemble final user context + const parts: string[] = [worldInfo]; + if (memoryLedger) parts.push(memoryLedger); + if (cognitiveBuffer) parts.push(cognitiveBuffer); + const userContext = parts.join("\n\n"); + + const components: PromptComponent[] = [ + { label: "System Prompt", type: "system", content: systemPrompt }, + { label: "World Info", type: "world", content: worldInfo }, + ]; + if (memoryLedger) { + components.push({ + label: "Memory Ledger", + type: "memories", + content: memoryLedger, + }); + } + if (cognitiveBuffer) { + components.push({ + label: "Cognitive Buffer", + type: "events", + content: cognitiveBuffer, + }); } - return sections.join("\n\n"); + return { + userContext, + components, + }; } - private buildMemorySection( + private buildCognitiveBufferSection( entity: Entity, entries: BufferEntry[], now: Date, @@ -139,7 +179,7 @@ Guidelines: if (!this.bufferRepo) return null; if (entries.length === 0) { - return `=== RECENT EVENTS ===\n(No recent events recorded.)`; + return `=== COGNITIVE BUFFER ===\n(No entries recorded.)`; } const recent = entries.slice(-this.memoryLimit); @@ -147,9 +187,16 @@ Guidelines: let currentGroup: string | null = null; for (const entry of recent) { - const serialized = serializeSubjectiveBufferEntry(entry, entity); + let serialized = serializeSubjectiveBufferEntry(entry, entity); const when = naturalizeTime(now, new Date(entry.timestamp)); + if ( + entry.intent.actorId === entity.id && + entry.intent.type === "dialogue" + ) { + serialized = `I said: ${serialized}`; + } + if (when !== currentGroup) { currentGroup = when; const header = when.charAt(0).toUpperCase() + when.slice(1); @@ -159,7 +206,7 @@ Guidelines: groupedLines.push(` - ${serialized}`); } - return `=== RECENT EVENTS ===\n${groupedLines.join("\n")}`; + return `=== COGNITIVE BUFFER ===\n${groupedLines.join("\n")}`; } private buildLedgerSection( @@ -239,12 +286,7 @@ Guidelines: for (const entry of recalled) { const when = naturalizeTime(now, new Date(entry.timestamp)); - let content = entry.content; - // Resolve system IDs to subjective aliases in the content - for (const targetId of entry.involvedEntityIds) { - const alias = resolveAlias(entity, targetId); - content = content.replace(new RegExp(targetId, "g"), alias); - } + let content = hydrate(entry.content, entity); if (entry.locationId) { content += ` (at ${entry.locationId})`; } @@ -263,6 +305,6 @@ Guidelines: } } - return `=== YOUR MEMORIES ===\n${groupedLines.join("\n")}`; + return `=== MEMORY LEDGER ===\n${groupedLines.join("\n")}`; } } diff --git a/packages/actor/src/actor.ts b/packages/actor/src/actor.ts index a7528b7..405007b 100644 --- a/packages/actor/src/actor.ts +++ b/packages/actor/src/actor.ts @@ -1,5 +1,5 @@ import { Entity, WorldState } from "@omnia/core"; -import { ILLMProvider } from "@omnia/llm"; +import { ILLMProvider, PromptComponent } from "@omnia/llm"; import { BufferEntry, BufferRepository, LedgerRepository } from "@omnia/memory"; import { Intent, IntentDecoder, IntentSequence } from "@omnia/intent"; import { @@ -54,6 +54,9 @@ export interface ActorTurnResult { narrativeProse: string; /** The decoded intent sequence (split/classified from the prose). */ intents: IntentSequence; + systemPrompt?: string; + userContext?: string; + promptComponents?: PromptComponent[]; } /** @@ -76,7 +79,7 @@ export class ActorAgent { constructor( llmProvider: ILLMProvider | { actor: ILLMProvider; decoder: ILLMProvider }, - bufferRepo?: BufferRepository, + private bufferRepo?: BufferRepository, ledgerRepo?: LedgerRepository, memoryLimit?: number, generator?: IActorProseGenerator, @@ -116,7 +119,7 @@ export class ActorAgent { ); } - const { systemPrompt, userContext } = this.promptBuilder.build( + const { systemPrompt, userContext, components } = this.promptBuilder.build( worldState, entity, ); @@ -127,15 +130,27 @@ export class ActorAgent { userContext, ); + const recentEntries = this.bufferRepo + ? this.bufferRepo.listForOwner(entity.id) + : []; + const recentIntents = recentEntries + .filter((e) => e.intent.actorId !== entity.id) + .slice(-3) + .map((e) => e.intent); + const intents = await this.decoder.decode( worldState, entity.id, narrativeProse, + recentIntents, ); return { narrativeProse, intents, + systemPrompt, + userContext, + promptComponents: components, }; } } diff --git a/packages/actor/tests/actor-prompt-builder.test.ts b/packages/actor/tests/actor-prompt-builder.test.ts index 1c9a7c4..437a250 100644 --- a/packages/actor/tests/actor-prompt-builder.test.ts +++ b/packages/actor/tests/actor-prompt-builder.test.ts @@ -4,7 +4,7 @@ import { WorldState, Entity } from "@omnia/core"; import { BufferRepository, LedgerRepository } from "@omnia/memory"; import { ActorPromptBuilder } from "../src/actor-prompt-builder"; -describe("ActorPromptBuilder with Long-Term Memory Integration", () => { +describe("ActorPromptBuilder with Memory Ledger Integration", () => { let db: Database.Database; let bufferRepo: BufferRepository; let ledgerRepo: LedgerRepository; @@ -31,7 +31,7 @@ describe("ActorPromptBuilder with Long-Term Memory Integration", () => { db.close(); }); - it("should inject both recent memory and recalled long-term memory with subjective aliases resolved", () => { + it("should inject both Cognitive Buffer and recalled Memory Ledger entries with subjective aliases resolved", () => { const world = new WorldState( "world-123", new Date("2024-01-10T12:00:00.000Z"), @@ -55,19 +55,19 @@ describe("ActorPromptBuilder with Long-Term Memory Integration", () => { type: "dialogue", actorId: "alice", targetIds: ["bob"], - originalText: "Hello there", - description: "Alice greets Bob", + content: "entity@alice[I] say 'Hello there' to entity@bob[Bob]", + modifiers: [], }, }); - // 2. Populate ledger repository (long-term memory) + // 2. Populate ledger repository (Memory Ledger) ledgerRepo.save({ id: "ledger1", ownerId: "alice", timestamp: "2024-01-08T12:00:00.000Z", // 2 days ago locationId: "tavern", involvedEntityIds: ["bob"], - content: "alice met bob at the tavern.", + content: "entity@alice[Alice] met entity@bob[bob] at the tavern.", quotes: ["I am a ranger."], importance: 9, embedding: [], @@ -76,14 +76,14 @@ describe("ActorPromptBuilder with Long-Term Memory Integration", () => { const builder = new ActorPromptBuilder(bufferRepo, ledgerRepo, 20, 5); const { userContext } = builder.build(world, alice); - // Check recent memory exists - expect(userContext).toContain("=== RECENT EVENTS ==="); - expect(userContext).toContain("Alice greets Bob"); + // Check Cognitive Buffer exists + expect(userContext).toContain("=== COGNITIVE BUFFER ==="); + expect(userContext).toContain("I said: I say 'Hello there' to Strider"); - // Check long-term memory exists - expect(userContext).toContain("=== YOUR MEMORIES ==="); - // Bob should be resolved to Strider in the ledger content - expect(userContext).toContain("alice met Strider at the tavern."); + // Check Memory Ledger exists + expect(userContext).toContain("=== MEMORY LEDGER ==="); + // Bob should be resolved to Strider, and alice to I in the ledger content + expect(userContext).toContain("I met Strider at the tavern."); expect(userContext).toContain('Quote: "I am a ranger."'); }); @@ -98,7 +98,7 @@ describe("ActorPromptBuilder with Long-Term Memory Integration", () => { const builder = new ActorPromptBuilder(bufferRepo, ledgerRepo, 20, 5); const { userContext } = builder.build(world, alice); - expect(userContext).toContain("=== RECENT EVENTS ==="); - expect(userContext).not.toContain("=== YOUR MEMORIES ==="); + expect(userContext).toContain("=== COGNITIVE BUFFER ==="); + expect(userContext).not.toContain("=== MEMORY LEDGER ==="); }); }); diff --git a/packages/actor/tsconfig.json b/packages/actor/tsconfig.json index 2ac04d2..7dd4d9a 100644 --- a/packages/actor/tsconfig.json +++ b/packages/actor/tsconfig.json @@ -9,6 +9,7 @@ { "path": "../core" }, { "path": "../intent" }, { "path": "../llm" }, - { "path": "../memory" } + { "path": "../memory" }, + { "path": "../voice" } ] } diff --git a/packages/architect/package.json b/packages/architect/package.json index 4b815a5..2bf2e71 100644 --- a/packages/architect/package.json +++ b/packages/architect/package.json @@ -10,6 +10,7 @@ "@omnia/core": "workspace:*", "@omnia/llm": "workspace:*", "@omnia/intent": "workspace:*", + "@omnia/voice": "workspace:*", "zod": "^4.4.3" } } diff --git a/packages/architect/src/architect.ts b/packages/architect/src/architect.ts index 56234fc..22f8331 100644 --- a/packages/architect/src/architect.ts +++ b/packages/architect/src/architect.ts @@ -9,7 +9,7 @@ export interface ProcessResult extends ValidationResult { } export class Architect { - private validator: LLMValidator; + public validator: LLMValidator; private timeDeltaGenerator: TimeDeltaGenerator; constructor( @@ -47,22 +47,22 @@ export class Architect { * Processes, validates, generates deltas, applies them to the world state, * and persists the changes to the database. * - * "monologue" intents are internal thoughts — they bypass validation and + * "monologue" and "thought" intents are internal thoughts — they bypass validation and * time-delta generation entirely: the clock does not advance, the world * state is not mutated or persisted. The caller is responsible for writing - * the monologue to the actor's memory buffer. + * the monologue/thought to the actor's Cognitive Buffer. */ async processIntent( worldState: WorldState, intent: Intent, ): Promise { - // 0. Monologue intents are purely internal — short-circuit before any + // 0. Monologue/thought intents are purely internal — short-circuit before any // validation or world mutation. - if (intent.type === "monologue") { + if (intent.type === "monologue" || intent.type === "thought") { return { isValid: true, reason: - "Monologue intent bypasses validation (internal thought, not perceivable).", + "Monologue/thought intent bypasses validation (internal thought, not perceivable).", timeDelta: { minutesToAdvance: 0, explanation: "Internal thought — no time elapsed.", diff --git a/packages/architect/src/delta.ts b/packages/architect/src/delta.ts index bb6666e..7339e60 100644 --- a/packages/architect/src/delta.ts +++ b/packages/architect/src/delta.ts @@ -2,6 +2,7 @@ import { z } from "zod"; import { WorldState, serializeObjectiveWorldState } from "@omnia/core"; import { ILLMProvider } from "@omnia/llm"; import { Intent } from "@omnia/intent"; +import { hydrateObjective } from "@omnia/voice"; export const TimeDeltaSchema = z.object({ minutesToAdvance: z.number().int().nonnegative(), @@ -26,10 +27,11 @@ export class TimeDeltaGenerator implements IDeltaGenerator { explanation: "Dialogue action; 1 minute granted for quick exchange.", }; } - if (intent.type === "monologue") { + if (intent.type === "monologue" || intent.type === "thought") { return { minutesToAdvance: 0, - explanation: "Monologue action; no time advanced for internal thought.", + explanation: + "Monologue/thought action; no time advanced for internal thought.", }; } @@ -45,6 +47,8 @@ Return a structured JSON object containing: - "explanation": a brief explanation of why this amount of time is appropriate. `.trim(); + const objectiveContent = hydrateObjective(intent.content, worldState); + const userContext = ` === CURRENT WORLD STATE === Current Time: ${worldState.clock.get().toISOString()} @@ -54,8 +58,7 @@ ${serializeObjectiveWorldState(worldState)} === ACTION === Actor ID: ${intent.actorId} Type: ${intent.type} -Description: "${intent.description}" -Original Text: "${intent.originalText}" +Content: "${objectiveContent}" Target IDs: ${intent.targetIds.join(", ") || "(None)"} `.trim(); diff --git a/packages/architect/src/index.ts b/packages/architect/src/index.ts index 19e95a8..d5ad0bd 100644 --- a/packages/architect/src/index.ts +++ b/packages/architect/src/index.ts @@ -1,3 +1,4 @@ export * from "./llm-validator.js"; +export * from "./llm-validator-prompt-builder.js"; export * from "./architect.js"; export * from "./delta.js"; diff --git a/packages/architect/src/llm-validator-prompt-builder.ts b/packages/architect/src/llm-validator-prompt-builder.ts new file mode 100644 index 0000000..733a348 --- /dev/null +++ b/packages/architect/src/llm-validator-prompt-builder.ts @@ -0,0 +1,59 @@ +import { WorldState, serializeObjectiveWorldState } from "@omnia/core"; +import { Intent } from "@omnia/intent"; +import { hydrateObjective } from "@omnia/voice"; +import { PromptBreakdown, PromptComponent, IPromptBuilder } from "@omnia/llm"; + +/** + * Prompt builder for the LLM Validator (World Architect). + * Separates prompt generation, structure, and component breakdowns. + */ +export class LLMValidatorPromptBuilder implements IPromptBuilder< + [WorldState, Intent] +> { + build(worldState: WorldState, intent: Intent): PromptBreakdown { + const serializedWorld = serializeObjectiveWorldState(worldState); + + const systemPrompt = ` +You are the World Architect, a deterministic and objective judge of reality, physics, and narration for a simulation game. +Your task is to judge whether a proposed action (Intent) by an actor is physically and logically possible given the current objective state of the world. +Exempt dialogue or speech actions from validation (consider them always valid). +Enforce logical boundaries such as: +- Spatial boundaries (an actor cannot grab an object in another location unless they are there). +- Physical boundaries (an actor cannot open a locked drawer without a key or breaking it). +- State Boundaries (an actor cannot perform a task if their state doesn't allow them to do so). +- State/Attribute constraints. +- An actor can perform actions on themselves as long as it follows the boundaries stated above. + +You must respond with a JSON object containing: +- "isValid": boolean indicating if the action is possible/allowed. +- "reason": a very short explanation of why the action is allowed or denied. +`.trim(); + + const objectiveContent = hydrateObjective(intent.content, worldState); + + const worldStateSection = `=== CURRENT WORLD STATE ===\nCurrent Time: ${worldState.clock.get().toISOString()}\nEntities & Attributes:\n${serializedWorld}`; + const proposedActionSection = `=== PROPOSED ACTION ===\nActor ID: ${intent.actorId}\nType: ${intent.type}\nContent: "${objectiveContent}"\nTarget IDs: ${intent.targetIds.join(", ") || "(None)"}`; + + const userContext = `${worldStateSection}\n\n${proposedActionSection}\n\nDecide if the proposed action is logically valid and physically possible.`; + + const components: PromptComponent[] = [ + { label: "System Prompt", type: "system", content: systemPrompt }, + { + label: "Current World State", + type: "world", + content: worldStateSection, + }, + { + label: "Proposed Action", + type: "input", + content: proposedActionSection, + }, + ]; + + return { + systemPrompt, + userContext, + components, + }; + } +} diff --git a/packages/architect/src/llm-validator.ts b/packages/architect/src/llm-validator.ts index 0262d54..893c0db 100644 --- a/packages/architect/src/llm-validator.ts +++ b/packages/architect/src/llm-validator.ts @@ -1,7 +1,8 @@ import { z } from "zod"; -import { WorldState, serializeObjectiveWorldState } from "@omnia/core"; -import { ILLMProvider } from "@omnia/llm"; +import { WorldState } from "@omnia/core"; +import { ILLMProvider, PromptBreakdown } from "@omnia/llm"; import { Intent } from "@omnia/intent"; +import { LLMValidatorPromptBuilder } from "./llm-validator-prompt-builder.js"; export const ValidationResultSchema = z.object({ isValid: z.boolean(), @@ -11,12 +12,17 @@ export const ValidationResultSchema = z.object({ export type ValidationResult = z.infer; export class LLMValidator { - constructor(private llmProvider: ILLMProvider) {} + public lastResult: PromptBreakdown | null = null; + private promptBuilder: LLMValidatorPromptBuilder; + + constructor(private llmProvider: ILLMProvider) { + this.promptBuilder = new LLMValidatorPromptBuilder(); + } /** * Validates an action intent against the objective world state. * - * "monologue" intents must never reach this validator — they are internal + * "monologue" and "thought" intents must never reach this validator — they are internal * thoughts that bypass validation entirely (see Architect.processIntent). * This guard exists as a defensive safeguard. */ @@ -24,12 +30,14 @@ export class LLMValidator { worldState: WorldState, intent: Intent, ): Promise { - // Defensive guard: monologue intents bypass validation. - if (intent.type === "monologue") { + this.lastResult = null; + + // Defensive guard: monologue and thought intents bypass validation. + if (intent.type === "monologue" || intent.type === "thought") { return { isValid: true, reason: - "Monologue intents are internal thoughts and bypass validation.", + "Monologue/thought intents are internal thoughts and bypass validation.", }; } @@ -41,40 +49,16 @@ export class LLMValidator { }; } - // 1. Serialize the objective world state for the LLM - const serializedWorld = serializeObjectiveWorldState(worldState); + const { systemPrompt, userContext, components } = this.promptBuilder.build( + worldState, + intent, + ); - // 2. Build the prompts - const systemPrompt = ` -You are the World Architect, a deterministic and objective judge of reality, physics, and narration for a simulation game. -Your task is to judge whether a proposed action (Intent) by an actor is physically and logically possible given the current objective state of the world. -Exempt dialogue or speech actions from validation (consider them always valid). -Enforce logical boundaries such as: -- Spatial boundaries (an actor cannot grab an object in another location unless they are there). -- Physical boundaries (an actor cannot open a locked drawer without a key or breaking it). -- State Boundaries (an actor cannot perform a task if their state doesn't allow them to do so). -- State/Attribute constraints. - -You must respond with a JSON object containing: -- "isValid": boolean indicating if the action is possible/allowed. -- "reason": a concise explanation of why the action is allowed or denied. -`.trim(); - - const userContext = ` -=== CURRENT WORLD STATE === -Current Time: ${worldState.clock.get().toISOString()} -Entities & Attributes: -${serializedWorld} - -=== PROPOSED ACTION === -Actor ID: ${intent.actorId} -Type: ${intent.type} -Description: "${intent.description}" -Original Text: "${intent.originalText}" -Target IDs: ${intent.targetIds.join(", ") || "(None)"} - -Decide if the proposed action is logically valid and physically possible. -`.trim(); + this.lastResult = { + systemPrompt, + userContext, + components, + }; // structured call via the LLM provider const response = await this.llmProvider.generateStructuredResponse({ diff --git a/packages/architect/tests/architect.test.ts b/packages/architect/tests/architect.test.ts index a3cc481..149020f 100644 --- a/packages/architect/tests/architect.test.ts +++ b/packages/architect/tests/architect.test.ts @@ -1,5 +1,5 @@ -import { describe, test, expect } from "vitest"; import Database from "better-sqlite3"; +import { describe, test, expect } from "vitest"; import { WorldState, Entity, @@ -7,15 +7,16 @@ import { AttributeVisibility, } from "@omnia/core"; import { MockLLMProvider } from "@omnia/llm"; -import { Architect, AliasDeltaGenerator } from "@omnia/architect"; import { Intent } from "@omnia/intent"; +import { Architect, AliasDeltaGenerator } from "../src/index.js"; -describe("Architect & LLMValidator Unit Tests (Tier 1)", () => { - test("returns valid response when LLM validates intent as successful", async () => { +describe("World Architect Validation Tests (Tier 1)", () => { + test("returns valid response when LLM confirms the intent", async () => { const world = new WorldState("world-1"); const alice = new Entity("alice"); world.addEntity(alice); + // Setup mock LLM response const mockResponse = { isValid: true, reason: "Alice is in the room and the chest is unlocked.", @@ -25,9 +26,7 @@ describe("Architect & LLMValidator Unit Tests (Tier 1)", () => { const intent: Intent = { type: "action", - originalText: "open the chest and read the scroll", - description: "Open the chest and read the scroll", - selfDescription: "You open the chest and read the scroll.", + content: "entity@alice[I] open the chest and read the scroll", actorId: "alice", targetIds: [], modifiers: [], @@ -56,9 +55,7 @@ describe("Architect & LLMValidator Unit Tests (Tier 1)", () => { const intent: Intent = { type: "action", - originalText: "unlock the gate and escape", - description: "Unlock the gate and escape", - selfDescription: "You unlock the gate and escape.", + content: "entity@bob[I] unlock the gate and escape", actorId: "bob", targetIds: [], modifiers: [], @@ -79,9 +76,7 @@ describe("Architect & LLMValidator Unit Tests (Tier 1)", () => { const intent: Intent = { type: "action", - originalText: "haunt the mansion", - description: "Haunt the mansion", - selfDescription: "You haunt the mansion.", + content: "entity@ghost[I] haunt the mansion", actorId: "ghost", targetIds: [], modifiers: [], @@ -128,9 +123,7 @@ describe("TimeDeltaGenerator & Architect.processIntent Unit Tests (Tier 1)", () const intent: Intent = { type: "action", - originalText: "pick the lock of the wooden chest", - description: "Pick the lock of the wooden chest", - selfDescription: "You pick the lock of the wooden chest.", + content: "entity@alice[I] pick the lock of the wooden chest", actorId: "alice", targetIds: [], modifiers: [], @@ -177,9 +170,7 @@ describe("TimeDeltaGenerator & Architect.processIntent Unit Tests (Tier 1)", () const intent: Intent = { type: "action", - originalText: "run away", - description: "Run away", - selfDescription: "You run away.", + content: "entity@bob[I] run away", actorId: "bob", targetIds: [], modifiers: [], diff --git a/packages/architect/tsconfig.json b/packages/architect/tsconfig.json index 66a91d3..f11c151 100644 --- a/packages/architect/tsconfig.json +++ b/packages/architect/tsconfig.json @@ -8,6 +8,7 @@ "references": [ { "path": "../core" }, { "path": "../llm" }, - { "path": "../intent" } + { "path": "../intent" }, + { "path": "../voice" } ] } diff --git a/packages/intent/package.json b/packages/intent/package.json index 095cf86..3e0a9c7 100644 --- a/packages/intent/package.json +++ b/packages/intent/package.json @@ -9,6 +9,7 @@ "dependencies": { "@omnia/core": "workspace:*", "@omnia/llm": "workspace:*", + "@omnia/voice": "workspace:*", "zod": "^4.4.3" } } diff --git a/packages/intent/src/intent-decoder.ts b/packages/intent/src/intent-decoder.ts index 4c73b5c..b528d0a 100644 --- a/packages/intent/src/intent-decoder.ts +++ b/packages/intent/src/intent-decoder.ts @@ -1,73 +1,34 @@ import { WorldState } from "@omnia/core"; import { ILLMProvider } from "@omnia/llm"; -import { IntentSequence, LLMIntentSequenceSchema } from "./intent.js"; +import { dehydrate, expandContractions } from "@omnia/voice"; +import { Intent, IntentSequence, LLMIntentSequenceSchema } from "./intent.js"; +import { IntentDecoderPromptBuilder } from "./intent-prompt-builder.js"; export class IntentDecoder { - constructor(private llmProvider: ILLMProvider) {} + private promptBuilder: IntentDecoderPromptBuilder; + + constructor(private llmProvider: ILLMProvider) { + this.promptBuilder = new IntentDecoderPromptBuilder(); + } /** * Decodes narrative prose into an ordered sequence of structured intents. - * - * Responsibilities (from docs/intents.md): - * - Split prose into multiple intents when applicable. - * - Classify each intent as "dialogue", "action", or "monologue". - * - Parse narrative text into structured JSON with minimal information loss. - * - Contextually resolve receiving parties (targets). */ async decode( worldState: WorldState, actorId: string, narrativeProse: string, + recentIntents: Intent[] = [], ): Promise { - const entityIds = Array.from(worldState.entities.keys()); + const processedProse = expandContractions(narrativeProse); const actor = worldState.getEntity(actorId); - const aliasEntries = actor ? Array.from(actor.aliases.entries()) : []; - const aliasContext = - aliasEntries.length > 0 - ? aliasEntries - .map( - ([targetId, alias]) => - `- "${alias}" refers to entity ID: "${targetId}"`, - ) - .join("\n") - : "(No known aliases)"; - - const systemPrompt = ` -You are the Intent Decoder for a narrative simulation engine. -Your job is to take a block of narrative prose written by an actor agent and decompose it into an ordered sequence of discrete intents. - -For each intent you must: -1. Classify its type: - - "dialogue": if actor speaking, talking, whispering, murmuring, etc - - "action": Any physical or logical action performed in the world (e.g., moving, opening, looking). - - "monologue": An inner thought, reflection, or internal monologue/self narration. -2. Extract the original text fragment from the prose that corresponds to this intent. -3. Populate "description" and "selfDescription": - - "description": No subject or name — a bare third-person verb phrase only (e.g. "clears their throat", "shakes their head slowly") - - "selfDescription": The same event from the actor's own perspective, second person, complete sentence starting with "You" (e.g. "You clear your throat.", "You shake your head slowly."). This is shown directly in the actor's own memory — it must never say "the actor" or refer to them in the third person. - - In case of a dialogue, the description and self Description only stores the exact words said by the entity. (e.g. "I will do that later", "Are you serious right now?") -4. Identify targetIds — the entity IDs of the receiving parties. Use the "KNOWN ENTITY IDS" mapping to resolve any subjective names,or aliases used in the prose to their correct system entity IDs. If no specific target, use an empty array. -5. Identify modifiers — a list of strings representing additional qualities or modifiers extracted from the narrative prose. This includes emotions, tone of voice, speed, manner of action, or statement type (e.g., "question", "anxious", "whispering", "slowly", "quietly", "forcefully"). If no modifiers are present, use an empty array. -`.trim(); - - const userContext = ` -=== KNOWN ENTITY IDS === -${entityIds.length > 0 ? entityIds.join(", ") : "(No entities)"} - -=== ACTOR ALIASES === -The actor refers to other entities using these subjective names/aliases: -${aliasContext} - -=== WORLD STATE === -${serializeSimplifiedWorldState(worldState)} - -=== ACTOR === -Actor ID: ${actorId} - -=== NARRATIVE PROSE === -${narrativeProse} -`.trim(); + const { systemPrompt, userContext, components } = this.promptBuilder.build( + worldState, + actorId, + processedProse, + recentIntents, + ); const response = await this.llmProvider.generateStructuredResponse({ systemPrompt, @@ -81,42 +42,32 @@ ${narrativeProse} ); } - const fullIntents = response.data.intents.map((intent) => ({ - ...intent, - actorId, - })); + const aliasMap: Record = {}; + if (actor) { + for (const [targetId, alias] of actor.aliases.entries()) { + aliasMap[alias] = targetId; + } + } + + const fullIntents = response.data.intents.map((intent) => { + const dehydrated = dehydrate( + intent.content, + actorId, + intent.targetIds, + aliasMap, + ); + return { + ...intent, + content: dehydrated, + actorId, + }; + }); return { intents: fullIntents, + systemPrompt, + userContext, + promptComponents: components, }; } } - -function serializeSimplifiedWorldState(worldState: WorldState): string { - const lines: string[] = []; - - lines.push("Locations:"); - if (worldState.locations.size > 0) { - for (const loc of worldState.locations.values()) { - const parentId = (loc as { parentId?: string | null }).parentId; - const parentStr = parentId ? ` (Parent: ${parentId})` : ""; - lines.push(` - Location [ID: ${loc.id}]${parentStr}`); - } - } else { - lines.push(" (No locations)"); - } - - lines.push("Entities:"); - if (worldState.entities.size > 0) { - for (const entity of worldState.entities.values()) { - const locStr = entity.locationId - ? ` (Location: ${entity.locationId})` - : ""; - lines.push(` - Entity [ID: ${entity.id}]${locStr}`); - } - } else { - lines.push(" (No entities)"); - } - - return lines.join("\n"); -} diff --git a/packages/intent/src/intent-prompt-builder.ts b/packages/intent/src/intent-prompt-builder.ts new file mode 100644 index 0000000..316edb7 --- /dev/null +++ b/packages/intent/src/intent-prompt-builder.ts @@ -0,0 +1,102 @@ +import { WorldState, resolveAlias } from "@omnia/core"; +import { PromptBreakdown, PromptComponent, IPromptBuilder } from "@omnia/llm"; +import { Intent } from "./intent.js"; + +/** + * Prompt builder for the Intent Decoder. + * Separates prompt generation, structure, and component breakdowns. + */ +export class IntentDecoderPromptBuilder implements IPromptBuilder< + [WorldState, string, string, Intent[]] +> { + build( + worldState: WorldState, + actorId: string, + processedProse: string, + recentIntents: Intent[], + ): PromptBreakdown { + const actor = worldState.getEntity(actorId); + + // 1. Get other entities co-located in the same context + const otherEntitiesLines: string[] = []; + for (const otherEntity of worldState.entities.values()) { + if ( + otherEntity.id !== actorId && + otherEntity.locationId === actor?.locationId + ) { + const alias = actor + ? resolveAlias(actor, otherEntity.id) + : otherEntity.id; + otherEntitiesLines.push(` - Alias="${alias}" ID=${otherEntity.id}`); + } + } + const otherEntitiesContext = + otherEntitiesLines.length > 0 + ? otherEntitiesLines.join("\n") + : " (No other entities in context)"; + + // 2. Format historical context (2-3 recent intents received by the actor) + const historicalLines: string[] = []; + for (const prior of recentIntents) { + const targetIds = + prior.actorId !== actorId ? [prior.actorId] : prior.targetIds; + const targetsStr = targetIds + .map((tid) => { + const alias = actor ? resolveAlias(actor, tid) : tid; + return `(Alias="${alias}", ID="${tid}")`; + }) + .join(", "); + historicalLines.push( + ` - Content: "${prior.content}", Type: ${prior.type}, Target Entities: ${targetsStr || "None"}`, + ); + } + const historicalContext = + historicalLines.length > 0 + ? historicalLines.join("\n") + : " (No prior intents in context)"; + + const systemPrompt = ` +You are the Intent Decoder for a narrative simulation engine. +Your job is to take a block of narrative prose written by an actor agent and decompose it into an ordered sequence of discrete intents. + +For each intent you must: +1. Classify its type: + - "dialogue": if actor speaking, talking, whispering, murmuring, etc + - "action": Any physical action performed in the world (e.g., moving, opening, looking). DO NOT CLASSIFY SPEAKING MODIFIERS AS ACTIONS + - "monologue" (or "thought"): An inner thought, reflection, or monologue/self narration. +2. Extract the original narrative text fragment from the prose that corresponds to this intent and populate it as "content". Do not paraphrase, do not convert to third person, and do not convert to second person. Keep the original text fragment exactly as written in the prose (first-person voice). +3. Identify targetIds — the entity IDs of the receiving parties. Use the "Other entities in context" list to resolve any subjective names, aliases, or descriptions used in the prose to their correct entity IDs. If no specific target, use an empty array. +4. Identify modifiers — a list of strings representing additional qualities or modifiers extracted from the narrative prose. This includes emotions, tone of voice, speed, manner of action, or statement type (e.g., "question", "anxious", "whispering", "slowly", "quietly", "forcefully"). If no modifiers are present, use an empty array. +5. For dialogue intents always use the following format for content field: + I say "" (optionally: to him/her/alias). +`.trim(); + + const decoderContext = ` +Intent Source: ${actorId} +Other entities in context: +${otherEntitiesContext} +Historical Context: +${historicalContext} +`.trim(); + + const narrativeProseSection = `=== NARRATIVE PROSE ===\n${processedProse}`; + + const userContext = `${decoderContext}\n\n${narrativeProseSection}`; + + const components: PromptComponent[] = [ + { label: "System Prompt", type: "system", content: systemPrompt }, + { label: "Decoder Context", type: "world", content: decoderContext }, + { + label: "Narrative Prose", + type: "input", + content: narrativeProseSection, + }, + ]; + + return { + systemPrompt, + userContext, + components, + }; + } +} diff --git a/packages/intent/src/intent.ts b/packages/intent/src/intent.ts index dc42974..aa43b7e 100644 --- a/packages/intent/src/intent.ts +++ b/packages/intent/src/intent.ts @@ -6,9 +6,15 @@ import { z } from "zod"; * - "action": A physical or logical action performed in the world. * - "monologue": An inner thought or internal monologue. Not perceivable by * any other entity. Bypasses the Architect/validators entirely and is - * written directly to the actor's memory buffer with no outcome. + * written directly to the actor's Cognitive Buffer with no outcome. + * - "thought": Equivalent/alias to "monologue". */ -export const IntentTypeSchema = z.enum(["dialogue", "action", "monologue"]); +export const IntentTypeSchema = z.enum([ + "dialogue", + "action", + "monologue", + "thought", +]); export type IntentType = z.infer; /** @@ -18,19 +24,13 @@ export const LLMIntentSchema = z.object({ /** The type of intent. */ type: IntentTypeSchema, - /** The original narrative text fragment this intent was extracted from. */ - originalText: z.string(), - - /** A concise, structured description of the intent's action or dialogue. */ - description: z.string(), - - /** The same event from the actor's own perspective (second person, "You"). */ - selfDescription: z.string(), + /** The dehydrated canonical content of the intent. */ + content: z.string(), /** * Entity IDs of the receiving parties (e.g., who is being spoken to, * what object is being interacted with). Always an empty array for - * "monologue" intents, since they are not perceivable by anyone. + * "monologue" and "thought" intents, since they are not perceivable by anyone. */ targetIds: z.array(z.string()), @@ -56,8 +56,14 @@ export const LLMIntentSequenceSchema = z.object({ * The full output of the Intent Decoder: an ordered sequence of intents * extracted from a single narrative prose block. */ +import { PromptComponent } from "@omnia/llm"; + export const IntentSequenceSchema = z.object({ intents: z.array(IntentSchema), }); -export type IntentSequence = z.infer; +export type IntentSequence = z.infer & { + systemPrompt?: string; + userContext?: string; + promptComponents?: PromptComponent[]; +}; diff --git a/packages/intent/tests/intent.test.ts b/packages/intent/tests/intent.test.ts index 162b327..88a1e88 100644 --- a/packages/intent/tests/intent.test.ts +++ b/packages/intent/tests/intent.test.ts @@ -1,7 +1,7 @@ import { describe, test, expect } from "vitest"; import { WorldState, Entity } from "@omnia/core"; import { MockLLMProvider } from "@omnia/llm"; -import { IntentDecoder, IntentSequence } from "@omnia/intent"; +import { IntentDecoder } from "@omnia/intent"; describe("IntentDecoder Unit Tests (Tier 1)", () => { test("decodes prose with a single action intent", async () => { @@ -9,13 +9,11 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { const alice = new Entity("alice"); world.addEntity(alice); - const mockResponse: IntentSequence = { + const mockResponse = { intents: [ { type: "action", - originalText: "Alice opened the chest.", - description: "Open the wooden chest.", - selfDescription: "You open the wooden chest.", + content: "I open the wooden chest.", targetIds: [], modifiers: [], }, @@ -34,6 +32,7 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { expect(result.intents).toHaveLength(1); expect(result.intents[0].type).toBe("action"); expect(result.intents[0].actorId).toBe("alice"); + expect(result.intents[0].content).toContain("entity@alice[I]"); expect(result.intents[0].targetIds).toEqual([]); }); @@ -44,13 +43,11 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { world.addEntity(alice); world.addEntity(bob); - const mockResponse: IntentSequence = { + const mockResponse = { intents: [ { type: "dialogue", - originalText: '"Do you have the key?" Alice asked Bob.', - description: "Alice asks Bob if he has the key.", - selfDescription: "You ask Bob if he has the key.", + content: '"Do you have the key?" I asked Bob.', targetIds: ["bob"], modifiers: [], }, @@ -68,6 +65,7 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { expect(result.intents).toHaveLength(1); expect(result.intents[0].type).toBe("dialogue"); + expect(result.intents[0].content).toContain("entity@alice[I]"); expect(result.intents[0].targetIds).toEqual(["bob"]); }); @@ -78,21 +76,17 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { world.addEntity(alice); world.addEntity(bob); - const mockResponse: IntentSequence = { + const mockResponse = { intents: [ { type: "dialogue", - originalText: '"Cover me," Alice whispered to Bob.', - description: "Alice whispers to Bob requesting cover.", - selfDescription: "You whisper to Bob requesting cover.", + content: '"Cover me," I whispered to Bob.', targetIds: ["bob"], modifiers: [], }, { type: "action", - originalText: "She crept towards the door and pulled the handle.", - description: "Creep towards the door and pull the handle.", - selfDescription: "You creep towards the door and pull the handle.", + content: "I crept towards the door and pulled the handle.", targetIds: [], modifiers: [], }, @@ -113,6 +107,7 @@ describe("IntentDecoder Unit Tests (Tier 1)", () => { expect(result.intents[0].targetIds).toEqual(["bob"]); expect(result.intents[1].type).toBe("action"); expect(result.intents[1].actorId).toBe("alice"); + expect(result.intents[1].content).toContain("entity@alice[I]"); }); test("throws on LLM failure", async () => { diff --git a/packages/intent/tsconfig.json b/packages/intent/tsconfig.json index f601e59..88fa32b 100644 --- a/packages/intent/tsconfig.json +++ b/packages/intent/tsconfig.json @@ -5,5 +5,9 @@ "outDir": "dist" }, "include": ["src"], - "references": [{ "path": "../core" }, { "path": "../llm" }] + "references": [ + { "path": "../core" }, + { "path": "../llm" }, + { "path": "../voice" } + ] } diff --git a/packages/llm/src/base-provider.ts b/packages/llm/src/base-provider.ts index 4540b25..2d33436 100644 --- a/packages/llm/src/base-provider.ts +++ b/packages/llm/src/base-provider.ts @@ -105,6 +105,7 @@ export abstract class BaseLLMProvider implements ILLMProvider { systemPrompt: request.systemPrompt, userContext: request.userContext, usage, + response: parsed, }); return { success: true, data: parsed, usage }; diff --git a/packages/llm/src/llm.ts b/packages/llm/src/llm.ts index 7d136da..fb7b1ca 100644 --- a/packages/llm/src/llm.ts +++ b/packages/llm/src/llm.ts @@ -1,6 +1,22 @@ import { z } from "zod"; import { ProviderRegistry } from "./registry.js"; +export interface PromptComponent { + label: string; + type: "system" | "world" | "events" | "memories" | "input" | "other"; + content: string; +} + +export interface PromptBreakdown { + systemPrompt: string; + userContext: string; + components?: PromptComponent[]; +} + +export interface IPromptBuilder { + build(...args: TArgs): PromptBreakdown; +} + export interface LLMRequest { systemPrompt: string; userContext: string; @@ -33,6 +49,7 @@ export interface LLMCallRecord { providerInstanceName?: string; maxContext?: number; }; + response?: unknown; } export interface ILLMProvider { diff --git a/packages/llm/src/providers/mock.ts b/packages/llm/src/providers/mock.ts index b898133..8c2fde0 100644 --- a/packages/llm/src/providers/mock.ts +++ b/packages/llm/src/providers/mock.ts @@ -30,6 +30,7 @@ export class MockLLMProvider implements ILLMProvider { registerGenerative("mock", () => new MockLLMProvider([])); } + // eslint-disable-next-line @typescript-eslint/no-unused-vars static create(inst: ModelProviderInstance): ILLMProvider { return new MockLLMProvider([]); } @@ -48,15 +49,21 @@ export class MockLLMProvider implements ILLMProvider { return { success: false, error: "Mock responses exhausted" }; } const usage = { inputTokens: 100, outputTokens: 50, totalTokens: 150 }; - this.lastCalls.push({ - systemPrompt: request.systemPrompt, - userContext: request.userContext, - usage, - }); try { const parsed = request.schema.parse(next); + this.lastCalls.push({ + systemPrompt: request.systemPrompt, + userContext: request.userContext, + usage, + response: parsed, + }); return { success: true, data: parsed, usage }; } catch (e) { + this.lastCalls.push({ + systemPrompt: request.systemPrompt, + userContext: request.userContext, + usage, + }); return { success: false, error: e instanceof Error ? e.message : String(e), diff --git a/packages/llm/tests/openai.test.ts b/packages/llm/tests/openai.test.ts index b09f2a7..0a01413 100644 --- a/packages/llm/tests/openai.test.ts +++ b/packages/llm/tests/openai.test.ts @@ -66,6 +66,7 @@ vi.mock("@langchain/openai", () => { constructor(config: unknown) { this.config = config; } + // eslint-disable-next-line @typescript-eslint/no-unused-vars embedQuery = vi.fn().mockImplementation(async (text: string) => { return [0.1, 0.2, 0.3]; }); diff --git a/packages/llm/tests/openrouter.test.ts b/packages/llm/tests/openrouter.test.ts index 7f0fee8..6f941d4 100644 --- a/packages/llm/tests/openrouter.test.ts +++ b/packages/llm/tests/openrouter.test.ts @@ -127,6 +127,10 @@ describe("OpenRouterProvider Unit Tests (Tier 1)", () => { expect(provider.lastCalls[0]).toEqual({ systemPrompt: "system prompt", userContext: "user context", + response: { + name: "mocked response", + success: true, + }, usage: { inputTokens: 10, outputTokens: 5, diff --git a/packages/memory/package.json b/packages/memory/package.json index 069ced9..1406ba6 100644 --- a/packages/memory/package.json +++ b/packages/memory/package.json @@ -10,6 +10,7 @@ "@omnia/core": "workspace:*", "@omnia/intent": "workspace:*", "@omnia/llm": "workspace:*", + "@omnia/voice": "workspace:*", "zod": "^4.4.3" } } diff --git a/packages/memory/src/buffer.ts b/packages/memory/src/buffer.ts index 394cc56..a3c981e 100644 --- a/packages/memory/src/buffer.ts +++ b/packages/memory/src/buffer.ts @@ -1,10 +1,11 @@ import Database from "better-sqlite3"; -import { Entity, resolveAlias } from "@omnia/core"; +import { Entity } from "@omnia/core"; import { Intent } from "@omnia/intent"; +import { hydrate } from "@omnia/voice"; export interface BufferEntry { id: string; - ownerId: string; // Whose subjective memory buffer this lives in + ownerId: string; // Whose Cognitive Buffer this entry lives in timestamp: string; // WorldClock.get().toISOString() at write time locationId: string | null; // Actor's location when this happened @@ -23,32 +24,14 @@ export function serializeSubjectiveBufferEntry( entry: BufferEntry, viewer: Entity, ): string { - const isSelf = viewer.id === entry.intent.actorId; - - if (isSelf) { - let details = ( - entry.intent.selfDescription || - entry.intent.description || - entry.intent.originalText - ).trim(); - if (details.length > 0) { - details = details.charAt(0).toUpperCase() + details.slice(1); - } - if (entry.intent.type === "action" && entry.outcome) { - details += ` (Outcome: ${entry.outcome.isValid ? "Succeeded" : `Failed - ${entry.outcome.reason}`})`; - } - return details; + let details = hydrate(entry.intent.content, viewer).trim(); + if (details.length > 0) { + details = details.charAt(0).toUpperCase() + details.slice(1); } - - const actorAlias = resolveAlias(viewer, entry.intent.actorId); - const subjectStr = actorAlias.charAt(0).toUpperCase() + actorAlias.slice(1); - - let details = (entry.intent.description || entry.intent.originalText).trim(); if (entry.intent.type === "action" && entry.outcome) { details += ` (Outcome: ${entry.outcome.isValid ? "Succeeded" : `Failed - ${entry.outcome.reason}`})`; } - - return `${subjectStr} ${details}`; + return details; } export class BufferRepository { diff --git a/packages/memory/src/handoff-prompt-builder.ts b/packages/memory/src/handoff-prompt-builder.ts new file mode 100644 index 0000000..3a019bc --- /dev/null +++ b/packages/memory/src/handoff-prompt-builder.ts @@ -0,0 +1,61 @@ +import { Entity } from "@omnia/core"; +import { PromptBreakdown, PromptComponent, IPromptBuilder } from "@omnia/llm"; +import { BufferEntry, serializeSubjectiveBufferEntry } from "./buffer.js"; + +/** + * Prompt builder for the Handoff Engine. + * Separates prompt generation, structure, and component breakdowns. + */ +export class HandoffPromptBuilder implements IPromptBuilder< + [Entity, BufferEntry[], Date] +> { + build(entity: Entity, candidates: BufferEntry[], now: Date): PromptBreakdown { + const candidatesList = candidates + .map((entry) => { + const serialized = serializeSubjectiveBufferEntry(entry, entity); + return `ID: ${entry.id} | Timestamp: ${entry.timestamp} | Location: ${entry.locationId || "None"}\nContent: ${serialized}`; + }) + .join("\n---\n"); + + const systemPrompt = ` +You are the memory Handoff Engine. Your task is to process a list of Cognitive Buffer entries for an entity and select which memories to promote to the Memory Ledger, and which to forget or summarize. + +Instructions: +1. **Cluster** related consecutive buffer entries into high-level narrative beats or events (e.g. physical action and its outcome or trivial actions). Combine them into a single chunk. +2. **Write in the third-person** for the events of other entities. (eg. Alan did that. Sarah did this, etc) +2. **Write in first-person for the events that you yourself did. (eg. I did this, I did that.) +3. **verbatim Quotes**: Extract verbatim, high-salience quotes from dialogue if relevant. Do not modify or invent quotes. +4. **Determine Importance**: Assign an importance score from 1 (trivial, e.g. waking up) to 10 (life-altering, e.g. witnessing a crime). +4. Discard small body movements like looking around, sighing, etc that do not contextually hold any meaning after it is done. +5. **Involved Entities**: Identify all entity IDs involved in the memories in this chunk. +6. **Retain in Cognitive Buffer (Pinning)**: If a beat represents an unresolved high-stakes situation (e.g. a standing threat, an unanswered accusation, an ongoing chase or conflict), set "retainInBuffer" to true so it remains in the Cognitive Buffer for immediate context. Otherwise, set it to false so it is safely pruned from the Cognitive Buffer. +7. **Exclude stage business**: Glances, sighs, ambient noticing, and irrelevant sensory details should be ignored and not included in any promoted chunk. They will be forgotten. +8. **Forget by omission**: Any buffer entry ID that you do not include in any chunk's "sourceEntryIds" will be permanently deleted and forgotten. +`.trim(); + + const entityContext = ` +Subject Entity ID: ${entity.id} +Current Time: ${now.toISOString()} +`.trim(); + + const candidatesSection = `Cognitive Buffer Candidates for Handoff:\n${candidatesList}`; + + const userContext = `${entityContext}\n\n${candidatesSection}`; + + const components: PromptComponent[] = [ + { label: "System Prompt", type: "system", content: systemPrompt }, + { label: "Entity Context", type: "world", content: entityContext }, + { + label: "Cognitive Candidates", + type: "input", + content: candidatesSection, + }, + ]; + + return { + systemPrompt, + userContext, + components, + }; + } +} diff --git a/packages/memory/src/handoff.ts b/packages/memory/src/handoff.ts index 4e61028..a31309c 100644 --- a/packages/memory/src/handoff.ts +++ b/packages/memory/src/handoff.ts @@ -6,7 +6,8 @@ import { BufferRepository, } from "./buffer.js"; import { LedgerEntry, LedgerRepository } from "./ledger.js"; -import { ILLMProvider, IEmbeddingProvider } from "@omnia/llm"; +import { ILLMProvider, IEmbeddingProvider, PromptComponent } from "@omnia/llm"; +import { HandoffPromptBuilder } from "./handoff-prompt-builder.js"; export const HandoffChunkSchema = z.object({ sourceEntryIds: z.array(z.string()), // buffer rows this chunk consumes @@ -33,7 +34,7 @@ export function getMemorySectionLength( now: Date, ): number { if (entries.length === 0) { - return `=== RECENT EVENTS ===\n(No recent events recorded.)`.length; + return `=== COGNITIVE BUFFER ===\n(No entries recorded.)`.length; } const groupedLines: string[] = []; @@ -52,7 +53,7 @@ export function getMemorySectionLength( groupedLines.push(` - ${serialized}`); } - return `=== RECENT EVENTS ===\n${groupedLines.join("\n")}`.length; + return `=== COGNITIVE BUFFER ===\n${groupedLines.join("\n")}`.length; } function checkSceneExit(entity: Entity, bufferEntries: BufferEntry[]): boolean { @@ -85,7 +86,9 @@ function checkIdleDecay(bufferEntries: BufferEntry[]): boolean { // Check the last N entries const lastN = bufferEntries.slice(-N); - return lastN.every((e) => e.intent.type === "monologue"); + return lastN.every( + (e) => e.intent.type === "monologue" || e.intent.type === "thought", + ); } function checkAttributeTrigger(entity: Entity): boolean { @@ -200,53 +203,44 @@ export function splitBufferForHandoff( /** * HandoffEngine processes memory handoffs using LLM summarization and DB transactions. */ +export interface HandoffRunResult { + success: boolean; + systemPrompt?: string; + userContext?: string; + promptComponents?: PromptComponent[]; + response?: unknown; +} + export class HandoffEngine { + public lastResult: HandoffRunResult | null = null; + private promptBuilder: HandoffPromptBuilder; + constructor( private llmProvider: ILLMProvider, private embedProvider: IEmbeddingProvider, private bufferRepo: BufferRepository, private ledgerRepo: LedgerRepository, - ) {} + ) { + this.promptBuilder = new HandoffPromptBuilder(); + } async runHandoff( entity: Entity, bufferEntries: BufferEntry[], now: Date, ): Promise { + this.lastResult = null; const { candidates } = splitBufferForHandoff(bufferEntries, now); if (candidates.length === 0) { return false; } - const candidatesList = candidates - .map((entry) => { - const serialized = serializeSubjectiveBufferEntry(entry, entity); - return `ID: ${entry.id} | Timestamp: ${entry.timestamp} | Location: ${entry.locationId || "None"}\nContent: ${serialized}`; - }) - .join("\n---\n"); - - const systemPrompt = ` -You are the memory Handoff Engine. Your task is to process a list of recent working memory buffer entries for an entity and select which memories to promote to the long-term Ledger, and which to forget or summarize. - -Instructions: -1. **Cluster** related consecutive buffer entries into high-level narrative beats or events (e.g. a full back-and-forth conversation or a single physical action and its outcome). Combine them into a single summary chunk. -2. **Write in the third-person** for the "content" of each chunk (e.g. "John asked Mary for the key, and Mary reluctantly handed it over"). -3. **verbatim Quotes**: Extract verbatim, high-salience quotes from dialogue if relevant. Do not modify or invent quotes. -4. **Determine Importance**: Assign an importance score from 1 (trivial, e.g. waking up) to 10 (life-altering, e.g. witnessing a crime). -5. **Involved Entities**: Identify all entity IDs involved in the memories in this chunk. -6. **Retain in Buffer (Pinning)**: If a beat represents an unresolved high-stakes situation (e.g. a standing threat, an unanswered accusation, an ongoing chase or conflict), set "retainInBuffer" to true so it remains in the working memory buffer for immediate context. Otherwise, set it to false so it is safely pruned from the buffer. -7. **Exclude stage business**: Glances, sighs, ambient noticing, and irrelevant sensory details should be ignored and not included in any promoted chunk. They will be forgotten. -8. **Forget by omission**: Any buffer entry ID that you do not include in any chunk's "sourceEntryIds" will be permanently deleted and forgotten. -`.trim(); - - const userContext = ` -Subject Entity ID: ${entity.id} -Current Time: ${now.toISOString()} - -Working Memory Candidates for Handoff: -${candidatesList} -`.trim(); + const { systemPrompt, userContext, components } = this.promptBuilder.build( + entity, + candidates, + now, + ); const response = await this.llmProvider.generateStructuredResponse({ systemPrompt, @@ -255,11 +249,29 @@ ${candidatesList} }); if (!response.success || !response.data) { + this.lastResult = { + success: false, + systemPrompt, + userContext, + promptComponents: components, + }; return false; } + this.lastResult = { + success: true, + systemPrompt, + userContext, + promptComponents: components, + response: response.data, + }; + const result = response.data; - const db = (this.bufferRepo as any).db; + const db = ( + this.bufferRepo as unknown as { + db: { transaction: (fn: () => void) => () => void }; + } + ).db; const ledgerEntries: LedgerEntry[] = []; for (const chunk of result.chunks) { diff --git a/packages/memory/tests/handoff.test.ts b/packages/memory/tests/handoff.test.ts index bf0c1c2..4fe9727 100644 --- a/packages/memory/tests/handoff.test.ts +++ b/packages/memory/tests/handoff.test.ts @@ -1,20 +1,20 @@ -import { describe, test, expect } from "vitest"; import Database from "better-sqlite3"; +import { describe, test, expect } from "vitest"; import { Entity } from "@omnia/core"; import { MockLLMProvider, MockEmbeddingProvider } from "@omnia/llm"; import { BufferEntry, BufferRepository, LedgerRepository, - checkHandoffTrigger, - splitBufferForHandoff, HandoffEngine, + splitBufferForHandoff, + checkHandoffTrigger, } from "@omnia/memory"; -describe("Memory Handoff Tests (Tier 1)", () => { - const now = new Date("2026-07-07T12:00:00.000Z"); +const now = new Date("2026-07-09T08:00:00.000Z"); - test("splitBufferForHandoff correctly splits based on watermark and fresh buckets", () => { +describe("Memory Handoff Tests (Tier 1)", () => { + test("splitBufferForHandoff identifies candidate entries based on recency", () => { const entries: BufferEntry[] = []; // Add 12 older entries (older than 30 minutes) @@ -30,8 +30,7 @@ describe("Memory Handoff Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "dialogue", - originalText: `Old event ${i}`, - description: `does old thing ${i}`, + content: `entity@alice[I] do old thing ${i}`, actorId: "alice", targetIds: ["bob"], }, @@ -54,8 +53,7 @@ describe("Memory Handoff Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "dialogue", - originalText: `Fresh event ${idx}`, - description: `does fresh thing ${idx}`, + content: `entity@alice[I] do fresh thing ${idx}`, actorId: "alice", targetIds: ["bob"], }, @@ -84,8 +82,7 @@ describe("Memory Handoff Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "dialogue", - originalText: "hello", - description: "says hello", + content: "entity@alice[I] say hello", actorId: "alice", targetIds: [], }, @@ -100,8 +97,7 @@ describe("Memory Handoff Tests (Tier 1)", () => { locationId: "room-2", intent: { type: "monologue", - originalText: "think", - description: "thinks", + content: "entity@alice[I] think", actorId: "alice", targetIds: [], }, @@ -138,8 +134,7 @@ describe("Memory Handoff Tests (Tier 1)", () => { locationId: "room-1", intent: { type: i % 2 === 0 ? "dialogue" : "action", - originalText: `Event ${i}`, - description: `does thing ${i}`, + content: `entity@alice[I] do thing ${i}`, actorId: "alice", targetIds: ["bob"], }, diff --git a/packages/memory/tests/memory.test.ts b/packages/memory/tests/memory.test.ts index 0741c23..678e985 100644 --- a/packages/memory/tests/memory.test.ts +++ b/packages/memory/tests/memory.test.ts @@ -1,24 +1,14 @@ -import { describe, test, expect } from "vitest"; import Database from "better-sqlite3"; +import { describe, test, expect } from "vitest"; import { Entity, SQLiteRepository } from "@omnia/core"; import { Intent } from "@omnia/intent"; import { BufferEntry, BufferRepository, serializeSubjectiveBufferEntry, - resolveAlias, } from "@omnia/memory"; describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { - test("resolveAlias correctly handles self and fallbacks", () => { - const viewer = new Entity("alice"); - viewer.aliases.set("bob", "the hooded figure"); - - expect(resolveAlias(viewer, "alice")).toBe("you"); - expect(resolveAlias(viewer, "bob")).toBe("the hooded figure"); - expect(resolveAlias(viewer, "charlie")).toBe("an unfamiliar figure"); - }); - test("serializes dialogue intent substituting target/actor aliases", () => { const viewer = new Entity("alice"); viewer.aliases.set("bob", "the hooded figure"); @@ -31,9 +21,8 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "dialogue", - originalText: '"Hello there," Bob said to Charlie.', - description: "says, 'Hello there' to the bartender", - selfDescription: "You say, 'Hello there' to the bartender.", + content: + "entity@bob[I] say 'Hello there' to entity@charlie[the bartender]", actorId: "bob", targetIds: ["charlie"], modifiers: [], @@ -42,7 +31,7 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { const result = serializeSubjectiveBufferEntry(entry, viewer); expect(result).toBe( - "The hooded figure says, 'Hello there' to the bartender", + "The hooded figure says 'Hello there' to the bartender", ); }); @@ -57,9 +46,7 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "action", - originalText: "Bob tried to break the latch.", - description: "attempts to break the lock latch", - selfDescription: "You attempt to break the lock latch.", + content: "entity@bob[I] attempt to break the lock latch", actorId: "bob", targetIds: [], modifiers: [], @@ -86,9 +73,7 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "action", - originalText: "I opened the window.", - description: "open the window", - selfDescription: "You open the window.", + content: "entity@alice[I] open the window", actorId: "alice", targetIds: [], modifiers: [], @@ -96,7 +81,7 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { }; const resultSelf = serializeSubjectiveBufferEntry(entrySelf, viewer); - expect(resultSelf).toBe("You open the window."); + expect(resultSelf).toBe("I open the window"); const entryUnfamiliar: BufferEntry = { id: "entry-unfamiliar", @@ -105,9 +90,7 @@ describe("Subjective Buffer Entry Serializer Tests (Tier 1)", () => { locationId: "room-1", intent: { type: "action", - originalText: "Someone knocked.", - description: "knocks on the door", - selfDescription: "You knock on the door.", + content: "entity@stranger-1[I] knock on the door", actorId: "stranger-1", targetIds: [], modifiers: [], @@ -136,9 +119,7 @@ describe("BufferRepository Persistence Tests (Tier 1)", () => { const intent: Intent = { type: "action", - originalText: "Alice picked up a stick.", - description: "Alice gathers a stick", - selfDescription: "You gather a stick.", + content: "entity@alice[I] gather a stick", actorId: "alice", targetIds: [], modifiers: [], @@ -196,8 +177,7 @@ describe("BufferRepository Persistence Tests (Tier 1)", () => { locationId: "forest", intent: { type: "action", - originalText: "Alice sneezed.", - description: "Alice sneezes", + content: "entity@alice[I] sneeze", actorId: "alice", targetIds: [], }, diff --git a/packages/memory/tsconfig.json b/packages/memory/tsconfig.json index c3a3b11..3a80f01 100644 --- a/packages/memory/tsconfig.json +++ b/packages/memory/tsconfig.json @@ -8,6 +8,7 @@ "references": [ { "path": "../core" }, { "path": "../intent" }, - { "path": "../llm" } + { "path": "../llm" }, + { "path": "../voice" } ] } diff --git a/packages/scenario/src/loader.ts b/packages/scenario/src/loader.ts index ff31959..e4806cf 100644 --- a/packages/scenario/src/loader.ts +++ b/packages/scenario/src/loader.ts @@ -145,8 +145,14 @@ export class ScenarioLoader { timestamp: mem.timestamp, locationId: mem.locationId, intent: { - ...mem.intent, - selfDescription: mem.intent.selfDescription ?? "", + type: mem.intent.type, + content: + mem.intent.content || + mem.intent.description || + mem.intent.originalText || + "", + actorId: mem.intent.actorId, + targetIds: mem.intent.targetIds, modifiers: mem.intent.modifiers ?? [], }, outcome: mem.outcome, diff --git a/packages/scenario/src/schema.ts b/packages/scenario/src/schema.ts index bbfe4ec..c222f06 100644 --- a/packages/scenario/src/schema.ts +++ b/packages/scenario/src/schema.ts @@ -30,9 +30,10 @@ export const ScenarioMemoryEntrySchema = z.object({ timestamp: z.string(), // ISO string locationId: z.string().nullable(), intent: z.object({ - type: z.enum(["dialogue", "action", "monologue"]), - originalText: z.string(), - description: z.string(), + type: z.enum(["dialogue", "action", "monologue", "thought"]), + content: z.string().optional(), + originalText: z.string().optional(), + description: z.string().optional(), selfDescription: z.string().optional(), actorId: z.string(), targetIds: z.array(z.string()), diff --git a/packages/scenario/tests/scenario.test.ts b/packages/scenario/tests/scenario.test.ts index 2f9ea6a..6c60a60 100644 --- a/packages/scenario/tests/scenario.test.ts +++ b/packages/scenario/tests/scenario.test.ts @@ -139,7 +139,7 @@ describe("Scenario Validation & Schema Tests (Tier 1)", () => { expect(memories[0].id).toBe("mem-seed-1"); expect(memories[0].timestamp).toBe("2026-07-09T07:55:00.000Z"); expect(memories[0].locationId).toBe("lobby"); - expect(memories[0].intent.description).toBe("entered the house"); + expect(memories[0].intent.content).toBe("entered the house"); db.close(); }); diff --git a/packages/scenario/tests/talking-room.test.ts b/packages/scenario/tests/talking-room.test.ts index 9226fb0..88a5ed7 100644 --- a/packages/scenario/tests/talking-room.test.ts +++ b/packages/scenario/tests/talking-room.test.ts @@ -128,18 +128,24 @@ describe("Talking Room Demo Scenario Test (Tier 1)", () => { // 7. Assert initial pre-seeded memories const alphaMemories = bufferRepo.listForOwner(alphaId); - expect(alphaMemories).toHaveLength(1); - expect(alphaMemories[0].id).toBe("alpha-wake"); + expect(alphaMemories).toHaveLength(3); + expect(alphaMemories[0].id).toBe("ab3f29d2-cf11-4111-9a99-b13c126d123e"); expect(alphaMemories[0].intent.type).toBe("monologue"); - expect(alphaMemories[0].intent.originalText).toContain("jail"); - expect(alphaMemories[0].intent.description).toBe(""); + expect(alphaMemories[0].intent.content).toContain("jail"); + + expect(alphaMemories[1].id).toBe("10ak29d2-as11-9811-9a99-b13c126d123e"); + expect(alphaMemories[1].intent.type).toBe("action"); + expect(alphaMemories[1].intent.content).toContain("another man"); + + expect(alphaMemories[2].id).toBe("zz3f29d2-as11-9811-9a99-b13c126d123e"); + expect(alphaMemories[2].intent.type).toBe("action"); + expect(alphaMemories[2].intent.content).toContain("sleep"); const betaMemories = bufferRepo.listForOwner(betaId); expect(betaMemories).toHaveLength(1); - expect(betaMemories[0].id).toBe("beta-wake"); + expect(betaMemories[0].id).toBe("zx1f29d2-cf11-4111-9a99-b13c126d123e"); expect(betaMemories[0].intent.type).toBe("action"); - expect(betaMemories[0].intent.originalText).toContain("agreement"); - expect(betaMemories[0].intent.description).toBe(""); + expect(betaMemories[0].intent.content).toContain("unfamiliar"); db.close(); }); diff --git a/packages/voice/package.json b/packages/voice/package.json new file mode 100644 index 0000000..1492a2b --- /dev/null +++ b/packages/voice/package.json @@ -0,0 +1,13 @@ +{ + "name": "@omnia/voice", + "version": "0.0.0", + "private": true, + "type": "module", + "exports": { + ".": "./dist/index.js" + }, + "dependencies": { + "@omnia/core": "workspace:*", + "compromise": "^14.14.0" + } +} diff --git a/packages/voice/src/contractions.ts b/packages/voice/src/contractions.ts new file mode 100644 index 0000000..3f3f2f0 --- /dev/null +++ b/packages/voice/src/contractions.ts @@ -0,0 +1,82 @@ +import { splitQuotes } from "./dehydration.js"; + +/** + * Preprocessor that expands common contractions (e.g. "he's" -> "he is", "I'm" -> "I am") + * in non-quote segments of a text, keeping dialogue segments untouched. + */ +export function expandContractions(text: string): string { + if (!text) return ""; + + const contractionMap: Record = { + "i'm": "I am", + "you're": "you are", + "he's": "he is", + "she's": "she is", + "it's": "it is", + "we're": "we are", + "they're": "they are", + "i've": "I have", + "you've": "you have", + "we've": "we have", + "they've": "they have", + "i'd": "I would", + "you'd": "you would", + "he'd": "he would", + "she'd": "she would", + "we'd": "we would", + "they'd": "they would", + "i'll": "I will", + "you'll": "you will", + "he'll": "he will", + "she'll": "she will", + "we'll": "we will", + "they'll": "they will", + "isn't": "is not", + "aren't": "are not", + "wasn't": "was not", + "weren't": "were not", + "haven't": "have not", + "hasn't": "has not", + "hadn't": "had not", + "won't": "will not", + "wouldn't": "would not", + "don't": "do not", + "doesn't": "does not", + "didn't": "did not", + "can't": "cannot", + "couldn't": "could not", + "shouldn't": "should not", + "mightn't": "might not", + "mustn't": "must not", + }; + + const segments = splitQuotes(text); + const processed = segments.map((seg) => { + if (seg.isQuote) { + return `"${seg.text}"`; + } + + let chunk = seg.text; + Object.entries(contractionMap).forEach(([contraction, replacement]) => { + const escaped = contraction.replace("'", "'"); + const regex = new RegExp(`\\b${escaped}\\b`, "gi"); + chunk = chunk.replace(regex, (matched) => { + const isCapitalized = matched[0] === matched[0].toUpperCase(); + let finalRep = replacement; + if (isCapitalized) { + finalRep = finalRep[0].toUpperCase() + finalRep.slice(1); + } else { + if (finalRep.startsWith("I ")) { + // Keep I capitalized + } else { + finalRep = finalRep[0].toLowerCase() + finalRep.slice(1); + } + } + return finalRep; + }); + }); + return chunk; + }); + + return processed.join(""); +} diff --git a/packages/voice/src/dehydration.ts b/packages/voice/src/dehydration.ts new file mode 100644 index 0000000..d0de255 --- /dev/null +++ b/packages/voice/src/dehydration.ts @@ -0,0 +1,147 @@ +export interface Segment { + text: string; + isQuote: boolean; +} + +/** + * Splits text into quote and non-quote segments. + */ +export function splitQuotes(text: string): Segment[] { + const segments: Segment[] = []; + let current = ""; + let inQuote = false; + + for (let i = 0; i < text.length; i++) { + const char = text[i]; + if (char === '"') { + if (current) { + segments.push({ text: current, isQuote: inQuote }); + current = ""; + } + inQuote = !inQuote; + } else { + current += char; + } + } + + if (current) { + segments.push({ text: current, isQuote: inQuote }); + } + + return segments; +} + +/** + * Transforms standard narrative prose from the source actor's perspective + * into a dehydrated canonical form with entity@[original] placeholder tags. + */ +export function dehydrate( + content: string, + sourceId: string, + targetIds: string[], + aliasMap: Record, +): string { + if (!content) return ""; + + const segments = splitQuotes(content); + + const processedSegments = segments.map((seg) => { + if (seg.isQuote) { + return `"${seg.text}"`; + } + + let text = seg.text; + + // 1. Map lowercase aliases/names/IDs to IDs + const nameToId = new Map(); + + // Add target IDs and source ID themselves + nameToId.set(sourceId.toLowerCase(), sourceId); + targetIds.forEach((id) => { + nameToId.set(id.toLowerCase(), id); + }); + + // Add entries from aliasMap (mapped lowercased) + Object.entries(aliasMap).forEach(([name, id]) => { + nameToId.set(name.toLowerCase(), id); + }); + + // Sort names by length descending to match longest name first + const sortedNames = Array.from(nameToId.keys()).sort( + (a, b) => b.length - a.length, + ); + + // Track state of matched target IDs for pronoun lookback + const matchedTargetIds: string[] = []; + + // 2. Replace names and aliases with entity@[name] + sortedNames.forEach((name) => { + const id = nameToId.get(name)!; + const escapedName = name.replace( + new RegExp("[-/\\\\^$*+?.()|[\\]{}]", "g"), + "\\$&", + ); + const regex = new RegExp(`\\b${escapedName}\\b`, "gi"); + + text = text.replace(regex, (matched) => { + if (id !== sourceId) { + matchedTargetIds.push(id); + } + return `entity@${id}[${matched}]`; + }); + }); + + // 3. Replace first-person pronouns with source actor tag + const firstPersonPronouns = [ + { word: "i" }, + { word: "me" }, + { word: "my" }, + { word: "myself" }, + { word: "mine" }, + { word: "we" }, + { word: "us" }, + { word: "our" }, + { word: "ours" }, + { word: "ourselves" }, + ]; + + firstPersonPronouns.forEach(({ word }) => { + const regex = new RegExp(`\\b${word}\\b`, "gi"); + text = text.replace(regex, (matched) => { + return `entity@${sourceId}[${matched}]`; + }); + }); + + // 4. Replace third-person pronouns using state lookback + const thirdPersonPronouns = [ + "he", + "him", + "his", + "himself", + "she", + "her", + "hers", + "herself", + "they", + "them", + "their", + "theirs", + "themselves", + ]; + thirdPersonPronouns.forEach((pronoun) => { + const regex = new RegExp(`\\b${pronoun}\\b`, "gi"); + text = text.replace(regex, (matched) => { + const lastTargetId = + matchedTargetIds[matchedTargetIds.length - 1] || targetIds[0]; + if (lastTargetId) { + return `entity@${lastTargetId}[${matched}]`; + } + return matched; + }); + }); + + return text; + }); + + return processedSegments.join(""); +} diff --git a/packages/voice/src/hydration.ts b/packages/voice/src/hydration.ts new file mode 100644 index 0000000..be01709 --- /dev/null +++ b/packages/voice/src/hydration.ts @@ -0,0 +1,238 @@ +import nlp from "compromise"; +import { Entity, WorldState, resolveAlias } from "@omnia/core"; +import { splitQuotes } from "./dehydration.js"; + +/** + * Hydrates a dehydrated narration text containing entity@[original] symbol tags + * into natural language from a specific viewer's perspective. + */ +export function hydrate(content: string, viewer: Entity): string { + if (!content) return ""; + const segments = splitQuotes(content); + + const processedSegments = segments.map((seg) => { + if (seg.isQuote) { + return `'${seg.text}'`; + } + + // Match entity@[original] and optionally the following space and word + const regex = /entity@([a-zA-Z0-9-]+)\[([^\]]+)\](?:\s+([a-zA-Z]+))?/g; + + const firstPersonSet = new Set([ + "i", + "me", + "my", + "myself", + "mine", + "we", + "us", + "our", + "ours", + ]); + const thirdPersonSet = new Set([ + "he", + "him", + "his", + "himself", + "she", + "her", + "hers", + "herself", + "they", + "them", + "their", + "theirs", + "themselves", + ]); + + return seg.text.replace(regex, (matchStr, id, original, followingWord) => { + const isSelf = id === viewer.id; + const lowerOriginal = original.toLowerCase(); + let resolvedSubject: string; + let isThirdPersonSingular = false; + + if (isSelf) { + if (["his", "her", "their", "my", "its", "our"].includes(lowerOriginal)) + resolvedSubject = "my"; + else if (["hers", "theirs", "mine", "ours"].includes(lowerOriginal)) + resolvedSubject = "mine"; + else if ( + [ + "himself", + "herself", + "themselves", + "myself", + "itself", + "ourselves", + ].includes(lowerOriginal) + ) + resolvedSubject = "myself"; + else if (["he", "she", "they", "i", "we"].includes(lowerOriginal)) + resolvedSubject = "I"; + else if (["him", "her", "them", "me", "us"].includes(lowerOriginal)) + resolvedSubject = "me"; + else { + // Noun/alias mapped to self: check preceding/succeeding context + const matchIdx = seg.text.indexOf(matchStr); + const precedingText = seg.text.slice(0, matchIdx); + const prec = precedingText.trim(); + const words = prec.split(/\s+/); + const lastWord = words[words.length - 1]?.toLowerCase() || ""; + + const prepositions = [ + "to", + "with", + "for", + "at", + "by", + "from", + "in", + "on", + "about", + "between", + "of", + "under", + "over", + "behind", + "beside", + "through", + ]; + if (prepositions.includes(lastWord)) { + resolvedSubject = "me"; + } else { + resolvedSubject = "I"; + } + } + } else { + const alias = resolveAlias(viewer, id); + if (firstPersonSet.has(lowerOriginal)) { + if (["my", "our"].includes(lowerOriginal)) + resolvedSubject = `${alias}'s`; + else if (["mine", "ours"].includes(lowerOriginal)) + resolvedSubject = `${alias}'s`; + else if (["myself", "ourselves"].includes(lowerOriginal)) + resolvedSubject = "himself"; + else { + resolvedSubject = alias; + isThirdPersonSingular = true; + } + } else if (thirdPersonSet.has(lowerOriginal)) { + resolvedSubject = original; + if (["he", "she", "it"].includes(lowerOriginal)) { + isThirdPersonSingular = true; + } + } else { + resolvedSubject = alias; + isThirdPersonSingular = true; + } + } + + if (followingWord) { + if (isThirdPersonSingular) { + const conj = nlp(followingWord).verbs().conjugate()[0] as + { Infinitive?: string; PresentTense?: string } | undefined; + if (conj && conj.Infinitive === followingWord && conj.PresentTense) { + return `${resolvedSubject} ${conj.PresentTense}`; + } + } + return `${resolvedSubject} ${followingWord}`; + } + + return resolvedSubject; + }); + }); + + return processedSegments.join(""); +} + +/** + * Hydrates a dehydrated narration text containing entity@[original] symbol tags + * into natural language from an objective world perspective. + */ +export function hydrateObjective( + content: string, + worldState: WorldState, +): string { + if (!content) return ""; + const segments = splitQuotes(content); + + const processedSegments = segments.map((seg) => { + if (seg.isQuote) { + return `'${seg.text}'`; + } + + // Match entity@[original] and optionally the following space and word + const regex = /entity@([a-zA-Z0-9-]+)\[([^\]]+)\](?:\s+([a-zA-Z]+))?/g; + + const firstPersonSet = new Set([ + "i", + "me", + "my", + "myself", + "mine", + "we", + "us", + "our", + "ours", + ]); + const thirdPersonSet = new Set([ + "he", + "him", + "his", + "himself", + "she", + "her", + "hers", + "herself", + "they", + "them", + "their", + "theirs", + "themselves", + ]); + + return seg.text.replace(regex, (matchStr, id, original, followingWord) => { + const entity = worldState.getEntity(id); + const name = entity?.attributes.get("name")?.getValue() || id; + const lowerOriginal = original.toLowerCase(); + let resolvedSubject: string; + let isThirdPersonSingular = false; + + if (firstPersonSet.has(lowerOriginal)) { + if (["my", "our"].includes(lowerOriginal)) + resolvedSubject = `${name}'s`; + else if (["mine", "ours"].includes(lowerOriginal)) + resolvedSubject = `${name}'s`; + else if (["myself", "ourselves"].includes(lowerOriginal)) + resolvedSubject = "himself"; + else { + resolvedSubject = name; + isThirdPersonSingular = true; + } + } else if (thirdPersonSet.has(lowerOriginal)) { + resolvedSubject = original; + if (["he", "she", "it"].includes(lowerOriginal)) { + isThirdPersonSingular = true; + } + } else { + resolvedSubject = name; + isThirdPersonSingular = true; + } + + if (followingWord) { + if (isThirdPersonSingular) { + const conj = nlp(followingWord).verbs().conjugate()[0] as + { Infinitive?: string; PresentTense?: string } | undefined; + if (conj && conj.Infinitive === followingWord && conj.PresentTense) { + return `${resolvedSubject} ${conj.PresentTense}`; + } + } + return `${resolvedSubject} ${followingWord}`; + } + + return resolvedSubject; + }); + }); + + return processedSegments.join(""); +} diff --git a/packages/voice/src/index.ts b/packages/voice/src/index.ts new file mode 100644 index 0000000..629558a --- /dev/null +++ b/packages/voice/src/index.ts @@ -0,0 +1,3 @@ +export * from "./dehydration.js"; +export * from "./hydration.js"; +export * from "./contractions.js"; diff --git a/packages/voice/tsconfig.json b/packages/voice/tsconfig.json new file mode 100644 index 0000000..3182c75 --- /dev/null +++ b/packages/voice/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "dist" + }, + "include": ["src"], + "references": [{ "path": "../core" }] +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 17b33d7..88cd69b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -284,6 +284,9 @@ importers: "@types/node": specifier: ^20.19.43 version: 20.19.43 + compromise: + specifier: ^14.16.0 + version: 14.16.0 dotenv: specifier: ^17.4.2 version: 17.4.2 @@ -354,6 +357,9 @@ importers: "@omnia/spatial": specifier: workspace:* version: link:../../packages/spatial + "@omnia/voice": + specifier: workspace:* + version: link:../../packages/voice "@radix-ui/react-dialog": specifier: ^1.1.19 version: 1.1.19(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.7(react@19.2.7))(react@19.2.7) @@ -439,6 +445,9 @@ importers: "@omnia/memory": specifier: workspace:* version: link:../memory + "@omnia/voice": + specifier: workspace:* + version: link:../voice zod: specifier: ^4.4.3 version: 4.4.3 @@ -454,6 +463,9 @@ importers: "@omnia/llm": specifier: workspace:* version: link:../llm + "@omnia/voice": + specifier: workspace:* + version: link:../voice zod: specifier: ^4.4.3 version: 4.4.3 @@ -478,6 +490,9 @@ importers: "@omnia/llm": specifier: workspace:* version: link:../llm + "@omnia/voice": + specifier: workspace:* + version: link:../voice zod: specifier: ^4.4.3 version: 4.4.3 @@ -506,6 +521,9 @@ importers: "@omnia/llm": specifier: workspace:* version: link:../llm + "@omnia/voice": + specifier: workspace:* + version: link:../voice zod: specifier: ^4.4.3 version: 4.4.3 @@ -531,6 +549,15 @@ importers: specifier: workspace:* version: link:../core + packages/voice: + dependencies: + "@omnia/core": + specifier: workspace:* + version: link:../core + compromise: + specifier: ^14.14.0 + version: 14.16.0 + web/docs: dependencies: "@astrojs/starlight": @@ -4716,6 +4743,13 @@ packages: } engines: { node: ">= 18" } + compromise@14.16.0: + resolution: + { + integrity: sha512-4DFYl/Hl7sW4XWUDfx9S5vxqyYKpZDwwqrpXsQv5acdbVP+joKceIcIaLb0lhVWUpDBV0OnExk/o/dnYUwXnhQ==, + } + engines: { node: ">=12.0.0" } + conf@10.2.0: resolution: { @@ -5388,6 +5422,13 @@ packages: integrity: sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==, } + efrt@2.7.0: + resolution: + { + integrity: sha512-/RInbCy1d4P6Zdfa+TMVsf/ufZVotat5hCw3QXmWtjU+3pFEOvOQ7ibo3aIxyCJw2leIeAMjmPj+1SLJiCpdrQ==, + } + engines: { node: ">=12.0.0" } + electron-to-chromium@1.5.389: resolution: { @@ -6114,6 +6155,13 @@ packages: integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==, } + grad-school@0.0.5: + resolution: + { + integrity: sha512-rXunEHF9M9EkMydTBux7+IryYXEZinRk6g8OBOGDBzo/qWJjhTxy86i5q7lQYpCLHN8Sqv1XX3OIOc7ka2gtvQ==, + } + engines: { node: ">=8.0.0" } + groq-sdk@1.3.0: resolution: { @@ -8914,6 +8962,12 @@ packages: integrity: sha512-5Z9ZpRzfuH6l/UAvCPAPUo3665Nk2wLaZU3x+TLHKVzIz33+sbJqbtrYoC3KD4/uVOr2Zp+L0LySezP9OHV9yA==, } + suffix-thumb@5.0.2: + resolution: + { + integrity: sha512-I5PWXAFKx3FYnI9a+dQMWNqTxoRt6vdBdb0O+BJ1sxXCWtSoQCusc13E58f+9p4MYx/qCnEMkD5jac6K2j3dgA==, + } + supports-color@10.2.2: resolution: { @@ -12520,6 +12574,12 @@ snapshots: common-ancestor-path@2.0.0: {} + compromise@14.16.0: + dependencies: + efrt: 2.7.0 + grad-school: 0.0.5 + suffix-thumb: 5.0.2 + conf@10.2.0: dependencies: ajv: 8.20.0 @@ -12898,6 +12958,8 @@ snapshots: ee-first@1.1.1: {} + efrt@2.7.0: {} + electron-to-chromium@1.5.389: {} emoji-regex@10.6.0: {} @@ -13379,6 +13441,8 @@ snapshots: graceful-fs@4.2.11: {} + grad-school@0.0.5: {} + groq-sdk@1.3.0: {} h3@1.15.11: @@ -15488,6 +15552,8 @@ snapshots: stylis@4.4.0: {} + suffix-thumb@5.0.2: {} + supports-color@10.2.2: {} svgo@4.0.1: diff --git a/tests/integration/actor-monologue.test.ts b/tests/integration/actor-monologue.test.ts index 4e877bd..9f713cc 100644 --- a/tests/integration/actor-monologue.test.ts +++ b/tests/integration/actor-monologue.test.ts @@ -7,7 +7,6 @@ import { AttributeVisibility, } from "@omnia/core"; import { MockLLMProvider } from "@omnia/llm"; -import { IntentSequence } from "@omnia/intent"; import { Architect } from "@omnia/architect"; import { BufferRepository, BufferEntry } from "@omnia/memory"; import { @@ -56,34 +55,25 @@ describe("Actor Agent + Monologue Intent Integration (Tier 2)", () => { }; // 2. IntentDecoder splits that prose into 3 intents. - const mockDecodedSequence: IntentSequence = { + const mockDecodedSequence = { intents: [ { type: "monologue", - originalText: - "I can't believe Bob hasn't noticed me yet, Alice thought.", - description: - "Alice internally reflects that Bob has not noticed her.", - selfDescription: - "You internally reflect that Bob has not noticed you.", + content: "I internally reflect that Bob has not noticed me.", actorId: "alice", targetIds: [], modifiers: [], }, { type: "dialogue", - originalText: '"Hey Bob," she called out softly.', - description: "Alice softly calls out to Bob.", - selfDescription: "You softly call out to Bob.", + content: '"Hey Bob," I call out softly to Bob.', actorId: "alice", targetIds: ["bob"], modifiers: [], }, { type: "action", - originalText: "She reached for the ledger on the table.", - description: "Alice reaches for the ledger on the table.", - selfDescription: "You reach for the ledger on the table.", + content: "I reach for the ledger on the table.", actorId: "alice", targetIds: [], modifiers: [], @@ -166,7 +156,7 @@ describe("Actor Agent + Monologue Intent Integration (Tier 2)", () => { const expectedTime = new Date(startTime.getTime() + 3 * 60_000); expect(world.clock.get().toISOString()).toBe(expectedTime.toISOString()); - // 7. All three intents persisted to Alice's memory buffer. + // 7. All three intents persisted to Alice's Cognitive Buffer. const aliceMemory = bufferRepo.listForOwner("alice"); expect(aliceMemory).toHaveLength(3); expect(aliceMemory[0].intent.type).toBe("monologue"); diff --git a/tests/integration/game-loop.test.ts b/tests/integration/game-loop.test.ts index aedf050..852cadd 100644 --- a/tests/integration/game-loop.test.ts +++ b/tests/integration/game-loop.test.ts @@ -1,30 +1,21 @@ import { describe, test, expect } from "vitest"; import Database from "better-sqlite3"; -import { - WorldState, - Entity, - SQLiteRepository, - AttributeVisibility, -} from "@omnia/core"; +import { WorldState, Entity, SQLiteRepository } from "@omnia/core"; import { MockLLMProvider } from "@omnia/llm"; -import { IntentDecoder, IntentSequence } from "@omnia/intent"; +import { IntentDecoder } from "@omnia/intent"; import { Architect } from "@omnia/architect"; -describe("Omnia Integration Tests (Tier 2)", () => { - test("end-to-end intent decoding, validation, execution, and persistence", async () => { +describe("Game Loop Integration Tests (Tier 2)", () => { + test("successfully processes a sequence of dialogue and action intents", async () => { const db = new Database(":memory:"); const repo = new SQLiteRepository(db); const startTime = new Date("2026-07-06T12:00:00.000Z"); const world = new WorldState("world-abc", startTime); - world.addAttribute("location", "Dungeon Crawl", AttributeVisibility.PUBLIC); const alice = new Entity("alice", "room-1"); - alice.addAttribute("role", "rogue", AttributeVisibility.PUBLIC); - world.addEntity(alice); - const bob = new Entity("bob", "room-1"); - bob.addAttribute("role", "knight", AttributeVisibility.PUBLIC); + world.addEntity(alice); world.addEntity(bob); // Save initial state to database @@ -32,22 +23,18 @@ describe("Omnia Integration Tests (Tier 2)", () => { // Mock responses for LLM: // 1. IntentDecoder parses: `"Cover me," Alice whispered to Bob. She crept towards the door and pulled the handle.` - const mockIntentSequence: IntentSequence = { + const mockIntentSequence = { intents: [ { type: "dialogue", - originalText: '"Cover me," Alice whispered to Bob.', - description: "Alice whispers to Bob to cover her.", - selfDescription: "You whisper to Bob to cover you.", + content: '"Cover me," I whisper to Bob.', actorId: "alice", targetIds: ["bob"], modifiers: [], }, { type: "action", - originalText: "She crept towards the door and pulled the handle.", - description: "Alice creeps to the door and pulls the handle.", - selfDescription: "You creep to the door and pull the handle.", + content: "I creep to the door and pull the handle.", actorId: "alice", targetIds: [], modifiers: [], @@ -135,9 +122,7 @@ describe("Omnia Integration Tests (Tier 2)", () => { // 2. Valid dialogue: speaking to bob const intent1 = { type: "action" as const, - originalText: "She tries to unlock the gate with a hairpin.", - description: "Alice attempts to pick the lock with a hairpin.", - selfDescription: "You attempt to pick the lock with a hairpin.", + content: "entity@alice[I] attempt to pick the lock with a hairpin.", actorId: "alice", targetIds: [], modifiers: [], @@ -145,9 +130,7 @@ describe("Omnia Integration Tests (Tier 2)", () => { const intent2 = { type: "dialogue" as const, - originalText: '"This is useless," she mutters.', - description: "Alice mutters to herself.", - selfDescription: "You mutter to yourself.", + content: '"This is useless," entity@alice[I] mutter.', actorId: "alice", targetIds: [], modifiers: [], diff --git a/web/docs/src/assets/img/mem1.png b/web/docs/src/assets/img/mem1.png new file mode 100644 index 0000000..f5d037b Binary files /dev/null and b/web/docs/src/assets/img/mem1.png differ diff --git a/web/docs/src/assets/img/mem2.png b/web/docs/src/assets/img/mem2.png new file mode 100644 index 0000000..7529e7d Binary files /dev/null and b/web/docs/src/assets/img/mem2.png differ diff --git a/web/docs/src/content/docs/architecture/actor.md b/web/docs/src/content/docs/architecture/actor.md index f292217..161cc25 100644 --- a/web/docs/src/content/docs/architecture/actor.md +++ b/web/docs/src/content/docs/architecture/actor.md @@ -7,7 +7,7 @@ The Actor Agent is the system component that embodies a single entity and produc ## Design Principles -1. **Epistemic boundedness** — The actor only sees what its entity would perceive: public attributes of other entities, private attributes explicitly ACL'd to it, its own memory buffer, and co-located entities. It does not have system-level access to all world state. +1. **Epistemic boundedness** — The actor only sees what its entity would perceive: public attributes of other entities, private attributes explicitly ACL'd to it, its own Cognitive Buffer, and co-located entities. It does not have system-level access to all world state. 2. **Proposal, not mutation** — The actor generates a _proposal_ (narrative prose). It never mutates world state, persists to the database, or writes to memory directly. Validation, execution, and persistence are the Architect's job. @@ -38,7 +38,7 @@ Epistemically bounded, with these sections: | ---------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------------- | | Current moment | The subjective present time | `worldState.clock.get().toISOString()` | | The world as you perceive it | Self-visible attributes, co-located entities + their visible attributes, other presences elsewhere | `serializeSubjectiveWorldState()` | -| Your recent memory | Recent `BufferEntry`s, alias-substituted, with relative time phrasing | `serializeSubjectiveBufferEntry()` | +| Your cognitive buffer | Recent `BufferEntry`s, alias-substituted, with relative time phrasing | `serializeSubjectiveBufferEntry()` | No system UUIDs, no private attributes the entity lacks ACL access to, and no objective-world-state dump are present. diff --git a/web/docs/src/content/docs/architecture/handoff.md b/web/docs/src/content/docs/architecture/handoff.md index 3a6e11b..23db2d3 100644 --- a/web/docs/src/content/docs/architecture/handoff.md +++ b/web/docs/src/content/docs/architecture/handoff.md @@ -1,15 +1,15 @@ --- title: Memory Handoff Pipeline -description: The Tier 1 (Working Buffer) to Tier 2 (Long-Term Ledger) memory promotion architecture +description: The Cognitive Buffer to Memory Ledger promotion architecture sidebar: order: 6 --- :::tip[Status: Fully Implemented] -The handoff pipeline is fully integrated into the simulation step loop. The `HandoffEngine` and `checkHandoffTrigger` components process working memory entries, promoting them to the persistent episodic Ledger, while safely pruning the subjective working memory buffer. +The handoff pipeline is fully integrated into the simulation step loop. The `HandoffEngine` and `checkHandoffTrigger` components process Cognitive Buffer entries, promoting them to the persistent Memory Ledger, while safely pruning the Cognitive Buffer. ::: -The **Handoff** pipeline manages the promotion of subjective experiences from the short-term working memory (Tier 1 Buffer) to long-term episodic memory (Tier 2 Ledger). This process regulates context window utilization by deciding **when** promotion occurs, **how much** of the buffer is processed, and **what** details are preserved or pruned. +The **Handoff** pipeline manages the promotion of subjective experiences from the **Cognitive Buffer** to the **Memory Ledger**. This process regulates context window utilization by deciding **when** promotion occurs, **how much** of the Cognitive Buffer is processed, and **what** details are preserved or pruned. To maintain performance and prevent context bloat, the pipeline separates the process into three decoupled stages: @@ -46,7 +46,7 @@ Voluntary triggers identify natural narrative boundaries where the entity is ina Involuntary triggers enforce context constraints before building the next prompt to prevent token limit overflows: -- **Buffer Ceiling**: The serialized length of the working memory section (as generated by `getMemorySectionLength`) exceeds a configured fraction (default: 60%) of the provider's context window. +- **Buffer Ceiling**: The serialized length of the Cognitive Buffer section (as generated by `getMemorySectionLength`) exceeds a configured fraction (default: 60%) of the provider's context window. - **Event Velocity**: The number of buffer entries exceeds a safe threshold (default: 20), indicating an event-heavy scene generating history faster than voluntary boundaries occur. If the buffer is empty, trigger detection exits immediately with `"none"`. @@ -55,13 +55,13 @@ If the buffer is empty, trigger detection exits immediately with `"none"`. ## 2. Watermark Partitioning -To preserve short-term narrative continuity and avoid immediate memory decay, recent events are protected from handoff. The pipeline divides the active buffer into a protected **Watermark Tail** and a **Candidate Pool**. +To preserve narrative continuity and avoid immediate memory decay, recent events are protected from handoff. The pipeline divides the active Cognitive Buffer into a protected **Watermark Tail** and a **Candidate Pool**. The watermark boundary is computed as: $$\text{Watermark} = \max(K \text{ entries}, \text{entries bucketed as fresh by } \textit{naturalizeTime})$$ -- **Hard Floor ($K = 8$)**: The last 8 entries in the buffer are always preserved. +- **Hard Floor ($K = 8$)**: The last 8 entries in the Cognitive Buffer are always preserved. - **Temporal Freshness**: Any entries that `naturalizeTime` classifies within the immediate temporal horizon (`"just now"`, `"moments ago"`, `"a few minutes ago"`, or `"several minutes ago"`) are also protected. Entries older than the watermark boundary constitute the **Candidate Pool** and are eligible for handoff processing. @@ -100,8 +100,8 @@ The prompt instructs the model to apply the following cognitive operations: When a chunk represents an unresolved high-stakes situation (e.g., an active conflict or standing threat), the LLM sets `retainInBuffer: true`. -- The summary is committed to the long-term Ledger as normal. -- The source buffer entries are exempted from pruning, remaining in the working memory buffer with `pinned: true` until a future handoff pass determines the thread is resolved. +- The summary is committed to the Memory Ledger as normal. +- The source Cognitive Buffer entries are exempted from pruning, remaining in the Cognitive Buffer with `pinned: true` until a future handoff pass determines the thread is resolved. --- diff --git a/web/docs/src/content/docs/architecture/memory-tier2.md b/web/docs/src/content/docs/architecture/memory-tier2.md index f1ff716..2caeae4 100644 --- a/web/docs/src/content/docs/architecture/memory-tier2.md +++ b/web/docs/src/content/docs/architecture/memory-tier2.md @@ -1,15 +1,15 @@ --- -title: Tier 2 Memory (Long-Term Ledger) +title: Memory Ledger description: Persistent episodic memory storage, semantic indexing, and retrieval mechanics --- -**Tier 2 Memory (the Ledger)** represents an entity's long-term episodic memory. It archives historical summaries of past events, providing a persistent record of character experiences that can be retrieved and injected into LLM prompts as needed. +**Memory Ledger** represents an entity's persistent episodic memory. It archives historical summaries of past events, providing a durable record of character experiences that can be retrieved and injected into LLM prompts as needed. --- ## 1. Data Model -A long-term memory is represented by a `LedgerEntry`. It includes metadata for deterministic database-level filtering, structured narrative content, and vector embeddings for semantic similarity scoring. +A Memory Ledger entry is represented by a `LedgerEntry`. It includes metadata for deterministic database-level filtering, structured narrative content, and vector embeddings for semantic similarity scoring. ```ts interface LedgerEntry { @@ -30,7 +30,7 @@ interface LedgerEntry { ## 2. Storage Model -To support fast, low-latency queries across large historical datasets, Tier 2 memory is stored in standard SQLite tables. Vector embeddings are stored in raw binary format as `Float32Array` BLOBs. +To support fast, low-latency queries across large historical datasets, Memory Ledger entries are stored in standard SQLite tables. Vector embeddings are stored in raw binary format as `Float32Array` BLOBs. Standard secondary indices optimize query execution time to microseconds, eliminating the compilation and cross-platform installation overhead of native vector database extensions (such as `sqlite-vec` or `node-gyp` binaries). @@ -64,7 +64,7 @@ CREATE INDEX IF NOT EXISTS idx_ledger_involved_entity ON ledger_involved_entitie ## 3. The Handoff Pipeline -Working memory (Tier 1 Buffer) entries are promoted to the Ledger through the automated [Handoff Pipeline](./handoff). +**Cognitive Buffer** entries are promoted to the Memory Ledger through the automated [Handoff Pipeline](./handoff). During handoff: @@ -72,7 +72,7 @@ During handoff: 2. Ambient stage business and redundant details are pruned. 3. Chunks are synthesized into third-person summaries and assigned importance scores. 4. Text embeddings are generated for the summary. -5. The processed memories are committed to the SQLite store, and the short-term buffer is pruned. +5. The processed memories are committed to the SQLite store, and the Cognitive Buffer is pruned. --- @@ -116,13 +116,13 @@ Once the candidate pool is loaded, the ranking engine evaluates entries in appli ## 5. Active Focus & Attention Loop -In crowded settings (e.g., a room with many characters), retrieving long-term memories for all co-located entities would exhaust the prompt context window. To prevent this, retrieval uses an **Active Focus** selection strategy: +In crowded settings (e.g., a room with many characters), retrieving Memory Ledger entries for all co-located entities would exhaust the prompt context window. To prevent this, retrieval uses an **Active Focus** selection strategy: -- **Active Focus Set**: The prompt builder scans the last 10 entries of the character's working memory buffer. Any entity targeted by, spoken to, or mentioned in these entries is added to the Active Focus set. +- **Active Focus Set**: The prompt builder scans the last 10 entries of the character's Cognitive Buffer. Any entity targeted by, spoken to, or mentioned in these entries is added to the Active Focus set. - **Dynamic Capacity Limits**: - - If the number of co-located characters is small ($\le 3$), long-term memory is retrieved for all of them. - - If the environment is crowded ($> 3$), long-term retrieval is strictly restricted to the top 3 characters in the Active Focus set. -- This creates a realistic attention loop: when a new character interacts with the actor, they enter the working buffer, triggering the retrieval of their long-term history on the subsequent turn. + - If the number of co-located characters is small ($\le 3$), Memory Ledger entries are retrieved for all of them. + - If the environment is crowded ($> 3$), Memory Ledger retrieval is strictly restricted to the top 3 characters in the Active Focus set. +- This creates a realistic attention loop: when a new character interacts with the actor, they enter the Cognitive Buffer, triggering the retrieval of their Memory Ledger history on the subsequent turn. --- @@ -130,15 +130,15 @@ In crowded settings (e.g., a room with many characters), retrieving long-term me Retrieved ledger entries are formatted into the prompt chronologically and mapped to subjective aliases. Internal metrics (e.g., importance numbers and raw system IDs) are omitted to preserve immersion. -- Working memory entries are injected under `=== RECENT EVENTS ===` (narrated relative to the present moment). -- Recalled ledger entries are injected under `=== YOUR MEMORIES ===` (presented as long-term recollections). +- Cognitive Buffer entries are injected under `=== COGNITIVE BUFFER ===` (narrated relative to the present moment). +- Recalled Memory Ledger entries are injected under `=== MEMORY LEDGER ===` (presented as episodic recollections). ```text -=== RECENT EVENTS === +=== COGNITIVE BUFFER === Moments ago - You spoke to Strider: "Hello there" -=== YOUR MEMORIES === +=== MEMORY LEDGER === A couple days ago - You met a hooded figure named Strider at The Prancing Pony. Quote: "I can avoid being seen, if I wish, but to disappear entirely, that is a rare gift." diff --git a/web/docs/src/content/docs/architecture/memory.md b/web/docs/src/content/docs/architecture/memory.md index 88707ea..c36633c 100644 --- a/web/docs/src/content/docs/architecture/memory.md +++ b/web/docs/src/content/docs/architecture/memory.md @@ -36,14 +36,14 @@ Entity aliases are persisted in the `objects` table via the `aliases_json TEXT` ## Subjective Buffer Entry -A subjective `BufferEntry` records a discrete event from the perspective of an entity (the `owner`). It wraps a structured `Intent` and appends execution metadata. +A subjective `BufferEntry` records a discrete event from the perspective of an entity (the `owner`). It wraps a structured `Intent` and appends execution metadata. These entries make up the entity's **Cognitive Buffer**. ### The Shape of a Buffer Entry ```typescript interface BufferEntry { id: string; - ownerId: string; // Whose subjective memory buffer this lives in + ownerId: string; // Whose Cognitive Buffer this entry lives in timestamp: string; // WorldClock.get().toISOString() at write time locationId: string | null; // Actor's location when this happened @@ -61,7 +61,7 @@ interface BufferEntry { ## Buffer Serialization (Epistemic Substitute) -To prevent leaking system IDs, buffer memories are serialized using `serializeSubjectiveBufferEntry` with the `resolveAlias` helper: +To prevent leaking system IDs, Cognitive Buffer entries are serialized using `serializeSubjectiveBufferEntry` with the `resolveAlias` helper: ```typescript export function resolveAlias(viewer: Entity, targetId: string): string { @@ -89,7 +89,7 @@ CREATE TABLE IF NOT EXISTS buffer_entries ( ``` - **JSON Storage**: `intent` and `outcome` are serialized/deserialized as raw JSON, validated by Zod at creation time. -- **Cascade Deletes**: Deleting an entity removes all associated subjective memory entries. +- **Cascade Deletes**: Deleting an entity removes all associated Cognitive Buffer entries. ## Time Naturalization diff --git a/web/docs/src/content/docs/architecture/overview.md b/web/docs/src/content/docs/architecture/overview.md index a18f9ec..a71a1fc 100644 --- a/web/docs/src/content/docs/architecture/overview.md +++ b/web/docs/src/content/docs/architecture/overview.md @@ -12,7 +12,7 @@ omnia/ intent/ intent types (dialogue/action/monologue) and the prose decoder architect/ World Architect: LLM validation plus time-delta generation actor/ actor agent: epistemically-bounded prompts, pluggable prose generators - memory/ verbatim buffer; later the vector archive, dossier, and affect vectors + memory/ Cognitive Buffer; Memory Ledger (vector archive), dossier, and affect vectors spatial/ location and POI graph, portal-based perception llm/ ILLMProvider interface plus Gemini and deterministic mock implementations scenario/ scenario JSON schema and loader (JSON → SQLite) diff --git a/web/docs/src/content/docs/diary/2_Finalizing_memory.md b/web/docs/src/content/docs/diary/2_Finalizing_memory.md new file mode 100644 index 0000000..36a2395 --- /dev/null +++ b/web/docs/src/content/docs/diary/2_Finalizing_memory.md @@ -0,0 +1,42 @@ +--- +title: I'm done with confusing memory jargon +author: sortedcord +time: 2026-07-18T16:12:00+05:30 +--- + +Memory has always been a core part of Omnia's architecture. I've always leaned towards a three tier system and it feels good to finally have a system frozen that's not subject to any more dramatic shifts. + +Up until now, I had been juggling terms like "short-term", "long-term", "buffer", "ledger" and memory quite randomly and interchagebly. Not just that, I think even [NLAVS](https://github.com/sortedcord/NLAVS) is also guilty of this. + +## How it was + +![A simple diagram showing three stacked horizontal boxes with orange outlines and text, labeled "Tier 1: Short Term" at the top, "Tier 2: Medium-ish long term?" in the middle, and "Tier 3: Long Term" at the bottom.](../../../assets/img/mem1.png) + +Notice how the tier 2 bar is the longest here. This made sense to me. You'd have a short term memory structure, ideally for events that are happening in real time and this would be small. At any given time you can't realistically carry a lot of information. And then at the bottom we have the long term tier. And just to make the graph complete, there's this intermediate layer which would contain most of the memories of an agent. + +Sounded right on markdown and looked cool in excali. But once you start talking about how they actually work, how does a unit of memory actually go between these layers is when all hell breaks loose. This wasn't something I paid much attention towards. + +## A colorful diagram that's a wanna be complex flowchart + +While developing [Handoff](../architecture/handoff.md); which is the mechanism for promoting "short term" memories to "long term" memories I realized that blindly modelling this after how humans are supposed to remember isn't a good approach to this, or rather, I'd say is a naiive solution to a data engineering problem. + +![A flow chart illustrating a three-tiered system where "Tier 1: Cognitive Buffer" flows into a "Handoff Call" box, which then connects to both the "Tier 2: Memory Ledger" and the "DossierDelta Generator"; simultaneously, both the "Tier 1: Cognitive Buffer" and the "Tier 2: Memory Ledger" also feed into the "DossierDelta Generator," which subsequently leads to "Tier 3: Dossiers," a large vertical box that also has a feedback loop returning to the "Tier 2: Memory Ledger."](../../../assets/img/mem2.png) + +Don't let this fool you. I didn't just move boxes around, add new boxes and renamed stuff and called it a day (even though that's what i did ;). But the main realization I had and the main framework that Omnia will follow from now on out is that **memories for an agent shouldn't be discrete rather they should be episodic.** + +When one remembers something, they don't remember it in isolated bits and pieces. You remember them in sequences that are chained together. Remembering something often leads to remembering events that either caused it or were caused by it. Like a chain. This chain is the episode this is what makes Omnia's memory system different: **Memory isn't pure hierarchy.** + +Each tier of memory serves its own purpose and has its own benefits and trade-offs but integrating them well together allows them to complement each other quite well. + +Putting this diagram into words: + +1. [**Tier 1: Cognitive Buffer**](../architecture/memory.md) is basically how the agent perceives its surroundings as-is. It is analgoues to a vision to text thing where whatever the agent "sees" is written down here, in the sense that this is high resolution and very verbose. So it contains actions, dialogues of others and itself but also its own thoughts, feelings, monologues. +2. [**Handoff**](../architecture/handoff.md) is what determines how important each event in the cognitive buffer is to be promoted to Memories. It's also responsible for compressing multiple actions into a single Memory Ledger entry. +3. [**Tier 2: Memory Ledger**](../architecture/memory-tier2.md) contains the bulk of an agent's memory. All memories are stored by when it happened, where it happened and who/what were involved in it. Given the size it will grow to, there was a need to implement a retrieval system (more than 1 of them). +4. [**Tier 3: Dossiers**](../architecture/) is the component of Omnia that allows for a per-actor subjective reality mapping. Every fact an actor knows whether it's about themselves or something else is stored as a dossier. + +Dossiers also dictate what series of memories from the memory ledger populate the context for an actor allowing for episodic retreival of memories alongside the general ledger retriever that works on semantic similarity. + +The main difference between the cognitive buffer+memory ledger and dossiers is that the former are immutable. Once something happens, it stays in the memory. It can become less relevant or harder to remember overtime (through time decay) but it stays in the memory as an immutable form. + +Dossiers however, are mutable. They are an ever-changing object in the actor's memory that define what they think about something at that intance of time. This 'thought' or 'impression' can be about anything - themselves, another actor, location, abstract idea, etc.