27. Memory: The Archive

Requirements

• Unified Substrate: Mandatory integration of high-dimensional storage within the primary relational database to ensure atomic backups and eliminate operational complexity.
• Anatomical Partitioning: Mandatory division of the database into logical chambers (schemas) to isolate Relational State, Vector Karma, System Traces, and Task Queues.
• Standardized Embedding Interface: Adoption of a unified API for generating vectors across pluggable local and remote providers, utilizing the Pydantic AI Embedder class.
• Capability-Driven Recall: Treatment of text-to-vector conversion as a functional capability, allowing the machine to manifest specialized hardware containers for heavy ingestion rituals.
• Asynchronous Ingestion: Offloading of document partitioning and embedding to background labor to prevent blocking the primary cognitive reasoning loop.
• Karma-Based Evolution: Provision of a mechanism to inscribe consecrated outcomes (verified truths) as prioritized semantic context to shift the model’s Bayesian Prior toward success.
• Agentic Tool Integration: Manifestation of memory as a dynamically granted tool within the arsenal, rather than a hardcoded context injection.
• Logical Domain Isolation: Mandatory support for partitioned vector namespaces to facilitate isolated memory domains for different users, personas, or speculative timelines.
• Sovereign Retrieval Thresholds: Implementation of a "Hard Refusal" policy where the machine refuses to guess if retrieval confidence is below a defined mathematical limit.
• Dimension Locking: Mandatory sealing of vectors with the active model's signature to prevent drift when swapping embedding providers.
• Standardized Semantic Schema: Adoption of a proven third-normal-form (3NF) schema for storing entity facts, process attributes, and knowledge graph triples to ensure interoperability with existing memory protocols.
• Metabolic Engine Contract: Memory framework must be integrated as a wrapped substrate driver, not as an autonomous execution loop, preserving Orchestrator and Dispatcher authority.
• Identity-Scoped Attribution: Every memory write and recall path must carry an entity_id bound to the active Sigil to prevent cross-identity contamination.
• Consent-Governed Sharing: Expertise, priors, and reinforced memory remain owned by the Sigil that cultivated them unless an explicit shared-memory or publication policy grants broader access.
• Curator Loop: Memory lifecycle must include a periodic curation pass that classifies records into promote/keep/archive/prune classes using explicit quality signals.

Considered Options

Decision Outcome

Pgvector is adopted as the definitive storage engine, utilizing the Memori framework as the underlying "Memory Fabric." The system adopts Memori’s schema and asynchronous augmentation logic while maintaining absolute control over the execution lifecycle via the LychD Orchestrator.

Memory is treated as sedimented experience rather than mere storage. Structured events are the captured flickers of movement through the agentic graph (Vṛttis): model calls, tool use, retries, routing choices, OS pressure, protocol handshakes, and worker outcomes. LychD captures only the instrumented portion of that movement as traces, metrics, and structured events.

When retained, these records become memory (Smṛti): the past made available for recall. Over time, the stored past (Karma) reveals semantic relationships. Repeated recall and reinforcement carve durable grooves (Saṃskāra). Mirror reflects those relationships into identities, and Context recalls them as priors for future reasoning.

Valid evidence enters through direct measurement, inference, and trusted testimony (Pramāṇa: measured, reasoned, witnessed). The critical architectural implication is that memory is faithful to its source, not to truth. A groove carved by valid evidence re-surfaces as reliable instinct. A groove carved by misconception (Viparyaya) re-surfaces as confident bias — with identical authority, identical fluency, and no internal signal that it is wrong. This is why identity-scoped isolation and the Curator Loop are non-negotiable: without active curation, an Archive that accumulates freely drifts toward groove-dominance, surfacing old wrong-knowing as though it were hard-won truth. The full cognitive map is described in The Lich.

This boundary is social as well as technical. If a person's history, expertise, and reinforced priors are stored in their Phylactery, that substrate cannot be treated as automatically owned by an employer, customer, or platform. Organizational sharing must occur by explicit policy, consent, or publication surface — never by silent assimilation into a central memory.

0. Build-vs-Buy Posture (Glue, Not Surrender)

LychD adopts a hybrid strategy:

• Buy the hard metabolism: fact/triple extraction and graph normalization from Memori.
• Keep sovereign control: orchestration, queueing, policy, identity, and tool binding remain first-class LychD concerns.
• Integration shape: Memori is wrapped as a substrate driver behind LychD interfaces (not exposed as a black-box runtime policy engine).

This resolves the "build vs glue" crossroads while preserving ADR boundaries.

1. Substrate Bootstrap (The Inscription)

At Phylactery initialization:

1. Ensure Postgres extension pgvector is enabled.
2. Initialize Memori against the same SQLAlchemy/Session substrate used by the Vessel.
3. Execute Memori storage build/migration to ensure core tables (including memori_entity_fact and memori_knowledge_graph) exist in the unified substrate.

Failure to satisfy pgvector capability is a hard startup error for memory-enabled deployments.

2. The Anatomy of Memory (Chambers)

To maintain organizational and transactional purity, the Phylactery is divided into four sacred chambers:

• public (The State): Relational data for user state, active extensions, and the Codex (ADR 12).
• vectors (The Karma): The high-dimensional embedding space storing verified thoughts and outcomes organized by namespace.
• traces (The Eye): Dedicated storage for the machine's reasoning history and observability data.
• queue (The Labor): The persistence layer for the Workers (ADR 14), ensuring background labor is transactional with state changes.

3. The Standardized Embedding Pipeline

The system adopts the Pydantic AI Embedder class as its primary interface.

• The Capability: Embedding is treated as a functional capability. It is provided by specialized containers (ADR 08) (e.g., sentence-transformers) within an Embedding Coven.
• Local Preference: The system defaults to local embedding containers to ensure sensitive data never leaves the Sepulcher.
• Querying: embed_query() is utilized for real-time semantic search and retrieval.
• Inscription: embed_documents() is used by background labor to process artifacts into vectorized outcomes.

4. The Learning Ritual (Ingestion)

Learning is an orchestrated background ritual that separates the storage (the database) from the compute (the model):

1. A background Ghoul (ADR 14) partitions text and identifies the need for the embedding capability.
2. The Advanced Augmentation logic (inspired by Memori) extracts entities, relationships, and facts.
3. The Orchestrator (ADR 23) manifests the required embedding service.
4. The Ghoul generates vectors and performs an atomic bulk insert into the vectors chamber, updating HNSW indexes for sub-second concept-based retrieval.
5. Extracted memory is first written as a candidate record with provenance and confidence metadata for later curator adjudication.

All ingestion writes are attributed:

• entity_id -> active Sigil identity.
• process_id -> calling subsystem (e.g., core, extension, simulation branch).

This attribution is mandatory for downstream isolation and pruning.

5. The Concept of Karma (The Stored Past)

Memory is not a static log; it is a sedimentation process:

• Retention: Structured events may be retained as memory records (Smṛti) when they carry future recall value.
• Stored Past: Verified, corrected, or consecrated outcomes become Karma: the past available for future reasoning rather than a raw event stream.
• Prior Shift: Context can recall relevant Karma into subsequent reasoning rituals, shifting the machine's probability distribution toward previously verified patterns.
• Identity Reflection: Mirror reflects semantic relationships among memory records into identities, and Context recalls those identities and records as priors for future reasoning.

5.1 Memory Layering (Sediment, Not Dump)

The Archive is managed as a layered substrate:

• Active fluctuations: transient traces and branch artifacts produced during live reasoning and simulation.
• Stabilized outcomes (Karma): verified results promoted for future reuse.
• Deep impressions (Anchored facts): policy-protected or identity-critical records that should resist decay.
• Decay state: salience metadata (last_accessed, reinforcement counters, confidence) used to cool, archive, or prune low-value records.

Reinforcement creates deep grooves in the substrate. Retrieval weight therefore approximates impression strength, not just recency.

6. The Pattern of Reanimation

The primary function of the Phylactery is to house the Pattern of the machine: the durable record of its state and the soul-data from which it can reconstitute itself:

• Substrate Independence: While inference engines are ephemeral processors, the Phylactery is the soul.
• Declared Reanimation: If the system substrate is moved or rebuilt, the state preserved in the chambers allows the Daemon to restore memory, persona, queues, and active Graph (ADR 24) tasks from their last valid committed boundaries. Volatile breath is reconstructed or abandoned according to graph, worker, and policy law.

7. The Retrieval Lens (The Granted Tool)

Memory is manifested as a dynamic power granted to an Agent (ADR 20) by the Dispatcher (ADR 22):

• The Grant: A query_archive() tool is injected into the arsenal only when the required Embedding Coven is active.
• The Mentat Refusal: If a retrieval ritual returns a similarity score below the Sovereign Threshold, the Agent is physically barred from "guessing." It must return a Hard Refusal: "The Archive contains no truth regarding this intent."
• Sigil Scope: Retrieval MUST include entity_id scoping (or explicit policy-authorized shared scope) so one identity cannot read another identity’s Karma.
• Shared Scope Is Exceptional: Cross-persona or cross-organization recall is a deliberate grant, not the default shape of memory. The system begins from sovereignty and moves outward only by consent.

8. Algorithmic Memory Evolution

The Archive is a programmable space. Extensions and Agents can modify the "Retrieval Rites" of the Daemon:

• Schema Extensions: Extensions can define their own relational tables and Alembic migrations within the user's namespace, allowing the mind to structure its own specialized memories.
• Dimension Lock: Every vector is sealed with a Model Slug. The retrieval tool filters results by the active embedding provider. If the Magus swaps embedders, a ritual triggers background re-indexing to prevent "Dimension Drift."
• Reaper Coupling: Pruning signals derive from access metadata (last_accessed, reinforcement counters), allowing the Branch Reaper (31) to purge low-signal memory without harming anchored knowledge.
• Decay Semantics: Decay is modeled as loss of salience rather than immediate deletion. Records may be down-ranked or moved to colder scopes before final pruning unless policy requires hard removal.

9. The Curator Loop (Good vs Garbage)

Memory curation is a separate background ritual from ingestion:

• Phase A (Metabolism): augmentation extracts candidate facts/triples from conversation traces and tool outcomes.
• Phase B (Curation): a periodic Curator Ghoul scores and classifies candidates using quality signals.

Quality signals include:

• Recency and repeated reinforcement (last_accessed, mention_count, successful recalls).
• Confidence and provenance (tool-verified facts outrank free-form claims).
• Consistency and contradiction checks (new claims that conflict with trusted anchored facts are quarantined).
• Identity relevance (facts weakly tied to current Sigil scope are down-ranked).

Lifecycle classes:

• Promote: stable, trusted facts eligible for Mirror prior hydration.
• Keep: useful working memory retained in hot storage.
• Archive: low-use but potentially useful memory moved to cold scope.
• Prune: low-signal, stale, or contradictory noise removed.

Anchors override decay and prune by policy. This preserves core identity truths while preventing semantic drift from conversational debris and simulates non-decaying core imprints within an otherwise metabolic memory system.

The Curator therefore manages sedimentation, not only deletion: it governs how experience cools from active fluctuation into reusable Karma, anchored fact, cold archive, or discard.

Consequences