Skip to content

17. Packaging: The Synthetic Forge

Context and Problem Statement

The capability for autonomous evolution creates a fundamental substrate dilemma. A Lich is a composite organism, its physical body formed by merging disparate manifests (Python, Node, System) and infrastructure intents (Systemd Quadlets) into a single, cohesive runtime. Standard imperative container build cycles suffer from "Substrate Drift"—where external repository shifts or re-tagged base images cause the same source code to produce different binary artifacts over time. In the agentic era, the substrate must also preserve enough inspectable source for the Smith to repair coupled organs instead of freezing a premature extension ABI. A mechanism is required to resolve dependency conflicts and forge a new body for the Daemon that is both mathematically deterministic and synchronized with the machine's physical state.

Requirements

  • Multi-Manifest Synthesis: Discovery and merging of pyproject.toml (Python), package.json (Node), and tailwind.config.js from all active extensions into a single build context.
  • Infrastructure Inscription: Automatic generation of Systemd Quadlets (08) based on the Soulstone definitions in the Codex (12) and extension requirements.
  • Extension Injection: A formal hook mechanism allowing extensions to register system-level dependencies (e.g., C-libraries) and custom container requirements during the Federation (05) phase.
  • Deterministic Manifests: Generation of a "Synthesis Manifest"—a pinned record of every dependency and its cryptographic hash to ensure verifiable provenance.
  • Pluggable Forge Strategies: Support for both a Mundane Path (imperative Containerfile with Jinja-based injections) and a Absolute Path (Nix-based functional image construction).
  • The Great Seal: Explicitly Read-Only runner environments (chmod -R a-w) to prevent runtime tampering and enforce the separation between evolution and execution.
  • Source-Centric Assembly: Preservation of raw Python source files and docstrings to enable the runtime introspection required for self-reflection.
  • Assimilation Before ABI: Preservation of coupled source as the pre-v1 compatibility mechanism; public SDK/ABI surfaces are harvested only after stable patterns survive real Forge cycles.
  • Manual Transition Gate: Air-gapped activation of new images requiring a manual signal via the CLI to prevent autonomous "Infection and Restart" loops.

Considered Options

Option 1: Individual Extension Containers (Sidecars)

Running every extension in its own isolated container within the Pod.

  • Pros: Maximum isolation between logic components.
  • Cons: Extreme Resource Tax. Significant VRAM and CPU overhead for dozens of interpreters. It introduces network latency for internal calls and complicates the orchestration of Workers (14).

Option 2: Imperative Monolithic Build

Generating a single, giant Containerfile that installs extensions sequentially via shell scripts.

  • Pros: Conceptually simple; utilizes standard OCI tooling.
  • Cons: Non-Deterministic. Dependency conflicts between Extensions are only caught at runtime. It fails the standard for verifiable provenance required for a sovereign daemon.

Option 3: Two-Phase Synthetic Packaging

Utilizing a logical Synthesis phase followed by a pluggable Manifestation strategy.

  • Pros:
    • Logical Sanity: Resolves dependency math using native tools (uv and npm) before the physical build begins.
    • Infrastructure Synchronization: Ensures the Systemd Quadlets are regenerated to match the new code substrate.
    • Pluggable Evolution: Allows for a low-friction start with standard tools while providing an upgrade path to advanced functional construction.

Decision Outcome

Synthetic Functional Packaging is adopted as the definitive standard for the system substrate. The Forge operates in two distinct phases: logical convergence followed by physical binding.

1. The Synthesis Stage (Logical Convergence)

When a packaging ritual begins, the system performs a multi-dimensional synthesis by scanning both the Built-in registry and the Crypt (13) to prepare for the physical build:

  • Anatomical Grafting: The Manager discovers all Built-in Extensions and establishes the kernel's baseline runtime and substrate requirements through the in-tree registration path. This is an in-memory operation that defines the body the Forge must then manifest.
  • The Code Layer (Substrate Synthesis): All pyproject.toml (Python), package.json (Node), and tailwind.config.js manifests from active Crypt Extensions are merged with the core manifests. Near-term Crypt extensions are private coupled organs unless they explicitly target a future versioned public API. The system executes a frozen lock to create a single, deterministic source of truth for the Backend (11) environment.
  • Substrate Injection: During assimilation, extensions declare system-level dependencies (e.g., C-libraries like ffmpeg or specialized binaries) and custom container requirements as part of the composed-runtime law. These are collected into the global synthesis manifest to be Manifested during the Forge. The register(context) hook is only the boot-time grafting branch of that law.
  • The Infrastructure Layer: The system reads Soulstone intents from the Codex (12) and infrastructure requirements from all active Extensions. It dynamically calculates the lychd.pod configuration, aggregating all ExposePort requirements and hardware tags for Containers (08).
  • Global Arbitration: The Manager performs a mandatory conflict check across the entire manifest. It enforces the Law of Exclusivity, ensuring no port collisions, image-name overlaps, or dependency version deadlocks exist. Only upon successful arbitration are the "Dumb Blueprints" handed to the Quadlet Scribe to manifest the concrete Systemd Quadlet files.
  • Binary Organ Synthesis: Rust/PyO3 organs are a Forge-mediated future path. A compiled organ may be built into the composed image as a coupled organ, or later target a versioned public API once that product surface exists. The active runtime must not blindly scan .so files; binary artifacts require manifest pinning, platform validation, and explicit activation.

Why Multi-Language Synthesis Is Possible

Cross-language synthesis is possible because the Forge builds one composed runtime image and verifies it before promotion. It is not proof of a stable in-process ABI. Until LychD has a versioned public API and conformance suite, binary organs are either coupled to the composed image or isolated behind an external-service Animator boundary. An external-service Animator may expose model inference, tools, observability, peer delegation, or any other typed capability; the boundary is the service contract, not the presence of an LLM.

Pre-v1, packaging is the repair boundary: source and manifests are assembled together so the Smith can inspect and adapt the organ. At v1, repeated stable seams may be harvested into a public API. Post-v1, that API becomes a compatibility product rather than an assumption baked into infancy.

2. The Forge Strategies

The Mundane Path (Current Standard)

This is the primary mechanism for manifestation, utilizing a multi-stage Containerfile rendered via Jinja2.

  1. Injections: Extension-registered system dependencies are injected into the RUN apt-get install block of the template.
  2. Builder Stage: Mounts the uv binary and cache to perform a frozen sync of the synthesized manifests.
  3. Runner Stage: A hardened, non-root environment based on python-slim.
  4. The Seal: The /app directory is stripped of write permissions. PYTHONDONTWRITEBYTECODE=1 is set to ensure the source remains readable for Agentic introspection.

The Sovereign Path (The Nix Sigil)

This is the advanced, functional upgrade path for the image construction.

  1. Transmutation: Consumes the synthesized manifests and transmutes them into a functional derivation.
  2. Calculation: Nix calculates the filesystem structure into a local store, ensuring every binary is cryptographically pinned.
  3. OCI Construction: Nix manufactures the layered image directly from the store, bypassing the non-determinism of standard base images.

3. The Rebirth Gate

The resulting image is loaded into the local registry as lychd:custom. To ensure the Magus remains the ultimate arbiter, the activation of the new body is an air-gapped ritual. The system refuses to restart the container or apply the new Quadlets (08) until it receives a manual confirmation command via the CLI.

4. Dual-Plane Trust Delta

Packaging now emits two runtime classes:

  • Vessel image: trusted control plane.
  • The Tomb image: untrusted execution runtime. Carries Python, uv, nono, and common CLI tools only. No agent framework, no LLM client libraries, no graph runner dependencies.
  • Tomb dependency expansion uses curated cache/broker channels by default.

5. Authority Matrix

Dimension Vessel Artifact (Trusted Control Plane) The Tomb Artifact (Untrusted Execution Plane)
Secrets Runtime secret injection for control-plane duties only. No provider, signing, or control-plane secret injection. A Tomb worker profile may receive a queue-only SAQ/Postgres execution credential.
Mounts Trusted Codex/persistence mount contract. Minimal execution mounts; no writable Codex and no Codex-wide privileged mounts.
Network Controlled provider/control-plane access. Constrained egress; brokered resources preferred.
Queue Ownership Carries queue-capable control-plane components. Carries execution-plane queue claim/ack components only.
Authority Boundaries Participates in controlled rebirth signaling. Cannot trigger rebirth or infrastructure transitions.

Consequences

Positive

  • Mathematical Provenance: The system provides proof that the physical "Body" perfectly matches the instruction "Scroll."
  • Synchronized Reality: Infrastructure (Quadlets) and Logic (Code) are updated in a single, atomic ritual, preventing "Blindness" where code expects a port that is not published.
  • Predictable Evolution: Dependency conflicts between Extensions are caught at build-time, preventing runtime instability.

Negative

  • Build Latency: The synthesis and multi-stage build rituals are significantly slower than simple hot-loading.
  • Storage Pressure: Maintaining previous images and functional derivations increases the disk footprint of the Crypt.