8. Containers: Systemd Runes

Context and Problem Statement

The LychD architecture functions as a "Sepulcher"—a unified pod of interconnected services including the primary Vessel, the persistent Phylactery, and a dynamic federation of extensions. Orchestrating this complex environment on a single Linux host requires a system that is declarative, resilient, and natively integrated with the operating system's lifecycle. A primary challenge involves the management of finite hardware resources, specifically GPU VRAM; the system must group inference containers into atomic, mutually exclusive Operational States to prevent hardware contention and Out-of-Memory (OOM) failures. Furthermore, as AI services are often multi-faceted (e.g., providing both Vision and OCR), the infrastructure definition must be capable of expressing nuanced, overlapping capabilities while maintaining deterministic deployment and transactional safety.

Requirements

Host-Native Orchestration: Mandatory integration with the operating system's init system (Systemd) to manage service lifecycles and recovery.
Atomic Resource States: Capability to group container definitions into Covens—mutually exclusive operational states managed as single units.
Semantic Capability Tagging: Support for assigning multiple functional tags (e.g., vision, reasoning, stt) to a single container for intelligent discovery.
The Law of Exclusivity: Physical enforcement of container conflicts at the kernel/init level to ensure deterministic resource allocation.
Declarative Blueprinting: Automated generation of immutable infrastructure definitions (Runes) from the user's central configuration.
Identity Symmetry: Native resolution of the host/container UID permission mismatch to allow seamless interaction with persistent volumes.
Transactional Inscription: Infrastructure updates must be atomic; a failed configuration ritual must not leave the host in a non-bootable or inconsistent state.

Considered Options

Rejected: K3s (Kubernetes)

A lightweight Kubernetes distribution was considered for its robust, declarative orchestration.

Pros:
- Powerful, industry-standard orchestration with features like health checks and auto-restarts.
Cons:
- Excessive Complexity. Introduces a massive architectural overhead (control plane, networking overlays) for a single-host system, violating the Host-Native Orchestration requirement.
- Lifecycle Misalignment. Runs parallel to the host's init system rather than integrating with it, creating two separate sources of truth for service management.

Rejected: Docker Compose

The most common tool for defining multi-container applications.

Pros:
- Simple, widely understood YAML syntax.
- Large community and extensive documentation.
Cons:
- No Native Exclusivity. Lacks any mechanism to enforce the Law of Exclusivity at the kernel level. Managing resource states would require fragile, imperative scripting to stop and start compositions.
- External to Host Lifecycle. Operates outside the host's init system, failing the Host-Native Orchestration requirement and complicating recovery and boot-time management.

Chosen: Podman Quadlets (Systemd Runes)

Leveraging Podman's ability to generate Systemd unit files from a simple definition.

Pros:
- Deep OS Integration. Treats containers as first-class Systemd services, perfectly fulfilling the Host-Native Orchestration requirement.
- Kernel-Enforced Exclusivity. Natively implements the Law of Exclusivity using Systemd's Conflicts= directive, providing a deterministic, hardware-aware state management solution.
- Atomic & Declarative. Enables a fully declarative blueprint that can be updated transactionally, satisfying all core architectural requirements.
- Seamless Permissions. Inherently supports the UserNS=keep-id mapping for perfect Identity Symmetry.

Decision Outcome

Podman Quadlets are adopted as the exclusive orchestration mechanism. These definitions, referred to as Runes, serve as the physical blueprint of the Daemon. They are organized into Covens for state management and tagged with Capabilities for semantic discovery by the internal dispatcher.

1. The Runic Hierarchy

The Sepulcher is organized into a strict hierarchy managed by the host's init system:

The Pod (lychd.pod): A shared network and resource namespace that forms the physical boundary of the Sepulcher. It encapsulates all core and extension services.
The Coven Target (lychd-coven-*.target): An optional meta-unit generated only for Covens with multiple members. It provides a convenient "master switch" for the Orchestrator (23) to start or stop an entire group of services with a single command.
The Core Runes: Persistent services essential for the system's existence:
- vessel.container: The primary application kernel.
- phylactery.container: The persistent PostgreSQL/PgVector engine.
- oculus.container: The observability and tracing stack (Arize Phoenix).
The Extension Runes: Dynamic services defined by installed organs. Each Rune matches a logical Animator by name.
The Portals: Logical bridges to remote APIs. These are Animators but do not manifest as physical Runes.

2. Capabilities & Tags: The Soul of the Animator

A provider (Soulstone or Portal) is defined not merely by its physical manifestation, but by the abstract services it provides to the agentic cortex. This metadata is maintained at the logical Animator level within the Pydantic schemas, rather than being inscribed into the physical unit files.

Capabilities (Interfaces): The calling convention or protocol (e.g., OpenAIChatCompletions, TTS).
Tags (Abilities): Architectural features used for routing (e.g., vision, tools).

While Covens define the Rune's physical relationship to other services for the Orchestrator (23), these semantic identifiers define its logical function for the Dispatcher (22).

Discovery: This metadata is the primary data source used by the Dispatcher (22) to map an Agent's abstract intent to a specific Animator.
Nuanced Provisioning: This allows the cortex to identify when a single, powerful Rune (like a multimodal VLM) can satisfy multiple requirements simultaneously, minimizing unnecessary container startup overhead.

3. Covens: The Law of Exclusivity

To manage finite hardware, Runes are organized into Covens. The system operates under a principle of "Implicit Exclusivity, Explicit Alliances": by default, every Coven is hostile to every other Coven. The user's role is not to define conflicts, but to declare alliances.

The Coven (groups): A Rune belongs to one or more Covens, as defined by the groups key in its Codex (12) Soulstone definition. This is its primary identity.
The Alliance (alliances): The Magus declares which Covens are "friends" and permitted to run concurrently by listing them together in the global alliances array within lychd.toml. Any Coven not explicitly allied is considered mutually exclusive with all non-allied Covens.

This design minimizes user configuration while maximizing safety. The user only needs to think about what should run together; the system assumes everything else cannot.
Automated Conflict Resolution (The Hybrid Covenant): The Rune Scribe via (lychd bind) translates the user's alliances into an ironclad set of Systemd rules that are robust against both orchestrated commands and manual user error.

1. The Ambassador's Pact (Pragmatic .target Generation): To provide a convenient "master switch" for the Orchestrator (23), a Systemd .target is generated only for Covens with two or more members. "Covens of One" do not generate this extra file, preventing unnecessary bloat.

2. The Citizen's Oath (Explicit Conflicts): The true source of robustness lies within each individual Rune. The RuneScribe calculates every Coven's enemies based on the alliances list. It then forges a brutally explicit Conflicts= directive in each .container file. This directive intelligently lists: - The .target of any multi-member enemy Coven. - The individual .service of any single-member enemy Coven.
```
# Example: hermes.container (from a Coven of One: 'reasoning')
# The RuneScribe determined 'vision' (multi-member) and 'tts' (single-member) are enemies.
[Unit]
Conflicts=lychd-coven-vision.target lychd-soulstone-xtts.service
```
State Transition: This explicit, low-level conflict definition guarantees deterministic state transitions. A high-level command from the Orchestrator or a "rogue" manual start of a single service both trigger the same, unbreakable conflict check at the kernel level. This prevents resource overload and ensures the system's physical state can never become corrupted.

4. Intra-Coven Dependencies (The Chain of Command)

While Covens and Alliances define the high-level rules of mutual exclusivity, they do not specify the startup order within a Coven or between allied services. A complex pipeline, for example, may require a pre-processor to be active before the main analysis model starts.

To address this, the Magus can specify standard Systemd ordering and dependency directives directly within a Soulstone's definition in the Codex (12).

User-Defined Ordering: The user can add keys such as after, wants, and requires to their Soulstone TOML. The values should be a list of the full Systemd service names of the dependencies.

Example: A vision pipeline where an analyzer model requires a preprocessor to be running first.

# In soulstones/vision.toml
[vision-preprocessor]
image = "..."
groups = ["vision"]

[vision-analyzer]
image = "..."
groups = ["vision"]
# This service needs the preprocessor to be running first.
after = ["lychd-soulstone-vision-preprocessor.service"]
requires = ["lychd-soulstone-vision-preprocessor.service"]

Direct Translation: The Rune Scribe will directly translate these keys into their corresponding After=, Requires=, and Wants= directives in the generated .container file for that Rune.

This design adheres to the principle of User Sovereignty. LychD does not attempt to magically infer dependencies. It provides a transparent bridge to the powerful, battle-tested dependency management engine of Systemd, giving the Magus full and explicit control over their system's boot logic.
Target Interaction: This mechanism works seamlessly with Coven Targets. When a user runs systemctl start vision.target, Systemd resolves the entire dependency graph, respecting all After= and Requires= directives, and starts the services in the correct order.

5. Federated Rune Registration

The Runic Hierarchy is not limited to the Core kernel. The system supports Inversion of Control for infrastructure:

Registration Hook: Extensions provide their own infrastructure blueprints by invoking context.add_rune(RuneDefinition) during the assimilation phase.
Unified Scribing: The CLI (19) treats core runes and extension runes as a single, flattened manifest, ensuring that the Law of Exclusivity and Port Arbitration are enforced across the entire organism.

6. Networking and Port Arbitration

Network management is arbitrated by the Pod unit to ensure collision-free internal and external communication.

Shared Namespace: All containers within lychd.pod share the localhost interface, enabling high-performance internal communication via standard ports.
Dynamic Exposure: The ContainerRune schema includes an ExposePort flag. When enabled, the Scribe adds the Rune's port mapping to the PublishPort directive of the main lychd.pod, granting it visibility to the host or the Proxy (40).

7. Identity Symmetry

To solve the "Permission Paradox," all generated Runes utilize the UserNS=keep-id mapping.

Mechanism: This instructs Podman to map the host user's UID/GID directly to the same ID inside the container.
Result: The process inside the container runs with the exact identity of the Magus. This ensures that mounted volumes in the Crypt (13)—including source code and database files—are accessible without permission errors or insecure host-side permission overrides.

8. The Rite of Atomic Inscription

To prevent systemic corruption—where a crash during configuration leaves the host in an unbootable state—the Scribe implements a transactional update ritual.

The Shadow Phase: All new Runes are first inscribed into a temporary "Shadow" directory.
The Atomic Swap: Only upon the successful generation of the entire manifest does the Scribe perform a rapid cleanup and move the new files to the active Binding Site (~/.config/containers/systemd/). This ensures the system always transitions between two valid, bootable states.

9. Runes as Manifestations of Animators

Runes are the physical manifestation of the Soulstones defined in the Codex.

Mapping by Name: The Soulstone (logical) and ContainerRune (physical) are paired by their shared identifier. The Scribe ensures the container_name and service_name derive predictably from the Soulstone's name.
Metadata Decoupling: Physical Runes do not carry capability or tag metadata in their unit files. This logic is handled at the Animator layer for the Dispatcher.

Consequences

Positive

Intelligent Resource Scaling: Multi-capability tagging allows the system to satisfy complex intents with the minimum number of active containers.
Hardware Determinism: GPU VRAM is strictly managed by the host init system, preventing resource contention and unrecoverable OOM crashes.
Adaptable Security: The keep-id mapping provides a seamless security bridge, maintaining the rootless posture while enabling effortless filesystem interaction.
Operational Reliability: The Atomic Inscription ritual guarantees system integrity, even if the binding process is interrupted by power loss or failure.

Negative

Linux Ecology Lock-in: This architecture binds LychD irrevocably to Linux distributions utilizing Systemd and Podman.
Orchestration Overhead: The interplay between semantic Capabilities and physical Covens requires a sophisticated Orchestrator to mitigate state-swap latency.