sorcery-go/docs/ARCHITECTURE.md

Sorcery-Go Architecture

1. The Core Thesis

Traditional source-based package management suffers from sequential fragility: if a single shell script fails, the entire system state becomes ambiguous. Sorcery-Go replaces procedural "instructional scripts" with declarative state managed by Go-grade concurrency and ACID transactions.

2. The Three Pillars

I. Deterministic Isolation (The Sandbox)

• Old way: make install writes directly to the live root. If it fails, the system is tainted.
• Go logic: Every build occurs in an OverlayFS sandbox. The compiler sees the system, but the system never sees the compiler.
• Result: Installations are atomic — they either succeed completely and are committed, or they fail and vanish without a trace.

II. Relational Intelligence (The DAG)

• Old way: Procedural loops check dependencies one by one.
• Go logic: The entire Grimoire is indexed into a directed acyclic graph with feature-aware edges (Build / Runtime / Optional + sub-depends).
• Result: Parallelism is no longer a guess — the engine knows exactly which spells can be built simultaneously without a race condition.

III. The Single Source of Truth (BoltDB)

• Old way: Flat text files in /var/lib/sorcery that can be partially written or corrupted.
• Go logic: A transactional, ACID-compliant KV store (bbolt) journals every intent.
• Result: If the power cuts out, the recovery engine reads the journal and resumes the exact byte-stream where it left off.

3. System Map

                 +-----------------------------------------------+
                 |  The Coven (Firewall-Isolated Ley-Lines)       |
                 |  +----------+  +----------+  +--------+       |
                 |  | Master   |--| Worker A |--| Worker |       |
                 |  | Sanctum  |  | (x86_64) |  | B(arm) |       |
                 |  +----+-----+  +----------+  +--------+       |
                 |       |                                               |
                 |  +----v--------------------------------------------+  |
                 |  |  Cauldron (Cast Pipeline)                        |  |
                 |  |  +-- Sub-Depends Solver                          |  |
                 |  |  +-- Variant Hash (y/n + arch)                    |  |
                 |  |  +-- Sandbox (OverlayFS + NS)                    |  |
                 |  |  +-- Committer (atomic rename)                   |  |
                 |  +----+--------------------------------------------+  |
                 |       |                                               |
                 |  +----v-------+    +--------------------+             |
                 |  |  Tomb      |    |  Warding           |             |
                 |  | (Merkle    |<-->|  (eBPF Tomb Guard) |             |
                 |  |  CAS)      |    +--------------------+             |
                 |  +----+-------+                                      |
                 |       |                                               |
                 |  +----v--------------------------------------------+  |
                 |  |  Tablet (BoltDB)                                 |  |
                 |  |  +-- Journal (resume on reboot)                 |  |
                 |  |  +-- Manifests (file list/spell)                |  |
                 |  |  +-- Tablet (y/n answers)                       |  |
                 |  |  +-- Configs (variant hashes)                   |  |
                 |  +-------------------------------------------------+  |
                 |                                                        |
                 |  +--------------------+  +-----------+                  |
                 |  | Legal Sentinel     |  | Coven     |                  |
                 |  | (SPDX/SBOM)        |  | Mirror    |                  |
                 |  +--------------------+  | (WebUI)   |                  |
                 |                           +-----------+                  |
                 +--------------------------------------------------------+
                                  |
                                  v
                         +-----------------+
                         |  Sanctums        |
                         |  (LXC / Podman   |
                         |  / Firecracker   |
                         |  / baremetal)    |
                         +-----------------+

4. Lifecycle of a Spell

1. Query — User defines intent via CLI, TUI, or WebUI.
2. Resolve — Sub-Depends Solver ensures every library variant is compatible.
3. Forge — Master shards the build; workers execute in isolated namespaces. Distributed scheduling is delegated to Fester when a cluster is active.
4. Sign — Resulting binary is hashed and stored as a Sarcophagus in the Tomb.
5. Hydrate — Target Sanctums are atomically updated via reflink/hardlink swaps.
6. Verify — Warding recomputes the Merkle root on first execution; mismatch triggers quarantine.

5. Toolchain Synergy

Tool	Role	Logic
Quill	The Scribe	Type-safe metadata generation. No more sed hacking.
Sorcery	The Engine	High-concurrency worker-stealing build orchestration.
Cauldron	The Blacksmith	Image composition via JSON/YAML manifests.
Gaze	The Eye	Reverse-path inventory + SBOM export.
Warding	The Shield	eBPF Tomb Guard + OpenSnitch/Portmaster + firewall isolation.
Legal	The Lawyer	License policy + SBOM + attribution bundles.
BTC.sh	The Forge	Sovereign cross-compilation across 19 target architectures.

6. Why Go

• Static binaries — The entire toolchain is one binary. It can repair a system even if glibc or bash is broken.
• Concurrency — Goroutines handle thousands of package checks with minimal RAM.
• Low-level access — Direct syscall management of namespaces and mounts without external wrappers.

7. Deployment Targets

• LXC / LXD containers managed by Cockpit
• Podman rootless containers
• Firecracker micro-VMs
• Bare-metal Source Mage hosts (systemd or OpenRC)
• Immutable infrastructure (squashfs ISOs from the Cauldron)
• CI/CD pipelines (automated spell testing + essence promotion)

The eBPF Tomb Guard provides in-kernel enforcement that works uniformly across all of the above runtimes — no per-runtime profile syntax needed.

8. Architecture Summary

Layer	Component	Tech Stack
Logic	Sorcery-Go	Go (static binary)
Storage	Tomb	Content-addressable Merkle CAS
Scheduling	Fester	DAG-driven distributed build execution (port 8181)
Security	Warding	eBPF LSM + cgroup filters + firewall + OpenSnitch
Compliance	Legal Sentinel	SPDX / CycloneDX
Control	Coven Mirror	Cockpit + WebUI API (port 8080)