From b1e713336908b059f8740d970c7776b204d3a16b Mon Sep 17 00:00:00 2001 From: Jeremy Anderson Date: Sun, 12 Jul 2026 01:40:30 -0400 Subject: [PATCH] Sorcery-Go is a high-concurrency, source-based infrastructure management suite. It implements a spell format compatible with Source Mage GNU/Linux grimoires for convenience, but the two projects have separate maintainers, separate codebases, and separate governance. --- CHANGELOG.md | 9 +- Makefile | 4 +- docs/DISASTER_RECOVERY.md | 20 +- docs/ESSENCE_SPEC.md | 4 +- docs/GETTING_STARTED_SMGL_CHROOT.md | 2 +- docs/QUICKSTART.md | 38 ++-- docs/RITUAL_OF_CASTING.md | 10 +- docs/SECURITY.md | 4 +- manifests/lxc-sorcery-essence.apparmor | 33 ++- manifests/opensnitch-essence-sync.json | 4 +- manifests/sorcery-go.openrc | 19 +- pkg/tomb/reanimate.go | 92 ++++++++ pkg/tomb/storage.go | 302 +++++++++++++++++++++++++ pkg/tomb/storage_test.go | 102 +++++++++ scripts/deploy_grid.sh | 13 +- scripts/forge_first_sanctum.sh | 2 +- tablet/.gitkeep | 3 + tomb/.gitkeep | 4 + 18 files changed, 578 insertions(+), 87 deletions(-) create mode 100755 pkg/tomb/reanimate.go create mode 100755 pkg/tomb/storage.go create mode 100755 pkg/tomb/storage_test.go create mode 100755 tablet/.gitkeep create mode 100755 tomb/.gitkeep diff --git a/CHANGELOG.md b/CHANGELOG.md index 3dcbfc6..0ab71e4 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ All notable changes to Sorcery-Go are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [1.1.0] — 2026-07-12 ### Added - **BTC 0.4.0 multi-arch cross-compilation support** (`pkg/toolchain/btc.go`) @@ -32,12 +32,6 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `RetrieveArtifact` / `RetrieveArtifactToFile` — fetch cached artifacts - `Stats` — cache hit rate, utilization, artifact count - Auto-initialized when Fester integration is active -- **BTC.sh toolchain integration** (`pkg/toolchain/btc.go`) - - `Probe()` — detect BTC.sh golden images at /opt/BTC - - `BuildEnv()` — return BTC-aware CC/CXX/CFLAGS/LDFLAGS - - `VerifyStamp()` — read .note.BTC ELF note and xattr stamps - - `StampBinary()` — apply forensic stamps (ELF note, xattr, debug symbols) - - Pipe-delimited format matching BTC.sh and Fester - **Cauldron.BundleAndCache** — one-call bundle + CAS push method - **Scheduler.Dispatch now CAS-aware** — checks shared CAS before dispatching to Fester; skips builds for cached artifacts @@ -54,7 +48,6 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - **eBPF security** replaces AppArmor as primary enforcement mechanism - `tomb_guard.bpf.c` — LSM hooks for file/inode protection - `sorcery_filter.bpf.c` — cgroup filters for device/network whitelisting - - AppArmor profiles retained as fallback - **License** changed from GPL-3.0 to AGPL-3.0-or-later - **Fester cluster integration** — full delegation of distributed build scheduling, node telemetry, and build dispatch to Fester diff --git a/Makefile b/Makefile index d2b9f67..7c2f821 100755 --- a/Makefile +++ b/Makefile @@ -2,7 +2,7 @@ # "The Forge is hot. The Warding is strong. The Coven is ready." BINARY = sorcery -VERSION ?= 1.0.0 +VERSION ?= 1.1.0 BUILD_DIR := build GO_FLAGS := -ldflags="-s -w -X main.Version=$(VERSION)" CGO ?= 0 @@ -113,7 +113,7 @@ ebpf: # Source Mage chroot resurrection helpers # ============================================================================= # Thin wrappers around scripts/smgl-getting-started.sh so you can drive the -# whole 8-phase pipeline from make. See docs/GETTING_STARTED_SMGL_CHROOT.md +# whole 9-phase pipeline from make. See docs/GETTING_STARTED_SMGL_CHROOT.md # for the full walkthrough. # # Example: diff --git a/docs/DISASTER_RECOVERY.md b/docs/DISASTER_RECOVERY.md index aa29cc0..63e3e1d 100755 --- a/docs/DISASTER_RECOVERY.md +++ b/docs/DISASTER_RECOVERY.md @@ -40,7 +40,7 @@ allow traffic on the sorcery-go (8080) and Fester (8181) ports. ```bash # 1. Verify network connectivity to the new Master -sorcery coven pulse +sorcery-go coven pulse # 2. If workers were using a Fester controller, update its URL # (set in /etc/sorcery-go/config.yaml or SORCERY_GO_FESTER_URL env var) @@ -56,11 +56,11 @@ corrupted during the failure: ```bash # Deep Gaze — recompute the Merkle root of every file in the Tomb and # compare against the restored Tablet entries -sorcery tomb verify --all +sorcery-go tomb verify --all # Ghost Build — pick a core tool (like busybox) and re-forge it in a # temporary sandbox. The new binary must match the restored Essence. -sorcery cast busybox --static --default +sorcery-go cast busybox --static --default ``` ## IV. Restoring the Sanctums (LXC Containers) @@ -71,7 +71,7 @@ work for containers — the links are broken. Run the Re-Hydration Ritual: ```bash # Reads the Tablet to see which Essences belong in which containers, # then re-links them from the Tomb -sorcery reanimate --all-containers +sorcery-go reanimate --all-containers ``` ## V. Backup Strategy @@ -86,9 +86,9 @@ sorcery reanimate --all-containers After reconstruction, verify each layer: -- [ ] `sorcery coven pulse` — every node green -- [ ] `sorcery tomb verify --all` — zero Merkle mismatches -- [ ] `sorcery ward status` — eBPF Tomb Guard active -- [ ] `sorcery legal audit` — zero license violations -- [ ] `gaze whereis /usr/bin/bash` — owned by `coreutils` -- [ ] Cast a smoke-test spell: `sorcery cast busybox --static --default` \ No newline at end of file +- [ ] `sorcery-go coven pulse` — every node green +- [ ] `sorcery-go tomb verify --all` — zero Merkle mismatches +- [ ] `sorcery-go ward status` — eBPF Tomb Guard active +- [ ] `sorcery-go legal audit` — zero license violations +- [ ] `sorcery-go gaze whereis /usr/bin/bash` — owned by `coreutils` +- [ ] Cast a smoke-test spell: `sorcery-go cast busybox --static --default` \ No newline at end of file diff --git a/docs/ESSENCE_SPEC.md b/docs/ESSENCE_SPEC.md index c9a10a0..ee04301 100755 --- a/docs/ESSENCE_SPEC.md +++ b/docs/ESSENCE_SPEC.md @@ -86,7 +86,7 @@ x86_64 binaries into an AArch64 container: { "arch": "aarch64", "instruction_set": "ARMv8-A", - "merkle_root": "sha256:9f4e2a8b..." + "merkle_root": "9f4e2a8b..." } ``` @@ -97,7 +97,7 @@ expected to be PGP-signed by the High Mage: ```bash gpg --detach-sign -a grimoire/libs/openssl/DETAILS -sorcery cast openssl --verify-pgp +sorcery-go cast openssl --verify-pgp ``` The Tablet records who signed each spell — the WebUI shows a "Trusted Source" diff --git a/docs/GETTING_STARTED_SMGL_CHROOT.md b/docs/GETTING_STARTED_SMGL_CHROOT.md index f2ba6d7..d3b8998 100755 --- a/docs/GETTING_STARTED_SMGL_CHROOT.md +++ b/docs/GETTING_STARTED_SMGL_CHROOT.md @@ -35,7 +35,7 @@ to boot. --- -## The 8-phase pipeline +## The 9-phase pipeline | Phase | What | Where | Script subcommand | |-------|------|-------|-------------------| diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 26e2a95..28ab51e 100755 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -13,11 +13,11 @@ sudo make caps | Ritual | Command | Effect | |----------|--------------------------------------|-------------------------------------| -| Cast | `sorcery cast ` | Forge an Essence in the Cauldron | -| Reanimate| `sorcery reanimate --sanctum ` | Deploy an Essence from the Tomb | -| Gaze | `sorcery gaze install ` | Inspect every file owned by an Essence | -| Ward | `sorcery ward status` | Check firewall and alarm status | -| Banish | `sorcery dispel ` | Surgical removal from a Sanctum | +| Cast | `sorcery-go cast ` | Forge an Essence in the Cauldron | +| Reanimate| `sorcery-go reanimate --sanctum ` | Deploy an Essence from the Tomb | +| Gaze | `sorcery-go gaze install ` | Inspect every file owned by an Essence | +| Ward | `sorcery-go ward status` | Check firewall and alarm status | +| Banish | `sorcery-go dispel ` | Surgical removal from a Sanctum | ## 3. Cast Flags @@ -32,10 +32,10 @@ sudo make caps ## 4. Coven Management ```bash -sorcery coven join # Add a new Sanctum to the Coven -sorcery coven pulse # Live heartbeat of every node -sorcery coven drain # Migrate builds off a node for maintenance -sorcery coven list # Show all known Sanctums +sorcery-go coven join # Add a new Sanctum to the Coven +sorcery-go coven pulse # Live heartbeat of every node +sorcery-go coven drain # Migrate builds off a node for maintenance +sorcery-go coven list # Show all known Sanctums ``` Distributed build scheduling is delegated to Fester when a cluster is active. @@ -43,16 +43,16 @@ Distributed build scheduling is delegated to Fester when a cluster is active. ## 5. Tomb (Binary Storage) ```bash -sorcery tomb list # Show every Essence in the Tomb -sorcery tomb verify --all # Warding bit-rot check -sorcery tomb purge # Garbage-collect unreferenced blobs +sorcery-go tomb list # Show every Essence in the Tomb +sorcery-go tomb verify --all # Warding bit-rot check +sorcery-go tomb purge # Garbage-collect unreferenced blobs ``` ## 6. Warding (Security) ```bash -sorcery ward status # Active alarms + quarantined nodes -sorcery ward banish # Freeze a Sanctum and log the banishment +sorcery-go ward status # Active alarms + quarantined nodes +sorcery-go ward banish # Freeze a Sanctum and log the banishment ``` Transport security is handled at the network firewall layer (OPNsense / IPFire). @@ -61,10 +61,10 @@ There are no certificates or key material to manage. ## 7. Legal Sentinel ```bash -sorcery legal audit # Scan fleet for license violations -sorcery legal sbom # CycloneDX SBOM on stdout -sorcery legal set-posture lawless # strict_copyleft | corporate_lite | lawless -sorcery legal credits --out ./DIR # Collect every COPYING / LICENSE file +sorcery-go legal audit # Scan fleet for license violations +sorcery-go legal sbom # CycloneDX SBOM on stdout +sorcery-go legal set-posture lawless # strict_copyleft | corporate_lite | lawless +sorcery-go legal credits --out ./DIR # Collect every COPYING / LICENSE file ``` ## 8. Coven Mirror (WebUI) @@ -80,7 +80,7 @@ Tabs: Grimoire, Tomb, Pulse, Sanctum, Portable Bin, Compliance. ```bash # Forge a static, portable version of a tool -sorcery cast procps --static --target aarch64 --essence-out ./portable-top.ess +sorcery-go cast procps --static --target aarch64 --essence-out ./portable-top.ess # Generate the curated Emergency Kit cauldron emergency-kit --bundle ./emergency.svb diff --git a/docs/RITUAL_OF_CASTING.md b/docs/RITUAL_OF_CASTING.md index eb9d291..905942b 100755 --- a/docs/RITUAL_OF_CASTING.md +++ b/docs/RITUAL_OF_CASTING.md @@ -61,11 +61,11 @@ Now the Essence must be brought to life inside a Sanctum. When a ritual fails or a node acts strangely, use the Gaze to see the truth: ```bash -gaze install # "Show me every file this spell owns." -gaze tablet # "Show me exactly how this was configured." -gaze essence # "Which spell + flags produced this blob?" -gaze whereis /usr/bin/wget # "Who owns this file?" -sorcery ward status # "Is the Warding active? Any tainted Essences?" +sorcery-go gaze install # "Show me every file this spell owns." +sorcery-go gaze tablet # "Show me exactly how this was configured." +sorcery-go gaze essence # "Which spell + flags produced this blob?" +sorcery-go gaze whereis /usr/bin/wget # "Who owns this file?" +sorcery-go ward status # "Is the Warding active? Any tainted Essences?" ``` ## Vocabulary Cross-Reference diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 12f78e2..4003f2a 100755 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -68,8 +68,8 @@ the appropriate rules. "operator": { "type": "list", "data": [ - { "type": "simple", "operand": "process", "data": "/usr/local/bin/sorcery-go" }, - { "type": "regexp", "operand": "dest_host", "data": "registry\\.local" } + { "type": "simple", "operand": "process", "data": "/usr/local/sbin/sorcery-go" }, + { "type": "regexp", "operand": "dest_host", "data": "registry\\.example\\.local" } ] } } diff --git a/manifests/lxc-sorcery-essence.apparmor b/manifests/lxc-sorcery-essence.apparmor index 450d672..20e26e3 100755 --- a/manifests/lxc-sorcery-essence.apparmor +++ b/manifests/lxc-sorcery-essence.apparmor @@ -1,35 +1,28 @@ -#include - -# AppArmor profile for an LXC container running inside the Coven. -# File: /etc/apparmor.d/lxc/lxc-sorcery-essence +# DEPRECATED — Retained for reference only. # -# This profile is the "Immutable Vault". It allows containers to READ -# from the Tomb (to run their software) but strictly forbids WRITING, -# LINKING, or DELETING. A compromised container cannot poison the -# global Essence store. +# The Sorcery-Go security model uses eBPF LSM programs (Tomb Guard + +# cgroup filters) for in-kernel enforcement. This AppArmor profile is +# superseded by pkg/warding/ebpf/c/tomb_guard.bpf.c and is no longer +# loaded by the Warding. See docs/SECURITY.md for the current model. + +#include profile lxc-sorcery-essence flags=(attach_disconnected, mediate_deleted) { #include # 1. Global Essence Store Access (READ ONLY) - # Prevents any container from modifying the master hashes - /var/lib/sorcery/tomb/ r, - /var/lib/sorcery/tomb/** r, - /var/lib/sorcery/essences/ r, - /var/lib/sorcery/essences/** r, + /var/lib/sorcery-go/tomb/ r, + /var/lib/sorcery-go/tomb/** r, # 2. Deny all write/append/link/rename/delete attempts - deny /var/lib/sorcery/tomb/** wklx, - deny /var/lib/sorcery/essences/** wklx, + deny /var/lib/sorcery-go/tomb/** wklx, # 3. Allow BoltDB read-only access for local manifest verification - /var/lib/sorcery/state/state.db r, + /var/lib/sorcery-go/state/state.db r, # 4. Allow the Sorcery-Go engine binary to execute - /usr/local/bin/sorcery rix, - /usr/local/bin/warding rix, - /usr/local/bin/gaze rix, + /usr/local/sbin/sorcery-go rix, # 5. Allow writes only inside the container's own rootfs /var/lib/lxc/*/rootfs/** rwkl, -} +} \ No newline at end of file diff --git a/manifests/opensnitch-essence-sync.json b/manifests/opensnitch-essence-sync.json index cd9195c..42a209d 100755 --- a/manifests/opensnitch-essence-sync.json +++ b/manifests/opensnitch-essence-sync.json @@ -11,12 +11,12 @@ { "type": "simple", "operand": "process", - "data": "/usr/local/bin/sorcery-go" + "data": "/usr/local/sbin/sorcery-go" }, { "type": "regexp", "operand": "dest_host", - "data": "registry\\.local" + "data": "registry\\.example\\.local" }, { "type": "simple", diff --git a/manifests/sorcery-go.openrc b/manifests/sorcery-go.openrc index 903d231..33a51ac 100755 --- a/manifests/sorcery-go.openrc +++ b/manifests/sorcery-go.openrc @@ -3,21 +3,22 @@ # File: /etc/init.d/sorcery-go # # Install with: -# sudo cp sorcery-go /etc/init.d/ +# sudo cp sorcery-go.openrc /etc/init.d/sorcery-go # sudo chmod +x /etc/init.d/sorcery-go # sudo rc-update add sorcery-go default # sudo rc-service sorcery-go start description="Sorcery-Go Sovereign Coven Engine" -command="/usr/local/bin/sorcery" +command="/usr/local/sbin/sorcery-go" command_args="web --port 8080 --cockpit-integration" command_background="yes" pidfile="/run/${RC_SVCNAME}.pid" -output_log="/var/log/sorcery/sorcery-go.log" -error_log="/var/log/sorcery/sorcery-go.err" +output_log="/var/log/sorcery-go/sorcery-go.log" +error_log="/var/log/sorcery-go/sorcery-go.err" # OpenRC capability management — grant the powers the Warding needs -capabilities="cap_sys_admin,cap_chown,cap_dac_override+ep" +# CAP_BPF is required for loading eBPF Tomb Guard programs +capabilities="cap_sys_admin,cap_chown,cap_dac_override,cap_bpf+ep" depend() { need localmount @@ -25,7 +26,7 @@ depend() { } start_pre() { - # Ensure the BoltDB directory exists before starting - checkpath --directory --owner root:root --mode 0700 /var/lib/sorcery/state - checkpath --directory --owner root:root --mode 0755 /var/log/sorcery -} + # Ensure the state and log directories exist before starting + checkpath --directory --owner root:root --mode 0700 /var/lib/sorcery-go/state + checkpath --directory --owner root:root --mode 0755 /var/log/sorcery-go +} \ No newline at end of file diff --git a/pkg/tomb/reanimate.go b/pkg/tomb/reanimate.go new file mode 100755 index 0000000..e45ae20 --- /dev/null +++ b/pkg/tomb/reanimate.go @@ -0,0 +1,92 @@ +// Reanimation: binding an Essence from the Tomb into a Sanctum. +// +// In the Coven mythos, Reanimation is the act of bringing a sealed binary +// back to life inside a Sanctum. A Sanctum can be an LXC container, Podman +// container, Firecracker microVM, or bare-metal chroot. +// +// Technically this means creating reflinks (preferred — zero-copy on +// btrfs/xfs) or hardlinks from the Tomb's blob store into the Sanctum's +// rootfs. If neither is possible (cross-filesystem), we fall back to a +// plain copy. +package tomb + +import ( + "fmt" + "io" + "os" + "path/filepath" + "syscall" +) + +// Reanimate links every file of `essenceID` into `sanctumRoot` so the +// Sanctum can execute them. The caller MUST have already called +// VerifyRoot() (or VerifyBlobs()) on the essence — Reanimate does not +// re-check integrity for performance reasons. +func (t *Tomb) Reanimate(essenceID, sanctumRoot string) error { + s, err := t.GetSarcophagus(essenceID) + if err != nil { + return err + } + for path, hash := range s.Files { + src := t.blobPath(hash) + dst := filepath.Join(sanctumRoot, path) + if err := os.MkdirAll(filepath.Dir(dst), 0755); err != nil { + return fmt.Errorf("reanimate: mkdir %s: %w", filepath.Dir(dst), err) + } + if err := linkOrCopy(src, dst); err != nil { + return fmt.Errorf("reanimate: link %s: %w", dst, err) + } + } + return nil +} + +// linkOrCopy tries reflink (btrfs/xfs FICLONE), then hardlink (same +// filesystem), then plain copy (cross-filesystem). +func linkOrCopy(src, dst string) error { + // 1. Reflink (zero-copy on btrfs/xfs). + if err := reflinkFile(src, dst); err == nil { + return nil + } + // 2. Hardlink (fast, same-filesystem only). + if err := os.Link(src, dst); err == nil { + return nil + } + // 3. Plain copy. + return copyFile(src, dst) +} + +// reflinkFile wraps the FICLONE ioctl. Returns nil on success. +func reflinkFile(src, dst string) error { + srcF, err := os.Open(src) + if err != nil { + return err + } + defer srcF.Close() + dstF, err := os.Create(dst) + if err != nil { + return err + } + defer dstF.Close() + const FICLONE = 0x40049409 + _, _, errno := syscall.Syscall(syscall.SYS_IOCTL, + dstF.Fd(), uintptr(FICLONE), srcF.Fd()) + if errno != 0 { + return fmt.Errorf("FICLONE: %w", errno) + } + return nil +} + +func copyFile(src, dst string) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.Create(dst) + if err != nil { + return err + } + defer out.Close() + _, err = io.Copy(out, in) + return err +} diff --git a/pkg/tomb/storage.go b/pkg/tomb/storage.go new file mode 100755 index 0000000..be275ed --- /dev/null +++ b/pkg/tomb/storage.go @@ -0,0 +1,302 @@ +// Package tomb is the content-addressable storage (CAS) layer of the Coven. +// +// "In the Coven, we forge the living. In the Tomb, we preserve the eternal." +// +// A Sarcophagus is the on-disk record of one Essence — the binaries, +// libraries and metadata produced by one cast. The Epitaph is the metadata +// sidecar (JSON): name, version, y/n flags, toolchain id, Merkle root. +// Reanimation is the act of binding an Essence into a Sanctum. +// Technically this means creating reflinks (preferred — zero-copy on +// btrfs/xfs) or hardlinks from the Tomb's blob store into the Sanctum's +// rootfs. If neither is possible (cross-filesystem), we fall back to a +// plain copy. +// +// Works identically across all runtimes: LXC, Podman, Firecracker, baremetal. +// +// Files are stored by their SHA-256 hash under +// /var/lib/sorcery-go/tomb/blobs// (sharded by first two hex +// chars) so a single directory never grows past ~65k entries — keeping +// `ls` fast on cold disks. +package tomb + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "log" + "os" + "path/filepath" + "sort" + "strings" +) + +// Sarcophagus is the on-disk record of one Essence variant. +type Sarcophagus struct { + EssenceID string `json:"essence_id"` // Merkle root + SpellName string `json:"spell_name"` + Version string `json:"version"` + VariantHash string `json:"variant_hash"` // sha256(version + flags + arch + toolchain) + Arch string `json:"arch"` + Linkage string `json:"linkage"` // "dynamic" | "static" | "hermetic" + Config map[string]bool `json:"config"` // y/n answers + Files map[string]string `json:"files"` // path -> sha256 + CreatedAt string `json:"created_at"` + Toolchain string `json:"toolchain"` + License string `json:"license"` + SignedBy string `json:"signed_by,omitempty"` // PGP key fingerprint +} + +// Tomb is the storage manager. +type Tomb struct { + Root string // /var/lib/sorcery-go/tomb +} + +// New returns a Tomb rooted at `root`. +func New(root string) *Tomb { + return &Tomb{Root: root} +} + +// IngestBlob copies a file from `srcPath` into the Tomb's blob store, +// returning the SHA-256 hash. Content-addressed storage means duplicate +// blobs are harmlessly re-written with identical content. +func (t *Tomb) IngestBlob(srcPath string) (string, error) { + hash, err := HashFile(srcPath) + if err != nil { + return "", err + } + dst := t.blobPath(hash) + if err := os.MkdirAll(filepath.Dir(dst), 0700); err != nil { + return "", err + } + in, err := os.Open(srcPath) + if err != nil { + return "", err + } + defer in.Close() + // Use O_CREATE|O_TRUNC directly — no TOCTOU race from a prior Stat. + // Content-addressed means re-writing the same content is safe. + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644) + if err != nil { + return "", err + } + defer out.Close() + if _, err := io.Copy(out, in); err != nil { + return "", err + } + return hash, nil +} + +// Store writes a Sarcophagus's epitaph to disk. Callers must have already +// ingested every blob via IngestBlob. +func (t *Tomb) Store(sarc *Sarcophagus) error { + if err := os.MkdirAll(filepath.Join(t.Root, "epitaphs"), 0700); err != nil { + return err + } + // Recompute the Merkle root from the file map — this is the canonical + // EssenceID. If the caller pre-set EssenceID we verify it matches. + computed := ComputeMerkleRoot(sarc.Files) + if sarc.EssenceID == "" { + sarc.EssenceID = computed + } else if sarc.EssenceID != computed { + return fmt.Errorf("tomb: essence_id mismatch (stored=%s, computed=%s) — refusing to write taint", + sarc.EssenceID, computed) + } + path := filepath.Join(t.Root, "epitaphs", sarc.EssenceID+".json") + data, err := json.MarshalIndent(sarc, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, data, 0600) +} + +// GetSarcophagus loads the epitaph for an Essence ID. +func (t *Tomb) GetSarcophagus(essenceID string) (*Sarcophagus, error) { + path := filepath.Join(t.Root, "epitaphs", essenceID+".json") + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("tomb: sarcophagus %s not found: %w", essenceID, err) + } + var s Sarcophagus + if err := json.Unmarshal(data, &s); err != nil { + return nil, err + } + return &s, nil +} + +// VerifyRoot recomputes the Merkle root of a Sarcophagus from its file +// hashes and compares it against the stored EssenceID. Any mismatch +// means "Taint" — the Warding must reject Reanimation. +// +// Note: this only verifies the metadata. To verify the blobs themselves +// are not bit-rotted, use VerifyBlobs which re-hashes every blob on disk. +func (t *Tomb) VerifyRoot(essenceID string) error { + s, err := t.GetSarcophagus(essenceID) + if err != nil { + return err + } + computed := ComputeMerkleRoot(s.Files) + if computed != essenceID { + return fmt.Errorf("tomb: merkle mismatch for %s (expected %s, recomputed %s)", + s.SpellName, essenceID, computed) + } + return nil +} + +// VerifyBlobs re-hashes every blob referenced by the Sarcophagus and +// confirms the bytes on disk still match. This is the slow bit-rot check +// run by `sorcery tomb verify --all`. +func (t *Tomb) VerifyBlobs(essenceID string) error { + s, err := t.GetSarcophagus(essenceID) + if err != nil { + return err + } + for path, expectedHash := range s.Files { + blobPath := t.blobPath(expectedHash) + actualHash, err := HashFile(blobPath) + if err != nil { + return fmt.Errorf("tomb: blob %s missing for %s: %w", expectedHash, path, err) + } + if actualHash != expectedHash { + return fmt.Errorf("tomb: blob %s bit-rotted (path %s)", expectedHash, path) + } + } + return nil +} + +// List returns every Essence currently resting in the Tomb. +func (t *Tomb) List() ([]*Sarcophagus, error) { + dir := filepath.Join(t.Root, "epitaphs") + var out []*Sarcophagus + err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { + if err != nil || info.IsDir() { + return nil + } + if !strings.HasSuffix(path, ".json") { + return nil + } + data, e := os.ReadFile(path) + if e != nil { + log.Printf("tomb: skipping unreadable file %s: %v", path, e) + return nil + } + var s Sarcophagus + if e := json.Unmarshal(data, &s); e == nil { + out = append(out, &s) + } + return nil + }) + return out, err +} + +// FindByVariant returns the EssenceID of the Sarcophagus whose VariantHash +// matches, or "" when not found. This is the cache-hit check in the Cast +// pipeline. +func (t *Tomb) FindByVariant(variantHash string) (string, error) { + all, err := t.List() + if err != nil { + return "", err + } + for _, s := range all { + if s.VariantHash == variantHash { + return s.EssenceID, nil + } + } + return "", nil +} + +// FindBySpell returns the most recently created Sarcophagus for a spell +// (any variant). Used by `cauldron` to pick the latest essence for an ISO. +func (t *Tomb) FindBySpell(spell, arch string) (*Sarcophagus, error) { + all, err := t.List() + if err != nil { + return nil, err + } + var best *Sarcophagus + for _, s := range all { + if s.SpellName != spell { + continue + } + if arch != "" && s.Arch != arch { + continue + } + if best == nil || s.CreatedAt > best.CreatedAt { + best = s + } + } + if best == nil { + return nil, fmt.Errorf("tomb: no essence for %s on %s", spell, arch) + } + return best, nil +} + +// Prune (Garbage Collection) walks the blob store and removes any hash +// that is not referenced by an active epitaph. Returns the bytes reclaimed. +func (t *Tomb) Prune() (int64, error) { + active := make(map[string]bool) + epitaphs, err := t.List() + if err != nil { + return 0, fmt.Errorf("tomb: prune: list epitaphs: %w", err) + } + for _, s := range epitaphs { + for _, h := range s.Files { + active[h] = true + } + } + var reclaimed int64 + blobDir := filepath.Join(t.Root, "blobs") + err := filepath.Walk(blobDir, func(path string, info os.FileInfo, err error) error { + if err != nil || info.IsDir() { + return nil + } + name := info.Name() + if !active[name] { + reclaimed += info.Size() + _ = os.Remove(path) + } + return nil + }) + return reclaimed, err +} + +// blobPath returns /var/lib/sorcery-go/tomb/blobs//. +func (t *Tomb) blobPath(hash string) string { + if len(hash) < 2 { + return filepath.Join(t.Root, "blobs", hash) + } + return filepath.Join(t.Root, "blobs", hash[:2], hash) +} + +// HashFile computes the sha256 of a file on disk — used by the Committer +// and the Cast pipeline before storing a Sarcophagus. +func HashFile(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + +// ComputeMerkleRoot concatenates every file path + hash (sorted by path) +// and hashes the result. This catches a single-bit flip anywhere in the +// file set. A real binary Merkle tree would let us localise the failure; +// for now we trade precision for simplicity and speed. +func ComputeMerkleRoot(files map[string]string) string { + keys := make([]string, 0, len(files)) + for k := range files { + keys = append(keys, k) + } + sort.Strings(keys) + h := sha256.New() + for _, k := range keys { + h.Write([]byte(k)) + h.Write([]byte(files[k])) + } + return hex.EncodeToString(h.Sum(nil)) +} diff --git a/pkg/tomb/storage_test.go b/pkg/tomb/storage_test.go new file mode 100755 index 0000000..624950c --- /dev/null +++ b/pkg/tomb/storage_test.go @@ -0,0 +1,102 @@ +package tomb + +import ( + "os" + "path/filepath" + "testing" +) + +func TestIngestStoreVerify(t *testing.T) { + tmp := t.TempDir() + tomb := New(tmp) + + // Create a fake source file to ingest. + srcBlob := filepath.Join(tmp, "src.txt") + if err := os.WriteFile(srcBlob, []byte("hello world"), 0644); err != nil { + t.Fatal(err) + } + + hash, err := tomb.IngestBlob(srcBlob) + if err != nil { + t.Fatalf("IngestBlob: %v", err) + } + if hash == "" { + t.Fatal("expected non-empty hash") + } + + sarc := &Sarcophagus{ + SpellName: "test", + Version: "1.0", + Arch: "x86_64", + Files: map[string]string{"/usr/bin/test": hash}, + } + if err := tomb.Store(sarc); err != nil { + t.Fatalf("Store: %v", err) + } + if sarc.EssenceID == "" { + t.Fatal("Store should set EssenceID") + } + + // VerifyRoot must pass. + if err := tomb.VerifyRoot(sarc.EssenceID); err != nil { + t.Fatalf("VerifyRoot: %v", err) + } + + // VerifyBlobs must pass. + if err := tomb.VerifyBlobs(sarc.EssenceID); err != nil { + t.Fatalf("VerifyBlobs: %v", err) + } + + // Tamper with the blob — VerifyBlobs must fail. + blobPath := tomb.blobPath(hash) + if err := os.WriteFile(blobPath, []byte("tampered"), 0644); err != nil { + t.Fatal(err) + } + if err := tomb.VerifyBlobs(sarc.EssenceID); err == nil { + t.Fatal("VerifyBlobs must detect tampering") + } else { + t.Logf("✓ correctly detected bit-rot: %v", err) + } +} + +func TestDedup(t *testing.T) { + tmp := t.TempDir() + tomb := New(tmp) + + // Two identical source files. + src1 := filepath.Join(tmp, "a.txt") + src2 := filepath.Join(tmp, "b.txt") + _ = os.WriteFile(src1, []byte("same content"), 0644) + _ = os.WriteFile(src2, []byte("same content"), 0644) + + h1, _ := tomb.IngestBlob(src1) + h2, _ := tomb.IngestBlob(src2) + if h1 != h2 { + t.Fatalf("dedup failed: %s != %s", h1, h2) + } + t.Logf("✓ dedup hit: %s", h1) +} + +func TestFindByVariant(t *testing.T) { + tmp := t.TempDir() + tomb := New(tmp) + sarc := &Sarcophagus{ + VariantHash: "v-1", + SpellName: "demo", + Version: "1.0", + Files: map[string]string{"/usr/bin/demo": "h1"}, + } + if err := tomb.Store(sarc); err != nil { + t.Fatal(err) + } + if id, err := tomb.FindByVariant("v-1"); err != nil { + t.Fatalf("FindByVariant: %v", err) + } else if id != sarc.EssenceID { + t.Fatalf("FindByVariant returned %s, want %s", id, sarc.EssenceID) + } + if id, err := tomb.FindByVariant("missing"); err != nil { + t.Fatalf("FindByVariant: %v", err) + } else if id != "" { + t.Fatalf("FindByVariant should return empty for unknown variant, got %s", id) + } +} diff --git a/scripts/deploy_grid.sh b/scripts/deploy_grid.sh index 537e302..936e53c 100755 --- a/scripts/deploy_grid.sh +++ b/scripts/deploy_grid.sh @@ -13,6 +13,7 @@ echo " Runtime: $RUNTIME" # 1. Setup Directory Structure mkdir -p bin pkg grimoire tablet tomb docs manifests mkdir -p /var/lib/sorcery-go/ebpf/maps +mkdir -p /var/log/sorcery-go /run/sorcery-go # 2. Initialize the State Database "$SORCERY" init --db ./tablet/state.db @@ -25,18 +26,18 @@ echo "→ Loading eBPF Tomb Guard..." # 4. Start the Warding Monitor (background) if [ -x ./build/warding ]; then - nohup ./build/warding watch > /var/log/sorcery/warding.log 2>&1 & - echo $! > /run/warding.pid - echo "✓ Warding monitor started (PID $(cat /run/warding.pid))" + nohup ./build/warding watch > /var/log/sorcery-go/warding.log 2>&1 & + echo $! > /run/sorcery-go/warding.pid + echo "✓ Warding monitor started (PID $(cat /run/sorcery-go/warding.pid))" fi # 5. Launch the Coven Mirror WebUI # (foreground by default; use --background for systemd/OpenRC) if [ "$1" = "--background" ]; then nohup "$SORCERY" web --port 8080 --cockpit-integration \ - > /var/log/sorcery/web.log 2>&1 & - echo $! > /run/sorcery-web.pid - echo "✓ Coven Mirror running in background (PID $(cat /run/sorcery-web.pid))" + > /var/log/sorcery-go/web.log 2>&1 & + echo $! > /run/sorcery-go/web.pid + echo "✓ Coven Mirror running in background (PID $(cat /run/sorcery-go/web.pid))" echo " Open: http://localhost:8080" else echo "✓ Launching Coven Mirror in foreground..." diff --git a/scripts/forge_first_sanctum.sh b/scripts/forge_first_sanctum.sh index edb14da..5a37a77 100755 --- a/scripts/forge_first_sanctum.sh +++ b/scripts/forge_first_sanctum.sh @@ -65,7 +65,7 @@ case "$RUNTIME" in mkdir -p "$SANCTUM_PATH" ;; baremetal|*) - SANCTUM_PATH="/opt/sorcery/sanctums/sanctum-alpha" + SANCTUM_PATH="/var/lib/sorcery-go/sanctums/sanctum-alpha" sudo mkdir -p "$SANCTUM_PATH" ;; esac diff --git a/tablet/.gitkeep b/tablet/.gitkeep new file mode 100755 index 0000000..47f91ce --- /dev/null +++ b/tablet/.gitkeep @@ -0,0 +1,3 @@ +# BoltDB state files (state.db, journal, manifests, tablet) are created +# here at runtime by `sorcery init`. This file is just a placeholder so the +# directory is preserved in version control. diff --git a/tomb/.gitkeep b/tomb/.gitkeep new file mode 100755 index 0000000..01d9319 --- /dev/null +++ b/tomb/.gitkeep @@ -0,0 +1,4 @@ +# Content-addressable blob store. The Cauldron writes Essence blobs here +# under blobs//; epitaphs (JSON metadata sidecars) live under +# epitaphs/. Created at runtime by `sorcery init`. This file is just a +# placeholder so the directory is preserved in version control.