sorcery-go/docs/DISASTER_RECOVERY.md

Disaster Recovery Tome

If your Master Sanctum is physically destroyed, or the primary disk suffers a critical failure, this is how you rebuild the entire Coven using only your encrypted backups.

Sacred Artifacts

To perform this reconstruction you need two things:

1. The Tablet Backup — tablet.db.gpg — your configuration, y/n choices, and Merkle roots.
2. The Tomb Backup — tomb.tar.gz or a remote mirror — the actual forged binaries.

I. Reconstructing the Master Node

If the Master is gone, designate a new host.

# 1. Deploy the engine to the new host
make build && sudo make drop-in

# 2. Restore the Tablet (the "memory" of every spell ever cast)
gpg --decrypt tablet.db.gpg > /var/lib/sorcery-go/state/state.db

# 3. Re-seed the Tomb
tar xzf tomb.tar.gz -C /var/lib/sorcery-go/

# 4. Re-initialise the Cauldron — the engine scans the restored Tablet
#    and Tomb to rebuild the internal search indexes
sorcery-go init --force

II. Re-Establishing the Coven (Worker Nodes)

Workers are connected to the Master via plain HTTP behind the network firewall. No certificates or key exchange is needed — just ensure the firewall rules allow traffic on the sorcery-go (8080) and Fester (8181) ports.

# 1. Verify network connectivity to the new Master
sorcery coven pulse

# 2. If workers were using a Fester controller, update its URL
# (set in /etc/sorcery-go/config.yaml or SORCERY_GO_FESTER_URL env var)

If the workers respond with a green pulse, the Coven is restored.

III. The Shadow-Forge Validation

Once the Coven is back online, verify that the restored Essences were not corrupted during the failure:

# Deep Gaze — recompute the Merkle root of every file in the Tomb and
# compare against the restored Tablet entries
sorcery tomb verify --all

# Ghost Build — pick a core tool (like busybox) and re-forge it in a
# temporary sandbox. The new binary must match the restored Essence.
sorcery cast busybox --static --default

IV. Restoring the Sanctums (LXC Containers)

Because hydration uses reflinks or hardlinks, a simple file restore will not work for containers — the links are broken. Run the Re-Hydration Ritual:

# Reads the Tablet to see which Essences belong in which containers,
# then re-links them from the Tomb
sorcery reanimate --all-containers

V. Backup Strategy

Rule	Action	Frequency
Rule of Three	One local, one off-site, one cloud backup	Daily
Immutable Seal	Sign every tablet.db backup with a hardware key	Per change
Mirror Ritual	Use Fester CAS replication to keep warm standby	Real-time

VI. Verification Checklist

After reconstruction, verify each layer:

• sorcery coven pulse — every node green
• sorcery tomb verify --all — zero Merkle mismatches
• sorcery ward status — eBPF Tomb Guard active
• sorcery legal audit — zero license violations
• gaze whereis /usr/bin/bash — owned by coreutils
• Cast a smoke-test spell: sorcery cast busybox --static --default