# Disaster Recovery Tome

If your Master Sanctum is physically destroyed, or the primary disk suffers a
critical failure, this is how you rebuild the entire Coven using only your
encrypted backups.

## Sacred Artifacts

To perform this reconstruction you need two things:

1. **The Tablet Backup** — `tablet.db.gpg` — your configuration, y/n choices,
   and Merkle roots.
2. **The Tomb Backup** — `tomb.tar.gz` or a remote mirror — the actual
   forged binaries.

## I. Reconstructing the Master Node

If the Master is gone, designate a new host.

```bash
# 1. Deploy the engine to the new host
make build && sudo make drop-in

# 2. Restore the Tablet (the "memory" of every spell ever cast)
gpg --decrypt tablet.db.gpg > /var/lib/sorcery-go/state/state.db

# 3. Re-seed the Tomb
tar xzf tomb.tar.gz -C /var/lib/sorcery-go/

# 4. Re-initialise the Cauldron — the engine scans the restored Tablet
#    and Tomb to rebuild the internal search indexes
sorcery-go init --force
```

## II. Re-Establishing the Coven (Worker Nodes)

Workers are connected to the Master via plain HTTP behind the network firewall.
No certificates or key exchange is needed — just ensure the firewall rules
allow traffic on the sorcery-go (8080) and Fester (8181) ports.

```bash
# 1. Verify network connectivity to the new Master
sorcery-go coven pulse

# 2. If workers were using a Fester controller, update its URL
# (set in /etc/sorcery-go/config.yaml or SORCERY_GO_FESTER_URL env var)
```

If the workers respond with a green pulse, the Coven is restored.

## III. The Shadow-Forge Validation

Once the Coven is back online, verify that the restored Essences were not
corrupted during the failure:

```bash
# Deep Gaze — recompute the Merkle root of every file in the Tomb and
# compare against the restored Tablet entries
sorcery-go tomb verify --all

# Ghost Build — pick a core tool (like busybox) and re-forge it in a
# temporary sandbox. The new binary must match the restored Essence.
sorcery-go cast busybox --static --default
```

## IV. Restoring the Sanctums (LXC Containers)

Because hydration uses reflinks or hardlinks, a simple file restore will not
work for containers — the links are broken. Run the Re-Hydration Ritual:

```bash
# Reads the Tablet to see which Essences belong in which containers,
# then re-links them from the Tomb
sorcery-go reanimate --all-containers
```

## V. Backup Strategy

| Rule              | Action                                              | Frequency |
|-------------------|-----------------------------------------------------|-----------|
| Rule of Three     | One local, one off-site, one cloud backup           | Daily     |
| Immutable Seal    | Sign every `tablet.db` backup with a hardware key   | Per change|
| Mirror Ritual     | Use Fester CAS replication to keep warm standby     | Real-time |

## VI. Verification Checklist

After reconstruction, verify each layer:

- [ ] `sorcery-go coven pulse` — every node green
- [ ] `sorcery-go tomb verify --all` — zero Merkle mismatches
- [ ] `sorcery-go ward status` — eBPF Tomb Guard active
- [ ] `sorcery-go legal audit` — zero license violations
- [ ] `sorcery-go gaze whereis /usr/bin/bash` — owned by `coreutils`
- [ ] Cast a smoke-test spell: `sorcery-go cast busybox --static --default`