94 lines
3.2 KiB
Markdown
Executable File
94 lines
3.2 KiB
Markdown
Executable File
# Disaster Recovery Tome
|
|
|
|
If your Master Sanctum is physically destroyed, or the primary disk suffers a
|
|
critical failure, this is how you rebuild the entire Coven using only your
|
|
encrypted backups.
|
|
|
|
## Sacred Artifacts
|
|
|
|
To perform this reconstruction you need two things:
|
|
|
|
1. **The Tablet Backup** — `tablet.db.gpg` — your configuration, y/n choices,
|
|
and Merkle roots.
|
|
2. **The Tomb Backup** — `tomb.tar.gz` or a remote mirror — the actual
|
|
forged binaries.
|
|
|
|
## I. Reconstructing the Master Node
|
|
|
|
If the Master is gone, designate a new host.
|
|
|
|
```bash
|
|
# 1. Deploy the engine to the new host
|
|
make build && sudo make drop-in
|
|
|
|
# 2. Restore the Tablet (the "memory" of every spell ever cast)
|
|
gpg --decrypt tablet.db.gpg > /var/lib/sorcery-go/state/state.db
|
|
|
|
# 3. Re-seed the Tomb
|
|
tar xzf tomb.tar.gz -C /var/lib/sorcery-go/
|
|
|
|
# 4. Re-initialise the Cauldron — the engine scans the restored Tablet
|
|
# and Tomb to rebuild the internal search indexes
|
|
sorcery-go init --force
|
|
```
|
|
|
|
## II. Re-Establishing the Coven (Worker Nodes)
|
|
|
|
Workers are connected to the Master via plain HTTP behind the network firewall.
|
|
No certificates or key exchange is needed — just ensure the firewall rules
|
|
allow traffic on the sorcery-go (8080) and Fester (8181) ports.
|
|
|
|
```bash
|
|
# 1. Verify network connectivity to the new Master
|
|
sorcery-go coven pulse
|
|
|
|
# 2. If workers were using a Fester controller, update its URL
|
|
# (set in /etc/sorcery-go/config.yaml or SORCERY_GO_FESTER_URL env var)
|
|
```
|
|
|
|
If the workers respond with a green pulse, the Coven is restored.
|
|
|
|
## III. The Shadow-Forge Validation
|
|
|
|
Once the Coven is back online, verify that the restored Essences were not
|
|
corrupted during the failure:
|
|
|
|
```bash
|
|
# Deep Gaze — recompute the Merkle root of every file in the Tomb and
|
|
# compare against the restored Tablet entries
|
|
sorcery-go tomb verify --all
|
|
|
|
# Ghost Build — pick a core tool (like busybox) and re-forge it in a
|
|
# temporary sandbox. The new binary must match the restored Essence.
|
|
sorcery-go cast busybox --static --default
|
|
```
|
|
|
|
## IV. Restoring the Sanctums (LXC Containers)
|
|
|
|
Because hydration uses reflinks or hardlinks, a simple file restore will not
|
|
work for containers — the links are broken. Run the Re-Hydration Ritual:
|
|
|
|
```bash
|
|
# Reads the Tablet to see which Essences belong in which containers,
|
|
# then re-links them from the Tomb
|
|
sorcery-go reanimate --all-containers
|
|
```
|
|
|
|
## V. Backup Strategy
|
|
|
|
| Rule | Action | Frequency |
|
|
|-------------------|-----------------------------------------------------|-----------|
|
|
| Rule of Three | One local, one off-site, one cloud backup | Daily |
|
|
| Immutable Seal | Sign every `tablet.db` backup with a hardware key | Per change|
|
|
| Mirror Ritual | Use Fester CAS replication to keep warm standby | Real-time |
|
|
|
|
## VI. Verification Checklist
|
|
|
|
After reconstruction, verify each layer:
|
|
|
|
- [ ] `sorcery-go coven pulse` — every node green
|
|
- [ ] `sorcery-go tomb verify --all` — zero Merkle mismatches
|
|
- [ ] `sorcery-go ward status` — eBPF Tomb Guard active
|
|
- [ ] `sorcery-go legal audit` — zero license violations
|
|
- [ ] `sorcery-go gaze whereis /usr/bin/bash` — owned by `coreutils`
|
|
- [ ] Cast a smoke-test spell: `sorcery-go cast busybox --static --default` |