commit 0a94c580876bd1e384b69bb3dfd3e0d954eadcd5 Author: Jeremy Anderson Date: Wed Jul 15 00:08:50 2026 -0400 Fester: a distributed build orchestrator with thermal-aware scheduling, live DAG, event replay, and failure analysis diff --git a/.github/funding.yml b/.github/funding.yml new file mode 100755 index 0000000..e09faad --- /dev/null +++ b/.github/funding.yml @@ -0,0 +1,64 @@ +# .github/funding.yml +# ============================================================ +# GitHub Sponsors "Sponsor this project" button configuration. +# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository +# +# Each key below corresponds to a sponsored-developer account or platform +# that will appear as a "Sponsor" button on the repo's main page. +# +# Replace the placeholder values with your actual usernames/addresses. +# Comment out or delete any platforms you don't want to use. +# ============================================================ + +# --- GitHub Sponsors --- +# Your GitHub Sponsors username (the part after github.com/sponsors/) +github: REPLACE_WITH_YOUR_GITHUB_USERNAME + +# --- Patreon --- +# Your Patreon username (the part after patreon.com/) +patreon: REPLACE_WITH_YOUR_PATREON_USERNAME + +# --- Open Collective --- +# Your Open Collective slug (the part after opencollective.com/) +open_collective: REPLACE_WITH_YOUR_OPENCOLLECTIVE_SLUG + +# --- Ko-fi --- +# Your Ko-fi username (the part after ko-fi.com/) +ko_fi: REPLACE_WITH_YOUR_KOFI_USERNAME + +# --- Tidelift --- +# Your Tidelift package name (e.g. "npm/foo" or "pypi/bar") +tidelift: REPLACE_WITH_YOUR_TIDELIFT_PACKAGE + +# --- CommunityBridge --- +# Your CommunityBridge project name +community_bridge: REPLACE_WITH_YOUR_COMMUNITYBRIDGE_PROJECT + +# --- Liberapay --- +# Your Liberapay username (the part after liberapay.com/) +liberapay: REPLACE_WITH_YOUR_LIBERAPAY_USERNAME + +# --- IssueHunt --- +# Your IssueHunt username +issuehunt: REPLACE_WITH_YOUR_ISSUEHUNT_USERNAME + +# --- LFX Crowdfunding --- +# Your LFX project name +lfx_crowdfunding: REPLACE_WITH_YOUR_LFX_PROJECT + +# --- Polar --- +# Your Polar repository (e.g. "owner/repo") +polar: REPLACE_WITH_YOUR_GITHUB_ORG/fester + +# --- Custom platforms / direct links --- +# A list of up to 4 custom {label, url} entries that appear as buttons. +# Use these for crypto donation links, your own donate page, etc. +custom: + - label: "Donate" + url: "https://git.dcos.net/dcosnet/fester/src/branch/main/donate.md" + # - label: "Bitcoin" + # url: "https://your-blockchain-explorer/address/YOUR_BTC_ADDRESS" + # - label: "Monero" + # url: "https://your-xmr-explorer/address/YOUR_XMR_ADDRESS" + # - label: "PayPal" + # url: "https://www.paypal.com/paypalme/YOUR_PAYPAL_ME" diff --git a/.gitignore b/.gitignore new file mode 100755 index 0000000..77aa5d6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,139 @@ +# ============================================================ +# Fester .gitignore +# ============================================================ + +# ---- Python ---- +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# ---- Virtual environments ---- +.venv/ +venv/ +env/ +ENV/ +env.bak/ +venv.bak/ +.python-version + +# ---- Testing / coverage ---- +htmlcov/ +.tox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +.hypothesis/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ + +# ---- Fester runtime data ---- +*.db +*.db-journal +*.db-wal +*.db-shm +*.sqlite +*.sqlite-journal +*.sqlite-wal +*.sqlite-shm + +# Fester storage dirs (local dev) +.fester/ +fester-data/ +/var/lib/fester/ +snapshots/ +cas/ +cache/ +*.qcow2 +*.tar.gz + +# Config with secrets (keep config.yaml.example in git, ignore actual config) +config.local.yaml +config.runtime.yaml +storage.json +node_roles.json +.env +.env.local +.env.*.local + +# ---- Logs ---- +*.log +logs/ +fester.log +fester_backend.log +mock_agent.log + +# ---- IDE / Editor ---- +.idea/ +.vscode/ +*.swp +*.swo +*~ +.project +.classpath +.settings/ +*.sublime-project +*.sublime-workspace +.atom/ +.brackets.json + +# ---- OS ---- +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +ehthumbs.db +Thumbs.db +desktop.ini + +# ---- Docker ---- +docker-compose.override.yml + +# ---- Node (if you ever add a JS build step) ---- +node_modules/ +npm-debug.log* +yarn-debug.log* +yarn-error.log* +pnpm-debug.log* + +# ---- Build artifacts ---- +*.o +*.a +*.so +*.dylib +build.tar.gz +build/ + +# ---- Screenshots (keep in git, but ignore ad-hoc captures) ---- +# docs/screenshots/ is tracked — these are ad-hoc: +/tmp/ +*.tmp + +# ---- Misc ---- +*.bak +*.orig +*.rej +*.diff +*.patch diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100755 index 0000000..611492b --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,171 @@ +# Changelog + +All notable changes to Fester are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- **BTC 0.4.0 multi-arch cross-compilation support** — `backend/toolchain/btc.py` + - `probe_btc()` now loads BTC manifest JSON sidecars for structured metadata + (target_id, arch, clib, triple, cross_mode, target_march) + - `btc_build_env()` supports `target` parameter to select specific cross-compiler + golden images when multiple are installed on a node + - `list_btc_targets()` enumerates all available BTC golden images with + parsed SYS_LABEL and manifest data — usable by the Fester API + - ISA flag table (`_ISA_FLAGS`) for AVX512, AVX2, SSE4_2, NEON, MIPS32, TILE + - SSE4_2 ISA tier added for Intel Atom and AMD APU targets that lack AVX support + - 19 cross-compilation targets: Intel HEDT/Server (5), AMD Ryzen/EPYC (4), + AMD APU (4: apu-zn1 through apu-zn4), Intel Atom (4: silvermont, goldmont, + tremont, sierraforest), embedded (mipselr2, armv7, tilegx) + - SYS_LABEL parser updated for BTC 0.4.0 cross format: + `DCOSNET-{FAMILY}-{TARGET_ID}-{ISA}-CROSS` + - Per-node BTC target selection via `node_config.btc.target` + - New env vars: `BTC_TARGET_ID`, `BTC_CROSS`, `BTC_CLIB`, `BTC_TARGET_TRIPLE` +- **Shared Content-Addressable Store (CAS) API** — `backend/storage/cas_api.py` + - `PUT /api/cas/{sha256}` — store an artifact (SHA-256 verified server-side) + - `GET /api/cas/{sha256}` — retrieve an artifact (streamed file download) + - `HEAD /api/cas/{sha256}` — check existence (returns metadata JSON) + - `DELETE /api/cas/{sha256}` — remove an artifact + - `GET /api/cas/` — list all artifacts (paginated, filterable by target) + - `GET /api/cas/stats` — cache statistics (hits, misses, utilization %) + - SQLite-backed index for fast listing, LRU eviction, and cross-query filtering + - btrfs reflink support for zero-copy artifact storage + - DAG-aware cache integration in PipelineEngine — checks CAS before each action +- **BTC.sh toolchain integration** — `backend/toolchain/btc.py` + - `probe_btc()` — detect BTC golden images, parse SYS_LABEL + - `btc_build_env()` — return BTC-aware CC/CXX/CFLAGS/LDFLAGS + - `verify_btc_stamp()` — read .note.BTC ELF note + xattr stamps + - `apply_btc_stamp()` — inject forensic stamps via assembly + objcopy + xattr + - `merge_compiler_env()` — overlay BTC on ccache/distcc env + - Cross-project compatible: pipe-delimited .note.BTC format, type=1 (NT_VERSION), bare hex hash +- **Podman integration** — `backend/integrations/podman.py` (full PodmanManager) +- **Firecracker integration** — `backend/integrations/firecracker.py` (full FirecrackerManager) +- **Podman/Firecracker executor adapters** — `backend/executor/adapters.py` (run_podman, run_firecracker) +- **Runtime router** updated — `backend/executor/runtime_router.py` routes podman and firecracker runtimes +- Documentation: README.md, quickstart.md, CONTRIBUTING.md, CHANGELOG.md +- Tooling: bootstrap.sh, run.sh, pyproject.toml, .gitignore +- Docker support (Dockerfile) +- Screenshots for all 8 UI pages + +## [0.3.0] — 2026-06-28 + +### Added — Storage layer +- SQLite-backed persistence for builds, sessions, events, and node states +- Btrfs CoW reflink snapshots (`backend/storage/btrfs_cas.py`) +- QCOW2 workspace freezing with real mount + rsync (`backend/storage/qcow2_freeze.py`) +- tmpfs workspace acceleration +- Storage status + config + freeze + reflink + snapshots endpoints + +### Added — tmux integration (E9) +- TmuxManager with session create/list/inspect/capture-output/kill +- `runtime: "tmux"` action runtime — runs actions in detached tmux sessions +- Live output viewer in Sessions page UI +- `/api/actions/active`, `/api/actions/{name}/tmux/output`, `/tmux/kill` endpoints + +### Added — mosh/ssh shell attach (E10) +- MoshManager with command builder (mosh-via-SSH-bootstrap) +- `/api/nodes/{name}/shell` GET + POST endpoints +- "⌘ Shell" button on Dashboard node rows (copies command to clipboard) + +### Added — Real engine wiring +- BuildRunner wraps PipelineEngine with build lifecycle + persistence +- Real DAG execution with proper cwd + env passing +- Failure propagation: skipped actions on failed deps +- Critical path computation + tagging in events +- Debugger pause/resume/step actually controls running engines + +### Added — Cause graph + blast radius +- `/api/propagation/{action}` — forward blast radius computation +- `/api/cause/explain/{node}` — causal chain for a node +- Blast Radius panel in Cause Graph UI page +- Failure autopsy with proper session ID correlation + +### Added — Storage + target catalog +- `/api/targets` reads from `backend/targets/catalog.py` (6 systems: Gentoo, Buildroot, OpenWrt, ALFS, SourceMage, Lunar) +- `/api/targets/arches` + `/api/targets/runtimes` endpoints + +### Added — Health + metrics +- `/api/health` endpoint (version, bus_subscribers, ws_clients, builds_known, timeline_events) +- Live health indicator in topbar (all pages) +- Prometheus metrics aligned with grafana config names (`fester_node_cpu`, `fester_pipeline_actions_total`, etc.) +- Pipeline action counters + cache hit counters wired via bus subscriber + +### Added — UI enhancements +- New Timeline page (per-node event drill-down) +- Node policy dropdowns (preferred/avoid/neutral) on Dashboard +- Per-node "Probe" buttons + "Probe All" button +- Build cancel + replay buttons on Sessions page +- Event filter chips on Dashboard (toggle by event type) + +### Added — CLI overhaul +- 35+ subcommands (was 11) +- Nested subcommands: `node {list,set-policy,probe,probe-all}`, `cause {explain,trace,events}`, `replay {start,step,reset}`, `debugger {state,pause,resume,step}` +- New: `build`, `builds`, `build-info`, `cancel`, `release`, `targets`, `toggle-target`, `blast`, `timeline`, `rewind`, `autopsy`, `policy-list`, `health` +- `--watch` flag on `build` command (streams events via WebSocket) + +### Fixed — Critical backend bugs +- `api/api.py` order bug (`bus.subscribe()` before `bus = EventBus()`) +- 7 broken `_endpoint` symbol imports +- `cause_graph.attach_bus` + `emit_debug` defined at module scope (zero indentation) +- Three missing files: `timeline_store.py`, `failure_autopsy.py`, `failure_propagation.py` +- Two Python module/package name collisions (`scheduler.py` vs `scheduler/`, `nodes.py` vs `nodes/`, `cache.py` vs `cache/`) +- `execute_action` passing empty `{}` as cwd → `TypeError` +- Build env never reaching subprocess +- Debugger/pipeline-control `_ENGINE` always `None` +- `failure_propagation` looking for `event.data.deps` (legacy shape) +- Autopsy session ID mismatch +- `/api/release` was a pure stub +- `/api/policy/set` didn't persist +- `/ws-debugger` returned canned data, ignored commands +- `forgejo.on_push_event` wrong PipelineEngine arity +- `pipeline/feedback.py` PolicyEngine() with no args + +### Fixed — Standardization +- ~20 bare imports converted to `backend.*` prefix +- Two parallel event taxonomies unified (`types.py` re-exports from `schema.py`) +- `MinioCache.__init__` made lazy (was crashing on import if MinIO unreachable) +- `cache.py` graceful fallback for non-writable paths +- `roles_store.py` graceful fallback for non-writable `/etc/fester/` +- `FesterEvent` made `node`/`action`/`state` truly optional (had `Optional[str]` type but no default) + +## [0.2.0] — 2026-06-27 + +### Added — Real FastAPI backend +- `backend/main.py` — FastAPI app with 50+ routes +- All routers wired: replay, autopsy, timeline, debugger, pipeline_control, nodes, metrics, cause +- WebSocket hub with `/ws`, `/ws-targets`, `/ws-debugger` +- Ambient loop for node probing + drift +- SQLite storage layer (`backend/storage/sqlite_db.py`) +- Node probe module (`backend/nodes/probe.py`) +- BuildRunner (`backend/pipeline/runner.py`) + +### Added — UI overhaul (7 pages) +- Dashboard (`index.html`) — 3-column layout with cluster stats, live stream, quick actions +- Live DAG (`ui/live_dag.html`) — layered Sugiyama-lite layout, SVG edges, click-to-inspect +- Replay (`ui/replay.html`) — timeline scrubber, play/pause/step, structured inspector +- Sessions (`ui/sessions.html`) — build history + replay links +- Metrics (`ui/metrics.html`) — live charts + per-node cards +- Cause Graph (`ui/cause.html`) — radial graph visualization +- Debugger (`ui/debugger.html`) — step-through execution control + +### Added — Shared infrastructure +- `style.css` — full design system (dark theme tokens, panels, badges, buttons, scrollbars) +- `ui/app.js` — shared shell (WS auto-reconnect, FesterTargets global, toasts, topbar nav) +- `cockpit/fester-module/targets.js` — fixed to expose `FesterTargets` global + +### Fixed +- `live_dag.html` was broken (referenced undefined `FesterTargets`) +- `style.css` was 0 bytes +- `index.html` was a 30-line throwaway + +## [0.1.0] — Initial commit + +### Added +- Basic project structure +- Backend modules (many stubbed) +- Cockpit module +- Install scripts +- CHEATSHEET.md diff --git a/CHEATSHEET.md b/CHEATSHEET.md new file mode 100755 index 0000000..238c7f1 --- /dev/null +++ b/CHEATSHEET.md @@ -0,0 +1,98 @@ +# Fester Operator Cheatsheet + +Distributed Build + Scheduler + Observability System + +--- + +## Starting a Build + +```bash +fester build ./project.yaml +``` + +## Key Concepts + +**Nodes** — Machines participating in builds (physical, VM, or container). + +**Targets** — Compilation environments (x86_64-linux-gnu, aarch64-linux-gnu, +riscv64, embedded/toolchain targets). + +**Actions** — Graph nodes representing individual build steps. + +**Scheduler** — Chooses the best node based on load, temperature, policy +rules, and historical performance. + +## Build Systems View + +```text +Cockpit Module: Fester +Live DAG: /ui/live_dag.html +Replay View: /ui/replay.html +``` + +## Debugging + +```bash +# Failure Autopsy +GET /api/autopsy/ + +# Timeline Replay +GET /api/replay/ +``` + +## Cache System + +Supports multiple backends: + +- ccache (local compiler cache) +- MinIO (distributed cache) +- Btrfs CoW snapshots (reflink) +- tmpfs (in-memory acceleration) +- Shared CAS API (`/api/cas/`) + +## Snapshots + +Freeze build state using: + +- QCOW2 image snapshots +- Btrfs copy-on-write states + +## Node Control + +```bash +fester node list +fester node set-policy preferred +fester node set-policy avoid +``` + +## Scheduler Modes + +| Mode | Description | +|------------------|-------------------------------------------------| +| unified | Default weighted scoring | +| weighted | Thermal-aware throttling above 85 C | +| cache-first | Prioritize nodes with cached artifacts | +| target-isolated | Each target arch runs on designated nodes | +| intelligence | Experimental ML-driven scheduling | + +## Safety Notes + +- Do not run unrestricted builds on production nodes. +- Monitor thermal load in Grafana. +- Ensure cache integrity for distributed builds. +- Validate cross-compile toolchains before enabling targets. +- Use a reverse proxy with authentication for public exposure. + +## Design Principle + +Every build is reproducible, explainable, and replayable. + +## System Maturity + +| Level | Capability | +|-------|---------------------------------------------------| +| 1 | Distributed compiler system (distcc-like) | +| 2 | Smart scheduler (load/thermal/cache aware) | +| 3 | Observability system (metrics + Grafana + Cockpit)| +| 4 | Causal execution graph (why decisions happen) | +| 5 | Replayable build brain (session + timeline + autopsy)| \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100755 index 0000000..0bf1fc1 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,145 @@ +# Contributing to Fester + +Thanks for your interest in improving Fester! This document covers the basics. + +## 🚀 Quick Start for Contributors + +```bash +# 1. Fork + clone +git clone https://git.dcos.net/dcosnet/fester.git +cd fester + +# 2. Bootstrap dev environment (creates venv, installs deps + dev tools) +./install.sh all --dev + +# 3. Run the backend in dev mode (auto-reload on changes) +./install.sh run --dev + +# 4. Run tests (when we have them) +pytest + +# 5. Lint + format +ruff check . +ruff format . +``` + +## 🏗️ Architecture Overview + +Read **[README.md](README.md)** first — it has the full architecture diagram and project layout. + +The key insight: Fester has a strict separation between: +- **EventBus** — the singleton message bus (all events flow through it) +- **Subscribers** — bus listeners that index/journal/broadcast events (TimelineStore, cause_graph, FailurePropagator, the WS broadcaster) +- **Routers** — FastAPI APIRouters under `backend/api/` that expose REST endpoints +- **UI** — vanilla HTML/JS pages under `ui/` that subscribe to the WS stream and call REST endpoints + +When you add a new feature, ask: +1. Does this emit events? → Use `bus.emit(EventType.X, ...)` from `backend/events/schema.py` +2. Does this need persistence? → Add a method to `backend/storage/sqlite_db.py:Storage` +3. Does this need a UI? → Add a page under `ui/` and a nav entry in `ui/app.js:Fester.mountShell()` +4. Does this need a CLI command? → Add a subcommand to `cli/fester.py` + +## 📝 Coding Standards + +### Python + +- **Python 3.12+** — use modern syntax (match statements, type hints, etc.) +- **Type hints** — add them to new code; don't worry about retrofitting old code +- **Imports** — use `from backend.X.Y import Z` (not bare `from X.Y import Z`) +- **EventBus** — always emit events via `bus.emit()`, never mutate state directly from a request handler +- **Pydantic** — use `BaseModel` for request bodies in routers +- **Async** — use `async def` for endpoints that touch the DB or do I/O + +### JavaScript (UI) + +- **Vanilla JS** — no frameworks (React, Vue, etc.). Keep it dependency-free. +- **Shared shell** — every page calls `Fester.mountShell('page-id')` to get the topbar + WS connection +- **API calls** — use `Fester.api(path, opts)` (returns parsed JSON) +- **Toasts** — use `Fester.toast(msg, kind)` for user notifications +- **No build step** — pages load directly via ` + + + + + + +
+
+
+

Debugger

+
+ + + +
+
+ 

+    

+

+
+
+
+
diff --git a/config.test.yaml b/config.test.yaml
new file mode 100755
index 0000000..6ab2114
--- /dev/null
+++ b/config.test.yaml
@@ -0,0 +1,20 @@
+master:
+  name: fester-master
+  role: control
+
+nodes:
+  - name: x99-v3
+    host: 127.0.0.1
+    max_jobs: 24
+
+  - name: x99-v4
+    host: 127.0.0.1
+    max_jobs: 30
+
+projects:
+  - name: linux-tool
+    repo: https://forgejo.local/linux-tool.git
+    targets:
+      debian: "make clean && make debian"
+      arch: "make clean && make arch"
+      fedora: "make clean && make fedora"
diff --git a/config.yaml b/config.yaml
new file mode 100755
index 0000000..d6d2a79
--- /dev/null
+++ b/config.yaml
@@ -0,0 +1,163 @@
+master:
+  name: fester-master
+  role: control
+
+# Sorcery-go bridge — when enabled, Fester integrates with sorcery-go's
+# eBPF warding for container-level security enforcement. The sorcery-go
+# FesterClient connects to this Fester master for distributed scheduling.
+sorcery_go:
+  enabled: false
+  # URL of the sorcery-go Coven master (sorcery-go connects TO us)
+  # This is informational — sorcery-go uses FESTER_URL env var.
+  eBPF_warding: true   # sorcery-go eBPF cgroup filters on LXC/Podman
+  coven_sync: false    # sync node list from sorcery-go Coven
+
+# Default runtime for nodes that don't specify one.
+# Options: host, lxc, podman, firecracker, libvirt, tmux
+default_runtime: host
+
+# Firecracker defaults (used when a node has runtime: firecracker)
+firecracker:
+  bin: firecracker
+  kernel: /var/lib/fester/vmlinux
+  rootfs: /var/lib/fester/rootfs.ext4
+  ssh_key: /root/.ssh/fc_builder
+  ssh_port: 2222
+  vcpus: 2
+  memory_mb: 4096
+  socket_dir: /run/fester/firecracker
+
+nodes:
+  - name: x99-v3
+    host: 192.168.1.10
+    max_jobs: 24
+    runtime: host
+
+  - name: x99-v4
+    host: 192.168.1.11
+    max_jobs: 30
+    runtime: host
+
+  # Example LXC node — ARMv7 cross-compile via SourceMage chroot
+  # - name: lxc-arm-builder
+  #   host: 192.168.1.20
+  #   max_jobs: 8
+  #   runtime: lxc
+  #   container: smgl-armv7
+  #   toolchain:
+  #     provider: sourcemage
+  #     path: /var/lib/smgl-armv7
+  #     target_arch: arm
+
+  # Example Gentoo stage3 cross-compile node
+  # - name: gentoo-mips-builder
+  #   host: 192.168.1.22
+  #   max_jobs: 16
+  #   runtime: host
+  #   toolchain:
+  #     provider: gentoo
+  #     path: /var/lib/gentoo-chroot
+  #     target_arch: mipsel
+
+  # Example LEDE/OpenWrt router SDK node
+  # - name: lede-router-builder
+  #   host: 192.168.1.23
+  #   max_jobs: 4
+  #   runtime: host
+  #   toolchain:
+  #     provider: lede
+  #     path: /opt/lede-sdk
+
+  # Example Buildroot SDK node
+  # - name: buildroot-arm64
+  #   host: 192.168.1.24
+  #   max_jobs: 8
+  #   runtime: host
+  #   toolchain:
+  #     provider: buildroot
+  #     path: /opt/buildroot-arm64
+
+  # Example Lunar Linux chroot node
+  # - name: lunar-x86-builder
+  #   host: 192.168.1.25
+  #   max_jobs: 12
+  #   runtime: host
+  #   toolchain:
+  #     provider: lunar
+  #     path: /var/lib/lunar-chroot
+
+  # Example generic chroot node (any distro)
+  # - name: debian-chroot-builder
+  #   host: 192.168.1.26
+  #   max_jobs: 12
+  #   runtime: host
+  #   toolchain:
+  #     provider: chroot
+  #     path: /var/lib/debian-chroot
+
+  # Example Podman node (rootless)
+  # - name: podman-x86
+  #   host: 192.168.1.21
+  #   max_jobs: 12
+  #   runtime: podman
+  #   container: smgl-x86_64
+
+  # Example Firecracker microVM node
+  # - name: fc-builder-1
+  #   host: 192.168.1.30
+  #   max_jobs: 4
+  #   runtime: firecracker
+  #   firecracker:
+  #     kernel: /var/lib/fester/vmlinux-arm64
+  #     rootfs: /var/lib/fester/rootfs-arm64.ext4
+  #     ssh_port: 2222
+
+# ---------------------------------------------------------------------------
+# Toolchain provider system
+# ---------------------------------------------------------------------------
+# Controls which toolchain provider Fester uses for builds.
+#
+# provider: (optional) one of: native, btc, buildroot, sourcemage, gentoo,
+#            lede, lunar, chroot
+#   - If omitted, selection falls through: per-node -> global -> btc -> native
+#   - "native" = host gcc/g++ (default fallback)
+#   - "btc"    = BTC.sh sovereign-forged toolchain (if btc.enabled)
+#   - "chroot" = generic distro chroot (auto-detect)
+#   - Others are distro-specific chroot/buildroot providers.
+#
+# target_arch: (optional) for cross-compilation, select target architecture
+#   (arm, arm64, mipsel, x86_64, tilegx, riscv64, etc.)
+# ---------------------------------------------------------------------------
+# toolchain:
+#   provider: native
+
+# BTC Sovereign Toolchain Integration (BTC-0.4.0+ Cross-Compilation)
+# Set enabled: true to use the BTC.sh sovereign-forged toolchain.
+# This is checked AFTER the toolchain.provider setting above.
+#
+# BTC 0.4.0 supports multi-architecture cross-compilation.  The 'target'
+# field selects which cross-compiler golden image to use on this node.
+# If omitted, Fester auto-detects the first available golden image.
+#
+# Available target IDs (run 'BTC.sh --list' for the authoritative table):
+#   x86_64:  haswell, haswell-ep, skylake, skylake-x, skylake-server,
+#            znver1, znver2, znver3, znver4,
+#            apu-zn1, apu-zn2, apu-zn3, apu-zn4,
+#            atom-silvermont, atom-goldmont, atom-tremont, atom-sierraforest
+#   mipsel:  mipselr2
+#   arm:     armv7
+#   tilegx:  tilegx
+btc:
+  enabled: true
+  root: /opt/BTC
+  # target: haswell-ep    # optional: select a specific cross-compiler target
+
+projects:
+
+  - name: linux-tool
+    repo: https://forgejo.local/linux-tool.git
+
+    targets:
+      debian: "make clean && make debian"
+      arch: "make clean && make arch"
+      fedora: "make clean && make fedora"
\ No newline at end of file
diff --git a/debugger/interactive_engine.py b/debugger/interactive_engine.py
new file mode 100755
index 0000000..cf0195f
--- /dev/null
+++ b/debugger/interactive_engine.py
@@ -0,0 +1,124 @@
+# debugger/interactive_engine.py
+
+import asyncio
+
+class InteractiveDebuggerEngine:
+
+    def __init__(self, nodes, graph, scheduler, timeline, ws):
+
+        self.nodes = nodes
+        self.graph = graph
+        self.scheduler = scheduler
+        self.timeline = timeline
+        self.ws = ws
+
+        self.paused = True
+        self.step_mode = True
+        self.current_index = 0
+        self.actions = []
+
+    # -----------------------------
+    # LOAD EXECUTION PLAN
+    # -----------------------------
+    def load_plan(self, actions):
+        self.actions = actions
+        self.current_index = 0
+
+    # -----------------------------
+    # CONTROL
+    # -----------------------------
+    def pause(self):
+        self.paused = True
+
+    def resume(self):
+        self.paused = False
+
+    def step(self):
+        self.paused = False
+
+    # -----------------------------
+    # MAIN EXECUTION LOOP
+    # -----------------------------
+    async def run(self):
+
+        while self.current_index < len(self.actions):
+
+            action = self.actions[self.current_index]
+
+            # -----------------------------
+            # SCHEDULER PREVIEW (NO EXECUTION YET)
+            # -----------------------------
+            decision = self.scheduler.choose_best_node(
+                self.nodes,
+                action
+            )
+
+            preview_event = {
+                "type": "debugger_preview",
+                "data": {
+                    "action": action["name"],
+                    "selected_node": decision["best_node"]["name"],
+                    "alternatives": decision["explanation"]["top_candidates"],
+                    "state": "awaiting_step"
+                }
+            }
+
+            await self.ws.broadcast(preview_event)
+            self.timeline.record(preview_event)
+
+            # -----------------------------
+            # WAIT FOR USER STEP
+            # -----------------------------
+            while self.paused:
+                await asyncio.sleep(0.05)
+
+            # auto-pause again after step
+            self.paused = True
+
+            # -----------------------------
+            # EXECUTE ACTION
+            # -----------------------------
+            node = decision["best_node"]
+
+            exec_event = {
+                "type": "node",
+                "data": {
+                    "action": action["name"],
+                    "node": node["name"],
+                    "state": "running",
+                    "mode": "debug_step"
+                }
+            }
+
+            await self.ws.broadcast(exec_event)
+            self.timeline.record(exec_event)
+
+            # simulate execution hook (replace with real executor)
+            rc = await self.execute(action, node)
+
+            final_state = "done" if rc == 0 else "failed"
+
+            final_event = {
+                "type": "node",
+                "data": {
+                    "action": action["name"],
+                    "node": node["name"],
+                    "state": final_state,
+                    "mode": "debug_step"
+                }
+            }
+
+            await self.ws.broadcast(final_event)
+            self.timeline.record(final_event)
+
+            self.current_index += 1
+
+    # -----------------------------
+    # EXECUTION HOOK
+    # -----------------------------
+    async def execute(self, action, node):
+        """
+        Replace with distcc / lxc / libvirt execution layer
+        """
+        await asyncio.sleep(0.2)
+        return 0
diff --git a/docker-compose.yml b/docker-compose.yml
new file mode 100755
index 0000000..1a629e1
--- /dev/null
+++ b/docker-compose.yml
@@ -0,0 +1,87 @@
+# docker-compose.yml for Fester
+# ============================================================
+# Quick start:
+#   docker-compose up -d
+#
+# Then open http://localhost:8181
+#
+# Volumes:
+#   fester-db      — SQLite database (builds, sessions, events)
+#   fester-storage — snapshots, cache, CAS
+#   fester-config  — config.yaml, storage.json, node_roles.json
+# ============================================================
+
+version: "3.8"
+
+services:
+  fester:
+    build: .
+    image: fester:latest
+    container_name: fester
+    restart: unless-stopped
+    ports:
+      - "8181:8181"
+    volumes:
+      - fester-db:/var/lib/fester
+      - fester-storage:/var/lib/fester/storage
+      - fester-config:/etc/fester
+      - ./config.yaml:/app/config.yaml:ro
+    environment:
+      - FESTER_DB_PATH=/var/lib/fester/fester.db
+      - FESTER_CONFIG=/app/config.yaml
+      - FESTER_NO_DRIFT=0
+      - FESTER_AUTOBUILD=0
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8181/api/health"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+
+  # Optional: MinIO for distributed cache
+  # Uncomment to enable
+  # minio:
+  #   image: minio/minio:latest
+  #   container_name: fester-minio
+  #   restart: unless-stopped
+  #   ports:
+  #     - "9000:9000"
+  #     - "9001:9001"
+  #   volumes:
+  #     - minio-data:/data
+  #   environment:
+  #     - MINIO_ROOT_USER=minioadmin
+  #     - MINIO_ROOT_PASSWORD=minioadmin
+  #   command: server /data --console-address ":9001"
+
+  # Optional: Prometheus scraper
+  # prometheus:
+  #   image: prom/prometheus:latest
+  #   container_name: fester-prometheus
+  #   restart: unless-stopped
+  #   ports:
+  #     - "9090:9090"
+  #   volumes:
+  #     - ./add-to-prometheus-config.yml:/etc/prometheus/prometheus.yml:ro
+  #     - prometheus-data:/prometheus
+
+  # Optional: Grafana dashboard
+  # grafana:
+  #   image: grafana/grafana:latest
+  #   container_name: fester-grafana
+  #   restart: unless-stopped
+  #   ports:
+  #     - "3000:3000"
+  #   volumes:
+  #     - grafana-data:/var/lib/grafana
+  #     - ./add-to-grafana-config.json:/etc/grafana/provisioning/dashboards/fester.json:ro
+  #   environment:
+  #     - GF_SECURITY_ADMIN_PASSWORD=admin
+
+volumes:
+  fester-db:
+  fester-storage:
+  fester-config:
+  # minio-data:
+  # prometheus-data:
+  # grafana-data:
diff --git a/docs/screenshots/01_dashboard.png b/docs/screenshots/01_dashboard.png
new file mode 100755
index 0000000..d1a6f4f
Binary files /dev/null and b/docs/screenshots/01_dashboard.png differ
diff --git a/docs/screenshots/02_live_dag.png b/docs/screenshots/02_live_dag.png
new file mode 100755
index 0000000..eb107ef
Binary files /dev/null and b/docs/screenshots/02_live_dag.png differ
diff --git a/docs/screenshots/03_replay.png b/docs/screenshots/03_replay.png
new file mode 100755
index 0000000..bf9513a
Binary files /dev/null and b/docs/screenshots/03_replay.png differ
diff --git a/docs/screenshots/04_sessions.png b/docs/screenshots/04_sessions.png
new file mode 100755
index 0000000..d9bd107
Binary files /dev/null and b/docs/screenshots/04_sessions.png differ
diff --git a/docs/screenshots/05_metrics.png b/docs/screenshots/05_metrics.png
new file mode 100755
index 0000000..6935478
Binary files /dev/null and b/docs/screenshots/05_metrics.png differ
diff --git a/docs/screenshots/06_cause.png b/docs/screenshots/06_cause.png
new file mode 100755
index 0000000..9292c82
Binary files /dev/null and b/docs/screenshots/06_cause.png differ
diff --git a/docs/screenshots/07_timeline.png b/docs/screenshots/07_timeline.png
new file mode 100755
index 0000000..ec05bb2
Binary files /dev/null and b/docs/screenshots/07_timeline.png differ
diff --git a/docs/screenshots/08_debugger.png b/docs/screenshots/08_debugger.png
new file mode 100755
index 0000000..000445f
Binary files /dev/null and b/docs/screenshots/08_debugger.png differ
diff --git a/donate.md b/donate.md
new file mode 100755
index 0000000..5aa870b
--- /dev/null
+++ b/donate.md
@@ -0,0 +1,101 @@
+# 💚 Support Fester
+
+Fester is free, open-source software released under the AGPL-3.0. It's built and maintained by contributors on their own time and hardware.
+
+If Fester has saved you time, helped you ship a build, or just made your cluster more observable — **thank you for considering a donation**. Every contribution helps cover:
+
+- ⚡ **Build cluster electricity** (the test cluster runs 24/7)
+- 🖥️ **Hardware replacements** (fans, SSDs, thermal paste — builds are hard on hardware)
+- 🌐 **Domain + git hosting** (git.dcos.net, forgejo instances)
+- ☕ **Caffeine** (the fuel that keeps DAGs flowing)
+
+---
+
+## 💰 Ways to Donate
+
+### Cryptocurrency (preferred — no platform fees)
+
+| Currency | Network | Address |
+|----------|---------|---------|
+| **Bitcoin** | BTC (mainnet) | `REPLACE_WITH_YOUR_BTC_ADDRESS` |
+| **Bitcoin** | Lightning | `REPLACE_WITH_YOUR_LIGHTNING_ADDRESS_OR_LNURL` |
+| **Monero** | XMR (mainnet) | `REPLACE_WITH_YOUR_XMR_ADDRESS` |
+| **Ethereum** | ERC-20 | `REPLACE_WITH_YOUR_ETH_ADDRESS` |
+| **USDC** | ERC-20 | `REPLACE_WITH_YOUR_USDC_ADDRESS` |
+| **USDC** | Polygon | `REPLACE_WITH_YOUR_USDC_POLYGON_ADDRESS` |
+| **SOL** | Solana | `REPLACE_WITH_YOUR_SOL_ADDRESS` |
+
+### Fiat / Card
+
+| Platform | Link |
+|----------|------|
+| **GitHub Sponsors** | `REPLACE_WITH_YOUR_GITHUB_SPONSORS_URL` |
+| **Ko-fi** | `REPLACE_WITH_YOUR_KOFI_URL` |
+| **Liberapay** | `REPLACE_WITH_YOUR_LIBERAPAY_URL` |
+| **Open Collective** | `REPLACE_WITH_YOUR_OPENCOLLECTIVE_URL` |
+| **Patreon** | `REPLACE_WITH_YOUR_PATREON_URL` |
+
+### Other
+
+| Method | Details |
+|--------|---------|
+| **PayPal** | `REPLACE_WITH_YOUR_PAYPAL_EMAIL_OR_ME_LINK` |
+| **Bank transfer (SEPA)** | `REPLACE_WITH_YOUR_SEPA_IBAN` (or email for details) |
+| **Hardware donations** | Email `REPLACE_WITH_YOUR_CONTACT_EMAIL` — we always need more nodes! |
+
+---
+
+## 🎯 Sponsorship Tiers
+
+If you'd like ongoing sponsorship with visibility, get in touch:
+
+| Tier | Suggested Monthly | Perks |
+|------|-------------------|-------|
+| ☕ **Coffee** | $5 | Your name in `DONORS.md` |
+| 🖥️ **Node** | $25 | Above + priority issue triage |
+| 🏗️ **Cluster** | $100 | Above + logo on README + input on roadmap |
+| 🚀 **Sustaining** | $500+ | Above + co-branded release notes + dedicated support channel |
+
+Sponsors are listed in **[DONORS.md](DONORS.md)** (with their permission).
+
+---
+
+## 🏢 Corporate Sponsorship
+
+If Fester is critical to your business and you want dedicated support, SLAs, or custom features:
+
+- 📧 Email: `REPLACE_WITH_YOUR_BUSINESS_EMAIL`
+- We can invoice monthly/quarterly/annually
+- Custom feature development is available
+- On-prem deployment assistance
+
+---
+
+## 🛠️ Non-Financial Contributions
+
+Money isn't the only way to help:
+
+- ⭐ **Star the repo** on [git.dcos.net/dcosnet/fester](https://git.dcos.net/dcosnet/fester)
+- 🐛 **File bug reports** with clear reproduction steps
+- ✨ **Submit PRs** — see **[CONTRIBUTING.md](CONTRIBUTING.md)**
+- 📣 **Spread the word** — write a blog post, mention us on forums
+- 🧪 **Test on your hardware** — especially non-x86_64 arches (arm64, riscv64)
+- 📖 **Improve docs** — quickstart, cheatsheet, architecture notes
+
+---
+
+## 📋 Transparency
+
+- All donations are tracked publicly in **[DONORS.md](DONORS.md)** (unless you request anonymity)
+- Quarterly reports will be posted showing how funds were used
+- Donations do **not** grant special access or priority on the issue tracker (that's what sponsorship tiers are for)
+
+---
+
+## 🙏 Thank You
+
+To everyone who has donated, starred, contributed, or just tried Fester out — **you're the reason this project exists**. The build must flow.
+
+---
+
+*Donation addresses last updated: 2026-06-28. Always check this file for the most current addresses.*
diff --git a/fester.js b/fester.js
new file mode 100755
index 0000000..dc1db9e
--- /dev/null
+++ b/fester.js
@@ -0,0 +1,48 @@
+document.addEventListener("DOMContentLoaded", function () {
+
+    function color(heat) {
+        if (heat < 30) return "#2ecc71";
+        if (heat < 60) return "#f1c40f";
+        if (heat < 80) return "#e67e22";
+        return "#e74c3c";
+    }
+
+    document.getElementById("load").addEventListener("click", function () {
+
+        cockpit.spawn(
+            ["python3", "/usr/share/cockpit/fester/backend/nodes_cli.py"],
+            { superuser: "require" }
+        )
+        .then(data => {
+
+            const parsed = JSON.parse(data);
+
+            let html = `
Master

+                        
${JSON.stringify(parsed.master, null, 2)}

+                        
Nodes
`;
+
+            parsed.nodes.forEach(n => {
+
+                html += `
+                    

+                        ${n.name}

+                        state: ${n.state}

+                        heat: ${n.heat}

+                    

+                `;
+            });
+
+            document.getElementById("nodes").innerHTML = html;
+        })
+        .catch(err => {
+            document.getElementById("nodes").textContent = err.toString();
+        });
+    });
+
+});
diff --git a/index.html b/index.html
new file mode 100755
index 0000000..11cd491
--- /dev/null
+++ b/index.html
@@ -0,0 +1,618 @@
+
+
+
+
+
+Fester — Dashboard
+
+
+
+
+
+

+
+    
+    

+        

+            
Cluster

+            

+                
+                
+            

+        

+        

+            

+                

+                    
Nodes Online

+                    
—

+                

+                

+                    
Avg Heat

+                    
—

+                

+                

+                    
Active Jobs

+                    
—

+                

+                

+                    
Cache Hit %

+                    
—

+                

+            

+
+            
Nodes

+            

+
+            
Scheduler Mode

+            
+        

+    

+
+    
+    

+        

+            
Live Event Stream

+            

+                
+                
+                
+                
+            

+        

+        

+            

+            

+                

+            

+        

+    

+
+    
+    

+        

+            
Quick Actions

+        

+        

+            

+                
+                
+            

+            

+                
+                
+            

+            

+                
+            

+
+            

+                
+                
+            

+            
+
+            

+
+            
Targets

+            

+
+            

+
+            
Pipeline Control

+            

+                
+                
+                
+            

+            

+                
+                
+                
+                
+            

+
+            

+
+            
Release Control

+            
—

+        

+    

+
+

+
+
+
+
+
diff --git a/install-fester.sh b/install-fester.sh
new file mode 100755
index 0000000..b3ec2ae
--- /dev/null
+++ b/install-fester.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# ============================================================
+# DEPRECATED — kept only to not break existing docs / workflows.
+# Forwards to the unified install.sh. Will be removed in a future
+# release.
+# ============================================================
+set -euo pipefail
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+echo "[install-fester.sh] deprecated — forwarding to install.sh" >&2
+# install-fester.sh's default behaviour was a system-wide deploy.
+# Map that to: deploy  + systemd + observability.
+BASE_DIR="${BASE_DIR:-/usr/share/cockpit/fester}"
+exec "$REPO_ROOT/install.sh" deploy "$BASE_DIR"
diff --git a/install-fester2.sh b/install-fester2.sh
new file mode 100755
index 0000000..404239c
--- /dev/null
+++ b/install-fester2.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# ============================================================
+# DEPRECATED — kept only to not break existing docs / workflows.
+#
+# install-fester2.sh shipped a STUB cli/fester.py (19 lines) that
+# overwrote the real CLI. Do not run it directly. It now forwards
+# to install.sh's `all` subcommand (which preserves the real CLI).
+# Will be removed in a future release.
+# ============================================================
+set -euo pipefail
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+echo "[install-fester2.sh] deprecated — forwarding to install.sh all" >&2
+exec "$REPO_ROOT/install.sh" all "$@"
diff --git a/install.sh b/install.sh
new file mode 100755
index 0000000..4426f35
--- /dev/null
+++ b/install.sh
@@ -0,0 +1,475 @@
+#!/usr/bin/env bash
+# ============================================================
+# Fester — unified installer / lifecycle script
+# ============================================================
+# Replaces the previous bootstrap.sh / install-fester.sh /
+# install-fester2.sh trio with a single dynamic entry point.
+#
+# Every install / run concern maps to a subcommand. Run with no
+# args (or `all`) to do the standard in-place dev setup:
+#
+#   ./install.sh                       # = ./install.sh all
+#   ./install.sh deps                  # venv + pip install (core or --dev)
+#   ./install.sh config                # write config.yaml (interactive / NONINTERACTIVE=1)
+#   ./install.sh db                    # init SQLite
+#   ./install.sh observability         # write Prometheus + Grafana drop-ins
+#   ./install.sh systemd               # write /tmp/fester.service
+#   ./install.sh deploy [BASE_DIR]     # copy code to BASE_DIR (default: /usr/share/cockpit/fester)
+#   ./install.sh run [--dev] [--port N] [--host H] [--mock-agent]
+#   ./install.sh help
+#
+# Environment variables (all optional):
+#   NONINTERACTIVE=1              skip all prompts, take defaults
+#   FESTER_STORAGE_DIR            default: ~/.fester
+#   FESTER_DB_PATH                default: $FESTER_STORAGE_DIR/fester.db
+#   FESTER_CONFIG                 default: ./config.yaml
+#   FESTER_HOST / FESTER_PORT     default: 0.0.0.0 / 8181
+#   FESTER_MINIO_* / FESTER_PROM_URL / FESTER_FC_SSH_KEY  — runtime service endpoints
+# ============================================================
+set -euo pipefail
+
+# ------------------------------------------------------------------
+# Resolve repo root + defaults
+# ------------------------------------------------------------------
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+
+STORAGE_DIR="${FESTER_STORAGE_DIR:-$HOME/.fester}"
+DB_PATH="${FESTER_DB_PATH:-$STORAGE_DIR/fester.db}"
+CONFIG_PATH="${FESTER_CONFIG:-$REPO_ROOT/config.yaml}"
+VENV_DIR="$REPO_ROOT/.venv"
+
+# ------------------------------------------------------------------
+# Logging helpers (no emoji on stderr noise; colour only if tty)
+# ------------------------------------------------------------------
+if [[ -t 1 ]]; then
+    C_BOLD="\033[1m"; C_GREEN="\033[32m"; C_YELLOW="\033[33m"; C_RED="\033[31m"; C_RESET="\033[0m"
+else
+    C_BOLD=""; C_GREEN=""; C_YELLOW=""; C_RED=""; C_RESET=""
+fi
+log()  { printf "%b\n" "${C_BOLD}[$(date +%H:%M:%S)]${C_RESET} $*"; }
+ok()   { printf "%b\n" "${C_GREEN}ok${C_RESET}  $*"; }
+warn() { printf "%b\n" "${C_YELLOW}warn${C_RESET}  $*" >&2; }
+err()  { printf "%b\n" "${C_RED}error${C_RESET}  $*" >&2; }
+
+# ------------------------------------------------------------------
+# Yes/no prompt with default (respects NONINTERACTIVE=1)
+# ------------------------------------------------------------------
+ask() {
+    local prompt="$1" default="${2:-y}" reply
+    if [[ "${NONINTERACTIVE:-0}" == "1" ]]; then
+        echo "$default"
+        return
+    fi
+    local hint="y/n"; [[ "$default" == "y" ]] && hint="Y/n" || hint="y/N"
+    read -rp "$prompt [$hint] " reply
+    reply="${reply:-$default}"
+    case "$reply" in
+        y|Y|yes|YES) echo "y" ;;
+        *) echo "n" ;;
+    esac
+}
+
+ask_value() {
+    local prompt="$1" default="$2"
+    if [[ "${NONINTERACTIVE:-0}" == "1" ]]; then
+        echo "$default"
+        return
+    fi
+    read -rp "$prompt [$default]: " reply
+    echo "${reply:-$default}"
+}
+
+# ------------------------------------------------------------------
+# Subcommand: deps
+# ------------------------------------------------------------------
+cmd_deps() {
+    local dev=0
+    [[ "${1:-}" == "--dev" ]] && dev=1
+
+    log "Checking Python..."
+    command -v python3 >/dev/null 2>&1 || { err "python3 not found (need 3.12+)"; exit 1; }
+    local PY_VERSION PY_MAJOR PY_MINOR
+    PY_VERSION=$(python3 -c "import sys; print(f'{sys.version_info.major}.{sys.version_info.minor}')")
+    PY_MAJOR=${PY_VERSION%%.*}
+    PY_MINOR=${PY_VERSION#*.}; PY_MINOR=${PY_MINOR%%.*}
+    if (( PY_MAJOR < 3 )) || (( PY_MAJOR == 3 && PY_MINOR < 12 )); then
+        err "Python $PY_VERSION found, but 3.12+ required"
+        exit 1
+    fi
+    ok "Python $PY_VERSION"
+
+    log "Setting up virtualenv at $VENV_DIR ..."
+    if [[ ! -d "$VENV_DIR" ]]; then
+        python3 -m venv "$VENV_DIR"
+        ok "created venv"
+    else
+        ok "reusing existing venv"
+    fi
+    # shellcheck disable=SC1091
+    source "$VENV_DIR/bin/activate"
+    python3 -m pip install --upgrade pip --quiet
+    ok "pip upgraded"
+
+    log "Installing Python dependencies..."
+    local deps=(
+        "fastapi>=0.128"
+        "uvicorn[standard]>=0.30"
+        "websockets>=12"
+        "pydantic>=2"
+        "aiohttp>=3.9"
+        "minio>=7.2"
+        "prometheus-client>=0.20"
+        "PyYAML>=6"
+        "requests>=2.31"
+        "websocket-client>=1.7"
+    )
+    pip install --quiet "${deps[@]}"
+    ok "core deps installed"
+
+    if (( dev )); then
+        log "Installing dev tools (ruff, mypy, pytest, httpx)..."
+        pip install --quiet ruff mypy pytest pytest-asyncio httpx
+        ok "dev tools installed"
+    fi
+
+    log "Checking optional system tools..."
+    set +e
+    local tools=(
+        "tmux:live action output viewer"
+        "mosh:shell-into-node button"
+        "ssh:shell-into-node fallback"
+        "qemu-img:qcow2 workspace snapshots"
+        "qemu-nbd:qcow2 mount for freeze"
+        "rsync:workspace sync for freeze"
+        "curl:firecracker API calls"
+    )
+    for entry in "${tools[@]}"; do
+        local tool="${entry%%:*}" purpose="${entry#*:}"
+        if command -v "$tool" >/dev/null 2>&1; then
+            ok "$tool — $purpose"
+        else
+            warn "$tool missing — $purpose (optional)"
+        fi
+    done
+    set -e
+}
+
+# ------------------------------------------------------------------
+# Subcommand: config
+# ------------------------------------------------------------------
+cmd_config() {
+    log "Setting up configuration at $CONFIG_PATH ..."
+
+    if [[ -f "$CONFIG_PATH" ]]; then
+        ok "config.yaml already exists — leaving untouched"
+        return
+    fi
+
+    local CLUSTER_NAME LIBVIRT LXC DISTCC MINIO
+    CLUSTER_NAME=$(ask_value "Cluster name" "fester-cluster")
+    [[ "$(ask "Enable libvirt integration?" "y")" == "y" ]] && LIBVIRT=true || LIBVIRT=false
+    [[ "$(ask "Enable LXC integration?" "y")" == "y" ]]      && LXC=true      || LXC=false
+    [[ "$(ask "Enable distcc integration?" "y")" == "y" ]]   && DISTCC=true   || DISTCC=false
+    [[ "$(ask "Enable MinIO cache backend?" "y")" == "y" ]]  && MINIO=true    || MINIO=false
+
+    cat > "$CONFIG_PATH" <&1 || { warn "DB init issue — will be created on first backend run"; return; }
+    ok "database ready"
+}
+
+# ------------------------------------------------------------------
+# Subcommand: observability
+# ------------------------------------------------------------------
+cmd_observability() {
+    log "Writing Prometheus + Grafana drop-ins ..."
+
+    local out_dir="${1:-$REPO_ROOT}"
+    mkdir -p "$out_dir"
+
+    cat > "$out_dir/add-to-prometheus-config.yml" <<'EOF'
+scrape_configs:
+  - job_name: 'fester'
+    static_configs:
+      - targets: ['localhost:8181']
+    metrics_path: /metrics
+EOF
+    ok "wrote $out_dir/add-to-prometheus-config.yml"
+
+    # Grafana dashboard — wired to real Prometheus metric names exposed
+    # by backend/metrics/prometheus.py:
+    #   fester_node_cpu, fester_node_temp,
+    #   fester_pipeline_actions_total, fester_cache_hits_total,
+    #   fester_scheduler_score
+    cat > "$out_dir/add-to-grafana-config.json" <<'EOF'
+{
+  "dashboard": {
+    "title": "Fester Cluster Overview",
+    "schemaVersion": 39,
+    "version": 1,
+    "time": { "from": "now-1h", "to": "now" },
+    "panels": [
+      {
+        "type": "stat", "title": "Pipeline Actions (done)",
+        "gridPos": { "x": 0, "y": 0, "w": 6, "h": 4 },
+        "targets": [{ "expr": "sum(fester_pipeline_actions_total{state=\"done\"})", "legendFormat": "done" }]
+      },
+      {
+        "type": "stat", "title": "Pipeline Actions (failed)",
+        "gridPos": { "x": 6, "y": 0, "w": 6, "h": 4 },
+        "targets": [{ "expr": "sum(fester_pipeline_actions_total{state=\"failed\"})", "legendFormat": "failed" }]
+      },
+      {
+        "type": "stat", "title": "Cache Hits",
+        "gridPos": { "x": 12, "y": 0, "w": 6, "h": 4 },
+        "targets": [{ "expr": "sum(fester_cache_hits_total)", "legendFormat": "hits" }]
+      },
+      {
+        "type": "stat", "title": "Nodes Tracked",
+        "gridPos": { "x": 18, "y": 0, "w": 6, "h": 4 },
+        "targets": [{ "expr": "count(fester_node_cpu)", "legendFormat": "nodes" }]
+      },
+      {
+        "type": "timeseries", "title": "Per-node CPU Load",
+        "gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
+        "targets": [{ "expr": "fester_node_cpu", "legendFormat": "{{node}}" }]
+      },
+      {
+        "type": "timeseries", "title": "Per-node Temperature (C)",
+        "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
+        "targets": [{ "expr": "fester_node_temp", "legendFormat": "{{node}}" }]
+      },
+      {
+        "type": "timeseries", "title": "Scheduler Score by Node / Target",
+        "gridPos": { "x": 0, "y": 12, "w": 24, "h": 8 },
+        "targets": [{ "expr": "fester_scheduler_score", "legendFormat": "{{node}} / {{target}}" }]
+      }
+    ]
+  }
+}
+EOF
+    ok "wrote $out_dir/add-to-grafana-config.json"
+}
+
+# ------------------------------------------------------------------
+# Subcommand: systemd
+# ------------------------------------------------------------------
+cmd_systemd() {
+    local base_dir="${1:-$REPO_ROOT}"
+    log "Writing systemd unit for $base_dir ..."
+
+    local unit_path="/tmp/fester.service"
+    cat > "$unit_path" </dev/null; then
+            : # created without sudo
+        elif [[ "${EUID:-$(id -u)}" -eq 0 ]]; then
+            mkdir -p "$base_dir"
+        else
+            warn "creating $base_dir requires root — invoking sudo"
+            sudo mkdir -p "$base_dir"
+            sudo chown -R "$USER":"$USER" "$base_dir" 2>/dev/null || true
+        fi
+    fi
+
+    # Ensure we can write into base_dir. If we can't, bail with a clear
+    # message instead of letting cp fail one file at a time.
+    if [[ ! -w "$base_dir" ]]; then
+        err "$base_dir is not writable by $USER — re-run with sudo or pick a different BASE_DIR"
+        exit 1
+    fi
+
+    mkdir -p "$base_dir"/{backend,ui,cockpit/fester-module,cli,docs,config}
+
+    # Copy code (preserve structure)
+    cp -r "$REPO_ROOT/backend/"* "$base_dir/backend/" 2>/dev/null || true
+    cp -r "$REPO_ROOT/ui/"*       "$base_dir/ui/"      2>/dev/null || true
+    cp -r "$REPO_ROOT/cockpit/"*  "$base_dir/cockpit/" 2>/dev/null || true
+    cp    "$REPO_ROOT/cli/fester.py"     "$base_dir/cli/fester.py"
+    cp    "$REPO_ROOT/index.html"        "$base_dir/"        2>/dev/null || true
+    cp    "$REPO_ROOT/style.css"         "$base_dir/"        2>/dev/null || true
+    cp    "$REPO_ROOT/manifest.json"     "$base_dir/"        2>/dev/null || true
+    cp    "$REPO_ROOT/CHEATSHEET.md"     "$base_dir/docs/"   2>/dev/null || true
+    cp    "$REPO_ROOT/README.md"         "$base_dir/docs/"   2>/dev/null || true
+    chmod +x "$base_dir/cli/fester.py"
+    ok "code deployed to $base_dir"
+
+    # Carry config + observability drop-ins along
+    [[ -f "$CONFIG_PATH" ]] && cp "$CONFIG_PATH" "$base_dir/config.yaml"
+    cmd_observability "$base_dir"
+}
+
+# ------------------------------------------------------------------
+# Subcommand: run  (delegates to run.sh so we keep one launcher)
+# ------------------------------------------------------------------
+cmd_run() {
+    exec "$REPO_ROOT/run.sh" "$@"
+}
+
+# ------------------------------------------------------------------
+# Subcommand: all  (the default — does the standard in-place setup)
+# ------------------------------------------------------------------
+cmd_all() {
+    local dev=0
+    [[ "${1:-}" == "--dev" ]] && dev=1
+    log "Running full in-place setup (dev=$dev)"
+    if (( dev )); then cmd_deps --dev; else cmd_deps; fi
+    cmd_config
+    cmd_db
+    cmd_observability
+    chmod +x "$REPO_ROOT/run.sh" "$REPO_ROOT/cli/fester.py" 2>/dev/null || true
+    cat < [args]
+
+Subcommands:
+  all                     venv + deps + config + db + observability  (default)
+      --dev               also install ruff/mypy/pytest/httpx
+  deps                    venv + pip install (core deps only)
+      --dev               also install dev tools
+  config                  write config.yaml interactively (or NONINTERACTIVE=1)
+  db                      init SQLite at \$FESTER_DB_PATH (default: ~/.fester/fester.db)
+  observability [DIR]     write Prometheus + Grafana drop-ins (default: repo root)
+  systemd [BASE_DIR]      write /tmp/fester.service pointed at BASE_DIR
+  deploy [BASE_DIR]       copy code to BASE_DIR (default: /usr/share/cockpit/fester)
+  run [--dev] [--port N] [--host H] [--mock-agent]
+                          launch the FastAPI backend via run.sh
+  help                    this message
+
+Environment:
+  NONINTERACTIVE=1        skip prompts, take defaults
+  FESTER_STORAGE_DIR      default: ~/.fester
+  FESTER_DB_PATH          default: \$FESTER_STORAGE_DIR/fester.db
+  FESTER_CONFIG           default: ./config.yaml
+  FESTER_HOST / FESTER_PORT   default: 0.0.0.0 / 8181
+
+Examples:
+  $0                          # standard in-place dev setup
+  $0 all --dev                # + dev tools
+  NONINTERACTIVE=1 $0 all     # CI mode
+  $0 deploy /opt/fester       # system-wide deploy + systemd unit
+  $0 run --dev --port 9000
+EOF
+}
+
+# ------------------------------------------------------------------
+# Dispatch
+# ------------------------------------------------------------------
+sub="${1:-all}"; shift || true
+case "$sub" in
+    all)             cmd_all "$@" ;;
+    deps)            cmd_deps "$@" ;;
+    config)          cmd_config "$@" ;;
+    db)              cmd_db "$@" ;;
+    observability)   cmd_observability "$@" ;;
+    systemd)         cmd_systemd "$@" ;;
+    deploy)          cmd_deploy "$@" ;;
+    run)             cmd_run "$@" ;;
+    help|-h|--help)  cmd_help ;;
+    *)
+        err "unknown subcommand: $sub"
+        cmd_help
+        exit 1
+        ;;
+esac
diff --git a/manifest.json b/manifest.json
new file mode 100755
index 0000000..fbd6430
--- /dev/null
+++ b/manifest.json
@@ -0,0 +1,10 @@
+{
+  "version": 1,
+  "name": "fester",
+  "tools": {
+    "fester": {
+      "label": "Fester",
+      "path": "index.html"
+    }
+  }
+}
diff --git a/optimizer.py b/optimizer.py
new file mode 100755
index 0000000..902caa1
--- /dev/null
+++ b/optimizer.py
@@ -0,0 +1,12 @@
+from prometheus_client import Gauge
+
+target_score_thermal = Gauge("fester_target_score_thermal", "Thermal score", ["target"])
+target_score_cost = Gauge("fester_target_score_cost", "Cost score", ["target"])
+target_score_instability = Gauge("fester_target_score_instability", "Instability score", ["target"])
+
+
+def export_scores(target, score):
+
+    target_score_thermal.labels(target=target).set(score["thermal"])
+    target_score_cost.labels(target=target).set(score["cost"])
+    target_score_instability.labels(target=target).set(score["instability"])
diff --git a/project.schema.json b/project.schema.json
new file mode 100755
index 0000000..5ecdc40
--- /dev/null
+++ b/project.schema.json
@@ -0,0 +1,170 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Fester Project Definition",
+  "type": "object",
+  "required": ["name", "source", "targets"],
+  "additionalProperties": false,
+
+  "properties": {
+
+    "name": {
+      "type": "string",
+      "description": "Human-readable project name"
+    },
+
+    "version": {
+      "type": "string",
+      "description": "Optional project version tag (release tracking)",
+      "default": "0.0.0"
+    },
+
+    "source": {
+      "type": "object",
+      "required": ["type", "url"],
+      "additionalProperties": false,
+
+      "properties": {
+
+        "type": {
+          "type": "string",
+          "enum": ["git", "hg", "svn", "cvs"],
+          "description": "Source control system type (normalized into snapshot)"
+        },
+
+        "url": {
+          "type": "string",
+          "description": "Repository URL or path for checkout/export"
+        },
+
+        "ref": {
+          "type": "string",
+          "description": "Optional branch/tag/revision reference (git branch, svn rev, hg changeset)",
+          "default": "HEAD"
+        }
+      }
+    },
+
+    "targets": {
+      "type": "object",
+      "description": "Build targets mapped to shell commands",
+      "additionalProperties": {
+        "type": "string"
+      }
+    },
+
+    "resources": {
+      "type": "object",
+      "description": "Optional resource hints for scheduler optimization",
+
+      "properties": {
+
+        "max_parallel_builds": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 256,
+          "default": 64
+        },
+
+        "thermal_priority": {
+          "type": "string",
+          "enum": ["low", "balanced", "high_safety"],
+          "default": "high_safety"
+        },
+
+        "preferred_nodes": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          },
+          "description": "Optional node names preferred for this project"
+        },
+
+        "avoid_nodes": {
+          "type": "array",
+          "items": {
+            "type": "string"
+          },
+          "description": "Nodes to exclude from scheduling"
+        }
+      }
+    },
+
+    "build_env": {
+      "type": "object",
+      "description": "Environment variables injected into build runtime",
+
+      "additionalProperties": {
+        "type": "string"
+      }
+    },
+
+    "artifacts": {
+      "type": "object",
+      "description": "Output packaging configuration",
+
+      "properties": {
+
+        "enabled": {
+          "type": "boolean",
+          "default": true
+        },
+
+        "format": {
+          "type": "string",
+          "enum": ["tar", "tar.gz", "tar.zst", "deb", "rpm"],
+          "default": "tar.zst"
+        },
+
+        "output_dir": {
+          "type": "string",
+          "default": "./dist"
+        },
+
+        "naming": {
+          "type": "string",
+          "description": "Artifact naming template",
+          "default": "{name}-{version}-{target}-{hash}"
+        }
+      }
+    },
+
+    "cache": {
+      "type": "object",
+      "description": "Build cache behavior control",
+
+      "properties": {
+
+        "enabled": {
+          "type": "boolean",
+          "default": true
+        },
+
+        "reuse_snapshots": {
+          "type": "boolean",
+          "default": true
+        },
+
+        "skip_if_unchanged": {
+          "type": "boolean",
+          "default": true
+        }
+      }
+    },
+
+    "logging": {
+      "type": "object",
+      "properties": {
+
+        "verbose": {
+          "type": "boolean",
+          "default": true
+        },
+
+        "stream_to_ui": {
+          "type": "boolean",
+          "default": true
+        }
+      }
+    }
+  }
+}
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100755
index 0000000..6312395
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,177 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "fester"
+version = "0.3.0"
+description = "Distributed, DAG-driven build execution system with real-time scheduling, observability, and deterministic replay"
+readme = "README.md"
+license = { text = "AGPL-3.0-or-later", file = "LICENSE" }
+authors = [
+    { name = "Fester Contributors" },
+]
+requires-python = ">=3.12"
+keywords = ["build", "distributed", "scheduler", "dag", "compile", "distcc", "observability"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Environment :: Web Environment",
+    "Framework :: FastAPI",
+    "Intended Audience :: Developers",
+    "Intended Audience :: System Administrators",
+    "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Build Tools",
+    "Topic :: System :: Distributed Computing",
+]
+
+# Core runtime dependencies
+dependencies = [
+    "fastapi>=0.128",
+    "uvicorn[standard]>=0.30",
+    "websockets>=12",
+    "pydantic>=2",
+    "aiohttp>=3.9",
+    "minio>=7.2",
+    "prometheus-client>=0.20",
+    "PyYAML>=6",
+    "requests>=2.31",
+    "websocket-client>=1.7",
+]
+
+[project.optional-dependencies]
+# Install with: pip install -e ".[dev]"
+dev = [
+    "ruff>=0.5",
+    "mypy>=1.10",
+    "pytest>=8",
+    "pytest-asyncio>=0.23",
+    "httpx>=0.27",  # for FastAPI TestClient
+    "respx>=0.21",  # for mocking aiohttp in tests
+]
+# Install with: pip install -e ".[docker]"
+docker = [
+    "gunicorn>=22",
+]
+
+[project.scripts]
+# After `pip install -e .`, you can run `fester` from anywhere
+fester = "cli.fester:main"
+
+[project.urls]
+Homepage = "https://git.dcos.net/dcosnet/fester"
+Repository = "https://git.dcos.net/dcosnet/fester"
+Documentation = "https://git.dcos.net/dcosnet/fester/blob/main/README.md"
+Issues = "https://git.dcos.net/dcosnet/fester/issues"
+
+# ------------------------------------------------------------
+# Setuptools package discovery
+# ------------------------------------------------------------
+[tool.setuptools]
+# We're not a pure-Python package — just expose the CLI + backend modules
+py-modules = []
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = [
+    "backend*",
+    "cli*",
+    "cockpit*",
+]
+exclude = [
+    "tests*",
+    "docs*",
+    "scripts*",
+]
+
+[tool.setuptools.package-data]
+# Include non-Python files in the package
+"*" = ["*.html", "*.css", "*.js", "*.yaml", "*.json", "*.md"]
+
+# ------------------------------------------------------------
+# Ruff (linter + formatter)
+# ------------------------------------------------------------
+[tool.ruff]
+line-length = 100
+target-version = "py312"
+extend-exclude = [
+    "docs",
+    "scripts/mock_agent.py",
+]
+
+[tool.ruff.lint]
+# Enable common rule sets
+select = [
+    "E",    # pycodestyle errors
+    "W",    # pycodestyle warnings
+    "F",    # pyflakes
+    "I",    # isort
+    "B",    # flake8-bugbear
+    "C4",   # flake8-comprehensions
+    "UP",   # pyupgrade
+    "RUF",  # ruff-specific
+]
+ignore = [
+    "E501",   # line too long (we use 100 but don't want to fail on it)
+    "B008",   # do not perform function calls in argument defaults (FastAPI does this)
+    "B904",   # raise from (we have a lot of legacy code)
+    "RUF001", # ambiguous unicode (we use emojis)
+    "RUF002", # ambiguous unicode in docstring
+    "RUF003", # ambiguous unicode in comment
+]
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["F401"]  # re-exports
+"cli/fester.py" = ["E402"]  # module-level imports after argparse helpers
+
+[tool.ruff.format]
+quote-style = "double"
+indent-style = "space"
+
+# ------------------------------------------------------------
+# Mypy (type checking)
+# ------------------------------------------------------------
+[tool.mypy]
+python_version = "3.12"
+warn_return_any = false  # too noisy for this codebase
+warn_unused_configs = true
+disallow_untyped_defs = false  # we have legacy untyped code
+disallow_incomplete_defs = false
+check_untyped_defs = true
+disallow_untyped_decorators = false
+no_implicit_optional = true
+warn_redundant_casts = true
+warn_unused_ignores = true
+warn_no_return = true
+warn_unreachable = true
+strict_equality = true
+show_error_codes = true
+
+[[tool.mypy.overrides]]
+module = [
+    "minio.*",
+    "aiohttp.*",
+    "websocket.*",
+    "prometheus_client.*",
+]
+ignore_missing_imports = true
+
+# ------------------------------------------------------------
+# Pytest
+# ------------------------------------------------------------
+[tool.pytest.ini_options]
+minversion = "8.0"
+testpaths = ["tests"]
+asyncio_mode = "auto"
+filterwarnings = [
+    "ignore::DeprecationWarning",
+    "ignore::PendingDeprecationWarning",
+]
+markers = [
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+    "integration: marks tests that require external services (minio, etc.)",
+]
diff --git a/quickstart.md b/quickstart.md
new file mode 100755
index 0000000..1e45952
--- /dev/null
+++ b/quickstart.md
@@ -0,0 +1,334 @@
+# 🚀 Fester Quickstart
+
+**5 minutes to your first distributed build.**
+
+---
+
+## 1. Prerequisites
+
+You need:
+- **Python 3.12+** (`python3 --version`)
+- **pip** (usually ships with Python)
+- **git** (to clone the repo)
+- **curl** (to test the API)
+
+Optional (features activate automatically when installed):
+- `tmux` — live action output viewer
+- `mosh` + `ssh` — "Shell into node" UI button
+- `qemu-utils` + `rsync` — QCOW2 workspace snapshots
+- `btrfs-progs` — CoW reflink snapshots (requires btrfs partition)
+
+On Ubuntu/Debian:
+```bash
+sudo apt install python3 python3-pip git curl tmux mosh openssh-client qemu-utils rsync
+```
+
+## 2. Clone + Bootstrap
+
+```bash
+git clone https://git.dcos.net/dcosnet/fester.git
+cd fester
+
+# One-shot: creates venv, installs Python deps, sets up config + DB
+./install.sh           # = ./install.sh all
+```
+
+The install script will:
+1. Create a Python virtualenv at `.venv/`
+2. Install all Python dependencies (FastAPI, uvicorn, websockets, minio, aiohttp, etc.)
+3. Create a default `config.yaml` if one doesn't exist (interactive, or `NONINTERACTIVE=1` for defaults)
+4. Initialize the SQLite database at `~/.fester/fester.db`
+5. Write Prometheus + Grafana drop-ins into the repo root
+6. Print next steps
+
+Individual subcommands are also available — see `./install.sh help`:
+```
+./install.sh deps            # just venv + pip install
+./install.sh config          # just write config.yaml
+./install.sh db              # just init SQLite
+./install.sh deploy /opt/f   # system-wide copy + observability drop-ins
+./install.sh systemd         # write /tmp/fester.service
+./install.sh run --dev       # launch backend with auto-reload
+```
+
+The legacy `bootstrap.sh`, `install-fester.sh`, and `install-fester2.sh`
+are kept as deprecation shims that forward to `install.sh`.
+
+## 3. Start the Backend
+
+```bash
+./run.sh
+```
+
+You should see:
+```
+🧠 Fester backend starting...
+   Config:  config.yaml
+   DB:      /home/user/.fester/fester.db
+   URL:     http://0.0.0.0:8181
+   Press Ctrl+C to stop
+INFO:     Uvicorn running on http://0.0.0.0:8181
+```
+
+Verify it's up:
+```bash
+curl http://localhost:8181/api/health
+# → {"status":"ok","version":"0.2.0","bus_subscribers":6,...}
+```
+
+## 4. Open the UI
+
+Open **http://localhost:8181** in your browser.
+
+You'll see the **Dashboard** with:
+- **Cluster panel** (left): node list with heat bars, job counters, policy dropdowns, probe buttons
+- **Live Event Stream** (center): real-time events with type-filter chips
+- **Quick Actions** (right): build form, release form, target toggles, pipeline controls
+
+The topbar shows a live health indicator (version + WS clients + builds count + timeline events).
+
+## 5. Trigger Your First Build
+
+### Via the UI
+
+In the **Quick Actions** panel (right side of the dashboard):
+1. **Build Command**: `make -j$(nproc)` (or any shell command)
+2. **Working Directory**: `/tmp/test-build` (or your project dir)
+3. Click **▶ Run Build**
+
+Watch the **Live Event Stream** populate with `task_update` events as actions execute.
+
+### Via the CLI
+
+```bash
+# Activate the venv first (or use the full path)
+source .venv/bin/activate
+
+# Kick off a build
+fester build --cmd "make -j4" --dir /tmp/test-build --watch
+
+# The --watch flag streams events as they happen
+```
+
+### Via curl
+
+```bash
+curl -X POST http://localhost:8181/api/build \
+  -H "Content-Type: application/json" \
+  -d '{"cmd":"make -j4","dir":"/tmp/test-build","target":"linux-gnu"}'
+```
+
+## 6. Explore the UI
+
+### Live DAG (`/ui/live_dag.html`)
+
+Real-time visualization of the build DAG. Each action becomes a node; dependency edges are drawn as arrows. Node colors:
+- 🟡 **yellow border** = running
+- 🟢 **green border** = done
+- 🔴 **red border** = failed
+- 🔵 **blue border** = cache hit
+- 🟠 **yellow outline** = critical path
+
+Click any node to inspect its details. Use the search box to filter, and the toggles to show/hide critical-path and failed nodes.
+
+### Replay (`/ui/replay.html`)
+
+Scrub through past builds event-by-event:
+1. Click **Load Replay** to snapshot the current timeline
+2. Use the timeline slider, or **▶ Play** / **⏭ Step** / **⏮ Back** buttons
+3. Jump to the end with **⏭ End** to see the full DAG
+4. Click any node to see its state, deps, scheduler decision, and raw event
+
+### Sessions (`/ui/sessions.html`)
+
+Build history + active tmux sessions + qcow2 snapshots:
+- **Replay sessions** — click "▶ Open in Replay" to scrub through any past session
+- **Recent Builds** — cancel running builds, or replay completed ones
+- **Active Action Sessions** — view live tmux output of running actions
+- **Storage Snapshots** — list of qcow2 freeze images
+
+### Metrics (`/ui/metrics.html`)
+
+Live cluster metrics with two trend charts (heat + jobs) and per-node cards showing CPU/memory/heat/jobs/instability bars. Refreshes every 2 seconds.
+
+### Cause Graph (`/ui/cause.html`)
+
+Post-hoc reasoning: enter a node name, see its causal chain as a radial graph. The **Blast Radius** panel computes the downstream impact if any action fails.
+
+### Timeline (`/ui/timeline.html`)
+
+Per-node event drill-down: pick a node from the left, see every event that touched it.
+
+### Debugger (`/ui/debugger.html`)
+
+Step-through execution control: pause/resume/step a running build. Shows the live tmux timeline and the next action preview.
+
+## 7. Configure Your Cluster
+
+Edit `config.yaml`:
+
+```yaml
+master:
+  name: fester-master
+  role: control
+
+nodes:
+  - name: x99-v3
+    host: 192.168.1.10
+    max_jobs: 24
+  - name: x99-v4
+    host: 192.168.1.11
+    max_jobs: 30
+  - name: rpi-1
+    host: 192.168.1.20
+    max_jobs: 4
+
+projects:
+  - name: linux-tool
+    repo: https://forgejo.local/linux-tool.git
+    targets:
+      debian: "make clean && make debian"
+      arch: "make clean && make arch"
+```
+
+Restart `./run.sh` to pick up changes.
+
+## 8. Set Node Policies
+
+Via the UI (Dashboard → node row → policy dropdown):
+- **preferred** — scheduler gives this node a +20 score bonus
+- **neutral** — default
+- **avoid** — scheduler subtracts 50 from this node's score
+
+Via the CLI:
+```bash
+fester node set-policy x99-v3 preferred
+fester node set-policy rpi-1 avoid
+```
+
+Via the API:
+```bash
+curl -X POST http://localhost:8181/api/nodes/x99-v3/policy \
+  -H "Content-Type: application/json" \
+  -d '{"policy":"preferred"}'
+```
+
+## 9. Probe Node Agents (Optional)
+
+If you run a fester-agent on each node (port 8787), Fester will probe it every 4 seconds for real metrics instead of drifting synthetic values.
+
+A mock agent for testing:
+```bash
+python3 scripts/mock_agent.py 8787 &
+```
+
+Manual probe via the UI (Dashboard → node row → "Probe" button) or CLI:
+```bash
+fester node probe x99-v3
+fester node probe-all
+```
+
+## 10. Shell Into a Node
+
+Click the **⌘ Shell** button next to any node in the Dashboard. Fester builds the correct mosh or ssh command and copies it to your clipboard:
+
+```bash
+mosh --ssh "ssh -p 2222 -i ~/.ssh/id_ed25519" build@192.168.1.10
+```
+
+Paste into your terminal to attach.
+
+## 11. Run a Release Build
+
+Via the UI (Dashboard → Quick Actions → Run Release Build) or CLI:
+```bash
+fester release --repo https://forgejo.local/project.git --project my-proj
+```
+
+This kicks off a longer release-shaped pipeline: `fetch_source → configure → build_kernel → build_modules → package → release`.
+
+## 12. Investigate a Failure
+
+When a build fails:
+
+1. **Sessions page** → find the failed build → click **▶ Replay**
+2. In Replay, scrub to the failed action (red node)
+3. Click the failed node → see the failure reason in the inspector
+4. Switch to **Cause Graph** page → enter the node name → see the causal chain
+5. Use **Blast Radius** to see which downstream actions were impacted
+
+Via the CLI:
+```bash
+# Start a replay session
+SID=$(curl -sX POST http://localhost:8181/replay/start \
+  -H "Content-Type: application/json" \
+  -d '{"journal":"latest"}' | jq -r .session_id)
+
+# Run autopsy on the failed action
+fester autopsy $SID build_fedora
+```
+
+## 🎯 Next Steps
+
+- Read the **[CHEATSHEET.md](CHEATSHEET.md)** for the operator survival guide
+- Browse the **[full API](http://localhost:8181/docs)** (FastAPI auto-docs)
+- Set up **Prometheus + Grafana** using the configs in `add-to-prometheus-config.yml` and `add-to-grafana-config.json`
+- Install **tmux**, **mosh**, and **qemu-utils** to activate the advanced features
+- Configure **MinIO** for distributed cache (set the endpoint in `backend/cache/minio_cache.py`)
+
+## 🆘 Troubleshooting
+
+### Backend won't start
+
+```bash
+# Check the log
+tail -f /tmp/fester_backend.log
+
+# Common issues:
+# - "No module named 'fastapi'" → run ./install.sh deps again
+# - "Permission denied: /var/lib/fester" → use FESTER_DB_PATH=/tmp/fester.db
+# - "Address already in use" → another process is on port 8181
+```
+
+### UI shows "CONNECTING" forever
+
+The WebSocket can't reach the backend. Check:
+```bash
+curl http://localhost:8181/api/health
+# If this fails, the backend isn't running
+```
+
+### No events in the Live Event Stream
+
+Trigger a build:
+```bash
+curl -X POST http://localhost:8181/api/build \
+  -H "Content-Type: application/json" \
+  -d '{"cmd":"demo","dir":"/tmp"}'
+```
+
+### Nodes show 0° heat / 0 jobs
+
+No agent is running on the nodes. Either:
+- Start a mock agent: `python3 scripts/mock_agent.py 8787`
+- Or accept synthetic drift (default behavior when agents are unreachable)
+
+### Build fails immediately
+
+Check the build details:
+```bash
+fester builds
+fester build-info 
+```
+
+Common causes:
+- Working directory doesn't exist → create it first
+- Command not found → check PATH
+- Permission denied → check file permissions
+
+## 📚 More Info
+
+- **[README.md](README.md)** — full project overview
+- **[CHEATSHEET.md](CHEATSHEET.md)** — operator cheatsheet
+- **[CONTRIBUTING.md](CONTRIBUTING.md)** — how to contribute
+- **[API docs](http://localhost:8181/docs)** — interactive OpenAPI spec
diff --git a/release.js b/release.js
new file mode 100755
index 0000000..480be98
--- /dev/null
+++ b/release.js
@@ -0,0 +1,18 @@
+document.addEventListener("DOMContentLoaded", function () {
+
+    function runRelease() {
+
+        const repo = document.getElementById("repo").value;
+
+        cockpit.spawn(
+            ["python3", "/usr/share/cockpit/fester/backend/release.py", repo],
+            { superuser: "require" }
+        )
+        .stream(function (data) {
+            document.getElementById("release_output").textContent += data;
+        });
+    }
+
+    document.getElementById("run_release").addEventListener("click", runRelease);
+
+});
diff --git a/run.sh b/run.sh
new file mode 100755
index 0000000..1bc12a8
--- /dev/null
+++ b/run.sh
@@ -0,0 +1,168 @@
+#!/usr/bin/env bash
+# ============================================================
+# Fester Backend Runner
+# ============================================================
+# Starts the FastAPI backend with sensible defaults.
+#
+# Usage:
+#   ./run.sh                    # default: host=0.0.0.0, port=8181
+#   ./run.sh --dev              # dev mode with auto-reload
+#   ./run.sh --mock-agent       # also start a mock fester-agent on :8787
+#   ./run.sh --port 9000        # custom port
+#   ./run.sh --help
+# ============================================================
+set -euo pipefail
+
+# Resolve repo root
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+
+# Defaults
+HOST="${FESTER_HOST:-0.0.0.0}"
+PORT="${FESTER_PORT:-8181}"
+DEV_MODE=0
+START_MOCK_AGENT=0
+
+# Parse args
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --dev)
+            DEV_MODE=1
+            shift
+            ;;
+        --mock-agent)
+            START_MOCK_AGENT=1
+            shift
+            ;;
+        --port)
+            PORT="$2"
+            shift 2
+            ;;
+        --host)
+            HOST="$2"
+            shift 2
+            ;;
+        --help|-h)
+            cat </dev/null; then
+    echo "❌ FastAPI not installed. Run ./bootstrap.sh first."
+    exit 1
+fi
+
+# ------------------------------------------------------------
+# Optionally start a mock fester-agent
+# ------------------------------------------------------------
+MOCK_PID=""
+if [ "$START_MOCK_AGENT" = "1" ]; then
+    echo "🤖 Starting mock fester-agent on :8787..."
+    MOCK_SCRIPT="$REPO_ROOT/scripts/mock_agent.py"
+    if [ ! -f "$MOCK_SCRIPT" ]; then
+        # Fall back to the global scripts dir
+        MOCK_SCRIPT="/home/z/my-project/scripts/mock_agent.py"
+    fi
+    if [ -f "$MOCK_SCRIPT" ]; then
+        python3 "$MOCK_SCRIPT" 8787 &
+        MOCK_PID=$!
+        echo "  ✓ Mock agent started (PID $MOCK_PID)"
+        sleep 1
+    else
+        echo "  ⚠ mock_agent.py not found — skipping"
+    fi
+fi
+
+# ------------------------------------------------------------
+# Print startup banner
+# ------------------------------------------------------------
+CONFIG_PATH="${FESTER_CONFIG:-config.yaml}"
+DB_PATH="${FESTER_DB_PATH:-$HOME/.fester/fester.db}"
+
+echo ""
+echo "🧠 Fester backend starting..."
+echo "   Config:  $CONFIG_PATH"
+echo "   DB:      $DB_PATH"
+echo "   URL:     http://$HOST:$PORT"
+echo "   Mode:    $([ "$DEV_MODE" = "1" ] && echo "development (auto-reload)" || echo "production")"
+echo "   Drift:   $([ "${FESTER_NO_DRIFT:-0}" = "1" ] && echo "disabled" || echo "enabled (fallback)")"
+echo "   Autobuild: $([ "${FESTER_AUTOBUILD:-0}" = "1" ] && echo "enabled" || echo "disabled")"
+echo ""
+echo "   Press Ctrl+C to stop"
+echo "   UI: http://localhost:$PORT"
+echo "   API docs: http://localhost:$PORT/docs"
+echo "   Health: http://localhost:$PORT/api/health"
+echo ""
+
+# ------------------------------------------------------------
+# Cleanup on exit
+# ------------------------------------------------------------
+cleanup() {
+    echo ""
+    echo "🛑 Shutting down..."
+    if [ -n "$MOCK_PID" ]; then
+        kill "$MOCK_PID" 2>/dev/null || true
+        echo "  ✓ Mock agent stopped"
+    fi
+    echo "  ✓ Backend stopped"
+}
+trap cleanup EXIT INT TERM
+
+# ------------------------------------------------------------
+# Start uvicorn
+# ------------------------------------------------------------
+if [ "$DEV_MODE" = "1" ]; then
+    exec python3 -m uvicorn backend.main:app \
+        --host "$HOST" \
+        --port "$PORT" \
+        --reload \
+        --log-level info
+else
+    exec python3 -m uvicorn backend.main:app \
+        --host "$HOST" \
+        --port "$PORT" \
+        --log-level info
+fi
diff --git a/style.css b/style.css
new file mode 100755
index 0000000..0bb89a2
--- /dev/null
+++ b/style.css
@@ -0,0 +1,424 @@
+/* ============================================================
+   Fester UI — Shared Design System
+   Used by: index.html, ui/live_dag.html, ui/replay.html,
+            cockpit/fester-module/ui.html
+   ============================================================ */
+
+/* ---------- Design Tokens ---------- */
+:root {
+    /* surfaces */
+    --bg-0: #07090d;
+    --bg-1: #0b0f14;
+    --bg-2: #0e141d;
+    --bg-3: #131c28;
+    --bg-4: #1a2432;
+
+    /* lines */
+    --line: #1f2a38;
+    --line-2: #2c3b4d;
+
+    /* text */
+    --fg-0: #f4f6fa;
+    --fg-1: #d0d6df;
+    --fg-2: #8a95a5;
+    --fg-3: #5a6573;
+
+    /* status */
+    --ok: #66bb6a;
+    --ok-bg: rgba(102, 187, 106, 0.12);
+    --warn: #ffb300;
+    --warn-bg: rgba(255, 179, 0, 0.12);
+    --err: #ef5350;
+    --err-bg: rgba(239, 83, 80, 0.12);
+    --info: #42a5f5;
+    --info-bg: rgba(66, 165, 245, 0.12);
+    --accent: #ffd54f;
+    --accent-bg: rgba(255, 213, 79, 0.12);
+
+    /* heat ramp */
+    --heat-0: #2ecc71;
+    --heat-1: #f1c40f;
+    --heat-2: #e67e22;
+    --heat-3: #e74c3c;
+
+    /* layout */
+    --radius: 6px;
+    --radius-lg: 10px;
+    --gap: 12px;
+    --pad: 12px;
+    --topbar-h: 48px;
+
+    /* type */
+    --font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif;
+    --font-mono: "JetBrains Mono", "SF Mono", "Cascadia Code", Menlo, Consolas, monospace;
+}
+
+/* ---------- Reset ---------- */
+*, *::before, *::after { box-sizing: border-box; }
+html, body { margin: 0; padding: 0; height: 100%; }
+body {
+    font-family: var(--font-sans);
+    font-size: 13px;
+    color: var(--fg-1);
+    background: var(--bg-1);
+    line-height: 1.45;
+    -webkit-font-smoothing: antialiased;
+}
+a { color: var(--info); text-decoration: none; }
+a:hover { text-decoration: underline; }
+pre, code, .mono { font-family: var(--font-mono); }
+button, input, select, textarea { font-family: inherit; font-size: inherit; color: inherit; }
+h1, h2, h3, h4 { margin: 0 0 8px; font-weight: 600; color: var(--fg-0); }
+h1 { font-size: 18px; }
+h2 { font-size: 15px; }
+h3 { font-size: 13px; text-transform: uppercase; letter-spacing: 0.6px; color: var(--fg-2); }
+hr { border: 0; border-top: 1px solid var(--line); margin: 12px 0; }
+
+/* ---------- Topbar / Shell ---------- */
+.topbar {
+    height: var(--topbar-h);
+    background: var(--bg-2);
+    border-bottom: 1px solid var(--line);
+    display: flex;
+    align-items: center;
+    padding: 0 16px;
+    gap: 16px;
+    position: sticky;
+    top: 0;
+    z-index: 50;
+}
+.topbar .brand {
+    font-weight: 700;
+    color: var(--fg-0);
+    letter-spacing: 0.5px;
+    display: flex;
+    align-items: center;
+    gap: 8px;
+}
+.topbar .brand .mark {
+    width: 22px; height: 22px;
+    border-radius: 4px;
+    background: linear-gradient(135deg, var(--info), var(--accent));
+    display: inline-block;
+}
+.topbar nav { display: flex; gap: 4px; flex: 1; }
+.topbar nav a {
+    color: var(--fg-2);
+    padding: 6px 12px;
+    border-radius: var(--radius);
+    font-size: 12px;
+    text-decoration: none;
+    transition: background 0.15s, color 0.15s;
+}
+.topbar nav a:hover { background: var(--bg-3); color: var(--fg-0); text-decoration: none; }
+.topbar nav a.active { background: var(--bg-4); color: var(--fg-0); }
+
+/* connection pill */
+.conn {
+    display: inline-flex;
+    align-items: center;
+    gap: 6px;
+    padding: 4px 10px;
+    border-radius: 999px;
+    background: var(--bg-3);
+    border: 1px solid var(--line-2);
+    font-size: 11px;
+    font-family: var(--font-mono);
+    color: var(--fg-2);
+}
+.conn .dot { width: 7px; height: 7px; border-radius: 50%; background: var(--fg-3); }
+.conn.connecting .dot { background: var(--warn); animation: pulse 1.2s infinite; }
+.conn.online .dot { background: var(--ok); box-shadow: 0 0 6px var(--ok); }
+.conn.offline .dot { background: var(--err); }
+@keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.35; } }
+
+/* ---------- Buttons ---------- */
+button, .btn {
+    display: inline-flex;
+    align-items: center;
+    justify-content: center;
+    gap: 6px;
+    padding: 6px 12px;
+    background: var(--bg-4);
+    color: var(--fg-1);
+    border: 1px solid var(--line-2);
+    border-radius: var(--radius);
+    cursor: pointer;
+    transition: background 0.12s, border-color 0.12s, transform 0.05s;
+    font-size: 12px;
+    white-space: nowrap;
+}
+button:hover, .btn:hover { background: var(--bg-3); border-color: #3a4a5e; }
+button:active, .btn:active { transform: translateY(1px); }
+button:disabled { opacity: 0.4; cursor: not-allowed; }
+button.primary { background: var(--info); color: #061018; border-color: var(--info); }
+button.primary:hover { filter: brightness(1.1); }
+button.danger { background: var(--err); color: #1a0606; border-color: var(--err); }
+button.ghost { background: transparent; }
+button.full { width: 100%; }
+
+/* ---------- Form inputs ---------- */
+input[type="text"], input[type="number"], input[type="search"],
+input[type="url"], input[type="password"], select, textarea {
+    width: 100%;
+    padding: 7px 10px;
+    background: var(--bg-0);
+    border: 1px solid var(--line-2);
+    border-radius: var(--radius);
+    color: var(--fg-0);
+    font-family: var(--font-mono);
+    font-size: 12px;
+    outline: none;
+    transition: border-color 0.12s;
+}
+input:focus, select:focus, textarea:focus { border-color: var(--info); }
+label { display: block; font-size: 11px; color: var(--fg-2); margin-bottom: 4px; text-transform: uppercase; letter-spacing: 0.4px; }
+.field { margin-bottom: 10px; }
+
+/* range slider */
+input[type="range"] {
+    width: 100%;
+    -webkit-appearance: none;
+    background: transparent;
+    height: 24px;
+}
+input[type="range"]::-webkit-slider-runnable-track {
+    height: 4px;
+    background: var(--bg-4);
+    border-radius: 2px;
+}
+input[type="range"]::-webkit-slider-thumb {
+    -webkit-appearance: none;
+    width: 14px; height: 14px;
+    border-radius: 50%;
+    background: var(--info);
+    margin-top: -5px;
+    cursor: pointer;
+}
+
+/* ---------- Panels / Cards ---------- */
+.panel {
+    background: var(--bg-2);
+    border: 1px solid var(--line);
+    border-radius: var(--radius-lg);
+    overflow: hidden;
+}
+.panel > .head {
+    padding: 10px 14px;
+    border-bottom: 1px solid var(--line);
+    display: flex;
+    align-items: center;
+    justify-content: space-between;
+    background: var(--bg-3);
+}
+.panel > .head h3 { margin: 0; }
+.panel > .body { padding: var(--pad); }
+
+/* ---------- Badges / Pills ---------- */
+.badge {
+    display: inline-flex;
+    align-items: center;
+    gap: 4px;
+    padding: 2px 8px;
+    border-radius: 999px;
+    font-size: 10px;
+    font-family: var(--font-mono);
+    text-transform: uppercase;
+    letter-spacing: 0.5px;
+    border: 1px solid var(--line-2);
+    background: var(--bg-3);
+    color: var(--fg-2);
+}
+.badge.ok { color: var(--ok); background: var(--ok-bg); border-color: rgba(102,187,106,0.3); }
+.badge.warn { color: var(--warn); background: var(--warn-bg); border-color: rgba(255,179,0,0.3); }
+.badge.err { color: var(--err); background: var(--err-bg); border-color: rgba(239,83,80,0.3); }
+.badge.info { color: var(--info); background: var(--info-bg); border-color: rgba(66,165,245,0.3); }
+.badge.accent { color: var(--accent); background: var(--accent-bg); border-color: rgba(255,213,79,0.3); }
+
+/* ---------- Tables ---------- */
+table { width: 100%; border-collapse: collapse; font-size: 12px; }
+th, td { text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--line); }
+th { color: var(--fg-2); font-weight: 500; font-size: 11px; text-transform: uppercase; letter-spacing: 0.5px; }
+tbody tr:hover { background: var(--bg-3); }
+
+/* ---------- Lists ---------- */
+.list { list-style: none; padding: 0; margin: 0; }
+.list > li { padding: 8px 10px; border-bottom: 1px solid var(--line); }
+.list > li:last-child { border-bottom: 0; }
+
+/* ---------- Event Stream ---------- */
+.stream {
+    font-family: var(--font-mono);
+    font-size: 11px;
+    height: 100%;
+    overflow-y: auto;
+    padding: 8px;
+    background: var(--bg-0);
+    border-radius: var(--radius);
+    border: 1px solid var(--line);
+}
+.stream .row {
+    padding: 3px 6px;
+    border-radius: 3px;
+    display: grid;
+    grid-template-columns: 70px 80px 1fr;
+    gap: 8px;
+    align-items: baseline;
+    border-bottom: 1px dashed rgba(255,255,255,0.04);
+}
+.stream .row:hover { background: var(--bg-2); }
+.stream .ts { color: var(--fg-3); }
+.stream .tag { font-weight: 600; }
+.stream .tag.node { color: var(--info); }
+.stream .tag.pipeline { color: var(--warn); }
+.stream .tag.failure { color: var(--err); }
+.stream .tag.cache { color: var(--accent); }
+.stream .tag.debug { color: var(--fg-2); }
+.stream .tag.policy { color: #ab47bd; }
+.stream .tag.session { color: var(--ok); }
+.stream .msg { color: var(--fg-1); word-break: break-word; }
+
+/* ---------- DAG Node (shared) ---------- */
+.dag-node {
+    position: absolute;
+    min-width: 120px;
+    padding: 6px 10px;
+    font-size: 11px;
+    text-align: center;
+    border-radius: var(--radius);
+    background: var(--bg-3);
+    border: 1px solid var(--line-2);
+    color: var(--fg-0);
+    cursor: pointer;
+    user-select: none;
+    transition: transform 0.15s, box-shadow 0.15s, border-color 0.15s;
+    z-index: 2;
+}
+.dag-node:hover { transform: translateY(-1px); box-shadow: 0 4px 12px rgba(0,0,0,0.4); }
+.dag-node .label { font-weight: 600; }
+.dag-node .meta { font-family: var(--font-mono); font-size: 10px; color: var(--fg-2); margin-top: 2px; }
+
+.dag-node.pending  { border-color: var(--fg-3); }
+.dag-node.running  { border-color: var(--warn); box-shadow: 0 0 12px rgba(255,179,0,0.35); }
+.dag-node.done     { border-color: var(--ok); }
+.dag-node.failed   { border-color: var(--err); background: var(--err-bg); }
+.dag-node.cache    { border-color: var(--info); }
+.dag-node.critical { outline: 2px solid var(--accent); outline-offset: 2px; }
+.dag-node.heat     { box-shadow: 0 0 0 2px var(--heat-2); }
+.dag-node.active   { outline: 2px solid var(--accent); outline-offset: 2px; }
+
+/* SVG edges */
+svg.edges {
+    position: absolute;
+    top: 0; left: 0;
+    width: 100%; height: 100%;
+    pointer-events: none;
+    z-index: 1;
+}
+svg.edges line, svg.edges path {
+    stroke: var(--line-2);
+    stroke-width: 1.5;
+    fill: none;
+}
+svg.edges line.critical, svg.edges path.critical {
+    stroke: var(--accent);
+    stroke-width: 2.5;
+}
+svg.edges marker path { fill: var(--line-2); stroke: none; }
+
+/* ---------- Inspector / JSON viewer ---------- */
+pre.json {
+    margin: 0;
+    padding: 10px;
+    background: var(--bg-0);
+    border-radius: var(--radius);
+    border: 1px solid var(--line);
+    font-family: var(--font-mono);
+    font-size: 11px;
+    color: var(--fg-1);
+    overflow: auto;
+    white-space: pre-wrap;
+    word-break: break-word;
+}
+
+/* ---------- Scrollbars ---------- */
+::-webkit-scrollbar { width: 8px; height: 8px; }
+::-webkit-scrollbar-track { background: var(--bg-0); }
+::-webkit-scrollbar-thumb { background: var(--line-2); border-radius: 4px; }
+::-webkit-scrollbar-thumb:hover { background: #3a4a5e; }
+
+/* ---------- Utility ---------- */
+.row { display: flex; gap: var(--gap); }
+.col { display: flex; flex-direction: column; gap: var(--gap); }
+.flex-1 { flex: 1; }
+.grow { flex-grow: 1; }
+.muted { color: var(--fg-2); }
+.dim { color: var(--fg-3); }
+.right { text-align: right; }
+.center { text-align: center; }
+.mono { font-family: var(--font-mono); }
+.nowrap { white-space: nowrap; }
+.hidden { display: none !important; }
+.spacer { flex: 1; }
+
+/* heat meter */
+.heat-meter {
+    height: 6px;
+    border-radius: 3px;
+    background: linear-gradient(90deg, var(--heat-0) 0%, var(--heat-1) 50%, var(--heat-2) 80%, var(--heat-3) 100%);
+    position: relative;
+}
+.heat-meter::after {
+    content: "";
+    position: absolute;
+    top: -2px; bottom: -2px;
+    left: var(--heat, 0%);
+    width: 2px;
+    background: var(--fg-0);
+}
+
+/* progress bar */
+.progress {
+    height: 8px;
+    background: var(--bg-0);
+    border-radius: 4px;
+    overflow: hidden;
+    border: 1px solid var(--line);
+}
+.progress > .bar {
+    height: 100%;
+    background: linear-gradient(90deg, var(--info), var(--ok));
+    transition: width 0.3s ease;
+}
+
+/* toasts */
+#toasts {
+    position: fixed;
+    bottom: 16px;
+    right: 16px;
+    z-index: 1000;
+    display: flex;
+    flex-direction: column;
+    gap: 8px;
+    max-width: 360px;
+}
+#toasts .toast {
+    background: var(--bg-3);
+    border: 1px solid var(--line-2);
+    border-left: 3px solid var(--info);
+    border-radius: var(--radius);
+    padding: 10px 14px;
+    font-size: 12px;
+    box-shadow: 0 8px 24px rgba(0,0,0,0.5);
+    animation: slide-in 0.2s ease;
+}
+#toasts .toast.ok { border-left-color: var(--ok); }
+#toasts .toast.err { border-left-color: var(--err); }
+#toasts .toast.warn { border-left-color: var(--warn); }
+@keyframes slide-in { from { transform: translateX(20px); opacity: 0; } to { transform: translateX(0); opacity: 1; } }
+
+/* responsive */
+@media (max-width: 900px) {
+    .topbar nav a { padding: 6px 8px; font-size: 11px; }
+    .topbar .brand .label { display: none; }
+}
diff --git a/ui/app.js b/ui/app.js
new file mode 100755
index 0000000..370fcde
--- /dev/null
+++ b/ui/app.js
@@ -0,0 +1,314 @@
+/* ============================================================
+   Fester UI — Shared application shell & utilities
+   Used by: index.html, ui/live_dag.html, ui/replay.html
+   Exposes a single global: window.Fester
+   ============================================================ */
+
+(function () {
+    'use strict';
+
+    const Fester = {
+        ws: null,
+        wsUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws',
+        wsDebuggerUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws-debugger',
+        wsTargetsUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws-targets',
+        wsState: 'connecting',
+        _subs: new Set(),
+        _reconnectTimer: null,
+        _reconnectDelay: 800,
+    };
+
+    /* ---------- Event subscription ----------
+       callback(event) — event is the parsed JSON object from /ws
+       returns an unsubscribe function */
+    Fester.subscribe = function (cb) {
+        Fester._subs.add(cb);
+        return () => Fester._subs.delete(cb);
+    };
+
+    function dispatch(event) {
+        Fester._subs.forEach(cb => {
+            try { cb(event); } catch (e) { console.error('subscriber error', e); }
+        });
+    }
+
+    /* ---------- WebSocket with auto-reconnect ---------- */
+    function setConnState(state) {
+        Fester.wsState = state;
+        document.querySelectorAll('[data-conn]').forEach(el => {
+            el.classList.remove('online', 'offline', 'connecting');
+            el.classList.add(state);
+            const txt = el.querySelector('[data-conn-text]');
+            if (txt) txt.textContent = state.toUpperCase();
+        });
+    }
+
+    function connect() {
+        setConnState('connecting');
+        try {
+            const ws = new WebSocket(Fester.wsUrl);
+            Fester.ws = ws;
+
+            ws.onopen = () => {
+                setConnState('online');
+                Fester._reconnectDelay = 800;
+                Fester.toast('Connected to Fester event stream', 'ok');
+            };
+
+            ws.onmessage = (msg) => {
+                try {
+                    const event = JSON.parse(msg.data);
+                    dispatch(event);
+                } catch (e) {
+                    console.error('bad ws payload', e, msg.data);
+                }
+            };
+
+            ws.onerror = (e) => {
+                console.warn('ws error', e);
+            };
+
+            ws.onclose = () => {
+                setConnState('offline');
+                if (!Fester._reconnectTimer) {
+                    Fester._reconnectTimer = setTimeout(() => {
+                        Fester._reconnectTimer = null;
+                        connect();
+                    }, Fester._reconnectDelay);
+                    Fester._reconnectDelay = Math.min(Fester._reconnectDelay * 1.6, 8000);
+                }
+            };
+        } catch (e) {
+            setConnState('offline');
+            console.error('ws connect failed', e);
+        }
+    }
+
+    Fester.connect = connect;
+
+    Fester.send = function (obj) {
+        if (Fester.ws && Fester.ws.readyState === WebSocket.OPEN) {
+            Fester.ws.send(JSON.stringify(obj));
+            return true;
+        }
+        return false;
+    };
+
+    /* ---------- REST helpers ---------- */
+    Fester.api = async function (path, opts = {}) {
+        const res = await fetch(path, {
+            headers: { 'Content-Type': 'application/json' },
+            ...opts,
+        });
+        if (!res.ok) throw new Error(`${res.status} ${res.statusText}`);
+        const ct = res.headers.get('content-type') || '';
+        return ct.includes('application/json') ? res.json() : res.text();
+    };
+
+    /* ---------- State → color helpers (mirrors backend fester.js logic) ---------- */
+    Fester.heatColor = function (heat) {
+        if (heat == null) return 'var(--fg-3)';
+        if (heat < 30) return 'var(--heat-0)';
+        if (heat < 60) return 'var(--heat-1)';
+        if (heat < 80) return 'var(--heat-2)';
+        return 'var(--heat-3)';
+    };
+
+    Fester.stateColor = function (state) {
+        switch ((state || '').toLowerCase()) {
+            case 'done': case 'ok': case 'success': return 'var(--ok)';
+            case 'running': case 'active': return 'var(--warn)';
+            case 'failed': case 'error': return 'var(--err)';
+            case 'cache': case 'hit': return 'var(--info)';
+            default: return 'var(--fg-3)';
+        }
+    };
+
+    Fester.stateBadge = function (state) {
+        const s = (state || 'pending').toLowerCase();
+        const map = {
+            done: 'ok', ok: 'ok', success: 'ok',
+            running: 'warn', active: 'warn',
+            failed: 'err', error: 'err',
+            cache: 'info', hit: 'info',
+            pending: '', queued: '',
+        };
+        const cls = map[s] || '';
+        return `${state || 'pending'}`;
+    };
+
+    /* ---------- Toasts ---------- */
+    Fester.toast = function (msg, kind = 'info', ms = 3200) {
+        let host = document.getElementById('toasts');
+        if (!host) {
+            host = document.createElement('div');
+            host.id = 'toasts';
+            document.body.appendChild(host);
+        }
+        const el = document.createElement('div');
+        el.className = `toast ${kind}`;
+        el.textContent = msg;
+        host.appendChild(el);
+        setTimeout(() => {
+            el.style.transition = 'opacity 0.3s';
+            el.style.opacity = '0';
+            setTimeout(() => el.remove(), 300);
+        }, ms);
+    };
+
+    /* ---------- Time formatting ---------- */
+    Fester.fmtTime = function (ts) {
+        if (!ts) return '—';
+        const d = new Date(ts * 1000 || ts);
+        const pad = n => String(n).padStart(2, '0');
+        return `${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}`;
+    };
+
+    Fester.fmtRel = function (ts) {
+        if (!ts) return '—';
+        const diff = Math.max(0, (Date.now() - (ts * 1000 || ts)) / 1000);
+        if (diff < 60) return Math.floor(diff) + 's ago';
+        if (diff < 3600) return Math.floor(diff / 60) + 'm ago';
+        return Math.floor(diff / 3600) + 'h ago';
+    };
+
+    /* ---------- Topbar shell ----------
+       Injects shared topbar so every page has consistent nav + WS status.
+       Call Fester.mountShell('dashboard') etc. */
+    Fester.mountShell = function (active) {
+        const pages = [
+            { id: 'dashboard', href: '/index.html',          label: 'Dashboard' },
+            { id: 'live',      href: '/ui/live_dag.html',    label: 'Live DAG' },
+            { id: 'replay',    href: '/ui/replay.html',      label: 'Replay' },
+            { id: 'sessions',  href: '/ui/sessions.html',    label: 'Sessions' },
+            { id: 'metrics',   href: '/ui/metrics.html',     label: 'Metrics' },
+            { id: 'cause',     href: '/ui/cause.html',       label: 'Cause Graph' },
+            { id: 'timeline',  href: '/ui/timeline.html',    label: 'Timeline' },
+            { id: 'debugger',  href: '/ui/debugger.html',    label: 'Debugger' },
+        ];
+        const links = pages.map(p =>
+            `${p.label}`
+        ).join('');
+
+        const shell = document.createElement('div');
+        shell.className = 'topbar';
+        shell.innerHTML = `
+            

+                
+                FESTER
+            

+            
+            

+                
+                CONNECTING
+            

+            

+                —
+            

+        `;
+        document.body.prepend(shell);
+
+        // Start health polling
+        Fester._startHealthPolling();
+    };
+
+    /* ---------- Health polling ---------- */
+    Fester._healthTimer = null;
+    Fester._startHealthPolling = function () {
+        if (Fester._healthTimer) return;
+        const tick = async () => {
+            try {
+                const h = await Fester.api('/api/health');
+                const el = document.querySelector('[data-health]');
+                if (el) {
+                    const subs = h.bus_subscribers || 0;
+                    const ws = h.ws_clients || 0;
+                    const builds = h.builds_known || 0;
+                    const events = h.timeline_events || 0;
+                    el.innerHTML = `
+                        ●
+                        v${h.version}
+                        ·
+                        ${ws}ws
+                        ·
+                        ${builds}b
+                        ·
+                        ${events}e
+                    `;
+                    el.style.color = 'var(--fg-1)';
+                }
+            } catch (e) {
+                const el = document.querySelector('[data-health]');
+                if (el) {
+                    el.innerHTML = `●unreachable`;
+                }
+            }
+        };
+        tick();
+        Fester._healthTimer = setInterval(tick, 5000);
+    };
+
+    /* ---------- FesterTargets global ----------
+       The missing API referenced by live_dag.html.
+       Maintains an in-memory snapshot of nodes + edges,
+       notifies subscribers on update. */
+    const snapshot = { nodes: [], edges: [], byId: {} };
+    const targetSubs = new Set();
+
+    Fester.targets = {
+        get() { return snapshot; },
+
+        onUpdate(cb) {
+            targetSubs.add(cb);
+            // immediately emit current snapshot
+            try { cb(snapshot); } catch (e) { console.error(e); }
+            return () => targetSubs.delete(cb);
+        },
+
+        update(patch) {
+            if (patch.nodes) {
+                patch.nodes.forEach(n => {
+                    const existing = snapshot.byId[n.id];
+                    snapshot.byId[n.id] = existing ? { ...existing, ...n } : n;
+                });
+                snapshot.nodes = Object.values(snapshot.byId);
+            }
+            if (patch.edges) {
+                // merge by from+to
+                const seen = new Set(snapshot.edges.map(e => e.from + '→' + e.to));
+                patch.edges.forEach(e => {
+                    const k = e.from + '→' + e.to;
+                    if (!seen.has(k)) { snapshot.edges.push(e); seen.add(k); }
+                });
+            }
+            targetSubs.forEach(cb => {
+                try { cb(snapshot); } catch (e) { console.error(e); }
+            });
+        },
+
+        reset() {
+            snapshot.nodes = [];
+            snapshot.edges = [];
+            snapshot.byId = {};
+            targetSubs.forEach(cb => cb(snapshot));
+        },
+    };
+
+    // legacy alias used by live_dag.html
+    window.FesterTargets = Fester.targets;
+
+    /* ---------- Auto-init on DOMContentLoaded ---------- */
+    document.addEventListener('DOMContentLoaded', () => {
+        // mount shell if a marker exists
+        const marker = document.querySelector('[data-fester-shell]');
+        if (marker) {
+            Fester.mountShell(marker.getAttribute('data-fester-shell'));
+        }
+        // auto-connect WS unless page opts out
+        if (!document.body.hasAttribute('data-no-ws')) {
+            connect();
+        }
+    });
+
+    window.Fester = Fester;
+})();
diff --git a/ui/cause.html b/ui/cause.html
new file mode 100755
index 0000000..073701a
--- /dev/null
+++ b/ui/cause.html
@@ -0,0 +1,292 @@
+
+
+
+
+
+Fester — Cause Graph
+
+
+
+
+
+

+
+    
+    

+        
Lookup

+        

+            

+                
+            

+            

+                
+                
+            

+
+            

+
+            
Blast Radius

+            

+                
+                
+            

+            

+
+            

+
+            
Causal Chain

+            
    
+                
  • Enter a node name and click Explain
    • 
+            

+
+            

+
+            
Recent Events

+            
+            
—

+        

+    

+
+    
+    

+        

+            
Cause Graph

+            idle
+        

+        

+            

+                
+            

+        

+    

+
+

+
+
+
+
+
diff --git a/ui/debugger.html b/ui/debugger.html
new file mode 100755
index 0000000..06376ee
--- /dev/null
+++ b/ui/debugger.html
@@ -0,0 +1,244 @@
+
+
+
+
+
+Fester — Debugger
+
+
+
+
+
+

+
+    
+    

+        

+            
Debugger

+            idle
+        

+        

+            

+                
+                
+                
+                
+            

+
+            

+
+            
State

+            
—

+
+            

+
+            
Next Action Preview

+            
—

+        

+    

+
+    
+    

+        

+            
Execution Timeline

+            step 0
+        

+        

+            

+        

+    

+
+    
+    

+        
Inspector

+        

+            
Click a timeline row to inspect.

+        

+    

+
+

+
+
+
+
+
diff --git a/ui/live_dag.html b/ui/live_dag.html
new file mode 100755
index 0000000..7682881
--- /dev/null
+++ b/ui/live_dag.html
@@ -0,0 +1,403 @@
+
+
+
+
+
+Fester — Live DAG
+
+
+
+
+
+

+    

+    
+        
+            
+                
+            
+        
+    
+
+    

+        
+        
+        
+        
+        
+        
+    

+
+    
+
+    

+        
 Pending

+        
 Running

+        
 Done

+        
 Failed

+        
 Cache hit

+        
 Critical path

+    

+

+
+
+
+
+
diff --git a/ui/metrics.html b/ui/metrics.html
new file mode 100755
index 0000000..8b816d3
--- /dev/null
+++ b/ui/metrics.html
@@ -0,0 +1,257 @@
+
+
+
+
+
+Fester — Metrics
+
+
+
+
+
+

+    

+        

+            
Nodes Online

+            
—

+            
—

+        

+        

+            
Avg Heat

+            
—

+            
—

+        

+        

+            
Active Jobs

+            
—

+            
—

+        

+        

+            
Instability

+            
—

+            
—

+        

+    

+
+    

+        

+            
Heat Trend (last 60s)

+            
+        

+        

+            
Active Jobs Trend (last 60s)

+            
+        

+    

+
+    

+        

+            
Per-Node Breakdown

+            
+        

+        

+            

+        

+    

+

+
+
+
+
+
diff --git a/ui/replay.html b/ui/replay.html
new file mode 100755
index 0000000..dcaaf3c
--- /dev/null
+++ b/ui/replay.html
@@ -0,0 +1,573 @@
+
+
+
+
+
+Fester — Replay Observatory
+
+
+
+
+
+

+
+    
+    

+        
Replay Controls

+        

+            

+                
+                
+            

+            
+
+            

+
+            

+                
+                
+                
+                
+                
+                
+            

+
+            
+            

+                

+            

+            
+            

+                0 / 0
+                —
+            

+
+            
+            

+                
+                
+                
+                
+            

+
+            

+            
+            
+        

+    

+
+    
+    

+        

+            
Execution Graph

+            

+                idle
+            

+        

+        

+            

+                
+                    
+                        
+                            
+                        
+                    
+                
+            

+        

+    

+
+    
+    

+        
Inspector

+        

+            
Click a node to inspect.

+        

+    

+
+

+
+
+

+    Failure Autopsy:
+    
+    
+    
+    Traces dependency chain backward + last scheduler decision
+

+

+    
—

+

+
+
+
+
+
diff --git a/ui/sessions.html b/ui/sessions.html
new file mode 100755
index 0000000..74fae08
--- /dev/null
+++ b/ui/sessions.html
@@ -0,0 +1,347 @@
+
+
+
+
+
+Fester — Sessions
+
+
+
+
+
+

+    

+        
Sessions

+        
+        
+    

+
+    

+        

+            
Replay Sessions

+            —
+        

+        

+            
+                
+                    
+                        
+                        
+                        
+                        
+                        
+                    
+                
+                
+                    
+                
+            
Session IDSourceEventsCreated
Loading…

+        

+    

+
+    

+        
Recent Builds

+        

+            
+                
+                    
+                        
+                        
+                        
+                        
+                        
+                        
+                        
+                    
+                
+                
+                    
+                
+            
Build IDCommandDirStatusStartedDurationActions
Loading…

+        

+    

+
+    

+        

+            
Active Action Sessions (tmux)

+            
+        

+        

+            
+                
+                    
+                        
+                        
+                        
+                        
+                        
+                    
+                
+                
+                    
+                
+            
SessionStartedCurrent CmdAttachedActions
Loading…

+        

+    

+
+    

+        

+            
Storage Snapshots (qcow2)

+            
+        

+        

+            
+                
+                    
+                        
+                        
+                        
+                        
+                    
+                
+                
+                    
+                
+            
NameSizeCreatedVirtual Size
Loading…

+        

+    

+

+
+
+
+
+
+
+
+
diff --git a/ui/timeline.html b/ui/timeline.html
new file mode 100755
index 0000000..61423f0
--- /dev/null
+++ b/ui/timeline.html
@@ -0,0 +1,194 @@
+
+
+
+
+
+Fester — Timeline
+
+
+
+
+
+

+
+    
+    

+        

+            
Nodes

+            
+        

+        

+            

+                
Loading…

+            

+        

+    

+
+    
+    

+        

+            
Events for —

+            

+                0
+                
+            

+        

+        

+            

+                
Select a node

+            

+        

+    

+
+

+
+
+
+
+