Fester

A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.

---

Screenshots

Dashboard	Live DAG	Replay

Sessions	Metrics	Cause Graph

Timeline	Debugger

---

What Fester Does

Fester turns a cluster of machines into a single, observable build brain:

• DAG-driven execution — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
• Smart scheduler — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
• Real-time observability — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
• Deterministic replay — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
• Failure autopsy — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
• Cause graph — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
• Cache layer — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots + shared CAS API

---

Architecture

+------------------------------------------------------------------+
|                         Fester Cluster                            |
+------------------------------------------------------------------+
|                                                                   |
|  +-------------+    +-------------+    +---------------------+   |
|  |   Backend   |<-->|  EventBus   |<-->|   PipelineEngine    |   |
|  |  (FastAPI)  |    |  (singleton)|    |  (DAG executor)     |   |
|  +------+------+    +------+------+    +----------+----------+   |
|         |                  |                      |              |
|         |                  |  +-------------------+----------+   |
|         |                  |  |                   |          |   |
|         |                  v  v                   v          |   |
|  +------+------+    +-------------+    +-----------------+  |   |
|  |  WebSocket  |    |  Timeline   |    |   Scheduler     |  |   |
|  |   Stream    |    |   Store     |    | (weighted/thermal|  |   |
|  | (/ws, etc)  |    |  (SQLite)   |    |   /cache-aware) |  |   |
|  +------+------+    +------+------+    +--------+--------+  |   |
|         |                  |                    |           |   |
|         |                  |            +-------+--------+   |   |
|         |                  |            | Node Registry  |   |   |
|         |                  |            | (probe + drift) |   |   |
|         v                  v            +----------------+   |   |
|  +------------------------------------------------------+  |   |
|  |                    UI (8 pages)                       |  |   |
|  |  Dashboard | Live DAG | Replay | Sessions | Metrics   |  |   |
|  |  Cause Graph | Timeline | Debugger                  |  |   |
|  +------------------------------------------------------+  |   |
|                                                              |
|  +------------+  +------------+  +------------+  +--------+  |
|  | MinIO Cache|  | Btrfs CAS  |  | QCOW2 Snap |  |  tmux  |  |
|  | (optional) |  | (optional) |  | (optional) |  | runtime|  |
|  +------------+  +------------+  +------------+  +--------+  |
|                                                              |
+------------------------------------------------------------------+

                              ^
                              | HTTP :8787/status (probe)
                              |
              +---------------+--------------------------------+
              |              Cluster Nodes                      |
              |  +--------+  +--------+  +--------+  +--------+ |
              |  | x99-v3 |  | x99-v4 |  | rpi-1  |  | ryzen-1| |
              |  |(x86_64)|  |(x86_64)|  |(arm64) |  |(x86_64)| |
              |  +--------+  +--------+  +--------+  +--------+ |
              +------------------------------------------------+

Project Layout

fester/
+-- backend/                    # Python backend (FastAPI)
|   +-- main.py                 # App entrypoint + 71 routes
|   +-- api/                    # REST routers + WebSocket hub
|   +-- pipeline/               # Build execution (DAG engine + runner)
|   +-- scheduler/              # Node selection (weighted scoring)
|   +-- toolchain/              # BTC.sh cross-compilation integration
|   |   +-- btc.py              #   Probe, build env, stamp verification
|   |   +-- chroot.py           #   Chroot-based build environment
|   +-- analysis/               # Cause graph, timeline, failure autopsy
|   +-- events/                 # Event infrastructure (bus + schema)
|   +-- nodes/                  # Node management (probe, roles, state)
|   +-- storage/                # SQLite, Btrfs CAS, QCOW2, tmpfs
|   +-- integrations/           # tmux, mosh, LXC, libvirt, Forgejo, Podman, Firecracker
|   +-- executor/               # Runtime router (host/lxc/libvirt/tmux/podman/firecracker)
|   +-- targets/                # Build target catalog (Gentoo, Buildroot, OpenWrt, etc.)
|   +-- metrics/                # Prometheus exporter + observability hub
|   +-- policy/                 # Policy engine (rules + overrides)
+-- ui/                         # Frontend (vanilla JS, no build step, ~12KB)
+-- cli/                        # CLI tool (35+ subcommands)
+-- cockpit/                    # Cockpit module (optional)
+-- docs/                       # Documentation + screenshots
+-- pyproject.toml              # Python packaging
+-- install.sh                  # Unified installer (deps / config / db / deploy / run)
+-- run.sh                      # Start the backend (also via `./install.sh run`)
+-- config.yaml                 # Cluster configuration

Quick Start

git clone https://git.dcos.net/dcosnet/fester.git
cd fester
./install.sh            # = ./install.sh all  (venv + deps + config + db)
./install.sh run        # start backend on :8181
# Open http://localhost:8181

See quickstart.md for the full walkthrough.

Prerequisites

• Python 3.12+
• pip (for dependency installation)

Optional (features activate automatically when installed):

Feature	Requires
Live action output	tmux
Remote shell	mosh + ssh
Workspace snapshots	qemu-utils + rsync
CoW snapshots	btrfs-progs (btrfs fs)
Distributed cache	minio server

Tech Stack

Layer	Technology
Backend	Python 3.12, FastAPI 0.128, uvicorn, pydantic 2
Realtime	WebSockets (3 channels: /ws, /ws-targets, /ws-debugger)
Storage	SQLite (WAL) for state; MinIO for distributed cache
Frontend	Vanilla HTML/CSS/JS (no build step, no framework)
Observability	Prometheus exposition + Grafana dashboard configs
CLI	Python argparse (35+ subcommands)
Integrations	tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache, Podman, Firecracker

BTC.sh Cross-Compilation Integration

Fester integrates with BTC.sh for multi-architecture cross-compilation. The backend/toolchain/btc.py module probes for BTC golden images, parses their manifest sidecars, and configures the build environment for 19 supported targets:

Family	Targets	ISA Tiers
Intel HEDT/Server	haswell, haswell-ep, skylake, skylake-x, skylake-server	AVX2, AVX512
AMD Ryzen/EPYC	znver1, znver2, znver3, znver4	AVX2, AVX512
AMD APU	apu-zn1, apu-zn2, apu-zn3, apu-zn4	AVX2
Intel Atom	atom-silvermont, atom-goldmont, atom-tremont, atom-sierraforest	SSE4_2
Embedded	mipselr2, armv7, tilegx	MIPS32, NEON, TILE

Per-node BTC target selection is configured in config.yaml under nodes[].btc.target. When a BTC toolchain is active, Fester sets CC, CXX, CFLAGS, LDFLAGS, and cross-compiler environment variables automatically.

Configuration

Fester reads from config.yaml (or FESTER_CONFIG env var) at startup:

master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
    btc:
      target: znver3           # Use BTC.sh znver3 cross-toolchain on this node

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"

Environment Variables

Variable	Default	Purpose
FESTER_CONFIG	config.yaml	Path to YAML config
FESTER_DB_PATH	/var/lib/fester/fester.db	SQLite database location
FESTER_NO_DRIFT	unset	Disable synthetic drift
FESTER_AUTOBUILD	unset	Auto-trigger builds every 60s
FESTER_ROLE_DB	/etc/fester/node_roles.json	Node role overrides
FESTER_CACHE_DIR	/var/lib/fester/cache	Local cache directory
FESTER_STORAGE_CONFIG	/etc/fester/storage.json	Storage layer config
FESTER_API	http://localhost:8181	CLI: backend URL
FESTER_WS	ws://localhost:8181/ws	CLI: WebSocket URL

API Surface

71 routes total. Key endpoints:

Method	Path	Purpose
POST	/api/build	Kick off a build
GET	/api/builds	Build history
POST	/api/builds/{id}/cancel	Cancel a running build
GET	/api/nodes	List cluster nodes (with live metrics)
POST	/api/nodes/{name}/policy	Set node policy
POST	/api/nodes/{name}/probe	Manually probe a node
GET	/api/metrics/json	JSON metrics snapshot
GET	/metrics	Prometheus exposition
GET	/api/cause/explain/{node}	Causal chain for a node
GET	/api/propagation/{action}	Blast radius if action fails
POST	/replay/start	Start a replay session
GET	/autopsy/{sid}/{action}	Failure autopsy
PUT	/api/cas/{sha256}	Store artifact in CAS
GET	/api/cas/{sha256}	Retrieve artifact from CAS
GET	/api/cas/stats	Cache statistics
WS	/ws	Main event stream
WS	/ws-debugger	Debugger control channel
WS	/ws-targets	Target toggle notifications

Full route list: curl http://localhost:8181/openapi.json | jq '.paths | keys[]'

CLI

fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch
fester builds
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3
fester cause explain x99-v3 build_kernel
fester blast build_kernel
fester stream
fester health

Run fester --help for the full list of 35+ subcommands.

Documentation

• quickstart.md — 5-minute getting started guide
• CHEATSHEET.md — Operator survival guide
• CONTRIBUTING.md — How to contribute
• CHANGELOG.md — Version history

Docker

docker build -t fester .
docker run -p 8181:8181 -v fester-data:/var/lib/fester fester

Security

Fester runs behind your existing network controls (OPNsense, IPFire, or any firewall appliance). There is no built-in authentication — assume the network is trusted. For public exposure, place it behind a reverse proxy with authentication (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).

License

GNU Affero General Public License v3.0 — see LICENSE.

Software is provided "as is", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.