# Fester A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging. [![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE) [![Python: 3.12+](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://www.python.org/) [![FastAPI](https://img.shields.io/badge/FastAPI-0.128+-green.svg)](https://fastapi.tiangolo.com/) --- ## Screenshots | Dashboard | Live DAG | Replay | |-----------|----------|--------| | ![Dashboard](docs/screenshots/01_dashboard.png) | ![Live DAG](docs/screenshots/02_live_dag.png) | ![Replay](docs/screenshots/03_replay.png) | | Sessions | Metrics | Cause Graph | |----------|---------|-------------| | ![Sessions](docs/screenshots/04_sessions.png) | ![Metrics](docs/screenshots/05_metrics.png) | ![Cause](docs/screenshots/06_cause.png) | | Timeline | Debugger | |----------|----------| | ![Timeline](docs/screenshots/07_timeline.png) | ![Debugger](docs/screenshots/08_debugger.png) | --- ## What Fester Does Fester turns a cluster of machines into a single, observable build brain: - **DAG-driven execution** — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently - **Smart scheduler** — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability - **Real-time observability** — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket - **Deterministic replay** — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why - **Failure autopsy** — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius - **Cause graph** — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?" - **Cache layer** — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots + shared CAS API --- ## Architecture ``` +------------------------------------------------------------------+ | Fester Cluster | +------------------------------------------------------------------+ | | | +-------------+ +-------------+ +---------------------+ | | | Backend |<-->| EventBus |<-->| PipelineEngine | | | | (FastAPI) | | (singleton)| | (DAG executor) | | | +------+------+ +------+------+ +----------+----------+ | | | | | | | | | +-------------------+----------+ | | | | | | | | | | v v v | | | +------+------+ +-------------+ +-----------------+ | | | | WebSocket | | Timeline | | Scheduler | | | | | Stream | | Store | | (weighted/thermal| | | | | (/ws, etc) | | (SQLite) | | /cache-aware) | | | | +------+------+ +------+------+ +--------+--------+ | | | | | | | | | | | +-------+--------+ | | | | | | Node Registry | | | | | | | (probe + drift) | | | | v v +----------------+ | | | +------------------------------------------------------+ | | | | UI (8 pages) | | | | | Dashboard | Live DAG | Replay | Sessions | Metrics | | | | | Cause Graph | Timeline | Debugger | | | | +------------------------------------------------------+ | | | | | +------------+ +------------+ +------------+ +--------+ | | | MinIO Cache| | Btrfs CAS | | QCOW2 Snap | | tmux | | | | (optional) | | (optional) | | (optional) | | runtime| | | +------------+ +------------+ +------------+ +--------+ | | | +------------------------------------------------------------------+ ^ | HTTP :8787/status (probe) | +---------------+--------------------------------+ | Cluster Nodes | | +--------+ +--------+ +--------+ +--------+ | | | x99-v3 | | x99-v4 | | rpi-1 | | ryzen-1| | | |(x86_64)| |(x86_64)| |(arm64) | |(x86_64)| | | +--------+ +--------+ +--------+ +--------+ | +------------------------------------------------+ ``` ## Project Layout ``` fester/ +-- backend/ # Python backend (FastAPI) | +-- main.py # App entrypoint + 71 routes | +-- api/ # REST routers + WebSocket hub | +-- pipeline/ # Build execution (DAG engine + runner) | +-- scheduler/ # Node selection (weighted scoring) | +-- toolchain/ # BTC.sh cross-compilation integration | | +-- btc.py # Probe, build env, stamp verification | | +-- chroot.py # Chroot-based build environment | +-- analysis/ # Cause graph, timeline, failure autopsy | +-- events/ # Event infrastructure (bus + schema) | +-- nodes/ # Node management (probe, roles, state) | +-- storage/ # SQLite, Btrfs CAS, QCOW2, tmpfs | +-- integrations/ # tmux, mosh, LXC, libvirt, Forgejo, Podman, Firecracker | +-- executor/ # Runtime router (host/lxc/libvirt/tmux/podman/firecracker) | +-- targets/ # Build target catalog (Gentoo, Buildroot, OpenWrt, etc.) | +-- metrics/ # Prometheus exporter + observability hub | +-- policy/ # Policy engine (rules + overrides) +-- ui/ # Frontend (vanilla JS, no build step, ~12KB) +-- cli/ # CLI tool (35+ subcommands) +-- cockpit/ # Cockpit module (optional) +-- docs/ # Documentation + screenshots +-- pyproject.toml # Python packaging +-- install.sh # Unified installer (deps / config / db / deploy / run) +-- run.sh # Start the backend (also via `./install.sh run`) +-- config.yaml # Cluster configuration ``` ## Quick Start ```bash git clone https://git.dcos.net/dcosnet/fester.git cd fester ./install.sh # = ./install.sh all (venv + deps + config + db) ./install.sh run # start backend on :8181 # Open http://localhost:8181 ``` See [quickstart.md](quickstart.md) for the full walkthrough. ## Prerequisites - **Python 3.12+** - **pip** (for dependency installation) Optional (features activate automatically when installed): | Feature | Requires | |-------------------|-----------------------------| | Live action output| `tmux` | | Remote shell | `mosh` + `ssh` | | Workspace snapshots| `qemu-utils` + `rsync` | | CoW snapshots | `btrfs-progs` (btrfs fs) | | Distributed cache | `minio` server | ## Tech Stack | Layer | Technology | |------------|---------------------------------------------------------------| | Backend | Python 3.12, FastAPI 0.128, uvicorn, pydantic 2 | | Realtime | WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`)| | Storage | SQLite (WAL) for state; MinIO for distributed cache | | Frontend | Vanilla HTML/CSS/JS (no build step, no framework) | | Observability| Prometheus exposition + Grafana dashboard configs | | CLI | Python argparse (35+ subcommands) | | Integrations| tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache, Podman, Firecracker | ## BTC.sh Cross-Compilation Integration Fester integrates with BTC.sh for multi-architecture cross-compilation. The `backend/toolchain/btc.py` module probes for BTC golden images, parses their manifest sidecars, and configures the build environment for 19 supported targets: | Family | Targets | ISA Tiers | |------------------|----------------------------------------------------------------|------------------| | Intel HEDT/Server| haswell, haswell-ep, skylake, skylake-x, skylake-server | AVX2, AVX512 | | AMD Ryzen/EPYC | znver1, znver2, znver3, znver4 | AVX2, AVX512 | | AMD APU | apu-zn1, apu-zn2, apu-zn3, apu-zn4 | AVX2 | | Intel Atom | atom-silvermont, atom-goldmont, atom-tremont, atom-sierraforest| SSE4_2 | | Embedded | mipselr2, armv7, tilegx | MIPS32, NEON, TILE| Per-node BTC target selection is configured in `config.yaml` under `nodes[].btc.target`. When a BTC toolchain is active, Fester sets `CC`, `CXX`, `CFLAGS`, `LDFLAGS`, and cross-compiler environment variables automatically. ## Configuration Fester reads from `config.yaml` (or `FESTER_CONFIG` env var) at startup: ```yaml master: name: fester-master role: control nodes: - name: x99-v3 host: 192.168.1.10 max_jobs: 24 btc: target: znver3 # Use BTC.sh znver3 cross-toolchain on this node projects: - name: linux-tool repo: https://forgejo.local/linux-tool.git targets: debian: "make clean && make debian" arch: "make clean && make arch" ``` ### Environment Variables | Variable | Default | Purpose | |-----------------------|----------------------------------|---------------------------------| | `FESTER_CONFIG` | `config.yaml` | Path to YAML config | | `FESTER_DB_PATH` | `/var/lib/fester/fester.db` | SQLite database location | | `FESTER_NO_DRIFT` | unset | Disable synthetic drift | | `FESTER_AUTOBUILD` | unset | Auto-trigger builds every 60s | | `FESTER_ROLE_DB` | `/etc/fester/node_roles.json` | Node role overrides | | `FESTER_CACHE_DIR` | `/var/lib/fester/cache` | Local cache directory | | `FESTER_STORAGE_CONFIG`| `/etc/fester/storage.json` | Storage layer config | | `FESTER_API` | `http://localhost:8181` | CLI: backend URL | | `FESTER_WS` | `ws://localhost:8181/ws` | CLI: WebSocket URL | ## API Surface 71 routes total. Key endpoints: | Method | Path | Purpose | |--------|---------------------------------|----------------------------------------| | `POST` | `/api/build` | Kick off a build | | `GET` | `/api/builds` | Build history | | `POST` | `/api/builds/{id}/cancel` | Cancel a running build | | `GET` | `/api/nodes` | List cluster nodes (with live metrics) | | `POST` | `/api/nodes/{name}/policy` | Set node policy | | `POST` | `/api/nodes/{name}/probe` | Manually probe a node | | `GET` | `/api/metrics/json` | JSON metrics snapshot | | `GET` | `/metrics` | Prometheus exposition | | `GET` | `/api/cause/explain/{node}` | Causal chain for a node | | `GET` | `/api/propagation/{action}` | Blast radius if action fails | | `POST` | `/replay/start` | Start a replay session | | `GET` | `/autopsy/{sid}/{action}` | Failure autopsy | | `PUT` | `/api/cas/{sha256}` | Store artifact in CAS | | `GET` | `/api/cas/{sha256}` | Retrieve artifact from CAS | | `GET` | `/api/cas/stats` | Cache statistics | | `WS` | `/ws` | Main event stream | | `WS` | `/ws-debugger` | Debugger control channel | | `WS` | `/ws-targets` | Target toggle notifications | Full route list: `curl http://localhost:8181/openapi.json | jq '.paths | keys[]'` ## CLI ```bash fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch fester builds fester node list fester node set-policy x99-v3 preferred fester node probe x99-v3 fester cause explain x99-v3 build_kernel fester blast build_kernel fester stream fester health ``` Run `fester --help` for the full list of 35+ subcommands. ## Documentation - [quickstart.md](quickstart.md) — 5-minute getting started guide - [CHEATSHEET.md](CHEATSHEET.md) — Operator survival guide - [CONTRIBUTING.md](CONTRIBUTING.md) — How to contribute - [CHANGELOG.md](CHANGELOG.md) — Version history ## Docker ```bash docker build -t fester . docker run -p 8181:8181 -v fester-data:/var/lib/fester fester ``` ## Security Fester runs behind your existing network controls (OPNsense, IPFire, or any firewall appliance). There is no built-in authentication — assume the network is trusted. For public exposure, place it behind a reverse proxy with authentication (nginx + OAuth2 Proxy, Traefik + Authelia, etc.). ## License GNU Affero General Public License v3.0 — see [LICENSE](LICENSE). Software is provided "as is", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.