🧠 Fester

A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.

---

📸 Screenshots

Dashboard	Live DAG	Replay

Sessions	Metrics	Cause Graph

Timeline	Debugger

---

🎯 What Fester Does

Fester turns a cluster of machines into a single, observable build brain:

• DAG-driven execution — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
• Smart scheduler — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
• Real-time observability — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
• Deterministic replay — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
• Failure autopsy — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
• Cause graph — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
• Cache layer — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                         Fester Cluster                            │
├──────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────┐   │
│  │   Backend   │◄──►│  EventBus   │◄──►│   PipelineEngine    │   │
│  │  (FastAPI)  │    │  (singleton)│    │  (DAG executor)     │   │
│  └──────┬──────┘    └──────┬──────┘    └──────────┬──────────┘   │
│         │                  │                      │              │
│         │                  │  ┌───────────────────┼──────────┐   │
│         │                  │  │                   │          │   │
│         │                  ▼  ▼                   ▼          │   │
│  ┌──────┴──────┐    ┌─────────────┐    ┌─────────────────┐   │   │
│  │  WebSocket  │    │  Timeline   │    │   Scheduler     │   │   │
│  │   Stream    │    │   Store     │    │ (weighted/thermal│   │   │
│  │ (/ws, etc)  │    │  (SQLite)   │    │   /cache-aware) │   │   │
│  └──────┬──────┘    └──────┬──────┘    └────────┬────────┘   │   │
│         │                  │                    │            │   │
│         │                  │            ┌───────┴────────┐   │   │
│         │                  │            │ Node Registry  │   │   │
│         │                  │            │ (probe + drift)│   │   │
│         ▼                  ▼            └────────────────┘   │   │
│  ┌──────────────────────────────────────────────────────┐   │   │
│  │                    UI (8 pages)                       │   │   │
│  │  Dashboard │ Live DAG │ Replay │ Sessions │ Metrics   │   │   │
│  │  Cause Graph │ Timeline │ Debugger                  │   │   │
│  └──────────────────────────────────────────────────────┘   │   │
│                                                              │   │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────┐│   │
│  │ MinIO Cache│  │ Btrfs CAS │  │ QCOW2 Snap │  │  tmux  ││   │
│  │ (optional) │  │ (optional)│  │ (optional) │  │ runtime││   │
│  └────────────┘  └────────────┘  └────────────┘  └────────┘│   │
│                                                              │   │
└──────────────────────────────────────────────────────────────┘   │
                                                                    │
                              ▲                                     │
                              │ HTTP :8787/status (probe)           │
                              │                                     │
              ┌───────────────┴────────────────────────────────┐   │
              │              Cluster Nodes                      │   │
              │  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐│   │
              │  │ x99-v3 │  │ x99-v4 │  │ rpi-1  │  │ ryzen-1││   │
              │  │(x86_64)│  │(x86_64)│  │(arm64) │  │(x86_64)││   │
              │  └────────┘  └────────┘  └────────┘  └────────┘│   │
              └────────────────────────────────────────────────┘   │

📁 Project Layout

fester/
├── backend/                    # Python backend (FastAPI)
│   ├── main.py                 # App entrypoint + 71 routes
│   ├── api/                    # REST routers + WebSocket hub
│   │   ├── build.py            #   /api/build
│   │   ├── nodes.py            #   /api/nodes (list, policy, probe)
│   │   ├── replay.py           #   /replay/start, /events
│   │   ├── autopsy.py          #   /autopsy/{sid}/{action}
│   │   ├── timeline.py         #   /timeline/{node}, /rewind/{i}
│   │   ├── debugger.py         #   /debugger/{pause,resume,step}
│   │   ├── pipeline_control.py #   /api/pipeline/{state,retry,...}
│   │   ├── metrics.py          #   /metrics (Prometheus) + /api/metrics/json
│   │   └── cause.py            #   /api/cause/{explain,trace,events}
│   ├── pipeline/               # Build execution
│   │   ├── engine.py           #   DAG executor with pause/step/critical-path
│   │   └── runner.py           #   BuildRunner (lifecycle + persistence)
│   ├── scheduler/              # Node selection
│   │   └── optimizer.py        #   choose_best_node (weighted scoring)
│   ├── analysis/               # Reasoning layer
│   │   ├── cause_graph.py      #   Causal graph from events
│   │   ├── timeline_store.py   #   SQLite-backed event journal
│   │   ├── failure_autopsy.py  #   Backward dependency tracing
│   │   └── failure_propagation.py # Forward blast radius
│   ├── events/                 # Event infrastructure
│   │   ├── bus.py              #   Singleton EventBus
│   │   └── schema.py           #   FesterEvent dataclass + EventType enum
│   ├── nodes/                  # Node management
│   │   ├── probe.py            #   HTTP agent probing (async + sync)
│   │   ├── roles.py            #   NodeRole dataclass
│   │   └── state_model.py      #   NodeStateRegistry
│   ├── storage/                # Storage layer
│   │   ├── btrfs_cas.py        #   CoW reflink snapshots
│   │   ├── qcow2_freeze.py     #   QCOW2 workspace freezing
│   │   ├── tmpfs.py            #   tmpfs workspace acceleration
│   │   └── sqlite_db.py        #   Persistent storage (builds, sessions, events)
│   ├── integrations/           # External tool integrations
│   │   ├── tmux.py             #   Detached tmux action runtime
│   │   ├── mosh.py             #   mosh/ssh shell command builder
│   │   ├── forgejo.py          #   Forgejo push webhook handler
│   │   ├── lxc.py              #   LXC container execution
│   │   └── libvirt.py          #   libvirt VM execution
│   ├── executor/               # Runtime router (host/lxc/libvirt/tmux)
│   ├── cache/                  # MinIO distributed cache
│   ├── graph/                  # DAG construction + critical path
│   ├── metrics/                # Prometheus exporter + observability hub
│   ├── policy/                 # Policy engine (rules + overrides)
│   └── targets/                # Build target catalog (Gentoo, Buildroot, ...)
├── ui/                         # Frontend (vanilla JS, no build step)
│   ├── app.js                  # Shared shell (WS, toasts, topbar, health)
│   ├── live_dag.html           # Real-time DAG with layered layout
│   ├── replay.html             # Timeline scrubber + autopsy
│   ├── sessions.html           # Build history + tmux output viewer
│   ├── metrics.html            # Live charts + per-node cards
│   ├── cause.html              # Cause graph + blast radius
│   ├── timeline.html           # Per-node event drill-down
│   └── debugger.html           # Step-through execution control
├── cli/                        # CLI tool (35+ subcommands)
│   └── fester.py
├── cockpit/                    # Cockpit module (optional)
├── docs/                       # Documentation + screenshots
├── scripts/                    # Helper scripts (mock agent, syntax checker)
├── pyproject.toml              # Python packaging
├── bootstrap.sh                # One-shot dependency installer
├── run.sh                      # Start the backend
└── config.yaml                 # Cluster configuration

🚀 Quick Start

# 1. Clone
git clone https://git.dcos.net/dcosnet/fester.git
cd fester

# 2. Bootstrap (creates venv, installs deps, sets up config)
./bootstrap.sh

# 3. Run
./run.sh

# 4. Open the UI
open http://localhost:8080

See quickstart.md for the full 5-minute walkthrough.

🛠️ Prerequisites

• Python 3.12+
• pip (for dependency installation)
• Optional (features activate automatically when installed):
  • tmux — for the tmux action runtime (live output viewing)
  • mosh + ssh — for the "Shell into node" UI button
  • qemu-utils — for QCOW2 workspace snapshots
  • rsync — for workspace freeze operations
  • btrfs-progs — for CoW reflink snapshots (requires btrfs filesystem)
  • minio server — for distributed cache (otherwise falls back to local)

📊 Tech Stack

Layer	Technology
Backend	Python 3.12, FastAPI 0.128, uvicorn, pydantic 2
Realtime	WebSockets (3 channels: /ws, /ws-targets, /ws-debugger)
Storage	SQLite (WAL mode) for builds/sessions/events; MinIO for distributed cache
Frontend	Vanilla HTML/CSS/JS (no build step, no framework, ~12KB total)
Observability	Prometheus exposition + Grafana dashboard configs
CLI	Python argparse (35+ subcommands)
Integrations	tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache

🎛️ Configuration

Fester reads from config.yaml (or FESTER_CONFIG env var) at startup:

master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
  - name: x99-v4
    host: 192.168.1.11
    max_jobs: 30

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"

Environment Variables

Variable	Default	Purpose
FESTER_CONFIG	config.yaml	Path to YAML config
FESTER_DB_PATH	/var/lib/fester/fester.db (with fallbacks)	SQLite database location
FESTER_NO_DRIFT	unset	Set to 1 to disable synthetic drift when agents unreachable
FESTER_AUTOBUILD	unset	Set to 1 to auto-trigger a build every 60s
FESTER_ROLE_DB	/etc/fester/node_roles.json	Node role overrides
FESTER_CACHE_DIR	/var/lib/fester/cache	Local cache directory
FESTER_STORAGE_CONFIG	/etc/fester/storage.json	Storage layer config
FESTER_API	http://localhost:8080	CLI: backend URL
FESTER_WS	ws://localhost:8080/ws	CLI: WebSocket URL

🔌 API Surface

71 routes total. Key endpoints:

Method	Path	Purpose
POST	/api/build	Kick off a build
GET	/api/builds	Build history
POST	/api/builds/{id}/cancel	Cancel a running build
GET	/api/nodes	List cluster nodes (with live metrics)
POST	/api/nodes/{name}/policy	Set node policy (preferred/avoid/neutral)
POST	/api/nodes/{name}/probe	Manually probe a node's agent
POST	/api/nodes/{name}/shell	Build mosh/ssh shell command
GET	/api/metrics/json	JSON metrics snapshot
GET	/metrics	Prometheus exposition
GET	/api/cause/explain/{node}	Causal chain for a node
GET	/api/propagation/{action}	Blast radius if action fails
POST	/replay/start	Start a replay session
GET	/replay/events	Full event journal
GET	/autopsy/{sid}/{action}	Failure autopsy
GET	/api/storage/status	Storage capabilities
POST	/api/storage/freeze	Freeze workdir into qcow2
GET	/api/actions/active	List active tmux sessions
WS	/ws	Main event stream
WS	/ws-debugger	Debugger control channel
WS	/ws-targets	Target toggle notifications

Full route list: curl http://localhost:8080/openapi.json | jq '.paths | keys[]'

🖥️ CLI

# Build + watch
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch

# List builds
fester builds

# Node management
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3

# Cause analysis
fester cause explain x99-v3 build_kernel
fester blast build_kernel

# Live event stream
fester stream

# Health check
fester health

Run fester --help for the full list of 35+ subcommands.

📚 Documentation

• quickstart.md — 5-minute getting started guide
• CHEATSHEET.md — Operator survival guide
• CONTRIBUTING.md — How to contribute
• CHANGELOG.md — Version history
• docs/screenshots/ — UI screenshots

🐳 Docker

docker build -t fester .
docker run -p 8080:8080 -v fester-data:/var/lib/fester fester

See Dockerfile for details.

🔐 Security

Fester is designed to run behind your existing network controls (NAT, firewalls like OPNsense, etc.). There is no built-in auth — assume the network is trusted. If you need to expose it publicly, put it behind a reverse proxy with auth (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).

⚠️ License

GNU Affero General Public License v3.0 — see LICENSE.

Software is provided "AS IS", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.