17 KiB

Raw Blame History

🧠 Fester

A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.

📸 Screenshots

Dashboard	Live DAG	Replay

Sessions	Metrics	Cause Graph

Timeline	Debugger

🎯 What Fester Does

Fester turns a cluster of machines into a single, observable build brain:

DAG-driven execution — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
Smart scheduler — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
Real-time observability — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
Deterministic replay — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
Failure autopsy — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
Cause graph — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
Cache layer — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                         Fester Cluster                            │
├──────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────┐   │
│  │   Backend   │◄──►│  EventBus   │◄──►│   PipelineEngine    │   │
│  │  (FastAPI)  │    │  (singleton)│    │  (DAG executor)     │   │
│  └──────┬──────┘    └──────┬──────┘    └──────────┬──────────┘   │
│         │                  │                      │              │
│         │                  │  ┌───────────────────┼──────────┐   │
│         │                  │  │                   │          │   │
│         │                  ▼  ▼                   ▼          │   │
│  ┌──────┴──────┐    ┌─────────────┐    ┌─────────────────┐   │   │
│  │  WebSocket  │    │  Timeline   │    │   Scheduler     │   │   │
│  │   Stream    │    │   Store     │    │ (weighted/thermal│   │   │
│  │ (/ws, etc)  │    │  (SQLite)   │    │   /cache-aware) │   │   │
│  └──────┬──────┘    └──────┬──────┘    └────────┬────────┘   │   │
│         │                  │                    │            │   │
│         │                  │            ┌───────┴────────┐   │   │
│         │                  │            │ Node Registry  │   │   │
│         │                  │            │ (probe + drift)│   │   │
│         ▼                  ▼            └────────────────┘   │   │
│  ┌──────────────────────────────────────────────────────┐   │   │
│  │                    UI (8 pages)                       │   │   │
│  │  Dashboard │ Live DAG │ Replay │ Sessions │ Metrics   │   │   │
│  │  Cause Graph │ Timeline │ Debugger                  │   │   │
│  └──────────────────────────────────────────────────────┘   │   │
│                                                              │   │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────┐│   │
│  │ MinIO Cache│  │ Btrfs CAS │  │ QCOW2 Snap │  │  tmux  ││   │
│  │ (optional) │  │ (optional)│  │ (optional) │  │ runtime││   │
│  └────────────┘  └────────────┘  └────────────┘  └────────┘│   │
│                                                              │   │
└──────────────────────────────────────────────────────────────┘   │
                                                                    │
                              ▲                                     │
                              │ HTTP :8787/status (probe)           │
                              │                                     │
              ┌───────────────┴────────────────────────────────┐   │
              │              Cluster Nodes                      │   │
              │  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐│   │
              │  │ x99-v3 │  │ x99-v4 │  │ rpi-1  │  │ ryzen-1││   │
              │  │(x86_64)│  │(x86_64)│  │(arm64) │  │(x86_64)││   │
              │  └────────┘  └────────┘  └────────┘  └────────┘│   │
              └────────────────────────────────────────────────┘   │

📁 Project Layout

fester/
├── backend/                    # Python backend (FastAPI)
│   ├── main.py                 # App entrypoint + 71 routes
│   ├── api/                    # REST routers + WebSocket hub
│   │   ├── build.py            #   /api/build
│   │   ├── nodes.py            #   /api/nodes (list, policy, probe)
│   │   ├── replay.py           #   /replay/start, /events
│   │   ├── autopsy.py          #   /autopsy/{sid}/{action}
│   │   ├── timeline.py         #   /timeline/{node}, /rewind/{i}
│   │   ├── debugger.py         #   /debugger/{pause,resume,step}
│   │   ├── pipeline_control.py #   /api/pipeline/{state,retry,...}
│   │   ├── metrics.py          #   /metrics (Prometheus) + /api/metrics/json
│   │   └── cause.py            #   /api/cause/{explain,trace,events}
│   ├── pipeline/               # Build execution
│   │   ├── engine.py           #   DAG executor with pause/step/critical-path
│   │   └── runner.py           #   BuildRunner (lifecycle + persistence)
│   ├── scheduler/              # Node selection
│   │   └── optimizer.py        #   choose_best_node (weighted scoring)
│   ├── analysis/               # Reasoning layer
│   │   ├── cause_graph.py      #   Causal graph from events
│   │   ├── timeline_store.py   #   SQLite-backed event journal
│   │   ├── failure_autopsy.py  #   Backward dependency tracing
│   │   └── failure_propagation.py # Forward blast radius
│   ├── events/                 # Event infrastructure
│   │   ├── bus.py              #   Singleton EventBus
│   │   └── schema.py           #   FesterEvent dataclass + EventType enum
│   ├── nodes/                  # Node management
│   │   ├── probe.py            #   HTTP agent probing (async + sync)
│   │   ├── roles.py            #   NodeRole dataclass
│   │   └── state_model.py      #   NodeStateRegistry
│   ├── storage/                # Storage layer
│   │   ├── btrfs_cas.py        #   CoW reflink snapshots
│   │   ├── qcow2_freeze.py     #   QCOW2 workspace freezing
│   │   ├── tmpfs.py            #   tmpfs workspace acceleration
│   │   └── sqlite_db.py        #   Persistent storage (builds, sessions, events)
│   ├── integrations/           # External tool integrations
│   │   ├── tmux.py             #   Detached tmux action runtime
│   │   ├── mosh.py             #   mosh/ssh shell command builder
│   │   ├── forgejo.py          #   Forgejo push webhook handler
│   │   ├── lxc.py              #   LXC container execution
│   │   └── libvirt.py          #   libvirt VM execution
│   ├── executor/               # Runtime router (host/lxc/libvirt/tmux)
│   ├── cache/                  # MinIO distributed cache
│   ├── graph/                  # DAG construction + critical path
│   ├── metrics/                # Prometheus exporter + observability hub
│   ├── policy/                 # Policy engine (rules + overrides)
│   └── targets/                # Build target catalog (Gentoo, Buildroot, ...)
├── ui/                         # Frontend (vanilla JS, no build step)
│   ├── app.js                  # Shared shell (WS, toasts, topbar, health)
│   ├── live_dag.html           # Real-time DAG with layered layout
│   ├── replay.html             # Timeline scrubber + autopsy
│   ├── sessions.html           # Build history + tmux output viewer
│   ├── metrics.html            # Live charts + per-node cards
│   ├── cause.html              # Cause graph + blast radius
│   ├── timeline.html           # Per-node event drill-down
│   └── debugger.html           # Step-through execution control
├── cli/                        # CLI tool (35+ subcommands)
│   └── fester.py
├── cockpit/                    # Cockpit module (optional)
├── docs/                       # Documentation + screenshots
├── scripts/                    # Helper scripts (mock agent, syntax checker)
├── pyproject.toml              # Python packaging
├── bootstrap.sh                # One-shot dependency installer
├── run.sh                      # Start the backend
└── config.yaml                 # Cluster configuration

🚀 Quick Start

# 1. Clone
git clone https://git.dcos.net/dcosnet/fester.git
cd fester

# 2. Bootstrap (creates venv, installs deps, sets up config)
./bootstrap.sh

# 3. Run
./run.sh

# 4. Open the UI
open http://localhost:8080

See quickstart.md for the full 5-minute walkthrough.

🛠️ Prerequisites

Python 3.12+
pip (for dependency installation)
Optional (features activate automatically when installed):
- tmux — for the tmux action runtime (live output viewing)
- mosh + ssh — for the "Shell into node" UI button
- qemu-utils — for QCOW2 workspace snapshots
- rsync — for workspace freeze operations
- btrfs-progs — for CoW reflink snapshots (requires btrfs filesystem)
- minio server — for distributed cache (otherwise falls back to local)

📊 Tech Stack

Layer	Technology
Backend	Python 3.12, FastAPI 0.128, uvicorn, pydantic 2
Realtime	WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`)
Storage	SQLite (WAL mode) for builds/sessions/events; MinIO for distributed cache
Frontend	Vanilla HTML/CSS/JS (no build step, no framework, ~12KB total)
Observability	Prometheus exposition + Grafana dashboard configs
CLI	Python argparse (35+ subcommands)
Integrations	tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache

🎛️ Configuration

Fester reads from config.yaml (or FESTER_CONFIG env var) at startup:

master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
  - name: x99-v4
    host: 192.168.1.11
    max_jobs: 30

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"

Environment Variables

Variable	Default	Purpose
`FESTER_CONFIG`	`config.yaml`	Path to YAML config
`FESTER_DB_PATH`	`/var/lib/fester/fester.db` (with fallbacks)	SQLite database location
`FESTER_NO_DRIFT`	unset	Set to `1` to disable synthetic drift when agents unreachable
`FESTER_AUTOBUILD`	unset	Set to `1` to auto-trigger a build every 60s
`FESTER_ROLE_DB`	`/etc/fester/node_roles.json`	Node role overrides
`FESTER_CACHE_DIR`	`/var/lib/fester/cache`	Local cache directory
`FESTER_STORAGE_CONFIG`	`/etc/fester/storage.json`	Storage layer config
`FESTER_API`	`http://localhost:8080`	CLI: backend URL
`FESTER_WS`	`ws://localhost:8080/ws`	CLI: WebSocket URL

🔌 API Surface

71 routes total. Key endpoints:

Method	Path	Purpose
`POST`	`/api/build`	Kick off a build
`GET`	`/api/builds`	Build history
`POST`	`/api/builds/{id}/cancel`	Cancel a running build
`GET`	`/api/nodes`	List cluster nodes (with live metrics)
`POST`	`/api/nodes/{name}/policy`	Set node policy (preferred/avoid/neutral)
`POST`	`/api/nodes/{name}/probe`	Manually probe a node's agent
`POST`	`/api/nodes/{name}/shell`	Build mosh/ssh shell command
`GET`	`/api/metrics/json`	JSON metrics snapshot
`GET`	`/metrics`	Prometheus exposition
`GET`	`/api/cause/explain/{node}`	Causal chain for a node
`GET`	`/api/propagation/{action}`	Blast radius if action fails
`POST`	`/replay/start`	Start a replay session
`GET`	`/replay/events`	Full event journal
`GET`	`/autopsy/{sid}/{action}`	Failure autopsy
`GET`	`/api/storage/status`	Storage capabilities
`POST`	`/api/storage/freeze`	Freeze workdir into qcow2
`GET`	`/api/actions/active`	List active tmux sessions
`WS`	`/ws`	Main event stream
`WS`	`/ws-debugger`	Debugger control channel
`WS`	`/ws-targets`	Target toggle notifications

Full route list: curl http://localhost:8080/openapi.json | jq '.paths | keys[]'

🖥️ CLI

# Build + watch
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch

# List builds
fester builds

# Node management
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3

# Cause analysis
fester cause explain x99-v3 build_kernel
fester blast build_kernel

# Live event stream
fester stream

# Health check
fester health

Run fester --help for the full list of 35+ subcommands.

📚 Documentation

quickstart.md — 5-minute getting started guide
CHEATSHEET.md — Operator survival guide
CONTRIBUTING.md — How to contribute
CHANGELOG.md — Version history
docs/screenshots/ — UI screenshots

🐳 Docker

docker build -t fester .
docker run -p 8080:8080 -v fester-data:/var/lib/fester fester

See Dockerfile for details.

🔐 Security

Fester is designed to run behind your existing network controls (NAT, firewalls like OPNsense, etc.). There is no built-in auth — assume the network is trusted. If you need to expose it publicly, put it behind a reverse proxy with auth (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).

⚠️ License

GNU Affero General Public License v3.0 — see LICENSE.

Software is provided "AS IS", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.

17 KiB Raw Blame History