323 lines
17 KiB
Markdown
323 lines
17 KiB
Markdown
# 🧠 Fester
|
|
|
|
**A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.**
|
|
|
|
[](LICENSE)
|
|
[](https://www.python.org/)
|
|
[](https://fastapi.tiangolo.com/)
|
|
|
|
---
|
|
|
|
## 📸 Screenshots
|
|
|
|
| Dashboard | Live DAG | Replay |
|
|
|-----------|----------|--------|
|
|
|  |  |  |
|
|
|
|
| Sessions | Metrics | Cause Graph |
|
|
|----------|---------|-------------|
|
|
|  |  |  |
|
|
|
|
| Timeline | Debugger |
|
|
|----------|----------|
|
|
|  |  |
|
|
|
|
---
|
|
|
|
## 🎯 What Fester Does
|
|
|
|
Fester turns a cluster of machines into a single, observable build brain:
|
|
|
|
- **DAG-driven execution** — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
|
|
- **Smart scheduler** — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
|
|
- **Real-time observability** — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
|
|
- **Deterministic replay** — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
|
|
- **Failure autopsy** — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
|
|
- **Cause graph** — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
|
|
- **Cache layer** — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots
|
|
|
|
## 🏗️ Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ Fester Cluster │
|
|
├──────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
|
│ │ Backend │◄──►│ EventBus │◄──►│ PipelineEngine │ │
|
|
│ │ (FastAPI) │ │ (singleton)│ │ (DAG executor) │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
|
|
│ │ │ │ │
|
|
│ │ │ ┌───────────────────┼──────────┐ │
|
|
│ │ │ │ │ │ │
|
|
│ │ ▼ ▼ ▼ │ │
|
|
│ ┌──────┴──────┐ ┌─────────────┐ ┌─────────────────┐ │ │
|
|
│ │ WebSocket │ │ Timeline │ │ Scheduler │ │ │
|
|
│ │ Stream │ │ Store │ │ (weighted/thermal│ │ │
|
|
│ │ (/ws, etc) │ │ (SQLite) │ │ /cache-aware) │ │ │
|
|
│ └──────┬──────┘ └──────┬──────┘ └────────┬────────┘ │ │
|
|
│ │ │ │ │ │
|
|
│ │ │ ┌───────┴────────┐ │ │
|
|
│ │ │ │ Node Registry │ │ │
|
|
│ │ │ │ (probe + drift)│ │ │
|
|
│ ▼ ▼ └────────────────┘ │ │
|
|
│ ┌──────────────────────────────────────────────────────┐ │ │
|
|
│ │ UI (8 pages) │ │ │
|
|
│ │ Dashboard │ Live DAG │ Replay │ Sessions │ Metrics │ │ │
|
|
│ │ Cause Graph │ Timeline │ Debugger │ │ │
|
|
│ └──────────────────────────────────────────────────────┘ │ │
|
|
│ │ │
|
|
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────┐│ │
|
|
│ │ MinIO Cache│ │ Btrfs CAS │ │ QCOW2 Snap │ │ tmux ││ │
|
|
│ │ (optional) │ │ (optional)│ │ (optional) │ │ runtime││ │
|
|
│ └────────────┘ └────────────┘ └────────────┘ └────────┘│ │
|
|
│ │ │
|
|
└──────────────────────────────────────────────────────────────┘ │
|
|
│
|
|
▲ │
|
|
│ HTTP :8787/status (probe) │
|
|
│ │
|
|
┌───────────────┴────────────────────────────────┐ │
|
|
│ Cluster Nodes │ │
|
|
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐│ │
|
|
│ │ x99-v3 │ │ x99-v4 │ │ rpi-1 │ │ ryzen-1││ │
|
|
│ │(x86_64)│ │(x86_64)│ │(arm64) │ │(x86_64)││ │
|
|
│ └────────┘ └────────┘ └────────┘ └────────┘│ │
|
|
└────────────────────────────────────────────────┘ │
|
|
```
|
|
|
|
## 📁 Project Layout
|
|
|
|
```
|
|
fester/
|
|
├── backend/ # Python backend (FastAPI)
|
|
│ ├── main.py # App entrypoint + 71 routes
|
|
│ ├── api/ # REST routers + WebSocket hub
|
|
│ │ ├── build.py # /api/build
|
|
│ │ ├── nodes.py # /api/nodes (list, policy, probe)
|
|
│ │ ├── replay.py # /replay/start, /events
|
|
│ │ ├── autopsy.py # /autopsy/{sid}/{action}
|
|
│ │ ├── timeline.py # /timeline/{node}, /rewind/{i}
|
|
│ │ ├── debugger.py # /debugger/{pause,resume,step}
|
|
│ │ ├── pipeline_control.py # /api/pipeline/{state,retry,...}
|
|
│ │ ├── metrics.py # /metrics (Prometheus) + /api/metrics/json
|
|
│ │ └── cause.py # /api/cause/{explain,trace,events}
|
|
│ ├── pipeline/ # Build execution
|
|
│ │ ├── engine.py # DAG executor with pause/step/critical-path
|
|
│ │ └── runner.py # BuildRunner (lifecycle + persistence)
|
|
│ ├── scheduler/ # Node selection
|
|
│ │ └── optimizer.py # choose_best_node (weighted scoring)
|
|
│ ├── analysis/ # Reasoning layer
|
|
│ │ ├── cause_graph.py # Causal graph from events
|
|
│ │ ├── timeline_store.py # SQLite-backed event journal
|
|
│ │ ├── failure_autopsy.py # Backward dependency tracing
|
|
│ │ └── failure_propagation.py # Forward blast radius
|
|
│ ├── events/ # Event infrastructure
|
|
│ │ ├── bus.py # Singleton EventBus
|
|
│ │ └── schema.py # FesterEvent dataclass + EventType enum
|
|
│ ├── nodes/ # Node management
|
|
│ │ ├── probe.py # HTTP agent probing (async + sync)
|
|
│ │ ├── roles.py # NodeRole dataclass
|
|
│ │ └── state_model.py # NodeStateRegistry
|
|
│ ├── storage/ # Storage layer
|
|
│ │ ├── btrfs_cas.py # CoW reflink snapshots
|
|
│ │ ├── qcow2_freeze.py # QCOW2 workspace freezing
|
|
│ │ ├── tmpfs.py # tmpfs workspace acceleration
|
|
│ │ └── sqlite_db.py # Persistent storage (builds, sessions, events)
|
|
│ ├── integrations/ # External tool integrations
|
|
│ │ ├── tmux.py # Detached tmux action runtime
|
|
│ │ ├── mosh.py # mosh/ssh shell command builder
|
|
│ │ ├── forgejo.py # Forgejo push webhook handler
|
|
│ │ ├── lxc.py # LXC container execution
|
|
│ │ └── libvirt.py # libvirt VM execution
|
|
│ ├── executor/ # Runtime router (host/lxc/libvirt/tmux)
|
|
│ ├── cache/ # MinIO distributed cache
|
|
│ ├── graph/ # DAG construction + critical path
|
|
│ ├── metrics/ # Prometheus exporter + observability hub
|
|
│ ├── policy/ # Policy engine (rules + overrides)
|
|
│ └── targets/ # Build target catalog (Gentoo, Buildroot, ...)
|
|
├── ui/ # Frontend (vanilla JS, no build step)
|
|
│ ├── app.js # Shared shell (WS, toasts, topbar, health)
|
|
│ ├── live_dag.html # Real-time DAG with layered layout
|
|
│ ├── replay.html # Timeline scrubber + autopsy
|
|
│ ├── sessions.html # Build history + tmux output viewer
|
|
│ ├── metrics.html # Live charts + per-node cards
|
|
│ ├── cause.html # Cause graph + blast radius
|
|
│ ├── timeline.html # Per-node event drill-down
|
|
│ └── debugger.html # Step-through execution control
|
|
├── cli/ # CLI tool (35+ subcommands)
|
|
│ └── fester.py
|
|
├── cockpit/ # Cockpit module (optional)
|
|
├── docs/ # Documentation + screenshots
|
|
├── scripts/ # Helper scripts (mock agent, syntax checker)
|
|
├── pyproject.toml # Python packaging
|
|
├── bootstrap.sh # One-shot dependency installer
|
|
├── run.sh # Start the backend
|
|
└── config.yaml # Cluster configuration
|
|
```
|
|
|
|
## 🚀 Quick Start
|
|
|
|
```bash
|
|
# 1. Clone
|
|
git clone https://git.dcos.net/dcosnet/fester.git
|
|
cd fester
|
|
|
|
# 2. Bootstrap (creates venv, installs deps, sets up config)
|
|
./bootstrap.sh
|
|
|
|
# 3. Run
|
|
./run.sh
|
|
|
|
# 4. Open the UI
|
|
open http://localhost:8080
|
|
```
|
|
|
|
See **[quickstart.md](quickstart.md)** for the full 5-minute walkthrough.
|
|
|
|
## 🛠️ Prerequisites
|
|
|
|
- **Python 3.12+**
|
|
- **pip** (for dependency installation)
|
|
- **Optional** (features activate automatically when installed):
|
|
- `tmux` — for the tmux action runtime (live output viewing)
|
|
- `mosh` + `ssh` — for the "Shell into node" UI button
|
|
- `qemu-utils` — for QCOW2 workspace snapshots
|
|
- `rsync` — for workspace freeze operations
|
|
- `btrfs-progs` — for CoW reflink snapshots (requires btrfs filesystem)
|
|
- `minio` server — for distributed cache (otherwise falls back to local)
|
|
|
|
## 📊 Tech Stack
|
|
|
|
| Layer | Technology |
|
|
|-------|------------|
|
|
| Backend | Python 3.12, FastAPI 0.128, uvicorn, pydantic 2 |
|
|
| Realtime | WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`) |
|
|
| Storage | SQLite (WAL mode) for builds/sessions/events; MinIO for distributed cache |
|
|
| Frontend | Vanilla HTML/CSS/JS (no build step, no framework, ~12KB total) |
|
|
| Observability | Prometheus exposition + Grafana dashboard configs |
|
|
| CLI | Python argparse (35+ subcommands) |
|
|
| Integrations | tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache |
|
|
|
|
## 🎛️ Configuration
|
|
|
|
Fester reads from `config.yaml` (or `FESTER_CONFIG` env var) at startup:
|
|
|
|
```yaml
|
|
master:
|
|
name: fester-master
|
|
role: control
|
|
|
|
nodes:
|
|
- name: x99-v3
|
|
host: 192.168.1.10
|
|
max_jobs: 24
|
|
- name: x99-v4
|
|
host: 192.168.1.11
|
|
max_jobs: 30
|
|
|
|
projects:
|
|
- name: linux-tool
|
|
repo: https://forgejo.local/linux-tool.git
|
|
targets:
|
|
debian: "make clean && make debian"
|
|
arch: "make clean && make arch"
|
|
```
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Default | Purpose |
|
|
|----------|---------|---------|
|
|
| `FESTER_CONFIG` | `config.yaml` | Path to YAML config |
|
|
| `FESTER_DB_PATH` | `/var/lib/fester/fester.db` (with fallbacks) | SQLite database location |
|
|
| `FESTER_NO_DRIFT` | unset | Set to `1` to disable synthetic drift when agents unreachable |
|
|
| `FESTER_AUTOBUILD` | unset | Set to `1` to auto-trigger a build every 60s |
|
|
| `FESTER_ROLE_DB` | `/etc/fester/node_roles.json` | Node role overrides |
|
|
| `FESTER_CACHE_DIR` | `/var/lib/fester/cache` | Local cache directory |
|
|
| `FESTER_STORAGE_CONFIG` | `/etc/fester/storage.json` | Storage layer config |
|
|
| `FESTER_API` | `http://localhost:8080` | CLI: backend URL |
|
|
| `FESTER_WS` | `ws://localhost:8080/ws` | CLI: WebSocket URL |
|
|
|
|
## 🔌 API Surface
|
|
|
|
71 routes total. Key endpoints:
|
|
|
|
| Method | Path | Purpose |
|
|
|--------|------|---------|
|
|
| `POST` | `/api/build` | Kick off a build |
|
|
| `GET` | `/api/builds` | Build history |
|
|
| `POST` | `/api/builds/{id}/cancel` | Cancel a running build |
|
|
| `GET` | `/api/nodes` | List cluster nodes (with live metrics) |
|
|
| `POST` | `/api/nodes/{name}/policy` | Set node policy (preferred/avoid/neutral) |
|
|
| `POST` | `/api/nodes/{name}/probe` | Manually probe a node's agent |
|
|
| `POST` | `/api/nodes/{name}/shell` | Build mosh/ssh shell command |
|
|
| `GET` | `/api/metrics/json` | JSON metrics snapshot |
|
|
| `GET` | `/metrics` | Prometheus exposition |
|
|
| `GET` | `/api/cause/explain/{node}` | Causal chain for a node |
|
|
| `GET` | `/api/propagation/{action}` | Blast radius if action fails |
|
|
| `POST` | `/replay/start` | Start a replay session |
|
|
| `GET` | `/replay/events` | Full event journal |
|
|
| `GET` | `/autopsy/{sid}/{action}` | Failure autopsy |
|
|
| `GET` | `/api/storage/status` | Storage capabilities |
|
|
| `POST` | `/api/storage/freeze` | Freeze workdir into qcow2 |
|
|
| `GET` | `/api/actions/active` | List active tmux sessions |
|
|
| `WS` | `/ws` | Main event stream |
|
|
| `WS` | `/ws-debugger` | Debugger control channel |
|
|
| `WS` | `/ws-targets` | Target toggle notifications |
|
|
|
|
Full route list: `curl http://localhost:8080/openapi.json | jq '.paths | keys[]'`
|
|
|
|
## 🖥️ CLI
|
|
|
|
```bash
|
|
# Build + watch
|
|
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch
|
|
|
|
# List builds
|
|
fester builds
|
|
|
|
# Node management
|
|
fester node list
|
|
fester node set-policy x99-v3 preferred
|
|
fester node probe x99-v3
|
|
|
|
# Cause analysis
|
|
fester cause explain x99-v3 build_kernel
|
|
fester blast build_kernel
|
|
|
|
# Live event stream
|
|
fester stream
|
|
|
|
# Health check
|
|
fester health
|
|
```
|
|
|
|
Run `fester --help` for the full list of 35+ subcommands.
|
|
|
|
## 📚 Documentation
|
|
|
|
- **[quickstart.md](quickstart.md)** — 5-minute getting started guide
|
|
- **[CHEATSHEET.md](CHEATSHEET.md)** — Operator survival guide
|
|
- **[CONTRIBUTING.md](CONTRIBUTING.md)** — How to contribute
|
|
- **[CHANGELOG.md](CHANGELOG.md)** — Version history
|
|
- **[docs/screenshots/](docs/screenshots/)** — UI screenshots
|
|
|
|
## 🐳 Docker
|
|
|
|
```bash
|
|
docker build -t fester .
|
|
docker run -p 8080:8080 -v fester-data:/var/lib/fester fester
|
|
```
|
|
|
|
See **[Dockerfile](Dockerfile)** for details.
|
|
|
|
## 🔐 Security
|
|
|
|
Fester is designed to run behind your existing network controls (NAT, firewalls like OPNsense, etc.). There is **no built-in auth** — assume the network is trusted. If you need to expose it publicly, put it behind a reverse proxy with auth (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).
|
|
|
|
## ⚠️ License
|
|
|
|
GNU Affero General Public License v3.0 — see **[LICENSE](LICENSE)**.
|
|
|
|
Software is provided "AS IS", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.
|