dcosnet
/
fester
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
fester/README.md

323 lines
17 KiB
Markdown
Raw Permalink Blame History

# 🧠 Fester
**A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.**
[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE)
[![Python: 3.12+](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://www.python.org/)
[![FastAPI](https://img.shields.io/badge/FastAPI-0.128+-green.svg)](https://fastapi.tiangolo.com/)
---
## 📸 Screenshots
| Dashboard | Live DAG | Replay |
|-----------|----------|--------|
| ![Dashboard](docs/screenshots/01_dashboard.png) | ![Live DAG](docs/screenshots/02_live_dag.png) | ![Replay](docs/screenshots/03_replay.png) |
| Sessions | Metrics | Cause Graph |
|----------|---------|-------------|
| ![Sessions](docs/screenshots/04_sessions.png) | ![Metrics](docs/screenshots/05_metrics.png) | ![Cause](docs/screenshots/06_cause.png) |
| Timeline | Debugger |
|----------|----------|
| ![Timeline](docs/screenshots/07_timeline.png) | ![Debugger](docs/screenshots/08_debugger.png) |
---
## 🎯 What Fester Does
Fester turns a cluster of machines into a single, observable build brain:
- **DAG-driven execution** — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
- **Smart scheduler** — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
- **Real-time observability** — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
- **Deterministic replay** — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
- **Failure autopsy** — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
- **Cause graph** — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
- **Cache layer** — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots
## 🏗️ Architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Fester Cluster │
├──────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ Backend │◄──►│ EventBus │◄──►│ PipelineEngine │ │
│ │ (FastAPI) │ │ (singleton)│ │ (DAG executor) │ │
│ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │
│ │ │ │ │
│ │ │ ┌───────────────────┼──────────┐ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ ┌──────┴──────┐ ┌─────────────┐ ┌─────────────────┐ │ │
│ │ WebSocket │ │ Timeline │ │ Scheduler │ │ │
│ │ Stream │ │ Store │ │ (weighted/thermal│ │ │
│ │ (/ws, etc) │ │ (SQLite) │ │ /cache-aware) │ │ │
│ └──────┬──────┘ └──────┬──────┘ └────────┬────────┘ │ │
│ │ │ │ │ │
│ │ │ ┌───────┴────────┐ │ │
│ │ │ │ Node Registry │ │ │
│ │ │ │ (probe + drift)│ │ │
│ ▼ ▼ └────────────────┘ │ │
│ ┌──────────────────────────────────────────────────────┐ │ │
│ │ UI (8 pages) │ │ │
│ │ Dashboard │ Live DAG │ Replay │ Sessions │ Metrics │ │ │
│ │ Cause Graph │ Timeline │ Debugger │ │ │
│ └──────────────────────────────────────────────────────┘ │ │
│ │ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────┐│ │
│ │ MinIO Cache│ │ Btrfs CAS │ │ QCOW2 Snap │ │ tmux ││ │
│ │ (optional) │ │ (optional)│ │ (optional) │ │ runtime││ │
│ └────────────┘ └────────────┘ └────────────┘ └────────┘│ │
│ │ │
└──────────────────────────────────────────────────────────────┘ │
│
▲ │
│ HTTP :8787/status (probe) │
│ │
┌───────────────┴────────────────────────────────┐ │
│ Cluster Nodes │ │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐│ │
│ │ x99-v3 │ │ x99-v4 │ │ rpi-1 │ │ ryzen-1││ │
│ │(x86_64)│ │(x86_64)│ │(arm64) │ │(x86_64)││ │
│ └────────┘ └────────┘ └────────┘ └────────┘│ │
└────────────────────────────────────────────────┘ │
```
## 📁 Project Layout
```
fester/
├── backend/ # Python backend (FastAPI)
│ ├── main.py # App entrypoint + 71 routes
│ ├── api/ # REST routers + WebSocket hub
│ │ ├── build.py # /api/build
│ │ ├── nodes.py # /api/nodes (list, policy, probe)
│ │ ├── replay.py # /replay/start, /events
│ │ ├── autopsy.py # /autopsy/{sid}/{action}
│ │ ├── timeline.py # /timeline/{node}, /rewind/{i}
│ │ ├── debugger.py # /debugger/{pause,resume,step}
│ │ ├── pipeline_control.py # /api/pipeline/{state,retry,...}
│ │ ├── metrics.py # /metrics (Prometheus) + /api/metrics/json
│ │ └── cause.py # /api/cause/{explain,trace,events}
│ ├── pipeline/ # Build execution
│ │ ├── engine.py # DAG executor with pause/step/critical-path
│ │ └── runner.py # BuildRunner (lifecycle + persistence)
│ ├── scheduler/ # Node selection
│ │ └── optimizer.py # choose_best_node (weighted scoring)
│ ├── analysis/ # Reasoning layer
│ │ ├── cause_graph.py # Causal graph from events
│ │ ├── timeline_store.py # SQLite-backed event journal
│ │ ├── failure_autopsy.py # Backward dependency tracing
│ │ └── failure_propagation.py # Forward blast radius
│ ├── events/ # Event infrastructure
│ │ ├── bus.py # Singleton EventBus
│ │ └── schema.py # FesterEvent dataclass + EventType enum
│ ├── nodes/ # Node management
│ │ ├── probe.py # HTTP agent probing (async + sync)
│ │ ├── roles.py # NodeRole dataclass
│ │ └── state_model.py # NodeStateRegistry
│ ├── storage/ # Storage layer
│ │ ├── btrfs_cas.py # CoW reflink snapshots
│ │ ├── qcow2_freeze.py # QCOW2 workspace freezing
│ │ ├── tmpfs.py # tmpfs workspace acceleration
│ │ └── sqlite_db.py # Persistent storage (builds, sessions, events)
│ ├── integrations/ # External tool integrations
│ │ ├── tmux.py # Detached tmux action runtime
│ │ ├── mosh.py # mosh/ssh shell command builder
│ │ ├── forgejo.py # Forgejo push webhook handler
│ │ ├── lxc.py # LXC container execution
│ │ └── libvirt.py # libvirt VM execution
│ ├── executor/ # Runtime router (host/lxc/libvirt/tmux)
│ ├── cache/ # MinIO distributed cache
│ ├── graph/ # DAG construction + critical path
│ ├── metrics/ # Prometheus exporter + observability hub
│ ├── policy/ # Policy engine (rules + overrides)
│ └── targets/ # Build target catalog (Gentoo, Buildroot, ...)
├── ui/ # Frontend (vanilla JS, no build step)
│ ├── app.js # Shared shell (WS, toasts, topbar, health)
│ ├── live_dag.html # Real-time DAG with layered layout
│ ├── replay.html # Timeline scrubber + autopsy
│ ├── sessions.html # Build history + tmux output viewer
│ ├── metrics.html # Live charts + per-node cards
│ ├── cause.html # Cause graph + blast radius
│ ├── timeline.html # Per-node event drill-down
│ └── debugger.html # Step-through execution control
├── cli/ # CLI tool (35+ subcommands)
│ └── fester.py
├── cockpit/ # Cockpit module (optional)
├── docs/ # Documentation + screenshots
├── scripts/ # Helper scripts (mock agent, syntax checker)
├── pyproject.toml # Python packaging
├── bootstrap.sh # One-shot dependency installer
├── run.sh # Start the backend
└── config.yaml # Cluster configuration
```
## 🚀 Quick Start
```bash
# 1. Clone
git clone https://git.dcos.net/dcosnet/fester.git
cd fester
# 2. Bootstrap (creates venv, installs deps, sets up config)
./bootstrap.sh
# 3. Run
./run.sh
# 4. Open the UI
open http://localhost:8080
```
See **[quickstart.md](quickstart.md)** for the full 5-minute walkthrough.
## 🛠️ Prerequisites
- **Python 3.12+**
- **pip** (for dependency installation)
- **Optional** (features activate automatically when installed):
- `tmux` — for the tmux action runtime (live output viewing)
- `mosh` + `ssh` — for the "Shell into node" UI button
- `qemu-utils` — for QCOW2 workspace snapshots
- `rsync` — for workspace freeze operations
- `btrfs-progs` — for CoW reflink snapshots (requires btrfs filesystem)
- `minio` server — for distributed cache (otherwise falls back to local)
## 📊 Tech Stack
| Layer | Technology |
|-------|------------|
| Backend | Python 3.12, FastAPI 0.128, uvicorn, pydantic 2 |
| Realtime | WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`) |
| Storage | SQLite (WAL mode) for builds/sessions/events; MinIO for distributed cache |
| Frontend | Vanilla HTML/CSS/JS (no build step, no framework, ~12KB total) |
| Observability | Prometheus exposition + Grafana dashboard configs |
| CLI | Python argparse (35+ subcommands) |
| Integrations | tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache |
## 🎛️ Configuration
Fester reads from `config.yaml` (or `FESTER_CONFIG` env var) at startup:
```yaml
master:
name: fester-master
role: control
nodes:
- name: x99-v3
host: 192.168.1.10
max_jobs: 24
- name: x99-v4
host: 192.168.1.11
max_jobs: 30
projects:
- name: linux-tool
repo: https://forgejo.local/linux-tool.git
targets:
debian: "make clean && make debian"
arch: "make clean && make arch"
```
### Environment Variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `FESTER_CONFIG` | `config.yaml` | Path to YAML config |
| `FESTER_DB_PATH` | `/var/lib/fester/fester.db` (with fallbacks) | SQLite database location |
| `FESTER_NO_DRIFT` | unset | Set to `1` to disable synthetic drift when agents unreachable |
| `FESTER_AUTOBUILD` | unset | Set to `1` to auto-trigger a build every 60s |
| `FESTER_ROLE_DB` | `/etc/fester/node_roles.json` | Node role overrides |
| `FESTER_CACHE_DIR` | `/var/lib/fester/cache` | Local cache directory |
| `FESTER_STORAGE_CONFIG` | `/etc/fester/storage.json` | Storage layer config |
| `FESTER_API` | `http://localhost:8080` | CLI: backend URL |
| `FESTER_WS` | `ws://localhost:8080/ws` | CLI: WebSocket URL |
## 🔌 API Surface
71 routes total. Key endpoints:
| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/api/build` | Kick off a build |
| `GET` | `/api/builds` | Build history |
| `POST` | `/api/builds/{id}/cancel` | Cancel a running build |
| `GET` | `/api/nodes` | List cluster nodes (with live metrics) |
| `POST` | `/api/nodes/{name}/policy` | Set node policy (preferred/avoid/neutral) |
| `POST` | `/api/nodes/{name}/probe` | Manually probe a node's agent |
| `POST` | `/api/nodes/{name}/shell` | Build mosh/ssh shell command |
| `GET` | `/api/metrics/json` | JSON metrics snapshot |
| `GET` | `/metrics` | Prometheus exposition |
| `GET` | `/api/cause/explain/{node}` | Causal chain for a node |
| `GET` | `/api/propagation/{action}` | Blast radius if action fails |
| `POST` | `/replay/start` | Start a replay session |
| `GET` | `/replay/events` | Full event journal |
| `GET` | `/autopsy/{sid}/{action}` | Failure autopsy |
| `GET` | `/api/storage/status` | Storage capabilities |
| `POST` | `/api/storage/freeze` | Freeze workdir into qcow2 |
| `GET` | `/api/actions/active` | List active tmux sessions |
| `WS` | `/ws` | Main event stream |
| `WS` | `/ws-debugger` | Debugger control channel |
| `WS` | `/ws-targets` | Target toggle notifications |
Full route list: `curl http://localhost:8080/openapi.json | jq '.paths | keys[]'`
## 🖥️ CLI
```bash
# Build + watch
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch
# List builds
fester builds
# Node management
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3
# Cause analysis
fester cause explain x99-v3 build_kernel
fester blast build_kernel
# Live event stream
fester stream
# Health check
fester health
```
Run `fester --help` for the full list of 35+ subcommands.
## 📚 Documentation
- **[quickstart.md](quickstart.md)** — 5-minute getting started guide
- **[CHEATSHEET.md](CHEATSHEET.md)** — Operator survival guide
- **[CONTRIBUTING.md](CONTRIBUTING.md)** — How to contribute
- **[CHANGELOG.md](CHANGELOG.md)** — Version history
- **[docs/screenshots/](docs/screenshots/)** — UI screenshots
## 🐳 Docker
```bash
docker build -t fester .
docker run -p 8080:8080 -v fester-data:/var/lib/fester fester
```
See **[Dockerfile](Dockerfile)** for details.
## 🔐 Security
Fester is designed to run behind your existing network controls (NAT, firewalls like OPNsense, etc.). There is **no built-in auth** — assume the network is trusted. If you need to expose it publicly, put it behind a reverse proxy with auth (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).
## ⚠️ License
GNU Affero General Public License v3.0 — see **[LICENSE](LICENSE)**.
Software is provided "AS IS", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.
Reference in New Issue View Git Blame Copy Permalink