dcosnet
/
fester
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
fester/README.md

289 lines
14 KiB
Markdown
Executable File
Raw Blame History

# Fester
A distributed, DAG-driven build execution system with real-time scheduling,
thermal/load awareness, cache-aware execution, and deterministic replay/debugging.
[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE)
[![Python: 3.12+](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://www.python.org/)
[![FastAPI](https://img.shields.io/badge/FastAPI-0.128+-green.svg)](https://fastapi.tiangolo.com/)
---
## Screenshots
| Dashboard | Live DAG | Replay |
|-----------|----------|--------|
| ![Dashboard](docs/screenshots/01_dashboard.png) | ![Live DAG](docs/screenshots/02_live_dag.png) | ![Replay](docs/screenshots/03_replay.png) |
| Sessions | Metrics | Cause Graph |
|----------|---------|-------------|
| ![Sessions](docs/screenshots/04_sessions.png) | ![Metrics](docs/screenshots/05_metrics.png) | ![Cause](docs/screenshots/06_cause.png) |
| Timeline | Debugger |
|----------|----------|
| ![Timeline](docs/screenshots/07_timeline.png) | ![Debugger](docs/screenshots/08_debugger.png) |
---
## What Fester Does
Fester turns a cluster of machines into a single, observable build brain:
- **DAG-driven execution** — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
- **Smart scheduler** — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
- **Real-time observability** — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
- **Deterministic replay** — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
- **Failure autopsy** — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
- **Cause graph** — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
- **Cache layer** — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots + shared CAS API
---
## Architecture
```
+------------------------------------------------------------------+
| Fester Cluster |
+------------------------------------------------------------------+
| |
| +-------------+ +-------------+ +---------------------+ |
| | Backend |<-->| EventBus |<-->| PipelineEngine | |
| | (FastAPI) | | (singleton)| | (DAG executor) | |
| +------+------+ +------+------+ +----------+----------+ |
| | | | |
| | | +-------------------+----------+ |
| | | | | | |
| | v v v | |
| +------+------+ +-------------+ +-----------------+ | |
| | WebSocket | | Timeline | | Scheduler | | |
| | Stream | | Store | | (weighted/thermal| | |
| | (/ws, etc) | | (SQLite) | | /cache-aware) | | |
| +------+------+ +------+------+ +--------+--------+ | |
| | | | | |
| | | +-------+--------+ | |
| | | | Node Registry | | |
| | | | (probe + drift) | | |
| v v +----------------+ | |
| +------------------------------------------------------+ | |
| | UI (8 pages) | | |
| | Dashboard | Live DAG | Replay | Sessions | Metrics | | |
| | Cause Graph | Timeline | Debugger | | |
| +------------------------------------------------------+ | |
| |
| +------------+ +------------+ +------------+ +--------+ |
| | MinIO Cache| | Btrfs CAS | | QCOW2 Snap | | tmux | |
| | (optional) | | (optional) | | (optional) | | runtime| |
| +------------+ +------------+ +------------+ +--------+ |
| |
+------------------------------------------------------------------+
^
| HTTP :8787/status (probe)
|
+---------------+--------------------------------+
| Cluster Nodes |
| +--------+ +--------+ +--------+ +--------+ |
| | x99-v3 | | x99-v4 | | rpi-1 | | ryzen-1| |
| |(x86_64)| |(x86_64)| |(arm64) | |(x86_64)| |
| +--------+ +--------+ +--------+ +--------+ |
+------------------------------------------------+
```
## Project Layout
```
fester/
+-- backend/ # Python backend (FastAPI)
| +-- main.py # App entrypoint + 71 routes
| +-- api/ # REST routers + WebSocket hub
| +-- pipeline/ # Build execution (DAG engine + runner)
| +-- scheduler/ # Node selection (weighted scoring)
| +-- toolchain/ # BTC.sh cross-compilation integration
| | +-- btc.py # Probe, build env, stamp verification
| | +-- chroot.py # Chroot-based build environment
| +-- analysis/ # Cause graph, timeline, failure autopsy
| +-- events/ # Event infrastructure (bus + schema)
| +-- nodes/ # Node management (probe, roles, state)
| +-- storage/ # SQLite, Btrfs CAS, QCOW2, tmpfs
| +-- integrations/ # tmux, mosh, LXC, libvirt, Forgejo, Podman, Firecracker
| +-- executor/ # Runtime router (host/lxc/libvirt/tmux/podman/firecracker)
| +-- targets/ # Build target catalog (Gentoo, Buildroot, OpenWrt, etc.)
| +-- metrics/ # Prometheus exporter + observability hub
| +-- policy/ # Policy engine (rules + overrides)
+-- ui/ # Frontend (vanilla JS, no build step, ~12KB)
+-- cli/ # CLI tool (35+ subcommands)
+-- cockpit/ # Cockpit module (optional)
+-- docs/ # Documentation + screenshots
+-- pyproject.toml # Python packaging
+-- install.sh # Unified installer (deps / config / db / deploy / run)
+-- run.sh # Start the backend (also via `./install.sh run`)
+-- config.yaml # Cluster configuration
```
## Quick Start
```bash
git clone https://git.dcos.net/dcosnet/fester.git
cd fester
./install.sh # = ./install.sh all (venv + deps + config + db)
./install.sh run # start backend on :8181
# Open http://localhost:8181
```
See [quickstart.md](quickstart.md) for the full walkthrough.
## Prerequisites
- **Python 3.12+**
- **pip** (for dependency installation)
Optional (features activate automatically when installed):
| Feature | Requires |
|-------------------|-----------------------------|
| Live action output| `tmux` |
| Remote shell | `mosh` + `ssh` |
| Workspace snapshots| `qemu-utils` + `rsync` |
| CoW snapshots | `btrfs-progs` (btrfs fs) |
| Distributed cache | `minio` server |
## Tech Stack
| Layer | Technology |
|------------|---------------------------------------------------------------|
| Backend | Python 3.12, FastAPI 0.128, uvicorn, pydantic 2 |
| Realtime | WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`)|
| Storage | SQLite (WAL) for state; MinIO for distributed cache |
| Frontend | Vanilla HTML/CSS/JS (no build step, no framework) |
| Observability| Prometheus exposition + Grafana dashboard configs |
| CLI | Python argparse (35+ subcommands) |
| Integrations| tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache, Podman, Firecracker |
## BTC.sh Cross-Compilation Integration
Fester integrates with BTC.sh for multi-architecture cross-compilation. The
`backend/toolchain/btc.py` module probes for BTC golden images, parses their
manifest sidecars, and configures the build environment for 19 supported targets:
| Family | Targets | ISA Tiers |
|------------------|----------------------------------------------------------------|------------------|
| Intel HEDT/Server| haswell, haswell-ep, skylake, skylake-x, skylake-server | AVX2, AVX512 |
| AMD Ryzen/EPYC | znver1, znver2, znver3, znver4 | AVX2, AVX512 |
| AMD APU | apu-zn1, apu-zn2, apu-zn3, apu-zn4 | AVX2 |
| Intel Atom | atom-silvermont, atom-goldmont, atom-tremont, atom-sierraforest| SSE4_2 |
| Embedded | mipselr2, armv7, tilegx | MIPS32, NEON, TILE|
Per-node BTC target selection is configured in `config.yaml` under
`nodes[].btc.target`. When a BTC toolchain is active, Fester sets `CC`,
`CXX`, `CFLAGS`, `LDFLAGS`, and cross-compiler environment variables
automatically.
## Configuration
Fester reads from `config.yaml` (or `FESTER_CONFIG` env var) at startup:
```yaml
master:
name: fester-master
role: control
nodes:
- name: x99-v3
host: 192.168.1.10
max_jobs: 24
btc:
target: znver3 # Use BTC.sh znver3 cross-toolchain on this node
projects:
- name: linux-tool
repo: https://forgejo.local/linux-tool.git
targets:
debian: "make clean && make debian"
arch: "make clean && make arch"
```
### Environment Variables
| Variable | Default | Purpose |
|-----------------------|----------------------------------|---------------------------------|
| `FESTER_CONFIG` | `config.yaml` | Path to YAML config |
| `FESTER_DB_PATH` | `/var/lib/fester/fester.db` | SQLite database location |
| `FESTER_NO_DRIFT` | unset | Disable synthetic drift |
| `FESTER_AUTOBUILD` | unset | Auto-trigger builds every 60s |
| `FESTER_ROLE_DB` | `/etc/fester/node_roles.json` | Node role overrides |
| `FESTER_CACHE_DIR` | `/var/lib/fester/cache` | Local cache directory |
| `FESTER_STORAGE_CONFIG`| `/etc/fester/storage.json` | Storage layer config |
| `FESTER_API` | `http://localhost:8181` | CLI: backend URL |
| `FESTER_WS` | `ws://localhost:8181/ws` | CLI: WebSocket URL |
## API Surface
71 routes total. Key endpoints:
| Method | Path | Purpose |
|--------|---------------------------------|----------------------------------------|
| `POST` | `/api/build` | Kick off a build |
| `GET` | `/api/builds` | Build history |
| `POST` | `/api/builds/{id}/cancel` | Cancel a running build |
| `GET` | `/api/nodes` | List cluster nodes (with live metrics) |
| `POST` | `/api/nodes/{name}/policy` | Set node policy |
| `POST` | `/api/nodes/{name}/probe` | Manually probe a node |
| `GET` | `/api/metrics/json` | JSON metrics snapshot |
| `GET` | `/metrics` | Prometheus exposition |
| `GET` | `/api/cause/explain/{node}` | Causal chain for a node |
| `GET` | `/api/propagation/{action}` | Blast radius if action fails |
| `POST` | `/replay/start` | Start a replay session |
| `GET` | `/autopsy/{sid}/{action}` | Failure autopsy |
| `PUT` | `/api/cas/{sha256}` | Store artifact in CAS |
| `GET` | `/api/cas/{sha256}` | Retrieve artifact from CAS |
| `GET` | `/api/cas/stats` | Cache statistics |
| `WS` | `/ws` | Main event stream |
| `WS` | `/ws-debugger` | Debugger control channel |
| `WS` | `/ws-targets` | Target toggle notifications |
Full route list: `curl http://localhost:8181/openapi.json | jq '.paths | keys[]'`
## CLI
```bash
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch
fester builds
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3
fester cause explain x99-v3 build_kernel
fester blast build_kernel
fester stream
fester health
```
Run `fester --help` for the full list of 35+ subcommands.
## Documentation
- [quickstart.md](quickstart.md) — 5-minute getting started guide
- [CHEATSHEET.md](CHEATSHEET.md) — Operator survival guide
- [CONTRIBUTING.md](CONTRIBUTING.md) — How to contribute
- [CHANGELOG.md](CHANGELOG.md) — Version history
## Docker
```bash
docker build -t fester .
docker run -p 8181:8181 -v fester-data:/var/lib/fester fester
```
## Security
Fester runs behind your existing network controls (OPNsense, IPFire, or any
firewall appliance). There is no built-in authentication — assume the network
is trusted. For public exposure, place it behind a reverse proxy with
authentication (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).
## License
GNU Affero General Public License v3.0 — see [LICENSE](LICENSE).
Software is provided "as is", without warranty of any kind. This is a
high-concurrency distributed execution system — review the safety notes in
the license before production use.
Reference in New Issue View Git Blame Copy Permalink