# 🧠 Fester

**A distributed, DAG-driven build execution system with real-time scheduling, thermal/load awareness, cache-aware execution, and deterministic replay/debugging.**

[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-blue.svg)](LICENSE)
[![Python: 3.12+](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://www.python.org/)
[![FastAPI](https://img.shields.io/badge/FastAPI-0.128+-green.svg)](https://fastapi.tiangolo.com/)

---

## 📸 Screenshots

| Dashboard | Live DAG | Replay |
|-----------|----------|--------|
| ![Dashboard](docs/screenshots/01_dashboard.png) | ![Live DAG](docs/screenshots/02_live_dag.png) | ![Replay](docs/screenshots/03_replay.png) |

| Sessions | Metrics | Cause Graph |
|----------|---------|-------------|
| ![Sessions](docs/screenshots/04_sessions.png) | ![Metrics](docs/screenshots/05_metrics.png) | ![Cause](docs/screenshots/06_cause.png) |

| Timeline | Debugger |
|----------|----------|
| ![Timeline](docs/screenshots/07_timeline.png) | ![Debugger](docs/screenshots/08_debugger.png) |

---

## 🎯 What Fester Does

Fester turns a cluster of machines into a single, observable build brain:

- **DAG-driven execution** — every build is compiled into a directed acyclic graph of actions, with dependencies honored and parallel branches run concurrently
- **Smart scheduler** — picks the best node for each action based on CPU load, temperature, policy constraints, and historical instability
- **Real-time observability** — every event (node state, schedule decision, task lifecycle, cache hit/miss, failure) streams live to the UI via WebSocket
- **Deterministic replay** — every build session is journaled to SQLite; scrub back through the timeline to see exactly what happened, when, and why
- **Failure autopsy** — when an action fails, trace its dependency chain backward, see the last scheduler decision, and compute the forward blast radius
- **Cause graph** — a post-hoc reasoning layer that builds a causal graph from system events, so you can ask "why did this node make this decision?"
- **Cache layer** — MinIO distributed cache + optional Btrfs CoW reflinks + QCOW2 workspace snapshots

## 🏗️ Architecture

```
┌──────────────────────────────────────────────────────────────────┐
│                         Fester Cluster                            │
├──────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────┐   │
│  │   Backend   │◄──►│  EventBus   │◄──►│   PipelineEngine    │   │
│  │  (FastAPI)  │    │  (singleton)│    │  (DAG executor)     │   │
│  └──────┬──────┘    └──────┬──────┘    └──────────┬──────────┘   │
│         │                  │                      │              │
│         │                  │  ┌───────────────────┼──────────┐   │
│         │                  │  │                   │          │   │
│         │                  ▼  ▼                   ▼          │   │
│  ┌──────┴──────┐    ┌─────────────┐    ┌─────────────────┐   │   │
│  │  WebSocket  │    │  Timeline   │    │   Scheduler     │   │   │
│  │   Stream    │    │   Store     │    │ (weighted/thermal│   │   │
│  │ (/ws, etc)  │    │  (SQLite)   │    │   /cache-aware) │   │   │
│  └──────┬──────┘    └──────┬──────┘    └────────┬────────┘   │   │
│         │                  │                    │            │   │
│         │                  │            ┌───────┴────────┐   │   │
│         │                  │            │ Node Registry  │   │   │
│         │                  │            │ (probe + drift)│   │   │
│         ▼                  ▼            └────────────────┘   │   │
│  ┌──────────────────────────────────────────────────────┐   │   │
│  │                    UI (8 pages)                       │   │   │
│  │  Dashboard │ Live DAG │ Replay │ Sessions │ Metrics   │   │   │
│  │  Cause Graph │ Timeline │ Debugger                  │   │   │
│  └──────────────────────────────────────────────────────┘   │   │
│                                                              │   │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  ┌────────┐│   │
│  │ MinIO Cache│  │ Btrfs CAS │  │ QCOW2 Snap │  │  tmux  ││   │
│  │ (optional) │  │ (optional)│  │ (optional) │  │ runtime││   │
│  └────────────┘  └────────────┘  └────────────┘  └────────┘│   │
│                                                              │   │
└──────────────────────────────────────────────────────────────┘   │
                                                                    │
                              ▲                                     │
                              │ HTTP :8787/status (probe)           │
                              │                                     │
              ┌───────────────┴────────────────────────────────┐   │
              │              Cluster Nodes                      │   │
              │  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐│   │
              │  │ x99-v3 │  │ x99-v4 │  │ rpi-1  │  │ ryzen-1││   │
              │  │(x86_64)│  │(x86_64)│  │(arm64) │  │(x86_64)││   │
              │  └────────┘  └────────┘  └────────┘  └────────┘│   │
              └────────────────────────────────────────────────┘   │
```

## 📁 Project Layout

```
fester/
├── backend/                    # Python backend (FastAPI)
│   ├── main.py                 # App entrypoint + 71 routes
│   ├── api/                    # REST routers + WebSocket hub
│   │   ├── build.py            #   /api/build
│   │   ├── nodes.py            #   /api/nodes (list, policy, probe)
│   │   ├── replay.py           #   /replay/start, /events
│   │   ├── autopsy.py          #   /autopsy/{sid}/{action}
│   │   ├── timeline.py         #   /timeline/{node}, /rewind/{i}
│   │   ├── debugger.py         #   /debugger/{pause,resume,step}
│   │   ├── pipeline_control.py #   /api/pipeline/{state,retry,...}
│   │   ├── metrics.py          #   /metrics (Prometheus) + /api/metrics/json
│   │   └── cause.py            #   /api/cause/{explain,trace,events}
│   ├── pipeline/               # Build execution
│   │   ├── engine.py           #   DAG executor with pause/step/critical-path
│   │   └── runner.py           #   BuildRunner (lifecycle + persistence)
│   ├── scheduler/              # Node selection
│   │   └── optimizer.py        #   choose_best_node (weighted scoring)
│   ├── analysis/               # Reasoning layer
│   │   ├── cause_graph.py      #   Causal graph from events
│   │   ├── timeline_store.py   #   SQLite-backed event journal
│   │   ├── failure_autopsy.py  #   Backward dependency tracing
│   │   └── failure_propagation.py # Forward blast radius
│   ├── events/                 # Event infrastructure
│   │   ├── bus.py              #   Singleton EventBus
│   │   └── schema.py           #   FesterEvent dataclass + EventType enum
│   ├── nodes/                  # Node management
│   │   ├── probe.py            #   HTTP agent probing (async + sync)
│   │   ├── roles.py            #   NodeRole dataclass
│   │   └── state_model.py      #   NodeStateRegistry
│   ├── storage/                # Storage layer
│   │   ├── btrfs_cas.py        #   CoW reflink snapshots
│   │   ├── qcow2_freeze.py     #   QCOW2 workspace freezing
│   │   ├── tmpfs.py            #   tmpfs workspace acceleration
│   │   └── sqlite_db.py        #   Persistent storage (builds, sessions, events)
│   ├── integrations/           # External tool integrations
│   │   ├── tmux.py             #   Detached tmux action runtime
│   │   ├── mosh.py             #   mosh/ssh shell command builder
│   │   ├── forgejo.py          #   Forgejo push webhook handler
│   │   ├── lxc.py              #   LXC container execution
│   │   └── libvirt.py          #   libvirt VM execution
│   ├── executor/               # Runtime router (host/lxc/libvirt/tmux)
│   ├── cache/                  # MinIO distributed cache
│   ├── graph/                  # DAG construction + critical path
│   ├── metrics/                # Prometheus exporter + observability hub
│   ├── policy/                 # Policy engine (rules + overrides)
│   └── targets/                # Build target catalog (Gentoo, Buildroot, ...)
├── ui/                         # Frontend (vanilla JS, no build step)
│   ├── app.js                  # Shared shell (WS, toasts, topbar, health)
│   ├── live_dag.html           # Real-time DAG with layered layout
│   ├── replay.html             # Timeline scrubber + autopsy
│   ├── sessions.html           # Build history + tmux output viewer
│   ├── metrics.html            # Live charts + per-node cards
│   ├── cause.html              # Cause graph + blast radius
│   ├── timeline.html           # Per-node event drill-down
│   └── debugger.html           # Step-through execution control
├── cli/                        # CLI tool (35+ subcommands)
│   └── fester.py
├── cockpit/                    # Cockpit module (optional)
├── docs/                       # Documentation + screenshots
├── scripts/                    # Helper scripts (mock agent, syntax checker)
├── pyproject.toml              # Python packaging
├── bootstrap.sh                # One-shot dependency installer
├── run.sh                      # Start the backend
└── config.yaml                 # Cluster configuration
```

## 🚀 Quick Start

```bash
# 1. Clone
git clone https://git.dcos.net/dcosnet/fester.git
cd fester

# 2. Bootstrap (creates venv, installs deps, sets up config)
./bootstrap.sh

# 3. Run
./run.sh

# 4. Open the UI
open http://localhost:8080
```

See **[quickstart.md](quickstart.md)** for the full 5-minute walkthrough.

## 🛠️ Prerequisites

- **Python 3.12+**
- **pip** (for dependency installation)
- **Optional** (features activate automatically when installed):
  - `tmux` — for the tmux action runtime (live output viewing)
  - `mosh` + `ssh` — for the "Shell into node" UI button
  - `qemu-utils` — for QCOW2 workspace snapshots
  - `rsync` — for workspace freeze operations
  - `btrfs-progs` — for CoW reflink snapshots (requires btrfs filesystem)
  - `minio` server — for distributed cache (otherwise falls back to local)

## 📊 Tech Stack

| Layer | Technology |
|-------|------------|
| Backend | Python 3.12, FastAPI 0.128, uvicorn, pydantic 2 |
| Realtime | WebSockets (3 channels: `/ws`, `/ws-targets`, `/ws-debugger`) |
| Storage | SQLite (WAL mode) for builds/sessions/events; MinIO for distributed cache |
| Frontend | Vanilla HTML/CSS/JS (no build step, no framework, ~12KB total) |
| Observability | Prometheus exposition + Grafana dashboard configs |
| CLI | Python argparse (35+ subcommands) |
| Integrations | tmux, mosh, LXC, libvirt, Forgejo, distcc, ccache |

## 🎛️ Configuration

Fester reads from `config.yaml` (or `FESTER_CONFIG` env var) at startup:

```yaml
master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
  - name: x99-v4
    host: 192.168.1.11
    max_jobs: 30

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"
```

### Environment Variables

| Variable | Default | Purpose |
|----------|---------|---------|
| `FESTER_CONFIG` | `config.yaml` | Path to YAML config |
| `FESTER_DB_PATH` | `/var/lib/fester/fester.db` (with fallbacks) | SQLite database location |
| `FESTER_NO_DRIFT` | unset | Set to `1` to disable synthetic drift when agents unreachable |
| `FESTER_AUTOBUILD` | unset | Set to `1` to auto-trigger a build every 60s |
| `FESTER_ROLE_DB` | `/etc/fester/node_roles.json` | Node role overrides |
| `FESTER_CACHE_DIR` | `/var/lib/fester/cache` | Local cache directory |
| `FESTER_STORAGE_CONFIG` | `/etc/fester/storage.json` | Storage layer config |
| `FESTER_API` | `http://localhost:8080` | CLI: backend URL |
| `FESTER_WS` | `ws://localhost:8080/ws` | CLI: WebSocket URL |

## 🔌 API Surface

71 routes total. Key endpoints:

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/api/build` | Kick off a build |
| `GET` | `/api/builds` | Build history |
| `POST` | `/api/builds/{id}/cancel` | Cancel a running build |
| `GET` | `/api/nodes` | List cluster nodes (with live metrics) |
| `POST` | `/api/nodes/{name}/policy` | Set node policy (preferred/avoid/neutral) |
| `POST` | `/api/nodes/{name}/probe` | Manually probe a node's agent |
| `POST` | `/api/nodes/{name}/shell` | Build mosh/ssh shell command |
| `GET` | `/api/metrics/json` | JSON metrics snapshot |
| `GET` | `/metrics` | Prometheus exposition |
| `GET` | `/api/cause/explain/{node}` | Causal chain for a node |
| `GET` | `/api/propagation/{action}` | Blast radius if action fails |
| `POST` | `/replay/start` | Start a replay session |
| `GET` | `/replay/events` | Full event journal |
| `GET` | `/autopsy/{sid}/{action}` | Failure autopsy |
| `GET` | `/api/storage/status` | Storage capabilities |
| `POST` | `/api/storage/freeze` | Freeze workdir into qcow2 |
| `GET` | `/api/actions/active` | List active tmux sessions |
| `WS` | `/ws` | Main event stream |
| `WS` | `/ws-debugger` | Debugger control channel |
| `WS` | `/ws-targets` | Target toggle notifications |

Full route list: `curl http://localhost:8080/openapi.json | jq '.paths | keys[]'`

## 🖥️ CLI

```bash
# Build + watch
fester build --cmd "make -j$(nproc)" --dir /home/user/linux --watch

# List builds
fester builds

# Node management
fester node list
fester node set-policy x99-v3 preferred
fester node probe x99-v3

# Cause analysis
fester cause explain x99-v3 build_kernel
fester blast build_kernel

# Live event stream
fester stream

# Health check
fester health
```

Run `fester --help` for the full list of 35+ subcommands.

## 📚 Documentation

- **[quickstart.md](quickstart.md)** — 5-minute getting started guide
- **[CHEATSHEET.md](CHEATSHEET.md)** — Operator survival guide
- **[CONTRIBUTING.md](CONTRIBUTING.md)** — How to contribute
- **[CHANGELOG.md](CHANGELOG.md)** — Version history
- **[docs/screenshots/](docs/screenshots/)** — UI screenshots

## 🐳 Docker

```bash
docker build -t fester .
docker run -p 8080:8080 -v fester-data:/var/lib/fester fester
```

See **[Dockerfile](Dockerfile)** for details.

## 🔐 Security

Fester is designed to run behind your existing network controls (NAT, firewalls like OPNsense, etc.). There is **no built-in auth** — assume the network is trusted. If you need to expose it publicly, put it behind a reverse proxy with auth (nginx + OAuth2 Proxy, Traefik + Authelia, etc.).

## ⚠️ License

GNU Affero General Public License v3.0 — see **[LICENSE](LICENSE)**.

Software is provided "AS IS", without warranty of any kind. This is a high-concurrency distributed execution system — review the safety notes in the license before production use.