# Fester Operator Cheatsheet

Distributed Build + Scheduler + Observability System

---

## Starting a Build

```bash
fester build ./project.yaml
```

## Key Concepts

**Nodes** — Machines participating in builds (physical, VM, or container).

**Targets** — Compilation environments (x86_64-linux-gnu, aarch64-linux-gnu,
riscv64, embedded/toolchain targets).

**Actions** — Graph nodes representing individual build steps.

**Scheduler** — Chooses the best node based on load, temperature, policy
rules, and historical performance.

## Build Systems View

```text
Cockpit Module: Fester
Live DAG:       /ui/live_dag.html
Replay View:    /ui/replay.html
```

## Debugging

```bash
# Failure Autopsy
GET /api/autopsy/<build_id>

# Timeline Replay
GET /api/replay/<session_id>
```

## Cache System

Supports multiple backends:

- ccache (local compiler cache)
- MinIO (distributed cache)
- Btrfs CoW snapshots (reflink)
- tmpfs (in-memory acceleration)
- Shared CAS API (`/api/cas/`)

## Snapshots

Freeze build state using:

- QCOW2 image snapshots
- Btrfs copy-on-write states

## Node Control

```bash
fester node list
fester node set-policy <node> preferred
fester node set-policy <node> avoid
```

## Scheduler Modes

| Mode             | Description                                     |
|------------------|-------------------------------------------------|
| unified          | Default weighted scoring                        |
| weighted         | Thermal-aware throttling above 85 C             |
| cache-first      | Prioritize nodes with cached artifacts          |
| target-isolated  | Each target arch runs on designated nodes       |
| intelligence     | Experimental ML-driven scheduling               |

## Safety Notes

- Do not run unrestricted builds on production nodes.
- Monitor thermal load in Grafana.
- Ensure cache integrity for distributed builds.
- Validate cross-compile toolchains before enabling targets.
- Use a reverse proxy with authentication for public exposure.

## Design Principle

Every build is reproducible, explainable, and replayable.

## System Maturity

| Level | Capability                                        |
|-------|---------------------------------------------------|
| 1     | Distributed compiler system (distcc-like)         |
| 2     | Smart scheduler (load/thermal/cache aware)        |
| 3     | Observability system (metrics + Grafana + Cockpit)|
| 4     | Causal execution graph (why decisions happen)     |
| 5     | Replayable build brain (session + timeline + autopsy)|