fester/quickstart.md

# 🚀 Fester Quickstart

**5 minutes to your first distributed build.**

---

## 1. Prerequisites

You need:
- **Python 3.12+** (`python3 --version`)
- **pip** (usually ships with Python)
- **git** (to clone the repo)
- **curl** (to test the API)

Optional (features activate automatically when installed):
- `tmux` — live action output viewer
- `mosh` + `ssh` — "Shell into node" UI button
- `qemu-utils` + `rsync` — QCOW2 workspace snapshots
- `btrfs-progs` — CoW reflink snapshots (requires btrfs partition)

On Ubuntu/Debian:
```bash
sudo apt install python3 python3-pip git curl tmux mosh openssh-client qemu-utils rsync
```

## 2. Clone + Bootstrap

```bash
git clone https://git.dcos.net/dcosnet/fester.git
cd fester

# One-shot: creates venv, installs Python deps, sets up config
./bootstrap.sh
```

The bootstrap script will:
1. Create a Python virtualenv at `.venv/`
2. Install all Python dependencies (FastAPI, uvicorn, websockets, minio, aiohttp, etc.)
3. Create a default `config.yaml` if one doesn't exist
4. Initialize the SQLite database at `~/.fester/fester.db`
5. Print next steps

## 3. Start the Backend

```bash
./run.sh
```

You should see:
```
🧠 Fester backend starting...
   Config:  config.yaml
   DB:      /home/user/.fester/fester.db
   URL:     http://0.0.0.0:8080
   Press Ctrl+C to stop
INFO:     Uvicorn running on http://0.0.0.0:8080
```

Verify it's up:
```bash
curl http://localhost:8080/api/health
# → {"status":"ok","version":"0.2.0","bus_subscribers":6,...}
```

## 4. Open the UI

Open **http://localhost:8080** in your browser.

You'll see the **Dashboard** with:
- **Cluster panel** (left): node list with heat bars, job counters, policy dropdowns, probe buttons
- **Live Event Stream** (center): real-time events with type-filter chips
- **Quick Actions** (right): build form, release form, target toggles, pipeline controls

The topbar shows a live health indicator (version + WS clients + builds count + timeline events).

## 5. Trigger Your First Build

### Via the UI

In the **Quick Actions** panel (right side of the dashboard):
1. **Build Command**: `make -j$(nproc)` (or any shell command)
2. **Working Directory**: `/tmp/test-build` (or your project dir)
3. Click **▶ Run Build**

Watch the **Live Event Stream** populate with `task_update` events as actions execute.

### Via the CLI

```bash
# Activate the venv first (or use the full path)
source .venv/bin/activate

# Kick off a build
fester build --cmd "make -j4" --dir /tmp/test-build --watch

# The --watch flag streams events as they happen
```

### Via curl

```bash
curl -X POST http://localhost:8080/api/build \
  -H "Content-Type: application/json" \
  -d '{"cmd":"make -j4","dir":"/tmp/test-build","target":"linux-gnu"}'
```

## 6. Explore the UI

### Live DAG (`/ui/live_dag.html`)

Real-time visualization of the build DAG. Each action becomes a node; dependency edges are drawn as arrows. Node colors:
- 🟡 **yellow border** = running
- 🟢 **green border** = done
- 🔴 **red border** = failed
- 🔵 **blue border** = cache hit
- 🟠 **yellow outline** = critical path

Click any node to inspect its details. Use the search box to filter, and the toggles to show/hide critical-path and failed nodes.

### Replay (`/ui/replay.html`)

Scrub through past builds event-by-event:
1. Click **Load Replay** to snapshot the current timeline
2. Use the timeline slider, or **▶ Play** / **⏭ Step** / **⏮ Back** buttons
3. Jump to the end with **⏭ End** to see the full DAG
4. Click any node to see its state, deps, scheduler decision, and raw event

### Sessions (`/ui/sessions.html`)

Build history + active tmux sessions + qcow2 snapshots:
- **Replay sessions** — click "▶ Open in Replay" to scrub through any past session
- **Recent Builds** — cancel running builds, or replay completed ones
- **Active Action Sessions** — view live tmux output of running actions
- **Storage Snapshots** — list of qcow2 freeze images

### Metrics (`/ui/metrics.html`)

Live cluster metrics with two trend charts (heat + jobs) and per-node cards showing CPU/memory/heat/jobs/instability bars. Refreshes every 2 seconds.

### Cause Graph (`/ui/cause.html`)

Post-hoc reasoning: enter a node name, see its causal chain as a radial graph. The **Blast Radius** panel computes the downstream impact if any action fails.

### Timeline (`/ui/timeline.html`)

Per-node event drill-down: pick a node from the left, see every event that touched it.

### Debugger (`/ui/debugger.html`)

Step-through execution control: pause/resume/step a running build. Shows the live tmux timeline and the next action preview.

## 7. Configure Your Cluster

Edit `config.yaml`:

```yaml
master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
  - name: x99-v4
    host: 192.168.1.11
    max_jobs: 30
  - name: rpi-1
    host: 192.168.1.20
    max_jobs: 4

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"
```

Restart `./run.sh` to pick up changes.

## 8. Set Node Policies

Via the UI (Dashboard → node row → policy dropdown):
- **preferred** — scheduler gives this node a +20 score bonus
- **neutral** — default
- **avoid** — scheduler subtracts 50 from this node's score

Via the CLI:
```bash
fester node set-policy x99-v3 preferred
fester node set-policy rpi-1 avoid
```

Via the API:
```bash
curl -X POST http://localhost:8080/api/nodes/x99-v3/policy \
  -H "Content-Type: application/json" \
  -d '{"policy":"preferred"}'
```

## 9. Probe Node Agents (Optional)

If you run a fester-agent on each node (port 8787), Fester will probe it every 4 seconds for real metrics instead of drifting synthetic values.

A mock agent for testing:
```bash
python3 scripts/mock_agent.py 8787 &
```

Manual probe via the UI (Dashboard → node row → "Probe" button) or CLI:
```bash
fester node probe x99-v3
fester node probe-all
```

## 10. Shell Into a Node

Click the **⌘ Shell** button next to any node in the Dashboard. Fester builds the correct mosh or ssh command and copies it to your clipboard:

```bash
mosh --ssh "ssh -p 2222 -i ~/.ssh/id_ed25519" build@192.168.1.10
```

Paste into your terminal to attach.

## 11. Run a Release Build

Via the UI (Dashboard → Quick Actions → Run Release Build) or CLI:
```bash
fester release --repo https://forgejo.local/project.git --project my-proj
```

This kicks off a longer release-shaped pipeline: `fetch_source → configure → build_kernel → build_modules → package → release`.

## 12. Investigate a Failure

When a build fails:

1. **Sessions page** → find the failed build → click **▶ Replay**
2. In Replay, scrub to the failed action (red node)
3. Click the failed node → see the failure reason in the inspector
4. Switch to **Cause Graph** page → enter the node name → see the causal chain
5. Use **Blast Radius** to see which downstream actions were impacted

Via the CLI:
```bash
# Start a replay session
SID=$(curl -sX POST http://localhost:8080/replay/start \
  -H "Content-Type: application/json" \
  -d '{"journal":"latest"}' | jq -r .session_id)

# Run autopsy on the failed action
fester autopsy $SID build_fedora
```

## 🎯 Next Steps

- Read the **[CHEATSHEET.md](CHEATSHEET.md)** for the operator survival guide
- Browse the **[full API](http://localhost:8080/docs)** (FastAPI auto-docs)
- Set up **Prometheus + Grafana** using the configs in `add-to-prometheus-config.yml` and `add-to-grafana-config.json`
- Install **tmux**, **mosh**, and **qemu-utils** to activate the advanced features
- Configure **MinIO** for distributed cache (set the endpoint in `backend/cache/minio_cache.py`)

## 🆘 Troubleshooting

### Backend won't start

```bash
# Check the log
tail -f /tmp/fester_backend.log

# Common issues:
# - "No module named 'fastapi'" → run ./bootstrap.sh again
# - "Permission denied: /var/lib/fester" → use FESTER_DB_PATH=/tmp/fester.db
# - "Address already in use" → another process is on port 8080
```

### UI shows "CONNECTING" forever

The WebSocket can't reach the backend. Check:
```bash
curl http://localhost:8080/api/health
# If this fails, the backend isn't running
```

### No events in the Live Event Stream

Trigger a build:
```bash
curl -X POST http://localhost:8080/api/build \
  -H "Content-Type: application/json" \
  -d '{"cmd":"demo","dir":"/tmp"}'
```

### Nodes show 0° heat / 0 jobs

No agent is running on the nodes. Either:
- Start a mock agent: `python3 scripts/mock_agent.py 8787`
- Or accept synthetic drift (default behavior when agents are unreachable)

### Build fails immediately

Check the build details:
```bash
fester builds
fester build-info <build_id>
```

Common causes:
- Working directory doesn't exist → create it first
- Command not found → check PATH
- Permission denied → check file permissions

## 📚 More Info

- **[README.md](README.md)** — full project overview
- **[CHEATSHEET.md](CHEATSHEET.md)** — operator cheatsheet
- **[CONTRIBUTING.md](CONTRIBUTING.md)** — how to contribute
- **[API docs](http://localhost:8080/docs)** — interactive OpenAPI spec