9.0 KiB

Raw Blame History

🚀 Fester Quickstart

5 minutes to your first distributed build.

1. Prerequisites

You need:

Python 3.12+ (python3 --version)
pip (usually ships with Python)
git (to clone the repo)
curl (to test the API)

Optional (features activate automatically when installed):

tmux — live action output viewer
mosh + ssh — "Shell into node" UI button
qemu-utils + rsync — QCOW2 workspace snapshots
btrfs-progs — CoW reflink snapshots (requires btrfs partition)

On Ubuntu/Debian:

sudo apt install python3 python3-pip git curl tmux mosh openssh-client qemu-utils rsync

2. Clone + Bootstrap

git clone https://git.dcos.net/dcosnet/fester.git
cd fester

# One-shot: creates venv, installs Python deps, sets up config
./bootstrap.sh

The bootstrap script will:

Create a Python virtualenv at .venv/
Install all Python dependencies (FastAPI, uvicorn, websockets, minio, aiohttp, etc.)
Create a default config.yaml if one doesn't exist
Initialize the SQLite database at ~/.fester/fester.db
Print next steps

3. Start the Backend

./run.sh

You should see:

🧠 Fester backend starting...
   Config:  config.yaml
   DB:      /home/user/.fester/fester.db
   URL:     http://0.0.0.0:8080
   Press Ctrl+C to stop
INFO:     Uvicorn running on http://0.0.0.0:8080

Verify it's up:

curl http://localhost:8080/api/health
# → {"status":"ok","version":"0.2.0","bus_subscribers":6,...}

4. Open the UI

Open http://localhost:8080 in your browser.

You'll see the Dashboard with:

Cluster panel (left): node list with heat bars, job counters, policy dropdowns, probe buttons
Live Event Stream (center): real-time events with type-filter chips
Quick Actions (right): build form, release form, target toggles, pipeline controls

The topbar shows a live health indicator (version + WS clients + builds count + timeline events).

5. Trigger Your First Build

Via the UI

In the Quick Actions panel (right side of the dashboard):

Build Command: make -j$(nproc) (or any shell command)
Working Directory: /tmp/test-build (or your project dir)
Click ▶ Run Build

Watch the Live Event Stream populate with task_update events as actions execute.

Via the CLI

# Activate the venv first (or use the full path)
source .venv/bin/activate

# Kick off a build
fester build --cmd "make -j4" --dir /tmp/test-build --watch

# The --watch flag streams events as they happen

Via curl

curl -X POST http://localhost:8080/api/build \
  -H "Content-Type: application/json" \
  -d '{"cmd":"make -j4","dir":"/tmp/test-build","target":"linux-gnu"}'

6. Explore the UI

Live DAG (`/ui/live_dag.html`)

Real-time visualization of the build DAG. Each action becomes a node; dependency edges are drawn as arrows. Node colors:

🟡 yellow border = running
🟢 green border = done
🔴 red border = failed
🔵 blue border = cache hit
🟠 yellow outline = critical path

Click any node to inspect its details. Use the search box to filter, and the toggles to show/hide critical-path and failed nodes.

Replay (`/ui/replay.html`)

Scrub through past builds event-by-event:

Click Load Replay to snapshot the current timeline
Use the timeline slider, or ▶ Play / ⏭ Step / ⏮ Back buttons
Jump to the end with ⏭ End to see the full DAG
Click any node to see its state, deps, scheduler decision, and raw event

Sessions (`/ui/sessions.html`)

Build history + active tmux sessions + qcow2 snapshots:

Replay sessions — click "▶ Open in Replay" to scrub through any past session
Recent Builds — cancel running builds, or replay completed ones
Active Action Sessions — view live tmux output of running actions
Storage Snapshots — list of qcow2 freeze images

Metrics (`/ui/metrics.html`)

Live cluster metrics with two trend charts (heat + jobs) and per-node cards showing CPU/memory/heat/jobs/instability bars. Refreshes every 2 seconds.

Cause Graph (`/ui/cause.html`)

Post-hoc reasoning: enter a node name, see its causal chain as a radial graph. The Blast Radius panel computes the downstream impact if any action fails.

Timeline (`/ui/timeline.html`)

Per-node event drill-down: pick a node from the left, see every event that touched it.

Debugger (`/ui/debugger.html`)

Step-through execution control: pause/resume/step a running build. Shows the live tmux timeline and the next action preview.

7. Configure Your Cluster

Edit config.yaml:

master:
  name: fester-master
  role: control

nodes:
  - name: x99-v3
    host: 192.168.1.10
    max_jobs: 24
  - name: x99-v4
    host: 192.168.1.11
    max_jobs: 30
  - name: rpi-1
    host: 192.168.1.20
    max_jobs: 4

projects:
  - name: linux-tool
    repo: https://forgejo.local/linux-tool.git
    targets:
      debian: "make clean && make debian"
      arch: "make clean && make arch"

Restart ./run.sh to pick up changes.

8. Set Node Policies

Via the UI (Dashboard → node row → policy dropdown):

preferred — scheduler gives this node a +20 score bonus
neutral — default
avoid — scheduler subtracts 50 from this node's score

Via the CLI:

fester node set-policy x99-v3 preferred
fester node set-policy rpi-1 avoid

Via the API:

curl -X POST http://localhost:8080/api/nodes/x99-v3/policy \
  -H "Content-Type: application/json" \
  -d '{"policy":"preferred"}'

9. Probe Node Agents (Optional)

If you run a fester-agent on each node (port 8787), Fester will probe it every 4 seconds for real metrics instead of drifting synthetic values.

A mock agent for testing:

python3 scripts/mock_agent.py 8787 &

Manual probe via the UI (Dashboard → node row → "Probe" button) or CLI:

fester node probe x99-v3
fester node probe-all

10. Shell Into a Node

Click the ⌘ Shell button next to any node in the Dashboard. Fester builds the correct mosh or ssh command and copies it to your clipboard:

mosh --ssh "ssh -p 2222 -i ~/.ssh/id_ed25519" build@192.168.1.10

Paste into your terminal to attach.

11. Run a Release Build

Via the UI (Dashboard → Quick Actions → Run Release Build) or CLI:

fester release --repo https://forgejo.local/project.git --project my-proj

This kicks off a longer release-shaped pipeline: fetch_source → configure → build_kernel → build_modules → package → release.

12. Investigate a Failure

When a build fails:

Sessions page → find the failed build → click ▶ Replay
In Replay, scrub to the failed action (red node)
Click the failed node → see the failure reason in the inspector
Switch to Cause Graph page → enter the node name → see the causal chain
Use Blast Radius to see which downstream actions were impacted

Via the CLI:

# Start a replay session
SID=$(curl -sX POST http://localhost:8080/replay/start \
  -H "Content-Type: application/json" \
  -d '{"journal":"latest"}' | jq -r .session_id)

# Run autopsy on the failed action
fester autopsy $SID build_fedora

🎯 Next Steps

Read the CHEATSHEET.md for the operator survival guide
Browse the full API (FastAPI auto-docs)
Set up Prometheus + Grafana using the configs in add-to-prometheus-config.yml and add-to-grafana-config.json
Install tmux, mosh, and qemu-utils to activate the advanced features
Configure MinIO for distributed cache (set the endpoint in backend/cache/minio_cache.py)

🆘 Troubleshooting

Backend won't start

# Check the log
tail -f /tmp/fester_backend.log

# Common issues:
# - "No module named 'fastapi'" → run ./bootstrap.sh again
# - "Permission denied: /var/lib/fester" → use FESTER_DB_PATH=/tmp/fester.db
# - "Address already in use" → another process is on port 8080

UI shows "CONNECTING" forever

The WebSocket can't reach the backend. Check:

curl http://localhost:8080/api/health
# If this fails, the backend isn't running

No events in the Live Event Stream

Trigger a build:

curl -X POST http://localhost:8080/api/build \
  -H "Content-Type: application/json" \
  -d '{"cmd":"demo","dir":"/tmp"}'

Nodes show 0° heat / 0 jobs

No agent is running on the nodes. Either:

Start a mock agent: python3 scripts/mock_agent.py 8787
Or accept synthetic drift (default behavior when agents are unreachable)

Build fails immediately

Check the build details:

fester builds
fester build-info <build_id>

Common causes:

Working directory doesn't exist → create it first
Command not found → check PATH
Permission denied → check file permissions

📚 More Info

README.md — full project overview
CHEATSHEET.md — operator cheatsheet
CONTRIBUTING.md — how to contribute
API docs — interactive OpenAPI spec

9.0 KiB Raw Blame History