commit 0a9cb8fd97761b2b76b238fff0327a6b6f6b8ea8 Author: Jeremy Anderson Date: Sun Jun 28 12:21:06 2026 -0400 Fester: a distributed build orchestrator with thermal-aware scheduling, live DAG, event replay, and failure analysis diff --git a/.github/funding.yml b/.github/funding.yml new file mode 100644 index 0000000..e09faad --- /dev/null +++ b/.github/funding.yml @@ -0,0 +1,64 @@ +# .github/funding.yml +# ============================================================ +# GitHub Sponsors "Sponsor this project" button configuration. +# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/displaying-a-sponsor-button-in-your-repository +# +# Each key below corresponds to a sponsored-developer account or platform +# that will appear as a "Sponsor" button on the repo's main page. +# +# Replace the placeholder values with your actual usernames/addresses. +# Comment out or delete any platforms you don't want to use. +# ============================================================ + +# --- GitHub Sponsors --- +# Your GitHub Sponsors username (the part after github.com/sponsors/) +github: REPLACE_WITH_YOUR_GITHUB_USERNAME + +# --- Patreon --- +# Your Patreon username (the part after patreon.com/) +patreon: REPLACE_WITH_YOUR_PATREON_USERNAME + +# --- Open Collective --- +# Your Open Collective slug (the part after opencollective.com/) +open_collective: REPLACE_WITH_YOUR_OPENCOLLECTIVE_SLUG + +# --- Ko-fi --- +# Your Ko-fi username (the part after ko-fi.com/) +ko_fi: REPLACE_WITH_YOUR_KOFI_USERNAME + +# --- Tidelift --- +# Your Tidelift package name (e.g. "npm/foo" or "pypi/bar") +tidelift: REPLACE_WITH_YOUR_TIDELIFT_PACKAGE + +# --- CommunityBridge --- +# Your CommunityBridge project name +community_bridge: REPLACE_WITH_YOUR_COMMUNITYBRIDGE_PROJECT + +# --- Liberapay --- +# Your Liberapay username (the part after liberapay.com/) +liberapay: REPLACE_WITH_YOUR_LIBERAPAY_USERNAME + +# --- IssueHunt --- +# Your IssueHunt username +issuehunt: REPLACE_WITH_YOUR_ISSUEHUNT_USERNAME + +# --- LFX Crowdfunding --- +# Your LFX project name +lfx_crowdfunding: REPLACE_WITH_YOUR_LFX_PROJECT + +# --- Polar --- +# Your Polar repository (e.g. "owner/repo") +polar: REPLACE_WITH_YOUR_GITHUB_ORG/fester + +# --- Custom platforms / direct links --- +# A list of up to 4 custom {label, url} entries that appear as buttons. +# Use these for crypto donation links, your own donate page, etc. +custom: + - label: "Donate" + url: "https://git.dcos.net/dcosnet/fester/src/branch/main/donate.md" + # - label: "Bitcoin" + # url: "https://your-blockchain-explorer/address/YOUR_BTC_ADDRESS" + # - label: "Monero" + # url: "https://your-xmr-explorer/address/YOUR_XMR_ADDRESS" + # - label: "PayPal" + # url: "https://www.paypal.com/paypalme/YOUR_PAYPAL_ME" diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..77aa5d6 --- /dev/null +++ b/.gitignore @@ -0,0 +1,139 @@ +# ============================================================ +# Fester .gitignore +# ============================================================ + +# ---- Python ---- +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# ---- Virtual environments ---- +.venv/ +venv/ +env/ +ENV/ +env.bak/ +venv.bak/ +.python-version + +# ---- Testing / coverage ---- +htmlcov/ +.tox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +.hypothesis/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ + +# ---- Fester runtime data ---- +*.db +*.db-journal +*.db-wal +*.db-shm +*.sqlite +*.sqlite-journal +*.sqlite-wal +*.sqlite-shm + +# Fester storage dirs (local dev) +.fester/ +fester-data/ +/var/lib/fester/ +snapshots/ +cas/ +cache/ +*.qcow2 +*.tar.gz + +# Config with secrets (keep config.yaml.example in git, ignore actual config) +config.local.yaml +config.runtime.yaml +storage.json +node_roles.json +.env +.env.local +.env.*.local + +# ---- Logs ---- +*.log +logs/ +fester.log +fester_backend.log +mock_agent.log + +# ---- IDE / Editor ---- +.idea/ +.vscode/ +*.swp +*.swo +*~ +.project +.classpath +.settings/ +*.sublime-project +*.sublime-workspace +.atom/ +.brackets.json + +# ---- OS ---- +.DS_Store +.DS_Store? +._* +.Spotlight-V100 +.Trashes +ehthumbs.db +Thumbs.db +desktop.ini + +# ---- Docker ---- +docker-compose.override.yml + +# ---- Node (if you ever add a JS build step) ---- +node_modules/ +npm-debug.log* +yarn-debug.log* +yarn-error.log* +pnpm-debug.log* + +# ---- Build artifacts ---- +*.o +*.a +*.so +*.dylib +build.tar.gz +build/ + +# ---- Screenshots (keep in git, but ignore ad-hoc captures) ---- +# docs/screenshots/ is tracked — these are ad-hoc: +/tmp/ +*.tmp + +# ---- Misc ---- +*.bak +*.orig +*.rej +*.diff +*.patch diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..398d331 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,134 @@ +# Changelog + +All notable changes to Fester are documented here. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- Documentation: README.md, quickstart.md, CONTRIBUTING.md, CHANGELOG.md +- Tooling: bootstrap.sh, run.sh, pyproject.toml, .gitignore +- Docker support (Dockerfile) +- Screenshots for all 8 UI pages + +## [0.3.0] — 2026-06-28 + +### Added — Storage layer +- SQLite-backed persistence for builds, sessions, events, and node states +- Btrfs CoW reflink snapshots (`backend/storage/btrfs_cas.py`) +- QCOW2 workspace freezing with real mount + rsync (`backend/storage/qcow2_freeze.py`) +- tmpfs workspace acceleration +- Storage status + config + freeze + reflink + snapshots endpoints + +### Added — tmux integration (E9) +- TmuxManager with session create/list/inspect/capture-output/kill +- `runtime: "tmux"` action runtime — runs actions in detached tmux sessions +- Live output viewer in Sessions page UI +- `/api/actions/active`, `/api/actions/{name}/tmux/output`, `/tmux/kill` endpoints + +### Added — mosh/ssh shell attach (E10) +- MoshManager with command builder (mosh-via-SSH-bootstrap) +- `/api/nodes/{name}/shell` GET + POST endpoints +- "⌘ Shell" button on Dashboard node rows (copies command to clipboard) + +### Added — Real engine wiring +- BuildRunner wraps PipelineEngine with build lifecycle + persistence +- Real DAG execution with proper cwd + env passing +- Failure propagation: skipped actions on failed deps +- Critical path computation + tagging in events +- Debugger pause/resume/step actually controls running engines + +### Added — Cause graph + blast radius +- `/api/propagation/{action}` — forward blast radius computation +- `/api/cause/explain/{node}` — causal chain for a node +- Blast Radius panel in Cause Graph UI page +- Failure autopsy with proper session ID correlation + +### Added — Storage + target catalog +- `/api/targets` reads from `backend/targets/catalog.py` (6 systems: Gentoo, Buildroot, OpenWrt, ALFS, SourceMage, Lunar) +- `/api/targets/arches` + `/api/targets/runtimes` endpoints + +### Added — Health + metrics +- `/api/health` endpoint (version, bus_subscribers, ws_clients, builds_known, timeline_events) +- Live health indicator in topbar (all pages) +- Prometheus metrics aligned with grafana config names (`fester_node_cpu`, `fester_pipeline_actions_total`, etc.) +- Pipeline action counters + cache hit counters wired via bus subscriber + +### Added — UI enhancements +- New Timeline page (per-node event drill-down) +- Node policy dropdowns (preferred/avoid/neutral) on Dashboard +- Per-node "Probe" buttons + "Probe All" button +- Build cancel + replay buttons on Sessions page +- Event filter chips on Dashboard (toggle by event type) + +### Added — CLI overhaul +- 35+ subcommands (was 11) +- Nested subcommands: `node {list,set-policy,probe,probe-all}`, `cause {explain,trace,events}`, `replay {start,step,reset}`, `debugger {state,pause,resume,step}` +- New: `build`, `builds`, `build-info`, `cancel`, `release`, `targets`, `toggle-target`, `blast`, `timeline`, `rewind`, `autopsy`, `policy-list`, `health` +- `--watch` flag on `build` command (streams events via WebSocket) + +### Fixed — Critical backend bugs +- `api/api.py` order bug (`bus.subscribe()` before `bus = EventBus()`) +- 7 broken `_endpoint` symbol imports +- `cause_graph.attach_bus` + `emit_debug` defined at module scope (zero indentation) +- Three missing files: `timeline_store.py`, `failure_autopsy.py`, `failure_propagation.py` +- Two Python module/package name collisions (`scheduler.py` vs `scheduler/`, `nodes.py` vs `nodes/`, `cache.py` vs `cache/`) +- `execute_action` passing empty `{}` as cwd → `TypeError` +- Build env never reaching subprocess +- Debugger/pipeline-control `_ENGINE` always `None` +- `failure_propagation` looking for `event.data.deps` (legacy shape) +- Autopsy session ID mismatch +- `/api/release` was a pure stub +- `/api/policy/set` didn't persist +- `/ws-debugger` returned canned data, ignored commands +- `forgejo.on_push_event` wrong PipelineEngine arity +- `pipeline/feedback.py` PolicyEngine() with no args + +### Fixed — Standardization +- ~20 bare imports converted to `backend.*` prefix +- Two parallel event taxonomies unified (`types.py` re-exports from `schema.py`) +- `MinioCache.__init__` made lazy (was crashing on import if MinIO unreachable) +- `cache.py` graceful fallback for non-writable paths +- `roles_store.py` graceful fallback for non-writable `/etc/fester/` +- `FesterEvent` made `node`/`action`/`state` truly optional (had `Optional[str]` type but no default) + +## [0.2.0] — 2026-06-27 + +### Added — Real FastAPI backend +- `backend/main.py` — FastAPI app with 50+ routes +- All routers wired: replay, autopsy, timeline, debugger, pipeline_control, nodes, metrics, cause +- WebSocket hub with `/ws`, `/ws-targets`, `/ws-debugger` +- Ambient loop for node probing + drift +- SQLite storage layer (`backend/storage/sqlite_db.py`) +- Node probe module (`backend/nodes/probe.py`) +- BuildRunner (`backend/pipeline/runner.py`) + +### Added — UI overhaul (7 pages) +- Dashboard (`index.html`) — 3-column layout with cluster stats, live stream, quick actions +- Live DAG (`ui/live_dag.html`) — layered Sugiyama-lite layout, SVG edges, click-to-inspect +- Replay (`ui/replay.html`) — timeline scrubber, play/pause/step, structured inspector +- Sessions (`ui/sessions.html`) — build history + replay links +- Metrics (`ui/metrics.html`) — live charts + per-node cards +- Cause Graph (`ui/cause.html`) — radial graph visualization +- Debugger (`ui/debugger.html`) — step-through execution control + +### Added — Shared infrastructure +- `style.css` — full design system (dark theme tokens, panels, badges, buttons, scrollbars) +- `ui/app.js` — shared shell (WS auto-reconnect, FesterTargets global, toasts, topbar nav) +- `cockpit/fester-module/targets.js` — fixed to expose `FesterTargets` global + +### Fixed +- `live_dag.html` was broken (referenced undefined `FesterTargets`) +- `style.css` was 0 bytes +- `index.html` was a 30-line throwaway + +## [0.1.0] — Initial commit + +### Added +- Basic project structure +- Backend modules (many stubbed) +- Cockpit module +- Install scripts +- CHEATSHEET.md diff --git a/CHEATSHEET.md b/CHEATSHEET.md new file mode 100644 index 0000000..8c14ec2 --- /dev/null +++ b/CHEATSHEET.md @@ -0,0 +1,153 @@ +# Fester Cluster Build System — Cheatsheet + +## 🧠 Core Concept + +A distributed, DAG-driven build execution system with: + +- real-time scheduling +- node-aware load balancing +- thermal + policy constraints +- cache-aware execution +- deterministic replay/debugging + +--- + +## 🧩 System Components + +### 1. Scheduler +Chooses best node per action using: + +- CPU load +- temperature +- policy rules +- historical intelligence feedback + +--- + +### 2. Pipeline Engine +Executes DAG actions sequentially or step-debugged. + +Supports: +- live execution +- interactive stepping +- replay mode + +--- + +### 3. Timeline Store +Immutable event log of entire system. + +Used for: +- replay +- debugging +- autopsy analysis + +--- + +### 4. Failure System +Detects and propagates failure: + +- backward → root cause candidates +- forward → downstream impact + +--- + +### 5. Critical Path Analyzer +Identifies bottleneck chain in DAG execution. + +--- + +### 6. Cache Layer +Supports: +- distributed artifact reuse +- MinIO backend +- future: Btrfs/QCOW2 snapshot acceleration + +--- + +### 7. Debugger Mode +Interactive execution control: + +- pause +- step +- resume +- scheduler preview before execution + +--- + +## 🧭 Execution Flow + +1. Build graph generated from spec +2. Scheduler selects node per action +3. Timeline records decision +4. Executor runs action on node +5. Events streamed to UI +6. Failures propagate through DAG +7. Optional debugger intercepts execution + +--- + +## 📡 Event Types + +- `node` +- `pipeline` +- `schedule_decision` +- `failure_propagation` +- `debugger_preview` + +--- + +## 🖥 UI Modes + +### Live Mode +- real-time DAG rendering +- node state updates + +### Debug Mode +- step execution +- scheduler inspection +- manual control + +### Autopsy Mode +- failure root cause analysis +- dependency tracing +- critical path overlay + +--- + +## ⚙️ Node Selection Model + +Score-based weighted system: + +- CPU availability +- thermal headroom +- policy constraints +- historical instability penalties + +--- + +## 🧪 Execution Backends (planned) + +- distcc +- LXC +- libvirt +- native execution +- cross-compilation toolchains + +--- + +## 📦 Cache Strategy + +- local ccache +- shared MinIO cache +- future: Btrfs/QCOW2 snapshot acceleration + +--- + +## 🔐 Design Principle + +> The system must always be replayable, explainable, and deterministic. + +No hidden state. +No opaque scheduling. +Everything is observable. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..2970579 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,145 @@ +# Contributing to Fester + +Thanks for your interest in improving Fester! This document covers the basics. + +## 🚀 Quick Start for Contributors + +```bash +# 1. Fork + clone +git clone https://git.dcos.net/dcosnet/fester.git +cd fester + +# 2. Bootstrap dev environment (creates venv, installs deps + dev tools) +./bootstrap.sh --dev + +# 3. Run the backend in dev mode (auto-reload on changes) +./run.sh --dev + +# 4. Run tests (when we have them) +pytest + +# 5. Lint + format +ruff check . +ruff format . +``` + +## 🏗️ Architecture Overview + +Read **[README.md](README.md)** first — it has the full architecture diagram and project layout. + +The key insight: Fester has a strict separation between: +- **EventBus** — the singleton message bus (all events flow through it) +- **Subscribers** — bus listeners that index/journal/broadcast events (TimelineStore, cause_graph, FailurePropagator, the WS broadcaster) +- **Routers** — FastAPI APIRouters under `backend/api/` that expose REST endpoints +- **UI** — vanilla HTML/JS pages under `ui/` that subscribe to the WS stream and call REST endpoints + +When you add a new feature, ask: +1. Does this emit events? → Use `bus.emit(EventType.X, ...)` from `backend/events/schema.py` +2. Does this need persistence? → Add a method to `backend/storage/sqlite_db.py:Storage` +3. Does this need a UI? → Add a page under `ui/` and a nav entry in `ui/app.js:Fester.mountShell()` +4. Does this need a CLI command? → Add a subcommand to `cli/fester.py` + +## 📝 Coding Standards + +### Python + +- **Python 3.12+** — use modern syntax (match statements, type hints, etc.) +- **Type hints** — add them to new code; don't worry about retrofitting old code +- **Imports** — use `from backend.X.Y import Z` (not bare `from X.Y import Z`) +- **EventBus** — always emit events via `bus.emit()`, never mutate state directly from a request handler +- **Pydantic** — use `BaseModel` for request bodies in routers +- **Async** — use `async def` for endpoints that touch the DB or do I/O + +### JavaScript (UI) + +- **Vanilla JS** — no frameworks (React, Vue, etc.). Keep it dependency-free. +- **Shared shell** — every page calls `Fester.mountShell('page-id')` to get the topbar + WS connection +- **API calls** — use `Fester.api(path, opts)` (returns parsed JSON) +- **Toasts** — use `Fester.toast(msg, kind)` for user notifications +- **No build step** — pages load directly via ` + + + + + + +
+
+
+

Debugger

+
+ + + +
+
+

+    
+
+ + + + diff --git a/config.test.yaml b/config.test.yaml new file mode 100644 index 0000000..6ab2114 --- /dev/null +++ b/config.test.yaml @@ -0,0 +1,20 @@ +master: + name: fester-master + role: control + +nodes: + - name: x99-v3 + host: 127.0.0.1 + max_jobs: 24 + + - name: x99-v4 + host: 127.0.0.1 + max_jobs: 30 + +projects: + - name: linux-tool + repo: https://forgejo.local/linux-tool.git + targets: + debian: "make clean && make debian" + arch: "make clean && make arch" + fedora: "make clean && make fedora" diff --git a/config.yaml b/config.yaml new file mode 100644 index 0000000..5e87db2 --- /dev/null +++ b/config.yaml @@ -0,0 +1,23 @@ +master: + name: fester-master + role: control + +nodes: + - name: x99-v3 + host: 192.168.1.10 + max_jobs: 24 + + - name: x99-v4 + host: 192.168.1.11 + max_jobs: 30 + + +projects: + + - name: linux-tool + repo: https://forgejo.local/linux-tool.git + + targets: + debian: "make clean && make debian" + arch: "make clean && make arch" + fedora: "make clean && make fedora" diff --git a/debugger/interactive_engine.py b/debugger/interactive_engine.py new file mode 100644 index 0000000..cf0195f --- /dev/null +++ b/debugger/interactive_engine.py @@ -0,0 +1,124 @@ +# debugger/interactive_engine.py + +import asyncio + +class InteractiveDebuggerEngine: + + def __init__(self, nodes, graph, scheduler, timeline, ws): + + self.nodes = nodes + self.graph = graph + self.scheduler = scheduler + self.timeline = timeline + self.ws = ws + + self.paused = True + self.step_mode = True + self.current_index = 0 + self.actions = [] + + # ----------------------------- + # LOAD EXECUTION PLAN + # ----------------------------- + def load_plan(self, actions): + self.actions = actions + self.current_index = 0 + + # ----------------------------- + # CONTROL + # ----------------------------- + def pause(self): + self.paused = True + + def resume(self): + self.paused = False + + def step(self): + self.paused = False + + # ----------------------------- + # MAIN EXECUTION LOOP + # ----------------------------- + async def run(self): + + while self.current_index < len(self.actions): + + action = self.actions[self.current_index] + + # ----------------------------- + # SCHEDULER PREVIEW (NO EXECUTION YET) + # ----------------------------- + decision = self.scheduler.choose_best_node( + self.nodes, + action + ) + + preview_event = { + "type": "debugger_preview", + "data": { + "action": action["name"], + "selected_node": decision["best_node"]["name"], + "alternatives": decision["explanation"]["top_candidates"], + "state": "awaiting_step" + } + } + + await self.ws.broadcast(preview_event) + self.timeline.record(preview_event) + + # ----------------------------- + # WAIT FOR USER STEP + # ----------------------------- + while self.paused: + await asyncio.sleep(0.05) + + # auto-pause again after step + self.paused = True + + # ----------------------------- + # EXECUTE ACTION + # ----------------------------- + node = decision["best_node"] + + exec_event = { + "type": "node", + "data": { + "action": action["name"], + "node": node["name"], + "state": "running", + "mode": "debug_step" + } + } + + await self.ws.broadcast(exec_event) + self.timeline.record(exec_event) + + # simulate execution hook (replace with real executor) + rc = await self.execute(action, node) + + final_state = "done" if rc == 0 else "failed" + + final_event = { + "type": "node", + "data": { + "action": action["name"], + "node": node["name"], + "state": final_state, + "mode": "debug_step" + } + } + + await self.ws.broadcast(final_event) + self.timeline.record(final_event) + + self.current_index += 1 + + # ----------------------------- + # EXECUTION HOOK + # ----------------------------- + async def execute(self, action, node): + """ + Replace with distcc / lxc / libvirt execution layer + """ + await asyncio.sleep(0.2) + return 0 diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..9ce2b39 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,87 @@ +# docker-compose.yml for Fester +# ============================================================ +# Quick start: +# docker-compose up -d +# +# Then open http://localhost:8080 +# +# Volumes: +# fester-db — SQLite database (builds, sessions, events) +# fester-storage — snapshots, cache, CAS +# fester-config — config.yaml, storage.json, node_roles.json +# ============================================================ + +version: "3.8" + +services: + fester: + build: . + image: fester:latest + container_name: fester + restart: unless-stopped + ports: + - "8080:8080" + volumes: + - fester-db:/var/lib/fester + - fester-storage:/var/lib/fester/storage + - fester-config:/etc/fester + - ./config.yaml:/app/config.yaml:ro + environment: + - FESTER_DB_PATH=/var/lib/fester/fester.db + - FESTER_CONFIG=/app/config.yaml + - FESTER_NO_DRIFT=0 + - FESTER_AUTOBUILD=0 + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8080/api/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + + # Optional: MinIO for distributed cache + # Uncomment to enable + # minio: + # image: minio/minio:latest + # container_name: fester-minio + # restart: unless-stopped + # ports: + # - "9000:9000" + # - "9001:9001" + # volumes: + # - minio-data:/data + # environment: + # - MINIO_ROOT_USER=minioadmin + # - MINIO_ROOT_PASSWORD=minioadmin + # command: server /data --console-address ":9001" + + # Optional: Prometheus scraper + # prometheus: + # image: prom/prometheus:latest + # container_name: fester-prometheus + # restart: unless-stopped + # ports: + # - "9090:9090" + # volumes: + # - ./add-to-prometheus-config.yml:/etc/prometheus/prometheus.yml:ro + # - prometheus-data:/prometheus + + # Optional: Grafana dashboard + # grafana: + # image: grafana/grafana:latest + # container_name: fester-grafana + # restart: unless-stopped + # ports: + # - "3000:3000" + # volumes: + # - grafana-data:/var/lib/grafana + # - ./add-to-grafana-config.json:/etc/grafana/provisioning/dashboards/fester.json:ro + # environment: + # - GF_SECURITY_ADMIN_PASSWORD=admin + +volumes: + fester-db: + fester-storage: + fester-config: + # minio-data: + # prometheus-data: + # grafana-data: diff --git a/docs/screenshots/01_dashboard.png b/docs/screenshots/01_dashboard.png new file mode 100644 index 0000000..d1a6f4f Binary files /dev/null and b/docs/screenshots/01_dashboard.png differ diff --git a/docs/screenshots/02_live_dag.png b/docs/screenshots/02_live_dag.png new file mode 100644 index 0000000..eb107ef Binary files /dev/null and b/docs/screenshots/02_live_dag.png differ diff --git a/docs/screenshots/03_replay.png b/docs/screenshots/03_replay.png new file mode 100644 index 0000000..bf9513a Binary files /dev/null and b/docs/screenshots/03_replay.png differ diff --git a/docs/screenshots/04_sessions.png b/docs/screenshots/04_sessions.png new file mode 100644 index 0000000..d9bd107 Binary files /dev/null and b/docs/screenshots/04_sessions.png differ diff --git a/docs/screenshots/05_metrics.png b/docs/screenshots/05_metrics.png new file mode 100644 index 0000000..6935478 Binary files /dev/null and b/docs/screenshots/05_metrics.png differ diff --git a/docs/screenshots/06_cause.png b/docs/screenshots/06_cause.png new file mode 100644 index 0000000..9292c82 Binary files /dev/null and b/docs/screenshots/06_cause.png differ diff --git a/docs/screenshots/07_timeline.png b/docs/screenshots/07_timeline.png new file mode 100644 index 0000000..ec05bb2 Binary files /dev/null and b/docs/screenshots/07_timeline.png differ diff --git a/docs/screenshots/08_debugger.png b/docs/screenshots/08_debugger.png new file mode 100644 index 0000000..000445f Binary files /dev/null and b/docs/screenshots/08_debugger.png differ diff --git a/donate.md b/donate.md new file mode 100644 index 0000000..5aa870b --- /dev/null +++ b/donate.md @@ -0,0 +1,101 @@ +# 💚 Support Fester + +Fester is free, open-source software released under the AGPL-3.0. It's built and maintained by contributors on their own time and hardware. + +If Fester has saved you time, helped you ship a build, or just made your cluster more observable — **thank you for considering a donation**. Every contribution helps cover: + +- ⚡ **Build cluster electricity** (the test cluster runs 24/7) +- 🖥️ **Hardware replacements** (fans, SSDs, thermal paste — builds are hard on hardware) +- 🌐 **Domain + git hosting** (git.dcos.net, forgejo instances) +- ☕ **Caffeine** (the fuel that keeps DAGs flowing) + +--- + +## 💰 Ways to Donate + +### Cryptocurrency (preferred — no platform fees) + +| Currency | Network | Address | +|----------|---------|---------| +| **Bitcoin** | BTC (mainnet) | `REPLACE_WITH_YOUR_BTC_ADDRESS` | +| **Bitcoin** | Lightning | `REPLACE_WITH_YOUR_LIGHTNING_ADDRESS_OR_LNURL` | +| **Monero** | XMR (mainnet) | `REPLACE_WITH_YOUR_XMR_ADDRESS` | +| **Ethereum** | ERC-20 | `REPLACE_WITH_YOUR_ETH_ADDRESS` | +| **USDC** | ERC-20 | `REPLACE_WITH_YOUR_USDC_ADDRESS` | +| **USDC** | Polygon | `REPLACE_WITH_YOUR_USDC_POLYGON_ADDRESS` | +| **SOL** | Solana | `REPLACE_WITH_YOUR_SOL_ADDRESS` | + +### Fiat / Card + +| Platform | Link | +|----------|------| +| **GitHub Sponsors** | `REPLACE_WITH_YOUR_GITHUB_SPONSORS_URL` | +| **Ko-fi** | `REPLACE_WITH_YOUR_KOFI_URL` | +| **Liberapay** | `REPLACE_WITH_YOUR_LIBERAPAY_URL` | +| **Open Collective** | `REPLACE_WITH_YOUR_OPENCOLLECTIVE_URL` | +| **Patreon** | `REPLACE_WITH_YOUR_PATREON_URL` | + +### Other + +| Method | Details | +|--------|---------| +| **PayPal** | `REPLACE_WITH_YOUR_PAYPAL_EMAIL_OR_ME_LINK` | +| **Bank transfer (SEPA)** | `REPLACE_WITH_YOUR_SEPA_IBAN` (or email for details) | +| **Hardware donations** | Email `REPLACE_WITH_YOUR_CONTACT_EMAIL` — we always need more nodes! | + +--- + +## 🎯 Sponsorship Tiers + +If you'd like ongoing sponsorship with visibility, get in touch: + +| Tier | Suggested Monthly | Perks | +|------|-------------------|-------| +| ☕ **Coffee** | $5 | Your name in `DONORS.md` | +| 🖥️ **Node** | $25 | Above + priority issue triage | +| 🏗️ **Cluster** | $100 | Above + logo on README + input on roadmap | +| 🚀 **Sustaining** | $500+ | Above + co-branded release notes + dedicated support channel | + +Sponsors are listed in **[DONORS.md](DONORS.md)** (with their permission). + +--- + +## 🏢 Corporate Sponsorship + +If Fester is critical to your business and you want dedicated support, SLAs, or custom features: + +- 📧 Email: `REPLACE_WITH_YOUR_BUSINESS_EMAIL` +- We can invoice monthly/quarterly/annually +- Custom feature development is available +- On-prem deployment assistance + +--- + +## 🛠️ Non-Financial Contributions + +Money isn't the only way to help: + +- ⭐ **Star the repo** on [git.dcos.net/dcosnet/fester](https://git.dcos.net/dcosnet/fester) +- 🐛 **File bug reports** with clear reproduction steps +- ✨ **Submit PRs** — see **[CONTRIBUTING.md](CONTRIBUTING.md)** +- 📣 **Spread the word** — write a blog post, mention us on forums +- 🧪 **Test on your hardware** — especially non-x86_64 arches (arm64, riscv64) +- 📖 **Improve docs** — quickstart, cheatsheet, architecture notes + +--- + +## 📋 Transparency + +- All donations are tracked publicly in **[DONORS.md](DONORS.md)** (unless you request anonymity) +- Quarterly reports will be posted showing how funds were used +- Donations do **not** grant special access or priority on the issue tracker (that's what sponsorship tiers are for) + +--- + +## 🙏 Thank You + +To everyone who has donated, starred, contributed, or just tried Fester out — **you're the reason this project exists**. The build must flow. + +--- + +*Donation addresses last updated: 2026-06-28. Always check this file for the most current addresses.* diff --git a/fester.js b/fester.js new file mode 100644 index 0000000..dc1db9e --- /dev/null +++ b/fester.js @@ -0,0 +1,48 @@ +document.addEventListener("DOMContentLoaded", function () { + + function color(heat) { + if (heat < 30) return "#2ecc71"; + if (heat < 60) return "#f1c40f"; + if (heat < 80) return "#e67e22"; + return "#e74c3c"; + } + + document.getElementById("load").addEventListener("click", function () { + + cockpit.spawn( + ["python3", "/usr/share/cockpit/fester/backend/nodes_cli.py"], + { superuser: "require" } + ) + .then(data => { + + const parsed = JSON.parse(data); + + let html = `

Master

+
${JSON.stringify(parsed.master, null, 2)}
+

Nodes

`; + + parsed.nodes.forEach(n => { + + html += ` +
+ ${n.name}
+ state: ${n.state}
+ heat: ${n.heat}
+
+ `; + }); + + document.getElementById("nodes").innerHTML = html; + }) + .catch(err => { + document.getElementById("nodes").textContent = err.toString(); + }); + }); + +}); diff --git a/index.html b/index.html new file mode 100644 index 0000000..11cd491 --- /dev/null +++ b/index.html @@ -0,0 +1,618 @@ + + + + + +Fester — Dashboard + + + + + +
+ + +
+
+

Cluster

+
+ + +
+
+
+
+
+
Nodes Online
+
+
+
+
Avg Heat
+
+
+
+
Active Jobs
+
+
+
+
Cache Hit %
+
+
+
+ +

Nodes

+
+ +

Scheduler Mode

+ +
+
+ + +
+
+

Live Event Stream

+
+ + + + +
+
+
+
+
+
+
+
+
+ + +
+
+

Quick Actions

+
+
+
+ + +
+
+ + +
+
+ +
+ +
+ + +
+ + +
+ +

Targets

+
+ +
+ +

Pipeline Control

+
+ + + +
+
+ + + + +
+ +
+ +

Release Control

+
+
+
+ +
+ + + + + diff --git a/install-fester.sh b/install-fester.sh new file mode 100755 index 0000000..5a6229f --- /dev/null +++ b/install-fester.sh @@ -0,0 +1,250 @@ +#!/usr/bin/env bash +# ============================================================ +# Fester Bootstrap Installer +# ============================================================ +# Deploys the Fester codebase to a target directory, runs the +# first-run config wizard, and writes the systemd unit + cockpit +# manifest. Does NOT overwrite the real CLI with a stub. +# +# Usage: +# ./install-fester.sh # interactive +# BASE_DIR=/opt/fester ./install-fester.sh # custom target +# NONINTERACTIVE=1 ./install-fester.sh # CI mode (defaults) +# ============================================================ +set -euo pipefail + +# Resolve the repo root (directory containing this script) +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +BASE_DIR="${BASE_DIR:-/usr/share/cockpit/fester}" +NONINTERACTIVE="${NONINTERACTIVE:-0}" + +echo "============================================================" +echo " Fester Bootstrap Installer" +echo " Source: $REPO_ROOT" +echo " Target: $BASE_DIR" +echo "============================================================" + +# ------------------------------------------------------------ +# Dependency checks +# ------------------------------------------------------------ +echo "" +echo "🔍 Checking dependencies..." + +check_dep() { + if command -v "$1" >/dev/null 2>&1; then + echo " ✓ $1" + return 0 + else + echo " ✗ $1 NOT FOUND" + return 1 + fi +} + +MISSING=0 +check_dep python3 || MISSING=1 + +# Python packages +PYTHON_OK=1 +python3 -c "import fastapi" 2>/dev/null && echo " ✓ python: fastapi" || { echo " ✗ python: fastapi missing"; PYTHON_OK=0; } +python3 -c "import uvicorn" 2>/dev/null && echo " ✓ python: uvicorn" || { echo " ✗ python: uvicorn missing"; PYTHON_OK=0; } +python3 -c "import websockets" 2>/dev/null && echo " ✓ python: websockets" || { echo " ✗ python: websockets missing"; PYTHON_OK=0; } +python3 -c "import prometheus_client" 2>/dev/null && echo " ✓ python: prometheus_client" || { echo " ✗ python: prometheus_client missing"; PYTHON_OK=0; } + +if [ "$MISSING" -eq 1 ]; then + echo "" + echo "❌ Required system tools missing. Install them first." + exit 1 +fi + +if [ "$PYTHON_OK" -eq 0 ]; then + echo "" + echo "⚠️ Some Python packages missing. Installing..." + pip3 install --break-system-packages fastapi uvicorn websockets prometheus_client minio pydantic 2>/dev/null || true +fi + +# ------------------------------------------------------------ +# Create directory structure +# ------------------------------------------------------------ +echo "" +echo "📁 Creating directory structure at $BASE_DIR..." +sudo mkdir -p "$BASE_DIR" +sudo chown -R "$USER":"$USER" "$BASE_DIR" 2>/dev/null || true +mkdir -p "$BASE_DIR"/{backend,ui,cockpit/fester-module,cli,docs,config} + +# ------------------------------------------------------------ +# Copy code from repo +# ------------------------------------------------------------ +echo "📦 Copying code from repo..." + +# Backend +cp -r "$REPO_ROOT/backend/"* "$BASE_DIR/backend/" + +# UI +cp "$REPO_ROOT/style.css" "$BASE_DIR/" +cp "$REPO_ROOT/index.html" "$BASE_DIR/" +cp -r "$REPO_ROOT/ui/"* "$BASE_DIR/ui/" + +# Cockpit module +cp -r "$REPO_ROOT/cockpit/"* "$BASE_DIR/cockpit/" + +# CLI (the REAL one, not a stub) +cp "$REPO_ROOT/cli/fester.py" "$BASE_DIR/cli/fester.py" +chmod +x "$BASE_DIR/cli/fester.py" + +# Top-level files +cp "$REPO_ROOT/manifest.json" "$BASE_DIR/" +cp "$REPO_ROOT/project.schema.json" "$BASE_DIR/" +cp "$REPO_ROOT/CHEATSHEET.md" "$BASE_DIR/docs/" +cp "$REPO_ROOT/README" "$BASE_DIR/LICENSE" 2>/dev/null || true + +# ------------------------------------------------------------ +# Config wizard +# ------------------------------------------------------------ +echo "" +echo "⚙️ Configuration" + +if [ "$NONINTERACTIVE" = "1" ]; then + CLUSTER_NAME="fester-cluster" + LIBVIRT="y" + LXC="y" + DISTCC="y" + MINIO="y" +else + read -rp "Cluster name [fester-cluster]: " CLUSTER_NAME + CLUSTER_NAME=${CLUSTER_NAME:-fester-cluster} + + read -rp "Enable libvirt integration? (y/n) [y]: " LIBVIRT + LIBVIRT=${LIBVIRT:-y} + + read -rp "Enable LXC integration? (y/n) [y]: " LXC + LXC=${LXC:-y} + + read -rp "Enable distcc integration? (y/n) [y]: " DISTCC + DISTCC=${DISTCC:-y} + + read -rp "Enable MinIO cache backend? (y/n) [y]: " MINIO + MINIO=${MINIO:-y} +fi + +# Write the canonical config.yaml — aligned with the project.schema.json +cat > "$BASE_DIR/config.yaml" </dev/null && sudo chown "$USER":"$USER" /etc/fester 2>/dev/null || true +echo '{}' > /etc/fester/node_roles.json 2>/dev/null || true + +# ------------------------------------------------------------ +# Cache dir +# ------------------------------------------------------------ +mkdir -p /var/lib/fester/cache 2>/dev/null && sudo chown "$USER":"$USER" /var/lib/fester 2>/dev/null || true + +# ------------------------------------------------------------ +# Prometheus scrape config (drop-in) +# ------------------------------------------------------------ +cat > "$BASE_DIR/add-to-prometheus-config.yml" <<'EOF' +scrape_configs: + - job_name: 'fester' + static_configs: + - targets: ['localhost:8080'] + metrics_path: /metrics +EOF + +# ------------------------------------------------------------ +# Grafana dashboard stub +# ------------------------------------------------------------ +cat > "$BASE_DIR/add-to-grafana-config.json" <<'EOF' +{ + "dashboard": { + "title": "Fester Cluster Overview", + "panels": [ + { "type": "stat", "title": "Nodes Online", "targets": [{ "expr": "fester_nodes_online" }] }, + { "type": "stat", "title": "Active Jobs", "targets": [{ "expr": "fester_cluster_active_jobs" }] }, + { "type": "graph", "title": "Avg Heat", "targets": [{ "expr": "fester_cluster_avg_heat" }] } + ] + } +} +EOF + +# ------------------------------------------------------------ +# Systemd unit (optional) +# ------------------------------------------------------------ +if command -v systemctl >/dev/null 2>&1; then + cat > /tmp/fester.service < "$BASE_DIR/config.yaml" < "$BASE_DIR/LICENSE" <<'EOF' +Fester Distributed Build System + +Licensed under GNU AGPL v3 + +This program is free software: you can redistribute it and/or modify +it under the terms of the GNU Affero General Public License. + +This software is provided "AS IS", without warranty of any kind. +EOF + + +# ============================================================ +# CHEATSHEET +# ============================================================ + +cat > "$BASE_DIR/CHEATSHEET.md" <<'EOF' +# 🧠 Fester Cheatsheet + +## Build +fester build + +## Node Control +fester node list +fester node set-policy preferred|avoid + +## Debug +/api/autopsy/ +/api/replay/ + +## UI +- Live DAG: /ui/live_dag.html +- Replay: /ui/replay.html +EOF + + +# ============================================================ +# INDEX PAGE (COCKPIT ENTRY) +# ============================================================ + +cat > "$BASE_DIR/index.html" <<'EOF' + +Fester Control Plane + +

Fester Cluster Control

+ + + +EOF + + +# ============================================================ +# UI: LIVE DAG (FULL FILE) +# ============================================================ + +cat > "$BASE_DIR/ui_live_dag.html" <<'EOF' + + + + + + +
+ + + + + +EOF + + +# ============================================================ +# PROMETHEUS CONFIG +# ============================================================ + +cat > "$BASE_DIR/add-to-prometheus-config.yml" <<'EOF' +scrape_configs: + - job_name: 'fester' + static_configs: + - targets: ['localhost:9100'] +EOF + + +# ============================================================ +# GRAFANA CONFIG +# ============================================================ + +cat > "$BASE_DIR/add-to-grafana-config.json" <<'EOF' +{ + "dashboard": { + "title": "Fester Cluster Overview" + } +} +EOF + + +# ============================================================ +# CLI TOOL (FULL FILE) +# ============================================================ + +mkdir -p "$BASE_DIR/cli" + +cat > "$BASE_DIR/cli/fester.py" <<'EOF' +#!/usr/bin/env python3 + +import argparse + +def main(): + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="cmd") + + build = sub.add_parser("build") + build.add_argument("project") + + node = sub.add_parser("node") + node.add_argument("action") + node.add_argument("target", nargs="*") + + args = parser.parse_args() + + if args.cmd == "build": + print("Building:", args.project) + + elif args.cmd == "node": + print("Node action:", args.action, args.target) + + else: + print("Fester CLI ready") + +if __name__ == "__main__": + main() +EOF + +chmod +x "$BASE_DIR/cli/fester.py" + + +# ============================================================ +# SIMPLE COCKPIT MANIFEST +# ============================================================ + +cat > "$BASE_DIR/manifest.json" <<'EOF' +{ + "version": 1, + "name": "fester", + "label": "Fester Cluster Control", + "entrypoint": "index.html" +} +EOF + + +# ============================================================ +# FINAL OUTPUT +# ============================================================ + +echo "" +echo "------------------------------------------------" +echo "✅ Fester installation complete" +echo "📍 Installed at: $BASE_DIR" +echo "" +echo "Next steps:" +echo " 1. Open Cockpit" +echo " 2. Select Fester module" +echo " 3. Open Live DAG" +echo "------------------------------------------------" diff --git a/manifest.json b/manifest.json new file mode 100644 index 0000000..fbd6430 --- /dev/null +++ b/manifest.json @@ -0,0 +1,10 @@ +{ + "version": 1, + "name": "fester", + "tools": { + "fester": { + "label": "Fester", + "path": "index.html" + } + } +} diff --git a/optimizer.py b/optimizer.py new file mode 100644 index 0000000..902caa1 --- /dev/null +++ b/optimizer.py @@ -0,0 +1,12 @@ +from prometheus_client import Gauge + +target_score_thermal = Gauge("fester_target_score_thermal", "Thermal score", ["target"]) +target_score_cost = Gauge("fester_target_score_cost", "Cost score", ["target"]) +target_score_instability = Gauge("fester_target_score_instability", "Instability score", ["target"]) + + +def export_scores(target, score): + + target_score_thermal.labels(target=target).set(score["thermal"]) + target_score_cost.labels(target=target).set(score["cost"]) + target_score_instability.labels(target=target).set(score["instability"]) diff --git a/project.schema.json b/project.schema.json new file mode 100644 index 0000000..5ecdc40 --- /dev/null +++ b/project.schema.json @@ -0,0 +1,170 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Fester Project Definition", + "type": "object", + "required": ["name", "source", "targets"], + "additionalProperties": false, + + "properties": { + + "name": { + "type": "string", + "description": "Human-readable project name" + }, + + "version": { + "type": "string", + "description": "Optional project version tag (release tracking)", + "default": "0.0.0" + }, + + "source": { + "type": "object", + "required": ["type", "url"], + "additionalProperties": false, + + "properties": { + + "type": { + "type": "string", + "enum": ["git", "hg", "svn", "cvs"], + "description": "Source control system type (normalized into snapshot)" + }, + + "url": { + "type": "string", + "description": "Repository URL or path for checkout/export" + }, + + "ref": { + "type": "string", + "description": "Optional branch/tag/revision reference (git branch, svn rev, hg changeset)", + "default": "HEAD" + } + } + }, + + "targets": { + "type": "object", + "description": "Build targets mapped to shell commands", + "additionalProperties": { + "type": "string" + } + }, + + "resources": { + "type": "object", + "description": "Optional resource hints for scheduler optimization", + + "properties": { + + "max_parallel_builds": { + "type": "integer", + "minimum": 1, + "maximum": 256, + "default": 64 + }, + + "thermal_priority": { + "type": "string", + "enum": ["low", "balanced", "high_safety"], + "default": "high_safety" + }, + + "preferred_nodes": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Optional node names preferred for this project" + }, + + "avoid_nodes": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Nodes to exclude from scheduling" + } + } + }, + + "build_env": { + "type": "object", + "description": "Environment variables injected into build runtime", + + "additionalProperties": { + "type": "string" + } + }, + + "artifacts": { + "type": "object", + "description": "Output packaging configuration", + + "properties": { + + "enabled": { + "type": "boolean", + "default": true + }, + + "format": { + "type": "string", + "enum": ["tar", "tar.gz", "tar.zst", "deb", "rpm"], + "default": "tar.zst" + }, + + "output_dir": { + "type": "string", + "default": "./dist" + }, + + "naming": { + "type": "string", + "description": "Artifact naming template", + "default": "{name}-{version}-{target}-{hash}" + } + } + }, + + "cache": { + "type": "object", + "description": "Build cache behavior control", + + "properties": { + + "enabled": { + "type": "boolean", + "default": true + }, + + "reuse_snapshots": { + "type": "boolean", + "default": true + }, + + "skip_if_unchanged": { + "type": "boolean", + "default": true + } + } + }, + + "logging": { + "type": "object", + "properties": { + + "verbose": { + "type": "boolean", + "default": true + }, + + "stream_to_ui": { + "type": "boolean", + "default": true + } + } + } + } +} diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..6312395 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,177 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "fester" +version = "0.3.0" +description = "Distributed, DAG-driven build execution system with real-time scheduling, observability, and deterministic replay" +readme = "README.md" +license = { text = "AGPL-3.0-or-later", file = "LICENSE" } +authors = [ + { name = "Fester Contributors" }, +] +requires-python = ">=3.12" +keywords = ["build", "distributed", "scheduler", "dag", "compile", "distcc", "observability"] +classifiers = [ + "Development Status :: 4 - Beta", + "Environment :: Console", + "Environment :: Web Environment", + "Framework :: FastAPI", + "Intended Audience :: Developers", + "Intended Audience :: System Administrators", + "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", + "Operating System :: POSIX :: Linux", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Software Development :: Build Tools", + "Topic :: System :: Distributed Computing", +] + +# Core runtime dependencies +dependencies = [ + "fastapi>=0.128", + "uvicorn[standard]>=0.30", + "websockets>=12", + "pydantic>=2", + "aiohttp>=3.9", + "minio>=7.2", + "prometheus-client>=0.20", + "PyYAML>=6", + "requests>=2.31", + "websocket-client>=1.7", +] + +[project.optional-dependencies] +# Install with: pip install -e ".[dev]" +dev = [ + "ruff>=0.5", + "mypy>=1.10", + "pytest>=8", + "pytest-asyncio>=0.23", + "httpx>=0.27", # for FastAPI TestClient + "respx>=0.21", # for mocking aiohttp in tests +] +# Install with: pip install -e ".[docker]" +docker = [ + "gunicorn>=22", +] + +[project.scripts] +# After `pip install -e .`, you can run `fester` from anywhere +fester = "cli.fester:main" + +[project.urls] +Homepage = "https://git.dcos.net/dcosnet/fester" +Repository = "https://git.dcos.net/dcosnet/fester" +Documentation = "https://git.dcos.net/dcosnet/fester/blob/main/README.md" +Issues = "https://git.dcos.net/dcosnet/fester/issues" + +# ------------------------------------------------------------ +# Setuptools package discovery +# ------------------------------------------------------------ +[tool.setuptools] +# We're not a pure-Python package — just expose the CLI + backend modules +py-modules = [] + +[tool.setuptools.packages.find] +where = ["."] +include = [ + "backend*", + "cli*", + "cockpit*", +] +exclude = [ + "tests*", + "docs*", + "scripts*", +] + +[tool.setuptools.package-data] +# Include non-Python files in the package +"*" = ["*.html", "*.css", "*.js", "*.yaml", "*.json", "*.md"] + +# ------------------------------------------------------------ +# Ruff (linter + formatter) +# ------------------------------------------------------------ +[tool.ruff] +line-length = 100 +target-version = "py312" +extend-exclude = [ + "docs", + "scripts/mock_agent.py", +] + +[tool.ruff.lint] +# Enable common rule sets +select = [ + "E", # pycodestyle errors + "W", # pycodestyle warnings + "F", # pyflakes + "I", # isort + "B", # flake8-bugbear + "C4", # flake8-comprehensions + "UP", # pyupgrade + "RUF", # ruff-specific +] +ignore = [ + "E501", # line too long (we use 100 but don't want to fail on it) + "B008", # do not perform function calls in argument defaults (FastAPI does this) + "B904", # raise from (we have a lot of legacy code) + "RUF001", # ambiguous unicode (we use emojis) + "RUF002", # ambiguous unicode in docstring + "RUF003", # ambiguous unicode in comment +] + +[tool.ruff.lint.per-file-ignores] +"__init__.py" = ["F401"] # re-exports +"cli/fester.py" = ["E402"] # module-level imports after argparse helpers + +[tool.ruff.format] +quote-style = "double" +indent-style = "space" + +# ------------------------------------------------------------ +# Mypy (type checking) +# ------------------------------------------------------------ +[tool.mypy] +python_version = "3.12" +warn_return_any = false # too noisy for this codebase +warn_unused_configs = true +disallow_untyped_defs = false # we have legacy untyped code +disallow_incomplete_defs = false +check_untyped_defs = true +disallow_untyped_decorators = false +no_implicit_optional = true +warn_redundant_casts = true +warn_unused_ignores = true +warn_no_return = true +warn_unreachable = true +strict_equality = true +show_error_codes = true + +[[tool.mypy.overrides]] +module = [ + "minio.*", + "aiohttp.*", + "websocket.*", + "prometheus_client.*", +] +ignore_missing_imports = true + +# ------------------------------------------------------------ +# Pytest +# ------------------------------------------------------------ +[tool.pytest.ini_options] +minversion = "8.0" +testpaths = ["tests"] +asyncio_mode = "auto" +filterwarnings = [ + "ignore::DeprecationWarning", + "ignore::PendingDeprecationWarning", +] +markers = [ + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "integration: marks tests that require external services (minio, etc.)", +] diff --git a/quickstart.md b/quickstart.md new file mode 100644 index 0000000..f014fd0 --- /dev/null +++ b/quickstart.md @@ -0,0 +1,320 @@ +# 🚀 Fester Quickstart + +**5 minutes to your first distributed build.** + +--- + +## 1. Prerequisites + +You need: +- **Python 3.12+** (`python3 --version`) +- **pip** (usually ships with Python) +- **git** (to clone the repo) +- **curl** (to test the API) + +Optional (features activate automatically when installed): +- `tmux` — live action output viewer +- `mosh` + `ssh` — "Shell into node" UI button +- `qemu-utils` + `rsync` — QCOW2 workspace snapshots +- `btrfs-progs` — CoW reflink snapshots (requires btrfs partition) + +On Ubuntu/Debian: +```bash +sudo apt install python3 python3-pip git curl tmux mosh openssh-client qemu-utils rsync +``` + +## 2. Clone + Bootstrap + +```bash +git clone https://git.dcos.net/dcosnet/fester.git +cd fester + +# One-shot: creates venv, installs Python deps, sets up config +./bootstrap.sh +``` + +The bootstrap script will: +1. Create a Python virtualenv at `.venv/` +2. Install all Python dependencies (FastAPI, uvicorn, websockets, minio, aiohttp, etc.) +3. Create a default `config.yaml` if one doesn't exist +4. Initialize the SQLite database at `~/.fester/fester.db` +5. Print next steps + +## 3. Start the Backend + +```bash +./run.sh +``` + +You should see: +``` +🧠 Fester backend starting... + Config: config.yaml + DB: /home/user/.fester/fester.db + URL: http://0.0.0.0:8080 + Press Ctrl+C to stop +INFO: Uvicorn running on http://0.0.0.0:8080 +``` + +Verify it's up: +```bash +curl http://localhost:8080/api/health +# → {"status":"ok","version":"0.2.0","bus_subscribers":6,...} +``` + +## 4. Open the UI + +Open **http://localhost:8080** in your browser. + +You'll see the **Dashboard** with: +- **Cluster panel** (left): node list with heat bars, job counters, policy dropdowns, probe buttons +- **Live Event Stream** (center): real-time events with type-filter chips +- **Quick Actions** (right): build form, release form, target toggles, pipeline controls + +The topbar shows a live health indicator (version + WS clients + builds count + timeline events). + +## 5. Trigger Your First Build + +### Via the UI + +In the **Quick Actions** panel (right side of the dashboard): +1. **Build Command**: `make -j$(nproc)` (or any shell command) +2. **Working Directory**: `/tmp/test-build` (or your project dir) +3. Click **▶ Run Build** + +Watch the **Live Event Stream** populate with `task_update` events as actions execute. + +### Via the CLI + +```bash +# Activate the venv first (or use the full path) +source .venv/bin/activate + +# Kick off a build +fester build --cmd "make -j4" --dir /tmp/test-build --watch + +# The --watch flag streams events as they happen +``` + +### Via curl + +```bash +curl -X POST http://localhost:8080/api/build \ + -H "Content-Type: application/json" \ + -d '{"cmd":"make -j4","dir":"/tmp/test-build","target":"linux-gnu"}' +``` + +## 6. Explore the UI + +### Live DAG (`/ui/live_dag.html`) + +Real-time visualization of the build DAG. Each action becomes a node; dependency edges are drawn as arrows. Node colors: +- 🟡 **yellow border** = running +- 🟢 **green border** = done +- 🔴 **red border** = failed +- 🔵 **blue border** = cache hit +- 🟠 **yellow outline** = critical path + +Click any node to inspect its details. Use the search box to filter, and the toggles to show/hide critical-path and failed nodes. + +### Replay (`/ui/replay.html`) + +Scrub through past builds event-by-event: +1. Click **Load Replay** to snapshot the current timeline +2. Use the timeline slider, or **▶ Play** / **⏭ Step** / **⏮ Back** buttons +3. Jump to the end with **⏭ End** to see the full DAG +4. Click any node to see its state, deps, scheduler decision, and raw event + +### Sessions (`/ui/sessions.html`) + +Build history + active tmux sessions + qcow2 snapshots: +- **Replay sessions** — click "▶ Open in Replay" to scrub through any past session +- **Recent Builds** — cancel running builds, or replay completed ones +- **Active Action Sessions** — view live tmux output of running actions +- **Storage Snapshots** — list of qcow2 freeze images + +### Metrics (`/ui/metrics.html`) + +Live cluster metrics with two trend charts (heat + jobs) and per-node cards showing CPU/memory/heat/jobs/instability bars. Refreshes every 2 seconds. + +### Cause Graph (`/ui/cause.html`) + +Post-hoc reasoning: enter a node name, see its causal chain as a radial graph. The **Blast Radius** panel computes the downstream impact if any action fails. + +### Timeline (`/ui/timeline.html`) + +Per-node event drill-down: pick a node from the left, see every event that touched it. + +### Debugger (`/ui/debugger.html`) + +Step-through execution control: pause/resume/step a running build. Shows the live tmux timeline and the next action preview. + +## 7. Configure Your Cluster + +Edit `config.yaml`: + +```yaml +master: + name: fester-master + role: control + +nodes: + - name: x99-v3 + host: 192.168.1.10 + max_jobs: 24 + - name: x99-v4 + host: 192.168.1.11 + max_jobs: 30 + - name: rpi-1 + host: 192.168.1.20 + max_jobs: 4 + +projects: + - name: linux-tool + repo: https://forgejo.local/linux-tool.git + targets: + debian: "make clean && make debian" + arch: "make clean && make arch" +``` + +Restart `./run.sh` to pick up changes. + +## 8. Set Node Policies + +Via the UI (Dashboard → node row → policy dropdown): +- **preferred** — scheduler gives this node a +20 score bonus +- **neutral** — default +- **avoid** — scheduler subtracts 50 from this node's score + +Via the CLI: +```bash +fester node set-policy x99-v3 preferred +fester node set-policy rpi-1 avoid +``` + +Via the API: +```bash +curl -X POST http://localhost:8080/api/nodes/x99-v3/policy \ + -H "Content-Type: application/json" \ + -d '{"policy":"preferred"}' +``` + +## 9. Probe Node Agents (Optional) + +If you run a fester-agent on each node (port 8787), Fester will probe it every 4 seconds for real metrics instead of drifting synthetic values. + +A mock agent for testing: +```bash +python3 scripts/mock_agent.py 8787 & +``` + +Manual probe via the UI (Dashboard → node row → "Probe" button) or CLI: +```bash +fester node probe x99-v3 +fester node probe-all +``` + +## 10. Shell Into a Node + +Click the **⌘ Shell** button next to any node in the Dashboard. Fester builds the correct mosh or ssh command and copies it to your clipboard: + +```bash +mosh --ssh "ssh -p 2222 -i ~/.ssh/id_ed25519" build@192.168.1.10 +``` + +Paste into your terminal to attach. + +## 11. Run a Release Build + +Via the UI (Dashboard → Quick Actions → Run Release Build) or CLI: +```bash +fester release --repo https://forgejo.local/project.git --project my-proj +``` + +This kicks off a longer release-shaped pipeline: `fetch_source → configure → build_kernel → build_modules → package → release`. + +## 12. Investigate a Failure + +When a build fails: + +1. **Sessions page** → find the failed build → click **▶ Replay** +2. In Replay, scrub to the failed action (red node) +3. Click the failed node → see the failure reason in the inspector +4. Switch to **Cause Graph** page → enter the node name → see the causal chain +5. Use **Blast Radius** to see which downstream actions were impacted + +Via the CLI: +```bash +# Start a replay session +SID=$(curl -sX POST http://localhost:8080/replay/start \ + -H "Content-Type: application/json" \ + -d '{"journal":"latest"}' | jq -r .session_id) + +# Run autopsy on the failed action +fester autopsy $SID build_fedora +``` + +## 🎯 Next Steps + +- Read the **[CHEATSHEET.md](CHEATSHEET.md)** for the operator survival guide +- Browse the **[full API](http://localhost:8080/docs)** (FastAPI auto-docs) +- Set up **Prometheus + Grafana** using the configs in `add-to-prometheus-config.yml` and `add-to-grafana-config.json` +- Install **tmux**, **mosh**, and **qemu-utils** to activate the advanced features +- Configure **MinIO** for distributed cache (set the endpoint in `backend/cache/minio_cache.py`) + +## 🆘 Troubleshooting + +### Backend won't start + +```bash +# Check the log +tail -f /tmp/fester_backend.log + +# Common issues: +# - "No module named 'fastapi'" → run ./bootstrap.sh again +# - "Permission denied: /var/lib/fester" → use FESTER_DB_PATH=/tmp/fester.db +# - "Address already in use" → another process is on port 8080 +``` + +### UI shows "CONNECTING" forever + +The WebSocket can't reach the backend. Check: +```bash +curl http://localhost:8080/api/health +# If this fails, the backend isn't running +``` + +### No events in the Live Event Stream + +Trigger a build: +```bash +curl -X POST http://localhost:8080/api/build \ + -H "Content-Type: application/json" \ + -d '{"cmd":"demo","dir":"/tmp"}' +``` + +### Nodes show 0° heat / 0 jobs + +No agent is running on the nodes. Either: +- Start a mock agent: `python3 scripts/mock_agent.py 8787` +- Or accept synthetic drift (default behavior when agents are unreachable) + +### Build fails immediately + +Check the build details: +```bash +fester builds +fester build-info +``` + +Common causes: +- Working directory doesn't exist → create it first +- Command not found → check PATH +- Permission denied → check file permissions + +## 📚 More Info + +- **[README.md](README.md)** — full project overview +- **[CHEATSHEET.md](CHEATSHEET.md)** — operator cheatsheet +- **[CONTRIBUTING.md](CONTRIBUTING.md)** — how to contribute +- **[API docs](http://localhost:8080/docs)** — interactive OpenAPI spec diff --git a/release.js b/release.js new file mode 100644 index 0000000..480be98 --- /dev/null +++ b/release.js @@ -0,0 +1,18 @@ +document.addEventListener("DOMContentLoaded", function () { + + function runRelease() { + + const repo = document.getElementById("repo").value; + + cockpit.spawn( + ["python3", "/usr/share/cockpit/fester/backend/release.py", repo], + { superuser: "require" } + ) + .stream(function (data) { + document.getElementById("release_output").textContent += data; + }); + } + + document.getElementById("run_release").addEventListener("click", runRelease); + +}); diff --git a/run.sh b/run.sh new file mode 100755 index 0000000..17a85a6 --- /dev/null +++ b/run.sh @@ -0,0 +1,168 @@ +#!/usr/bin/env bash +# ============================================================ +# Fester Backend Runner +# ============================================================ +# Starts the FastAPI backend with sensible defaults. +# +# Usage: +# ./run.sh # default: host=0.0.0.0, port=8080 +# ./run.sh --dev # dev mode with auto-reload +# ./run.sh --mock-agent # also start a mock fester-agent on :8787 +# ./run.sh --port 9000 # custom port +# ./run.sh --help +# ============================================================ +set -euo pipefail + +# Resolve repo root +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$REPO_ROOT" + +# Defaults +HOST="${FESTER_HOST:-0.0.0.0}" +PORT="${FESTER_PORT:-8080}" +DEV_MODE=0 +START_MOCK_AGENT=0 + +# Parse args +while [[ $# -gt 0 ]]; do + case "$1" in + --dev) + DEV_MODE=1 + shift + ;; + --mock-agent) + START_MOCK_AGENT=1 + shift + ;; + --port) + PORT="$2" + shift 2 + ;; + --host) + HOST="$2" + shift 2 + ;; + --help|-h) + cat </dev/null; then + echo "❌ FastAPI not installed. Run ./bootstrap.sh first." + exit 1 +fi + +# ------------------------------------------------------------ +# Optionally start a mock fester-agent +# ------------------------------------------------------------ +MOCK_PID="" +if [ "$START_MOCK_AGENT" = "1" ]; then + echo "🤖 Starting mock fester-agent on :8787..." + MOCK_SCRIPT="$REPO_ROOT/scripts/mock_agent.py" + if [ ! -f "$MOCK_SCRIPT" ]; then + # Fall back to the global scripts dir + MOCK_SCRIPT="/home/z/my-project/scripts/mock_agent.py" + fi + if [ -f "$MOCK_SCRIPT" ]; then + python3 "$MOCK_SCRIPT" 8787 & + MOCK_PID=$! + echo " ✓ Mock agent started (PID $MOCK_PID)" + sleep 1 + else + echo " ⚠ mock_agent.py not found — skipping" + fi +fi + +# ------------------------------------------------------------ +# Print startup banner +# ------------------------------------------------------------ +CONFIG_PATH="${FESTER_CONFIG:-config.yaml}" +DB_PATH="${FESTER_DB_PATH:-$HOME/.fester/fester.db}" + +echo "" +echo "🧠 Fester backend starting..." +echo " Config: $CONFIG_PATH" +echo " DB: $DB_PATH" +echo " URL: http://$HOST:$PORT" +echo " Mode: $([ "$DEV_MODE" = "1" ] && echo "development (auto-reload)" || echo "production")" +echo " Drift: $([ "${FESTER_NO_DRIFT:-0}" = "1" ] && echo "disabled" || echo "enabled (fallback)")" +echo " Autobuild: $([ "${FESTER_AUTOBUILD:-0}" = "1" ] && echo "enabled" || echo "disabled")" +echo "" +echo " Press Ctrl+C to stop" +echo " UI: http://localhost:$PORT" +echo " API docs: http://localhost:$PORT/docs" +echo " Health: http://localhost:$PORT/api/health" +echo "" + +# ------------------------------------------------------------ +# Cleanup on exit +# ------------------------------------------------------------ +cleanup() { + echo "" + echo "🛑 Shutting down..." + if [ -n "$MOCK_PID" ]; then + kill "$MOCK_PID" 2>/dev/null || true + echo " ✓ Mock agent stopped" + fi + echo " ✓ Backend stopped" +} +trap cleanup EXIT INT TERM + +# ------------------------------------------------------------ +# Start uvicorn +# ------------------------------------------------------------ +if [ "$DEV_MODE" = "1" ]; then + exec python3 -m uvicorn backend.main:app \ + --host "$HOST" \ + --port "$PORT" \ + --reload \ + --log-level info +else + exec python3 -m uvicorn backend.main:app \ + --host "$HOST" \ + --port "$PORT" \ + --log-level info +fi diff --git a/scheduler/optimizer.py b/scheduler/optimizer.py new file mode 100644 index 0000000..4b70ab0 --- /dev/null +++ b/scheduler/optimizer.py @@ -0,0 +1,96 @@ +# scheduler/optimizer.py + +def choose_best_node(nodes, action, intelligence=None): + + best_node = None + best_score = -10**9 + + ranked = [] + + # ----------------------------- + # SCORE EACH NODE + # ----------------------------- + for node in nodes: + + breakdown = { + "cpu": 0, + "thermal": 0, + "policy": 0, + "intelligence": 0, + "final": 0 + } + + # CPU availability + cpu_score = (100 - node.get("cpu_load", 0)) + breakdown["cpu"] = cpu_score + + # Thermal penalty + thermal_penalty = node.get("temp", 0) + breakdown["thermal"] = -thermal_penalty + + # Policy rules + policy_score = 0 + + if node.get("policy") == "preferred": + policy_score += 20 + elif node.get("policy") == "avoid": + policy_score -= 50 + + breakdown["policy"] = policy_score + + # Intelligence feedback + intel_score = 0 + + if intelligence: + stats = intelligence.profiles.get(action["target"]) + + if stats: + latest = stats.finalize() + + intel_score -= latest["score"].get("thermal", 0) * 0.3 + intel_score -= latest["score"].get("instability", 0) * 0.2 + + breakdown["intelligence"] = intel_score + + # Final score + final = cpu_score + policy_score + intel_score + breakdown["thermal"] + breakdown["final"] = final + + ranked.append({ + "node": node, + "score": breakdown + }) + + if final > best_score: + best_score = final + best_node = node + + # ----------------------------- + # SORT FOR EXPLANATION + # ----------------------------- + ranked_sorted = sorted( + ranked, + key=lambda x: x["score"]["final"], + reverse=True + ) + + # ----------------------------- + # EXPLANATION OBJECT + # ----------------------------- + explanation = { + "selected": best_node["name"] if best_node else None, + "top_candidates": [ + { + "node": r["node"]["name"], + "final_score": r["score"]["final"], + "breakdown": r["score"] + } + for r in ranked_sorted[:5] + ] + } + + return { + "best_node": best_node, + "ranked": ranked, + "explanation": explanation + } diff --git a/style.css b/style.css new file mode 100644 index 0000000..0bb89a2 --- /dev/null +++ b/style.css @@ -0,0 +1,424 @@ +/* ============================================================ + Fester UI — Shared Design System + Used by: index.html, ui/live_dag.html, ui/replay.html, + cockpit/fester-module/ui.html + ============================================================ */ + +/* ---------- Design Tokens ---------- */ +:root { + /* surfaces */ + --bg-0: #07090d; + --bg-1: #0b0f14; + --bg-2: #0e141d; + --bg-3: #131c28; + --bg-4: #1a2432; + + /* lines */ + --line: #1f2a38; + --line-2: #2c3b4d; + + /* text */ + --fg-0: #f4f6fa; + --fg-1: #d0d6df; + --fg-2: #8a95a5; + --fg-3: #5a6573; + + /* status */ + --ok: #66bb6a; + --ok-bg: rgba(102, 187, 106, 0.12); + --warn: #ffb300; + --warn-bg: rgba(255, 179, 0, 0.12); + --err: #ef5350; + --err-bg: rgba(239, 83, 80, 0.12); + --info: #42a5f5; + --info-bg: rgba(66, 165, 245, 0.12); + --accent: #ffd54f; + --accent-bg: rgba(255, 213, 79, 0.12); + + /* heat ramp */ + --heat-0: #2ecc71; + --heat-1: #f1c40f; + --heat-2: #e67e22; + --heat-3: #e74c3c; + + /* layout */ + --radius: 6px; + --radius-lg: 10px; + --gap: 12px; + --pad: 12px; + --topbar-h: 48px; + + /* type */ + --font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; + --font-mono: "JetBrains Mono", "SF Mono", "Cascadia Code", Menlo, Consolas, monospace; +} + +/* ---------- Reset ---------- */ +*, *::before, *::after { box-sizing: border-box; } +html, body { margin: 0; padding: 0; height: 100%; } +body { + font-family: var(--font-sans); + font-size: 13px; + color: var(--fg-1); + background: var(--bg-1); + line-height: 1.45; + -webkit-font-smoothing: antialiased; +} +a { color: var(--info); text-decoration: none; } +a:hover { text-decoration: underline; } +pre, code, .mono { font-family: var(--font-mono); } +button, input, select, textarea { font-family: inherit; font-size: inherit; color: inherit; } +h1, h2, h3, h4 { margin: 0 0 8px; font-weight: 600; color: var(--fg-0); } +h1 { font-size: 18px; } +h2 { font-size: 15px; } +h3 { font-size: 13px; text-transform: uppercase; letter-spacing: 0.6px; color: var(--fg-2); } +hr { border: 0; border-top: 1px solid var(--line); margin: 12px 0; } + +/* ---------- Topbar / Shell ---------- */ +.topbar { + height: var(--topbar-h); + background: var(--bg-2); + border-bottom: 1px solid var(--line); + display: flex; + align-items: center; + padding: 0 16px; + gap: 16px; + position: sticky; + top: 0; + z-index: 50; +} +.topbar .brand { + font-weight: 700; + color: var(--fg-0); + letter-spacing: 0.5px; + display: flex; + align-items: center; + gap: 8px; +} +.topbar .brand .mark { + width: 22px; height: 22px; + border-radius: 4px; + background: linear-gradient(135deg, var(--info), var(--accent)); + display: inline-block; +} +.topbar nav { display: flex; gap: 4px; flex: 1; } +.topbar nav a { + color: var(--fg-2); + padding: 6px 12px; + border-radius: var(--radius); + font-size: 12px; + text-decoration: none; + transition: background 0.15s, color 0.15s; +} +.topbar nav a:hover { background: var(--bg-3); color: var(--fg-0); text-decoration: none; } +.topbar nav a.active { background: var(--bg-4); color: var(--fg-0); } + +/* connection pill */ +.conn { + display: inline-flex; + align-items: center; + gap: 6px; + padding: 4px 10px; + border-radius: 999px; + background: var(--bg-3); + border: 1px solid var(--line-2); + font-size: 11px; + font-family: var(--font-mono); + color: var(--fg-2); +} +.conn .dot { width: 7px; height: 7px; border-radius: 50%; background: var(--fg-3); } +.conn.connecting .dot { background: var(--warn); animation: pulse 1.2s infinite; } +.conn.online .dot { background: var(--ok); box-shadow: 0 0 6px var(--ok); } +.conn.offline .dot { background: var(--err); } +@keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.35; } } + +/* ---------- Buttons ---------- */ +button, .btn { + display: inline-flex; + align-items: center; + justify-content: center; + gap: 6px; + padding: 6px 12px; + background: var(--bg-4); + color: var(--fg-1); + border: 1px solid var(--line-2); + border-radius: var(--radius); + cursor: pointer; + transition: background 0.12s, border-color 0.12s, transform 0.05s; + font-size: 12px; + white-space: nowrap; +} +button:hover, .btn:hover { background: var(--bg-3); border-color: #3a4a5e; } +button:active, .btn:active { transform: translateY(1px); } +button:disabled { opacity: 0.4; cursor: not-allowed; } +button.primary { background: var(--info); color: #061018; border-color: var(--info); } +button.primary:hover { filter: brightness(1.1); } +button.danger { background: var(--err); color: #1a0606; border-color: var(--err); } +button.ghost { background: transparent; } +button.full { width: 100%; } + +/* ---------- Form inputs ---------- */ +input[type="text"], input[type="number"], input[type="search"], +input[type="url"], input[type="password"], select, textarea { + width: 100%; + padding: 7px 10px; + background: var(--bg-0); + border: 1px solid var(--line-2); + border-radius: var(--radius); + color: var(--fg-0); + font-family: var(--font-mono); + font-size: 12px; + outline: none; + transition: border-color 0.12s; +} +input:focus, select:focus, textarea:focus { border-color: var(--info); } +label { display: block; font-size: 11px; color: var(--fg-2); margin-bottom: 4px; text-transform: uppercase; letter-spacing: 0.4px; } +.field { margin-bottom: 10px; } + +/* range slider */ +input[type="range"] { + width: 100%; + -webkit-appearance: none; + background: transparent; + height: 24px; +} +input[type="range"]::-webkit-slider-runnable-track { + height: 4px; + background: var(--bg-4); + border-radius: 2px; +} +input[type="range"]::-webkit-slider-thumb { + -webkit-appearance: none; + width: 14px; height: 14px; + border-radius: 50%; + background: var(--info); + margin-top: -5px; + cursor: pointer; +} + +/* ---------- Panels / Cards ---------- */ +.panel { + background: var(--bg-2); + border: 1px solid var(--line); + border-radius: var(--radius-lg); + overflow: hidden; +} +.panel > .head { + padding: 10px 14px; + border-bottom: 1px solid var(--line); + display: flex; + align-items: center; + justify-content: space-between; + background: var(--bg-3); +} +.panel > .head h3 { margin: 0; } +.panel > .body { padding: var(--pad); } + +/* ---------- Badges / Pills ---------- */ +.badge { + display: inline-flex; + align-items: center; + gap: 4px; + padding: 2px 8px; + border-radius: 999px; + font-size: 10px; + font-family: var(--font-mono); + text-transform: uppercase; + letter-spacing: 0.5px; + border: 1px solid var(--line-2); + background: var(--bg-3); + color: var(--fg-2); +} +.badge.ok { color: var(--ok); background: var(--ok-bg); border-color: rgba(102,187,106,0.3); } +.badge.warn { color: var(--warn); background: var(--warn-bg); border-color: rgba(255,179,0,0.3); } +.badge.err { color: var(--err); background: var(--err-bg); border-color: rgba(239,83,80,0.3); } +.badge.info { color: var(--info); background: var(--info-bg); border-color: rgba(66,165,245,0.3); } +.badge.accent { color: var(--accent); background: var(--accent-bg); border-color: rgba(255,213,79,0.3); } + +/* ---------- Tables ---------- */ +table { width: 100%; border-collapse: collapse; font-size: 12px; } +th, td { text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--line); } +th { color: var(--fg-2); font-weight: 500; font-size: 11px; text-transform: uppercase; letter-spacing: 0.5px; } +tbody tr:hover { background: var(--bg-3); } + +/* ---------- Lists ---------- */ +.list { list-style: none; padding: 0; margin: 0; } +.list > li { padding: 8px 10px; border-bottom: 1px solid var(--line); } +.list > li:last-child { border-bottom: 0; } + +/* ---------- Event Stream ---------- */ +.stream { + font-family: var(--font-mono); + font-size: 11px; + height: 100%; + overflow-y: auto; + padding: 8px; + background: var(--bg-0); + border-radius: var(--radius); + border: 1px solid var(--line); +} +.stream .row { + padding: 3px 6px; + border-radius: 3px; + display: grid; + grid-template-columns: 70px 80px 1fr; + gap: 8px; + align-items: baseline; + border-bottom: 1px dashed rgba(255,255,255,0.04); +} +.stream .row:hover { background: var(--bg-2); } +.stream .ts { color: var(--fg-3); } +.stream .tag { font-weight: 600; } +.stream .tag.node { color: var(--info); } +.stream .tag.pipeline { color: var(--warn); } +.stream .tag.failure { color: var(--err); } +.stream .tag.cache { color: var(--accent); } +.stream .tag.debug { color: var(--fg-2); } +.stream .tag.policy { color: #ab47bd; } +.stream .tag.session { color: var(--ok); } +.stream .msg { color: var(--fg-1); word-break: break-word; } + +/* ---------- DAG Node (shared) ---------- */ +.dag-node { + position: absolute; + min-width: 120px; + padding: 6px 10px; + font-size: 11px; + text-align: center; + border-radius: var(--radius); + background: var(--bg-3); + border: 1px solid var(--line-2); + color: var(--fg-0); + cursor: pointer; + user-select: none; + transition: transform 0.15s, box-shadow 0.15s, border-color 0.15s; + z-index: 2; +} +.dag-node:hover { transform: translateY(-1px); box-shadow: 0 4px 12px rgba(0,0,0,0.4); } +.dag-node .label { font-weight: 600; } +.dag-node .meta { font-family: var(--font-mono); font-size: 10px; color: var(--fg-2); margin-top: 2px; } + +.dag-node.pending { border-color: var(--fg-3); } +.dag-node.running { border-color: var(--warn); box-shadow: 0 0 12px rgba(255,179,0,0.35); } +.dag-node.done { border-color: var(--ok); } +.dag-node.failed { border-color: var(--err); background: var(--err-bg); } +.dag-node.cache { border-color: var(--info); } +.dag-node.critical { outline: 2px solid var(--accent); outline-offset: 2px; } +.dag-node.heat { box-shadow: 0 0 0 2px var(--heat-2); } +.dag-node.active { outline: 2px solid var(--accent); outline-offset: 2px; } + +/* SVG edges */ +svg.edges { + position: absolute; + top: 0; left: 0; + width: 100%; height: 100%; + pointer-events: none; + z-index: 1; +} +svg.edges line, svg.edges path { + stroke: var(--line-2); + stroke-width: 1.5; + fill: none; +} +svg.edges line.critical, svg.edges path.critical { + stroke: var(--accent); + stroke-width: 2.5; +} +svg.edges marker path { fill: var(--line-2); stroke: none; } + +/* ---------- Inspector / JSON viewer ---------- */ +pre.json { + margin: 0; + padding: 10px; + background: var(--bg-0); + border-radius: var(--radius); + border: 1px solid var(--line); + font-family: var(--font-mono); + font-size: 11px; + color: var(--fg-1); + overflow: auto; + white-space: pre-wrap; + word-break: break-word; +} + +/* ---------- Scrollbars ---------- */ +::-webkit-scrollbar { width: 8px; height: 8px; } +::-webkit-scrollbar-track { background: var(--bg-0); } +::-webkit-scrollbar-thumb { background: var(--line-2); border-radius: 4px; } +::-webkit-scrollbar-thumb:hover { background: #3a4a5e; } + +/* ---------- Utility ---------- */ +.row { display: flex; gap: var(--gap); } +.col { display: flex; flex-direction: column; gap: var(--gap); } +.flex-1 { flex: 1; } +.grow { flex-grow: 1; } +.muted { color: var(--fg-2); } +.dim { color: var(--fg-3); } +.right { text-align: right; } +.center { text-align: center; } +.mono { font-family: var(--font-mono); } +.nowrap { white-space: nowrap; } +.hidden { display: none !important; } +.spacer { flex: 1; } + +/* heat meter */ +.heat-meter { + height: 6px; + border-radius: 3px; + background: linear-gradient(90deg, var(--heat-0) 0%, var(--heat-1) 50%, var(--heat-2) 80%, var(--heat-3) 100%); + position: relative; +} +.heat-meter::after { + content: ""; + position: absolute; + top: -2px; bottom: -2px; + left: var(--heat, 0%); + width: 2px; + background: var(--fg-0); +} + +/* progress bar */ +.progress { + height: 8px; + background: var(--bg-0); + border-radius: 4px; + overflow: hidden; + border: 1px solid var(--line); +} +.progress > .bar { + height: 100%; + background: linear-gradient(90deg, var(--info), var(--ok)); + transition: width 0.3s ease; +} + +/* toasts */ +#toasts { + position: fixed; + bottom: 16px; + right: 16px; + z-index: 1000; + display: flex; + flex-direction: column; + gap: 8px; + max-width: 360px; +} +#toasts .toast { + background: var(--bg-3); + border: 1px solid var(--line-2); + border-left: 3px solid var(--info); + border-radius: var(--radius); + padding: 10px 14px; + font-size: 12px; + box-shadow: 0 8px 24px rgba(0,0,0,0.5); + animation: slide-in 0.2s ease; +} +#toasts .toast.ok { border-left-color: var(--ok); } +#toasts .toast.err { border-left-color: var(--err); } +#toasts .toast.warn { border-left-color: var(--warn); } +@keyframes slide-in { from { transform: translateX(20px); opacity: 0; } to { transform: translateX(0); opacity: 1; } } + +/* responsive */ +@media (max-width: 900px) { + .topbar nav a { padding: 6px 8px; font-size: 11px; } + .topbar .brand .label { display: none; } +} diff --git a/ui/app.js b/ui/app.js new file mode 100644 index 0000000..370fcde --- /dev/null +++ b/ui/app.js @@ -0,0 +1,314 @@ +/* ============================================================ + Fester UI — Shared application shell & utilities + Used by: index.html, ui/live_dag.html, ui/replay.html + Exposes a single global: window.Fester + ============================================================ */ + +(function () { + 'use strict'; + + const Fester = { + ws: null, + wsUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws', + wsDebuggerUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws-debugger', + wsTargetsUrl: (location.protocol === 'https:' ? 'wss://' : 'ws://') + location.host + '/ws-targets', + wsState: 'connecting', + _subs: new Set(), + _reconnectTimer: null, + _reconnectDelay: 800, + }; + + /* ---------- Event subscription ---------- + callback(event) — event is the parsed JSON object from /ws + returns an unsubscribe function */ + Fester.subscribe = function (cb) { + Fester._subs.add(cb); + return () => Fester._subs.delete(cb); + }; + + function dispatch(event) { + Fester._subs.forEach(cb => { + try { cb(event); } catch (e) { console.error('subscriber error', e); } + }); + } + + /* ---------- WebSocket with auto-reconnect ---------- */ + function setConnState(state) { + Fester.wsState = state; + document.querySelectorAll('[data-conn]').forEach(el => { + el.classList.remove('online', 'offline', 'connecting'); + el.classList.add(state); + const txt = el.querySelector('[data-conn-text]'); + if (txt) txt.textContent = state.toUpperCase(); + }); + } + + function connect() { + setConnState('connecting'); + try { + const ws = new WebSocket(Fester.wsUrl); + Fester.ws = ws; + + ws.onopen = () => { + setConnState('online'); + Fester._reconnectDelay = 800; + Fester.toast('Connected to Fester event stream', 'ok'); + }; + + ws.onmessage = (msg) => { + try { + const event = JSON.parse(msg.data); + dispatch(event); + } catch (e) { + console.error('bad ws payload', e, msg.data); + } + }; + + ws.onerror = (e) => { + console.warn('ws error', e); + }; + + ws.onclose = () => { + setConnState('offline'); + if (!Fester._reconnectTimer) { + Fester._reconnectTimer = setTimeout(() => { + Fester._reconnectTimer = null; + connect(); + }, Fester._reconnectDelay); + Fester._reconnectDelay = Math.min(Fester._reconnectDelay * 1.6, 8000); + } + }; + } catch (e) { + setConnState('offline'); + console.error('ws connect failed', e); + } + } + + Fester.connect = connect; + + Fester.send = function (obj) { + if (Fester.ws && Fester.ws.readyState === WebSocket.OPEN) { + Fester.ws.send(JSON.stringify(obj)); + return true; + } + return false; + }; + + /* ---------- REST helpers ---------- */ + Fester.api = async function (path, opts = {}) { + const res = await fetch(path, { + headers: { 'Content-Type': 'application/json' }, + ...opts, + }); + if (!res.ok) throw new Error(`${res.status} ${res.statusText}`); + const ct = res.headers.get('content-type') || ''; + return ct.includes('application/json') ? res.json() : res.text(); + }; + + /* ---------- State → color helpers (mirrors backend fester.js logic) ---------- */ + Fester.heatColor = function (heat) { + if (heat == null) return 'var(--fg-3)'; + if (heat < 30) return 'var(--heat-0)'; + if (heat < 60) return 'var(--heat-1)'; + if (heat < 80) return 'var(--heat-2)'; + return 'var(--heat-3)'; + }; + + Fester.stateColor = function (state) { + switch ((state || '').toLowerCase()) { + case 'done': case 'ok': case 'success': return 'var(--ok)'; + case 'running': case 'active': return 'var(--warn)'; + case 'failed': case 'error': return 'var(--err)'; + case 'cache': case 'hit': return 'var(--info)'; + default: return 'var(--fg-3)'; + } + }; + + Fester.stateBadge = function (state) { + const s = (state || 'pending').toLowerCase(); + const map = { + done: 'ok', ok: 'ok', success: 'ok', + running: 'warn', active: 'warn', + failed: 'err', error: 'err', + cache: 'info', hit: 'info', + pending: '', queued: '', + }; + const cls = map[s] || ''; + return `${state || 'pending'}`; + }; + + /* ---------- Toasts ---------- */ + Fester.toast = function (msg, kind = 'info', ms = 3200) { + let host = document.getElementById('toasts'); + if (!host) { + host = document.createElement('div'); + host.id = 'toasts'; + document.body.appendChild(host); + } + const el = document.createElement('div'); + el.className = `toast ${kind}`; + el.textContent = msg; + host.appendChild(el); + setTimeout(() => { + el.style.transition = 'opacity 0.3s'; + el.style.opacity = '0'; + setTimeout(() => el.remove(), 300); + }, ms); + }; + + /* ---------- Time formatting ---------- */ + Fester.fmtTime = function (ts) { + if (!ts) return '—'; + const d = new Date(ts * 1000 || ts); + const pad = n => String(n).padStart(2, '0'); + return `${pad(d.getHours())}:${pad(d.getMinutes())}:${pad(d.getSeconds())}`; + }; + + Fester.fmtRel = function (ts) { + if (!ts) return '—'; + const diff = Math.max(0, (Date.now() - (ts * 1000 || ts)) / 1000); + if (diff < 60) return Math.floor(diff) + 's ago'; + if (diff < 3600) return Math.floor(diff / 60) + 'm ago'; + return Math.floor(diff / 3600) + 'h ago'; + }; + + /* ---------- Topbar shell ---------- + Injects shared topbar so every page has consistent nav + WS status. + Call Fester.mountShell('dashboard') etc. */ + Fester.mountShell = function (active) { + const pages = [ + { id: 'dashboard', href: '/index.html', label: 'Dashboard' }, + { id: 'live', href: '/ui/live_dag.html', label: 'Live DAG' }, + { id: 'replay', href: '/ui/replay.html', label: 'Replay' }, + { id: 'sessions', href: '/ui/sessions.html', label: 'Sessions' }, + { id: 'metrics', href: '/ui/metrics.html', label: 'Metrics' }, + { id: 'cause', href: '/ui/cause.html', label: 'Cause Graph' }, + { id: 'timeline', href: '/ui/timeline.html', label: 'Timeline' }, + { id: 'debugger', href: '/ui/debugger.html', label: 'Debugger' }, + ]; + const links = pages.map(p => + `${p.label}` + ).join(''); + + const shell = document.createElement('div'); + shell.className = 'topbar'; + shell.innerHTML = ` +
+ + FESTER +
+ +
+ + CONNECTING +
+
+ +
+ `; + document.body.prepend(shell); + + // Start health polling + Fester._startHealthPolling(); + }; + + /* ---------- Health polling ---------- */ + Fester._healthTimer = null; + Fester._startHealthPolling = function () { + if (Fester._healthTimer) return; + const tick = async () => { + try { + const h = await Fester.api('/api/health'); + const el = document.querySelector('[data-health]'); + if (el) { + const subs = h.bus_subscribers || 0; + const ws = h.ws_clients || 0; + const builds = h.builds_known || 0; + const events = h.timeline_events || 0; + el.innerHTML = ` + + v${h.version} + · + ${ws}ws + · + ${builds}b + · + ${events}e + `; + el.style.color = 'var(--fg-1)'; + } + } catch (e) { + const el = document.querySelector('[data-health]'); + if (el) { + el.innerHTML = `unreachable`; + } + } + }; + tick(); + Fester._healthTimer = setInterval(tick, 5000); + }; + + /* ---------- FesterTargets global ---------- + The missing API referenced by live_dag.html. + Maintains an in-memory snapshot of nodes + edges, + notifies subscribers on update. */ + const snapshot = { nodes: [], edges: [], byId: {} }; + const targetSubs = new Set(); + + Fester.targets = { + get() { return snapshot; }, + + onUpdate(cb) { + targetSubs.add(cb); + // immediately emit current snapshot + try { cb(snapshot); } catch (e) { console.error(e); } + return () => targetSubs.delete(cb); + }, + + update(patch) { + if (patch.nodes) { + patch.nodes.forEach(n => { + const existing = snapshot.byId[n.id]; + snapshot.byId[n.id] = existing ? { ...existing, ...n } : n; + }); + snapshot.nodes = Object.values(snapshot.byId); + } + if (patch.edges) { + // merge by from+to + const seen = new Set(snapshot.edges.map(e => e.from + '→' + e.to)); + patch.edges.forEach(e => { + const k = e.from + '→' + e.to; + if (!seen.has(k)) { snapshot.edges.push(e); seen.add(k); } + }); + } + targetSubs.forEach(cb => { + try { cb(snapshot); } catch (e) { console.error(e); } + }); + }, + + reset() { + snapshot.nodes = []; + snapshot.edges = []; + snapshot.byId = {}; + targetSubs.forEach(cb => cb(snapshot)); + }, + }; + + // legacy alias used by live_dag.html + window.FesterTargets = Fester.targets; + + /* ---------- Auto-init on DOMContentLoaded ---------- */ + document.addEventListener('DOMContentLoaded', () => { + // mount shell if a marker exists + const marker = document.querySelector('[data-fester-shell]'); + if (marker) { + Fester.mountShell(marker.getAttribute('data-fester-shell')); + } + // auto-connect WS unless page opts out + if (!document.body.hasAttribute('data-no-ws')) { + connect(); + } + }); + + window.Fester = Fester; +})(); diff --git a/ui/cause.html b/ui/cause.html new file mode 100644 index 0000000..073701a --- /dev/null +++ b/ui/cause.html @@ -0,0 +1,292 @@ + + + + + +Fester — Cause Graph + + + + + +
+ + +
+

Lookup

+
+
+ +
+
+ + +
+ +
+ +

Blast Radius

+
+ + +
+
+ +
+ +

Causal Chain

+
    +
  • Enter a node name and click Explain
  • +
+ +
+ +

Recent Events

+ +
+
+
+ + +
+
+

Cause Graph

+ idle +
+
+
+ +
+
+
+ +
+ + + + + diff --git a/ui/debugger.html b/ui/debugger.html new file mode 100644 index 0000000..06376ee --- /dev/null +++ b/ui/debugger.html @@ -0,0 +1,244 @@ + + + + + +Fester — Debugger + + + + + +
+ + +
+
+

Debugger

+ idle +
+
+
+ + + + +
+ +
+ +

State

+
+ +
+ +

Next Action Preview

+
+
+
+ + +
+
+

Execution Timeline

+ step 0 +
+
+
+
+
+ + +
+

Inspector

+
+
Click a timeline row to inspect.
+
+
+ +
+ + + + + diff --git a/ui/live_dag.html b/ui/live_dag.html new file mode 100644 index 0000000..7682881 --- /dev/null +++ b/ui/live_dag.html @@ -0,0 +1,403 @@ + + + + + +Fester — Live DAG + + + + + +
+
+ + + + + + + + +
+ + + + + + +
+ + + +
+
Pending
+
Running
+
Done
+
Failed
+
Cache hit
+
Critical path
+
+
+ + + + + diff --git a/ui/metrics.html b/ui/metrics.html new file mode 100644 index 0000000..8b816d3 --- /dev/null +++ b/ui/metrics.html @@ -0,0 +1,257 @@ + + + + + +Fester — Metrics + + + + + +
+
+
+
Nodes Online
+
+
+
+
+
Avg Heat
+
+
+
+
+
Active Jobs
+
+
+
+
+
Instability
+
+
+
+
+ +
+
+

Heat Trend (last 60s)

+ +
+
+

Active Jobs Trend (last 60s)

+ +
+
+ +
+
+

Per-Node Breakdown

+ +
+
+
+
+
+
+ + + + + diff --git a/ui/replay.html b/ui/replay.html new file mode 100644 index 0000000..dcaaf3c --- /dev/null +++ b/ui/replay.html @@ -0,0 +1,573 @@ + + + + + +Fester — Replay Observatory + + + + + +
+ + +
+

Replay Controls

+
+
+ + +
+ + +
+ +
+ + + + + + +
+ + +
+
+
+ +
+ 0 / 0 + +
+ + +
+ + + + +
+ +
+ + +
+
+ + +
+
+

Execution Graph

+
+ idle +
+
+
+
+ + + + + + + +
+
+
+ + +
+

Inspector

+
+
Click a node to inspect.
+
+
+ +
+ + +
+ Failure Autopsy: + + + + Traces dependency chain backward + last scheduler decision +
+
+
+
+ + + + + diff --git a/ui/sessions.html b/ui/sessions.html new file mode 100644 index 0000000..74fae08 --- /dev/null +++ b/ui/sessions.html @@ -0,0 +1,347 @@ + + + + + +Fester — Sessions + + + + + +
+
+

Sessions

+ + +
+ +
+
+

Replay Sessions

+ +
+
+ + + + + + + + + + + + + +
Session IDSourceEventsCreated
Loading…
+
+
+ +
+

Recent Builds

+
+ + + + + + + + + + + + + + + +
Build IDCommandDirStatusStartedDurationActions
Loading…
+
+
+ +
+
+

Active Action Sessions (tmux)

+ +
+
+ + + + + + + + + + + + + +
SessionStartedCurrent CmdAttachedActions
Loading…
+
+
+ +
+
+

Storage Snapshots (qcow2)

+ +
+
+ + + + + + + + + + + + +
NameSizeCreatedVirtual Size
Loading…
+
+
+
+ + + + + + + + diff --git a/ui/timeline.html b/ui/timeline.html new file mode 100644 index 0000000..61423f0 --- /dev/null +++ b/ui/timeline.html @@ -0,0 +1,194 @@ + + + + + +Fester — Timeline + + + + + +
+ + +
+
+

Nodes

+ +
+
+
+
Loading…
+
+
+
+ + +
+
+

Events for

+
+ 0 + +
+
+
+
+
Select a node
+
+
+
+ +
+ + + + +