fester/CHANGELOG.md

Changelog

All notable changes to Fester are documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

• BTC 0.4.0 multi-arch cross-compilation support — backend/toolchain/btc.py
  • probe_btc() now loads BTC manifest JSON sidecars for structured metadata (target_id, arch, clib, triple, cross_mode, target_march)
  • btc_build_env() supports target parameter to select specific cross-compiler golden images when multiple are installed on a node
  • list_btc_targets() enumerates all available BTC golden images with parsed SYS_LABEL and manifest data — usable by the Fester API
  • ISA flag table (_ISA_FLAGS) for AVX512, AVX2, SSE4_2, NEON, MIPS32, TILE
  • SSE4_2 ISA tier added for Intel Atom and AMD APU targets that lack AVX support
  • 19 cross-compilation targets: Intel HEDT/Server (5), AMD Ryzen/EPYC (4), AMD APU (4: apu-zn1 through apu-zn4), Intel Atom (4: silvermont, goldmont, tremont, sierraforest), embedded (mipselr2, armv7, tilegx)
  • SYS_LABEL parser updated for BTC 0.4.0 cross format: DCOSNET-{FAMILY}-{TARGET_ID}-{ISA}-CROSS
  • Per-node BTC target selection via node_config.btc.target
  • New env vars: BTC_TARGET_ID, BTC_CROSS, BTC_CLIB, BTC_TARGET_TRIPLE
• Shared Content-Addressable Store (CAS) API — backend/storage/cas_api.py
  • PUT /api/cas/{sha256} — store an artifact (SHA-256 verified server-side)
  • GET /api/cas/{sha256} — retrieve an artifact (streamed file download)
  • HEAD /api/cas/{sha256} — check existence (returns metadata JSON)
  • DELETE /api/cas/{sha256} — remove an artifact
  • GET /api/cas/ — list all artifacts (paginated, filterable by target)
  • GET /api/cas/stats — cache statistics (hits, misses, utilization %)
  • SQLite-backed index for fast listing, LRU eviction, and cross-query filtering
  • btrfs reflink support for zero-copy artifact storage
  • DAG-aware cache integration in PipelineEngine — checks CAS before each action
• BTC.sh toolchain integration — backend/toolchain/btc.py
  • probe_btc() — detect BTC golden images, parse SYS_LABEL
  • btc_build_env() — return BTC-aware CC/CXX/CFLAGS/LDFLAGS
  • verify_btc_stamp() — read .note.BTC ELF note + xattr stamps
  • apply_btc_stamp() — inject forensic stamps via assembly + objcopy + xattr
  • merge_compiler_env() — overlay BTC on ccache/distcc env
  • Cross-project compatible: pipe-delimited .note.BTC format, type=1 (NT_VERSION), bare hex hash
• Podman integration — backend/integrations/podman.py (full PodmanManager)
• Firecracker integration — backend/integrations/firecracker.py (full FirecrackerManager)
• Podman/Firecracker executor adapters — backend/executor/adapters.py (run_podman, run_firecracker)
• Runtime router updated — backend/executor/runtime_router.py routes podman and firecracker runtimes
• Documentation: README.md, quickstart.md, CONTRIBUTING.md, CHANGELOG.md
• Tooling: bootstrap.sh, run.sh, pyproject.toml, .gitignore
• Docker support (Dockerfile)
• Screenshots for all 8 UI pages

[0.3.0] — 2026-06-28

Added — Storage layer

• SQLite-backed persistence for builds, sessions, events, and node states
• Btrfs CoW reflink snapshots (backend/storage/btrfs_cas.py)
• QCOW2 workspace freezing with real mount + rsync (backend/storage/qcow2_freeze.py)
• tmpfs workspace acceleration
• Storage status + config + freeze + reflink + snapshots endpoints

Added — tmux integration (E9)

• TmuxManager with session create/list/inspect/capture-output/kill
• runtime: "tmux" action runtime — runs actions in detached tmux sessions
• Live output viewer in Sessions page UI
• /api/actions/active, /api/actions/{name}/tmux/output, /tmux/kill endpoints

Added — mosh/ssh shell attach (E10)

• MoshManager with command builder (mosh-via-SSH-bootstrap)
• /api/nodes/{name}/shell GET + POST endpoints
• "⌘ Shell" button on Dashboard node rows (copies command to clipboard)

Added — Real engine wiring

• BuildRunner wraps PipelineEngine with build lifecycle + persistence
• Real DAG execution with proper cwd + env passing
• Failure propagation: skipped actions on failed deps
• Critical path computation + tagging in events
• Debugger pause/resume/step actually controls running engines

Added — Cause graph + blast radius

• /api/propagation/{action} — forward blast radius computation
• /api/cause/explain/{node} — causal chain for a node
• Blast Radius panel in Cause Graph UI page
• Failure autopsy with proper session ID correlation

Added — Storage + target catalog

• /api/targets reads from backend/targets/catalog.py (6 systems: Gentoo, Buildroot, OpenWrt, ALFS, SourceMage, Lunar)
• /api/targets/arches + /api/targets/runtimes endpoints

Added — Health + metrics

• /api/health endpoint (version, bus_subscribers, ws_clients, builds_known, timeline_events)
• Live health indicator in topbar (all pages)
• Prometheus metrics aligned with grafana config names (fester_node_cpu, fester_pipeline_actions_total, etc.)
• Pipeline action counters + cache hit counters wired via bus subscriber

Added — UI enhancements

• New Timeline page (per-node event drill-down)
• Node policy dropdowns (preferred/avoid/neutral) on Dashboard
• Per-node "Probe" buttons + "Probe All" button
• Build cancel + replay buttons on Sessions page
• Event filter chips on Dashboard (toggle by event type)

Added — CLI overhaul

• 35+ subcommands (was 11)
• Nested subcommands: node {list,set-policy,probe,probe-all}, cause {explain,trace,events}, replay {start,step,reset}, debugger {state,pause,resume,step}
• New: build, builds, build-info, cancel, release, targets, toggle-target, blast, timeline, rewind, autopsy, policy-list, health
• --watch flag on build command (streams events via WebSocket)

Fixed — Critical backend bugs

• api/api.py order bug (bus.subscribe() before bus = EventBus())
• 7 broken _endpoint symbol imports
• cause_graph.attach_bus + emit_debug defined at module scope (zero indentation)
• Three missing files: timeline_store.py, failure_autopsy.py, failure_propagation.py
• Two Python module/package name collisions (scheduler.py vs scheduler/, nodes.py vs nodes/, cache.py vs cache/)
• execute_action passing empty {} as cwd → TypeError
• Build env never reaching subprocess
• Debugger/pipeline-control _ENGINE always None
• failure_propagation looking for event.data.deps (legacy shape)
• Autopsy session ID mismatch
• /api/release was a pure stub
• /api/policy/set didn't persist
• /ws-debugger returned canned data, ignored commands
• forgejo.on_push_event wrong PipelineEngine arity
• pipeline/feedback.py PolicyEngine() with no args

Fixed — Standardization

• ~20 bare imports converted to backend.* prefix
• Two parallel event taxonomies unified (types.py re-exports from schema.py)
• MinioCache.__init__ made lazy (was crashing on import if MinIO unreachable)
• cache.py graceful fallback for non-writable paths
• roles_store.py graceful fallback for non-writable /etc/fester/
• FesterEvent made node/action/state truly optional (had Optional[str] type but no default)

[0.2.0] — 2026-06-27

Added — Real FastAPI backend

• backend/main.py — FastAPI app with 50+ routes
• All routers wired: replay, autopsy, timeline, debugger, pipeline_control, nodes, metrics, cause
• WebSocket hub with /ws, /ws-targets, /ws-debugger
• Ambient loop for node probing + drift
• SQLite storage layer (backend/storage/sqlite_db.py)
• Node probe module (backend/nodes/probe.py)
• BuildRunner (backend/pipeline/runner.py)

Added — UI overhaul (7 pages)

• Dashboard (index.html) — 3-column layout with cluster stats, live stream, quick actions
• Live DAG (ui/live_dag.html) — layered Sugiyama-lite layout, SVG edges, click-to-inspect
• Replay (ui/replay.html) — timeline scrubber, play/pause/step, structured inspector
• Sessions (ui/sessions.html) — build history + replay links
• Metrics (ui/metrics.html) — live charts + per-node cards
• Cause Graph (ui/cause.html) — radial graph visualization
• Debugger (ui/debugger.html) — step-through execution control

Added — Shared infrastructure

• style.css — full design system (dark theme tokens, panels, badges, buttons, scrollbars)
• ui/app.js — shared shell (WS auto-reconnect, FesterTargets global, toasts, topbar nav)
• cockpit/fester-module/targets.js — fixed to expose FesterTargets global

Fixed

• live_dag.html was broken (referenced undefined FesterTargets)
• style.css was 0 bytes
• index.html was a 30-line throwaway

[0.1.0] — Initial commit

Added

• Basic project structure
• Backend modules (many stubbed)
• Cockpit module
• Install scripts
• CHEATSHEET.md