# QCrows All the Way Down: One VM-Image Format, Three Projects, Zero Trust Between Them *How SysDeck 0.4.6 became a first-class consumer of the in-house QCrows VM container image format — parsing image manifests straight out of tar.gz in memory, re-implementing the master toolchain's verification checks without extracting a single byte to disk, and rendering real image metadata in the Kata panel. The same format is produced by cockpit-kata's qcrows-pack and by AI-LSC's stack exporter; SysDeck verifies both with the same code path.* A QCrows image (`.qcrows`, spec v0.2.0) is a self-describing bundle for VM-based container runtimes: one tarball carrying a `metadata.toml` manifest, a Cockpit `menu.toml` entry, a `hashes.sha256` integrity list, the guest `rootfs.tar.gz`, and — mandatory since v0.2 — the guest kernel and its `.config`. The format's master implementation lives in the cockpit-kata project (`qcrows-pack`, `qcrows-verify`, `qcrows-inspect`, `qcrows-export`), and a second producer now exists: AI-LSC, the local AI stack manager, exports its active tool stack directly as `.qcrows` archives. SysDeck's Kata panel has listed whatever sat in `/usr/share/sysdeck/kata/qcrows/` since the v0.0.43 production rewrite — but until now it only ever looked at filenames. A bundle could be a Kata-tuned Alpine image or someone's holiday photo archive; the panel treated both as "a file with a size." v0.4.6 closes that gap by making the bridge a real format consumer. `qcrows-list` opens every archive in the directory with Python's `tarfile` in `r:*` mode and reads `metadata.toml` and `menu.toml` into memory — no extraction, no temp directories, nothing executable ever materialized from an image the operator hasn't chosen to trust. The TOML parser is deliberately small: QCrows metadata is a flat shape of `[ section ]` headers and scalar/array values, so ~40 lines of section-aware parsing cover it — and being section-aware is not cosmetic. The master `qcrows-inspect` greps first-matching keys, which reports the kernel's `size_mb` on the rootfs row of its own output; SysDeck's parser tracks the current section and gets `kernel.size_mb` and `rootfs.size_mb` right on the same files the master tool mislabels. The new `qcrows-verify` subcommand mirrors the master verifier check-for-check — required files, rootfs presence, kernel binary with a real magic test (bzImage's `HdrS` at offset 0x202, or ELF), the kernel `.config` with the five Kata-required options, and a full `sha256sum -c`-style walk of every entry in `hashes.sha256` — with two deliberate divergences in *how*, not *what*. First, the kernel check classifies bytes directly instead of shelling out to `file`: same verdicts, no subprocess. Second, the semantics are calibrated to the master's own exit codes: a missing kernel is a hard failure, but a shortfall of `CONFIG_VSOCKETS=y`-style options is a *warning*, exactly as in qcrows-verify, so a host-kernel-built bundle reports ok with advisories rather than failing a check the master would pass. The hash walk streams members in 1 MiB chunks and caps parsed text members at 1 MiB, header counts at ten thousand — tar-bomb posture inherited from the bridge's CVE-lesson hardening style. Everything runs through the same security gate the rest of the Kata bridge uses: filenames validated against the strict allowlist regex, paths resolved with `realpath` containment under the bundle directory, output sanitized before it reaches JSON. The panel card grew accordingly — image name and version, kernel version and format, architecture, hypervisor compatibility list, per-row Verify and Inspect actions whose output lands in the existing operation-output card, all rendered through `textContent`/`escapeHtml` per the panel's no-innerHTML rule. Legacy non-QCrows tarballs that happen to sit in the directory degrade to a stat-only row with an honest "(not a QCrows archive)" label instead of being silently miscounted. The test discipline is where the three-project story pays off. The new suite builds synthetic spec-conformant archives in a temp directory — including one whose `rootfs.tar.gz` is byte-flipped *after* its hashes are computed, which is what an integrity violation actually is — and additionally exercises the bridge against real images from both external producers: an AI-LSC stack export and a bundle from cockpit-kata's own `qcrows-pack`. Both parse, both verify clean, the tampered one fails on exactly the checksum check. One image format, two independent producers, one consumer that trusts none of them until the hashes say otherwise — which is the whole point of a self-describing format. --- # SysDeck and the Cockpit Inheritance: One Module Catalog, Two Frontends, and the Case for Real Host State *A technical walkthrough of how SysDeck 0.4.4 — a standalone Linux operations console (thirty-one panels, one Bun process, no Cockpit required) that also ships as a drop-in Cockpit plugin suite — authenticates against the host's own Unix accounts through PAM exactly the way Cockpit does, keeps every module working in both frontends, holds itself to a compiler-enforced no-demo contract where every panel reads real host state or fails honestly, and resolves every fork in the road with a step-down through the system's real tools: ten package managers from pacman to sorcery, nftables before iptables, lm-sensors before raw sysfs. Grounded in the source code, not in marketing claims.* There is a particular category of Linux server console that treats the operator as an implementation detail. The shared-password panels — one login for whoever holds the string, no per-user audit trail, no privilege boundary between reading a sensor list and rewriting the firewall — go back to the Webmin era, and the pattern keeps getting reinvented because it is easy to build. A second category is softer but just as corrosive: the console that demos beautifully and operates poorly, because half its panels are wired to fixture data that was never meant to see a production host. Both categories share a root cause: the console stopped being a view onto the system and became a self-contained application with opinions of its own. Cockpit solved the first problem a decade ago and has kept solving it since: the login is the host's own PAM stack, the session belongs to a real Unix account, privileged operations ride a polkit-authenticated superuser channel, and every page is a thin view onto systemd, the firewall, and the package manager as they actually are. SysDeck is built inside that inheritance. Its default shape is the standalone Next.js console (`web/` in the master tarball) that runs with no Cockpit installed at all — one process, a bundled SQLite store, an optional Fester sidecar — and, for hosts that already run Cockpit, the same catalog ships as a cockpit-native plugin suite: static HTML+JS+CSS plus a Python bridge package installed under `/usr/share/cockpit/sysdeck-*/`, discovered automatically by the cockpit-bridge. The two frontends share one module catalog of twenty-seven domain modules — containers, firewall, network security, integrity auditing, service mesh, encryption vaults, fleet compute, Kata Containers, firmware, image building, mining, theme engine, hardware authentication, build orchestration, monitoring, the Glances system monitor, sensors, benchmarking, packages, policy and permissions, database control, media servers, photo managers, remote filesystems, the service/port editor, the AI gateway panel, plus a third-party Cockpit-module installer — and one contract: real host state or an honest empty. The web edition's TypeScript bridge layer (`web/src/lib/sysdeck/bridge/*.ts`) mirrors the cockpit-side Python bridges (`bridge/*.py`) command-for-command; the packages bridge runs the same ten-manager step-down on both sides, and the sensors bridge runs the same `sensors -j` → sysfs chain. The inheritance also runs both directions: the web console scans `/usr/share/cockpit` and `/usr/local/share/cockpit` (extra roots via `SYSDECK_COCKPIT_SCAN`) and loads every installed third-party Cockpit module into its own sidebar, so cockpit-machines or a 45Drives plugin appears in the Next.js console the same way SysDeck's own modules appear inside Cockpit. This post is a technical walkthrough of how those pieces fit together, grounded in the source code rather than marketing claims. It is organized around the four decisions that most heavily shape SysDeck's identity: the decision to authenticate with the host's own Unix accounts through PAM rather than a private user database, the decision to keep one module catalog behind two frontends with real parity between them, the decision to make "real host state" a compiler-enforced contract instead of a reviewer's good intentions, and the decision to resolve every fork in the road — package manager, firewall backend, sensor source, privilege path — with an explicit step-down through the system's real tools rather than an abstraction layer that hides them. --- ## The Authentication Model: Unix Accounts via PAM, the Way Cockpit Does It The load-bearing decision in the web edition is that it owns no account system. `web/scripts/pam-auth.py` is a stdlib-only ctypes shim that loads `libpam` directly and hands the username and password to the host's PAM stack, the same mechanism a Cockpit login uses. The password arrives on stdin as one JSON document — never argv, which is world-readable through `/proc//cmdline` — and the PAM conversation callback answers only `PAM_PROMPT_ECHO_OFF` and `PAM_PROMPT_ECHO_ON` messages, acknowledging anything else with an empty reply so an exotic stack cannot fish for extra data. `pam_acct_mgmt()` runs after authentication, because an account that is expired, locked, or outside its allowed login hours must not pass just because its password was right. The service name steps down the same way the rest of the system does: `SYSDECK_PAM_SERVICE` or the request's `service` field (default `sysdeck`) first, so an operator can ship a tailored `/etc/pam.d/sysdeck` stack; when that stack does not exist the helper falls back to `login` — the stack the console TTY uses, which is what "log in like at the console" means on a stock distro. Sessions are user-bound HMAC tokens, `v2...` in `web/src/lib/sysdeck/session.ts`, minted from a per-install secret and bound to one account name; the login route rate-limits attempts per IP *and* per username, and audits the actor it actually authenticated. `SYSDECK_AUTH_MODE` selects the deployment posture as a step-down — `pam` (the Cockpit default; run the service as root so any Unix account can sign in), `pam+local` (PAM first, locally-stored scrypt-hashed console accounts as fallback, which works unprivileged because `unix_chkpwd` serves the invoking uid), or `local` (console accounts only). This is the same honesty Cockpit applies to its own root posture: the docs state plainly that arbitrary-user PAM verification requires the service to run as root, and the operator picks the rung that matches the deployment. ## One Module Catalog, Two Frontends The catalog is a single declarative source: `scripts/generate-plugins.py` enumerates the modules, and adding one means appending an entry and dropping a plugin directory — no other wiring. On the Cockpit side each plugin is a directory under `plugins/` with a `manifest.json` registered under the `index` menu key, and its panels reach the system exclusively through the Python bridges, invoked as `python3 /usr/lib/sysdeck/bridge/.py [args]`. Privilege rides the cockpit superuser channel — the JS passes `{ superuser: 'try' }` to `cockpit.spawn`, the operator authenticates once through polkit against the shipped `org.sysdeck.policy` action domains (`packages.modify`, `firewall.modify`, `builder.modify`, and so on), and the bridge then runs as root. There is no sudo shell-out from JavaScript anywhere in the suite. The web edition is a full Next.js console over the same catalog. Its bridge layer replaces the Python helpers with TypeScript modules that exec the same system commands — `ss`, `nft`, `podman`, `systemctl`, the package managers — and speak the same JSON envelope. Mutations there run only with real privilege (root or passwordless `sudo -n`) and, since 0.4.3, only from an admin session (wheel/sudo/adm or uid 0), with `SYSDECK_MUTATIONS=any` documented for single-operator consoles where every login *is* the operator. Module-level parity is treated as a release property, not an aspiration: the 0.4.3 QA pass explicitly hardened the Python side to match the web side (the sensors chain, the `dnf check-update` exit-100-is-data rule, timeouts on every spawn), and the 0.4.4 pass did the same for the packages bridge — the ten-manager step-down, the detection probes, and the parser fixtures now live in both trees. The third-party module installer completes the loop in both directions: it pulls 45Drives Navigator, cockpit-pacman, cockpit-identities, and friends on demand with license, developer, source URL, and homepage shown inline next to a one-click install button, and the web console's cockpit-module detection (`bridge/cockpitmodules.ts`) reports the scanned roots honestly in its envelope when no Cockpit tree exists — `unavailable`, with a note saying exactly what was scanned and how to extend it. ## Real Host State or an Honest Empty: The Zero-Demo Contract The most important type in the web edition is three strings wide: ```ts export type DataSource = 'live' | 'hybrid' | 'unavailable' ``` Every bridge envelope carries a `source` field, and `'demo'` is not a member of the union. A panel cannot fabricate rows without lying in a field the type system refuses to produce — the contract is checked by the compiler, not by reviewer vigilance. `'live'` means the data was read from this host; `'hybrid'` means live data merged with an operator-managed registry (the service/port editor cross-references `ss -H -tlnp` output against its `SERVICES_REGISTRY` of known services); `'unavailable'` means the backend is absent and the panel says so, with install guidance, instead of inventing an inventory. The concrete behavior this forces is worth spelling out, because it is the difference between a console and a demo. Sensors reads the canonical source — `sensors -j` from lm-sensors — with the raw sysfs collectors (`/sys/class/hwmon`, thermal zones) as the dependency-free fallback, and a host with no sensors gets an empty list, never a made-up chip set. Network-security bans are enforced for real: banning an address loads an atomic nftables batch (`table inet sysdeck`, a `blacklist` set with 30-day timeouts) or an iptables `INPUT DROP` rule when only iptables exists, the ban list merges the live fail2ban state when fail2ban runs, and unbanning a fail2ban row executes the real `fail2ban-client set unbanip` and reports its actual exit code. `dnf check-update` exiting 100 (updates exist) is parsed as data, because treating it as failure would fabricate an empty update list on every RPM host. A firewall template apply marks the ruleset `ACTIVE` only after the apply exits zero. A database backup is a real `dump | gzip > file` binary-safe pipeline, not a text buffer with a `.sql.gz` name. Every spawn from every bridge runs under a scrubbed `LC_ALL=C` environment with a hard timeout and an output cap, so parsed output stays locale-stable and a runaway command cannot exhaust the console. ## Ten Package Managers, One Step-Down The packages module is where the step-down philosophy is most visible, because the fork it faces has ten tines. The alternative — a cross-distro abstraction like PackageKit — was rejected for the same reason the SCP browser in a certain Rust SSH client shells out to `ssh`: the abstraction would carry its own daemon, its own policy layer, and its own divergence from what the operator actually types at a root shell. SysDeck wraps the distro's own manager instead: pacman on Arch, emerge on Gentoo, lunar on Lunar, sorcery on SourceMage, xbps on Void, apk on Alpine, zypper on openSUSE, dnf and yum on RPM hosts, apt on Debian. Detection is a `shutil.which` step-down in a fixed order, most specific first, with two corroboration rules that matter: `emerge` only claims the host when `/var/db/pkg` also exists (a Gentoo box always carries the vdb), and Void is probed through `xbps-query` because Void ships no bare `xbps` binary. Each backend then reads the manager's real state — pacman `-Q`, the rpm database for zypper, a direct `/var/db/pkg//-` directory scan for emerge (no emerge invocation needed), `lvu installed` with the `/var/state/lunar/packages` file as the dependency-free fallback, `gaze installed` with `/var/state/sorcery/packages` likewise. The parsers are built to survive the tools' actual output formats: zypper tables are parsed by locating `Name`/`Current`/`Available` columns from the header row, because zypper prefixes its tables with status and repository columns whose count varies by subcommand and release; the emerge update preview anchors its capture *after* the class bracket (`[ebuild U ] cat/pkg-1.2.3 [1.2.2]`), because portage pads the class field with spaces and a looser regex captures the bracket itself and silently drops every row. That failure mode is the exact bug the zero-demo contract exists to prevent — a parser that returns nothing looks identical to "no updates available" unless someone makes the distinction explicit. Where a manager genuinely lacks an operation, the module reports that instead of approximating it: `lvu` has no update-preview subcommand, so the lunar backend returns an honest empty and the summary carries a note — "lunar has no update-preview subcommand — run lunar update to fetch + rebuild" — rather than a count of zero that would read as "all current." Single-module update has no lunar equivalent, so the mutation refuses with the real instruction; every other manager maps to its exact argv (`pacman -S --noconfirm`, `emerge --unmerge`, `cast`, `dispel`, `xbps-remove -y`, `zypper --non-interactive install`, …) from one `MUTATION_CMDS` table shared by install, remove, update, update-all, and the dry-run preview. Package names pass an argument-injection guard first — no leading dash (a name like `--config=` becomes a manager *option*), no URL scheme (dnf would fetch a remote RPM), no whitespace, bounded length — because the name travels to the system package manager as one argv element, which makes this argument injection, not shell injection. ## The Firewall: Seven Topologies, the Live Ruleset, and Privilege on stdin The firewall module treats the host's kernel firewall as the only source of truth and the panel as a view. A live-ruleset tab reads the actual active ruleset — `nft -j list ruleset`, or `iptables-save` when only iptables exists — rendered straight from the binary and refreshed every 15 seconds. The template catalog ships seven full topologies (public-webserver, vps-webserver, ai-llm, remote-admin, no-services, cilium, and the unified zone firewall sysdeck-fw, which takes its zone model from Smoothwall Express and IPFire), each a real script the operator can read before running. The apply path is where the privilege discipline lives. Rulesets ride stdin: the bridge pipes the synthesized nft or iptables script to `nft -f -` / `iptables-restore` directly, so no predictable `/tmp` file exists to hijack — the same hardening the shipped templates carry via `mktemp` staging. A dry-run renders the exact script the apply would execute, verbatim, so preview and execution cannot diverge. Rule comments are injection-guarded and escaped at render time, because a comment containing a quote would otherwise escape its string literal inside the generated script. The ruleset is marked `ACTIVE` only after the apply exits zero. And the network-security module's ban/unban operations share this exact privilege chain on the web side — root or `sudo -n`, honest refusal otherwise — while the cockpit side gates the same operations through the polkit `firewall.modify` domain. Secrets follow the same stdin rule everywhere: the smartcard PIN in the hardware-auth module never touches argv, so `/proc//cmdline` cannot leak it to other local users. ## Performance Without Fabrication A console that polls the host every ten seconds is a console that can DDoS itself, so the web edition's shared bridge layer (`web/src/lib/sysdeck/bridge/shared.ts`) runs a TTL plus single-flight cache: identical probes inside one window collapse into a single subprocess sweep whose settled result serves every panel, an explicit `invalidateCache()` sweeps it after mutations, and errors are never cached — a failed probe is retried, not enshrined. Binary presence probes (`which`) carry their own short TTL because a missing binary is a slow-moving fact. Every child runs with a scrubbed environment and an 8 MB output cap, and supports stdin for the privileged-write paths above. The step-down rule applies to performance too. The services module rewrote its listener enumeration as an async `ss -H -tlnp` parse with a `/proc` step-down for hosts without ss, replacing a synchronous per-process-per-file walk that blocked the event loop on every poll. Fleet reachability probes fan out in parallel. The Fester journal viewer folds its replay in O(delta) over a WeakMap-keyed running fold instead of re-reducing the whole log on every render. None of this fabricates anything — it changes how often the truth is sampled, never what the truth is. ## Putting It All Together The canonical workflow ties the pieces together: 1. **Install** drops the cockpit tree under `/usr/share/cockpit/` (plugin panels, Python bridges, polkit policies) and optionally unpacks the web edition with its two systemd units; `make check` gates the build on manifest consistency, bridge subcommand coverage, parser unit tests, and version sync across every release surface. 2. **Sign in** with a real Unix account — PAM answers, the session mints a user-bound `v2` token, and the login route's per-IP and per-username rate limits stand guard. 3. **The catalog loads**: twenty-seven SysDeck domain modules in whichever frontend you opened (the web console adds its Overview, runbook, hardware-alerts, and cockpit-modules panels), plus every installed third-party Cockpit module the scanner found under the scan roots. 4. **Every panel probes its real backend** — one cached subprocess sweep per window serves them all; absent backends render honest empties with install hints. 5. **A mutation gates on privilege**: polkit on the cockpit side, an admin session plus root or `sudo -n` on the web side, with rulesets and secrets riding stdin and the executed command reported back verbatim. 6. **The result is auditable**: the envelope's `source` field says where every row came from, dry-runs show exactly what will run, and the firewall panel reads back the kernel's own ruleset rather than its own intentions. Each piece is independently simple — `pam` is `libpam`, the firewall is `nft`, the package managers are the distro's own tools, and the bridges are thin parsers over their output. The value SysDeck adds is not in reimplementing any of them; it is in the catalog that organizes them, the parity that makes every module work identically in Cockpit and in the standalone console, the type-level contract that keeps every panel honest, and the step-down discipline that keeps every fork resolved in favor of the system's real tools. --- *SysDeck 0.4.4 is developed by Jeremy Anderson at dcos.net and released under the MIT license. Cockpit is developed by the Cockpit Project and the larger freedesktop.org community; SysDeck's module model, PAM login posture, and superuser-channel privilege discipline follow Cockpit's design, and the plugin edition runs inside it. The vendored third-party components (klanker-gate by TykoDev, the 45Drives and cockpit-* modules offered through the installer, Fester) are credited with their licenses in `THIRD_PARTY.md`. This article was written in 2026 against the 0.4.4 release.*