corbel/QUICKSTART.md

# Build & Operation Guide

This document is the authoritative build guide and operator reference
for CorbelPurge. For the project overview and detection model, see
[README.md](README.md).

## Prerequisites

| Tool | Version | Purpose |
|---|---|---|
| **Rust** | 1.70+ stable | Compiles the scanner, CLI, and GUI |
| **Python 3** | any | Only needed to regenerate test fixtures |

Install Rust via [rustup](https://rustup.rs/):

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
```

The headless CLI has zero GUI dependencies and no native libraries.
The GUI adds `iced`, `rfd`, and `tokio` (all pure-Rust).

## Step 1 — Get the source

From a release tarball:

```bash
tar xzf corbel-purge-0.4.2.tar.gz
cd corbel-purge-0.4.2
```

From the repository:

```bash
git clone https://git.dcos.net/dcosnet/corbel.git
cd corbel
```

## Step 2 — Build the CLI

```bash
cargo build --release
```

The binary lands at `target/release/corbel-purge`. Verify it works:

```bash
./target/release/corbel-purge --help
```

You should see the usage banner listing `scan`, `scan-dir`, `study`,
and the supported flags (`--workspace`, `--abort-on-threat`, `--quiet`,
`--recursive`, `--preserve-format`, `--rules`, `--cve-db`,
`--no-clean-output`, `--move-clean`).

### Building the GUI (optional)

The iced 0.13 dashboard GUI is behind the `gui` feature flag:

```bash
cargo build --release --features gui --bin corbel-purge-gui
./target/release/corbel-purge-gui
```

### Build profiles

The release profile (in `Cargo.toml`) is tuned for production:

```toml
[profile.release]
opt-level = 3
lto = "thin"
codegen-units = 1
strip = "symbols"
```

For faster debug builds (no optimizations, faster compile):

```bash
cargo build            # debug profile
```

## Step 3 — Run the test suite

```bash
cargo test
```

Expected output: 154 lib tests + 24 pipeline integration tests + 4
zip-bomb defense tests, all passing. The pipeline integration tests
include the corpus regression test
(`benign_realworld_pdf_produces_zero_findings`) which asserts that a
3-page PDF with 13 hyperlinks (kernel.org, linuxfromscratch.org,
github.com/microsoft/vscode, .ru URLs, mailto:, tel:, and URLs
containing "support", "account", "verify" in their paths) produces
zero findings.

### Regenerating test fixtures

The fixtures in `tests/fixtures/` are checked in. Regenerate them
only when changing the parser or scanner behavior:

```bash
pip install pypdf reportlab python-docx
python3 scripts/gen_fixtures.py
python3 scripts/gen_md_epub_fixtures.py
python3 scripts/gen_docx_fixtures.py
python3 scripts/gen_zip_bomb_fixtures.py
python3 scripts/gen_benign_realworld_pdf.py
```

## Step 4 — Scan your first file

```bash
./target/release/corbel-purge scan suspicious_document.pdf
```

If the file contains threats, the summary block prints to stdout and
the pipeline writes:

- A quarantine tarball in `./corbel_quarantine/`
  (`quarantine_<ts>_<sha>.tar.gz`) containing `original.<ext>`,
  `report.json`, `report.md`, and one `.bin` per carved payload
  (plus paired `.hex` and `.info` files for each payload).
- A cleansed Markdown derivative in `./corbel_clean/`
  (`cleansed_<ts>_<sha>.md`).
- Standalone `report_<ts>_<sha>.json` and `.md` for programmatic
  access in `./corbel_quarantine/`.

If the file is clean (zero malicious findings), the pipeline writes:

- A clean-output copy in `./corbel_clean/`
  (`clean_<ts>_<sha>.<ext>`) — the source file byte-for-byte.
- Standalone `report_<ts>_<sha>.json` and `.md` confirming the scan
  ran and the document was clean.

## Step 5 — Try PreserveFormat mode

For a cleaned version that keeps the original format (e.g. a cleaned
`.epub` you can read in an e-reader):

```bash
./target/release/corbel-purge scan research_paper.epub --preserve-format
```

Produces `cleansed_<ts>_<sha>.epub` with malicious entries stripped
from the ZIP container but chapter text preserved. Works for PDF and
DOCX too.

## Step 6 — Pipeline-stage mode

The scanner acts as a pipeline stage: input files flow through and
clean ones end up in the output folder alongside the cleansed
derivatives of malicious ones.

```bash
# Default: copy clean files to ./corbel_clean/
./target/release/corbel-purge scan inbox/file.pdf

# Queue-draining: move (not copy) clean files to output, remove source
./target/release/corbel-purge scan inbox/file.pdf --move-clean

# Reports only, no clean-output copy
./target/release/corbel-purge scan inbox/file.pdf --no-clean-output
```

Downstream processing can then operate on the contents of
`corbel_clean/` without inspecting each file's report — every file
there has been verified clean.

## Step 7 — Scan a directory

```bash
./target/release/corbel-purge scan-dir /path/to/inbox --recursive --workspace /tmp/corbel
```

The directory walker picks up `.pdf`, `.epub`, `.md`, `.markdown`, and
`.docx` files. Each file is logged with a `[OK]`, `[MALICIOUS]`, or
`[ERROR]` tag.

## Step 8 — CI gate

Use `--abort-on-threat` to make CorbelPurge a CI gate. Exit code 2
means threats were found:

```bash
./target/release/corbel-purge scan incoming_document.pdf --abort-on-threat --quiet
# exit 0: clean
# exit 2: has threats -> fail the build
# exit 1: hard error (parse failure, IO, etc.)
```

`--quiet` suppresses the summary block and prints only the JSON
report path, which is handy for piping into downstream tooling.

## Step 9 — External threat-intel feeds (optional)

The built-in signature tables and CVE database are static. Layer your
own on top at runtime:

```bash
# Load additional signature rules
./target/release/corbel-purge scan suspicious.pdf --rules my_rules.json

# Load additional CVE signature entries
./target/release/corbel-purge scan suspicious.docx --cve-db my_cve_db.json

# Or set them via environment variables
export CORBEL_EXTERNAL_RULES=/etc/corbel/rules.json
export CORBEL_EXTERNAL_CVE_DB=/etc/corbel/cve_db.json
./target/release/corbel-purge scan suspicious.pdf
```

External rules are matched alongside the built-in tables; nothing is
overridden. See `MANIFEST.md` for the JSON schema. Supported
external-rule types:

- `homograph-host-list` — additional exact-match homograph host strings
- `signature-list` — additional `(offset, magic_bytes, name)` entries
- `shellcode-list` — additional shellcode prologue byte patterns

(Legacy `tld-list`, `keyword-list`, and `brand-list` rule types are
silently skipped — the scanner no longer consults those tables.)

## Step 10 — Study a document in place

The `study` subcommand renders the source document to a single
annotated HTML file with malicious regions wrapped in inline `<span>`
tags, color-coded by classification. Use it when you want to see
exactly where the exploit sits in context, without leaving the source
format:

```bash
./target/release/corbel-purge study suspicious.epub
```

Output lands at `study_<ts>_<sha>.html` in the workspace. Quiet mode
(`-q`) prints just the path.

## CLI flags reference

| Flag | Default | Purpose |
|---|---|---|
| `--workspace <dir>` | CWD | Root for `corbel_quarantine/` and `corbel_clean/` output dirs |
| `--abort-on-threat` | off | Exit with code 2 if any malicious finding fires |
| `--quiet`, `-q` | off | Print only the JSON report path on success |
| `--recursive`, `-r` | off | Recurse into subdirectories (`scan-dir`) |
| `--preserve-format` | off | Repackage cleansed document in original format |
| `--rules <path>` | unset | Path to external signature-rules JSON |
| `--cve-db <path>` | unset | Path to external CVE database JSON |
| `--no-clean-output` | off | Do NOT copy clean documents to output folder |
| `--move-clean` | off | Move (not copy) clean source files to output folder |

## Exit codes

| Code | Meaning |
|---|---|
| `0` | No threats found |
| `1` | Hard error (parse failure, IO failure, etc.) |
| `2` | One or more malicious findings (file was processed) |

## Environment variables

| Variable | Default | Purpose |
|---|---|---|
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | Quarantine tarball + reports output dir |
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | Cleansed document + clean-output copy dir |
| `CORBEL_ABORT_ON_THREAT` | `false` | Abort on first malicious finding |
| `CORBEL_EMIT_SUSPICIOUS` | `false` | Include Suspicious findings (no default detector produces any) |
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | Generate Markdown report alongside JSON |
| `CORBEL_EMIT_CLEAN_OUTPUT` | `true` | Copy clean documents to output folder |
| `CORBEL_MOVE_CLEAN_TO_OUTPUT` | `false` | Move (not copy) clean source files to output |
| `CORBEL_TOTAL_ARCHIVE_SCAN_CAP` | `268435456` (256 MiB) | Cumulative cap across all entries in a multi-entry archive |
| `CORBEL_EXTERNAL_RULES` | (unset) | Path to external signature-rules JSON |
| `CORBEL_EXTERNAL_CVE_DB` | (unset) | Path to external CVE database JSON |

## Expected output examples

### Clean file

```
────────────────────────────────────────────────────────────
scan complete: pdf
  source:      benign.pdf
  sha256:      9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
  text nodes:  12
  vectors:     0
  findings:    0 malicious, 0 educational, 0 total
  cleansed:    ./corbel_clean/clean_20260801T120000_9f86d081.pdf
  json report: ./corbel_quarantine/report_20260801T120000_9f86d081.json
  md report:   ./corbel_quarantine/report_20260801T120000_9f86d081.md
────────────────────────────────────────────────────────────
```

### Threat found

```
────────────────────────────────────────────────────────────
scan complete: pdf
  source:      suspicious.pdf
  sha256:      a1b2c3...
  text nodes:  8
  vectors:     1
  findings:    1 malicious, 0 educational, 1 total
  quarantine:  ./corbel_quarantine/quarantine_20260801T120000_abc12345.tar.gz
  cleansed:    ./corbel_clean/cleansed_20260801T120000_abc12345.pdf
  json report: ./corbel_quarantine/report_20260801T120000_abc12345.json
  md report:   ./corbel_quarantine/report_20260801T120000_abc12345.md
────────────────────────────────────────────────────────────
```

Exit code is `2` when any malicious finding is produced.

## Troubleshooting

### Build fails on `lopdf` or `zip`

These crates occasionally need a newer Rust than the MSRV declared in
their `Cargo.toml`. Update Rust:

```bash
rustup update stable
```

### GUI binary not found

The GUI is behind the `gui` feature flag. Build with:

```bash
cargo build --release --features gui --bin corbel-purge-gui
```

### Scanner reports zero findings on a document I expected to be flagged

Confirm the document actually contains a Category 1 or Category 2
detector trigger (see [README.md](README.md) for the table). The
scanner does not flag based on reputation, file source, or filename.
Run with `cargo run -- scan file.pdf` for verbose output.

### Quarantine tarball missing

The tarball is written only when `malicious_count() > 0`. Clean
documents produce only `report_<ts>_<sha>.{json,md}` and a
`clean_<ts>_<sha>.<ext>` copy in the output folder.

### Test failure on `benign_realworld_pdf_produces_zero_findings`

This is the corpus regression test. If it fails, a detector is
wrong by construction — the detector fired on a clean document. Inspect
the test's failure output for the specific detector that fired and
tighten that detector's rule.

## Installation

After building, install the binary to a system path:

```bash
cargo install --path .
# or
sudo cp target/release/corbel-purge /usr/local/bin/
```

For system-wide configuration, set environment variables in
`/etc/corbel/env` or your shell profile:

```bash
export CORBEL_QUARANTINE_DIR=/var/lib/corbel/quarantine
export CORBEL_CLEANSE_DIR=/var/lib/corbel/clean
export CORBEL_EXTERNAL_RULES=/etc/corbel/rules.json
export CORBEL_EXTERNAL_CVE_DB=/etc/corbel/cve_db.json
```

## Next steps

- Read [README.md](README.md) for the project overview and detection model
- Read [MANIFEST.md](MANIFEST.md) for the detailed technical specification
- Read [TODO.md](TODO.md) for the development roadmap
- Read [FIX-NOTES-false-positive-redesign.md](FIX-NOTES-false-positive-redesign.md)
  for the design notes on the two-category detector model

## Contact

**Jeremy Anderson** — [dcos.net](https://dcos.net) — [info@dcos.net](mailto:info@dcos.net)