# Project Manifest: CorbelPurge (v0.3.0)

## Strict Rust Secure Document Sanitizer & Threat Neutralizer

**Author:** Jeremy Anderson — https://git.dcos.net/dcosnet/corbel
**License:** GPL-3.0-or-later

**Status:** v0.2.0 MVP shipped. v0.3.0 planning: study mode, payload carving v2,
scanner extensibility, GUI navigation improvements.

---

## 1. Executive Summary

CorbelPurge is a high-assurance document sanitizer written in strict Rust
(`#![forbid(unsafe_code)]`) for **PDF, EPUB, Markdown, and DOCX** formats.
It parses documents into a unified intermediate representation, runs a
layered contextual scanner that distinguishes educational security
literature from active malicious injections, then produces cleansed
derivatives with all executable content stripped. Malicious payloads are
carved into quarantine tarballs with full forensic reports.

**Threat model:** CorbelPurge is a researcher's tool for studying old
exploits, not a target itself. The scanner safely ingests adversarial
samples (zip bombs, malicious docs from malware corpora) without
choking, so the researcher can study them. The tool itself is assumed
to not be the target.

---

## 2. Core Specifications & Non-Goals

### Core Features

- **Multi-Format Support:** Native parsing pipelines for PDF (`lopdf`),
  EPUB (`zip`), Markdown (`pulldown-cmark`), DOCX (`zip` + XML regex).
- **Unified Intermediate Representation:** All parsers emit the same
  `Document` struct (`src/core/types.rs`). Downstream modules never
touch format-specific structures.
- **Automated Threat Scanning:** Two-pass engine — executable vectors
  go through `heuristics::classify_vector()`, text nodes go through
  `context_filter::evaluate()`, then findings are annotated by
  `cve_tags::match_cve()`.
- **Context-Aware Heuristics:** Differentiation between static text
  content (including code blocks discussing vulnerabilities) and
  executable/malicious injection vectors. Academic markers (CVE IDs,
  "remediation", code-block context) whitelist educational content.
- **Weaponization Detection:** Long hex-encoded runs (16+ `\xNN`),
  64+ char base64 blobs, NOP sleds, Metasploit stager prologues are
  flagged as malicious regardless of context.
- **CVE Tagging:** 7 known exploits (CVE-2010-0188, CVE-2018-4990,
  CVE-2017-11882, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199,
  EPUB-SCRIPT-INJECTION) with byte-signature and vector-type matching.
- **Forensic Extraction & Reporting:** JSON + Markdown reports with
  SHA-256, vector location, classification, CVE tag, context notes.
  Payload carving into standalone `.bin` files inside quarantine tarballs.
- **Secure Quarantine:** Compressed `quarantine_<ts>_<sha>.tar.gz`
  containing original file, report, and extracted payloads.
- **Document Cleansing (two modes):**
  - *Markdown mode* (default): produces a safe Markdown derivative via
    `cleanse::sanitizer::sanitize()`. A Markdown file cannot carry
    executable content by definition.
  - *PreserveFormat mode* (`--preserve-format`): repackages the document
    in its original format (PDF/EPUB/DOCX) with malicious entries
    stripped via `cleanse::repackage::repackage()`.
- **Zip-Bomb Defense:** `util::read_with_cap()` counts actual decompressed
  bytes rather than trusting ZIP central directory size headers.
  Configurable via `Config::epub_entry_scan_cap` (default 8 MiB).
- **Optional GUI:** iced 0.13 dashboard (`src/bin/gui.rs`) with dark
  theme, collapsible panels, console log, sidebar stats, file picker
  (`rfd::AsyncFileDialog`).
- **No-arg-parsing-crates:** The CLI (`src/main.rs`) parses its own
  arguments to keep the dependency tree minimal — a defensive measure
  for security-critical software.

### Explicit Non-Goals

- No document editing, annotation writing, form-filling, or signature generation.
- No execution of embedded active scripting (PDF JS, EPUB scripts, VBA macros).
- No sandbox hardening or fuzzing of the tool itself (it is not the target).
- No bundled historical exploit corpus (too risky to ship).
- No magic-byte sniffing for format detection (extension-only is the safe
  failure mode — a mismatched extension causes a parse error).

---

## 3. System Architecture & Tech Stack

CorbelPurge uses safe Rust primitives, relying on memory-safe parsers to
avoid buffer overflow vulnerabilities common in legacy C/C++ document
viewers. The `#![forbid(unsafe_code)]` crate attribute is enforced at
compile time.

### Dependency Table

| Purpose | Crate | Version |
|---------|-------|---------|
| PDF structure inspection | `lopdf` | 0.34 |
| EPUB / DOCX containers | `zip` | 2 |
| Markdown parsing | `pulldown-cmark` | 0.12 |
| Quarantine packaging | `tar` + `flate2` | 0.4 / 1 |
| Serialization | `serde` + `serde_json` | 1 |
| Hashing | `sha2` + `hex` | 0.10 / 0.4 |
| Datetime | `chrono` | 0.4 |
| Context filter | `regex` | 1 |
| Error handling | `thiserror` | 2 |
| GUI (optional) | `iced` | 0.13 |
| File picker (optional) | `rfd` | 0.15 |
| Async runtime (optional) | `tokio` | 1 |
| Test utilities | `tempfile` + `pretty_assertions` | 3 / 1 |

### Feature Flags

- `default` — headless CLI + library only. No GUI dependencies.
- `gui` — adds the iced GUI binary (`corbel-purge-gui`) and depends
  on `iced`, `rfd`, `tokio`.

---

## 4. Unified Intermediate Representation

Every parser emits the same [`Document`](src/core/types.rs) struct.
The scanner, quarantine, and cleanse modules consume only this UIR —
they never touch format-specific structures. Adding a new format
requires writing exactly one new `DocumentParser` impl and adding a
match arm in `Dispatcher::parse()` (`src/parsers/mod.rs`).

Key types defined in `src/core/types.rs`:

- **`DocumentFormat`** — `Pdf`, `Epub`, `Markdown`, `Docx`. Detected by
  file extension via `DocumentFormat::from_path()` in `pipeline.rs`.
- **`Location`** — per-format source location (PDF object/stream IDs,
  EPUB ZIP entry paths, Markdown line:col). Used in findings to report
  exactly where a threat was found.
- **`TextContext`** — semantic context of extracted text (`Paragraph`,
  `Heading`, `CodeBlock`, `CodeSpan`, `Hyperlink`, `BlockQuote`,
  `Metadata`, `ExecutableHook`). The heart of the context-aware scanner.
- **`TextNode`** — extracted text with location + context. Emitted by
  parsers for all rendered/static content.
- **`ExecutableVector`** — a potentially dangerous element (JS action,
  embedded file, script tag, OLE object, external link) with raw bytes
  and optional decoded preview.
- **`VectorType`** — 15 variants covering PDF (JS, Launch, URI, GoToR,
  EmbeddedFile, WidgetAction, AcroForm), EPUB (Script, ExternalResource,
  Object), Markdown (Hyperlink), DOCX (Macro, ExternalLink,
  EmbeddedObject, ActiveX), plus UnknownPayload.
- **`Document`** — the top-level IR: format, raw bytes, SHA-256,
  metadata, text nodes, executable vectors.
- **`ThreatClassification`** — `Benign`, `Suspicious`, `EducationalContent`,
  `Malicious(MaliciousType)`.
- **`MaliciousType`** — 8 specific categories (ActiveJavaScriptInjection,
  LaunchAction, MaliciousEmbeddedFile, ObfuscatedShellcode,
  SuspiciousUri, EpubActiveScript, DocxActiveContent, Other).
- **`Finding`** — classification, location, vector type, payload preview,
  context notes, recommendation.
- **`Recommendation`** — `Allow`, `WhitelistAsEducational`, `Quarantine`,
  `QuarantineAndCleanse`.
- **`ScanReport`** — aggregate of all findings with counts and metadata.
- **`CorbelError`** — exhaustive error enum (Io, Serde, PdfParse,
  EpubParse, MarkdownParse, UnknownFormat, ThreatDetected, Quarantine,
  Cleanse, Internal).

---

## 5. Security Scanning Pipeline

The pipeline is orchestrated by `Pipeline::run()` in `src/core/pipeline.rs`.
Given an input path, it: reads bytes, detects format, dispatches to the
correct parser, runs the scanner, then conditionally quarantines and
cleanses.

### Step 0: Weaponization Check

`context_filter::has_weaponization_indicators()` runs first on every text
node. Pure obfuscation indicators (16+ `\xNN` runs, 64+ char base64
blobs, 2+ shell commands in non-code context) are flagged as
`ObfuscatedShellcode` regardless of surrounding context. Code blocks
and block quotes get an exception — weaponized-looking content there is
almost certainly a research writeup.

### Step 1: Structural vs. Textual Isolation

The scanner in `scanner::scan()` walks two separate lists:

- **Executable Vectors** (untrusted by default) — dispatched to
  `heuristics::inspect_vector()` which calls `classify_vector()`.
  This function matches against `signatures.rs` tables (file signatures,
  shellcode patterns, phishing TLDs, brand homographs, URL shorteners,
  suspicious keywords) and the `Config::allowed_uri_schemes` whitelist.
- **Static Text Nodes** (context-dependent) — dispatched to
  `context_filter::evaluate()` which looks for suspicious signatures
  (`/JavaScript`, `eval(`, `shellcode`, `exploit`, etc.) and then
  checks whether the surrounding context is educational.

### Step 2: Context-Aware Filtering

`context_filter::looks_educational()` checks for:

- **Structural signals:** code blocks, code spans, block quotes — always
  educational.
- **Lexical signals:** presence of academic markers ("CVE-",
  "vulnerability", "remediation", "patch", "mitigation", "advisory",
  "for example", "proof of concept", etc.)

**Decision:** A signature in an `ExecutableHook` context is flagged as
malicious immediately. A signature in a code block describing a CVE
with remediation steps is whitelisted as `EducationalContent`.

### Step 3: CVE Tagging

After classification, `scanner::scan()` calls `cve_tags::match_cve()`
for each finding. If a CVE matches, the ID and name are appended to the
finding's `context_notes` field. The forensic reporter (`reporter.rs`)
includes the full CVE description.

| CVE ID | Name | Detection Signal |
|--------|------|------------------|
| CVE-2017-11882 | Equation Editor RCE | DOCX OLE + "Equation" in payload |
| CVE-2018-0802 | Equation Editor RCE (variant) | DOCX OLE + PE signature, no "Equation" |
| CVE-2017-8570 | Office RTF OLE Object RCE | DOCX OLE + RTF magic bytes |
| CVE-2017-0199 | Office OLE2Link RCE | DOCX external link + `.hta` target |
| CVE-2018-4990 | Adobe Reader JS RCE | PDF JS action >500 bytes |
| CVE-2010-0188 | PDF LibTiff Buffer Overflow | PDF embedded file + TIFF magic |
| EPUB-SCRIPT-INJECTION | EPUB Active Script Injection | EPUB `<script>` (malicious) |

### Heuristic Signature Tables (`signatures.rs`)

Centralized static threat-intel tables:

- **PHISHING_TLDS** — 30+ TLDs overrepresented in phishing URLs.
- **SUSPICIOUS_URL_KEYWORDS** — 30+ path/host keywords (login, signin,
  verify, password, credential, etc.).
- **URL_SHORTENER_DOMAINS** — 25+ shortener domains.
- **COMMON_PHISHING_BRANDS** — 50+ brand names with homograph variants
  (micros0ft, paypa1, app1e, etc.). Two-pass matching: homograph
  variants always flag; canonical spellings are suppressed if the host
  is a genuine brand domain.
- **KNOWN_FILE_SIGNATURES** — 15+ magic-byte patterns (PE, ELF, Mach-O,
  OLE2, RTF, VBA, LNK, SWF, Java, Python, HTA).
- **SHELLCODE_PATTERNS** — 9 prologues (NOP sled, Metasploit stager,
  x86_64 syscall, shikata_ga_nai, etc.).

---

## 6. Quarantine & Cleansing Workflow

When the scanner identifies a malicious finding, the pipeline conditionally
activates quarantine and cleansing.

### Pipeline flow (from `pipeline.rs`)

```text
 input path
     │
     ▼
 DocumentFormat::from_path()  ──►  Dispatcher::parse()  ──►  Document (UIR)
                                                                    │
                                                                    ▼
                                                              scanner::scan()
                                                                    │
                                                         ScanReport
                                                       (findings, counts)
                                                                    │
                              ┌─────────────────────────┼──────────────────────┐
                              │                         │                      │
                        no malicious           malicious > 0          quarantine +
                         findings               AND abort              cleanse
                              │                   on threat                  │
                              ▼                        ▼                     ▼
                      return result            Err(ThreatDetected)  quarantine::handle()
                                                                    + cleanse::cleanse()
                                                                    │
                                                                    ▼
                                                              PipelineResult
                                                     (paths to tarball,
                                                      report, cleansed)
```

### Quarantine (`quarantine/`)

1. **Payload Extraction** (`extractor.rs`): For each `Malicious` finding,
   locates the corresponding `ExecutableVector` by matching `Location`
   values, carves its `raw_payload` into an `ExtractedPayload` with a
   sanitized filename (e.g. `payload_pdf_000_obj42_pdf-javascript.bin`).
2. **Forensic Reporting** (`reporter.rs`): Builds a JSON report (schema
   version 1) with source metadata, finding details, CVE descriptions,
   and extracted payload info. Also builds a Markdown report with the
   `# CorbelPurge Forensic Report` header.
3. **Tarball Packaging** (`mod.rs`): Writes `quarantine_<ts>_<sha>.tar.gz`
   containing `original.<ext>`, `report.json`, `report.md` (if enabled),
   and one `.bin` per carved payload.

### Cleansing (`cleanse/`)

Dispatched by `cleanse::cleanse()` based on `Config::cleanse_mode`:

- **Markdown mode** (`sanitizer.rs`): Emits a header banner with SHA-256
  and format, a `CorbelPurge Notice` blockquote, the original title, then
  all text nodes whose locations don't match a malicious finding. Code
  blocks are fenced, block quotes are prefixed with `>`, hyperlinks
  drop their destinations (only text is kept).

- **PreserveFormat mode** (`repackage.rs`):
  - **EPUB**: Rebuilds the ZIP, skipping entries whose paths match
    malicious findings. Cleans inline `<script>`, `<iframe>`, `<object>`,
    `<embed>` tags from remaining XHTML entries. Neutralizes
    `src="http..."` to `src="#"`.
  - **DOCX**: Strips `word/vbaProject.xml`, `word/embeddings/*`,
    `word/activeX/*`, and entries matching malicious findings.
    Rewrites `word/_rels/document.xml.rels` to drop
    `TargetMode="External"` relationships.
  - **PDF**: Deletes objects matching malicious findings via
    `lopdf::Document::delete_objects()`. Strips `/OpenAction`, `/AA`,
    `/Names`, `/AcroForm` from the root catalog. Rebuilds xref on save.
  - **Markdown**: Falls back to the Markdown sanitizer.

---

## 7. Zip-Bomb Defense

`util::read_with_cap()` is the zip-bomb defense for ZIP-container formats
(EPUB, DOCX). It reads in 8 KiB chunks and counts **actual decompressed
bytes** rather than trusting the ZIP central directory size header.

- **Lying size header:** ZIP declares 100 bytes, actually decompresses
  to 1 MiB — detected and truncated at `cap` (default 8 MiB).
- **Decompression-ratio bomb:** 42 KB ZIP decompresses to petabytes —
  stopped at `cap` bytes and never allocates more.
- **What is NOT defended:** Many small entries (10,000 × 1 MiB = 10 GiB)
  — the per-entry cap doesn't help here. This is tracked as a TODO item
  (`total_archive_scan_cap`).

Truncated bytes are still emitted as `UnknownPayload` vectors so the
file-signature matcher can inspect the prefix.

---

## 8. GUI Architecture

The iced 0.13 dashboard (`src/bin/gui.rs`) uses a dark theme:

- **48px header** — brand, version, GPL-3.0 badge.
- **Collapsible left panel** — PATHS (input/output with folder pickers
  via `rfd::AsyncFileDialog`), OPTIONS (toggles: preserve-format,
  strip-metadata, recursive, abort-on-threat), CONSOLE (timestamped
  log with [OK]/[ERROR] tags).
- **280px right sidebar** — Unicode block-character progress gauge
  (`█`/`░`), stats (CLEANED/COPIED/ERRORS), RUN CONFIG summary,
  LAST SCAN findings.
- **56px footer** — START PROCESSING (gold), STOP, CLEAR LOG (teal),
  ABOUT/LICENSE buttons + status bar.

The GUI spawns the pipeline via `tokio::spawn_blocking` and streams
results to the console log.

---

## 9. Public API Surface

The crate root (`lib.rs`) re-exports:

```rust
pub use core::{config::{CleanseMode, Config}, pipeline::{Pipeline, PipelineResult}, types::*};
pub use util::sha256_hex;
```

The `Pipeline` struct is the only thing most callers need:

```rust
use corbel_purge::{Pipeline, Config, CleanseMode};

let config = Config::with_workspace("/tmp/output")
    .override_from_env();
let pipeline = Pipeline::with_config(config);
let result = pipeline.run("suspicious.pdf")?;
```

The library can also be embedded as an in-memory scanner (no filesystem
access) via `Pipeline::run_on_bytes()`.

---

## 10. Configuration

All tunables live in `Config` (`src/core/config.rs`). Runtime overrides
via environment variables (applied by `Config::override_from_env()` and
by the CLI's `apply_env_overrides()`):

| Variable | Default | Field |
|----------|---------|-------|
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | `quarantine_dir` |
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | `cleanse_dir` |
| `CORBEL_ABORT_ON_THREAT` | `false` | `abort_on_threat` |
| `CORBEL_EMIT_SUSPICIOUS` | `true` | `emit_suspicious` |
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | `emit_markdown_report` |

CLI flags (`src/main.rs`):
`--workspace <dir>`, `--abort-on-threat`, `--quiet` / `-q`,
`--recursive` / `-r`, `--preserve-format`.

---

## 11. Test Suite

- **Unit tests:** In every module (config, types, pipeline, parsers,
  scanner submodules, quarantine, cleanse, util). Cover format detection,
  signature matching, context filtering, CVE tagging, payload extraction,
  sanitizer output, zip-bomb defense.
- **Integration tests** (`tests/pipeline_integration.rs`): 22 tests
  running the full pipeline against fixture files — benign and malicious
  PDFs, EPUBs, DOCXes, Markdown files. Covers quarantine, cleansing,
  PreserveFormat repackage, abort-on-threat, educational whitelisting,
  JSON/Markdown report content.
- **Zip-bomb tests** (`tests/zip_bomb_defense.rs`): Defense against lying
  size headers and honest-but-oversized entries.
- **Fixtures** (`tests/fixtures/`): Generated by Python scripts in
  `scripts/`. Requires `pypdf`, `reportlab`, `python-docx`.

---

## 12. Planned Future Work (v0.3.0+)

See `TODO.md` for the full task list. Key directions:

- **Study mode** — annotated HTML output showing exploit locations in
  context, reusing the existing `ScanReport` and `Location` types.
- **Payload carving v2** — standalone `.hex` dumps and `.info` JSON
  metadata alongside the existing `.bin` files.
- **Total-memory budget** for multi-entry archives (cross-entry cap).
- **PDF repackage: dangling reference cleanup** after object deletion.
- **OPF manifest cleanup** after EPUB repackage.

---

## 13. License

GPL-3.0-or-later. Copyright (c) 2025 Jeremy Anderson.
https://git.dcos.net/dcosnet/corbel