439 lines
19 KiB
Markdown
Executable File
439 lines
19 KiB
Markdown
Executable File
# Project Manifest: CorbelPurge (v0.3.0)
|
||
|
||
## Strict Rust Secure Document Sanitizer & Threat Neutralizer
|
||
|
||
**Author:** Jeremy Anderson — https://git.dcos.net/dcosnet/corbel
|
||
**License:** GPL-3.0-or-later
|
||
|
||
**Status:** v0.2.0 MVP shipped. v0.3.0 planning: study mode, payload carving v2,
|
||
scanner extensibility, GUI navigation improvements.
|
||
|
||
---
|
||
|
||
## 1. Executive Summary
|
||
|
||
CorbelPurge is a high-assurance document sanitizer written in strict Rust
|
||
(`#![forbid(unsafe_code)]`) for **PDF, EPUB, Markdown, and DOCX** formats.
|
||
It parses documents into a unified intermediate representation, runs a
|
||
layered contextual scanner that distinguishes educational security
|
||
literature from active malicious injections, then produces cleansed
|
||
derivatives with all executable content stripped. Malicious payloads are
|
||
carved into quarantine tarballs with full forensic reports.
|
||
|
||
**Threat model:** CorbelPurge is a researcher's tool for studying old
|
||
exploits, not a target itself. The scanner safely ingests adversarial
|
||
samples (zip bombs, malicious docs from malware corpora) without
|
||
choking, so the researcher can study them. The tool itself is assumed
|
||
to not be the target.
|
||
|
||
---
|
||
|
||
## 2. Core Specifications & Non-Goals
|
||
|
||
### Core Features
|
||
|
||
- **Multi-Format Support:** Native parsing pipelines for PDF (`lopdf`),
|
||
EPUB (`zip`), Markdown (`pulldown-cmark`), DOCX (`zip` + XML regex).
|
||
- **Unified Intermediate Representation:** All parsers emit the same
|
||
`Document` struct (`src/core/types.rs`). Downstream modules never
|
||
touch format-specific structures.
|
||
- **Automated Threat Scanning:** Two-pass engine — executable vectors
|
||
go through `heuristics::classify_vector()`, text nodes go through
|
||
`context_filter::evaluate()`, then findings are annotated by
|
||
`cve_tags::match_cve()`.
|
||
- **Context-Aware Heuristics:** Differentiation between static text
|
||
content (including code blocks discussing vulnerabilities) and
|
||
executable/malicious injection vectors. Academic markers (CVE IDs,
|
||
"remediation", code-block context) whitelist educational content.
|
||
- **Weaponization Detection:** Long hex-encoded runs (16+ `\xNN`),
|
||
64+ char base64 blobs, NOP sleds, Metasploit stager prologues are
|
||
flagged as malicious regardless of context.
|
||
- **CVE Tagging:** 7 known exploits (CVE-2010-0188, CVE-2018-4990,
|
||
CVE-2017-11882, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199,
|
||
EPUB-SCRIPT-INJECTION) with byte-signature and vector-type matching.
|
||
- **Forensic Extraction & Reporting:** JSON + Markdown reports with
|
||
SHA-256, vector location, classification, CVE tag, context notes.
|
||
Payload carving into standalone `.bin` files inside quarantine tarballs.
|
||
- **Secure Quarantine:** Compressed `quarantine_<ts>_<sha>.tar.gz`
|
||
containing original file, report, and extracted payloads.
|
||
- **Document Cleansing (two modes):**
|
||
- *Markdown mode* (default): produces a safe Markdown derivative via
|
||
`cleanse::sanitizer::sanitize()`. A Markdown file cannot carry
|
||
executable content by definition.
|
||
- *PreserveFormat mode* (`--preserve-format`): repackages the document
|
||
in its original format (PDF/EPUB/DOCX) with malicious entries
|
||
stripped via `cleanse::repackage::repackage()`.
|
||
- **Zip-Bomb Defense:** `util::read_with_cap()` counts actual decompressed
|
||
bytes rather than trusting ZIP central directory size headers.
|
||
Configurable via `Config::epub_entry_scan_cap` (default 8 MiB).
|
||
- **Optional GUI:** iced 0.13 dashboard (`src/bin/gui.rs`) with dark
|
||
theme, collapsible panels, console log, sidebar stats, file picker
|
||
(`rfd::AsyncFileDialog`).
|
||
- **No-arg-parsing-crates:** The CLI (`src/main.rs`) parses its own
|
||
arguments to keep the dependency tree minimal — a defensive measure
|
||
for security-critical software.
|
||
|
||
### Explicit Non-Goals
|
||
|
||
- No document editing, annotation writing, form-filling, or signature generation.
|
||
- No execution of embedded active scripting (PDF JS, EPUB scripts, VBA macros).
|
||
- No sandbox hardening or fuzzing of the tool itself (it is not the target).
|
||
- No bundled historical exploit corpus (too risky to ship).
|
||
- No magic-byte sniffing for format detection (extension-only is the safe
|
||
failure mode — a mismatched extension causes a parse error).
|
||
|
||
---
|
||
|
||
## 3. System Architecture & Tech Stack
|
||
|
||
CorbelPurge uses safe Rust primitives, relying on memory-safe parsers to
|
||
avoid buffer overflow vulnerabilities common in legacy C/C++ document
|
||
viewers. The `#![forbid(unsafe_code)]` crate attribute is enforced at
|
||
compile time.
|
||
|
||
### Dependency Table
|
||
|
||
| Purpose | Crate | Version |
|
||
|---------|-------|---------|
|
||
| PDF structure inspection | `lopdf` | 0.34 |
|
||
| EPUB / DOCX containers | `zip` | 2 |
|
||
| Markdown parsing | `pulldown-cmark` | 0.12 |
|
||
| Quarantine packaging | `tar` + `flate2` | 0.4 / 1 |
|
||
| Serialization | `serde` + `serde_json` | 1 |
|
||
| Hashing | `sha2` + `hex` | 0.10 / 0.4 |
|
||
| Datetime | `chrono` | 0.4 |
|
||
| Context filter | `regex` | 1 |
|
||
| Error handling | `thiserror` | 2 |
|
||
| GUI (optional) | `iced` | 0.13 |
|
||
| File picker (optional) | `rfd` | 0.15 |
|
||
| Async runtime (optional) | `tokio` | 1 |
|
||
| Test utilities | `tempfile` + `pretty_assertions` | 3 / 1 |
|
||
|
||
### Feature Flags
|
||
|
||
- `default` — headless CLI + library only. No GUI dependencies.
|
||
- `gui` — adds the iced GUI binary (`corbel-purge-gui`) and depends
|
||
on `iced`, `rfd`, `tokio`.
|
||
|
||
---
|
||
|
||
## 4. Unified Intermediate Representation
|
||
|
||
Every parser emits the same [`Document`](src/core/types.rs) struct.
|
||
The scanner, quarantine, and cleanse modules consume only this UIR —
|
||
they never touch format-specific structures. Adding a new format
|
||
requires writing exactly one new `DocumentParser` impl and adding a
|
||
match arm in `Dispatcher::parse()` (`src/parsers/mod.rs`).
|
||
|
||
Key types defined in `src/core/types.rs`:
|
||
|
||
- **`DocumentFormat`** — `Pdf`, `Epub`, `Markdown`, `Docx`. Detected by
|
||
file extension via `DocumentFormat::from_path()` in `pipeline.rs`.
|
||
- **`Location`** — per-format source location (PDF object/stream IDs,
|
||
EPUB ZIP entry paths, Markdown line:col). Used in findings to report
|
||
exactly where a threat was found.
|
||
- **`TextContext`** — semantic context of extracted text (`Paragraph`,
|
||
`Heading`, `CodeBlock`, `CodeSpan`, `Hyperlink`, `BlockQuote`,
|
||
`Metadata`, `ExecutableHook`). The heart of the context-aware scanner.
|
||
- **`TextNode`** — extracted text with location + context. Emitted by
|
||
parsers for all rendered/static content.
|
||
- **`ExecutableVector`** — a potentially dangerous element (JS action,
|
||
embedded file, script tag, OLE object, external link) with raw bytes
|
||
and optional decoded preview.
|
||
- **`VectorType`** — 15 variants covering PDF (JS, Launch, URI, GoToR,
|
||
EmbeddedFile, WidgetAction, AcroForm), EPUB (Script, ExternalResource,
|
||
Object), Markdown (Hyperlink), DOCX (Macro, ExternalLink,
|
||
EmbeddedObject, ActiveX), plus UnknownPayload.
|
||
- **`Document`** — the top-level IR: format, raw bytes, SHA-256,
|
||
metadata, text nodes, executable vectors.
|
||
- **`ThreatClassification`** — `Benign`, `Suspicious`, `EducationalContent`,
|
||
`Malicious(MaliciousType)`.
|
||
- **`MaliciousType`** — 8 specific categories (ActiveJavaScriptInjection,
|
||
LaunchAction, MaliciousEmbeddedFile, ObfuscatedShellcode,
|
||
SuspiciousUri, EpubActiveScript, DocxActiveContent, Other).
|
||
- **`Finding`** — classification, location, vector type, payload preview,
|
||
context notes, recommendation.
|
||
- **`Recommendation`** — `Allow`, `WhitelistAsEducational`, `Quarantine`,
|
||
`QuarantineAndCleanse`.
|
||
- **`ScanReport`** — aggregate of all findings with counts and metadata.
|
||
- **`CorbelError`** — exhaustive error enum (Io, Serde, PdfParse,
|
||
EpubParse, MarkdownParse, UnknownFormat, ThreatDetected, Quarantine,
|
||
Cleanse, Internal).
|
||
|
||
---
|
||
|
||
## 5. Security Scanning Pipeline
|
||
|
||
The pipeline is orchestrated by `Pipeline::run()` in `src/core/pipeline.rs`.
|
||
Given an input path, it: reads bytes, detects format, dispatches to the
|
||
correct parser, runs the scanner, then conditionally quarantines and
|
||
cleanses.
|
||
|
||
### Step 0: Weaponization Check
|
||
|
||
`context_filter::has_weaponization_indicators()` runs first on every text
|
||
node. Pure obfuscation indicators (16+ `\xNN` runs, 64+ char base64
|
||
blobs, 2+ shell commands in non-code context) are flagged as
|
||
`ObfuscatedShellcode` regardless of surrounding context. Code blocks
|
||
and block quotes get an exception — weaponized-looking content there is
|
||
almost certainly a research writeup.
|
||
|
||
### Step 1: Structural vs. Textual Isolation
|
||
|
||
The scanner in `scanner::scan()` walks two separate lists:
|
||
|
||
- **Executable Vectors** (untrusted by default) — dispatched to
|
||
`heuristics::inspect_vector()` which calls `classify_vector()`.
|
||
This function matches against `signatures.rs` tables (file signatures,
|
||
shellcode patterns, phishing TLDs, brand homographs, URL shorteners,
|
||
suspicious keywords) and the `Config::allowed_uri_schemes` whitelist.
|
||
- **Static Text Nodes** (context-dependent) — dispatched to
|
||
`context_filter::evaluate()` which looks for suspicious signatures
|
||
(`/JavaScript`, `eval(`, `shellcode`, `exploit`, etc.) and then
|
||
checks whether the surrounding context is educational.
|
||
|
||
### Step 2: Context-Aware Filtering
|
||
|
||
`context_filter::looks_educational()` checks for:
|
||
|
||
- **Structural signals:** code blocks, code spans, block quotes — always
|
||
educational.
|
||
- **Lexical signals:** presence of academic markers ("CVE-",
|
||
"vulnerability", "remediation", "patch", "mitigation", "advisory",
|
||
"for example", "proof of concept", etc.)
|
||
|
||
**Decision:** A signature in an `ExecutableHook` context is flagged as
|
||
malicious immediately. A signature in a code block describing a CVE
|
||
with remediation steps is whitelisted as `EducationalContent`.
|
||
|
||
### Step 3: CVE Tagging
|
||
|
||
After classification, `scanner::scan()` calls `cve_tags::match_cve()`
|
||
for each finding. If a CVE matches, the ID and name are appended to the
|
||
finding's `context_notes` field. The forensic reporter (`reporter.rs`)
|
||
includes the full CVE description.
|
||
|
||
| CVE ID | Name | Detection Signal |
|
||
|--------|------|------------------|
|
||
| CVE-2017-11882 | Equation Editor RCE | DOCX OLE + "Equation" in payload |
|
||
| CVE-2018-0802 | Equation Editor RCE (variant) | DOCX OLE + PE signature, no "Equation" |
|
||
| CVE-2017-8570 | Office RTF OLE Object RCE | DOCX OLE + RTF magic bytes |
|
||
| CVE-2017-0199 | Office OLE2Link RCE | DOCX external link + `.hta` target |
|
||
| CVE-2018-4990 | Adobe Reader JS RCE | PDF JS action >500 bytes |
|
||
| CVE-2010-0188 | PDF LibTiff Buffer Overflow | PDF embedded file + TIFF magic |
|
||
| EPUB-SCRIPT-INJECTION | EPUB Active Script Injection | EPUB `<script>` (malicious) |
|
||
|
||
### Heuristic Signature Tables (`signatures.rs`)
|
||
|
||
Centralized static threat-intel tables:
|
||
|
||
- **PHISHING_TLDS** — 30+ TLDs overrepresented in phishing URLs.
|
||
- **SUSPICIOUS_URL_KEYWORDS** — 30+ path/host keywords (login, signin,
|
||
verify, password, credential, etc.).
|
||
- **URL_SHORTENER_DOMAINS** — 25+ shortener domains.
|
||
- **COMMON_PHISHING_BRANDS** — 50+ brand names with homograph variants
|
||
(micros0ft, paypa1, app1e, etc.). Two-pass matching: homograph
|
||
variants always flag; canonical spellings are suppressed if the host
|
||
is a genuine brand domain.
|
||
- **KNOWN_FILE_SIGNATURES** — 15+ magic-byte patterns (PE, ELF, Mach-O,
|
||
OLE2, RTF, VBA, LNK, SWF, Java, Python, HTA).
|
||
- **SHELLCODE_PATTERNS** — 9 prologues (NOP sled, Metasploit stager,
|
||
x86_64 syscall, shikata_ga_nai, etc.).
|
||
|
||
---
|
||
|
||
## 6. Quarantine & Cleansing Workflow
|
||
|
||
When the scanner identifies a malicious finding, the pipeline conditionally
|
||
activates quarantine and cleansing.
|
||
|
||
### Pipeline flow (from `pipeline.rs`)
|
||
|
||
```text
|
||
input path
|
||
│
|
||
▼
|
||
DocumentFormat::from_path() ──► Dispatcher::parse() ──► Document (UIR)
|
||
│
|
||
▼
|
||
scanner::scan()
|
||
│
|
||
ScanReport
|
||
(findings, counts)
|
||
│
|
||
┌─────────────────────────┼──────────────────────┐
|
||
│ │ │
|
||
no malicious malicious > 0 quarantine +
|
||
findings AND abort cleanse
|
||
│ on threat │
|
||
▼ ▼ ▼
|
||
return result Err(ThreatDetected) quarantine::handle()
|
||
+ cleanse::cleanse()
|
||
│
|
||
▼
|
||
PipelineResult
|
||
(paths to tarball,
|
||
report, cleansed)
|
||
```
|
||
|
||
### Quarantine (`quarantine/`)
|
||
|
||
1. **Payload Extraction** (`extractor.rs`): For each `Malicious` finding,
|
||
locates the corresponding `ExecutableVector` by matching `Location`
|
||
values, carves its `raw_payload` into an `ExtractedPayload` with a
|
||
sanitized filename (e.g. `payload_pdf_000_obj42_pdf-javascript.bin`).
|
||
2. **Forensic Reporting** (`reporter.rs`): Builds a JSON report (schema
|
||
version 1) with source metadata, finding details, CVE descriptions,
|
||
and extracted payload info. Also builds a Markdown report with the
|
||
`# CorbelPurge Forensic Report` header.
|
||
3. **Tarball Packaging** (`mod.rs`): Writes `quarantine_<ts>_<sha>.tar.gz`
|
||
containing `original.<ext>`, `report.json`, `report.md` (if enabled),
|
||
and one `.bin` per carved payload.
|
||
|
||
### Cleansing (`cleanse/`)
|
||
|
||
Dispatched by `cleanse::cleanse()` based on `Config::cleanse_mode`:
|
||
|
||
- **Markdown mode** (`sanitizer.rs`): Emits a header banner with SHA-256
|
||
and format, a `CorbelPurge Notice` blockquote, the original title, then
|
||
all text nodes whose locations don't match a malicious finding. Code
|
||
blocks are fenced, block quotes are prefixed with `>`, hyperlinks
|
||
drop their destinations (only text is kept).
|
||
|
||
- **PreserveFormat mode** (`repackage.rs`):
|
||
- **EPUB**: Rebuilds the ZIP, skipping entries whose paths match
|
||
malicious findings. Cleans inline `<script>`, `<iframe>`, `<object>`,
|
||
`<embed>` tags from remaining XHTML entries. Neutralizes
|
||
`src="http..."` to `src="#"`.
|
||
- **DOCX**: Strips `word/vbaProject.xml`, `word/embeddings/*`,
|
||
`word/activeX/*`, and entries matching malicious findings.
|
||
Rewrites `word/_rels/document.xml.rels` to drop
|
||
`TargetMode="External"` relationships.
|
||
- **PDF**: Deletes objects matching malicious findings via
|
||
`lopdf::Document::delete_objects()`. Strips `/OpenAction`, `/AA`,
|
||
`/Names`, `/AcroForm` from the root catalog. Rebuilds xref on save.
|
||
- **Markdown**: Falls back to the Markdown sanitizer.
|
||
|
||
---
|
||
|
||
## 7. Zip-Bomb Defense
|
||
|
||
`util::read_with_cap()` is the zip-bomb defense for ZIP-container formats
|
||
(EPUB, DOCX). It reads in 8 KiB chunks and counts **actual decompressed
|
||
bytes** rather than trusting the ZIP central directory size header.
|
||
|
||
- **Lying size header:** ZIP declares 100 bytes, actually decompresses
|
||
to 1 MiB — detected and truncated at `cap` (default 8 MiB).
|
||
- **Decompression-ratio bomb:** 42 KB ZIP decompresses to petabytes —
|
||
stopped at `cap` bytes and never allocates more.
|
||
- **What is NOT defended:** Many small entries (10,000 × 1 MiB = 10 GiB)
|
||
— the per-entry cap doesn't help here. This is tracked as a TODO item
|
||
(`total_archive_scan_cap`).
|
||
|
||
Truncated bytes are still emitted as `UnknownPayload` vectors so the
|
||
file-signature matcher can inspect the prefix.
|
||
|
||
---
|
||
|
||
## 8. GUI Architecture
|
||
|
||
The iced 0.13 dashboard (`src/bin/gui.rs`) uses a dark theme:
|
||
|
||
- **48px header** — brand, version, GPL-3.0 badge.
|
||
- **Collapsible left panel** — PATHS (input/output with folder pickers
|
||
via `rfd::AsyncFileDialog`), OPTIONS (toggles: preserve-format,
|
||
strip-metadata, recursive, abort-on-threat), CONSOLE (timestamped
|
||
log with [OK]/[ERROR] tags).
|
||
- **280px right sidebar** — Unicode block-character progress gauge
|
||
(`█`/`░`), stats (CLEANED/COPIED/ERRORS), RUN CONFIG summary,
|
||
LAST SCAN findings.
|
||
- **56px footer** — START PROCESSING (gold), STOP, CLEAR LOG (teal),
|
||
ABOUT/LICENSE buttons + status bar.
|
||
|
||
The GUI spawns the pipeline via `tokio::spawn_blocking` and streams
|
||
results to the console log.
|
||
|
||
---
|
||
|
||
## 9. Public API Surface
|
||
|
||
The crate root (`lib.rs`) re-exports:
|
||
|
||
```rust
|
||
pub use core::{config::{CleanseMode, Config}, pipeline::{Pipeline, PipelineResult}, types::*};
|
||
pub use util::sha256_hex;
|
||
```
|
||
|
||
The `Pipeline` struct is the only thing most callers need:
|
||
|
||
```rust
|
||
use corbel_purge::{Pipeline, Config, CleanseMode};
|
||
|
||
let config = Config::with_workspace("/tmp/output")
|
||
.override_from_env();
|
||
let pipeline = Pipeline::with_config(config);
|
||
let result = pipeline.run("suspicious.pdf")?;
|
||
```
|
||
|
||
The library can also be embedded as an in-memory scanner (no filesystem
|
||
access) via `Pipeline::run_on_bytes()`.
|
||
|
||
---
|
||
|
||
## 10. Configuration
|
||
|
||
All tunables live in `Config` (`src/core/config.rs`). Runtime overrides
|
||
via environment variables (applied by `Config::override_from_env()` and
|
||
by the CLI's `apply_env_overrides()`):
|
||
|
||
| Variable | Default | Field |
|
||
|----------|---------|-------|
|
||
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | `quarantine_dir` |
|
||
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | `cleanse_dir` |
|
||
| `CORBEL_ABORT_ON_THREAT` | `false` | `abort_on_threat` |
|
||
| `CORBEL_EMIT_SUSPICIOUS` | `true` | `emit_suspicious` |
|
||
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | `emit_markdown_report` |
|
||
|
||
CLI flags (`src/main.rs`):
|
||
`--workspace <dir>`, `--abort-on-threat`, `--quiet` / `-q`,
|
||
`--recursive` / `-r`, `--preserve-format`.
|
||
|
||
---
|
||
|
||
## 11. Test Suite
|
||
|
||
- **Unit tests:** In every module (config, types, pipeline, parsers,
|
||
scanner submodules, quarantine, cleanse, util). Cover format detection,
|
||
signature matching, context filtering, CVE tagging, payload extraction,
|
||
sanitizer output, zip-bomb defense.
|
||
- **Integration tests** (`tests/pipeline_integration.rs`): 22 tests
|
||
running the full pipeline against fixture files — benign and malicious
|
||
PDFs, EPUBs, DOCXes, Markdown files. Covers quarantine, cleansing,
|
||
PreserveFormat repackage, abort-on-threat, educational whitelisting,
|
||
JSON/Markdown report content.
|
||
- **Zip-bomb tests** (`tests/zip_bomb_defense.rs`): Defense against lying
|
||
size headers and honest-but-oversized entries.
|
||
- **Fixtures** (`tests/fixtures/`): Generated by Python scripts in
|
||
`scripts/`. Requires `pypdf`, `reportlab`, `python-docx`.
|
||
|
||
---
|
||
|
||
## 12. Planned Future Work (v0.3.0+)
|
||
|
||
See `TODO.md` for the full task list. Key directions:
|
||
|
||
- **Study mode** — annotated HTML output showing exploit locations in
|
||
context, reusing the existing `ScanReport` and `Location` types.
|
||
- **Payload carving v2** — standalone `.hex` dumps and `.info` JSON
|
||
metadata alongside the existing `.bin` files.
|
||
- **Total-memory budget** for multi-entry archives (cross-entry cap).
|
||
- **PDF repackage: dangling reference cleanup** after object deletion.
|
||
- **OPF manifest cleanup** after EPUB repackage.
|
||
|
||
---
|
||
|
||
## 13. License
|
||
|
||
GPL-3.0-or-later. Copyright (c) 2025 Jeremy Anderson.
|
||
https://git.dcos.net/dcosnet/corbel
|