19 KiB

Executable File

Raw Permalink Blame History

Project Manifest: CorbelPurge (v0.3.0)

Strict Rust Secure Document Sanitizer & Threat Neutralizer

Author: Jeremy Anderson — https://git.dcos.net/dcosnet/corbel License: GPL-3.0-or-later

Status: v0.2.0 MVP shipped. v0.3.0 planning: study mode, payload carving v2, scanner extensibility, GUI navigation improvements.

1. Executive Summary

CorbelPurge is a high-assurance document sanitizer written in strict Rust (#![forbid(unsafe_code)]) for PDF, EPUB, Markdown, and DOCX formats. It parses documents into a unified intermediate representation, runs a layered contextual scanner that distinguishes educational security literature from active malicious injections, then produces cleansed derivatives with all executable content stripped. Malicious payloads are carved into quarantine tarballs with full forensic reports.

Threat model: CorbelPurge is a researcher's tool for studying old exploits, not a target itself. The scanner safely ingests adversarial samples (zip bombs, malicious docs from malware corpora) without choking, so the researcher can study them. The tool itself is assumed to not be the target.

2. Core Specifications & Non-Goals

Core Features

Multi-Format Support: Native parsing pipelines for PDF (lopdf), EPUB (zip), Markdown (pulldown-cmark), DOCX (zip + XML regex).
Unified Intermediate Representation: All parsers emit the same Document struct (src/core/types.rs). Downstream modules never touch format-specific structures.
Automated Threat Scanning: Two-pass engine — executable vectors go through heuristics::classify_vector(), text nodes go through context_filter::evaluate(), then findings are annotated by cve_tags::match_cve().
Context-Aware Heuristics: Differentiation between static text content (including code blocks discussing vulnerabilities) and executable/malicious injection vectors. Academic markers (CVE IDs, "remediation", code-block context) whitelist educational content.
Weaponization Detection: Long hex-encoded runs (16+ \xNN), 64+ char base64 blobs, NOP sleds, Metasploit stager prologues are flagged as malicious regardless of context.
CVE Tagging: 7 known exploits (CVE-2010-0188, CVE-2018-4990, CVE-2017-11882, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199, EPUB-SCRIPT-INJECTION) with byte-signature and vector-type matching.
Forensic Extraction & Reporting: JSON + Markdown reports with SHA-256, vector location, classification, CVE tag, context notes. Payload carving into standalone .bin files inside quarantine tarballs.
Secure Quarantine: Compressed quarantine_<ts>_<sha>.tar.gz containing original file, report, and extracted payloads.
Document Cleansing (two modes):
- Markdown mode (default): produces a safe Markdown derivative via cleanse::sanitizer::sanitize(). A Markdown file cannot carry executable content by definition.
- PreserveFormat mode (--preserve-format): repackages the document in its original format (PDF/EPUB/DOCX) with malicious entries stripped via cleanse::repackage::repackage().
Zip-Bomb Defense: util::read_with_cap() counts actual decompressed bytes rather than trusting ZIP central directory size headers. Configurable via Config::epub_entry_scan_cap (default 8 MiB).
Optional GUI: iced 0.13 dashboard (src/bin/gui.rs) with dark theme, collapsible panels, console log, sidebar stats, file picker (rfd::AsyncFileDialog).
No-arg-parsing-crates: The CLI (src/main.rs) parses its own arguments to keep the dependency tree minimal — a defensive measure for security-critical software.

Explicit Non-Goals

No document editing, annotation writing, form-filling, or signature generation.
No execution of embedded active scripting (PDF JS, EPUB scripts, VBA macros).
No sandbox hardening or fuzzing of the tool itself (it is not the target).
No bundled historical exploit corpus (too risky to ship).
No magic-byte sniffing for format detection (extension-only is the safe failure mode — a mismatched extension causes a parse error).

3. System Architecture & Tech Stack

CorbelPurge uses safe Rust primitives, relying on memory-safe parsers to avoid buffer overflow vulnerabilities common in legacy C/C++ document viewers. The #![forbid(unsafe_code)] crate attribute is enforced at compile time.

Dependency Table

Purpose	Crate	Version
PDF structure inspection	`lopdf`	0.34
EPUB / DOCX containers	`zip`	2
Markdown parsing	`pulldown-cmark`	0.12
Quarantine packaging	`tar` + `flate2`	0.4 / 1
Serialization	`serde` + `serde_json`	1
Hashing	`sha2` + `hex`	0.10 / 0.4
Datetime	`chrono`	0.4
Context filter	`regex`	1
Error handling	`thiserror`	2
GUI (optional)	`iced`	0.13
File picker (optional)	`rfd`	0.15
Async runtime (optional)	`tokio`	1
Test utilities	`tempfile` + `pretty_assertions`	3 / 1

Feature Flags

default — headless CLI + library only. No GUI dependencies.
gui — adds the iced GUI binary (corbel-purge-gui) and depends on iced, rfd, tokio.

4. Unified Intermediate Representation

Every parser emits the same Document struct. The scanner, quarantine, and cleanse modules consume only this UIR — they never touch format-specific structures. Adding a new format requires writing exactly one new DocumentParser impl and adding a match arm in Dispatcher::parse() (src/parsers/mod.rs).

Key types defined in src/core/types.rs:

DocumentFormat — Pdf, Epub, Markdown, Docx. Detected by file extension via DocumentFormat::from_path() in pipeline.rs.
Location — per-format source location (PDF object/stream IDs, EPUB ZIP entry paths, Markdown line:col). Used in findings to report exactly where a threat was found.
TextContext — semantic context of extracted text (Paragraph, Heading, CodeBlock, CodeSpan, Hyperlink, BlockQuote, Metadata, ExecutableHook). The heart of the context-aware scanner.
TextNode — extracted text with location + context. Emitted by parsers for all rendered/static content.
ExecutableVector — a potentially dangerous element (JS action, embedded file, script tag, OLE object, external link) with raw bytes and optional decoded preview.
VectorType — 15 variants covering PDF (JS, Launch, URI, GoToR, EmbeddedFile, WidgetAction, AcroForm), EPUB (Script, ExternalResource, Object), Markdown (Hyperlink), DOCX (Macro, ExternalLink, EmbeddedObject, ActiveX), plus UnknownPayload.
Document — the top-level IR: format, raw bytes, SHA-256, metadata, text nodes, executable vectors.
ThreatClassification — Benign, Suspicious, EducationalContent, Malicious(MaliciousType).
MaliciousType — 8 specific categories (ActiveJavaScriptInjection, LaunchAction, MaliciousEmbeddedFile, ObfuscatedShellcode, SuspiciousUri, EpubActiveScript, DocxActiveContent, Other).
Finding — classification, location, vector type, payload preview, context notes, recommendation.
Recommendation — Allow, WhitelistAsEducational, Quarantine, QuarantineAndCleanse.
ScanReport — aggregate of all findings with counts and metadata.
CorbelError — exhaustive error enum (Io, Serde, PdfParse, EpubParse, MarkdownParse, UnknownFormat, ThreatDetected, Quarantine, Cleanse, Internal).

5. Security Scanning Pipeline

The pipeline is orchestrated by Pipeline::run() in src/core/pipeline.rs. Given an input path, it: reads bytes, detects format, dispatches to the correct parser, runs the scanner, then conditionally quarantines and cleanses.

Step 0: Weaponization Check

context_filter::has_weaponization_indicators() runs first on every text node. Pure obfuscation indicators (16+ \xNN runs, 64+ char base64 blobs, 2+ shell commands in non-code context) are flagged as ObfuscatedShellcode regardless of surrounding context. Code blocks and block quotes get an exception — weaponized-looking content there is almost certainly a research writeup.

Step 1: Structural vs. Textual Isolation

The scanner in scanner::scan() walks two separate lists:

Executable Vectors (untrusted by default) — dispatched to heuristics::inspect_vector() which calls classify_vector(). This function matches against signatures.rs tables (file signatures, shellcode patterns, phishing TLDs, brand homographs, URL shorteners, suspicious keywords) and the Config::allowed_uri_schemes whitelist.
Static Text Nodes (context-dependent) — dispatched to context_filter::evaluate() which looks for suspicious signatures (/JavaScript, eval(, shellcode, exploit, etc.) and then checks whether the surrounding context is educational.

Step 2: Context-Aware Filtering

context_filter::looks_educational() checks for:

Structural signals: code blocks, code spans, block quotes — always educational.
Lexical signals: presence of academic markers ("CVE-", "vulnerability", "remediation", "patch", "mitigation", "advisory", "for example", "proof of concept", etc.)

Decision: A signature in an ExecutableHook context is flagged as malicious immediately. A signature in a code block describing a CVE with remediation steps is whitelisted as EducationalContent.

Step 3: CVE Tagging

After classification, scanner::scan() calls cve_tags::match_cve() for each finding. If a CVE matches, the ID and name are appended to the finding's context_notes field. The forensic reporter (reporter.rs) includes the full CVE description.

CVE ID	Name	Detection Signal
CVE-2017-11882	Equation Editor RCE	DOCX OLE + "Equation" in payload
CVE-2018-0802	Equation Editor RCE (variant)	DOCX OLE + PE signature, no "Equation"
CVE-2017-8570	Office RTF OLE Object RCE	DOCX OLE + RTF magic bytes
CVE-2017-0199	Office OLE2Link RCE	DOCX external link + `.hta` target
CVE-2018-4990	Adobe Reader JS RCE	PDF JS action >500 bytes
CVE-2010-0188	PDF LibTiff Buffer Overflow	PDF embedded file + TIFF magic
EPUB-SCRIPT-INJECTION	EPUB Active Script Injection	EPUB `<script>` (malicious)

Heuristic Signature Tables (`signatures.rs`)

Centralized static threat-intel tables:

PHISHING_TLDS — 30+ TLDs overrepresented in phishing URLs.
SUSPICIOUS_URL_KEYWORDS — 30+ path/host keywords (login, signin, verify, password, credential, etc.).
URL_SHORTENER_DOMAINS — 25+ shortener domains.
COMMON_PHISHING_BRANDS — 50+ brand names with homograph variants (micros0ft, paypa1, app1e, etc.). Two-pass matching: homograph variants always flag; canonical spellings are suppressed if the host is a genuine brand domain.
KNOWN_FILE_SIGNATURES — 15+ magic-byte patterns (PE, ELF, Mach-O, OLE2, RTF, VBA, LNK, SWF, Java, Python, HTA).
SHELLCODE_PATTERNS — 9 prologues (NOP sled, Metasploit stager, x86_64 syscall, shikata_ga_nai, etc.).

6. Quarantine & Cleansing Workflow

When the scanner identifies a malicious finding, the pipeline conditionally activates quarantine and cleansing.

Pipeline flow (from `pipeline.rs`)

 input path
     │
     ▼
 DocumentFormat::from_path()  ──►  Dispatcher::parse()  ──►  Document (UIR)
                                                                    │
                                                                    ▼
                                                              scanner::scan()
                                                                    │
                                                         ScanReport
                                                       (findings, counts)
                                                                    │
                              ┌─────────────────────────┼──────────────────────┐
                              │                         │                      │
                        no malicious           malicious > 0          quarantine +
                         findings               AND abort              cleanse
                              │                   on threat                  │
                              ▼                        ▼                     ▼
                      return result            Err(ThreatDetected)  quarantine::handle()
                                                                    + cleanse::cleanse()
                                                                    │
                                                                    ▼
                                                              PipelineResult
                                                     (paths to tarball,
                                                      report, cleansed)

Quarantine (`quarantine/`)

Payload Extraction (extractor.rs): For each Malicious finding, locates the corresponding ExecutableVector by matching Location values, carves its raw_payload into an ExtractedPayload with a sanitized filename (e.g. payload_pdf_000_obj42_pdf-javascript.bin).
Forensic Reporting (reporter.rs): Builds a JSON report (schema version 1) with source metadata, finding details, CVE descriptions, and extracted payload info. Also builds a Markdown report with the # CorbelPurge Forensic Report header.
Tarball Packaging (mod.rs): Writes quarantine_<ts>_<sha>.tar.gz containing original.<ext>, report.json, report.md (if enabled), and one .bin per carved payload.

Cleansing (`cleanse/`)

Dispatched by cleanse::cleanse() based on Config::cleanse_mode:

Markdown mode (sanitizer.rs): Emits a header banner with SHA-256 and format, a CorbelPurge Notice blockquote, the original title, then all text nodes whose locations don't match a malicious finding. Code blocks are fenced, block quotes are prefixed with >, hyperlinks drop their destinations (only text is kept).
PreserveFormat mode (repackage.rs):
- EPUB: Rebuilds the ZIP, skipping entries whose paths match malicious findings. Cleans inline <script>, <iframe>, <object>, <embed> tags from remaining XHTML entries. Neutralizes src="http..." to src="#".
- DOCX: Strips word/vbaProject.xml, word/embeddings/*, word/activeX/*, and entries matching malicious findings. Rewrites word/_rels/document.xml.rels to drop TargetMode="External" relationships.
- PDF: Deletes objects matching malicious findings via lopdf::Document::delete_objects(). Strips /OpenAction, /AA, /Names, /AcroForm from the root catalog. Rebuilds xref on save.
- Markdown: Falls back to the Markdown sanitizer.

7. Zip-Bomb Defense

util::read_with_cap() is the zip-bomb defense for ZIP-container formats (EPUB, DOCX). It reads in 8 KiB chunks and counts actual decompressed bytes rather than trusting the ZIP central directory size header.

Lying size header: ZIP declares 100 bytes, actually decompresses to 1 MiB — detected and truncated at cap (default 8 MiB).
Decompression-ratio bomb: 42 KB ZIP decompresses to petabytes — stopped at cap bytes and never allocates more.
What is NOT defended: Many small entries (10,000 × 1 MiB = 10 GiB) — the per-entry cap doesn't help here. This is tracked as a TODO item (total_archive_scan_cap).

Truncated bytes are still emitted as UnknownPayload vectors so the file-signature matcher can inspect the prefix.

8. GUI Architecture

The iced 0.13 dashboard (src/bin/gui.rs) uses a dark theme:

48px header — brand, version, GPL-3.0 badge.
Collapsible left panel — PATHS (input/output with folder pickers via rfd::AsyncFileDialog), OPTIONS (toggles: preserve-format, strip-metadata, recursive, abort-on-threat), CONSOLE (timestamped log with [OK]/[ERROR] tags).
280px right sidebar — Unicode block-character progress gauge (█/░), stats (CLEANED/COPIED/ERRORS), RUN CONFIG summary, LAST SCAN findings.
56px footer — START PROCESSING (gold), STOP, CLEAR LOG (teal), ABOUT/LICENSE buttons + status bar.

The GUI spawns the pipeline via tokio::spawn_blocking and streams results to the console log.

9. Public API Surface

The crate root (lib.rs) re-exports:

pub use core::{config::{CleanseMode, Config}, pipeline::{Pipeline, PipelineResult}, types::*};
pub use util::sha256_hex;

The Pipeline struct is the only thing most callers need:

use corbel_purge::{Pipeline, Config, CleanseMode};

let config = Config::with_workspace("/tmp/output")
    .override_from_env();
let pipeline = Pipeline::with_config(config);
let result = pipeline.run("suspicious.pdf")?;

The library can also be embedded as an in-memory scanner (no filesystem access) via Pipeline::run_on_bytes().

10. Configuration

All tunables live in Config (src/core/config.rs). Runtime overrides via environment variables (applied by Config::override_from_env() and by the CLI's apply_env_overrides()):

Variable	Default	Field
`CORBEL_QUARANTINE_DIR`	`./corbel_quarantine`	`quarantine_dir`
`CORBEL_CLEANSE_DIR`	`./corbel_clean`	`cleanse_dir`
`CORBEL_ABORT_ON_THREAT`	`false`	`abort_on_threat`
`CORBEL_EMIT_SUSPICIOUS`	`true`	`emit_suspicious`
`CORBEL_EMIT_MARKDOWN_REPORT`	`true`	`emit_markdown_report`

CLI flags (src/main.rs): --workspace <dir>, --abort-on-threat, --quiet / -q, --recursive / -r, --preserve-format.

11. Test Suite

Unit tests: In every module (config, types, pipeline, parsers, scanner submodules, quarantine, cleanse, util). Cover format detection, signature matching, context filtering, CVE tagging, payload extraction, sanitizer output, zip-bomb defense.
Integration tests (tests/pipeline_integration.rs): 22 tests running the full pipeline against fixture files — benign and malicious PDFs, EPUBs, DOCXes, Markdown files. Covers quarantine, cleansing, PreserveFormat repackage, abort-on-threat, educational whitelisting, JSON/Markdown report content.
Zip-bomb tests (tests/zip_bomb_defense.rs): Defense against lying size headers and honest-but-oversized entries.
Fixtures (tests/fixtures/): Generated by Python scripts in scripts/. Requires pypdf, reportlab, python-docx.

12. Planned Future Work (v0.3.0+)

See TODO.md for the full task list. Key directions:

Study mode — annotated HTML output showing exploit locations in context, reusing the existing ScanReport and Location types.
Payload carving v2 — standalone .hex dumps and .info JSON metadata alongside the existing .bin files.
Total-memory budget for multi-entry archives (cross-entry cap).
PDF repackage: dangling reference cleanup after object deletion.
OPF manifest cleanup after EPUB repackage.

19 KiB Executable File Raw Permalink Blame History Unescape Escape