dcosnet
/
corbel
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

CorbelPurge parses documents into a unified intermediate representation (Document struct), runs a layered contextual scanner that distinguishes educational security literature from active malicious injections, and produces cleansed derivatives with all executable content stripped. Malicious payloads are carved into quarantine tarballs with full forensic reports.'

This commit is contained in:
Jeremy Anderson 2026-08-12 16:52:57 -04:00
commit 7992a48d10
59 changed files with 13272 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
BLOG.md Normal file
View File

@ -0,0 +1,83 @@
# Announcing CorbelPurge v0.4.2
*A defensive document research and cleaning tool for security analysts.*
## Why This Exists
Security researchers have a problem. When a suspicious PDF lands in your inbox or a weaponized DOCX turns up in a malware corpus, your options for safely inspecting it are limited. Open it in Adobe Reader and you might trigger the payload. Open it in LibreOffice and the macro runs. Send it to VirusTotal and you get a score but not an understanding.
CorbelPurge is built to fill that gap. It is a strict-Rust document sanitizer that never executes embedded scripts, macros, or active content -- but still gives you a full understanding of what is inside the file, then carves the malicious bits into a quarantine tarball with forensic reports and leaves behind a cleansed derivative you can actually read.
## What CorbelPurge Does
- **Parses** PDF, EPUB, DOCX, and Markdown files into a unified intermediate representation (`Document` struct with `TextNode` and `ExecutableVector` items)
- **Scans** with a two-pass contextual engine: executable vectors go through `heuristics::classify_vector()` against file-signature, shellcode, phishing, and URI-allowlist tables; text nodes go through `context_filter::evaluate()` which distinguishes weaponized content from educational security literature (CVE writeups, code blocks, academic language)
- **Tags** known exploits automatically -- 11 entries in the built-in CVE table (CVE-2017-11882, CVE-2018-4990, CVE-2010-0188, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199, CVE-2012-0158, CVE-2015-2545, CVE-2021-40444, CVE-2022-30190, and EPUB-SCRIPT-INJECTION)
- **Quarantines** extracted payloads into a compressed `quarantine_<ts>_<sha>.tar.gz` containing the original file, a JSON report, a Markdown report, and one `.bin` per carved payload -- each paired with a `.hex` annotated hex dump and a `.info` JSON metadata file
- **Cleanses** documents so you can read them safely, either as Markdown (default) or in the original format via `--preserve-format` (EPUB entries stripped from the ZIP, DOCX macros and embeddings removed, PDF objects deleted via lopdf)
- **Studies** documents in place via `corbel-purge study <path>`, which renders the original to a single annotated HTML file with malicious regions wrapped in inline `<span>` tags color-coded by classification
- **Defends** against zip-bomb attacks with streaming byte-counting (`util::read_with_cap()`) that counts actual decompressed bytes rather than trusting ZIP central-directory size headers, plus a cumulative `total_archive_scan_cap` (default 256 MiB) across all entries in a multi-entry archive
## What Is New in v0.4.2
### Glyphfix (carried from v0.4.1)
The iced 0.13 GUI previously rendered 11 Unicode glyphs as tofu boxes because iced's default embedded font (a subset of DejaVu Sans) does not cover them. Every problematic glyph was swapped for an ASCII equivalent that is guaranteed to render: shield emoji became `[+]`, block characters became `#` and `-`, the warning sign became `!`, and so on. No new dependencies, no embedded fonts, no binary-size regression.
### About / License overlay (carried from v0.4.1)
The ABOUT / LICENSE button in the GUI footer is now wired up. Clicking it opens a floating info panel anchored to the top-right corner -- dark panel, gold border, drop shadow, close button in the header, structured metadata (title, version, description, author, website, license, tech stack, copyright). A 55%-opacity black backdrop dims the underlying UI while the panel is open.
### What else changed
- **Version bumped** to 0.4.2
- **Docs** (`README.md`, `MANIFEST.md`, `TODO.md`, `FIX-NOTES-glyphfix.md`, the `pdf_parser.rs` header comment) all updated to drop pdf-render references
- **Test suite** unchanged: 154 tests still pass (127 unit + 23 integration + 4 zip-bomb defense)
## What Came Before
- **v0.4.1:** Glyphfix patch, About / License overlay wired up
- **v0.3.0:** Study mode (`corbel-purge study`), payload carving v2 (`.hex` + `.info` files alongside `.bin`), EPUB/Markdown cleansed-document viewer in the GUI, external threat-intel feeds (`--rules` and `--cve-db` flags + `ExternalRulesData` static storage), expanded CVE table (4 new entries: CVE-2012-0158, CVE-2015-2545, CVE-2021-40444, CVE-2022-30190), total-memory budget for multi-entry archives, PDF repackage dangling-reference cleanup, OPF manifest cleanup after EPUB repackage, GUI CVE badges parsed from `Finding.context_notes`
- **v0.2.0:** 4-format parsing (PDF via `lopdf`, EPUB via `zip`, Markdown via `pulldown-cmark`, DOCX via `zip` + XML regex), unified intermediate representation, layered contextual scanner, CVE tagging, quarantine packaging, document cleansing (Markdown + PreserveFormat modes), zip-bomb defense, iced 0.13 GUI dashboard
## Who Is This For?
- **Incident responders** who need to safely inspect suspicious attachments
- **Threat researchers** studying document-based exploits from malware corpora
- **Security teams** who want CI/CD scanning for incoming documents (`--abort-on-threat` exits 2 on any malicious finding)
- **Academics** analyzing CVE write-ups and exploit techniques (the context filter whitelists educational content so a CVE writeup that quotes shellcode does not get flagged)
- **Anyone** who receives a document from an untrusted source and wants to read it without risking code execution
## What CorbelPurge Is Not
CorbelPurge does not create, generate, weaponize, or distribute exploits. It does not execute embedded scripts, macros, or active content. It is not an offensive security tool. It is a defensive research instrument.
## Get Started
```bash
tar xzf corbel-purge-0.4.2.tar.gz
cd corbel-purge-0.4.2
cargo build --release
./target/release/corbel-purge scan path/to/suspicious.pdf
```
See [QUICKSTART.md](QUICKSTART.md) for the full step-by-step guide, including
the `study` subcommand, PreserveFormat mode, CI integration, external
threat-intel feeds, and the optional iced GUI build.
## Roadmap Preview
The highest-priority items currently on the deferred list:
- **Bundled historical exploit corpus** -- too risky to ship; would bloat the repo
- **Sandbox hardening / fuzzing the tool itself** -- the tool is not the target per the threat model
- **Encrypted quarantine tarballs** -- unnecessary for the research use case; disk encryption is the operator's responsibility
- **Full OOXML schema validation** -- the current coarse regex extraction in `docx_parser.rs` is sufficient for the threat model
- **Magic-byte sniffing** -- `DocumentFormat::from_path()` uses extension only; a mismatched extension causes a parse error (safe failure mode)
- **Full syntect syntax highlighting** -- the cleansed-document viewer uses a lightweight dark-panel approach; `syntect` would add a heavy dependency to a security-critical crate
See [TODO.md](TODO.md) for the full development roadmap.
## License
GNU General Public License v3.0-or-later. Free software for research, security analysis, and academic study. See [LICENSE](LICENSE) for full terms.

71
Cargo.toml Executable file
View File

@ -0,0 +1,71 @@
[package]
name = "corbel-purge"
version = "0.4.2"
edition = "2021"
description = "Strict Rust secure document sanitizer & threat neutralizer for PDF, EPUB, Markdown, and DOCX"
license = "GPL-3.0-or-later"
authors = ["Jeremy Anderson"]
repository = "https://git.dcos.net/dcosnet/corbel"
homepage = "https://git.dcos.net/dcosnet/corbel"
[[bin]]
name = "corbel-purge"
path = "src/main.rs"
[[bin]]
name = "corbel-purge-gui"
path = "src/bin/gui.rs"
required-features = ["gui"]
[lib]
name = "corbel_purge"
path = "src/lib.rs"
[features]
default = []
# Enable the iced GUI binary. When disabled, the crate compiles as a headless
# CLI + library only — useful for CI and server-side scanning pipelines.
gui = ["dep:iced", "dep:rfd", "dep:tokio"]
[dependencies]
# --- PDF structure inspection ---
lopdf = "0.34"
# --- EPUB (ZIP container) inspection ---
# Using `zip` directly gives us byte-level control over the EPUB container,
# which the security scanner needs (we have to inspect raw stream bytes
# inside each ZIP entry, not just rendered XHTML).
zip = "2"
# --- Markdown parsing ---
pulldown-cmark = { version = "0.12", default-features = false }
# --- Quarantine packaging ---
tar = "0.4"
flate2 = "1"
# --- Serialization & hashing ---
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sha2 = "0.10"
hex = "0.4"
# --- Misc utilities ---
thiserror = "2"
regex = "1"
chrono = { version = "0.4", features = ["serde"] }
# --- Optional GUI (iced) ---
iced = { version = "0.13", optional = true, features = ["tokio", "debug"] }
rfd = { version = "0.15", optional = true }
tokio = { version = "1", optional = true, features = ["rt-multi-thread", "macros", "fs"] }
[dev-dependencies]
tempfile = "3"
pretty_assertions = "1"
[profile.release]
opt-level = 3
lto = "thin"
codegen-units = 1
strip = "symbols"

133
FIX-NOTES-glyphfix.md Normal file
View File

@ -0,0 +1,133 @@
# Patch notes — 0.4.1-glyphfix + about-overlay
## Patch 1: Glyphfix
The GUI (`src/bin/gui.rs`) used 11 Unicode glyphs that are not present in
iced 0.13's default embedded font (a subset of DejaVu Sans). Every missing
glyph rendered as a tofu box (□) on screen.
Affected locations (visible in the original screenshot):
| Glyph | Code point | Where it appeared |
|-------|------------|-------------------|
| 🛡 | U+1F6E1 | Brand icon, before "CORBELPURGE" |
| ▴ ▾ | U+25B4 / U+25BE | Chevrons before PATHS / OPTIONS / CONSOLE |
| █ | U+2588 | CONSOLE section label + gauge filled cells |
| ░ | U+2591 | Gauge empty cells (the 16-box string) |
| ✨ | U+2728 | CLEANED stat |
| 📄 | U+1F4C4 | COPIED stat |
| ⚠ | U+26A0 | ERRORS stat |
| ⏻ | U+23FB | START PROCESSING button |
| 🧹 | U+1F9F9 | CLEAR LOG button |
| ℹ | U+2139 | ABOUT / LICENSE button |
| • | U+2022 | Findings bullet (only visible with findings) |
### Note on the `pdf-render` glyphs
An earlier version of this patch also documented three glyphs that only
appeared when the optional `pdf-render` feature was enabled (📁 on the
IN/OUT browse buttons, ◀ ▶ on PDF prev/next page buttons, 🖱 on the
"view original PDF" button). The `pdf-render` feature was removed in
v0.4.2 — CorbelPurge scans four formats (PDF/EPUB/MD/DOCX) but only
PDF had a visual renderer, which was an inconsistency that wasn't worth
the pdfium dynamic-library dependency. Those three rows are kept out of
the table above since the code paths no longer exist.
### Fix
Swapped every problematic glyph for an ASCII equivalent that is guaranteed
to render in iced's default font. No new dependencies, no embedded fonts,
no binary-size regression.
| Old | New | Rationale |
|---------------|------|-----------|
| 🛡 | `[+]` | "protected" badge feel |
| ▴ (open) | `-` | CLI-standard "expanded" marker |
| ▾ (closed) | `+` | CLI-standard "collapsed" marker |
| █ (filled) | `#` | block character everyone has |
| ░ (empty) | `-` | clean empty-track look |
| ✨ | `*` | clean/done marker |
| 📄 | `>` | "copied out" arrow |
| ⚠ | `!` | universal warning |
| ⏻ | `>` | "start" arrow |
| 🧹 | `x` | universal clear/delete |
| ℹ | `i` | universal info |
| • | `*` | bullet, safe everywhere |
| 📁 | `...` | standard "browse" indicator |
### Alternative (not applied)
If you want prettier icons later, embed the `iced_fonts` crate
(https://crates.io/crates/iced_fonts) which bundles Noto Sans Symbols 2
and Bootstrap/Material icon fonts. Then restore the original glyphs and
apply `.font(iced_fonts::REQUIRED_FONT)` to each `text()` call. This adds
~600 KB to the binary but gives you proper iconography.
---
## Patch 2: About / License overlay
### Problem
The ABOUT / LICENSE button in the footer was wired up to nothing — it had
no `.on_press(...)` handler, so clicking it did nothing.
### Fix
Added a floating info panel overlay, modelled on the ferret about-panel
screenshot. Visual style:
- Dark panel background (`colors::panel()`)
- Gold border (`colors::gold()`, 1.5px, 6px rounded corners)
- Drop shadow (offset 4px down, 12px blur, 60% opacity)
- Close (X) button in the top-right corner of the panel header
- Structured content:
- Title "CORBELPURGE" + version (from `CARGO_PKG_VERSION`)
- Two-line description (from `Cargo.toml` description)
- Metadata rows: Author / Website / License (last two from
`CARGO_PKG_REPOSITORY` and `CARGO_PKG_LICENSE`)
- Thin separator
- Footer: tech stack + copyright
### Behavior
- Clicking ABOUT / LICENSE toggles the overlay open/closed.
- Clicking the X button (or pressing ABOUT / LICENSE again) closes it.
- When open, the main UI is dimmed (55% opacity black backdrop).
- No click-outside-to-close — see Implementation notes below.
### Implementation notes
iced 0.13 has no `Stack` widget (true non-modal overlays landed in
0.14). To emulate the ferret "floating panel over content" look without
adding a dependency, we use a full-window dim backdrop with the panel
positioned in the top-right via a `row + Space::Fill` layout.
Click-outside-to-close was considered but dropped: wrapping the panel
in a no-op `Button` would make every label inside the panel close the
modal on click (since iced 0.13 buttons don't stop event propagation to
their parent). ESC-key handling requires a keyboard subscription, which
is a separate feature.
### Future enhancements
1. **Upgrade to iced 0.14+** to get the `Stack` widget, enabling true
non-modal floating panels with click-outside-to-close.
2. **Add a keyboard subscription** for the Escape key to close the
overlay without needing to click the X button.
3. **Embed `iced_fonts`** and restore the original emoji glyphs for a
richer visual style.
4. **Make the website URL clickable** — currently it's just colored
text. iced 0.13 doesn't have a native hyperlink widget, but you
could shell out to `xdg-open` via a button.
### Files changed
- `src/bin/gui.rs`:
- Added `about_open: bool` to `CorbelGui` state
- Added `ToggleAbout` and `CloseAbout` Message variants + update handlers
- Wired ABOUT / LICENSE button's `on_press`
- Added `build_about_panel()` and `build_about_overlay()` functions
- Added `about_dim_style()`, `about_panel_style()`, `about_close_btn_style()` style helpers
- Modified `view()` to render the overlay when `about_open` is true
- No changes to `Cargo.toml` — no new dependencies added

438
MANIFEST.md Executable file
View File

@ -0,0 +1,438 @@
# Project Manifest: CorbelPurge (v0.3.0)
## Strict Rust Secure Document Sanitizer & Threat Neutralizer
**Author:** Jeremy Anderson — https://git.dcos.net/dcosnet/corbel
**License:** GPL-3.0-or-later
**Status:** v0.2.0 MVP shipped. v0.3.0 planning: study mode, payload carving v2,
scanner extensibility, GUI navigation improvements.
---
## 1. Executive Summary
CorbelPurge is a high-assurance document sanitizer written in strict Rust
(`#![forbid(unsafe_code)]`) for **PDF, EPUB, Markdown, and DOCX** formats.
It parses documents into a unified intermediate representation, runs a
layered contextual scanner that distinguishes educational security
literature from active malicious injections, then produces cleansed
derivatives with all executable content stripped. Malicious payloads are
carved into quarantine tarballs with full forensic reports.
**Threat model:** CorbelPurge is a researcher's tool for studying old
exploits, not a target itself. The scanner safely ingests adversarial
samples (zip bombs, malicious docs from malware corpora) without
choking, so the researcher can study them. The tool itself is assumed
to not be the target.
---
## 2. Core Specifications & Non-Goals
### Core Features
- **Multi-Format Support:** Native parsing pipelines for PDF (`lopdf`),
EPUB (`zip`), Markdown (`pulldown-cmark`), DOCX (`zip` + XML regex).
- **Unified Intermediate Representation:** All parsers emit the same
`Document` struct (`src/core/types.rs`). Downstream modules never
touch format-specific structures.
- **Automated Threat Scanning:** Two-pass engine — executable vectors
go through `heuristics::classify_vector()`, text nodes go through
`context_filter::evaluate()`, then findings are annotated by
`cve_tags::match_cve()`.
- **Context-Aware Heuristics:** Differentiation between static text
content (including code blocks discussing vulnerabilities) and
executable/malicious injection vectors. Academic markers (CVE IDs,
"remediation", code-block context) whitelist educational content.
- **Weaponization Detection:** Long hex-encoded runs (16+ `\xNN`),
64+ char base64 blobs, NOP sleds, Metasploit stager prologues are
flagged as malicious regardless of context.
- **CVE Tagging:** 7 known exploits (CVE-2010-0188, CVE-2018-4990,
CVE-2017-11882, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199,
EPUB-SCRIPT-INJECTION) with byte-signature and vector-type matching.
- **Forensic Extraction & Reporting:** JSON + Markdown reports with
SHA-256, vector location, classification, CVE tag, context notes.
Payload carving into standalone `.bin` files inside quarantine tarballs.
- **Secure Quarantine:** Compressed `quarantine_<ts>_<sha>.tar.gz`
containing original file, report, and extracted payloads.
- **Document Cleansing (two modes):**
- *Markdown mode* (default): produces a safe Markdown derivative via
`cleanse::sanitizer::sanitize()`. A Markdown file cannot carry
executable content by definition.
- *PreserveFormat mode* (`--preserve-format`): repackages the document
in its original format (PDF/EPUB/DOCX) with malicious entries
stripped via `cleanse::repackage::repackage()`.
- **Zip-Bomb Defense:** `util::read_with_cap()` counts actual decompressed
bytes rather than trusting ZIP central directory size headers.
Configurable via `Config::epub_entry_scan_cap` (default 8 MiB).
- **Optional GUI:** iced 0.13 dashboard (`src/bin/gui.rs`) with dark
theme, collapsible panels, console log, sidebar stats, file picker
(`rfd::AsyncFileDialog`).
- **No-arg-parsing-crates:** The CLI (`src/main.rs`) parses its own
arguments to keep the dependency tree minimal — a defensive measure
for security-critical software.
### Explicit Non-Goals
- No document editing, annotation writing, form-filling, or signature generation.
- No execution of embedded active scripting (PDF JS, EPUB scripts, VBA macros).
- No sandbox hardening or fuzzing of the tool itself (it is not the target).
- No bundled historical exploit corpus (too risky to ship).
- No magic-byte sniffing for format detection (extension-only is the safe
failure mode — a mismatched extension causes a parse error).
---
## 3. System Architecture & Tech Stack
CorbelPurge uses safe Rust primitives, relying on memory-safe parsers to
avoid buffer overflow vulnerabilities common in legacy C/C++ document
viewers. The `#![forbid(unsafe_code)]` crate attribute is enforced at
compile time.
### Dependency Table
| Purpose | Crate | Version |
|---------|-------|---------|
| PDF structure inspection | `lopdf` | 0.34 |
| EPUB / DOCX containers | `zip` | 2 |
| Markdown parsing | `pulldown-cmark` | 0.12 |
| Quarantine packaging | `tar` + `flate2` | 0.4 / 1 |
| Serialization | `serde` + `serde_json` | 1 |
| Hashing | `sha2` + `hex` | 0.10 / 0.4 |
| Datetime | `chrono` | 0.4 |
| Context filter | `regex` | 1 |
| Error handling | `thiserror` | 2 |
| GUI (optional) | `iced` | 0.13 |
| File picker (optional) | `rfd` | 0.15 |
| Async runtime (optional) | `tokio` | 1 |
| Test utilities | `tempfile` + `pretty_assertions` | 3 / 1 |
### Feature Flags
- `default` — headless CLI + library only. No GUI dependencies.
- `gui` — adds the iced GUI binary (`corbel-purge-gui`) and depends
on `iced`, `rfd`, `tokio`.
---
## 4. Unified Intermediate Representation
Every parser emits the same [`Document`](src/core/types.rs) struct.
The scanner, quarantine, and cleanse modules consume only this UIR —
they never touch format-specific structures. Adding a new format
requires writing exactly one new `DocumentParser` impl and adding a
match arm in `Dispatcher::parse()` (`src/parsers/mod.rs`).
Key types defined in `src/core/types.rs`:
- **`DocumentFormat`** — `Pdf`, `Epub`, `Markdown`, `Docx`. Detected by
file extension via `DocumentFormat::from_path()` in `pipeline.rs`.
- **`Location`** — per-format source location (PDF object/stream IDs,
EPUB ZIP entry paths, Markdown line:col). Used in findings to report
exactly where a threat was found.
- **`TextContext`** — semantic context of extracted text (`Paragraph`,
`Heading`, `CodeBlock`, `CodeSpan`, `Hyperlink`, `BlockQuote`,
`Metadata`, `ExecutableHook`). The heart of the context-aware scanner.
- **`TextNode`** — extracted text with location + context. Emitted by
parsers for all rendered/static content.
- **`ExecutableVector`** — a potentially dangerous element (JS action,
embedded file, script tag, OLE object, external link) with raw bytes
and optional decoded preview.
- **`VectorType`** — 15 variants covering PDF (JS, Launch, URI, GoToR,
EmbeddedFile, WidgetAction, AcroForm), EPUB (Script, ExternalResource,
Object), Markdown (Hyperlink), DOCX (Macro, ExternalLink,
EmbeddedObject, ActiveX), plus UnknownPayload.
- **`Document`** — the top-level IR: format, raw bytes, SHA-256,
metadata, text nodes, executable vectors.
- **`ThreatClassification`** — `Benign`, `Suspicious`, `EducationalContent`,
`Malicious(MaliciousType)`.
- **`MaliciousType`** — 8 specific categories (ActiveJavaScriptInjection,
LaunchAction, MaliciousEmbeddedFile, ObfuscatedShellcode,
SuspiciousUri, EpubActiveScript, DocxActiveContent, Other).
- **`Finding`** — classification, location, vector type, payload preview,
context notes, recommendation.
- **`Recommendation`** — `Allow`, `WhitelistAsEducational`, `Quarantine`,
`QuarantineAndCleanse`.
- **`ScanReport`** — aggregate of all findings with counts and metadata.
- **`CorbelError`** — exhaustive error enum (Io, Serde, PdfParse,
EpubParse, MarkdownParse, UnknownFormat, ThreatDetected, Quarantine,
Cleanse, Internal).
---
## 5. Security Scanning Pipeline
The pipeline is orchestrated by `Pipeline::run()` in `src/core/pipeline.rs`.
Given an input path, it: reads bytes, detects format, dispatches to the
correct parser, runs the scanner, then conditionally quarantines and
cleanses.
### Step 0: Weaponization Check
`context_filter::has_weaponization_indicators()` runs first on every text
node. Pure obfuscation indicators (16+ `\xNN` runs, 64+ char base64
blobs, 2+ shell commands in non-code context) are flagged as
`ObfuscatedShellcode` regardless of surrounding context. Code blocks
and block quotes get an exception — weaponized-looking content there is
almost certainly a research writeup.
### Step 1: Structural vs. Textual Isolation
The scanner in `scanner::scan()` walks two separate lists:
- **Executable Vectors** (untrusted by default) — dispatched to
`heuristics::inspect_vector()` which calls `classify_vector()`.
This function matches against `signatures.rs` tables (file signatures,
shellcode patterns, phishing TLDs, brand homographs, URL shorteners,
suspicious keywords) and the `Config::allowed_uri_schemes` whitelist.
- **Static Text Nodes** (context-dependent) — dispatched to
`context_filter::evaluate()` which looks for suspicious signatures
(`/JavaScript`, `eval(`, `shellcode`, `exploit`, etc.) and then
checks whether the surrounding context is educational.
### Step 2: Context-Aware Filtering
`context_filter::looks_educational()` checks for:
- **Structural signals:** code blocks, code spans, block quotes — always
educational.
- **Lexical signals:** presence of academic markers ("CVE-",
"vulnerability", "remediation", "patch", "mitigation", "advisory",
"for example", "proof of concept", etc.)
**Decision:** A signature in an `ExecutableHook` context is flagged as
malicious immediately. A signature in a code block describing a CVE
with remediation steps is whitelisted as `EducationalContent`.
### Step 3: CVE Tagging
After classification, `scanner::scan()` calls `cve_tags::match_cve()`
for each finding. If a CVE matches, the ID and name are appended to the
finding's `context_notes` field. The forensic reporter (`reporter.rs`)
includes the full CVE description.
| CVE ID | Name | Detection Signal |
|--------|------|------------------|
| CVE-2017-11882 | Equation Editor RCE | DOCX OLE + "Equation" in payload |
| CVE-2018-0802 | Equation Editor RCE (variant) | DOCX OLE + PE signature, no "Equation" |
| CVE-2017-8570 | Office RTF OLE Object RCE | DOCX OLE + RTF magic bytes |
| CVE-2017-0199 | Office OLE2Link RCE | DOCX external link + `.hta` target |
| CVE-2018-4990 | Adobe Reader JS RCE | PDF JS action >500 bytes |
| CVE-2010-0188 | PDF LibTiff Buffer Overflow | PDF embedded file + TIFF magic |
| EPUB-SCRIPT-INJECTION | EPUB Active Script Injection | EPUB `<script>` (malicious) |
### Heuristic Signature Tables (`signatures.rs`)
Centralized static threat-intel tables:
- **PHISHING_TLDS** — 30+ TLDs overrepresented in phishing URLs.
- **SUSPICIOUS_URL_KEYWORDS** — 30+ path/host keywords (login, signin,
verify, password, credential, etc.).
- **URL_SHORTENER_DOMAINS** — 25+ shortener domains.
- **COMMON_PHISHING_BRANDS** — 50+ brand names with homograph variants
(micros0ft, paypa1, app1e, etc.). Two-pass matching: homograph
variants always flag; canonical spellings are suppressed if the host
is a genuine brand domain.
- **KNOWN_FILE_SIGNATURES** — 15+ magic-byte patterns (PE, ELF, Mach-O,
OLE2, RTF, VBA, LNK, SWF, Java, Python, HTA).
- **SHELLCODE_PATTERNS** — 9 prologues (NOP sled, Metasploit stager,
x86_64 syscall, shikata_ga_nai, etc.).
---
## 6. Quarantine & Cleansing Workflow
When the scanner identifies a malicious finding, the pipeline conditionally
activates quarantine and cleansing.
### Pipeline flow (from `pipeline.rs`)
```text
input path
│
▼
DocumentFormat::from_path() ──► Dispatcher::parse() ──► Document (UIR)
│
▼
scanner::scan()
│
ScanReport
(findings, counts)
│
┌─────────────────────────┼──────────────────────┐
│ │ │
no malicious malicious > 0 quarantine +
findings AND abort cleanse
│ on threat │
▼ ▼ ▼
return result Err(ThreatDetected) quarantine::handle()
+ cleanse::cleanse()
│
▼
PipelineResult
(paths to tarball,
report, cleansed)
```
### Quarantine (`quarantine/`)
1. **Payload Extraction** (`extractor.rs`): For each `Malicious` finding,
locates the corresponding `ExecutableVector` by matching `Location`
values, carves its `raw_payload` into an `ExtractedPayload` with a
sanitized filename (e.g. `payload_pdf_000_obj42_pdf-javascript.bin`).
2. **Forensic Reporting** (`reporter.rs`): Builds a JSON report (schema
version 1) with source metadata, finding details, CVE descriptions,
and extracted payload info. Also builds a Markdown report with the
`# CorbelPurge Forensic Report` header.
3. **Tarball Packaging** (`mod.rs`): Writes `quarantine_<ts>_<sha>.tar.gz`
containing `original.<ext>`, `report.json`, `report.md` (if enabled),
and one `.bin` per carved payload.
### Cleansing (`cleanse/`)
Dispatched by `cleanse::cleanse()` based on `Config::cleanse_mode`:
- **Markdown mode** (`sanitizer.rs`): Emits a header banner with SHA-256
and format, a `CorbelPurge Notice` blockquote, the original title, then
all text nodes whose locations don't match a malicious finding. Code
blocks are fenced, block quotes are prefixed with `>`, hyperlinks
drop their destinations (only text is kept).
- **PreserveFormat mode** (`repackage.rs`):
- **EPUB**: Rebuilds the ZIP, skipping entries whose paths match
malicious findings. Cleans inline `<script>`, `<iframe>`, `<object>`,
`<embed>` tags from remaining XHTML entries. Neutralizes
`src="http..."` to `src="#"`.
- **DOCX**: Strips `word/vbaProject.xml`, `word/embeddings/*`,
`word/activeX/*`, and entries matching malicious findings.
Rewrites `word/_rels/document.xml.rels` to drop
`TargetMode="External"` relationships.
- **PDF**: Deletes objects matching malicious findings via
`lopdf::Document::delete_objects()`. Strips `/OpenAction`, `/AA`,
`/Names`, `/AcroForm` from the root catalog. Rebuilds xref on save.
- **Markdown**: Falls back to the Markdown sanitizer.
---
## 7. Zip-Bomb Defense
`util::read_with_cap()` is the zip-bomb defense for ZIP-container formats
(EPUB, DOCX). It reads in 8 KiB chunks and counts **actual decompressed
bytes** rather than trusting the ZIP central directory size header.
- **Lying size header:** ZIP declares 100 bytes, actually decompresses
to 1 MiB — detected and truncated at `cap` (default 8 MiB).
- **Decompression-ratio bomb:** 42 KB ZIP decompresses to petabytes —
stopped at `cap` bytes and never allocates more.
- **What is NOT defended:** Many small entries (10,000 × 1 MiB = 10 GiB)
— the per-entry cap doesn't help here. This is tracked as a TODO item
(`total_archive_scan_cap`).
Truncated bytes are still emitted as `UnknownPayload` vectors so the
file-signature matcher can inspect the prefix.
---
## 8. GUI Architecture
The iced 0.13 dashboard (`src/bin/gui.rs`) uses a dark theme:
- **48px header** — brand, version, GPL-3.0 badge.
- **Collapsible left panel** — PATHS (input/output with folder pickers
via `rfd::AsyncFileDialog`), OPTIONS (toggles: preserve-format,
strip-metadata, recursive, abort-on-threat), CONSOLE (timestamped
log with [OK]/[ERROR] tags).
- **280px right sidebar** — Unicode block-character progress gauge
(`█`/`░`), stats (CLEANED/COPIED/ERRORS), RUN CONFIG summary,
LAST SCAN findings.
- **56px footer** — START PROCESSING (gold), STOP, CLEAR LOG (teal),
ABOUT/LICENSE buttons + status bar.
The GUI spawns the pipeline via `tokio::spawn_blocking` and streams
results to the console log.
---
## 9. Public API Surface
The crate root (`lib.rs`) re-exports:
```rust
pub use core::{config::{CleanseMode, Config}, pipeline::{Pipeline, PipelineResult}, types::*};
pub use util::sha256_hex;
```
The `Pipeline` struct is the only thing most callers need:
```rust
use corbel_purge::{Pipeline, Config, CleanseMode};
let config = Config::with_workspace("/tmp/output")
.override_from_env();
let pipeline = Pipeline::with_config(config);
let result = pipeline.run("suspicious.pdf")?;
```
The library can also be embedded as an in-memory scanner (no filesystem
access) via `Pipeline::run_on_bytes()`.
---
## 10. Configuration
All tunables live in `Config` (`src/core/config.rs`). Runtime overrides
via environment variables (applied by `Config::override_from_env()` and
by the CLI's `apply_env_overrides()`):
| Variable | Default | Field |
|----------|---------|-------|
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | `quarantine_dir` |
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | `cleanse_dir` |
| `CORBEL_ABORT_ON_THREAT` | `false` | `abort_on_threat` |
| `CORBEL_EMIT_SUSPICIOUS` | `true` | `emit_suspicious` |
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | `emit_markdown_report` |
CLI flags (`src/main.rs`):
`--workspace <dir>`, `--abort-on-threat`, `--quiet` / `-q`,
`--recursive` / `-r`, `--preserve-format`.
---
## 11. Test Suite
- **Unit tests:** In every module (config, types, pipeline, parsers,
scanner submodules, quarantine, cleanse, util). Cover format detection,
signature matching, context filtering, CVE tagging, payload extraction,
sanitizer output, zip-bomb defense.
- **Integration tests** (`tests/pipeline_integration.rs`): 22 tests
running the full pipeline against fixture files — benign and malicious
PDFs, EPUBs, DOCXes, Markdown files. Covers quarantine, cleansing,
PreserveFormat repackage, abort-on-threat, educational whitelisting,
JSON/Markdown report content.
- **Zip-bomb tests** (`tests/zip_bomb_defense.rs`): Defense against lying
size headers and honest-but-oversized entries.
- **Fixtures** (`tests/fixtures/`): Generated by Python scripts in
`scripts/`. Requires `pypdf`, `reportlab`, `python-docx`.
---
## 12. Planned Future Work (v0.3.0+)
See `TODO.md` for the full task list. Key directions:
- **Study mode** — annotated HTML output showing exploit locations in
context, reusing the existing `ScanReport` and `Location` types.
- **Payload carving v2** — standalone `.hex` dumps and `.info` JSON
metadata alongside the existing `.bin` files.
- **Total-memory budget** for multi-entry archives (cross-entry cap).
- **PDF repackage: dangling reference cleanup** after object deletion.
- **OPF manifest cleanup** after EPUB repackage.
---
## 13. License
GPL-3.0-or-later. Copyright (c) 2025 Jeremy Anderson.
https://git.dcos.net/dcosnet/corbel

195
QUICKSTART.md Normal file
View File

@ -0,0 +1,195 @@
# Quick Start Guide
Get CorbelPurge built and running in under five minutes.
## Prerequisites
- **Rust** 1.70+ (install via [rustup](https://rustup.rs/))
- **Python 3** (only needed if you want to regenerate test fixtures)
That is it. The headless CLI has zero GUI dependencies and no native libraries.
## Step 1: Get the Source
```bash
tar xzf corbel-purge-0.4.2.tar.gz
cd corbel-purge-0.4.2
```
## Step 2: Build the CLI
```bash
cargo build --release
```
The binary lands at `target/release/corbel-purge`. Verify it works:
```bash
./target/release/corbel-purge --help
```
You should see the usage banner listing `scan`, `scan-dir`, `study`, and the
supported flags (`--workspace`, `--abort-on-threat`, `--quiet`, `--recursive`,
`--preserve-format`, `--rules`, `--cve-db`).
## Step 3: Scan Your First File
```bash
# Scan a suspicious PDF you received
./target/release/corbel-purge scan suspicious_document.pdf
```
If the file contains threats, you will see a summary block printed to your
terminal (findings count, classification, SHA-256, output paths), and the
pipeline will write:
- A quarantine tarball in `./corbel_quarantine/` (`quarantine_<ts>_<sha>.tar.gz`)
containing `original.<ext>`, `report.json`, `report.md`, and one `.bin` per
carved payload (plus paired `.hex` and `.info` files for each payload)
- A cleansed Markdown derivative in `./corbel_clean/`
- Standalone `report_<timestamp>_<sha>.json` and `.md` for programmatic access
If the file is clean, the summary block simply reports zero findings and no
quarantine output is written.
## Step 4: Try PreserveFormat Mode
If you want a cleaned version that keeps the original format (e.g. a cleaned
`.epub` you can actually read in an e-reader):
```bash
./target/release/corbel-purge scan research_paper.epub --preserve-format
```
This produces a `cleansed_<ts>_<sha>.epub` with malicious entries stripped from
the ZIP container but chapter text preserved. Works for PDF and DOCX too.
## Step 5: Study a Document In Place
The `study` subcommand renders the original document to a single annotated HTML
file with malicious regions wrapped in inline `<span>` tags, color-coded by
classification. Use it when you want to see exactly where the exploit sits in
context, without leaving the source format:
```bash
./target/release/corbel-purge study suspicious.epub
```
Output lands at `study_<ts>_<sha>.html` in the workspace. Quiet mode (`-q`)
prints just the path.
## Step 6: Scan a Directory (Optional)
```bash
# Recursively scan an inbox directory
./target/release/corbel-purge scan-dir /path/to/inbox --recursive --workspace /tmp/corbel
```
The directory walker picks up `.pdf`, `.epub`, `.md`, `.markdown`, and `.docx`
files. Each file is logged with a `[OK]`, `[MALICIOUS]`, or `[ERROR]` tag.
## Step 7: CI Integration (Optional)
Use `--abort-on-threat` to make CorbelPurge a CI gate. Exit code 2 means
threats were found:
```bash
# In your CI pipeline
./target/release/corbel-purge scan incoming_document.pdf --abort-on-threat --quiet
# exit 0: clean
# exit 2: has threats -> fail the build
# exit 1: hard error (parse failure, IO, etc.)
```
`--quiet` suppresses the summary block and prints only the JSON report path,
which is handy for piping into downstream tooling.
## Step 8: Plug In External Threat-Intel Feeds (Optional)
The built-in signature tables and CVE database are static, but you can layer
your own on top at runtime:
```bash
# Load additional YARA-style signature rules
./target/release/corbel-purge scan suspicious.pdf --rules my_rules.json
# Load additional CVE signature entries
./target/release/corbel-purge scan suspicious.docx --cve-db my_cve_db.json
# Or set them via environment variables
export CORBEL_EXTERNAL_RULES=/etc/corbel/rules.json
export CORBEL_EXTERNAL_CVE_DB=/etc/corbel/cve_db.json
./target/release/corbel-purge scan suspicious.pdf
```
External rules are matched alongside the built-in tables; nothing is
overridden. See `MANIFEST.md` for the JSON schema.
## Step 9: Build the GUI (Optional)
The iced 0.13 dashboard GUI is behind the `gui` feature flag:
```bash
cargo build --release --features gui --bin corbel-purge-gui
./target/release/corbel-purge-gui
```
The GUI provides file pickers, toggle switches for preserve-format /
abort-on-threat / recursive, a timestamped console log, a sidebar with a
Unicode progress gauge and per-file stats, and a cleansed-document viewer.
All scanning runs through the same `Pipeline` the CLI uses, via
`tokio::spawn_blocking`.
## What You Should See
### Clean file output:
```
────────────────────────────────────────────────────────
scan complete: PDF
source: benign.pdf
sha256: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
text nodes: 12
vectors: 0
findings: 0 malicious, 0 educational, 0 total
────────────────────────────────────────────────────────
```
### Threat found output:
```
────────────────────────────────────────────────────────
scan complete: PDF
source: suspicious.pdf
sha256: a1b2c3...
text nodes: 8
vectors: 1
findings: 1 malicious, 0 educational, 1 total
quarantine: ./corbel_quarantine/quarantine_20260801T120000_abc12345.tar.gz
cleansed: ./corbel_clean/cleansed_20260801T120000_abc12345.md
json report: ./corbel_quarantine/report_20260801T120000_abc12345.json
md report: ./corbel_quarantine/report_20260801T120000_abc12345.md
────────────────────────────────────────────────────────
```
Exit code is `2` when any malicious finding is produced.
## Key Environment Variables
| Variable | Default | When to Change It |
|----------|---------|-------------------|
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | Point at a shared quarantine volume |
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | Point at an output directory for cleaned files |
| `CORBEL_ABORT_ON_THREAT` | `false` | Set to `true` in CI pipelines |
| `CORBEL_EMIT_SUSPICIOUS` | `true` | Set to `false` to only report Malicious (not Suspicious) |
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | Set to `false` if you only need JSON |
| `CORBEL_TOTAL_ARCHIVE_SCAN_CAP` | `268435456` (256 MiB) | Cumulative cap across all entries in a multi-entry archive |
| `CORBEL_EXTERNAL_RULES` | (unset) | Path to an external signature-rules JSON file |
| `CORBEL_EXTERNAL_CVE_DB` | (unset) | Path to an external CVE database JSON file |
## Next Steps
- Read [README.md](README.md) for the full feature overview and security model
- Read [MANIFEST.md](MANIFEST.md) for the detailed technical specification
- Read [TODO.md](TODO.md) for the development roadmap
- Run `cargo test` to verify all 154 tests pass in your environment

191
README.md Executable file
View File

@ -0,0 +1,191 @@
# CorbelPurge
> Strict Rust document sanitizer & threat neutralizer for PDF, EPUB, Markdown, and DOCX.
**Author:** Jeremy Anderson — [https://git.dcos.net/dcosnet/corbel](https://git.dcos.net/dcosnet/corbel)
**License:** GPL-3.0-or-later
![Corbel-Purge-ss](./corbel-purge-ss.png)
CorbelPurge parses documents into a unified intermediate representation (`Document` struct), runs a layered contextual scanner that distinguishes educational security literature from active malicious injections, and produces cleansed derivatives with all executable content stripped. Malicious payloads are carved into quarantine tarballs with full forensic reports.
## What it does
Given a file path, `Pipeline::run()` in `src/core/pipeline.rs`:
1. **Detects format** via `DocumentFormat::from_path()` (extension-based: `.pdf`, `.epub`, `.md`/`.markdown`, `.docx`).
2. **Parses** via `parsers::Dispatcher` into a `Document` containing `TextNode` (static text with semantic context) and `ExecutableVector` (active content like JS streams, embedded files, script tags, VBA macros) items.
3. **Scans** with a two-pass engine:
- `heuristics::inspect_vector()` classifies every executable vector against file-signature tables, shellcode patterns, phishing heuristics, and URI allowlists.
- `context_filter::evaluate()` checks text nodes for suspicious signatures and determines whether the surrounding context is educational (code blocks, CVE writeups, academic language) or weaponized.
- `cve_tags::match_cve()` annotates findings with known exploit IDs.
4. **Quarantines** (when malicious findings exist) — `quarantine::handle()` carves payloads into `quarantine_<ts>_<sha>.tar.gz` with `original.<ext>`, `report.json`, `report.md`, and one `.bin` per payload.
5. **Cleanses** (when recommended) — produces a sanitized derivative:
- **Markdown mode** (default): `cleanse::sanitizer::sanitize()` emits a safe Markdown file. Text nodes at malicious locations are stripped; hyperlinks lose their destinations.
- **PreserveFormat mode** (`--preserve-format`): `cleanse::repackage::repackage()` rebuilds the original format with malicious entries removed. EPUB entries are stripped from the ZIP, DOCX macros/embeddings/external-links are removed, PDF objects are deleted via lopdf.
Supported formats: **PDF**, **EPUB**, **Markdown**, **DOCX**.
## Build
```bash
# Headless CLI (default — no GUI deps)
cargo build --release
# With iced GUI
cargo build --release --features gui --bin corbel-purge-gui
```
Requires Rust 1.70+ (stable).
Binaries:
- `target/release/corbel-purge` — headless CLI
- `target/release/corbel-purge-gui` — iced dashboard GUI (`gui` feature)
## CLI usage
The CLI is in `src/main.rs` and parses its own arguments (no `clap` dependency).
```bash
# Scan a single file
corbel-purge scan path/to/suspicious.pdf
# Scan with format-preserving output (keeps original format)
corbel-purge scan path/to/book.epub --preserve-format
# Scan with a custom workspace
corbel-purge scan path/to/file.pdf --workspace /tmp/corbel
# Scan a directory recursively
corbel-purge scan-dir path/to/inbox --recursive --workspace /tmp/corbel
# CI gate: exit non-zero on threat
corbel-purge scan path/to/file.pdf --abort-on-threat
# Quiet mode: print only the JSON report path
corbel-purge scan path/to/file.pdf --quiet
# Version
corbel-purge --version
```
### Exit codes
| Code | Meaning |
|------|----------|
| 0 | No threats found |
| 1 | Hard error (parse failure, IO, etc.) |
| 2 | One or more malicious findings |
### Environment variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `CORBEL_QUARANTINE_DIR` | `./corbel_quarantine` | Quarantine tarball output dir |
| `CORBEL_CLEANSE_DIR` | `./corbel_clean` | Cleansed document output dir |
| `CORBEL_ABORT_ON_THREAT` | `false` | Abort on first malicious finding |
| `CORBEL_EMIT_SUSPICIOUS` | `true` | Include Suspicious findings in report |
| `CORBEL_EMIT_MARKDOWN_REPORT` | `true` | Generate Markdown report alongside JSON |
## Library usage
```rust
use corbel_purge::{Pipeline, Config, CleanseMode};
// Scan a file on disk
let pipeline = Pipeline::with_config(Config::with_workspace("/tmp/output"));
let result = pipeline.run("suspicious.pdf")?;
println!("malicious: {}", result.scan_report.malicious_count());
// Scan in-memory bytes (e.g. from an email gateway)
let bytes = std::fs::read("upload.docx")?;
let result = pipeline.run_on_bytes(
bytes,
corbel_purge::DocumentFormat::Docx,
Some("upload.docx".into()),
)?;
// PreserveFormat mode
let mut config = Config::default();
config.cleanse_mode = CleanseMode::PreserveFormat;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run("book.epub")?;
```
## Source layout
```text
src/
├── main.rs # CLI: scan, scan-dir, --version, arg parsing
├── lib.rs # Crate root, CorbelError enum, public re-exports
├── util.rs # sha256_hex(), truncate_with_ellipsis(), read_with_cap()
├── bin/
│ └── gui.rs # iced 0.13 dashboard GUI (gui feature)
├── core/
│ ├── types.rs # Document, TextNode, ExecutableVector, Finding, ScanReport,
│ │ # Location, TextContext, VectorType, ThreatClassification,
│ │ # MaliciousType, Recommendation, DocumentMetadata
│ ├── config.rs # Config, CleanseMode, env overrides, workspace dirs
│ └── pipeline.rs # Pipeline, PipelineResult, DocumentFormat::from_path()
├── parsers/
│ ├── mod.rs # DocumentParser trait, Dispatcher
│ ├── pdf_parser.rs # lopdf-based PDF structure walker
│ ├── epub_parser.rs # ZIP-based EPUB container inspector
│ ├── md_parser.rs # pulldown-cmark text extractor
│ └── docx_parser.rs # OOXML ZIP inspector (regex XML extraction)
├── scanner/
│ ├── mod.rs # scan() entrypoint, CVE tag injection
│ ├── heuristics.rs # classify_vector() for executable vectors
│ ├── context_filter.rs # evaluate() for text nodes, educational vs weaponized
│ ├── signatures.rs # PHISHING_TLDS, KNOWN_FILE_SIGNATURES,
│ │ # SHELLCODE_PATTERNS, COMMON_PHISHING_BRANDS,
│ │ # SUSPICIOUS_URL_KEYWORDS, URL_SHORTENER_DOMAINS
│ └── cve_tags.rs # CVE_TABLE (7 entries), match_cve(), cve_tag()
├── quarantine/
│ ├── mod.rs # handle() — tarball writer, QuarantineOutcome
│ ├── extractor.rs # extract_payloads(), ExtractedPayload, filename sanitization
│ └── reporter.rs # build_json_report(), build_markdown_report()
├── cleanse/
│ ├── mod.rs # cleanse() dispatcher (Markdown vs PreserveFormat)
│ ├── sanitizer.rs # sanitize() — Markdown re-serializer, build_clean_markdown()
│ └── repackage.rs # repackage() — format-preserving (EPUB/DOCX/PDF)
└── ui/
├── mod.rs # GUI module gate (behind `gui` feature)
├── viewer.rs # Stub
└── alert_modal.rs # Stub
```
## Testing
```bash
# Regenerate test fixtures (requires pypdf, reportlab, python-docx)
pip install pypdf reportlab python-docx
python3 scripts/gen_fixtures.py
python3 scripts/gen_md_epub_fixtures.py
python3 scripts/gen_docx_fixtures.py
python3 scripts/gen_zip_bomb_fixtures.py
# Run all tests
cargo test
```
## Adding a new format
1. Create `src/parsers/<format>_parser.rs` implementing `DocumentParser`.
2. Add a variant to `DocumentFormat` in `src/core/types.rs` (and its `Display` impl).
3. Add a match arm in `Dispatcher::parse()` in `src/parsers/mod.rs`.
4. Add an extension check in `DocumentFormat::from_path()` in `src/core/pipeline.rs`.
5. Add repackage support in `src/cleanse/repackage.rs` (optional).
The scanner, quarantine, and cleanse modules consume only the `Document` UIR and require no changes for basic format support.
## Non-goals
- No document editing or annotation.
- No execution of embedded scripting (PDF JS, EPUB scripts, VBA macros).
- No network access during scanning — all analysis is local.
- No sandbox hardening of the tool itself.
## License
GPL-3.0-or-later. Copyright (c) 2026 Jeremy Anderson. https://git.dcos.net/dcosnet/corbel

117
TODO.md Executable file
View File

@ -0,0 +1,117 @@
# CorbelPurge v0.4.0 — TODO
Things to build next, grounded in what actually exists in the source.
Each item names the real files and functions involved.
---
## High priority
_(None — all high-priority items from v0.3.0 were completed.)_
---
## Medium priority
_(None — all medium-priority items from v0.3.0 were completed.)_
---
## Low priority
_(None — all low-priority items from v0.3.0 were completed.)_
---
## Deferred
- **Bundled historical exploit corpus** — too risky to ship; would also
bloat the repo.
- **Sandbox hardening / fuzzing the tool itself** — the tool is not the
target per the threat model.
- **Encrypted quarantine tarballs** — unnecessary for the research use case;
disk encryption is the operator's responsibility.
- **Full OOXML schema validation** — the current coarse extraction in
`docx_parser.rs` (regex on XML text) is sufficient for the threat model.
- **Magic-byte sniffing** — `DocumentFormat::from_path()` uses extension
only. A mismatched extension causes a parse error (safe failure mode).
- **Full syntect syntax highlighting** — the current cleansed-document viewer
uses a lightweight approach (dark-panel code blocks). Adding `syntect`
would provide richer highlighting but adds a heavy dependency to a
security-critical crate.
---
## Done (v0.3.0)
- Study mode: `corbel-purge study <path>` subcommand producing annotated HTML
with inline `<span>` wrappers color-coded by classification.
Files: `src/study/mod.rs`, `src/study/annotator.rs`.
- Payload carving v2: standalone `.hex` (annotated hex dump) and `.info`
(JSON metadata with vector type, location, classification, CVE tag,
SHA-256) files alongside existing `.bin`.
Files: `src/quarantine/hexdump.rs`, `src/quarantine/mod.rs`.
- Multi-page PDF navigation in GUI: prev/next buttons, page number input,
LRU page cache, wired to `PdfiumRenderer::render_page(document, page_index)`.
Files: `src/bin/gui.rs`, `src/pdfium_render.rs`. **Removed in v0.4.2** —
the PDF renderer was the only format being visually rendered (MD/DOCX/EPUB
were text-only), so the inconsistency was cut. PDF scanning, quarantine,
and cleansing are unaffected.
- EPUB/Markdown visual rendering in GUI: cleansed document viewer
using scrollable text widget in the sidebar.
Files: `src/bin/gui.rs`.
- Syntax highlighting in cleansed-document viewer: lightweight code-block
distinction using the dark-panel background.
Files: `src/bin/gui.rs`.
- External threat-intel feed integration: `--rules <path>` and
`--cve-db <path>` CLI flags, `ExternalRulesData` static storage,
external TLD/brand/keyword/signature/shellcode matching.
Files: `src/core/config.rs`, `src/scanner/signatures.rs`,
`src/scanner/cve_tags.rs`, `src/main.rs`.
- Expanded CVE table: 4 new entries (CVE-2012-0158, CVE-2015-2545,
CVE-2021-40444, CVE-2022-30190) — total 11 entries.
Files: `src/scanner/cve_tags.rs`.
- Total-memory budget for multi-entry archives: `total_archive_scan_cap`
field in `Config` (default 256 MiB), enforced in `epub_parser.rs`
and `docx_parser.rs`.
Files: `src/core/config.rs`, `src/parsers/epub_parser.rs`,
`src/parsers/docx_parser.rs`.
- PDF repackage dangling reference cleanup: `sweep_dangling_references()`
nullifies `N 0 R` patterns in remaining PDF objects after deletion.
Files: `src/cleanse/repackage.rs`.
- OPF manifest cleanup after EPUB repackage: `clean_epub_opf()` rewrites
the OPF XML to remove stale `<manifest>` and `<spine>` entries.
Files: `src/cleanse/repackage.rs`.
- PDF PreserveFormat integration tests: `preserve_format_pdf_strips_javascript`
and `preserve_format_pdf_strips_launch` in pipeline_integration.rs.
Files: `tests/pipeline_integration.rs`.
- GUI CVE badges: CVE tags parsed from `Finding.context_notes` and
rendered as teal badges in the findings list.
Files: `src/bin/gui.rs`.
## Done (v0.2.0 and earlier)
- 4-format parsing (PDF via `lopdf`, EPUB via `zip`, Markdown via
`pulldown-cmark`, DOCX via `zip` + regex XML extraction)
- Unified intermediate representation (`Document` struct with `TextNode`
and `ExecutableVector`)
- `DocumentParser` trait + `Dispatcher` in `parsers/mod.rs`
- Layered scanner: `heuristics::classify_vector()` for executable vectors,
`context_filter::evaluate()` for text nodes, `cve_tags::match_cve()`
for CVE annotation
- Threat signature tables in `signatures.rs`: 30+ phishing TLDs, 50+
brand homographs, 15+ file signatures, 9 shellcode prologues
- CVE table: 7 entries (CVE-2010-0188, CVE-2018-4990, CVE-2017-11882,
CVE-2018-0802, CVE-2017-8570, CVE-2017-0199, EPUB-SCRIPT-INJECTION)
- Quarantine: payload carving (`extractor.rs`), JSON + Markdown forensic
reports (`reporter.rs`), tarball packaging (`quarantine/mod.rs`)
- Document cleansing: Markdown sanitizer (`sanitizer.rs`),
format-preserving repackage for EPUB/DOCX/PDF (`repackage.rs`)
- Zip-bomb defense: `util::read_with_cap()` with actual-decompressed-byte
counting, configurable via `Config::epub_entry_scan_cap`
- CLI: `scan` (single file), `scan-dir` (directory, `--recursive`),
`--abort-on-threat`, `--quiet`, `--preserve-format`, `--workspace`,
env-var overrides
- iced 0.13 GUI dashboard with dark theme, collapsible panels, file
picker (`rfd::AsyncFileDialog`), console log, sidebar stats
- 22+ integration tests + zip-bomb defense tests + unit tests across all modules

BIN
corbel-purge-ss.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

138
scripts/gen_docx_fixtures.py Executable file
View File

@ -0,0 +1,138 @@
#!/usr/bin/env python3
"""Generate DOCX test fixtures for CorbelPurge.
Creates:
- benign.docx — clean DOCX with just text
- malicious_macro.docx — DOCX with a VBA macro stub
- malicious_ole.docx — DOCX with an embedded OLE object (PE)
- malicious_link.docx — DOCX with an external phishing hyperlink
"""
from pathlib import Path
import zipfile
from docx import Document
from docx.opc.constants import RELATIONSHIP_TYPE as RT
FIXTURES_DIR = Path(__file__).parent.parent / "tests" / "fixtures"
FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
def make_benign_docx():
"""A clean DOCX with just text — no macros, no embedded objects."""
path = FIXTURES_DIR / "benign.docx"
doc = Document()
doc.core_properties.title = "Benign Test DOCX"
doc.core_properties.author = "CorbelPurge Tests"
doc.core_properties.subject = "Test subject"
doc.add_paragraph("Hello, this is a benign DOCX.")
doc.add_paragraph("Second paragraph of benign content.")
doc.save(str(path))
return path
def make_malicious_macro_docx():
"""A DOCX with a VBA macro stub injected into the ZIP."""
# First create a normal DOCX.
base_path = FIXTURES_DIR / "_base_macro.docx"
doc = Document()
doc.core_properties.title = "Malicious Macro DOCX"
doc.add_paragraph("This DOCX contains a VBA macro.")
doc.save(str(base_path))
# Now copy the ZIP and inject a fake word/vbaProject.xml.
path = FIXTURES_DIR / "malicious_macro.docx"
with zipfile.ZipFile(base_path, "r") as src, zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as dst:
for item in src.infolist():
dst.writestr(item, src.read(item.filename))
# Inject the macro file.
dst.writestr(
"word/vbaProject.xml",
"<?xml version='1.0'?>"
"<vbaProject><module name='Module1'>"
"Sub AutoOpen()\n"
" MsgBox \"Hello from VBA\"\n"
"End Sub"
"</module></vbaProject>",
)
base_path.unlink()
return path
def make_malicious_ole_docx():
"""A DOCX with an embedded OLE object (fake PE)."""
base_path = FIXTURES_DIR / "_base_ole.docx"
doc = Document()
doc.core_properties.title = "Malicious OLE DOCX"
doc.add_paragraph("This DOCX contains an embedded OLE object.")
doc.save(str(base_path))
path = FIXTURES_DIR / "malicious_ole.docx"
with zipfile.ZipFile(base_path, "r") as src, zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as dst:
for item in src.infolist():
dst.writestr(item, src.read(item.filename))
# Inject an embedded OLE object with PE signature.
# MZ + dummy DOS header.
pe_bytes = b"MZ\x90\x00\x03\x00\x00\x00" + b"\x00" * 56 + b"PE\x00\x00"
dst.writestr("word/embeddings/oleObject1.bin", pe_bytes)
base_path.unlink()
return path
def make_malicious_link_docx():
"""A DOCX with an external phishing hyperlink (micros0ft homograph)."""
path = FIXTURES_DIR / "malicious_link.docx"
doc = Document()
doc.core_properties.title = "Malicious Link DOCX"
doc.add_paragraph("This DOCX contains a suspicious external hyperlink:")
# Add a paragraph with an external hyperlink.
para = doc.add_paragraph()
run = para.add_run("Click here to verify your account")
# python-docx doesn't directly expose external hyperlinks, so we
# post-process the XML.
doc.save(str(path))
# Post-process: add a hyperlink relationship and wrap the run.
import re
from docx.opc.packuri import PackURI
from docx.opc.part import Part
from docx.opc.constants import CONTENT_TYPE as CT
# Simpler approach: just unzip, edit document.xml.rels to add an
# external hyperlink relationship. The parser will detect it.
base_path2 = FIXTURES_DIR / "_base_link.docx"
Path(path).rename(base_path2)
with zipfile.ZipFile(base_path2, "r") as src, zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as dst:
for item in src.infolist():
data = src.read(item.filename)
if item.filename == "word/_rels/document.xml.rels":
# Inject a new external-link relationship.
text = data.decode("utf-8")
new_rel = (
'<Relationship Id="rIdEvil" '
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink" '
'Target="https://micros0ft.com/account/verify" '
'TargetMode="External"/>'
)
text = text.replace("</Relationships>", new_rel + "</Relationships>")
data = text.encode("utf-8")
dst.writestr(item, data)
base_path2.unlink()
return path
def main():
paths = [
make_benign_docx(),
make_malicious_macro_docx(),
make_malicious_ole_docx(),
make_malicious_link_docx(),
]
for p in paths:
print(f" wrote {p} ({p.stat().st_size} bytes)")
if __name__ == "__main__":
main()

147
scripts/gen_fixtures.py Executable file
View File

@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""Generate test fixture PDFs for CorbelPurge.
Creates four PDFs:
- benign.pdf — a clean PDF with just text
- malicious_js.pdf — a PDF with a /JavaScript action
- malicious_launch.pdf — a PDF with a /Launch action
- cve_writeup.pdf — an educational PDF that mentions JS / shellcode
Uses pypdf for xref-safe object injection.
"""
from pathlib import Path
from reportlab.pdfgen import canvas
from reportlab.lib.pagesizes import letter
import pypdf
from pypdf.generic import (
ArrayObject,
DictionaryObject,
NameObject,
NumberObject,
TextStringObject,
IndirectObject,
)
FIXTURES_DIR = Path(__file__).parent.parent / "tests" / "fixtures"
FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
def make_benign_pdf():
"""A clean PDF with just text — no JavaScript, no embedded files."""
path = FIXTURES_DIR / "benign.pdf"
c = canvas.Canvas(str(path), pagesize=letter)
c.drawString(100, 700, "Hello, this is a benign PDF.")
c.drawString(100, 680, "It contains only text and no active content.")
c.setTitle("Benign Test PDF")
c.setAuthor("CorbelPurge Tests")
c.save()
return path
def make_malicious_js_pdf():
"""A PDF with a /JavaScript action attached to the catalog's /OpenAction."""
# First, generate a base PDF with reportlab.
base_path = FIXTURES_DIR / "_base_js.pdf"
c = canvas.Canvas(str(base_path), pagesize=letter)
c.drawString(100, 700, "This PDF has a JavaScript action.")
c.setTitle("Malicious JS Test PDF")
c.save()
# Now use pypdf to inject a /JavaScript action safely.
path = FIXTURES_DIR / "malicious_js.pdf"
reader = pypdf.PdfReader(str(base_path))
writer = pypdf.PdfWriter()
# Copy all pages.
for page in reader.pages:
writer.add_page(page)
# Add a new /JavaScript action object.
js_action = DictionaryObject({
NameObject("/Type"): NameObject("/Action"),
NameObject("/S"): NameObject("/JavaScript"),
NameObject("/JS"): TextStringObject("app.alert('XSS from PDF');"),
})
js_action_ref = writer._add_object(js_action)
# Attach it to the catalog's /OpenAction.
writer._root_object[NameObject("/OpenAction")] = js_action_ref
with open(path, "wb") as f:
writer.write(f)
base_path.unlink()
return path
def make_malicious_launch_pdf():
"""A PDF with a /Launch action."""
base_path = FIXTURES_DIR / "_base_launch.pdf"
c = canvas.Canvas(str(base_path), pagesize=letter)
c.drawString(100, 700, "This PDF has a Launch action.")
c.setTitle("Malicious Launch Test PDF")
c.save()
path = FIXTURES_DIR / "malicious_launch.pdf"
reader = pypdf.PdfReader(str(base_path))
writer = pypdf.PdfWriter()
for page in reader.pages:
writer.add_page(page)
# Add a /Launch action.
launch_action = DictionaryObject({
NameObject("/Type"): NameObject("/Action"),
NameObject("/S"): NameObject("/Launch"),
NameObject("/F"): TextStringObject("/bin/sh"),
NameObject("/Win"): DictionaryObject({
NameObject("/F"): TextStringObject("cmd.exe"),
}),
})
launch_ref = writer._add_object(launch_action)
writer._root_object[NameObject("/OpenAction")] = launch_ref
with open(path, "wb") as f:
writer.write(f)
base_path.unlink()
return path
def make_cve_writeup_pdf():
"""A PDF that *mentions* JavaScript and shellcode in an educational context."""
path = FIXTURES_DIR / "cve_writeup.pdf"
c = canvas.Canvas(str(path), pagesize=letter)
c.setTitle("CVE-2024-1234 Writeup")
c.setAuthor("Security Researcher")
c.drawString(100, 750, "CVE-2024-1234: PDF JavaScript Injection Analysis")
c.drawString(100, 720, "Abstract")
text = c.beginText(100, 700)
text.setFont("Helvetica", 10)
text.textLines(
"In this paper we describe a vulnerability in which a malicious PDF\n"
"uses a /JavaScript action to execute arbitrary code. The eval()\n"
"function is called with attacker-controlled input. Remediation:\n"
"patch the reader to ignore /JavaScript actions in /OpenAction."
)
c.drawText(text)
c.save()
return path
def main():
paths = [
make_benign_pdf(),
make_malicious_js_pdf(),
make_malicious_launch_pdf(),
make_cve_writeup_pdf(),
]
for p in paths:
print(f" wrote {p} ({p.stat().st_size} bytes)")
if __name__ == "__main__":
main()

158
scripts/gen_md_epub_fixtures.py Executable file
View File

@ -0,0 +1,158 @@
#!/usr/bin/env python3
"""Generate Markdown and EPUB test fixtures for CorbelPurge.
Creates:
- benign.md — clean educational text
- cve_writeup.md — security literature with code samples (should whitelist)
- malicious.md — markdown with javascript: phishing links + shellcode
- benign.epub — clean EPUB
- malicious.epub — EPUB with <script> tag and external tracker
"""
from pathlib import Path
FIXTURES_DIR = Path(__file__).parent.parent / "tests" / "fixtures"
FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
def make_benign_md():
content = """# Benign Document
This is a paragraph of perfectly normal text. It discusses the weather,
the state of the economy, and other uncontroversial topics.
## Subsection
More text here. Nothing suspicious whatsoever.
"""
path = FIXTURES_DIR / "benign.md"
path.write_text(content)
return path
def make_cve_writeup_md():
"""Educational markdown that *mentions* exploits but in a literature context."""
content = """# CVE-2024-1234: PDF JavaScript Injection
## Abstract
In this paper we describe a vulnerability in which a malicious PDF
uses a /JavaScript action to execute arbitrary code. The eval() function
is called with attacker-controlled input.
## Proof of Concept
```python
# This is a PoC for the vulnerability described above.
import subprocess
# Note: this code is for educational purposes only.
payload = "eval('alert(1)')"
print(f"Payload: {payload}")
```
## Remediation
Patch the reader to ignore /JavaScript actions in /OpenAction.
"""
path = FIXTURES_DIR / "cve_writeup.md"
path.write_text(content)
return path
def make_malicious_md():
"""Markdown with phishing links and obfuscated shellcode in a paragraph."""
# 16+ hex-encoded bytes in a non-code context
shellcode = "\\x90" * 20
content = f"""# Click Here
Free money! [Click now](javascript:alert('xss'))
Run this: {shellcode}
"""
path = FIXTURES_DIR / "malicious.md"
path.write_text(content)
return path
def make_benign_epub():
"""Minimal clean EPUB with just text content."""
import zipfile
path = FIXTURES_DIR / "benign.epub"
with zipfile.ZipFile(path, "w", zipfile.ZIP_STORED) as z:
z.writestr("mimetype", "application/epub+zip")
z.writestr(
"OEBPS/content.opf",
"""<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Benign EPUB</dc:title>
<dc:author>CorbelPurge Tests</dc:author>
</metadata>
<manifest>
<item id="ch1" href="ch1.xhtml" media-type="application/xhtml+xml"/>
</manifest>
<spine>
<itemref idref="ch1"/>
</spine>
</package>""",
)
z.writestr(
"OEBPS/ch1.xhtml",
"""<?xml version="1.0"?>
<html><head><title>Ch1</title></head>
<body><p>Hello world.</p><p>Second paragraph.</p></body></html>""",
)
return path
def make_malicious_epub():
"""EPUB with a <script> tag and an external tracker image."""
import zipfile
path = FIXTURES_DIR / "malicious.epub"
with zipfile.ZipFile(path, "w", zipfile.ZIP_STORED) as z:
z.writestr("mimetype", "application/epub+zip")
z.writestr(
"OEBPS/content.opf",
"""<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Malicious EPUB</dc:title>
<dc:author>Attacker</dc:author>
</metadata>
<manifest>
<item id="ch1" href="ch1.xhtml" media-type="application/xhtml+xml"/>
</manifest>
<spine>
<itemref idref="ch1"/>
</spine>
</package>""",
)
z.writestr(
"OEBPS/ch1.xhtml",
"""<?xml version="1.0"?>
<html><head><title>Ch1</title></head>
<body>
<p>Hello.</p>
<script>alert('xss from epub');</script>
<img src="https://192.168.1.1/track.png" />
</body></html>""",
)
return path
def main():
paths = [
make_benign_md(),
make_cve_writeup_md(),
make_malicious_md(),
make_benign_epub(),
make_malicious_epub(),
]
for p in paths:
print(f" wrote {p} ({p.stat().st_size} bytes)")
if __name__ == "__main__":
main()

144
scripts/gen_zip_bomb_fixtures.py Executable file
View File

@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Generate a "lying" zip-bomb test fixture for CorbelPurge.
This crafts a ZIP archive where the central directory declares a small
uncompressed size (100 bytes) but the actual decompressed content is
much larger (1 MiB). This simulates a malicious archive that tries to
bypass size-header-based caps.
The ZIP format is hand-crafted (not via the `zip` library) so we can
lie about the size. The structure is:
[Local File Header][file data][Central Directory][End of Central Dir]
Each file header has both a "compressed size" and "uncompressed size"
field. We set the central directory's "uncompressed size" to 100, but
write 1 MiB of actual data. A naive reader that trusts the header
would only allocate 100 bytes; a streaming reader counts actual bytes
and detects the lie.
"""
import struct
import zlib
from pathlib import Path
FIXTURES_DIR = Path(__file__).parent.parent / "tests" / "fixtures"
FIXTURES_DIR.mkdir(parents=True, exist_ok=True)
def make_lying_zip_bomb():
"""Create a ZIP where the declared uncompressed size is 100 bytes
but the actual decompressed content is 1 MiB.
The ZIP is structurally valid (decompressors can read it) but the
central directory lies about the size. CorbelPurge's streaming
`read_with_cap` should detect this by counting actual bytes.
"""
# The actual content: 1 MiB of 'A' characters.
actual_content = b"A" * (1024 * 1024)
# Compress it with DEFLATE.
compressed = zlib.compress(actual_content, 9)
# The "lie": declare the uncompressed size as 100 bytes.
declared_uncompressed_size = 100
declared_compressed_size = len(compressed) # we don't lie about this
# CRC32 of the actual content (the decompressor will compute this
# and we need to match it for the CRC check to pass).
crc = zlib.crc32(actual_content) & 0xFFFFFFFF
# --- Local File Header ---
local_header = struct.pack(
"<IHHHHHIIIHH",
0x04034b50, # Local file header signature
20, # Version needed to extract (2.0)
0, # General purpose bit flag
8, # Compression method (DEFLATE)
0, # File last modification time
0, # File last modification date
crc, # CRC-32 of uncompressed data
declared_compressed_size, # Compressed size
declared_uncompressed_size, # Uncompressed size (THE LIE)
12, # File name length
0, # Extra field length
)
file_name = b"bomb.txt"
# --- Central Directory File Header ---
cd_header = struct.pack(
"<IHHHHHHIIIHHHHHII",
0x02014b50, # Central directory file header signature
20, # Version made by
20, # Version needed to extract
0, # General purpose bit flag
8, # Compression method (DEFLATE)
0, # File last modification time
0, # File last modification date
crc, # CRC-32
declared_compressed_size, # Compressed size
declared_uncompressed_size, # Uncompressed size (THE LIE)
12, # File name length
0, # Extra field length
0, # File comment length
0, # Disk number where file starts
0, # Internal file attributes
0, # External file attributes
0, # Relative offset of local file header
)
# --- End of Central Directory Record ---
local_header_size = len(local_header) + len(file_name) + len(compressed)
cd_size = len(cd_header) + len(file_name)
eocd = struct.pack(
"<IHHHHIIH",
0x06054b50, # End of central directory signature
0, # Number of this disk
0, # Disk where central directory starts
1, # Number of central directory records on this disk
1, # Total number of central directory records
cd_size, # Size of central directory (bytes)
local_header_size, # Offset of start of central directory
0, # Comment length
)
# Assemble the ZIP.
zip_bytes = (
local_header
+ file_name
+ compressed
+ cd_header
+ file_name
+ eocd
)
path = FIXTURES_DIR / "lying_zip_bomb.zip"
path.write_bytes(zip_bytes)
return path, len(actual_content), declared_uncompressed_size
def make_honest_zip_bomb():
"""Create a ZIP where the declared size is honest (1 MiB) but the
content is 1 MiB of 'B' characters. This tests the "honest but
oversized" case — the cap should still trigger based on the
declared size alone (the old behavior).
"""
import zipfile
path = FIXTURES_DIR / "honest_zip_bomb.zip"
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as z:
z.writestr("bomb.txt", b"B" * (1024 * 1024))
return path
def main():
lying_path, actual, declared = make_lying_zip_bomb()
print(f" wrote {lying_path} ({lying_path.stat().st_size} bytes)")
print(f" declared uncompressed size: {declared} bytes")
print(f" actual uncompressed size: {actual} bytes")
print(f" ratio: {actual / declared:.0f}x")
honest_path = make_honest_zip_bomb()
print(f" wrote {honest_path} ({honest_path.stat().st_size} bytes)")
if __name__ == "__main__":
main()

1053
src/bin/gui.rs Executable file
View File

File diff suppressed because it is too large Load Diff

37
src/cleanse/mod.rs Executable file
View File

@ -0,0 +1,37 @@
//! Cleansed-document builder: produces a sanitized derivative of the
//! original document containing only validated clean content.
//!
//! Two modes are supported (selected via [`crate::core::config::CleanseMode`]):
//!
//! - **Markdown** (default): produces a safe Markdown file. See [`sanitizer`].
//! - **PreserveFormat**: repackages the document in its original format
//! (PDF/EPUB/DOCX) with malicious entries stripped. See [`repackage`].
pub mod repackage;
pub mod sanitizer;
pub use repackage::repackage;
pub use sanitizer::sanitize;
use std::path::PathBuf;
use crate::core::config::{CleanseMode, Config};
use crate::core::types::{Document, ScanReport};
use crate::CorbelResult;
/// Top-level cleanse dispatcher. Called by the pipeline when
/// [`ScanReport::overall_recommendation`] returns
/// [`crate::core::types::Recommendation::QuarantineAndCleanse`].
///
/// Dispatches to [`sanitize`] (Markdown) or [`repackage`] (PreserveFormat)
/// based on `config.cleanse_mode`.
pub fn cleanse(
document: &Document,
scan_report: &ScanReport,
config: &Config,
) -> CorbelResult<PathBuf> {
match config.cleanse_mode {
CleanseMode::Markdown => sanitize(document, scan_report, config),
CleanseMode::PreserveFormat => repackage(document, scan_report, config),
}
}

1173
src/cleanse/repackage.rs Executable file
View File

File diff suppressed because it is too large Load Diff

286
src/cleanse/sanitizer.rs Executable file
View File

@ -0,0 +1,286 @@
//! Structural re-serializer: generates a brand-new, sanitized document
//! containing only the verified-clean text content from the source.
//!
//! The cleansed output is always Markdown, regardless of the source
//! format. This is intentional:
//!
//! 1. Markdown cannot carry executable content — there is no
//! `<script>`, no `/JavaScript`, no `/Launch` action. A cleansed
//! Markdown file is *definitionally* safe to render.
//! 2. Markdown preserves enough structure (headings, paragraphs, code
//! blocks) that the reader can still meaningfully display the
//! document's text content.
//! 3. Choosing a single output format keeps the sanitizer simple and
//! auditable. Re-serializing to PDF or EPUB would require pulling
//! in heavyweight writer crates (pdf-writer, epub-builder), each
//! of which is another attack surface.
use std::path::PathBuf;
use crate::core::config::Config;
use crate::core::types::{
Document, ScanReport, TextContext, TextNode,
ThreatClassification,
};
use crate::CorbelResult;
/// Top-level sanitize entrypoint. Called by the pipeline when
/// [`ScanReport::overall_recommendation`] returns
/// [`Recommendation::QuarantineAndCleanse`].
pub fn sanitize(
document: &Document,
scan_report: &ScanReport,
config: &Config,
) -> CorbelResult<PathBuf> {
let markdown = build_clean_markdown(document, scan_report);
let sha_prefix = &document.sha256[..8.min(document.sha256.len())];
let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S");
let filename = format!("cleansed_{timestamp}_{sha_prefix}.md");
let path = config.cleanse_dir.join(filename);
std::fs::write(&path, markdown)?;
Ok(path)
}
/// Build the cleansed Markdown string by:
///
/// 1. Emitting a header explaining what this file is.
/// 2. Emitting only text nodes whose corresponding finding (if any) is
/// not `Malicious`.
/// 3. Skipping any text node that contains a `Malicious` finding at
/// the same location.
pub fn build_clean_markdown(document: &Document, scan_report: &ScanReport) -> String {
let mut out = String::new();
// Header banner.
out.push_str("<!-- corbel: cleansed derivative — original was " );
out.push_str(&document.format.to_string());
out.push_str(", sha256=");
out.push_str(&document.sha256);
out.push_str(" -->\n\n");
out.push_str("> **CorbelPurge Notice**: This file was generated by stripping\n");
out.push_str("> all non-whitelisted objects (JavaScript streams, /Launch actions,\n");
out.push_str("> embedded files, `<script>` tags, etc.) from the source document.\n");
out.push_str("> ");
out.push_str(&format!(
"{} malicious finding(s) were quarantined. See the accompanying report.\n\n",
scan_report.malicious_count()
));
// Original metadata as a YAML-ish frontmatter block.
if let Some(title) = &document.metadata.title {
out.push_str(&format!("# {title}\n\n"));
}
// Build a set of (location → is_malicious) for quick filtering.
let malicious_locations: std::collections::HashSet<String> = scan_report
.findings
.iter()
.filter(|f| matches!(f.classification, ThreatClassification::Malicious(_)))
.map(|f| f.location.to_string())
.collect();
// Emit each text node in source order, skipping any whose location
// matches a malicious finding.
let mut emitted_any = false;
for node in &document.text_nodes {
if malicious_locations.contains(&node.location.to_string()) {
continue;
}
emit_node(&mut out, node);
emitted_any = true;
}
if !emitted_any {
out.push_str("_No clean content was recoverable from the source document._\n");
}
// Footer (without an artificial "end of report" marker).
out
}
/// Emit a single text node in the appropriate Markdown representation
/// for its context.
fn emit_node(out: &mut String, node: &TextNode) {
match node.context {
TextContext::Heading => {
out.push_str(&format!("## {}\n\n", node.content.trim()));
}
TextContext::CodeBlock => {
out.push_str("```\n");
out.push_str(&node.content);
if !node.content.ends_with('\n') {
out.push('\n');
}
out.push_str("```\n\n");
}
TextContext::CodeSpan => {
out.push_str(&format!("`{}`\n\n", node.content.trim()));
}
TextContext::BlockQuote => {
for line in node.content.lines() {
out.push_str(&format!("> {line}\n"));
}
out.push('\n');
}
TextContext::Hyperlink => {
// Drop the link destination; just emit the text.
// The scanner will have already flagged any malicious URIs.
out.push_str(&format!("{}\n\n", node.content.trim()));
}
TextContext::Paragraph | TextContext::Metadata | TextContext::ExecutableHook => {
out.push_str(&format!("{}\n\n", node.content.trim()));
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::*;
use tempfile::tempdir;
fn make_doc(nodes: Vec<TextNode>) -> Document {
Document {
format: DocumentFormat::Markdown,
source_path: None,
raw_bytes: Vec::new(),
sha256: "abcdef".to_string(),
size: 0,
metadata: DocumentMetadata::default(),
text_nodes: nodes,
executable_vectors: Vec::new(),
}
}
fn make_scan_with_malicious_at(loc: Location) -> ScanReport {
ScanReport {
source_sha256: "abcdef".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![Finding {
classification: ThreatClassification::Malicious(
MaliciousType::ActiveJavaScriptInjection,
),
location: loc,
vector_type: None,
payload_preview: "x".to_string(),
context_notes: "x".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}],
text_nodes_scanned: 1,
vectors_scanned: 0,
}
}
#[test]
fn clean_doc_passes_through() {
let doc = make_doc(vec![
TextNode {
location: Location::MarkdownLine { line: 1, col: 0 },
context: TextContext::Heading,
content: "Hello".to_string(),
},
TextNode {
location: Location::MarkdownLine { line: 3, col: 0 },
context: TextContext::Paragraph,
content: "World".to_string(),
},
]);
let scan = ScanReport {
source_sha256: "x".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![],
text_nodes_scanned: 2,
vectors_scanned: 0,
};
let tmp = tempdir().unwrap();
let mut config = Config::default();
config.cleanse_dir = tmp.path().to_path_buf();
let path = sanitize(&doc, &scan, &config).unwrap();
let content = std::fs::read_to_string(&path).unwrap();
assert!(content.contains("## Hello"));
assert!(content.contains("World"));
assert!(content.contains("CorbelPurge Notice"));
}
#[test]
fn malicious_node_is_stripped() {
let malicious_loc = Location::MarkdownLine { line: 3, col: 0 };
let doc = make_doc(vec![
TextNode {
location: Location::MarkdownLine { line: 1, col: 0 },
context: TextContext::Heading,
content: "Safe".to_string(),
},
TextNode {
location: malicious_loc.clone(),
context: TextContext::Paragraph,
content: "alert('xss')".to_string(),
},
TextNode {
location: Location::MarkdownLine { line: 5, col: 0 },
context: TextContext::Paragraph,
content: "Also safe".to_string(),
},
]);
let scan = make_scan_with_malicious_at(malicious_loc);
let tmp = tempdir().unwrap();
let mut config = Config::default();
config.cleanse_dir = tmp.path().to_path_buf();
let path = sanitize(&doc, &scan, &config).unwrap();
let content = std::fs::read_to_string(&path).unwrap();
assert!(content.contains("## Safe"));
assert!(!content.contains("alert"));
assert!(content.contains("Also safe"));
}
#[test]
fn empty_doc_emits_placeholder() {
let doc = make_doc(vec![]);
let scan = ScanReport {
source_sha256: "x".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![],
text_nodes_scanned: 0,
vectors_scanned: 0,
};
let tmp = tempdir().unwrap();
let mut config = Config::default();
config.cleanse_dir = tmp.path().to_path_buf();
let path = sanitize(&doc, &scan, &config).unwrap();
let content = std::fs::read_to_string(&path).unwrap();
assert!(content.contains("No clean content was recoverable"));
}
#[test]
fn code_block_is_emitted_as_fenced() {
let doc = make_doc(vec![TextNode {
location: Location::MarkdownLine { line: 1, col: 0 },
context: TextContext::CodeBlock,
content: "import os".to_string(),
}]);
let scan = ScanReport {
source_sha256: "x".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![],
text_nodes_scanned: 1,
vectors_scanned: 0,
};
let tmp = tempdir().unwrap();
let mut config = Config::default();
config.cleanse_dir = tmp.path().to_path_buf();
let path = sanitize(&doc, &scan, &config).unwrap();
let content = std::fs::read_to_string(&path).unwrap();
assert!(content.contains("```\nimport os\n```"));
}
}

237
src/core/config.rs Executable file
View File

@ -0,0 +1,237 @@
//! Configuration: security thresholds and workspace paths.
//!
//! All tunable parameters live here so that operators can override them
//! at runtime via [`Config::override_from_env`] or by constructing a
//! custom [`Config`] and passing it to [`crate::core::pipeline::Pipeline`].
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
/// Default maximum size for a payload preview string in the threat report.
const DEFAULT_MAX_PREVIEW_LEN: usize = 512;
/// Default maximum number of bytes of a ZIP-container entry (EPUB or
/// DOCX) to load into memory. Enforced via [`crate::util::read_with_cap`]
/// which counts **actual decompressed bytes** — not the size declared
/// in the ZIP central directory — so it defends against lying-size
/// zip bombs as well as honest-but-oversized entries.
const DEFAULT_EPUB_ENTRY_SCAN_CAP: usize = 8 * 1024 * 1024; // 8 MiB
/// Default total memory budget for all ZIP-container entries combined.
/// Prevents the "many small entries" attack where an archive with
/// thousands of entries each under the per-entry cap still exhausts memory.
const DEFAULT_TOTAL_ARCHIVE_SCAN_CAP: usize = 256 * 1024 * 1024; // 256 MiB
/// Default quarantine directory, relative to the current working dir.
const DEFAULT_QUARANTINE_DIR: &str = "corbel_quarantine";
/// Default cleansed-output directory, relative to the current working dir.
const DEFAULT_CLEANSE_DIR: &str = "corbel_clean";
/// How the cleansed document should be produced.
///
/// The default (`Markdown`) is the safest option — a Markdown file
/// cannot carry executable content by definition. `PreserveFormat`
/// repackages the document in its original format (PDF/EPUB/DOCX)
/// with the malicious entries stripped, which is more useful when
/// the document itself is the artifact of interest (e.g. a book
/// the researcher wants to actually read after cleaning).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Default)]
pub enum CleanseMode {
/// Produce a sanitized Markdown derivative (default, always safe).
#[default]
Markdown,
/// Repackage the document in its original format with malicious
/// entries stripped. The output is structurally similar to the
/// input but with all executable vectors removed.
PreserveFormat,
}
/// Top-level configuration for the CorbelPurge pipeline.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Config {
/// Where to write quarantine tarballs.
pub quarantine_dir: PathBuf,
/// Where to write cleansed document derivatives.
pub cleanse_dir: PathBuf,
/// Maximum length (in chars) of any payload preview in the report.
pub max_payload_preview_len: usize,
/// Maximum size (in bytes) of a single ZIP-container entry (EPUB
/// or DOCX) that will be loaded into memory. Entries that exceed
/// this cap are truncated mid-stream by [`crate::util::read_with_cap`]
/// and emitted as `UnknownPayload` vectors — the partial bytes are
/// still inspected for file signatures (PE, ELF, OLE2, etc.).
///
/// **This cap counts actual decompressed bytes, not the size
/// declared in the ZIP header.** A malicious archive that
/// declares `size = 100` but actually decompresses to gigabytes
/// is detected and truncated at this cap.
pub epub_entry_scan_cap: usize,
/// Total memory budget (in bytes) for all ZIP-container entries combined.
/// The parser tracks cumulative decompressed bytes across all entries
/// and stops processing entries once this budget is exhausted.
/// This prevents the "many small entries" attack where thousands of
/// entries each under `epub_entry_scan_cap` still exhaust memory.
pub total_archive_scan_cap: usize,
/// If `true`, the pipeline aborts with [`crate::CorbelError::ThreatDetected`]
/// the moment a malicious finding is produced, instead of proceeding
/// to quarantine + cleanse. Useful for automated CI gates.
pub abort_on_threat: bool,
/// If `true`, the scanner will emit `Suspicious` findings for any
/// executable vector it cannot confidently classify. If `false`,
/// only confidently-malicious vectors are reported.
pub emit_suspicious: bool,
/// List of URI schemes that are considered safe for hyperlink
/// navigation (e.g. `https`, `mailto`). Anything else is flagged.
pub allowed_uri_schemes: Vec<String>,
/// If `true`, generate a Markdown report alongside the JSON report.
pub emit_markdown_report: bool,
/// Path to an external YARA rules file. If set, the scanner loads
/// additional signature rules from this file at startup.
pub external_rules_path: Option<PathBuf>,
/// Path to an external CVE signature database file (JSON array).
pub external_cve_db_path: Option<PathBuf>,
/// How to produce the cleansed document derivative.
pub cleanse_mode: CleanseMode,
}
impl Default for Config {
fn default() -> Self {
Self {
quarantine_dir: PathBuf::from(DEFAULT_QUARANTINE_DIR),
cleanse_dir: PathBuf::from(DEFAULT_CLEANSE_DIR),
max_payload_preview_len: DEFAULT_MAX_PREVIEW_LEN,
epub_entry_scan_cap: DEFAULT_EPUB_ENTRY_SCAN_CAP,
total_archive_scan_cap: DEFAULT_TOTAL_ARCHIVE_SCAN_CAP,
abort_on_threat: false,
emit_suspicious: true,
allowed_uri_schemes: vec![
"https".to_string(),
"mailto".to_string(),
"ftp".to_string(),
],
emit_markdown_report: true,
external_rules_path: None,
external_cve_db_path: None,
cleanse_mode: CleanseMode::Markdown,
}
}
}
impl Config {
/// Construct a new config with default values, but with the
/// quarantine and cleanse directories rooted at `base`.
#[must_use]
pub fn with_workspace(base: impl AsRef<Path>) -> Self {
let base = base.as_ref();
Self {
quarantine_dir: base.join(DEFAULT_QUARANTINE_DIR),
cleanse_dir: base.join(DEFAULT_CLEANSE_DIR),
..Self::default()
}
}
/// Override select fields from environment variables.
///
/// Recognized variables:
/// - `CORBEL_QUARANTINE_DIR`
/// - `CORBEL_CLEANSE_DIR`
/// - `CORBEL_ABORT_ON_THREAT` (`1`/`true`/`yes` → true)
/// - `CORBEL_EMIT_SUSPICIOUS`
/// - `CORBEL_EMIT_MARKDOWN_REPORT`
/// - `CORBEL_EXTERNAL_RULES` (path to external YARA rules JSON)
/// - `CORBEL_EXTERNAL_CVE_DB` (path to external CVE DB JSON)
pub fn override_from_env(mut self) -> Self {
if let Ok(v) = std::env::var("CORBEL_QUARANTINE_DIR") {
self.quarantine_dir = PathBuf::from(v);
}
if let Ok(v) = std::env::var("CORBEL_CLEANSE_DIR") {
self.cleanse_dir = PathBuf::from(v);
}
if let Ok(v) = std::env::var("CORBEL_ABORT_ON_THREAT") {
self.abort_on_threat = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_EMIT_SUSPICIOUS") {
self.emit_suspicious = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_EMIT_MARKDOWN_REPORT") {
self.emit_markdown_report = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_TOTAL_ARCHIVE_SCAN_CAP") {
if let Ok(cap) = v.parse::<usize>() {
self.total_archive_scan_cap = cap;
}
}
if let Ok(v) = std::env::var("CORBEL_EXTERNAL_RULES") {
if !v.is_empty() {
self.external_rules_path = Some(PathBuf::from(v));
}
}
if let Ok(v) = std::env::var("CORBEL_EXTERNAL_CVE_DB") {
if !v.is_empty() {
self.external_cve_db_path = Some(PathBuf::from(v));
}
}
self
}
/// Ensure the workspace directories exist.
///
/// Called automatically by [`crate::core::pipeline::Pipeline::run`],
/// but exposed publicly for callers that want to pre-create them.
pub fn ensure_workspace(&self) -> std::io::Result<()> {
std::fs::create_dir_all(&self.quarantine_dir)?;
std::fs::create_dir_all(&self.cleanse_dir)?;
Ok(())
}
}
fn truthy(v: &str) -> bool {
matches!(v.to_ascii_lowercase().as_str(), "1" | "true" | "yes" | "on")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn defaults_are_sane() {
let c = Config::default();
assert!(c.quarantine_dir.ends_with(DEFAULT_QUARANTINE_DIR));
assert!(c.cleanse_dir.ends_with(DEFAULT_CLEANSE_DIR));
assert!(!c.abort_on_threat);
assert!(c.emit_suspicious);
assert!(c.allowed_uri_schemes.contains(&"https".to_string()));
}
#[test]
fn env_override_works() {
// Temporarily set env vars, build config, restore.
let old_q = std::env::var("CORBEL_QUARANTINE_DIR").ok();
let old_a = std::env::var("CORBEL_ABORT_ON_THREAT").ok();
std::env::set_var("CORBEL_QUARANTINE_DIR", "/tmp/corbel_q_test");
std::env::set_var("CORBEL_ABORT_ON_THREAT", "yes");
let c = Config::default().override_from_env();
assert_eq!(c.quarantine_dir, PathBuf::from("/tmp/corbel_q_test"));
assert!(c.abort_on_threat);
match old_q {
Some(v) => std::env::set_var("CORBEL_QUARANTINE_DIR", v),
None => std::env::remove_var("CORBEL_QUARANTINE_DIR"),
}
match old_a {
Some(v) => std::env::set_var("CORBEL_ABORT_ON_THREAT", v),
None => std::env::remove_var("CORBEL_ABORT_ON_THREAT"),
}
}
#[test]
fn workspace_with_base() {
let c = Config::with_workspace("/tmp/corbel_ws_test");
assert_eq!(c.quarantine_dir, PathBuf::from("/tmp/corbel_ws_test/corbel_quarantine"));
assert_eq!(c.cleanse_dir, PathBuf::from("/tmp/corbel_ws_test/corbel_clean"));
}
}

13
src/core/mod.rs Executable file
View File

@ -0,0 +1,13 @@
//! Core types, configuration, and the pipeline coordinator.
//!
//! This module owns the unified intermediate representation that all three
//! parsers (PDF, EPUB, Markdown) produce, and that the scanner, quarantine,
//! and cleanse modules consume.
pub mod config;
pub mod pipeline;
pub mod types;
pub use config::{CleanseMode, Config};
pub use pipeline::{Pipeline, PipelineResult};
pub use types::*;

220
src/core/pipeline.rs Executable file
View File

@ -0,0 +1,220 @@
//! The pipeline coordinator.
//!
//! [`Pipeline`] is the single entrypoint that ties together parsing,
//! scanning, quarantine, and cleanse. It is the only struct most
//! callers need to touch.
//!
//! ## Pipeline flow
//!
//! ```text
//! input path ──► parser ──► Document (UIR)
//! │
//! ▼
//! scanner ──► ScanReport
//! │
//! ┌────────────────────┼────────────────────┐
//! │ │ │
//! ▼ ▼ ▼
//! (no threats) (quarantine) (cleanse)
//! │ │ │
//! ▼ ▼ ▼
//! return tarball + report cleansed file
//! ```
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use crate::CorbelError::{self, ThreatDetected};
use crate::{quarantine, scanner, cleanse, CorbelResult};
use super::config::Config;
use super::types::{DocumentFormat, ScanReport};
/// The result of running the pipeline on a single document.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PipelineResult {
/// Source file path (if read from disk).
pub source_path: Option<PathBuf>,
/// SHA-256 of the source bytes.
pub source_sha256: String,
/// Detected format.
pub format: DocumentFormat,
/// Aggregate scan report.
pub scan_report: ScanReport,
/// Path to the generated quarantine tarball, if any.
pub quarantine_path: Option<PathBuf>,
/// Path to the generated cleansed document, if any.
pub cleansed_path: Option<PathBuf>,
/// Path to the JSON forensic report, if any.
pub json_report_path: Option<PathBuf>,
/// Path to the Markdown forensic report, if any.
pub markdown_report_path: Option<PathBuf>,
}
/// The pipeline coordinator. Owns the [`Config`] and dispatches to the
/// appropriate parser / scanner / quarantine / cleanse modules.
#[derive(Debug, Clone)]
pub struct Pipeline {
/// Pipeline configuration.
pub config: Config,
}
impl Default for Pipeline {
fn default() -> Self {
Self {
config: Config::default().override_from_env(),
}
}
}
impl Pipeline {
/// Construct a new pipeline with the given config.
#[must_use]
pub fn with_config(config: Config) -> Self {
Self { config }
}
/// Run the full pipeline against a file on disk.
///
/// This reads the file, parses it, scans it, and (depending on
/// findings) produces quarantine and cleansed outputs.
pub fn run(&self, path: impl AsRef<Path>) -> CorbelResult<PipelineResult> {
let path = path.as_ref();
let bytes = std::fs::read(path)?;
let format = DocumentFormat::from_path(path)?;
self.run_on_bytes(bytes, format, Some(path.to_path_buf()))
}
/// Run the full pipeline against an in-memory byte buffer.
///
/// Useful for tests and for embedding CorbelPurge as a library
/// inside a larger system (e.g. an email gateway).
pub fn run_on_bytes(
&self,
bytes: Vec<u8>,
format: DocumentFormat,
source_path: Option<PathBuf>,
) -> CorbelResult<PipelineResult> {
self.config.ensure_workspace()?;
// 1. Parse into the UIR.
let document = format.parse(&bytes, source_path.clone(), &self.config)?;
// 2. Scan.
let scan_report = scanner::scan(&document, &self.config);
// 3. Abort-on-threat short-circuit.
if self.config.abort_on_threat && scan_report.malicious_count() > 0 {
return Err(ThreatDetected(format!(
"found {} malicious finding(s) in {}",
scan_report.malicious_count(),
document
.source_path
.as_ref()
.map(|p| p.display().to_string())
.unwrap_or_else(|| format!("<{} buffer>", format)),
)));
}
// 4. Quarantine + report (if anything malicious was found).
let quarantine_outcome = if scan_report.malicious_count() > 0 {
Some(quarantine::handle(&document, &scan_report, &self.config)?)
} else {
None
};
// 5. Cleanse (if recommended).
let cleansed_path = if scan_report.overall_recommendation()
== super::types::Recommendation::QuarantineAndCleanse
{
Some(cleanse::cleanse(&document, &scan_report, &self.config)?)
} else {
None
};
Ok(PipelineResult {
source_path,
source_sha256: document.sha256.clone(),
format,
scan_report,
quarantine_path: quarantine_outcome.as_ref().map(|q| q.tarball_path.clone()),
cleansed_path,
json_report_path: quarantine_outcome.as_ref().map(|q| q.json_report_path.clone()),
markdown_report_path: quarantine_outcome
.as_ref()
.and_then(|q| q.markdown_report_path.clone()),
})
}
}
impl DocumentFormat {
/// Detect a document's format from its file extension.
///
/// We do not sniff magic bytes in this MVP — extension is sufficient
/// for the threat model. (A malicious file with a `.md` extension
/// but PDF bytes will simply fail to parse, which is the safe
/// failure mode.)
pub fn from_path(path: &Path) -> CorbelResult<Self> {
let ext = path
.extension()
.and_then(|e| e.to_str())
.map(|s| s.to_ascii_lowercase())
.ok_or_else(|| CorbelError::UnknownFormat(path.to_path_buf()))?;
match ext.as_str() {
"pdf" => Ok(Self::Pdf),
"epub" => Ok(Self::Epub),
"md" | "markdown" => Ok(Self::Markdown),
"docx" => Ok(Self::Docx),
_ => Err(CorbelError::UnknownFormat(path.to_path_buf())),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn format_detection_pdf() {
assert!(matches!(
DocumentFormat::from_path(Path::new("/tmp/foo.pdf")),
Ok(DocumentFormat::Pdf)
));
}
#[test]
fn format_detection_epub() {
assert!(matches!(
DocumentFormat::from_path(Path::new("/tmp/foo.epub")),
Ok(DocumentFormat::Epub)
));
}
#[test]
fn format_detection_markdown_variants() {
assert!(matches!(
DocumentFormat::from_path(Path::new("/tmp/foo.md")),
Ok(DocumentFormat::Markdown)
));
assert!(matches!(
DocumentFormat::from_path(Path::new("/tmp/foo.markdown")),
Ok(DocumentFormat::Markdown)
));
}
#[test]
fn format_detection_unknown() {
assert!(DocumentFormat::from_path(Path::new("/tmp/foo.exe")).is_err());
assert!(DocumentFormat::from_path(Path::new("/tmp/noext")).is_err());
}
#[test]
fn format_detection_case_insensitive() {
assert!(matches!(
DocumentFormat::from_path(Path::new("/tmp/FOO.PDF")),
Ok(DocumentFormat::Pdf)
));
}
}

426
src/core/types.rs Executable file
View File

@ -0,0 +1,426 @@
//! Core domain types shared across the pipeline.
//!
//! These types form the *unified intermediate representation* (UIR) that
//! every parser emits. The scanner, quarantine, and cleanse modules all
//! consume the UIR — they never touch format-specific structures directly.
//! This separation is what lets us add a fourth format later (e.g. DOCX)
//! without touching the scanner.
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
/// The on-disk format of a document, inferred from its extension and/or
/// magic bytes.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum DocumentFormat {
/// PDF — the highest threat surface due to JavaScript streams,
/// `/Launch` actions, `/EmbeddedFiles`, and action dictionaries.
Pdf,
/// EPUB — a ZIP container of XHTML files; can carry scripts and
/// external resources.
Epub,
/// Markdown — pure text, but can still embed malicious hyperlinks
/// or obfuscated payloads inside code blocks.
Markdown,
/// DOCX — Office Open XML; can carry VBA macros, embedded OLE
/// objects, ActiveX controls, and external hyperlinks.
Docx,
}
impl std::fmt::Display for DocumentFormat {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Pdf => write!(f, "pdf"),
Self::Epub => write!(f, "epub"),
Self::Markdown => write!(f, "markdown"),
Self::Docx => write!(f, "docx"),
}
}
}
/// A location inside a document, used by the scanner to report exactly
/// where a threat was found.
///
/// We deliberately keep this a string-based enum rather than a typed
/// (page/object/offset) tuple because each format has a different notion
/// of "location" (PDF uses object IDs, EPUB uses XHTML filenames,
/// Markdown uses line numbers).
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum Location {
/// A PDF object reference, e.g. `42 0` (object 42, generation 0).
PdfObject {
/// Object number.
id: u32,
/// Generation number.
gen: u32,
},
/// A PDF stream inside a specific object.
PdfStream {
/// Owning object id.
id: u32,
/// Filter chain (e.g. `["FlateDecode", "ASCIIHexDecode"]`).
filter: Vec<String>,
},
/// An EPUB entry inside the ZIP container, with optional anchor.
EpubEntry {
/// Path inside the ZIP, e.g. `OEBPS/chapter3.xhtml`.
path: String,
/// Optional fragment / XPath / line number.
anchor: Option<String>,
},
/// A Markdown source location.
MarkdownLine {
/// 1-indexed line number.
line: u32,
/// Column offset within the line (0-indexed).
col: u32,
},
}
impl std::fmt::Display for Location {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::PdfObject { id, gen } => write!(f, "pdf:{id} {gen} R"),
Self::PdfStream { id, filter } => {
write!(f, "pdf:stream(obj={id}, filter=[{}])", filter.join(","))
}
Self::EpubEntry { path, anchor } => match anchor {
Some(a) => write!(f, "epub:{path}#{a}"),
None => write!(f, "epub:{path}"),
},
Self::MarkdownLine { line, col } => write!(f, "md:{line}:{col}"),
}
}
}
/// The semantic context in which a piece of content was found.
///
/// This is the heart of the context-aware scanner: the same string
/// (`eval(...)`, `/JavaScript`, a hex blob) is benign inside an
/// educational code block and malicious inside a PDF action dictionary.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum TextContext {
/// A paragraph or body sentence.
Paragraph,
/// A heading or title.
Heading,
/// A fenced or indented code block (Markdown, EPUB `<pre>`, etc.).
CodeBlock,
/// An inline code span.
CodeSpan,
/// A hyperlink, anchor, or `/URI` action.
Hyperlink,
/// A quoted block.
BlockQuote,
/// Document metadata (title, author, subject).
Metadata,
/// An executable / active vector — only set when the content lives
/// inside a structural hook (PDF action dictionary, EPUB `<script>`,
/// etc.). This is the default-malicious bucket.
ExecutableHook,
}
/// A piece of static text extracted from the document.
///
/// "Static" means the text was found in a rendered content stream —
/// a paragraph, heading, code block, etc. — *not* inside an active
/// executable vector. The scanner's [`context_filter`] inspects these.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TextNode {
/// Where in the document this text lives.
pub location: Location,
/// The semantic context surrounding the text.
pub context: TextContext,
/// The text content itself (already decoded from any encoding).
pub content: String,
}
/// An executable or active vector embedded in the document.
///
/// These are treated as *untrusted by default* by the scanner. The most
/// common examples are PDF `/JavaScript` streams, `/Launch` actions,
/// `/EmbeddedFiles`, EPUB `<script>` tags, and external resource
/// references.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExecutableVector {
/// Where in the document this vector lives.
pub location: Location,
/// What kind of vector it is.
pub vector_type: VectorType,
/// The raw payload bytes (may be encoded/obfuscated; the scanner
/// inspects these for shellcode patterns, hex blobs, etc.).
pub raw_payload: Vec<u8>,
/// Optional decoded/preview form of the payload (UTF-8 lossy).
pub decoded_preview: Option<String>,
}
/// Classification of executable vector types.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum VectorType {
/// PDF `/JavaScript` or `/JS` action stream.
PdfJavaScript,
/// PDF `/Launch` action (execute an external program).
PdfLaunch,
/// PDF `/URI` action (open a URL — used for phishing).
PdfUri,
/// PDF `/GoToR` / `/GoTo` remote navigation.
PdfGoToR,
/// PDF `/EmbeddedFiles` attachment.
PdfEmbeddedFile,
/// PDF `/Annot` of subtype `/Widget` with an action.
PdfWidgetAction,
/// PDF form (`/AcroForm`) with JavaScript hooks.
PdfAcroForm,
/// EPUB `<script>` tag.
EpubScript,
/// EPUB external resource (`<link>`, `<img src=...>`, etc.).
EpubExternalResource,
/// EPUB `<object>` / `<embed>` tag.
EpubObject,
/// Markdown hyperlink (potential phishing / drive-by URL).
MarkdownHyperlink,
/// DOCX VBA macro (`word/vbaProject.xml`).
DocxMacro,
/// DOCX external hyperlink (from `word/_rels/document.xml.rels`
/// with `TargetMode="External"`).
DocxExternalLink,
/// DOCX embedded OLE object (`word/embeddings/*.bin`).
DocxEmbeddedObject,
/// DOCX ActiveX control (`word/activeX/*.xml`).
DocxActiveX,
/// Anything that looks like an obfuscated payload but doesn't fit
/// a known vector type (e.g. a hex blob in an unexpected stream).
UnknownPayload,
}
impl std::fmt::Display for VectorType {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let s = match self {
Self::PdfJavaScript => "pdf-javascript",
Self::PdfLaunch => "pdf-launch",
Self::PdfUri => "pdf-uri",
Self::PdfGoToR => "pdf-gotor",
Self::PdfEmbeddedFile => "pdf-embedded-file",
Self::PdfWidgetAction => "pdf-widget-action",
Self::PdfAcroForm => "pdf-acroform",
Self::EpubScript => "epub-script",
Self::EpubExternalResource => "epub-external-resource",
Self::EpubObject => "epub-object",
Self::MarkdownHyperlink => "md-hyperlink",
Self::DocxMacro => "docx-macro",
Self::DocxExternalLink => "docx-external-link",
Self::DocxEmbeddedObject => "docx-embedded-object",
Self::DocxActiveX => "docx-activex",
Self::UnknownPayload => "unknown-payload",
};
f.write_str(s)
}
}
/// Document-level metadata captured during parsing.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct DocumentMetadata {
/// Document title, if declared.
pub title: Option<String>,
/// Document author, if declared.
pub author: Option<String>,
/// Subject / description, if declared.
pub subject: Option<String>,
/// Declared producer tool (e.g. the PDF writer).
pub producer: Option<String>,
/// Declared authoring tool.
pub creator: Option<String>,
/// Creation timestamp (ISO 8601 string if present).
pub created: Option<String>,
/// Modification timestamp (ISO 8601 string if present).
pub modified: Option<String>,
}
/// The unified intermediate representation of a parsed document.
///
/// All three parsers (PDF, EPUB, Markdown) emit one of these. Downstream
/// modules (scanner, quarantine, cleanse) only ever consume this struct —
/// they do not know which parser produced it.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Document {
/// Source format.
pub format: DocumentFormat,
/// Original file path (may be `None` if parsed from an in-memory buffer).
pub source_path: Option<PathBuf>,
/// Raw file bytes (kept for forensic hashing and re-serialization).
pub raw_bytes: Vec<u8>,
/// SHA-256 hex digest of `raw_bytes`.
pub sha256: String,
/// File size in bytes.
pub size: u64,
/// Declared document metadata.
pub metadata: DocumentMetadata,
/// Static text nodes extracted from the document.
pub text_nodes: Vec<TextNode>,
/// Executable / active vectors found during structural inspection.
pub executable_vectors: Vec<ExecutableVector>,
}
impl Document {
/// Helper: count text nodes by context.
#[must_use]
pub fn text_node_counts(&self) -> std::collections::HashMap<TextContext, usize> {
let mut counts = std::collections::HashMap::new();
for node in &self.text_nodes {
*counts.entry(node.context).or_insert(0) += 1;
}
counts
}
/// Helper: total number of executable vectors of a given type.
#[must_use]
pub fn vector_count_of(&self, vt: VectorType) -> usize {
self.executable_vectors
.iter()
.filter(|v| v.vector_type == vt)
.count()
}
}
/// Final classification of a single finding produced by the scanner.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum ThreatClassification {
/// Content was inspected and found to be safe.
Benign,
/// Content is suspicious-looking but appears in an educational /
/// literature context (CVE writeup, textbook, defensive blog).
EducationalContent,
/// Content is suspicious but lacks enough signal to be flagged
/// as definitively malicious. The pipeline will still allow it
/// but log a warning.
Suspicious,
/// Content is malicious. The inner type narrows down the threat
/// category so the quarantine report can group findings.
Malicious(MaliciousType),
}
/// Specific categories of malicious content.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum MaliciousType {
/// An active PDF JavaScript injection inside an executable hook.
ActiveJavaScriptInjection,
/// A PDF `/Launch` action that would execute an external program.
LaunchAction,
/// A malicious embedded file (e.g. an EXE or macro-laden OLE object).
MaliciousEmbeddedFile,
/// An obfuscated payload (packed shellcode, excessive hex encoding)
/// hidden in an unexpected stream.
ObfuscatedShellcode,
/// A suspicious URI (e.g. known-bad TLD, credential harvesting pattern).
SuspiciousUri,
/// An EPUB `<script>` carrying active content.
EpubActiveScript,
/// A DOCX VBA macro or other active content.
DocxActiveContent,
/// Anything else that doesn't fit the above buckets.
Other,
}
impl std::fmt::Display for MaliciousType {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let s = match self {
Self::ActiveJavaScriptInjection => "active-javascript-injection",
Self::LaunchAction => "launch-action",
Self::MaliciousEmbeddedFile => "malicious-embedded-file",
Self::ObfuscatedShellcode => "obfuscated-shellcode",
Self::SuspiciousUri => "suspicious-uri",
Self::EpubActiveScript => "epub-active-script",
Self::DocxActiveContent => "docx-active-content",
Self::Other => "other",
};
f.write_str(s)
}
}
/// A single finding emitted by the scanner.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Finding {
/// Final classification.
pub classification: ThreatClassification,
/// Where in the document the finding was made.
pub location: Location,
/// What kind of vector produced the finding (if any).
pub vector_type: Option<VectorType>,
/// A short (truncated, UTF-8 lossy) preview of the offending bytes.
pub payload_preview: String,
/// Free-form notes from the context filter explaining the decision.
pub context_notes: String,
/// Recommended action for the pipeline coordinator.
pub recommendation: Recommendation,
}
/// Recommended action for a given finding.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum Recommendation {
/// Allow the content to render as-is.
Allow,
/// Whitelist as educational content (allow, but log).
WhitelistAsEducational,
/// Quarantine the payload but allow the cleansed document to render.
Quarantine,
/// Quarantine and rebuild the document without the offending content.
QuarantineAndCleanse,
}
/// Aggregate scan report for the whole document.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ScanReport {
/// SHA-256 of the source file.
pub source_sha256: String,
/// Source format.
pub format: DocumentFormat,
/// When the scan ran (ISO 8601).
pub scanned_at: String,
/// All findings, in document order.
pub findings: Vec<Finding>,
/// Total number of text nodes scanned.
pub text_nodes_scanned: usize,
/// Total number of executable vectors scanned.
pub vectors_scanned: usize,
}
impl ScanReport {
/// Number of findings classified as `Malicious`.
#[must_use]
pub fn malicious_count(&self) -> usize {
self.findings
.iter()
.filter(|f| matches!(f.classification, ThreatClassification::Malicious(_)))
.count()
}
/// Number of findings classified as `EducationalContent`.
#[must_use]
pub fn educational_count(&self) -> usize {
self.findings
.iter()
.filter(|f| f.classification == ThreatClassification::EducationalContent)
.count()
}
/// Highest-severity recommendation across all findings.
#[must_use]
pub fn overall_recommendation(&self) -> Recommendation {
if self
.findings
.iter()
.any(|f| f.recommendation == Recommendation::QuarantineAndCleanse)
{
return Recommendation::QuarantineAndCleanse;
}
if self
.findings
.iter()
.any(|f| f.recommendation == Recommendation::Quarantine)
{
return Recommendation::Quarantine;
}
Recommendation::Allow
}
}

101
src/lib.rs Executable file
View File

@ -0,0 +1,101 @@
//! # CorbelPurge
//!
//! Strict Rust secure document viewer & threat neutralizer for PDF, EPUB, and
//! Markdown formats.
//!
//! This crate exposes a headless security pipeline that:
//!
//! 1. Parses a document into a unified intermediate representation
//! ([`crate::core::Document`]) consisting of text nodes and executable
//! vectors.
//! 2. Runs a layered contextual scanner
//! ([`crate::scanner`]) that distinguishes between legitimate security
//! literature (CVE writeups, exploit code samples) and active malicious
//! injection vectors (PDF JavaScript streams, /Launch actions, embedded
//! executables).
//! 3. Optionally extracts, reports, and quarantines any identified threats
//! ([`crate::quarantine`]).
//! 4. Optionally produces a sanitized, cleansed derivative of the original
//! document containing only validated clean content
//! ([`crate::cleanse`]).
//!
//! The `gui` feature flag additionally enables an [`iced`]-based viewer,
//! which is intentionally stubbed in this MVP.
//!
//! See `MANIFEST.md` in the project root for the full design manifest.
#![forbid(unsafe_code)]
#![deny(missing_docs)]
#![warn(clippy::all, clippy::pedantic)]
#![allow(clippy::module_name_repetitions, clippy::missing_errors_doc)]
pub mod core;
pub mod parsers;
pub mod scanner;
pub mod quarantine;
pub mod cleanse;
pub mod util;
pub mod study;
#[cfg(feature = "gui")]
pub mod ui;
pub use core::{
config::{CleanseMode, Config},
pipeline::{Pipeline, PipelineResult},
types::*,
};
pub use util::sha256_hex;
/// Crate-level error type.
///
/// Every fallible public API in CorbelPurge returns [`Result<T, CorbelError>`]
/// so callers can pattern-match on a single, exhaustive error enum rather than
/// juggling per-module error types.
#[derive(Debug, thiserror::Error)]
pub enum CorbelError {
/// I/O failure (file not found, permission denied, disk full, etc.).
#[error("io error: {0}")]
Io(#[from] std::io::Error),
/// Serialization/deserialization failure.
#[error("serde error: {0}")]
Serde(#[from] serde_json::Error),
/// PDF structure could not be parsed by `lopdf`.
#[error("pdf parse error: {0}")]
PdfParse(String),
/// EPUB container (ZIP) was malformed or missing required entries.
#[error("epub parse error: {0}")]
EpubParse(String),
/// Markdown could not be tokenized.
#[error("markdown parse error: {0}")]
MarkdownParse(String),
/// The document format was not recognized from its extension/magic bytes.
#[error("unknown document format for path: {0}")]
UnknownFormat(std::path::PathBuf),
/// A threat was found and the caller configured the pipeline to abort
/// rather than quarantine.
#[error("threat detected and abort-on-threat is set: {0}")]
ThreatDetected(String),
/// Quarantine packaging failed (tarball write, compression, etc.).
#[error("quarantine error: {0}")]
Quarantine(String),
/// Sanitization / document reconstruction failed.
#[error("cleanse error: {0}")]
Cleanse(String),
/// Catch-all for errors that don't fit a more specific variant.
#[error("internal error: {0}")]
Internal(String),
}
/// Convenience alias used throughout the crate.
pub type CorbelResult<T> = Result<T, CorbelError>;

445
src/main.rs Executable file
View File

@ -0,0 +1,445 @@
//! corbel-purge CLI
//!
//! Headless command-line entrypoint for the CorbelPurge pipeline.
//! Used for scanning and studying documents without launching the GUI.
//!
//! Run with:
//! ```text
//! corbel-purge scan <path> [--workspace <dir>] [--abort-on-threat]
//! corbel-purge scan-dir <dir> [--recursive]
//! corbel-purge study <path> [--workspace <dir>]
//! corbel-purge --version
//! ```
//!
//! The CLI deliberately avoids `clap` and other heavy arg-parsing crates:
//! CorbelPurge is security-critical software, and keeping the dependency
//! tree small is itself a defensive measure.
use std::path::{Path, PathBuf};
use std::process::ExitCode;
use corbel_purge::{Config, Pipeline, PipelineResult};
fn main() -> ExitCode {
let args: Vec<String> = std::env::args().collect();
if args.len() < 2 {
print_usage();
return ExitCode::FAILURE;
}
match args[1].as_str() {
"study" => run_study(&args[2..]),
"scan" => run_scan(&args[2..]),
"scan-dir" => run_scan_dir(&args[2..]),
"--version" | "-V" | "version" => {
println!("corbel-purge {}", env!("CARGO_PKG_VERSION"));
ExitCode::SUCCESS
}
"--help" | "-h" | "help" => {
print_usage();
ExitCode::SUCCESS
}
other => {
eprintln!("unknown command: {other}");
print_usage();
ExitCode::FAILURE
}
}
}
fn print_usage() {
eprintln!(
"corbel-purge {version}
USAGE:
corbel-purge scan <path> [--workspace <dir>] [--abort-on-threat] [--quiet] [--preserve-format]
[--rules <path>] [--cve-db <path>]
corbel-purge scan-dir <dir> [--recursive] [--workspace <dir>] [--abort-on-threat] [--preserve-format]
[--rules <path>] [--cve-db <path>]
corbel-purge study <path> [--workspace <dir>]
corbel-purge --version
OPTIONS:
--workspace <dir> Root for quarantine/ and clean/ output dirs.
Defaults to the current working directory.
--abort-on-threat Exit immediately with code 2 if any malicious
finding is produced. No quarantine tarball written.
--quiet, -q Only print the JSON report path on success.
--recursive, -r Recurse into subdirectories (scan-dir).
--preserve-format Repackage the cleansed document in its original
format (PDF/EPUB/DOCX) instead of converting to
Markdown. Useful when the document itself is the
artifact of interest.
--rules <path> Path to an external YARA rules JSON file.
Additional signature rules are loaded from this
file and checked alongside the built-in tables.
--cve-db <path> Path to an external CVE signature database
(JSON array). Additional CVE entries are loaded
and matched alongside the built-in CVE table.
EXIT CODES:
0 No threats found.
2 One or more malicious findings (file was processed).
1 Hard error (could not parse, IO failure, etc.).",
version = env!("CARGO_PKG_VERSION")
);
}
/// Parse common scan args out of `args` (the slice after the subcommand).
struct ScanArgs {
path: PathBuf,
workspace: Option<PathBuf>,
abort_on_threat: bool,
quiet: bool,
recursive: bool,
preserve_format: bool,
external_rules_path: Option<PathBuf>,
external_cve_db_path: Option<PathBuf>,
}
fn parse_scan_args(args: &[String]) -> Result<ScanArgs, String> {
let mut path: Option<PathBuf> = None;
let mut workspace: Option<PathBuf> = None;
let mut abort_on_threat = false;
let mut quiet = false;
let mut recursive = false;
let mut preserve_format = false;
let mut external_rules_path: Option<PathBuf> = None;
let mut external_cve_db_path: Option<PathBuf> = None;
let mut i = 0;
while i < args.len() {
match args[i].as_str() {
"--workspace" => {
i += 1;
workspace = Some(PathBuf::from(
args.get(i).ok_or("--workspace requires a value")?,
));
}
"--abort-on-threat" => abort_on_threat = true,
"--quiet" | "-q" => quiet = true,
"--recursive" | "-r" => recursive = true,
"--preserve-format" => preserve_format = true,
"--rules" => {
i += 1;
external_rules_path = Some(PathBuf::from(
args.get(i).ok_or("--rules requires a value")?,
));
}
"--cve-db" => {
i += 1;
external_cve_db_path = Some(PathBuf::from(
args.get(i).ok_or("--cve-db requires a value")?,
));
}
"--help" | "-h" => {
print_usage();
std::process::exit(0);
}
other if other.starts_with("--") => {
return Err(format!("unknown flag: {other}"));
}
other => {
if path.is_none() {
path = Some(PathBuf::from(other));
} else {
return Err(format!("unexpected positional argument: {other}"));
}
}
}
i += 1;
}
let path = path.ok_or("missing required <path> argument")?;
Ok(ScanArgs {
path,
workspace,
abort_on_threat,
quiet,
recursive,
preserve_format,
external_rules_path,
external_cve_db_path,
})
}
fn run_scan(args: &[String]) -> ExitCode {
let parsed = match parse_scan_args(args) {
Ok(p) => p,
Err(e) => {
eprintln!("error: {e}");
return ExitCode::FAILURE;
}
};
let pipeline = build_pipeline(
parsed.workspace,
parsed.abort_on_threat,
parsed.preserve_format,
parsed.external_rules_path,
parsed.external_cve_db_path,
);
match pipeline.run(&parsed.path) {
Ok(result) => {
print_result(&result, parsed.quiet);
let malicious = result.scan_report.malicious_count();
if malicious > 0 {
ExitCode::from(2)
} else {
ExitCode::SUCCESS
}
}
Err(e) => {
eprintln!("error: {e}");
ExitCode::FAILURE
}
}
}
fn run_scan_dir(args: &[String]) -> ExitCode {
let parsed = match parse_scan_args(args) {
Ok(p) => p,
Err(e) => {
eprintln!("error: {e}");
return ExitCode::FAILURE;
}
};
let pipeline = build_pipeline(
parsed.workspace,
parsed.abort_on_threat,
parsed.preserve_format,
parsed.external_rules_path,
parsed.external_cve_db_path,
);
let mut found_malicious = false;
let mut had_error = false;
let files = match collect_supported_files(&parsed.path, parsed.recursive) {
Ok(f) => f,
Err(e) => {
eprintln!("error reading dir: {e}");
return ExitCode::FAILURE;
}
};
for path in files {
match pipeline.run(&path) {
Ok(result) => {
let malicious = result.scan_report.malicious_count();
if malicious > 0 {
found_malicious = true;
println!("[MALICIOUS] {} — {} finding(s)", path.display(), malicious);
if let Some(q) = &result.quarantine_path {
println!(" quarantined → {}", q.display());
}
if parsed.abort_on_threat {
return ExitCode::from(2);
}
} else {
println!("[OK] {}", path.display());
}
}
Err(e) => {
eprintln!("[ERROR] {}: {e}", path.display());
had_error = true;
}
}
}
if had_error {
ExitCode::FAILURE
} else if found_malicious {
ExitCode::from(2)
} else {
ExitCode::SUCCESS
}
}
fn collect_supported_files(
dir: &Path,
recursive: bool,
) -> std::io::Result<Vec<PathBuf>> {
const SUPPORTED: [&str; 5] = ["pdf", "epub", "md", "markdown", "docx"];
let mut out = Vec::new();
collect_supported_files_inner(dir, recursive, &mut out, &SUPPORTED)?;
out.sort();
Ok(out)
}
fn collect_supported_files_inner(
dir: &Path,
recursive: bool,
out: &mut Vec<PathBuf>,
supported: &[&str],
) -> std::io::Result<()> {
for entry in std::fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
let ft = entry.file_type()?;
if ft.is_dir() && recursive {
collect_supported_files_inner(&path, recursive, out, supported)?;
} else if ft.is_file() {
let ext_ok = path
.extension()
.and_then(|e| e.to_str())
.map(|s| {
let lower = s.to_ascii_lowercase();
supported.contains(&lower.as_str())
})
.unwrap_or(false);
if ext_ok {
out.push(path);
}
}
}
Ok(())
}
fn build_pipeline(
workspace: Option<PathBuf>,
abort_on_threat: bool,
preserve_format: bool,
external_rules_path: Option<PathBuf>,
external_cve_db_path: Option<PathBuf>,
) -> Pipeline {
let mut config = match workspace {
Some(p) => Config::with_workspace(p),
None => Config::default(),
};
config.abort_on_threat = abort_on_threat;
config.external_rules_path = external_rules_path;
config.external_cve_db_path = external_cve_db_path;
if preserve_format {
config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat;
}
// Apply env overrides last so they win.
apply_env_overrides(&mut config);
// Load external threat-intel feeds if configured.
if let Some(ref path) = config.external_rules_path {
if let Err(e) = corbel_purge::scanner::signatures::load_external_rules(path) {
eprintln!("warning: failed to load external rules from {}: {e}", path.display());
}
}
if let Some(ref path) = config.external_cve_db_path {
if let Err(e) = corbel_purge::scanner::cve_tags::load_external_cve_db(path) {
eprintln!("warning: failed to load external CVE db from {}: {e}", path.display());
}
}
Pipeline::with_config(config)
}
fn apply_env_overrides(config: &mut Config) {
if let Ok(v) = std::env::var("CORBEL_QUARANTINE_DIR") {
config.quarantine_dir = PathBuf::from(v);
}
if let Ok(v) = std::env::var("CORBEL_CLEANSE_DIR") {
config.cleanse_dir = PathBuf::from(v);
}
if let Ok(v) = std::env::var("CORBEL_ABORT_ON_THREAT") {
config.abort_on_threat = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_EMIT_SUSPICIOUS") {
config.emit_suspicious = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_EMIT_MARKDOWN_REPORT") {
config.emit_markdown_report = truthy(&v);
}
if let Ok(v) = std::env::var("CORBEL_TOTAL_ARCHIVE_SCAN_CAP") {
if let Ok(cap) = v.parse::<usize>() {
config.total_archive_scan_cap = cap;
}
}
if let Ok(v) = std::env::var("CORBEL_EXTERNAL_RULES") {
if !v.is_empty() {
config.external_rules_path = Some(PathBuf::from(v));
}
}
if let Ok(v) = std::env::var("CORBEL_EXTERNAL_CVE_DB") {
if !v.is_empty() {
config.external_cve_db_path = Some(PathBuf::from(v));
}
}
}
fn truthy(v: &str) -> bool {
matches!(
v.to_ascii_lowercase().as_str(),
"1" | "true" | "yes" | "on"
)
}
fn print_result(result: &PipelineResult, quiet: bool) {
if quiet {
if let Some(p) = &result.json_report_path {
println!("{}", p.display());
}
return;
}
let bar = "─".repeat(60);
println!("{bar}");
println!("scan complete: {}", result.format);
if let Some(p) = &result.source_path {
println!(" source: {}", p.display());
}
println!(" sha256: {}", result.source_sha256);
println!(
" text nodes: {}",
result.scan_report.text_nodes_scanned
);
println!(
" vectors: {}",
result.scan_report.vectors_scanned
);
println!(
" findings: {} malicious, {} educational, {} total",
result.scan_report.malicious_count(),
result.scan_report.educational_count(),
result.scan_report.findings.len()
);
if let Some(p) = &result.quarantine_path {
println!(" quarantine: {}", p.display());
}
if let Some(p) = &result.cleansed_path {
println!(" cleansed: {}", p.display());
}
if let Some(p) = &result.json_report_path {
println!(" json report: {}", p.display());
}
if let Some(p) = &result.markdown_report_path {
println!(" md report: {}", p.display());
}
println!("{bar}");
}
fn run_study(args: &[String]) -> ExitCode {
let parsed = parse_scan_args(args);
let parsed = match parsed {
Ok(p) => p,
Err(e) => {
eprintln!("error: {e}");
return ExitCode::FAILURE;
}
};
match corbel_purge::study::run_study(
&parsed.path,
parsed.workspace.as_deref(),
) {
Ok(html_path) => {
if !parsed.quiet {
println!("study output: {}", html_path.display());
} else {
println!("{}", html_path.display());
}
ExitCode::SUCCESS
}
Err(e) => {
eprintln!("error: {e}");
ExitCode::FAILURE
}
}
}

575
src/parsers/docx_parser.rs Executable file
View File

@ -0,0 +1,575 @@
//! DOCX parser.
//!
//! DOCX is a ZIP container of XML files (the Office Open XML format).
//! The main content lives in `word/document.xml`; macros live in
//! `word/vbaProject.xml`; embedded objects in `word/embeddings/`;
//! external relationships in `word/_rels/document.xml.rels`.
//!
//! This parser uses the [`zip`] crate to walk the container (same as
//! the EPUB parser) and extracts:
//!
//! - **Text nodes**: paragraphs (`<w:p>`) and their text runs (`<w:t>`).
//! - **Executable vectors**:
//! - VBA macros (`word/vbaProject.xml`) → [`VectorType::DocxMacro`]
//! - External hyperlinks (`r:id` with `targetMode="External"`) →
//! [`VectorType::DocxExternalLink`]
//! - Embedded objects (`word/embeddings/*.bin`) →
//! [`VectorType::DocxEmbeddedObject`]
//! - ActiveX controls (`word/activeX/*.xml`) →
//! [`VectorType::DocxActiveX`]
//!
//! We deliberately do **not** use a full OOXML schema parser — that
//! would require pulling in a heavyweight XML validation library. The
//! coarse regex-style extraction here is sufficient for the security
//! scanner's needs: it sees all text and all executable vectors.
//!
//! ## Zip-bomb defense
//!
//! Each ZIP entry is read via [`crate::util::read_with_cap`], which
//! counts **actual decompressed bytes** rather than trusting the
//! size declared in the ZIP central directory. See the EPUB parser
//! docs for details.
use std::path::PathBuf;
use zip::ZipArchive;
use crate::core::config::Config;
use crate::core::types::{
Document, DocumentFormat, DocumentMetadata, ExecutableVector, Location, TextContext, TextNode,
VectorType,
};
use crate::util::{read_with_cap, ReadOutcome};
use crate::CorbelError;
use crate::CorbelResult;
/// Concrete [`DocumentParser`] for DOCX.
pub struct DocxParser;
impl super::DocumentParser for DocxParser {
fn parse(
bytes: &[u8],
_format: DocumentFormat,
source_path: Option<PathBuf>,
config: &Config,
) -> CorbelResult<Document> {
let sha256 = crate::sha256_hex(bytes);
let size = bytes.len() as u64;
let cursor = std::io::Cursor::new(bytes.to_vec());
let mut archive = ZipArchive::new(cursor)
.map_err(|e| CorbelError::EpubParse(format!("docx zip open failed: {e}")))?;
let mut text_nodes = Vec::new();
let mut vectors = Vec::new();
let mut metadata = DocumentMetadata::default();
let mut total_decompressed: usize = 0;
for i in 0..archive.len() {
// Check total-memory budget before reading each entry.
if total_decompressed >= config.total_archive_scan_cap {
for j in i..archive.len() {
let entry = archive.by_index(j)
.map_err(|e| CorbelError::EpubParse(format!("docx entry {j} read failed: {e}")));
let entry_name = entry.map(|e| e.name().to_string()).unwrap_or_default();
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name,
anchor: None,
},
vector_type: VectorType::UnknownPayload,
raw_payload: Vec::new(),
decoded_preview: Some(format!(
"<total-archive-budget exhausted at {} bytes, entry skipped>",
total_decompressed
)),
});
}
break;
}
let mut entry = archive
.by_index(i)
.map_err(|e| CorbelError::EpubParse(format!("docx entry {i} read failed: {e}")))?;
let entry_name = entry.name().to_string();
let declared_size = entry.size() as usize;
// Streaming read with a hard cap — same zip-bomb defense
// as the EPUB parser. We don't trust `declared_size`.
let outcome = read_with_cap(&mut entry, config.epub_entry_scan_cap)
.map_err(|e| CorbelError::EpubParse(format!("docx entry {i} read failed: {e}")))?;
let (buf, was_truncated) = match outcome {
ReadOutcome::Complete(b) => {
total_decompressed = total_decompressed.saturating_add(b.len());
(b, false)
}
ReadOutcome::Truncated { bytes, bytes_read, cap } => {
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::UnknownPayload,
raw_payload: bytes.clone(),
decoded_preview: Some(format!(
"<oversized entry: declared={}, actual_read={}, cap={} — truncated>",
declared_size, bytes_read, cap
)),
});
(bytes, true)
}
};
if was_truncated {
continue;
}
let lower = entry_name.to_ascii_lowercase();
if lower == "word/document.xml" {
if let Ok(s) = std::str::from_utf8(&buf) {
extract_docx_text(s, &entry_name, &mut text_nodes);
extract_docx_external_links(s, &entry_name, &mut vectors);
// Extract core metadata from <dc:title> etc. if present
// (rare in document.xml — usually in app.xml/core.xml).
}
} else if lower == "docprops/core.xml" {
if let Ok(s) = std::str::from_utf8(&buf) {
extract_core_props(s, &mut metadata);
}
} else if lower == "word/vbaproject.xml" || lower == "word/vbaproject.bin" {
// VBA macros → always a vector.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::DocxMacro,
raw_payload: buf.clone(),
decoded_preview: String::from_utf8_lossy(&buf)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
} else if lower.starts_with("word/embeddings/") {
// Embedded OLE objects → always a vector.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::DocxEmbeddedObject,
raw_payload: buf.clone(),
decoded_preview: String::from_utf8_lossy(&buf)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
} else if lower.starts_with("word/activex/") {
// ActiveX controls → always a vector.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::DocxActiveX,
raw_payload: buf.clone(),
decoded_preview: String::from_utf8_lossy(&buf)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
} else if lower == "word/_rels/document.xml.rels" {
// Relationships file — extract external links.
if let Ok(s) = std::str::from_utf8(&buf) {
extract_rels_external_links(s, &mut vectors);
}
}
// Other entries (styles.xml, themes.xml, fonts, etc.) are
// skipped — they don't carry text or executable vectors
// that the scanner cares about.
}
Ok(Document {
format: DocumentFormat::Docx,
source_path,
raw_bytes: bytes.to_vec(),
sha256,
size,
metadata,
text_nodes,
executable_vectors: vectors,
})
}
}
/// Extract text from `word/document.xml`.
///
/// Walks `<w:p>` (paragraph) elements. Inside each paragraph, concatenates
/// the text of all `<w:t>` elements. Emits one [`TextNode`] per paragraph.
fn extract_docx_text(xml: &str, entry_name: &str, text_nodes: &mut Vec<TextNode>) {
// Split on `<w:p` to get paragraph chunks. This is a coarse
// regex-free approach — we don't need a full XML parser.
let mut search_from = 0;
while search_from < xml.len() {
let Some(p_start_rel) = xml[search_from..].find("<w:p") else {
break;
};
let abs_start = search_from + p_start_rel;
// Skip `<w:pPr` (paragraph properties) — only match real paragraphs.
// We require the char after `<w:p` to be `>` or ` ` (space).
let after = &xml[abs_start + 4..];
if !after.starts_with('>') && !after.starts_with(' ') {
search_from = abs_start + 4;
continue;
}
// Find the end of the paragraph (</w:p>).
let rest = &xml[abs_start..];
let Some(p_end_rel) = rest.find("</w:p>") else {
// No closing tag — stop.
break;
};
let p_end = abs_start + p_end_rel;
let paragraph_xml = &xml[abs_start..p_end];
// Extract all <w:t...>text</w:t> within this paragraph.
let mut paragraph_text = String::new();
let mut text_search = 0;
while let Some(t_start) = paragraph_xml[text_search..].find("<w:t") {
let abs_t_start = text_search + t_start;
// Skip past the opening tag (and any attributes).
let after_tag = &paragraph_xml[abs_t_start..];
let content_start = match after_tag.find('>') {
Some(p) => abs_t_start + p + 1,
None => break,
};
if content_start >= paragraph_xml.len() {
break;
}
let after_content = &paragraph_xml[content_start..];
let content_end = match after_content.find("</w:t>") {
Some(p) => content_start + p,
None => break,
};
paragraph_text.push_str(&paragraph_xml[content_start..content_end]);
text_search = content_end + 5; // length of "</w:t>"
}
if !paragraph_text.trim().is_empty() {
let decoded = decode_xml_entities(&paragraph_text);
text_nodes.push(TextNode {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: None,
},
context: TextContext::Paragraph,
content: decoded,
});
}
search_from = p_end + 6; // length of "</w:p>"
}
}
/// Extract external hyperlinks from `word/document.xml`.
///
/// Hyperlinks look like `<w:hyperlink r:id="rId1">text</w:hyperlink>`.
/// The actual URL is in the rels file, but we still emit a vector
/// here so the scanner knows there's an external link reference.
fn extract_docx_external_links(xml: &str, entry_name: &str, vectors: &mut Vec<ExecutableVector>) {
let mut search_from = 0;
while let Some(h_start) = xml[search_from..].find("<w:hyperlink") {
let abs_start = search_from + h_start;
// Find end of the hyperlink opening tag.
let rest = &xml[abs_start..];
let tag_end = match rest.find('>') {
Some(p) => abs_start + p + 1,
None => break,
};
// Find the closing </w:hyperlink>.
let after_tag = &xml[tag_end..];
let h_end = match after_tag.find("</w:hyperlink>") {
Some(p) => tag_end + p,
None => break,
};
// Extract r:id attribute value.
let opening_tag = &xml[abs_start..tag_end];
let rid = extract_attribute(opening_tag, "r:id");
// Extract the visible text of the hyperlink.
let inner = &xml[tag_end..h_end];
let mut visible_text = String::new();
let mut text_search = 0;
while let Some(t_start) = inner[text_search..].find("<w:t") {
let abs_t_start = text_search + t_start;
let after_tag = &inner[abs_t_start..];
let content_start = match after_tag.find('>') {
Some(p) => abs_t_start + p + 1,
None => break,
};
let after_content = &inner[content_start..];
let content_end = match after_content.find("</w:t>") {
Some(p) => content_start + p,
None => break,
};
visible_text.push_str(&inner[content_start..content_end]);
text_search = content_end + 5;
}
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: rid.clone(),
},
vector_type: VectorType::DocxExternalLink,
raw_payload: visible_text.as_bytes().to_vec(),
decoded_preview: Some(format!("rId={} text={}", rid.unwrap_or_default(), visible_text)),
});
search_from = h_end + 14; // length of "</w:hyperlink>"
}
}
/// Extract external relationships from `word/_rels/document.xml.rels`.
///
/// Each `<Relationship>` element has attributes `Id`, `Target`, and
/// `TargetMode`. If `TargetMode="External"`, the relationship points
/// to an external URL — emit a vector with the URL as the payload.
fn extract_rels_external_links(xml: &str, vectors: &mut Vec<ExecutableVector>) {
let mut search_from = 0;
while let Some(rel_start) = xml[search_from..].find("<Relationship") {
let abs_start = search_from + rel_start;
let rest = &xml[abs_start..];
let tag_end = match rest.find("/>").map(|p| abs_start + p + 2).or_else(|| {
rest.find('>').map(|p| abs_start + p + 1)
}) {
Some(p) => p,
None => break,
};
let opening_tag = &xml[abs_start..tag_end];
let target_mode = extract_attribute(opening_tag, "TargetMode");
let target = extract_attribute(opening_tag, "Target").unwrap_or_default();
let id = extract_attribute(opening_tag, "Id").unwrap_or_default();
if target_mode.as_deref() == Some("External") {
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: "word/_rels/document.xml.rels".to_string(),
anchor: Some(id.clone()),
},
vector_type: VectorType::DocxExternalLink,
raw_payload: target.as_bytes().to_vec(),
decoded_preview: Some(format!("rId={} target={}", id, target)),
});
}
search_from = tag_end;
}
}
/// Extract a `<attr="value">` or `<attr='value'>` attribute from an XML tag.
fn extract_attribute(tag: &str, attr_name: &str) -> Option<String> {
// Look for `attr_name="..."` or `attr_name='...'`.
let needle = format!("{attr_name}=\"");
if let Some(start) = tag.find(&needle) {
let value_start = start + needle.len();
if let Some(end) = tag[value_start..].find('"') {
return Some(tag[value_start..value_start + end].to_string());
}
}
let needle = format!("{attr_name}='");
if let Some(start) = tag.find(&needle) {
let value_start = start + needle.len();
if let Some(end) = tag[value_start..].find('\'') {
return Some(tag[value_start..value_start + end].to_string());
}
}
None
}
/// Decode the five predefined XML entities. Leaves other text unchanged.
fn decode_xml_entities(s: &str) -> String {
s.replace("&amp;", "&")
.replace("&lt;", "<")
.replace("&gt;", ">")
.replace("&quot;", "\"")
.replace("&apos;", "'")
}
/// Extract `<dc:title>`, `<dc:creator>`, `<dc:subject>` from `docProps/core.xml`.
fn extract_core_props(xml: &str, meta: &mut DocumentMetadata) {
meta.title = extract_xml_element_text(xml, "dc:title");
meta.author = extract_xml_element_text(xml, "dc:creator");
meta.subject = extract_xml_element_text(xml, "dc:subject");
}
/// Extract the text content of `<tag>...</tag>` (first occurrence).
fn extract_xml_element_text(xml: &str, tag: &str) -> Option<String> {
let open = format!("<{tag}");
let close = format!("</{tag}>");
let start = xml.find(&open)?;
let after_open = &xml[start..];
let content_start = after_open.find('>')? + 1 + start;
let after_content = &xml[content_start..];
let end = after_content.find(&close)?;
let raw = &xml[content_start..content_start + end];
Some(decode_xml_entities(raw))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parsers::DocumentParser;
use std::io::Write;
fn make_minimal_docx(entries: &[(&str, &[u8])]) -> Vec<u8> {
let mut buf = Vec::new();
{
let mut zip = zip::ZipWriter::new(std::io::Cursor::new(&mut buf));
let opts = zip::write::SimpleFileOptions::default()
.compression_method(zip::CompressionMethod::Stored);
for (name, data) in entries {
zip.start_file(name, opts).unwrap();
zip.write_all(data).unwrap();
}
zip.finish().unwrap();
}
buf
}
fn minimal_document_xml() -> &'static [u8] {
b"<?xml version=\"1.0\"?>
<w:document xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\">
<w:body>
<w:p><w:r><w:t>Hello, this is a benign DOCX.</w:t></w:r></w:p>
<w:p><w:r><w:t>Second paragraph.</w:t></w:r></w:p>
</w:body>
</w:document>"
}
fn minimal_rels_xml() -> &'static [u8] {
b"<?xml version=\"1.0\"?>
<Relationships xmlns=\"http://schemas.openxmlformats.org/package/2006/relationships\">
<Relationship Id=\"rId1\" Type=\"http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink\" Target=\"https://example.com\" TargetMode=\"External\"/>
<Relationship Id=\"rId2\" Type=\"http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles\" Target=\"styles.xml\"/>
</Relationships>"
}
fn minimal_core_xml() -> &'static [u8] {
b"<?xml version=\"1.0\"?>
<cp:coreProperties xmlns:cp=\"http://schemas.openxmlformats.org/package/2006/metadata/core-properties\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\">
<dc:title>Test DOCX</dc:title>
<dc:creator>CorbelPurge Tests</dc:creator>
<dc:subject>Test subject</dc:subject>
</cp:coreProperties>"
}
#[test]
fn parses_minimal_docx() {
let bytes = make_minimal_docx(&[
("[Content_Types].xml", b"<?xml version='1.0'?><Types xmlns='http://schemas.openxmlformats.org/package/2006/content-types'><Default Extension='rels' ContentType='application/vnd.openxmlformats-package.relationships+xml'/><Default Extension='xml' ContentType='application/xml'/></Types>"),
("word/document.xml", minimal_document_xml()),
("docProps/core.xml", minimal_core_xml()),
("word/_rels/document.xml.rels", minimal_rels_xml()),
]);
let doc = DocxParser::parse(&bytes, DocumentFormat::Docx, None, &Config::default()).unwrap();
assert_eq!(doc.format, DocumentFormat::Docx);
assert_eq!(doc.metadata.title.as_deref(), Some("Test DOCX"));
assert_eq!(doc.metadata.author.as_deref(), Some("CorbelPurge Tests"));
assert_eq!(doc.metadata.subject.as_deref(), Some("Test subject"));
// Two paragraphs.
assert_eq!(doc.text_nodes.len(), 2);
let combined: String = doc.text_nodes.iter().map(|n| n.content.as_str()).collect();
assert!(combined.contains("benign DOCX"));
assert!(combined.contains("Second paragraph"));
// One external hyperlink (from rels).
let external_links: Vec<_> = doc
.executable_vectors
.iter()
.filter(|v| v.vector_type == VectorType::DocxExternalLink)
.collect();
assert_eq!(external_links.len(), 1);
assert!(external_links[0].decoded_preview.as_ref().unwrap().contains("https://example.com"));
}
#[test]
fn flags_vba_macros() {
let bytes = make_minimal_docx(&[
("word/document.xml", minimal_document_xml()),
("word/vbaProject.xml", b"<?xml version='1.0'?><vbaProject><module name='evil'>Sub AutoOpen() ...</module></vbaProject>"),
]);
let doc = DocxParser::parse(&bytes, DocumentFormat::Docx, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::DocxMacro),
"should have flagged the VBA macro"
);
}
#[test]
fn flags_embedded_objects() {
let bytes = make_minimal_docx(&[
("word/document.xml", minimal_document_xml()),
("word/embeddings/oleObject1.bin", b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1rest of OLE"),
]);
let doc = DocxParser::parse(&bytes, DocumentFormat::Docx, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::DocxEmbeddedObject),
"should have flagged the embedded OLE object"
);
}
#[test]
fn flags_activex_controls() {
let bytes = make_minimal_docx(&[
("word/document.xml", minimal_document_xml()),
("word/activeX/activeX1.xml", b"<?xml version='1.0'?><activeX><classId>clsid:...</classId></activeX>"),
]);
let doc = DocxParser::parse(&bytes, DocumentFormat::Docx, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::DocxActiveX),
"should have flagged the ActiveX control"
);
}
#[test]
fn rejects_invalid_zip() {
let result = DocxParser::parse(b"not a zip", DocumentFormat::Docx, None, &Config::default());
assert!(result.is_err());
}
#[test]
fn decodes_xml_entities() {
let bytes = make_minimal_docx(&[
("word/document.xml", b"<?xml version='1.0'?>
<w:document xmlns:w='http://schemas.openxmlformats.org/wordprocessingml/2006/main'>
<w:body>
<w:p><w:r><w:t>5 &lt; 10 &amp; 10 &gt; 5</w:t></w:r></w:p>
</w:body>
</w:document>"),
]);
let doc = DocxParser::parse(&bytes, DocumentFormat::Docx, None, &Config::default()).unwrap();
assert_eq!(doc.text_nodes.len(), 1);
assert_eq!(doc.text_nodes[0].content, "5 < 10 & 10 > 5");
}
}

544
src/parsers/epub_parser.rs Executable file
View File

@ -0,0 +1,544 @@
//! EPUB parser.
//!
//! EPUB is a ZIP container of XHTML files plus metadata. We use the
//! [`zip`] crate directly (rather than the higher-level `epub` crate)
//! because the scanner needs byte-level access to every container
//! entry — including non-XHTML resources like embedded fonts, images,
//! and (critically) any unexpected binary blobs that could be a
//! malicious payload.
//!
//! ## Zip-bomb defense
//!
//! Each ZIP entry is read via [`crate::util::read_with_cap`], which
//! counts **actual decompressed bytes** rather than trusting the
//! size declared in the ZIP central directory. A malicious archive
//! that declares `size = 100` but actually decompresses to gigabytes
//! is detected and truncated at the configured cap
//! ([`Config::epub_entry_scan_cap`], default 8 MiB).
use std::path::PathBuf;
use zip::ZipArchive;
use crate::core::config::Config;
use crate::core::types::{
Document, DocumentFormat, DocumentMetadata, ExecutableVector, Location, TextContext, TextNode,
VectorType,
};
use crate::util::{read_with_cap, ReadOutcome};
use crate::CorbelError;
use crate::CorbelResult;
/// Concrete [`DocumentParser`] for EPUB.
pub struct EpubParser;
impl super::DocumentParser for EpubParser {
fn parse(
bytes: &[u8],
_format: DocumentFormat,
source_path: Option<PathBuf>,
config: &Config,
) -> CorbelResult<Document> {
let sha256 = crate::sha256_hex(bytes);
let size = bytes.len() as u64;
let cursor = std::io::Cursor::new(bytes.to_vec());
let mut archive = ZipArchive::new(cursor)
.map_err(|e| CorbelError::EpubParse(format!("zip open failed: {e}")))?;
let mut text_nodes = Vec::new();
let mut vectors = Vec::new();
let mut metadata = DocumentMetadata::default();
let mut total_decompressed: usize = 0;
for i in 0..archive.len() {
// Check total-memory budget before reading each entry.
if total_decompressed >= config.total_archive_scan_cap {
// Budget exhausted — emit remaining entries as oversized.
// We still record them as vectors so the scanner sees them.
for j in i..archive.len() {
let entry = archive.by_index(j)
.map_err(|e| CorbelError::EpubParse(format!("zip entry {j} read failed: {e}")))?;
let entry_name = entry.name().to_string();
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::UnknownPayload,
raw_payload: Vec::new(),
decoded_preview: Some(format!(
"<total-archive-budget exhausted at {} bytes, entry '{}' skipped>",
total_decompressed, entry_name
)),
});
}
break;
}
let mut entry = archive
.by_index(i)
.map_err(|e| CorbelError::EpubParse(format!("zip entry {i} read failed: {e}")))?;
let entry_name = entry.name().to_string();
let declared_size = entry.size() as usize;
// Streaming read with a hard cap. We deliberately do NOT
// trust `declared_size` — a malicious ZIP can declare any
// size in its central directory while the actual
// decompressed content is much larger. `read_with_cap`
// counts actual bytes consumed from the stream and
// aborts once we hit `epub_entry_scan_cap`.
let outcome = read_with_cap(&mut entry, config.epub_entry_scan_cap)
.map_err(|e| CorbelError::EpubParse(format!("zip entry {i} read failed: {e}")))?;
let (buf, was_truncated) = match outcome {
ReadOutcome::Complete(b) => {
total_decompressed = total_decompressed.saturating_add(b.len());
(b, false)
}
ReadOutcome::Truncated { bytes, bytes_read, cap } => {
// The entry exceeded the cap. Emit it as an
// UnknownPayload so the scanner still inspects the
// first `cap` bytes for file signatures (PE, ELF,
// OLE2, etc.). This is the zip-bomb defense: we
// never allocate more than `cap` bytes per entry,
// regardless of what the ZIP header claimed.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::UnknownPayload,
raw_payload: bytes.clone(),
decoded_preview: Some(format!(
"<oversized entry: declared={}, actual_read={}, cap={} — truncated>",
declared_size, bytes_read, cap
)),
});
(bytes, true)
}
};
// If we truncated, skip the normal classification — we
// already emitted an UnknownPayload above. The partial
// bytes are still useful for file-signature matching in
// the scanner, but we can't reliably parse them as
// XHTML/OPF/etc.
if was_truncated {
continue;
}
// Classify the entry by extension / mime.
let lower = entry_name.to_ascii_lowercase();
if lower.ends_with(".opf") {
// OPF = package metadata XML.
if let Ok(s) = std::str::from_utf8(&buf) {
extract_opf_metadata(s, &mut metadata);
}
} else if lower.ends_with(".ncx") {
// NCX = navigation. Skip text extraction (it's just ToC labels).
} else if lower.ends_with(".xhtml")
|| lower.ends_with(".html")
|| lower.ends_with(".htm")
{
// XHTML content — extract text + scan for <script> tags.
if let Ok(s) = std::str::from_utf8(&buf) {
extract_xhtml(
s,
&entry_name,
&mut text_nodes,
&mut vectors,
);
}
} else if lower.ends_with(".css") {
// Stylesheet — text, but not really prose. Skip.
} else if lower.ends_with(".js") {
// JavaScript file inside the EPUB — definitely a vector.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::EpubScript,
raw_payload: buf.clone(),
decoded_preview: String::from_utf8_lossy(&buf)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
} else if lower.ends_with(".ttf")
|| lower.ends_with(".otf")
|| lower.ends_with(".woff")
|| lower.ends_with(".woff2")
{
// Fonts — skip.
} else if lower.ends_with(".png")
|| lower.ends_with(".jpg")
|| lower.ends_with(".jpeg")
|| lower.ends_with(".gif")
|| lower.ends_with(".svg")
{
// Images — skip text extraction. SVG could carry scripts;
// we'll add an SVG-script detector later if needed.
if lower.ends_with(".svg") {
if let Ok(s) = std::str::from_utf8(&buf) {
if s.contains("<script") {
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::EpubScript,
raw_payload: buf.clone(),
decoded_preview: Some(s.chars().take(2048).collect()),
});
}
}
}
} else {
// Unknown entry — could be an embedded executable.
// Record as a vector (UnknownPayload) so the scanner
// can inspect it.
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.clone(),
anchor: None,
},
vector_type: VectorType::UnknownPayload,
raw_payload: buf.clone(),
decoded_preview: String::from_utf8_lossy(&buf)
.chars()
.take(512)
.collect::<String>()
.into(),
});
}
}
Ok(Document {
format: DocumentFormat::Epub,
source_path,
raw_bytes: bytes.to_vec(),
sha256,
size,
metadata,
text_nodes,
executable_vectors: vectors,
})
}
}
/// Extract `<dc:title>`, `<dc:creator>`, etc. from an OPF package file.
fn extract_opf_metadata(opf: &str, meta: &mut DocumentMetadata) {
// Very lightweight regex-free extraction. We're not building a
// fully-conformant OPF parser — just grabbing the obvious fields.
if let Some(start) = opf.find("<dc:title") {
if let Some(content_start) = opf[start..].find('>') {
let after = &opf[start + content_start + 1..];
if let Some(end) = after.find("</dc:title>") {
meta.title = Some(after[..end].trim().to_string());
}
}
}
if let Some(start) = opf.find("<dc:creator") {
if let Some(content_start) = opf[start..].find('>') {
let after = &opf[start + content_start + 1..];
if let Some(end) = after.find("</dc:creator>") {
meta.author = Some(after[..end].trim().to_string());
}
}
}
if let Some(start) = opf.find("<dc:description") {
if let Some(content_start) = opf[start..].find('>') {
let after = &opf[start + content_start + 1..];
if let Some(end) = after.find("</dc:description>") {
meta.subject = Some(after[..end].trim().to_string());
}
}
}
}
/// Extract text and detect active vectors from an XHTML content file.
fn extract_xhtml(
xhtml: &str,
entry_name: &str,
text_nodes: &mut Vec<TextNode>,
vectors: &mut Vec<ExecutableVector>,
) {
// 1. Detect <script> tags.
let mut search_from = 0;
while let Some(script_start) = xhtml[search_from..].find("<script") {
let abs_start = search_from + script_start;
// Find the closing </script>
if let Some(content_start) = xhtml[abs_start..].find('>') {
let after_open = abs_start + content_start + 1;
if let Some(end) = xhtml[after_open..].find("</script>") {
let script_body = &xhtml[after_open..after_open + end];
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: Some("script".to_string()),
},
vector_type: VectorType::EpubScript,
raw_payload: script_body.as_bytes().to_vec(),
decoded_preview: Some(script_body.chars().take(2048).collect()),
});
search_from = after_open + end + 9; // length of "</script>"
continue;
}
}
// Malformed <script> — bail this loop.
break;
}
// 2. Detect <object>, <embed>, <iframe>.
for tag_name in &["<object", "<embed", "<iframe"] {
let mut search_from = 0;
while let Some(rel_start) = xhtml[search_from..].find(tag_name) {
let abs_start = search_from + rel_start;
let snippet_end = xhtml[abs_start..]
.find('>')
.map(|p| abs_start + p + 1)
.unwrap_or(xhtml.len());
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: Some(tag_name.trim_start_matches('<').to_string()),
},
vector_type: VectorType::EpubObject,
raw_payload: xhtml[abs_start..snippet_end].as_bytes().to_vec(),
decoded_preview: Some(xhtml[abs_start..snippet_end].to_string()),
});
search_from = snippet_end;
}
}
// 3. Detect external resource references (src=, href= with http(s)://).
extract_external_resources(xhtml, entry_name, vectors);
// 4. Extract body text (very coarse: strip everything between < and >).
// This is intentionally lossy — we're not trying to preserve formatting,
// we just want paragraphs of text for the scanner to inspect.
extract_text_from_xhtml(xhtml, entry_name, text_nodes);
}
/// Extract `src="http..."` and `href="http..."` references as external-resource vectors.
fn extract_external_resources(
xhtml: &str,
entry_name: &str,
vectors: &mut Vec<ExecutableVector>,
) {
let mut search_from = 0;
while let Some(rel_start) = xhtml[search_from..].find("http") {
let abs_start = search_from + rel_start;
// Make sure this is inside an attribute (preceded by src=, href=, etc.)
let preceding = &xhtml[abs_start.saturating_sub(16)..abs_start];
if !preceding.contains("src=")
&& !preceding.contains("href=")
&& !preceding.contains("data=")
{
search_from = abs_start + 4;
continue;
}
// Find the end of the URL (next quote or whitespace).
let rest = &xhtml[abs_start..];
let url_end = rest
.find(|c: char| c.is_whitespace() || c == '"' || c == '\'')
.map(|p| abs_start + p)
.unwrap_or(abs_start + 256);
let url = &xhtml[abs_start..url_end.min(xhtml.len())];
if !url.is_empty() {
vectors.push(ExecutableVector {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: Some("external-resource".to_string()),
},
vector_type: VectorType::EpubExternalResource,
raw_payload: url.as_bytes().to_vec(),
decoded_preview: Some(url.to_string()),
});
}
search_from = url_end + 1;
}
}
/// Coarse text extraction: strip tags, split on block boundaries,
/// emit each non-empty chunk as a Paragraph-context TextNode.
fn extract_text_from_xhtml(
xhtml: &str,
entry_name: &str,
text_nodes: &mut Vec<TextNode>,
) {
// Extract content between <body> and </body>, if present.
let body = if let Some(b_start) = xhtml.to_ascii_lowercase().find("<body") {
let after = &xhtml[b_start..];
let body_content_start = after.find('>').map(|p| b_start + p + 1).unwrap_or(b_start);
if let Some(end) = xhtml[body_content_start..]
.to_ascii_lowercase()
.find("</body>")
{
&xhtml[body_content_start..body_content_start + end]
} else {
&xhtml[body_content_start..]
}
} else {
xhtml
};
// Split on block-level closing tags so each <p>, <div>, <h1>, etc.
// becomes its own text node.
let mut current = String::new();
let mut i = 0;
let bytes = body.as_bytes();
while i < bytes.len() {
if bytes[i] == b'<' {
// Find end of tag.
let rest = &body[i..];
let tag_end = rest.find('>').map(|p| i + p + 1).unwrap_or(body.len());
let tag = &body[i..tag_end];
let tag_lower = tag.to_ascii_lowercase();
// If this is a closing block tag, flush the buffer.
let is_block_close = [
"</p>", "</div>", "</h1>", "</h2>", "</h3>", "</h4>", "</h5>",
"</h6>", "</li>", "</blockquote>", "</pre>",
]
.iter()
.any(|t| tag_lower.starts_with(t));
if is_block_close {
let trimmed = current.trim();
if !trimmed.is_empty() {
text_nodes.push(TextNode {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: None,
},
context: TextContext::Paragraph,
content: trimmed.to_string(),
});
}
current.clear();
}
// If this is a <pre> opening tag, switch to CodeBlock context
// (handled by the next flush). For simplicity we treat all
// text inside <pre> as CodeBlock.
// (Skipping for now — the simple paragraph split is enough
// for the MVP scanner to do its job.)
i = tag_end;
} else {
// Append the character (decode entities minimally).
let ch = body[i..].chars().next().unwrap();
current.push(ch);
i += ch.len_utf8();
}
}
// Final flush.
let trimmed = current.trim();
if !trimmed.is_empty() {
text_nodes.push(TextNode {
location: Location::EpubEntry {
path: entry_name.to_string(),
anchor: None,
},
context: TextContext::Paragraph,
content: trimmed.to_string(),
});
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parsers::DocumentParser;
use std::io::Write;
fn make_minimal_epub(entries: &[(&str, &[u8])]) -> Vec<u8> {
// Use the zip crate to write a minimal in-memory EPUB.
let mut buf = Vec::new();
{
let mut zip = zip::ZipWriter::new(std::io::Cursor::new(&mut buf));
let opts =
zip::write::SimpleFileOptions::default().compression_method(zip::CompressionMethod::Stored);
for (name, data) in entries {
zip.start_file(name, opts).unwrap();
zip.write_all(data).unwrap();
}
zip.finish().unwrap();
}
buf
}
#[test]
fn parses_minimal_epub() {
let mimetype = b"application/epub+zip";
let opf = br#"<?xml version="1.0"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:title>Test Book</dc:title>
<dc:creator>Test Author</dc:creator>
</metadata>
</package>"#;
let xhtml = br#"<?xml version="1.0"?>
<html><head><title>Ch1</title></head>
<body><p>Hello world.</p><p>Second paragraph.</p></body></html>"#;
let bytes = make_minimal_epub(&[
("mimetype", mimetype),
("OEBPS/content.opf", opf),
("OEBPS/ch1.xhtml", xhtml),
]);
let doc = EpubParser::parse(&bytes, DocumentFormat::Epub, None, &Config::default()).unwrap();
assert_eq!(doc.format, DocumentFormat::Epub);
assert_eq!(doc.metadata.title.as_deref(), Some("Test Book"));
assert_eq!(doc.metadata.author.as_deref(), Some("Test Author"));
assert!(!doc.text_nodes.is_empty(), "should have extracted paragraph text");
let combined: String = doc.text_nodes.iter().map(|n| n.content.as_str()).collect();
assert!(combined.contains("Hello world"));
assert!(combined.contains("Second paragraph"));
}
#[test]
fn flags_scripts_as_vectors() {
let xhtml = br#"<html><body>
<p>Normal text.</p>
<script>alert('xss');</script>
</body></html>"#;
let bytes = make_minimal_epub(&[("OEBPS/ch1.xhtml", xhtml)]);
let doc = EpubParser::parse(&bytes, DocumentFormat::Epub, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::EpubScript),
"should have flagged the <script> tag"
);
}
#[test]
fn flags_external_resources() {
let xhtml = br#"<html><body>
<img src="https://evil.example.com/track.png" />
</body></html>"#;
let bytes = make_minimal_epub(&[("OEBPS/ch1.xhtml", xhtml)]);
let doc = EpubParser::parse(&bytes, DocumentFormat::Epub, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::EpubExternalResource),
"should have flagged the external resource"
);
}
#[test]
fn rejects_invalid_zip() {
let result = EpubParser::parse(b"not a zip", DocumentFormat::Epub, None, &Config::default());
assert!(result.is_err());
}
}

307
src/parsers/md_parser.rs Executable file
View File

@ -0,0 +1,307 @@
//! Markdown parser.
//!
//! Markdown is the simplest of the three formats: pure text, no
//! executable hooks. The only vectors we extract are hyperlinks
//! (which the scanner will inspect for phishing patterns).
use std::path::PathBuf;
use pulldown_cmark::{Event, Options, Parser, Tag, TagEnd};
use crate::core::config::Config;
use crate::core::types::{
Document, DocumentFormat, DocumentMetadata, ExecutableVector, Location, TextContext, TextNode,
VectorType,
};
use crate::CorbelResult;
/// Concrete [`DocumentParser`] for Markdown.
pub struct MarkdownParser;
impl super::DocumentParser for MarkdownParser {
fn parse(
bytes: &[u8],
_format: DocumentFormat,
source_path: Option<PathBuf>,
_config: &Config,
) -> CorbelResult<Document> {
// Markdown is text; lossy-convert to UTF-8.
let text = String::from_utf8_lossy(bytes).into_owned();
let sha256 = crate::sha256_hex(&text.as_bytes());
let size = bytes.len() as u64;
let mut text_nodes = Vec::new();
let mut vectors = Vec::new();
// Track the current semantic context as we walk events.
let mut ctx_stack: Vec<TextContext> = vec![TextContext::Paragraph];
let mut current_line: u32 = 1;
let mut current_col: u32 = 0;
// Buffer accumulating text under the current node, plus the
// line/col where accumulation started.
let mut buf = String::new();
let mut buf_start: Option<(u32, u32)> = None;
let opts = Options::ENABLE_TABLES
.union(Options::ENABLE_STRIKETHROUGH)
.union(Options::ENABLE_TASKLISTS);
let parser = Parser::new_ext(&text, opts);
for event in parser {
match event {
Event::Start(tag) => {
// If this is a link, capture its destination as a vector.
if let Tag::Link { dest_url, .. } = &tag {
vectors.push(ExecutableVector {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
vector_type: VectorType::MarkdownHyperlink,
raw_payload: dest_url.as_bytes().to_vec(),
decoded_preview: Some(dest_url.to_string()),
});
}
let new_ctx = context_for_tag(&tag);
ctx_stack.push(new_ctx);
// For code blocks / headings we want to capture
// a fresh text node, so flush any pending buf.
let ctx_for_flush = *ctx_stack.last().unwrap_or(&TextContext::Paragraph);
flush_buf(&mut buf, &mut buf_start, &ctx_for_flush, &mut text_nodes);
}
Event::End(tag_end) => {
let ended_ctx = *ctx_stack.last().unwrap_or(&TextContext::Paragraph);
flush_buf(&mut buf, &mut buf_start, &ended_ctx, &mut text_nodes);
// Pop the context stack — but make sure we don't pop
// the bottom Paragraph frame.
if ctx_stack.len() > 1 {
ctx_stack.pop();
}
// Reference `tag_end` so the compiler doesn't warn
// about unused variable (we don't need it for anything
// else; the link vector was already captured on Start).
let _ = tag_end;
}
Event::Text(t) => {
if buf_start.is_none() {
buf_start = Some((current_line, current_col));
}
// Update line/col by counting newlines in the text.
for ch in t.chars() {
if ch == '\n' {
current_line += 1;
current_col = 0;
} else {
current_col += 1;
}
}
buf.push_str(&t);
}
Event::Code(c) => {
// Inline code span — emit as its own text node so the
// scanner can see it even if no other text accumulated.
text_nodes.push(TextNode {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
context: TextContext::CodeSpan,
content: c.into_string(),
});
}
Event::InlineHtml(h) => {
// Inline HTML inside Markdown is suspicious by default —
// emit as a text node with CodeSpan context so the
// scanner can decide.
text_nodes.push(TextNode {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
context: TextContext::CodeSpan,
content: h.into_string(),
});
}
Event::DisplayMath(m) | Event::InlineMath(m) => {
text_nodes.push(TextNode {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
context: TextContext::CodeSpan,
content: m.into_string(),
});
}
Event::SoftBreak | Event::HardBreak => {
current_line += 1;
current_col = 0;
if !buf.is_empty() {
buf.push('\n');
}
}
Event::TaskListMarker(_) => {
// Ignore — TaskListMarker doesn't carry text.
}
Event::FootnoteReference(f) => {
text_nodes.push(TextNode {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
context: TextContext::Paragraph,
content: format!("[^{}]", f),
});
}
Event::Html(h) => {
// Block-level HTML — emit as its own paragraph-context
// text node so the scanner can inspect it.
text_nodes.push(TextNode {
location: Location::MarkdownLine {
line: current_line,
col: current_col,
},
context: TextContext::Paragraph,
content: h.into_string(),
});
}
Event::Rule => {
// Horizontal rule — no text to capture.
}
}
}
// Flush any trailing buffered text.
flush_buf(
&mut buf,
&mut buf_start,
ctx_stack.last().unwrap_or(&TextContext::Paragraph),
&mut text_nodes,
);
// Extract metadata from the first heading.
let metadata = extract_metadata(&text_nodes);
Ok(Document {
format: DocumentFormat::Markdown,
source_path,
raw_bytes: bytes.to_vec(),
sha256,
size,
metadata,
text_nodes,
executable_vectors: vectors,
})
}
}
fn context_for_tag(tag: &Tag) -> TextContext {
match tag {
Tag::Paragraph => TextContext::Paragraph,
Tag::Heading { .. } => TextContext::Heading,
Tag::CodeBlock(_) => TextContext::CodeBlock,
Tag::Emphasis | Tag::Strong | Tag::Strikethrough => TextContext::Paragraph,
Tag::Link { .. } => TextContext::Hyperlink,
Tag::Image { .. } => TextContext::Paragraph,
Tag::BlockQuote(_) => TextContext::BlockQuote,
Tag::List(_) | Tag::Item => TextContext::Paragraph,
_ => TextContext::Paragraph,
}
}
fn flush_buf(
buf: &mut String,
buf_start: &mut Option<(u32, u32)>,
ctx: &TextContext,
nodes: &mut Vec<TextNode>,
) {
if buf.is_empty() {
return;
}
let (line, col) = buf_start.unwrap_or((1, 0));
nodes.push(TextNode {
location: Location::MarkdownLine { line, col },
context: *ctx,
content: std::mem::take(buf),
});
*buf_start = None;
}
/// Best-effort extraction of document metadata from the first heading
/// and optional `<!-- corbel: ... -->` comment.
fn extract_metadata(nodes: &[TextNode]) -> DocumentMetadata {
let mut meta = DocumentMetadata::default();
// First H1 heading becomes the title.
if let Some(first_h1) = nodes
.iter()
.find(|n| n.context == TextContext::Heading)
.map(|n| n.content.trim().to_string())
.filter(|s| !s.is_empty())
{
meta.title = Some(first_h1);
}
meta
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parsers::DocumentParser;
#[test]
fn parses_simple_markdown() {
let md = b"# Hello\n\nThis is a paragraph.\n\n## Subhead\n\nMore text.";
let doc = MarkdownParser::parse(md, DocumentFormat::Markdown, None, &Config::default()).unwrap();
assert_eq!(doc.format, DocumentFormat::Markdown);
assert!(!doc.text_nodes.is_empty());
assert_eq!(doc.metadata.title.as_deref(), Some("Hello"));
}
#[test]
fn captures_code_blocks() {
let md = b"# Title\n\n```python\nimport os\nos.system('rm -rf /')\n```\n";
let doc = MarkdownParser::parse(md, DocumentFormat::Markdown, None, &Config::default()).unwrap();
let code_nodes: Vec<_> = doc
.text_nodes
.iter()
.filter(|n| n.context == TextContext::CodeBlock)
.collect();
assert!(!code_nodes.is_empty(), "should have at least one code block node");
let combined: String = code_nodes.iter().map(|n| n.content.as_str()).collect();
assert!(combined.contains("os.system"));
}
#[test]
fn captures_hyperlinks_as_vectors() {
let md = b"# Title\n\n[click me](https://evil.example.com/path)\n";
let doc = MarkdownParser::parse(md, DocumentFormat::Markdown, None, &Config::default()).unwrap();
assert_eq!(doc.executable_vectors.len(), 1);
assert_eq!(
doc.executable_vectors[0].vector_type,
VectorType::MarkdownHyperlink
);
assert_eq!(
doc.executable_vectors[0].decoded_preview.as_deref(),
Some("https://evil.example.com/path")
);
}
#[test]
fn sha256_is_stable() {
let md = b"# Hello\n";
let doc1 = MarkdownParser::parse(md, DocumentFormat::Markdown, None, &Config::default()).unwrap();
let doc2 = MarkdownParser::parse(md, DocumentFormat::Markdown, None, &Config::default()).unwrap();
assert_eq!(doc1.sha256, doc2.sha256);
assert_eq!(doc1.sha256.len(), 64);
}
}
// Suppress unused-import warning for `TagEnd` — we keep it imported so
// the parser intent stays explicit, even though we currently rely on
// context-stack pop rather than matching specific TagEnd variants.
#[allow(unused_imports)]
use TagEnd as _UnusedTagEnd;

87
src/parsers/mod.rs Executable file
View File

@ -0,0 +1,87 @@
//! Unified document parser trait and per-format implementations.
//!
//! The trait abstraction here is critical: every downstream module
//! (scanner, quarantine, cleanse) operates on the [`crate::core::Document`]
//! UIR only, so adding a new format later means writing one more
//! `DocumentParser` impl and nothing else.
pub mod pdf_parser;
pub mod epub_parser;
pub mod md_parser;
pub mod docx_parser;
use std::path::PathBuf;
use crate::core::config::Config;
use crate::core::types::{Document, DocumentFormat};
use crate::CorbelResult;
/// The unified parser interface. Every format implements this.
///
/// Implementations are responsible for:
///
/// 1. Hashing the raw bytes (SHA-256).
/// 2. Walking the document's structure and extracting:
/// - static [`TextNode`](crate::core::types::TextNode)s
/// - active [`ExecutableVector`](crate::core::types::ExecutableVector)s
/// 3. Capturing declared metadata (title, author, etc.).
///
/// Implementations must **not**:
/// - Execute any embedded script or active content.
/// - Follow any external URI.
/// - Allocate unbounded memory for decompressed streams. The `config`
/// parameter carries per-format caps (e.g.
/// [`Config::epub_entry_scan_cap`](crate::core::config::Config::epub_entry_scan_cap))
/// that parsers must respect via [`crate::util::read_with_cap`].
pub trait DocumentParser: Send + Sync {
/// Parse the given bytes into a [`Document`].
///
/// The `config` parameter is used to cap memory usage during
/// streaming reads of compressed container entries (EPUB, DOCX).
fn parse(
bytes: &[u8],
format: DocumentFormat,
source_path: Option<PathBuf>,
config: &Config,
) -> CorbelResult<Document>;
}
/// Dispatcher that picks the right concrete parser based on `format`.
pub struct Dispatcher;
impl DocumentParser for Dispatcher {
fn parse(
bytes: &[u8],
format: DocumentFormat,
source_path: Option<PathBuf>,
config: &Config,
) -> CorbelResult<Document> {
match format {
DocumentFormat::Pdf => {
pdf_parser::PdfParser::parse(bytes, format, source_path, config)
}
DocumentFormat::Epub => {
epub_parser::EpubParser::parse(bytes, format, source_path, config)
}
DocumentFormat::Markdown => {
md_parser::MarkdownParser::parse(bytes, format, source_path, config)
}
DocumentFormat::Docx => {
docx_parser::DocxParser::parse(bytes, format, source_path, config)
}
}
}
}
// Re-export the dispatcher as the top-level entrypoint.
impl DocumentFormat {
/// Dispatch to the correct parser for this format.
pub fn parse(
self,
bytes: &[u8],
source_path: Option<PathBuf>,
config: &Config,
) -> CorbelResult<Document> {
Dispatcher::parse(bytes, self, source_path, config)
}
}

713
src/parsers/pdf_parser.rs Executable file
View File

@ -0,0 +1,713 @@
//! PDF parser.
//!
//! Uses [`lopdf`] for low-level PDF structure inspection. We walk every
//! indirect object in the PDF and classify it:
//!
//! - **Executable vectors** (treated as untrusted-by-default):
//! - `/JavaScript` and `/JS` action streams
//! - `/Launch` actions
//! - `/URI` actions
//! - `/EmbeddedFiles` attachments
//! - `/Annot` dictionaries with `/AA` (additional actions)
//! - `/AcroForm` with `/AA` hooks
//! - Any stream whose dictionary contains `/JavaScript` or `/JS`
//!
//! - **Static text nodes** (passed through the context filter):
//! - Content-stream text operators (`Tj`, `TJ`, `'`, `"`)
//!
//! - **Metadata**:
//! - `/Info` dictionary (title, author, subject, producer, creator, dates)
use std::collections::BTreeMap;
use std::path::PathBuf;
use lopdf::{
content::{Content, Operation},
Document as LopdfDocument, Object, ObjectId,
};
use crate::core::config::Config;
use crate::core::types::{
Document, DocumentFormat, DocumentMetadata, ExecutableVector, Location, TextContext, TextNode,
VectorType,
};
use crate::CorbelError;
use crate::CorbelResult;
/// Concrete [`DocumentParser`] for PDF.
pub struct PdfParser;
impl super::DocumentParser for PdfParser {
fn parse(
bytes: &[u8],
_format: DocumentFormat,
source_path: Option<PathBuf>,
_config: &Config,
) -> CorbelResult<Document> {
let sha256 = crate::sha256_hex(bytes);
let size = bytes.len() as u64;
let doc = LopdfDocument::load_mem(bytes)
.map_err(|e| CorbelError::PdfParse(format!("lopdf load failed: {e}")))?;
let mut text_nodes = Vec::new();
let mut vectors = Vec::new();
let mut metadata = DocumentMetadata::default();
// 1. Extract /Info metadata.
extract_info_metadata(&doc, &mut metadata);
// 2. Walk every indirect object, classifying each one.
// We collect a snapshot of (ObjectId, Object) pairs first to
// avoid borrow issues during iteration.
let objects: Vec<(ObjectId, Object)> = doc
.objects
.iter()
.map(|(id, o)| (*id, o.clone()))
.collect();
for (obj_id, obj) in &objects {
// Look at dictionaries — these may contain /Action, /JavaScript,
// /EmbeddedFiles, /AA, etc.
if let Object::Dictionary(dict) = obj {
inspect_dictionary(dict, obj_id, &doc, &mut vectors);
}
// Look at streams directly — they may be /JavaScript streams
// or carry encoded payloads.
if let Object::Stream(stream) = obj {
let stream_dict = &stream.dict;
// Decompress the stream content for inspection.
// `decompressed_content()` applies the filter chain
// (FlateDecode, etc.) and returns the raw bytes.
let decompressed = stream
.decompressed_content()
.unwrap_or_else(|_| stream.content.clone());
// Detect /JavaScript stream
if let Ok(Object::Name(name)) = stream_dict.get(b"Type") {
if name.as_slice() == b"JavaScript" {
vectors.push(ExecutableVector {
location: Location::PdfStream {
id: obj_id.0,
filter: extract_filter_chain(stream_dict),
},
vector_type: VectorType::PdfJavaScript,
raw_payload: decompressed.clone(),
decoded_preview: String::from_utf8_lossy(&decompressed)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
continue;
}
}
// Detect stream with /JS in dictionary
if stream_dict.has(b"JS") || stream_dict.has(b"JavaScript") {
vectors.push(ExecutableVector {
location: Location::PdfStream {
id: obj_id.0,
filter: extract_filter_chain(stream_dict),
},
vector_type: VectorType::PdfJavaScript,
raw_payload: decompressed.clone(),
decoded_preview: String::from_utf8_lossy(&decompressed)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
}
}
}
// 3. Walk /Pages tree to extract text from page content streams.
extract_text_from_pages(&doc, &mut text_nodes);
Ok(Document {
format: DocumentFormat::Pdf,
source_path,
raw_bytes: bytes.to_vec(),
sha256,
size,
metadata,
text_nodes,
executable_vectors: vectors,
})
}
}
/// Extract `/Info`-dictionary metadata.
fn extract_info_metadata(doc: &LopdfDocument, meta: &mut DocumentMetadata) {
let Ok(info_ref) = doc.trailer.get(b"Info") else {
return;
};
let Ok((_info_id, info_obj)) = doc.dereference(info_ref) else {
return;
};
let Object::Dictionary(info_dict) = info_obj else {
return;
};
if let Some(s) = info_dict.get(b"Title").ok().and_then(|o| string_from_obj(o).ok()) {
meta.title = Some(s);
}
if let Some(s) = info_dict.get(b"Author").ok().and_then(|o| string_from_obj(o).ok()) {
meta.author = Some(s);
}
if let Some(s) = info_dict.get(b"Subject").ok().and_then(|o| string_from_obj(o).ok()) {
meta.subject = Some(s);
}
if let Some(s) = info_dict.get(b"Producer").ok().and_then(|o| string_from_obj(o).ok()) {
meta.producer = Some(s);
}
if let Some(s) = info_dict.get(b"Creator").ok().and_then(|o| string_from_obj(o).ok()) {
meta.creator = Some(s);
}
if let Some(s) = info_dict.get(b"CreationDate").ok().and_then(|o| string_from_obj(o).ok()) {
meta.created = Some(s);
}
if let Some(s) = info_dict.get(b"ModDate").ok().and_then(|o| string_from_obj(o).ok()) {
meta.modified = Some(s);
}
}
/// Helper: convert a `lopdf::Object` (String or Name) into a Rust `String`.
fn string_from_obj(obj: &Object) -> Result<String, ()> {
match obj {
Object::String(bytes, _) => Ok(String::from_utf8_lossy(bytes).into_owned()),
Object::Name(n) => Ok(String::from_utf8_lossy(n).into_owned()),
_ => Err(()),
}
}
/// Inspect a dictionary for executable vectors: /Action, /AA, /EmbeddedFiles, etc.
fn inspect_dictionary(
dict: &lopdf::Dictionary,
obj_id: &ObjectId,
doc: &LopdfDocument,
vectors: &mut Vec<ExecutableVector>,
) {
// /A or /AA (Additional Actions) → potentially a JavaScript / Launch action.
// We use a slice of byte-slices because each `b"..."` literal has a
// different length, so they can't go in the same fixed-size array.
let action_keys: &[&[u8]] = &[b"A", b"AA", b"OpenAction", b"Next"];
for action_key in action_keys {
if let Ok(action_ref) = dict.get(action_key) {
inspect_action(action_ref, obj_id, doc, vectors);
}
}
// /EmbeddedFiles → malicious attachments.
if let Ok(names_ref) = dict.get(b"Names") {
inspect_names_tree(names_ref, obj_id, doc, vectors);
}
if let Ok(ef_ref) = dict.get(b"EmbeddedFiles") {
inspect_names_tree(ef_ref, obj_id, doc, vectors);
}
// /URI actions inside /Annots
if let Ok(annots_ref) = dict.get(b"Annots") {
inspect_annots(annots_ref, obj_id, doc, vectors);
}
// /AcroForm with /AA
if let Ok(acroform_ref) = dict.get(b"AcroForm") {
if let Ok((_id, acroform_obj)) = doc.dereference(acroform_ref) {
if let Object::Dictionary(acro_dict) = acroform_obj {
if acro_dict.has(b"AA") || acro_dict.has(b"NeedAppearances") {
vectors.push(ExecutableVector {
location: Location::PdfObject { id: obj_id.0, gen: 0 },
vector_type: VectorType::PdfAcroForm,
raw_payload: Vec::new(),
decoded_preview: Some(format!(
"<AcroForm with AA={} NeedAppearances={}>",
acro_dict.has(b"AA"),
acro_dict.has(b"NeedAppearances")
)),
});
}
}
}
}
}
/// Inspect an `/Action` dictionary. May emit multiple vectors.
///
/// `obj_id` is the ID of the dictionary that *contains* the `/OpenAction`,
/// `/AA`, `/A`, or `/Next` entry pointing at this action — typically the
/// catalog or a page. We dereference `action_ref` to get the action's own
/// `ObjectId` and report *that* as the finding location, so that
/// `repackage_pdf` deletes the action object (not the container — deleting
/// the catalog would corrupt the entire PDF).
///
/// If the action is inline (a direct dictionary rather than an indirect
/// reference), `dereference` returns `None` for the ID and we fall back
/// to `obj_id` — the container — so the location is still well-defined.
fn inspect_action(action_ref: &Object, obj_id: &ObjectId, doc: &LopdfDocument, vectors: &mut Vec<ExecutableVector>) {
let Ok((action_id_opt, action_obj)) = doc.dereference(action_ref) else {
return;
};
let Object::Dictionary(action_dict) = action_obj else {
return;
};
let Ok(s_type_raw) = action_dict.get(b"S") else {
// No /S → not a recognizable action. Skip.
return;
};
let s_type: String = match s_type_raw {
Object::Name(n) => String::from_utf8_lossy(n).into_owned(),
Object::String(b, _) => String::from_utf8_lossy(b).into_owned(),
_ => return,
};
// Prefer the action's own ObjectId; fall back to the container's
// ID for inline (direct-dictionary) actions. We widen `gen` from
// `u16` (lopdf's ObjectId) to `u32` (Location's storage type).
let (loc_id, loc_gen) = action_id_opt
.map(|(id, gen)| (id, u32::from(gen)))
.unwrap_or((obj_id.0, u32::from(obj_id.1)));
let loc = Location::PdfObject { id: loc_id, gen: loc_gen };
match s_type.as_str() {
"JavaScript" | "JS" => {
let payload = action_dict
.get(b"JS")
.ok()
.and_then(|o| doc.dereference(o).ok())
.and_then(|(_, o)| match o {
Object::String(b, _) => Some(b.clone()),
_ => None,
})
.unwrap_or_default();
vectors.push(ExecutableVector {
location: loc,
vector_type: VectorType::PdfJavaScript,
raw_payload: payload.clone(),
decoded_preview: String::from_utf8_lossy(&payload)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
}
"Launch" => {
// /F (file), /Win, /Mac, /Unix parameters
let f_payload = action_dict
.get(b"F")
.ok()
.and_then(|o| doc.dereference(o).ok())
.and_then(|(_, o)| match o {
Object::String(b, _) => Some(b.clone()),
Object::Dictionary(d) => d
.get(b"F")
.ok()
.and_then(|f| doc.dereference(f).ok())
.and_then(|(_, f)| match f {
Object::String(b, _) => Some(b.clone()),
_ => None,
}),
_ => None,
})
.unwrap_or_default();
vectors.push(ExecutableVector {
location: loc,
vector_type: VectorType::PdfLaunch,
raw_payload: f_payload.clone(),
decoded_preview: String::from_utf8_lossy(&f_payload)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
}
"URI" => {
let uri = action_dict
.get(b"URI")
.ok()
.and_then(|o| doc.dereference(o).ok())
.and_then(|(_, o)| match o {
Object::String(b, _) => Some(b.clone()),
_ => None,
})
.unwrap_or_default();
vectors.push(ExecutableVector {
location: loc,
vector_type: VectorType::PdfUri,
raw_payload: uri.clone(),
decoded_preview: String::from_utf8_lossy(&uri)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
}
"GoToR" | "GoTo" => {
// Remote / local navigation — flag but lower priority.
vectors.push(ExecutableVector {
location: loc,
vector_type: VectorType::PdfGoToR,
raw_payload: Vec::new(),
decoded_preview: Some(format!("<GoTo/GoToR action in obj {loc_id}>")),
});
}
_ => {
// Unknown action type — emit as WidgetAction for visibility.
vectors.push(ExecutableVector {
location: loc,
vector_type: VectorType::PdfWidgetAction,
raw_payload: Vec::new(),
decoded_preview: Some(format!("<action type={s_type} in obj {loc_id}>")),
});
}
}
}
/// Inspect a Names tree (where /EmbeddedFiles live).
fn inspect_names_tree(names_ref: &Object, obj_id: &ObjectId, doc: &LopdfDocument, vectors: &mut Vec<ExecutableVector>) {
let Ok((_id, names_obj)) = doc.dereference(names_ref) else {
return;
};
let Object::Dictionary(names_dict) = names_obj else {
return;
};
// Names trees have /Kids (intermediate) or /Names (leaf with [name, file-spec, name, file-spec, ...]).
if let Ok(kids_ref) = names_dict.get(b"Kids") {
if let Ok((_id, kids_obj)) = doc.dereference(kids_ref) {
if let Object::Array(kids) = kids_obj {
for kid in kids {
inspect_names_tree(kid, obj_id, doc, vectors);
}
}
}
}
if let Ok(names_arr_ref) = names_dict.get(b"Names") {
if let Ok((_id, names_arr_obj)) = doc.dereference(names_arr_ref) {
if let Object::Array(arr) = names_arr_obj {
// Walk pairs: name (String), file-spec (Dictionary).
let mut idx = 0;
while idx + 1 < arr.len() {
if let Object::Dictionary(file_spec) = &arr[idx + 1] {
// File spec has /EF → embedded file stream.
if let Ok(ef_ref) = file_spec.get(b"EF") {
if let Ok((_id, ef_obj)) = doc.dereference(ef_ref) {
if let Object::Dictionary(ef_dict) = ef_obj {
// /F is the embedded file stream reference.
if let Ok(f_stream_ref) = ef_dict.get(b"F") {
if let Ok((_id, f_stream_obj)) = doc.dereference(f_stream_ref) {
if let Object::Stream(s) = f_stream_obj {
vectors.push(ExecutableVector {
location: Location::PdfObject { id: obj_id.0, gen: 0 },
vector_type: VectorType::PdfEmbeddedFile,
raw_payload: s.content.clone(),
decoded_preview: String::from_utf8_lossy(&s.content)
.chars()
.take(2048)
.collect::<String>()
.into(),
});
}
}
}
}
}
}
}
idx += 2;
}
}
}
}
}
/// Inspect /Annots array on a page.
fn inspect_annots(annots_ref: &Object, obj_id: &ObjectId, doc: &LopdfDocument, vectors: &mut Vec<ExecutableVector>) {
let Ok((_id, annots_obj)) = doc.dereference(annots_ref) else {
return;
};
let Object::Array(annots) = annots_obj else {
return;
};
for annot_ref in annots {
let Ok((_id, annot_obj)) = doc.dereference(annot_ref) else {
continue;
};
let Object::Dictionary(annot_dict) = annot_obj else {
continue;
};
// /Subtype /Widget with /AA → potential JS hook
if let Ok(Object::Name(subtype)) = annot_dict.get(b"Subtype") {
if subtype.as_slice() == b"Widget" && annot_dict.has(b"AA") {
vectors.push(ExecutableVector {
location: Location::PdfObject { id: obj_id.0, gen: 0 },
vector_type: VectorType::PdfWidgetAction,
raw_payload: Vec::new(),
decoded_preview: Some("<Widget with AA>".to_string()),
});
}
}
// /A → URI or Launch action
if let Ok(a_ref) = annot_dict.get(b"A") {
inspect_action(a_ref, obj_id, doc, vectors);
}
}
}
/// Walk /Pages tree and extract text from each page's content stream.
fn extract_text_from_pages(doc: &LopdfDocument, text_nodes: &mut Vec<TextNode>) {
// `get_pages()` returns `BTreeMap<u32, (u32, u16)>` — page number →
// (object_id, generation). We iterate and pull /Contents from each.
let pages: BTreeMap<u32, ObjectId> = doc.get_pages();
for (_page_num, page_obj_id) in &pages {
let Ok(page_obj) = doc.get_object(*page_obj_id) else {
continue;
};
let Object::Dictionary(page_dict) = page_obj else {
continue;
};
let Ok(contents_ref) = page_dict.get(b"Contents") else {
continue;
};
// Contents can be a single stream ref or an array of stream refs.
let contents_vec: Vec<Object> = match contents_ref {
Object::Reference(_) => vec![contents_ref.clone()],
Object::Array(arr) => arr.clone(),
_ => vec![],
};
for c_ref in contents_vec {
let Ok((_id, c_obj)) = doc.dereference(&c_ref) else {
continue;
};
let Object::Stream(stream) = c_obj else {
continue;
};
// Decompress the stream first. `stream.content` is the raw
// (possibly FlateDecode-compressed) bytes; `decompressed_content()`
// applies the filter chain and returns decompressed bytes.
// We fall back to the raw content if decompression fails.
let raw = stream
.decompressed_content()
.unwrap_or_else(|_| stream.content.clone());
let decoded = match Content::decode(&raw) {
Ok(c) => c,
Err(_) => continue,
};
extract_text_from_content(&decoded, page_obj_id, text_nodes);
}
}
}
/// Extract text operators (`Tj`, `TJ`, `'`, `"`) from a decoded content stream.
fn extract_text_from_content(
content: &Content,
page_id: &ObjectId,
text_nodes: &mut Vec<TextNode>,
) {
let mut current_text = String::new();
let mut text_started = false;
for op in &content.operations {
process_operation(op, page_id, &mut current_text, &mut text_started, text_nodes);
}
// Final flush.
if text_started && !current_text.is_empty() {
text_nodes.push(TextNode {
location: Location::PdfObject { id: page_id.0, gen: 0 },
context: TextContext::Paragraph,
content: current_text,
});
}
}
/// Process a single content-stream operation, flushing buffered text
/// when a new line / paragraph boundary is hit.
fn process_operation(
op: &Operation,
page_id: &ObjectId,
current_text: &mut String,
text_started: &mut bool,
text_nodes: &mut Vec<TextNode>,
) {
match op.operator.as_str() {
"Tj" => {
if let Some(Object::String(bytes, _)) = op.operands.first() {
current_text.push_str(&String::from_utf8_lossy(bytes));
*text_started = true;
}
}
"TJ" => {
// TJ operand is an array of [string, num, string, num, ...]
if let Some(Object::Array(arr)) = op.operands.first() {
for elem in arr {
if let Object::String(bytes, _) = elem {
current_text.push_str(&String::from_utf8_lossy(bytes));
}
}
*text_started = true;
}
}
"'" => {
// Move to next line and show string.
if let Some(Object::String(bytes, _)) = op.operands.first() {
if *text_started && !current_text.is_empty() {
text_nodes.push(TextNode {
location: Location::PdfObject { id: page_id.0, gen: 0 },
context: TextContext::Paragraph,
content: std::mem::take(current_text),
});
}
current_text.push_str(&String::from_utf8_lossy(bytes));
*text_started = true;
}
}
"\"" => {
// aw ac string — set word and char spacing, move to next line, show string.
if let Some(Object::String(bytes, _)) = op.operands.get(2) {
if *text_started && !current_text.is_empty() {
text_nodes.push(TextNode {
location: Location::PdfObject { id: page_id.0, gen: 0 },
context: TextContext::Paragraph,
content: std::mem::take(current_text),
});
}
current_text.push_str(&String::from_utf8_lossy(bytes));
*text_started = true;
}
}
"Td" | "TD" | "T*" | "Tm" => {
// Text positioning operators — treat as paragraph boundary.
if *text_started && !current_text.is_empty() {
text_nodes.push(TextNode {
location: Location::PdfObject { id: page_id.0, gen: 0 },
context: TextContext::Paragraph,
content: std::mem::take(current_text),
});
*text_started = false;
}
}
"BT" => {
current_text.clear();
*text_started = true;
}
"ET" => {
if *text_started && !current_text.is_empty() {
text_nodes.push(TextNode {
location: Location::PdfObject { id: page_id.0, gen: 0 },
context: TextContext::Paragraph,
content: std::mem::take(current_text),
});
}
*text_started = false;
}
_ => {
// Non-text operator — ignore.
}
}
}
/// Extract the /Filter chain from a stream dictionary as a Vec of names.
fn extract_filter_chain(dict: &lopdf::Dictionary) -> Vec<String> {
let Ok(filter_obj) = dict.get(b"Filter") else {
return Vec::new();
};
match filter_obj {
Object::Name(n) => vec![String::from_utf8_lossy(n).into_owned()],
Object::Array(arr) => arr
.iter()
.filter_map(|o| match o {
Object::Name(n) => Some(String::from_utf8_lossy(n).into_owned()),
_ => None,
})
.collect(),
_ => Vec::new(),
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parsers::DocumentParser;
/// Load a test fixture PDF by name from `tests/fixtures/`.
fn load_fixture(name: &str) -> Vec<u8> {
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures")
.join(name);
std::fs::read(&path)
.unwrap_or_else(|e| panic!("failed to read fixture {name}: {e}"))
}
#[test]
fn parses_benign_pdf() {
let bytes = load_fixture("benign.pdf");
let doc = PdfParser::parse(&bytes, DocumentFormat::Pdf, None, &Config::default()).unwrap();
assert_eq!(doc.format, DocumentFormat::Pdf);
assert_eq!(doc.metadata.title.as_deref(), Some("Benign Test PDF"));
assert_eq!(doc.metadata.author.as_deref(), Some("CorbelPurge Tests"));
// A benign PDF should produce NO executable vectors.
assert!(
doc.executable_vectors.is_empty(),
"benign PDF should have no executable vectors, got: {:?}",
doc.executable_vectors
);
// We should have extracted at least some text.
let combined: String = doc.text_nodes.iter().map(|n| n.content.as_str()).collect();
assert!(
combined.contains("benign PDF"),
"expected to find 'benign PDF' in extracted text, got: {combined}"
);
}
#[test]
fn parses_malicious_js_pdf_and_flags_vector() {
let bytes = load_fixture("malicious_js.pdf");
let doc = PdfParser::parse(&bytes, DocumentFormat::Pdf, None, &Config::default()).unwrap();
// The scanner should have picked up at least one PdfJavaScript vector.
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::PdfJavaScript),
"expected at least one PdfJavaScript vector, got: {:?}",
doc.executable_vectors
);
}
#[test]
fn parses_malicious_launch_pdf_and_flags_vector() {
let bytes = load_fixture("malicious_launch.pdf");
let doc = PdfParser::parse(&bytes, DocumentFormat::Pdf, None, &Config::default()).unwrap();
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::PdfLaunch),
"expected at least one PdfLaunch vector, got: {:?}",
doc.executable_vectors
);
}
#[test]
fn sha256_is_stable_for_pdf() {
let bytes = load_fixture("benign.pdf");
let doc1 = PdfParser::parse(&bytes, DocumentFormat::Pdf, None, &Config::default()).unwrap();
let doc2 = PdfParser::parse(&bytes, DocumentFormat::Pdf, None, &Config::default()).unwrap();
assert_eq!(doc1.sha256, doc2.sha256);
assert_eq!(doc1.sha256.len(), 64);
}
#[test]
fn rejects_non_pdf_bytes() {
let result = PdfParser::parse(b"not a pdf at all", DocumentFormat::Pdf, None, &Config::default());
assert!(result.is_err());
}
}

287
src/quarantine/extractor.rs Executable file
View File

@ -0,0 +1,287 @@
//! Payload carver: extracts malicious payloads out of a document's
//! raw structure so they can be quarantined independently.
//!
//! For each malicious finding in the [`ScanReport`], the extractor
//! locates the corresponding [`ExecutableVector`] (or text node) in
//! the document and carves its raw bytes into a standalone file.
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
use crate::core::config::Config;
use crate::core::types::{Document, Finding, Location, ScanReport, ThreatClassification};
/// One carved payload, ready to be written into the quarantine tarball.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ExtractedPayload {
/// Sanitized filename (e.g. `payload_pdf_42_js.bin`).
pub filename: String,
/// Full path on disk where the payload was written.
pub payload_path: PathBuf,
/// The raw carved bytes.
pub bytes: Vec<u8>,
/// The finding this payload was extracted from.
pub source_finding_index: usize,
}
impl ExtractedPayload {
/// Human-readable location string for use in .info files.
#[must_use]
pub fn location_str(&self) -> String {
// The location is stored implicitly via the filename's encoded
// path info. We derive it from the payload_path filename suffix.
self.filename
.strip_prefix("payload_")
.and_then(|rest| rest.split('_').nth(1))
.unwrap_or("")
.to_string()
}
}
/// Extract all malicious payloads from `document`, keyed by their
/// corresponding finding in `scan_report`.
///
/// Returns a list of [`ExtractedPayload`] entries — one per malicious
/// finding. Non-malicious findings produce no extracted payloads.
pub fn extract_payloads(
document: &Document,
scan_report: &ScanReport,
) -> Vec<ExtractedPayload> {
let config = Config::default();
extract_payloads_with_config(document, scan_report, &config)
}
/// Same as [`extract_payloads`] but accepts a config (used to pick the
/// output directory).
pub fn extract_payloads_with_config(
document: &Document,
scan_report: &ScanReport,
config: &Config,
) -> Vec<ExtractedPayload> {
let mut out = Vec::new();
for (idx, finding) in scan_report.findings.iter().enumerate() {
if !matches!(
finding.classification,
ThreatClassification::Malicious(_)
) {
continue;
}
// Locate the bytes to carve: prefer an executable vector at
// the same location; fall back to the finding's payload preview.
let bytes = locate_payload_bytes(document, finding);
let filename = sanitize_filename(document, finding, idx);
let payload_path = config.quarantine_dir.join(&filename);
out.push(ExtractedPayload {
filename,
payload_path,
bytes,
source_finding_index: idx,
});
}
out
}
/// Find the raw bytes that correspond to a given finding.
fn locate_payload_bytes(document: &Document, finding: &Finding) -> Vec<u8> {
// Try to match by location against executable vectors first.
for vector in &document.executable_vectors {
if locations_match(&vector.location, &finding.location) {
return vector.raw_payload.clone();
}
}
// Fall back to the finding's payload preview as bytes.
finding.payload_preview.as_bytes().to_vec()
}
/// Loose location equality — we don't require field-for-field match,
/// just that the two locations refer to the same document region.
fn locations_match(a: &Location, b: &Location) -> bool {
match (a, b) {
(
Location::PdfObject { id: ida, .. },
Location::PdfObject { id: idb, .. },
) => ida == idb,
(
Location::PdfStream { id: ida, .. },
Location::PdfStream { id: idb, .. },
) => ida == idb,
(
Location::PdfObject { id: ida, .. },
Location::PdfStream { id: idb, .. },
) |
(
Location::PdfStream { id: ida, .. },
Location::PdfObject { id: idb, .. },
) => ida == idb,
(
Location::EpubEntry { path: pa, .. },
Location::EpubEntry { path: pb, .. },
) => pa == pb,
(
Location::MarkdownLine { line: la, .. },
Location::MarkdownLine { line: lb, .. },
) => la == lb,
_ => false,
}
}
/// Build a safe filename for an extracted payload.
fn sanitize_filename(document: &Document, finding: &Finding, idx: usize) -> String {
let prefix = match document.format {
crate::core::types::DocumentFormat::Pdf => "pdf",
crate::core::types::DocumentFormat::Epub => "epub",
crate::core::types::DocumentFormat::Markdown => "md",
crate::core::types::DocumentFormat::Docx => "docx",
};
let suffix = match &finding.location {
Location::PdfObject { id, .. } => format!("obj{id}"),
Location::PdfStream { id, .. } => format!("stream{id}"),
Location::EpubEntry { path, .. } => {
// Use the basename of the path, sanitized.
let basename = path.rsplit('/').next().unwrap_or("entry");
let sanitized: String = basename
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
c
} else {
'_'
}
})
.collect();
sanitized
}
Location::MarkdownLine { line, .. } => format!("line{line}"),
};
let vt_suffix = finding
.vector_type
.map(|vt| format!("_{}", vt))
.unwrap_or_default();
format!("payload_{prefix}_{idx:03}_{suffix}{vt_suffix}.bin")
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::*;
use tempfile::tempdir;
fn make_doc_with_vector(vt: VectorType, payload: &[u8]) -> Document {
Document {
format: DocumentFormat::Pdf,
source_path: None,
raw_bytes: Vec::new(),
sha256: "abcdef".to_string(),
size: 0,
metadata: DocumentMetadata::default(),
text_nodes: Vec::new(),
executable_vectors: vec![ExecutableVector {
location: Location::PdfObject { id: 42, gen: 0 },
vector_type: vt,
raw_payload: payload.to_vec(),
decoded_preview: String::from_utf8_lossy(payload).to_string().into(),
}],
}
}
fn make_finding(loc: Location, vt: VectorType) -> Finding {
Finding {
classification: ThreatClassification::Malicious(
MaliciousType::ActiveJavaScriptInjection,
),
location: loc,
vector_type: Some(vt),
payload_preview: "alert('xss')".to_string(),
context_notes: "test".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}
}
#[test]
fn extracts_one_payload_per_malicious_finding() {
let doc = make_doc_with_vector(VectorType::PdfJavaScript, b"alert('xss')");
let scan = ScanReport {
source_sha256: "abcdef".to_string(),
format: DocumentFormat::Pdf,
scanned_at: "2026-01-01T00:00:00Z".to_string(),
findings: vec![make_finding(
Location::PdfObject { id: 42, gen: 0 },
VectorType::PdfJavaScript,
)],
text_nodes_scanned: 0,
vectors_scanned: 1,
};
let payloads = extract_payloads(&doc, &scan);
assert_eq!(payloads.len(), 1);
assert_eq!(payloads[0].bytes, b"alert('xss')");
assert!(payloads[0].filename.starts_with("payload_pdf_000_obj42_"));
}
#[test]
fn skips_non_malicious_findings() {
let doc = make_doc_with_vector(VectorType::PdfUri, b"https://example.com");
let scan = ScanReport {
source_sha256: "abcdef".to_string(),
format: DocumentFormat::Pdf,
scanned_at: "2026-01-01T00:00:00Z".to_string(),
findings: vec![Finding {
classification: ThreatClassification::Benign,
location: Location::PdfObject { id: 42, gen: 0 },
vector_type: Some(VectorType::PdfUri),
payload_preview: "https://example.com".to_string(),
context_notes: "benign".to_string(),
recommendation: Recommendation::Allow,
}],
text_nodes_scanned: 0,
vectors_scanned: 1,
};
let payloads = extract_payloads(&doc, &scan);
assert!(payloads.is_empty());
}
#[test]
fn sanitized_filename_is_safe() {
let doc = make_doc_with_vector(VectorType::PdfJavaScript, b"x");
let scan = ScanReport {
source_sha256: "x".to_string(),
format: DocumentFormat::Pdf,
scanned_at: "x".to_string(),
findings: vec![Finding {
classification: ThreatClassification::Malicious(
MaliciousType::ActiveJavaScriptInjection,
),
location: Location::EpubEntry {
path: "OEBPS/chapter 1.xhtml".to_string(),
anchor: None,
},
vector_type: Some(VectorType::EpubScript),
payload_preview: "x".to_string(),
context_notes: "x".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}],
text_nodes_scanned: 0,
vectors_scanned: 0,
};
let tmp = tempdir().unwrap();
let mut config = Config::default();
config.quarantine_dir = tmp.path().to_path_buf();
let payloads = extract_payloads_with_config(&doc, &scan, &config);
assert_eq!(payloads.len(), 1);
let name = &payloads[0].filename;
// Should contain no spaces.
assert!(!name.contains(' '), "filename should not contain spaces: {name}");
}
}

157
src/quarantine/hexdump.rs Executable file
View File

@ -0,0 +1,157 @@
//! Annotated hex dump generator for extracted payloads.
//!
//! Produces a side-by-side hex/ASCII dump similar to `xxd` or `hexdump -C`,
//! with a configurable bytes-per-line and offset column. Used by the
//! payload carving v2 feature to emit `.hex` files alongside `.bin` files.
/// Default number of hex bytes per line in the dump.
const BYTES_PER_LINE: usize = 16;
/// Generate an annotated hex dump of `data`.
///
/// Output format (similar to `xxd`):
///
/// ```text
/// 00000000: 4d5a 9000 0300 0000 0400 0000 ffff 0000 MZ..............
/// 00000010: b800 0000 0000 0000 4000 0000 0000 0000 ........@.......
/// ```
#[must_use]
pub fn hex_dump(data: &[u8]) -> String {
let mut out = String::new();
let mut offset = 0usize;
while offset < data.len() {
out.push_str(&format!("{:08x}: ", offset));
// Hex portion — split into groups of 2 bytes.
let chunk = &data[offset..std::cmp::min(offset + BYTES_PER_LINE, data.len())];
let mut hex = String::new();
let mut ascii = String::new();
for (i, &byte) in chunk.iter().enumerate() {
if i > 0 && i % 2 == 0 {
hex.push(' ');
}
hex.push_str(&format!("{:02x}", byte));
// Printable ASCII or '.'
if byte.is_ascii_graphic() || byte == b' ' {
ascii.push(byte as char);
} else {
ascii.push('.');
}
}
// Pad hex portion if the last line is short.
let full_hex_len = BYTES_PER_LINE * 2 + (BYTES_PER_LINE / 2 - 1);
if hex.len() < full_hex_len {
hex.push_str(&" ".repeat(full_hex_len - hex.len()));
}
out.push_str(&format!("{} {}\n", hex, ascii));
offset += BYTES_PER_LINE;
}
// Trailing offset line (total size).
out.push_str(&format!("{:08x}\n", data.len()));
out
}
/// Build a JSON info file for an extracted payload.
///
/// Returns a `serde_json::Value` suitable for writing to a `.info` file.
pub fn build_payload_info(
filename: &str,
finding_index: usize,
vector_type: Option<&str>,
location: &str,
classification: &str,
recommendation: &str,
context_notes: &str,
payload_sha256: &str,
payload_size: usize,
file_signature: Option<&str>,
cve_tag: Option<&str>,
) -> serde_json::Value {
let mut obj = serde_json::json!({
"filename": filename,
"finding_index": finding_index,
"payload_sha256": payload_sha256,
"payload_size_bytes": payload_size,
"location": location,
"classification": classification,
"recommendation": recommendation,
"context_notes": context_notes,
});
if let Some(vt) = vector_type {
obj["vector_type"] = serde_json::json!(vt);
}
if let Some(sig) = file_signature {
obj["file_signature"] = serde_json::json!(sig);
}
if let Some(cve) = cve_tag {
obj["cve_tag"] = serde_json::json!(cve);
}
obj
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hex_dump_basic() {
let data = b"MZ\x90\x00\x03\x00\x00\x00\x04\x00";
let dump = hex_dump(data);
assert!(dump.starts_with("00000000:"), "should start with offset");
assert!(dump.contains("4d5a"), "should contain 'MZ' as hex");
assert!(dump.contains("MZ"), "should contain 'MZ' in ASCII column");
}
#[test]
fn hex_dump_empty() {
let dump = hex_dump(b"");
assert_eq!(dump, "00000000\n");
}
#[test]
fn hex_dump_multiline() {
// 32 bytes → 2 full lines.
let data: Vec<u8> = (0..32).collect();
let dump = hex_dump(&data);
let lines: Vec<&str> = dump.lines().collect();
assert!(lines[0].starts_with("00000000:"));
assert!(lines[1].starts_with("00000010:"));
assert!(lines[2].starts_with("00000020"));
}
#[test]
fn hex_dump_non_printable_replaced_with_dot() {
let data = b"\x00\x01\x02ABC";
let dump = hex_dump(data);
assert!(dump.contains("...ABC"), "non-printable bytes should be dots");
}
#[test]
fn build_payload_info_structure() {
let info = build_payload_info(
"payload_pdf_000_obj42_pdf-javascript.bin",
0,
Some("pdf-javascript"),
"pdf:42 0 R",
"malicious:active-javascript-injection",
"quarantine-and-cleanse",
"test notes [CVE-2018-4990: Adobe Reader JavaScript RCE]",
"abcdef1234567890",
42,
None,
Some("[CVE-2018-4990: Adobe Reader JavaScript RCE]"),
);
assert_eq!(info["filename"], "payload_pdf_000_obj42_pdf-javascript.bin");
assert_eq!(info["payload_size_bytes"], 42);
assert_eq!(info["vector_type"], "pdf-javascript");
assert!(info["cve_tag"].as_str().unwrap().contains("CVE-2018-4990"));
}
}

249
src/quarantine/mod.rs Executable file
View File

@ -0,0 +1,249 @@
//! Quarantine: isolation & packaging manager.
//!
//! When the scanner identifies a malicious finding, the quarantine
//! module:
//!
//! 1. Carves the offending payload out of the document structure
//! ([`extractor`]).
//! 2. Generates a comprehensive forensic report ([`reporter`]).
//! 3. Bundles payload + report into a compressed `tar.gz` archive
//! at `<workspace>/corbel_quarantine/quarantine_<ts>_<sha256_prefix>.tar.gz`.
pub mod extractor;
pub mod hexdump;
pub mod reporter;
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
use crate::core::config::Config;
use crate::core::types::{Document, ScanReport};
use crate::CorbelResult;
/// The outcome of a successful quarantine operation.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct QuarantineOutcome {
/// Path to the generated `quarantine_<ts>_<sha256_prefix>.tar.gz` file.
pub tarball_path: PathBuf,
/// Path to the JSON forensic report.
pub json_report_path: PathBuf,
/// Path to the Markdown forensic report (if `emit_markdown_report` was set).
pub markdown_report_path: Option<PathBuf>,
/// Extracted payload files (one per malicious finding), keyed by
/// a sanitized filename derived from the finding location.
pub extracted_payload_paths: Vec<PathBuf>,
}
/// Top-level quarantine entrypoint. Called by the pipeline when the
/// scanner finds at least one malicious finding.
pub fn handle(
document: &Document,
scan_report: &ScanReport,
config: &Config,
) -> CorbelResult<QuarantineOutcome> {
// 1. Carve payloads out of the document.
//
// We pass `config` through so that `payload_path` is rooted at the
// caller's quarantine_dir. The no-config `extract_payloads` helper
// would fall back to a default Config whose `quarantine_dir` is the
// relative path `corbel_quarantine/` — fine in production (where
// CWD == workspace) but broken in tests (where the workspace is a
// tempdir) and any other embedded use case.
let extracted = extractor::extract_payloads_with_config(document, scan_report, config);
// 2. Generate reports.
let json_report = reporter::build_json_report(document, scan_report, &extracted);
let markdown_report = if config.emit_markdown_report {
Some(reporter::build_markdown_report(document, scan_report, &extracted))
} else {
None
};
// 3. Compose the quarantine tarball name.
let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S");
let sha_prefix = &document.sha256[..8.min(document.sha256.len())];
let tarball_name = format!("quarantine_{timestamp}_{sha_prefix}.tar.gz");
let tarball_path = config.quarantine_dir.join(&tarball_name);
let json_report_name = format!("report_{timestamp}_{sha_prefix}.json");
let json_report_path = config.quarantine_dir.join(&json_report_name);
let markdown_report_path = if markdown_report.is_some() {
Some(config.quarantine_dir.join(format!(
"report_{timestamp}_{sha_prefix}.md"
)))
} else {
None
};
// 4. Write the tarball.
write_tarball(
&tarball_path,
document,
&json_report,
markdown_report.as_deref(),
&extracted,
)?;
// 5. Write the JSON report as a standalone file (for easy programmatic access).
std::fs::write(&json_report_path, serde_json::to_string_pretty(&json_report)?)?;
// 6. Write the Markdown report as a standalone file too.
if let Some(md) = &markdown_report {
if let Some(md_path) = &markdown_report_path {
std::fs::write(md_path, md)?;
}
}
// 7. Write standalone payload carving v2 files (.hex + .info).
for payload in &extracted {
// .hex — annotated hex dump
let hex_content = hexdump::hex_dump(&payload.bytes);
let hex_path = payload.payload_path.with_extension("hex");
std::fs::write(&hex_path, hex_content)?;
// .info — JSON metadata
let finding = scan_report
.findings
.get(payload.source_finding_index);
let (classification_str, recommendation_str, context_notes, vector_type_str, cve_tag_str) =
if let Some(f) = finding {
(
match &f.classification {
crate::core::types::ThreatClassification::Benign => "benign".to_string(),
crate::core::types::ThreatClassification::EducationalContent => "educational".to_string(),
crate::core::types::ThreatClassification::Suspicious => "suspicious".to_string(),
crate::core::types::ThreatClassification::Malicious(t) => format!("malicious:{t}"),
},
match f.recommendation {
crate::core::types::Recommendation::Allow => "allow".to_string(),
crate::core::types::Recommendation::WhitelistAsEducational => "whitelist-as-educational".to_string(),
crate::core::types::Recommendation::Quarantine => "quarantine".to_string(),
crate::core::types::Recommendation::QuarantineAndCleanse => "quarantine-and-cleanse".to_string(),
},
f.context_notes.clone(),
f.vector_type.map(|v| v.to_string()),
extract_cve_tag_from_notes(&f.context_notes),
)
} else {
(String::new(), String::new(), String::new(), None, None)
};
let file_sig =
crate::scanner::signatures::match_file_signature(&payload.bytes)
.map(|s| s.to_string());
let info = hexdump::build_payload_info(
&payload.filename,
payload.source_finding_index,
vector_type_str.as_deref(),
&payload.location_str(),
&classification_str,
&recommendation_str,
&context_notes,
&crate::sha256_hex(&payload.bytes),
payload.bytes.len(),
file_sig.as_deref(),
cve_tag_str.as_deref(),
);
let info_path = payload.payload_path.with_extension("info");
std::fs::write(&info_path, serde_json::to_string_pretty(&info)?)?;
}
Ok(QuarantineOutcome {
tarball_path,
json_report_path,
markdown_report_path,
extracted_payload_paths: extracted
.iter()
.map(|p| p.payload_path.clone())
.collect(),
})
}
/// Extract a CVE tag (e.g. `[CVE-2017-11882: Equation Editor RCE]`)
/// from a finding's context_notes string, if present.
fn extract_cve_tag_from_notes(notes: &str) -> Option<String> {
let start = notes.find('[')?;
let end = notes.find(']')?;
if start < end {
Some(notes[start + 1..end].to_string())
} else {
None
}
}
/// Write the quarantine tarball containing:
/// - the forensic report (JSON + optional Markdown)
/// - each extracted payload
/// - a copy of the original file (for chain-of-custody)
fn write_tarball(
tarball_path: &std::path::Path,
document: &Document,
json_report: &serde_json::Value,
markdown_report: Option<&str>,
extracted: &[extractor::ExtractedPayload],
) -> CorbelResult<()> {
use std::io::Write;
let tar_gz = std::fs::File::create(tarball_path)?;
let enc = flate2::write::GzEncoder::new(tar_gz, flate2::Compression::default());
let mut tar = tar::Builder::new(enc);
// Add the original file under `original.<ext>`.
let ext = match document.format {
crate::core::types::DocumentFormat::Pdf => "pdf",
crate::core::types::DocumentFormat::Epub => "epub",
crate::core::types::DocumentFormat::Markdown => "md",
crate::core::types::DocumentFormat::Docx => "docx",
};
let original_name = format!("original.{ext}");
let mut header = tar::Header::new_gnu();
header.set_size(document.raw_bytes.len() as u64);
header.set_mode(0o644);
header.set_cksum();
tar.append_data(&mut header, &original_name, std::io::Cursor::new(&document.raw_bytes))?;
// Add the JSON report.
let json_bytes = serde_json::to_vec_pretty(json_report)?;
let mut header = tar::Header::new_gnu();
header.set_size(json_bytes.len() as u64);
header.set_mode(0o644);
header.set_cksum();
tar.append_data(&mut header, "report.json", std::io::Cursor::new(&json_bytes))?;
// Add the Markdown report if present.
if let Some(md) = markdown_report {
let md_bytes = md.as_bytes();
let mut header = tar::Header::new_gnu();
header.set_size(md_bytes.len() as u64);
header.set_mode(0o644);
header.set_cksum();
tar.append_data(&mut header, "report.md", std::io::Cursor::new(md_bytes))?;
}
// Add each extracted payload.
for payload in extracted {
let bytes = &payload.bytes;
let mut header = tar::Header::new_gnu();
header.set_size(bytes.len() as u64);
header.set_mode(0o644);
header.set_cksum();
// payload.payload_path is the full path under quarantine dir;
// we want just the filename inside the tarball.
let name = payload
.payload_path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("payload.bin");
tar.append_data(&mut header, name, std::io::Cursor::new(bytes))?;
}
// Finalize: flush the tar + gzip encoder.
let enc = tar.into_inner()?;
let mut file = enc.finish()?;
file.flush()?;
Ok(())
}

313
src/quarantine/reporter.rs Executable file
View File

@ -0,0 +1,313 @@
//! Forensic report generator.
//!
//! Produces both JSON and Markdown reports capturing file metadata,
//! vector locations, classifications, and context evaluation notes.
use serde_json::{json, Value};
use crate::core::types::{Document, Finding, MaliciousType, ScanReport, ThreatClassification};
use crate::quarantine::extractor::ExtractedPayload;
/// Build the JSON forensic report.
pub fn build_json_report(
document: &Document,
scan_report: &ScanReport,
extracted: &[ExtractedPayload],
) -> Value {
let findings: Vec<Value> = scan_report
.findings
.iter()
.enumerate()
.map(|(idx, f)| finding_to_json(idx, f, extracted))
.collect();
json!({
"schema_version": 1,
"scanned_at": scan_report.scanned_at,
"source": {
"format": scan_report.format.to_string(),
"sha256": scan_report.source_sha256,
"size_bytes": document.size,
"path": document.source_path.as_ref().map(|p| p.display().to_string()),
"metadata": metadata_to_json(&document.metadata),
},
"summary": {
"text_nodes_scanned": scan_report.text_nodes_scanned,
"executable_vectors_scanned": scan_report.vectors_scanned,
"total_findings": scan_report.findings.len(),
"malicious_findings": scan_report.malicious_count(),
"educational_findings": scan_report.educational_count(),
"overall_recommendation": recommendation_str(scan_report.overall_recommendation()),
},
"findings": findings,
})
}
/// Build the Markdown forensic report.
pub fn build_markdown_report(
document: &Document,
scan_report: &ScanReport,
extracted: &[ExtractedPayload],
) -> String {
let mut out = String::new();
out.push_str("# CorbelPurge Forensic Report\n\n");
out.push_str(&format!(
"Generated: {}\n\n",
scan_report.scanned_at
));
out.push_str("## Source\n\n");
out.push_str(&format!(
"- **Format**: {}\n",
scan_report.format
));
out.push_str(&format!("- **SHA-256**: `{}`\n", scan_report.source_sha256));
out.push_str(&format!("- **Size**: {} bytes\n", document.size));
if let Some(p) = &document.source_path {
out.push_str(&format!("- **Path**: `{}`\n", p.display()));
}
if let Some(t) = &document.metadata.title {
out.push_str(&format!("- **Title**: {t}\n"));
}
if let Some(a) = &document.metadata.author {
out.push_str(&format!("- **Author**: {a}\n"));
}
out.push('\n');
out.push_str("## Summary\n\n");
out.push_str(&format!(
"- **Text nodes scanned**: {}\n",
scan_report.text_nodes_scanned
));
out.push_str(&format!(
"- **Executable vectors scanned**: {}\n",
scan_report.vectors_scanned
));
out.push_str(&format!(
"- **Total findings**: {}\n",
scan_report.findings.len()
));
out.push_str(&format!(
"- **Malicious findings**: {}\n",
scan_report.malicious_count()
));
out.push_str(&format!(
"- **Educational findings**: {}\n",
scan_report.educational_count()
));
out.push_str(&format!(
"- **Overall recommendation**: {}\n\n",
recommendation_str(scan_report.overall_recommendation())
));
if scan_report.findings.is_empty() {
out.push_str("_No findings._\n");
return out;
}
out.push_str("## Findings\n\n");
for (idx, finding) in scan_report.findings.iter().enumerate() {
out.push_str(&format!("### Finding {idx}: {}\n\n", classification_str(&finding.classification)));
out.push_str(&format!("- **Location**: `{}`\n", finding.location));
if let Some(vt) = finding.vector_type {
out.push_str(&format!("- **Vector type**: `{vt}`\n"));
}
out.push_str(&format!(
"- **Recommendation**: {}\n",
recommendation_str(finding.recommendation)
));
out.push_str(&format!("- **Notes**: {}\n", finding.context_notes));
// If the finding's notes mention a CVE, look up the full
// description and include it in the report.
if let Some(cve_id) = extract_cve_id(&finding.context_notes) {
if let Some(desc) = crate::scanner::cve_tags::cve_description(cve_id) {
out.push_str(&format!("- **{} description**: {}\n", cve_id, desc));
}
}
out.push_str("- **Payload preview**:\n ```\n ");
// Indent each line of the preview.
let preview = finding.payload_preview.chars().take(512).collect::<String>();
let indented = preview.replace('\n', "\n ");
out.push_str(&indented);
out.push_str("\n ```\n\n");
}
if !extracted.is_empty() {
out.push_str("## Extracted Payloads\n\n");
for p in extracted {
out.push_str(&format!(
"- `{}` — {} bytes (from finding #{})\n",
p.filename,
p.bytes.len(),
p.source_finding_index
));
}
}
out
}
fn finding_to_json(idx: usize, f: &Finding, extracted: &[ExtractedPayload]) -> Value {
let matching_payload = extracted
.iter()
.find(|p| p.source_finding_index == idx)
.map(|p| json!({
"filename": p.filename,
"size_bytes": p.bytes.len(),
}));
json!({
"index": idx,
"classification": classification_str(&f.classification),
"location": f.location.to_string(),
"vector_type": f.vector_type.map(|v| v.to_string()),
"recommendation": recommendation_str(f.recommendation),
"context_notes": f.context_notes,
"payload_preview": f.payload_preview,
"extracted_payload": matching_payload,
})
}
fn metadata_to_json(meta: &crate::core::types::DocumentMetadata) -> Value {
json!({
"title": meta.title,
"author": meta.author,
"subject": meta.subject,
"producer": meta.producer,
"creator": meta.creator,
"created": meta.created,
"modified": meta.modified,
})
}
fn classification_str(c: &ThreatClassification) -> String {
match c {
ThreatClassification::Benign => "benign".to_string(),
ThreatClassification::EducationalContent => "educational".to_string(),
ThreatClassification::Suspicious => "suspicious".to_string(),
ThreatClassification::Malicious(t) => format!("malicious:{}", malicious_str(t)),
}
}
fn malicious_str(t: &MaliciousType) -> &'static str {
match t {
MaliciousType::ActiveJavaScriptInjection => "active-javascript-injection",
MaliciousType::LaunchAction => "launch-action",
MaliciousType::MaliciousEmbeddedFile => "malicious-embedded-file",
MaliciousType::ObfuscatedShellcode => "obfuscated-shellcode",
MaliciousType::SuspiciousUri => "suspicious-uri",
MaliciousType::EpubActiveScript => "epub-active-script",
MaliciousType::DocxActiveContent => "docx-active-content",
MaliciousType::Other => "other",
}
}
fn recommendation_str(r: crate::core::types::Recommendation) -> &'static str {
match r {
crate::core::types::Recommendation::Allow => "allow",
crate::core::types::Recommendation::WhitelistAsEducational => "whitelist-as-educational",
crate::core::types::Recommendation::Quarantine => "quarantine",
crate::core::types::Recommendation::QuarantineAndCleanse => "quarantine-and-cleanse",
}
}
/// Extract a CVE ID (e.g. `CVE-2017-11882`) from a string, if present.
///
/// Used to look up the full CVE description from the context_notes field
/// where the scanner appends `[CVE-XXXX-YYYYY: Name]` tags.
fn extract_cve_id(s: &str) -> Option<&str> {
// Look for "CVE-" followed by 4+ digits, a dash, 4+ digits, and a
// word boundary. We return a slice into the original string.
let lower = s.to_ascii_lowercase();
let cve_pos = lower.find("cve-")?;
let rest = &s[cve_pos..];
// Find the end of the CVE ID (next non-alphanumeric character
// other than dash).
let end = rest
.find(|c: char| !c.is_ascii_alphanumeric() && c != '-')
.unwrap_or(rest.len());
Some(&rest[..end])
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::*;
fn make_doc() -> Document {
Document {
format: DocumentFormat::Markdown,
source_path: Some(std::path::PathBuf::from("/tmp/test.md")),
raw_bytes: b"# test\n".to_vec(),
sha256: "abc123".to_string(),
size: 7,
metadata: DocumentMetadata {
title: Some("Test".to_string()),
..Default::default()
},
text_nodes: Vec::new(),
executable_vectors: Vec::new(),
}
}
fn make_scan() -> ScanReport {
ScanReport {
source_sha256: "abc123".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "2026-01-01T00:00:00Z".to_string(),
findings: vec![Finding {
classification: ThreatClassification::Malicious(
MaliciousType::ActiveJavaScriptInjection,
),
location: Location::MarkdownLine { line: 5, col: 0 },
vector_type: None,
payload_preview: "alert('xss')".to_string(),
context_notes: "test note".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}],
text_nodes_scanned: 10,
vectors_scanned: 1,
}
}
#[test]
fn json_report_has_expected_fields() {
let doc = make_doc();
let scan = make_scan();
let json = build_json_report(&doc, &scan, &[]);
assert_eq!(json["schema_version"], 1);
assert_eq!(json["source"]["sha256"], "abc123");
assert_eq!(json["summary"]["malicious_findings"], 1);
assert_eq!(json["findings"][0]["classification"], "malicious:active-javascript-injection");
}
#[test]
fn markdown_report_has_expected_sections() {
let doc = make_doc();
let scan = make_scan();
let md = build_markdown_report(&doc, &scan, &[]);
assert!(md.contains("# CorbelPurge Forensic Report"));
assert!(md.contains("## Source"));
assert!(md.contains("## Summary"));
assert!(md.contains("## Findings"));
assert!(md.contains("Finding 0"));
}
#[test]
fn empty_findings_produces_no_findings_section() {
let doc = make_doc();
let scan = ScanReport {
source_sha256: "abc".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![],
text_nodes_scanned: 0,
vectors_scanned: 0,
};
let md = build_markdown_report(&doc, &scan, &[]);
assert!(md.contains("_No findings._"));
}
}

349
src/scanner/context_filter.rs Executable file
View File

@ -0,0 +1,349 @@
//! Context filter: NLP / lexical checks for distinguishing security
//! literature from active malicious content.
//!
//! When the scanner encounters a suspicious signature inside a *static
//! text node* (paragraph, heading, code block), it asks the context
//! filter whether the surrounding context looks like:
//!
//! - **Educational content** (CVE writeups, exploit code samples in
//! defensive blog posts, textbook material) → whitelisted.
//! - **Weaponized content** (obfuscated shellcode, packed executables
//! in non-code contexts, embedded action triggers) → flagged.
//! - **Indeterminate** → emitted as `Suspicious` if configured.
use crate::core::config::Config;
use crate::core::types::{
Finding, MaliciousType, Recommendation, TextContext, TextNode, ThreatClassification,
};
/// Evaluate a single text node. Returns `Some(Finding)` if the node
/// contains a suspicious or malicious signature that survived the
/// context filter.
pub fn evaluate(node: &TextNode, config: &Config) -> Option<Finding> {
// Step 0: Check for weaponization indicators first — these are
// malicious regardless of whether a "signature" is present.
// Pure shellcode blobs, for example, contain no recognizable
// keyword but are still dangerous.
if has_weaponization_indicators(&node.content) {
let classification = if node.context == TextContext::CodeBlock
|| node.context == TextContext::CodeSpan
|| node.context == TextContext::BlockQuote
{
// Even weaponized-looking content inside a code block /
// blockquote is treated as educational — it's almost
// certainly a research writeup illustrating an attack.
ThreatClassification::EducationalContent
} else {
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
};
let recommendation = match classification {
ThreatClassification::EducationalContent => Recommendation::WhitelistAsEducational,
ThreatClassification::Malicious(_) => Recommendation::QuarantineAndCleanse,
_ => Recommendation::Allow,
};
let preview: String = node.content.chars().take(config.max_payload_preview_len).collect();
return Some(Finding {
classification,
location: node.location.clone(),
vector_type: None,
payload_preview: preview,
context_notes: format!(
"weaponization indicators detected in {} context",
context_name(node.context)
),
recommendation,
});
}
// Step 1: Does the node contain any suspicious signatures?
let signatures = find_signatures(&node.content);
if signatures.is_empty() {
return None;
}
// Step 2: What's the surrounding context?
let is_educational = looks_educational(node);
// Step 3: Decision matrix.
let classification = if is_educational {
ThreatClassification::EducationalContent
} else if node.context == TextContext::ExecutableHook {
// A suspicious signature inside an executable hook is malicious.
ThreatClassification::Malicious(MaliciousType::ActiveJavaScriptInjection)
} else if has_weaponization_indicators(&node.content) {
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
} else if config.emit_suspicious {
ThreatClassification::Suspicious
} else {
// Suspicious findings suppressed — drop it.
return None;
};
let recommendation = match classification {
ThreatClassification::Benign => Recommendation::Allow,
ThreatClassification::EducationalContent => Recommendation::WhitelistAsEducational,
ThreatClassification::Suspicious => Recommendation::Quarantine,
ThreatClassification::Malicious(_) => Recommendation::QuarantineAndCleanse,
};
let preview: String = node.content.chars().take(config.max_payload_preview_len).collect();
let notes = format!(
"found {} suspicious signature(s) [{}] in {} context{}",
signatures.len(),
signatures.join(", "),
context_name(node.context),
if is_educational { " (educational markers present)" } else { "" },
);
Some(Finding {
classification,
location: node.location.clone(),
vector_type: None,
payload_preview: preview,
context_notes: notes,
recommendation,
})
}
/// A signature is a string that *could* indicate malicious content
/// but is also commonly found in security literature.
const SUSPICIOUS_SIGNATURES: &[&str] = &[
"/JavaScript",
"/JS",
"/Launch",
"/EmbeddedFile",
"eval(",
"Function(",
"document.write",
"innerHTML",
"<script",
"<iframe",
"shellcode",
"exploit",
"payload",
"calc.exe",
"/bin/sh",
"powershell",
"cmd.exe",
"wget",
"curl http",
"rm -rf",
"Base64.decode",
"atob(",
"exec(",
];
/// Find all suspicious signatures present in `text`. Returns the
/// list of signatures found (deduplicated, in source order).
fn find_signatures(text: &str) -> Vec<&'static str> {
// Case-insensitive matching for some signatures, exact for others.
// For simplicity, we do case-sensitive matching first and let the
// context filter handle false positives.
let lower = text.to_ascii_lowercase();
SUSPICIOUS_SIGNATURES
.iter()
.copied()
.filter(|sig| {
// Signatures starting with `/` (PDF operators) are case-sensitive.
if sig.starts_with('/') {
text.contains(sig)
} else {
lower.contains(&sig.to_ascii_lowercase())
}
})
.collect()
}
/// Heuristic: does this node look like educational / literature content?
fn looks_educational(node: &TextNode) -> bool {
// Strong signal: it's inside a code block.
if matches!(node.context, TextContext::CodeBlock | TextContext::CodeSpan) {
return true;
}
// Strong signal: it's inside a block quote (often a cited excerpt).
if node.context == TextContext::BlockQuote {
return true;
}
// Weaker signal: the surrounding text contains academic / defensive
// markers (CVE IDs, "vulnerability", "remediation", etc.).
let lower = node.content.to_ascii_lowercase();
let academic_markers = [
"cve-",
"vulnerability",
"remediation",
"patch",
"mitigation",
"advisory",
"researchers",
"according to",
"for example",
"for instance",
"e.g.",
"i.e.",
"in this paper",
"we describe",
"we present",
"the following",
"the attack works as follows",
"proof of concept",
"shown below",
"listing",
];
if academic_markers.iter().any(|m| lower.contains(m)) {
return true;
}
false
}
/// Heuristic: does this node contain weaponization indicators?
///
/// Weaponization indicators are *not* found in educational content:
/// - Long runs of hex-encoded bytes (obfuscated shellcode)
/// - Base64 blobs of significant length (64+ chars, anywhere in text)
/// - Multiple concatenated shell-y commands with no explanatory text
fn has_weaponization_indicators(text: &str) -> bool {
// Long hex blob: 16+ consecutive \xNN tokens.
let hex_run = text.matches("\\x").count();
if hex_run >= 16 {
return true;
}
// Long base64 blob: 64+ consecutive base64 chars anywhere in text.
// (Not just whole lines — attackers like to embed base64 inline.)
let b64_re = regex::Regex::new(r"[A-Za-z0-9+/=]{64,}").unwrap();
if b64_re.is_match(text) {
return true;
}
// Multiple shell commands in a single non-code node.
let shell_indicators = ["rm -rf", "wget ", "curl ", "nc -", "/bin/sh", "powershell "];
let count = shell_indicators.iter().filter(|s| text.contains(*s)).count();
if count >= 2 {
return true;
}
false
}
/// Human-readable name for a context.
fn context_name(ctx: TextContext) -> &'static str {
match ctx {
TextContext::Paragraph => "paragraph",
TextContext::Heading => "heading",
TextContext::CodeBlock => "code-block",
TextContext::CodeSpan => "code-span",
TextContext::Hyperlink => "hyperlink",
TextContext::BlockQuote => "block-quote",
TextContext::Metadata => "metadata",
TextContext::ExecutableHook => "executable-hook",
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::Location;
fn make_node(ctx: TextContext, content: &str) -> TextNode {
TextNode {
location: Location::MarkdownLine { line: 1, col: 0 },
context: ctx,
content: content.to_string(),
}
}
#[test]
fn clean_text_produces_no_finding() {
let node = make_node(TextContext::Paragraph, "The quick brown fox jumps over the lazy dog.");
assert!(evaluate(&node, &Config::default()).is_none());
}
#[test]
fn cve_writeup_in_code_block_is_educational() {
let node = make_node(
TextContext::CodeBlock,
"eval('alert(1)') // PoC for CVE-2024-1234",
);
let finding = evaluate(&node, &Config::default()).unwrap();
assert_eq!(finding.classification, ThreatClassification::EducationalContent);
}
#[test]
fn pdf_javascript_in_executable_hook_is_malicious() {
let node = make_node(
TextContext::ExecutableHook,
"app.alert('hello'); /JavaScript",
);
let finding = evaluate(&node, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(_)
));
}
#[test]
fn obfuscated_shellcode_in_paragraph_is_malicious() {
let shellcode = "\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90\\x90";
let node = make_node(TextContext::Paragraph, shellcode);
let finding = evaluate(&node, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
));
}
#[test]
fn suspicious_in_paragraph_with_signature() {
let node = make_node(TextContext::Paragraph, "Run eval('alert(1)') now");
let finding = evaluate(&node, &Config::default()).unwrap();
assert_eq!(finding.classification, ThreatClassification::Suspicious);
}
#[test]
fn suspicious_can_be_suppressed() {
let node = make_node(TextContext::Paragraph, "Run eval('alert(1)') now");
let mut config = Config::default();
config.emit_suspicious = false;
assert!(evaluate(&node, &config).is_none());
}
#[test]
fn academic_text_with_signature_is_educational() {
let node = make_node(
TextContext::Paragraph,
"In this paper we describe the eval() vulnerability and its remediation.",
);
let finding = evaluate(&node, &Config::default()).unwrap();
assert_eq!(finding.classification, ThreatClassification::EducationalContent);
}
#[test]
fn long_base64_blob_is_weaponized() {
let b64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ABCDEFGH";
let node = make_node(TextContext::Paragraph, b64);
let finding = evaluate(&node, &Config::default());
// With the step-0 weaponization check, pure base64 (no signature)
// is now flagged as Malicious (ObfuscatedShellcode).
let finding = finding.expect("pure base64 blob should be flagged as weaponized");
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(_)
));
}
#[test]
fn long_base64_with_signature_is_malicious() {
let b64 = "eval(ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ABCDEFGH)";
let node = make_node(TextContext::Paragraph, b64);
let finding = evaluate(&node, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(_)
));
}
}

663
src/scanner/cve_tags.rs Executable file
View File

@ -0,0 +1,663 @@
//! CVE tagging: recognize known old exploits by their byte signatures
//! and tag findings with the CVE ID + a short description.
//!
//! This module is research-oriented — it helps a researcher studying
//! old exploits quickly identify "oh, this is CVE-2017-11882, the
//! Equation Editor vulnerability" rather than just seeing "embedded
//! OLE object with PE signature."
//!
//! ## What's in the table
//!
//! The table covers well-known old exploits (2010–2020 era) that a
//! researcher studying malware history would commonly encounter.
//! Each entry has:
//!
//! - A CVE ID
//! - A human-readable name
//! - A detection function (matches against the payload bytes / vector
//! type / context)
//! - A short description of what the exploit does
//!
//! ## How it's used
//!
//! The scanner calls [`match_cve`] after classifying a finding. If a
//! CVE matches, the CVE ID is appended to the finding's
//! `context_notes` field. The forensic report includes the full
//! description.
//!
//! ## External CVE databases
//!
//! Additional CVE entries can be loaded at runtime via
//! [`load_external_cve_db`]. Loaded entries are stored in a
//! process-wide static and checked by [`match_cve`] alongside the
//! built-in table.
use std::path::Path;
use std::sync::OnceLock;
use serde::Deserialize;
use crate::core::types::{ExecutableVector, Finding, MaliciousType, VectorType};
use crate::scanner::signatures::get_external_rules_storage;
use crate::CorbelResult;
/// A known CVE exploit entry.
#[derive(Debug, Clone)]
pub struct CveEntry {
/// The CVE identifier (e.g. `"CVE-2017-11882"`).
pub cve_id: &'static str,
/// Short human-readable name (e.g. `"Equation Editor RCE"`).
pub name: &'static str,
/// One-paragraph description of the exploit.
pub description: &'static str,
/// Detection function. Returns `true` if this CVE matches the
/// given vector + finding.
pub matches: fn(&ExecutableVector, &Finding) -> bool,
}
/// The full CVE table. Add new entries here as the corpus grows.
///
/// Order matters: more specific entries should come before more
/// general ones, since [`match_cve`] returns the first match.
pub static CVE_TABLE: &[CveEntry] = &[
// --- PDF exploits ---
CveEntry {
cve_id: "CVE-2010-0188",
name: "PDF LibTiff Buffer Overflow",
description: "Buffer overflow in the LibTiff library bundled with Adobe Reader \
9.x. Triggered by a crafted TIFF image embedded in a PDF. Allows \
remote code execution when the victim opens the PDF.",
matches: |v, _f| {
// Signature: PDF with an embedded TIFF image stream that
// has an unusually large StripByteCounts value. We can't
// easily detect the exact overflow, but a PDF embedded
// file with TIFF magic (II*\0 or MM*\0) is suspicious.
v.vector_type == VectorType::PdfEmbeddedFile
&& v.raw_payload.starts_with(b"II*\0")
},
},
CveEntry {
cve_id: "CVE-2018-4990",
name: "Adobe Reader JavaScript RCE",
description: "Heap-based buffer overflow in the JavaScript engine bundled with \
Adobe Reader. Triggered by a crafted /JavaScript action that \
manipulates the heap layout before triggering the overflow.",
matches: |v, f| {
// Signature: PDF JavaScript action with a payload longer
// than ~500 bytes (real exploit code is rarely short).
v.vector_type == VectorType::PdfJavaScript
&& v.raw_payload.len() > 500
&& matches!(
f.classification,
crate::core::types::ThreatClassification::Malicious(_)
)
},
},
// --- DOCX / Office exploits ---
CveEntry {
cve_id: "CVE-2017-11882",
name: "Equation Editor RCE",
description: "Remote code execution in Microsoft Office's Equation Editor \
(EQNEDT32.EXE). The exploit embeds an OLE object with a \
malformed Equation Editor stream that triggers a stack buffer \
overflow when the victim opens the document. One of the most \
widely exploited Office vulnerabilities of the 2017–2020 era.",
matches: |v, _f| {
// Signature: embedded OLE object whose class name mentions
// "Equation" or whose ProgID is "Equation.3".
if v.vector_type != VectorType::DocxEmbeddedObject {
return false;
}
let preview = v.decoded_preview.as_deref().unwrap_or("");
preview.contains("Equation") || preview.contains("EQNEDT32")
|| v.raw_payload.windows(8).any(|w| w == b"Equation")
},
},
CveEntry {
cve_id: "CVE-2018-0802",
name: "Equation Editor RCE (variant)",
description: "Variant of CVE-2017-11882. Uses a different Equation Editor \
COM object method to achieve code execution. Same vector \
(embedded OLE object) but different payload structure.",
matches: |v, f| {
// Same vector as CVE-2017-11882 but without the Equation
// string — fall back to "embedded OLE with PE signature"
// as a secondary match. We only match this if CVE-2017-11882
// didn't already match (which it would have if "Equation"
// was present).
v.vector_type == VectorType::DocxEmbeddedObject
&& v.raw_payload.starts_with(b"MZ")
&& !v.raw_payload.windows(8).any(|w| w == b"Equation")
&& matches!(
f.classification,
crate::core::types::ThreatClassification::Malicious(
MaliciousType::MaliciousEmbeddedFile
)
)
},
},
CveEntry {
cve_id: "CVE-2017-8570",
name: "Office RTF OLE Object RCE",
description: "Remote code execution via a malicious RTF file containing \
an OLE object with a crafted COM control. Affects all \
versions of Office that support RTF parsing.",
matches: |v, _f| {
// Signature: RTF content inside a DOCX embedded object that
// also contains the `\objdata` keyword (the RTF control word
// used to embed an OLE object). We require `\objdata` so that
// we don't shadow [`CVE-2012-0158`] (which matches any large
// RTF payload) or short non-exploit RTF fragments (which
// should match nothing).
//
// The check is case-insensitive because RTF control words
// are case-insensitive in practice.
v.vector_type == VectorType::DocxEmbeddedObject
&& v.raw_payload.starts_with(b"{\\rtf")
&& v.raw_payload
.windows(b"\\objdata".len())
.any(|w| w.eq_ignore_ascii_case(b"\\objdata"))
},
},
CveEntry {
cve_id: "CVE-2017-0199",
name: "Office OLE2Link RCE",
description: "Remote code execution via a malicious OLE2Link object in \
an Office document. The link points to a remote HTA file \
that gets downloaded and executed when the victim opens \
the document. Widely used in targeted attacks (2017–2019).",
matches: |v, _f| {
// Signature: DOCX external link with .hta or .html target,
// OR an embedded object whose payload references OLE2Link.
(v.vector_type == VectorType::DocxExternalLink
|| v.vector_type == VectorType::DocxEmbeddedObject)
&& v.decoded_preview
.as_deref()
.unwrap_or("")
.to_ascii_lowercase()
.contains(".hta")
},
},
// --- EPUB exploits (rare, but worth tagging) ---
CveEntry {
cve_id: "EPUB-SCRIPT-INJECTION",
name: "EPUB Active Script Injection",
description: "EPUB specification allows `<script>` tags inside XHTML \
content, but most readers either disable or sandbox them. \
A `<script>` tag in an EPUB is almost always malicious \
(or a poorly-designed interactive book). The payload is \
quarantined; the chapter text is preserved.",
matches: |v, f| {
v.vector_type == VectorType::EpubScript
&& matches!(
f.classification,
crate::core::types::ThreatClassification::Malicious(_)
)
},
},
// --- DOCX / Office exploits (expanded v0.3.0) ---
CveEntry {
cve_id: "CVE-2012-0158",
name: "Office RTF Stack Buffer Overflow",
description: "Stack-based buffer overflow in Microsoft Office when \
parsing a specially crafted RTF file. The vulnerability \
exists in the RTF parser and can be triggered by opening \
a malicious .rtf or a document that embeds RTF content. \
Widely exploited from 2012 through 2017.",
matches: |v, _f| {
// Signature: DOCX embedded object whose payload starts with \
// RTF magic and the raw payload is > 200 bytes (a full RTF \
// exploit, not just a minimal header).
v.vector_type == VectorType::DocxEmbeddedObject
&& v.raw_payload.starts_with(b"{\\rtf")
&& v.raw_payload.len() > 200
},
},
CveEntry {
cve_id: "CVE-2015-2545",
name: "Office OLE Packager Heap Corruption",
description: "Heap-based buffer corruption in Microsoft Office when \
handling OLE packager objects. Triggered by a crafted \
OLE2 link embedded in an Office document. Allows \
remote code execution when the victim opens the document. \
Exploited in targeted attacks throughout 2015-2017.",
matches: |v, f| {
// Signature: DOCX embedded OLE object with an OLE2 link \
// (contains \\x00\\x01Ole10Native) but NOT RTF magic (which \
// would be CVE-2012-0158 instead).
v.vector_type == VectorType::DocxEmbeddedObject
&& !v.raw_payload.starts_with(b"{\\rtf")
&& v.raw_payload.windows(12).any(|w| w == b"\x00\x01Ole10Native")
&& matches!(
f.classification,
crate::core::types::ThreatClassification::Malicious(
MaliciousType::MaliciousEmbeddedFile
)
)
},
},
CveEntry {
cve_id: "CVE-2021-40444",
name: "MSHTML Remote Code Execution",
description: "Remote code execution via the MSHTML component in \
Microsoft Office. A specially crafted Office document \
with an embedded ActiveX control can download and \
execute a payload via the MSHTML rendering engine. \
Widely exploited in the wild since September 2021.",
matches: |v, _f| {
// Signature: DOCX ActiveX control whose XML contains \
// classid references to MSHTML or contains an external \
// data reference.
v.vector_type == VectorType::DocxActiveX
&& v.decoded_preview
.as_deref()
.unwrap_or("")
.to_ascii_lowercase()
.contains("mshtml")
},
},
CveEntry {
cve_id: "CVE-2022-30190",
name: "Word SSTI via ms-msdt Protocol",
description: "Remote code execution via Microsoft Support Diagnostic \
Tool (MSDT) protocol URI in a Word document. A crafted \
document with an ms-msdt: URI can execute arbitrary \
commands when opened. Exploited in-the-wild in May \
2022 via weaponized documents distributed via email.",
matches: |v, _f| {
// Signature: DOCX external link pointing to ms-msdt: protocol.
v.vector_type == VectorType::DocxExternalLink
&& v.decoded_preview
.as_deref()
.unwrap_or("")
.to_ascii_lowercase()
.contains("ms-msdt:")
},
},
];
/// Try to match a finding against the CVE table.
///
/// Returns the first matching [`CveEntry`], or `None` if no CVE matches.
/// Checks both the built-in [`CVE_TABLE`] and any external entries
/// loaded via [`load_external_cve_db`].
/// Call this after the scanner has classified the finding.
#[must_use]
pub fn match_cve(vector: &ExecutableVector, finding: &Finding) -> Option<&'static CveEntry> {
// Built-in table first.
if let Some(entry) = CVE_TABLE.iter().find(|entry| (entry.matches)(vector, finding)) {
return Some(entry);
}
// External CVE database.
if let (Some(entries), Some(specs)) = (
EXTERNAL_CVE_ENTRIES.get(),
EXTERNAL_CVE_SPECS.get(),
) {
for (entry, spec) in entries.iter().zip(specs.iter()) {
if external_cve_matches(spec, vector, finding) {
return Some(entry);
}
}
}
None
}
/// Format a CVE tag for inclusion in the finding's context notes.
///
/// Returns a string like `"[CVE-2017-11882: Equation Editor RCE]"`
/// or an empty string if no CVE matched.
#[must_use]
pub fn cve_tag(vector: &ExecutableVector, finding: &Finding) -> String {
match match_cve(vector, finding) {
Some(entry) => format!("[{}: {}]", entry.cve_id, entry.name),
None => String::new(),
}
}
// ────────────────────────────────────────────────────────────────
// External CVE database support
// ────────────────────────────────────────────────────────────────
/// A single external CVE entry deserialized from a JSON threat-intel
/// database file.
#[derive(Debug, Clone, Deserialize)]
pub struct ExternalCveJson {
/// The CVE identifier (e.g. `"CVE-2023-XXXXX"`).
pub cve_id: String,
/// Short human-readable name.
pub name: String,
/// One-paragraph description of the exploit.
pub description: String,
/// How to match this CVE. One of `"keyword-list"`, `"brand-list"`,
/// `"signature-list"`, `"shellcode-list"` (optionally prefixed
/// with `"type:"`).
#[serde(rename = "matches_fn")]
pub matches_fn: String,
/// The key used during matching:
/// - For `"keyword-list"`: checked against `finding.context_notes`.
/// - For `"brand-list"`: checked against the `decoded_preview`.
/// - For `"signature-list"`/`"shellcode-list"`: the name of an
/// external rule whose bytes are checked against `raw_payload`.
pub payload_key: String,
}
/// Match specification for an external CVE entry.
///
/// Stored alongside the [`CveEntry`] in a parallel static so that
/// [`match_cve`] can perform the correct matching logic.
struct ExternalCveMatchSpec {
/// Normalized match type (e.g. `"keyword-list"`).
matches_type: String,
/// Key used during matching (meaning depends on `matches_type`).
payload_key: String,
}
/// Placeholder match function for external [`CveEntry`] instances.
///
/// This is never called directly — [`match_cve`] performs the
/// actual matching using [`ExternalCveMatchSpec`] instead.
fn external_cve_placeholder_matches(_v: &ExecutableVector, _f: &Finding) -> bool {
false
}
/// Process-wide storage for externally loaded CVE entries.
static EXTERNAL_CVE_ENTRIES: OnceLock<Vec<CveEntry>> = OnceLock::new();
/// Process-wide storage for external CVE match specifications.
static EXTERNAL_CVE_SPECS: OnceLock<Vec<ExternalCveMatchSpec>> = OnceLock::new();
/// Match an external CVE specification against a vector + finding.
fn external_cve_matches(
spec: &ExternalCveMatchSpec,
vector: &ExecutableVector,
finding: &Finding,
) -> bool {
match spec.matches_type.as_str() {
"keyword-list" => finding.context_notes.contains(&spec.payload_key),
"brand-list" => vector
.decoded_preview
.as_deref()
.unwrap_or("")
.contains(&spec.payload_key),
"signature-list" => {
if let Some(ext_rules) = get_external_rules_storage() {
for (offset, bytes, _name) in &ext_rules.signatures {
if vector.raw_payload.len() >= *offset + bytes.len() {
if &vector.raw_payload[*offset..*offset + bytes.len()] == bytes.as_slice() {
return true;
}
}
}
}
false
}
"shellcode-list" => {
if let Some(ext_rules) = get_external_rules_storage() {
for pattern in &ext_rules.shellcode {
if vector
.raw_payload
.windows(pattern.len())
.any(|w| w == pattern.as_slice())
{
return true;
}
}
}
false
}
_ => false,
}
}
/// Load an external CVE signature database from a JSON file.
///
/// The file must contain a JSON array of [`ExternalCveJson`] objects.
/// Entries are converted to [`CveEntry`] instances and stored in a
/// process-wide static. Subsequent calls to [`match_cve`] will
/// check both the built-in [`CVE_TABLE`] and the external entries.
///
/// Returns references to the newly created [`CveEntry`] instances.
///
/// # Errors
///
/// Returns [`CorbelError::Io`] if the file cannot be read, or
/// [`CorbelError::Serde`] if the JSON is malformed.
pub fn load_external_cve_db(path: &Path) -> CorbelResult<Vec<&'static CveEntry>> {
let data = std::fs::read_to_string(path)?;
let json_entries: Vec<ExternalCveJson> = serde_json::from_str(&data)?;
let mut cve_entries = Vec::with_capacity(json_entries.len());
let mut specs = Vec::with_capacity(json_entries.len());
for entry in json_entries {
// Normalize the matches_fn field (strip optional "type:" prefix).
let matches_type = entry
.matches_fn
.strip_prefix("type:")
.unwrap_or(&entry.matches_fn)
.to_string();
// Leak the strings to obtain `'static` references.
// This is intentional: the data lives for the remainder of
// the process and is only populated once.
let cve_id: &'static str = Box::leak(entry.cve_id.into_boxed_str());
let name: &'static str = Box::leak(entry.name.into_boxed_str());
let description: &'static str = Box::leak(entry.description.into_boxed_str());
cve_entries.push(CveEntry {
cve_id,
name,
description,
matches: external_cve_placeholder_matches,
});
specs.push(ExternalCveMatchSpec {
matches_type,
payload_key: entry.payload_key,
});
}
let _ = EXTERNAL_CVE_ENTRIES.set(cve_entries);
let _ = EXTERNAL_CVE_SPECS.set(specs);
// SAFETY: we just set the lock above, so get() is guaranteed Some.
let entries = EXTERNAL_CVE_ENTRIES.get().unwrap();
let refs: Vec<&'static CveEntry> = entries.iter().collect();
Ok(refs)
}
/// Get the full description for a CVE ID, if known.
///
/// Checks both the built-in [`CVE_TABLE`] and any external entries
/// loaded via [`load_external_cve_db`].
#[must_use]
pub fn cve_description(cve_id: &str) -> Option<&'static str> {
if let Some(entry) = CVE_TABLE.iter().find(|e| e.cve_id == cve_id) {
return Some(entry.description);
}
if let Some(entries) = EXTERNAL_CVE_ENTRIES.get() {
if let Some(entry) = entries.iter().find(|e| e.cve_id == cve_id) {
return Some(entry.description);
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::*;
fn make_vector(vt: VectorType, payload: &[u8]) -> ExecutableVector {
ExecutableVector {
location: Location::PdfObject { id: 1, gen: 0 },
vector_type: vt,
raw_payload: payload.to_vec(),
decoded_preview: String::from_utf8_lossy(payload).to_string().into(),
}
}
fn make_malicious_finding(malicious_type: MaliciousType) -> Finding {
Finding {
classification: ThreatClassification::Malicious(malicious_type),
location: Location::PdfObject { id: 1, gen: 0 },
vector_type: None,
payload_preview: "x".to_string(),
context_notes: "x".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}
}
#[test]
fn detects_cve_2017_11882_equation_editor() {
let payload = b"Equation Editor OLE stream...\x00\x01\x02";
let v = make_vector(VectorType::DocxEmbeddedObject, payload);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2017-11882");
assert!(cve.description.contains("Equation Editor"));
}
#[test]
fn detects_cve_2018_0802_variant() {
// PE payload without "Equation" string → CVE-2018-0802.
let payload = b"MZ\x90\x00\x03\x00\x00\x00rest of PE";
let v = make_vector(VectorType::DocxEmbeddedObject, payload);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2018-0802");
}
#[test]
fn detects_cve_2017_8570_rtf() {
let payload = b"{\\rtf1\\ansi\\objdata ...}";
let v = make_vector(VectorType::DocxEmbeddedObject, payload);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2017-8570");
}
#[test]
fn detects_cve_2017_0199_hta_link() {
let v = ExecutableVector {
location: Location::EpubEntry {
path: "word/_rels/document.xml.rels".to_string(),
anchor: Some("rId1".to_string()),
},
vector_type: VectorType::DocxExternalLink,
raw_payload: b"https://evil.example.com/payload.hta".to_vec(),
decoded_preview: Some("target=https://evil.example.com/payload.hta".to_string()),
};
let f = make_malicious_finding(MaliciousType::SuspiciousUri);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2017-0199");
}
#[test]
fn detects_cve_2018_4990_long_js() {
// PDF JavaScript with >500 bytes → CVE-2018-4990.
let payload = vec![b'a'; 600];
let v = make_vector(VectorType::PdfJavaScript, &payload);
let f = make_malicious_finding(MaliciousType::ActiveJavaScriptInjection);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2018-4990");
}
#[test]
fn detects_epub_script_injection() {
let v = make_vector(VectorType::EpubScript, b"alert('xss')");
let f = make_malicious_finding(MaliciousType::EpubActiveScript);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "EPUB-SCRIPT-INJECTION");
}
#[test]
fn no_cve_for_clean_content() {
// A benign PDF URI shouldn't match any CVE.
let v = make_vector(VectorType::PdfUri, b"https://example.com");
let f = Finding {
classification: ThreatClassification::Benign,
location: Location::PdfObject { id: 1, gen: 0 },
vector_type: Some(VectorType::PdfUri),
payload_preview: "https://example.com".to_string(),
context_notes: "benign".to_string(),
recommendation: Recommendation::Allow,
};
assert!(match_cve(&v, &f).is_none());
}
#[test]
fn cve_tag_formats_correctly() {
let payload = b"Equation Editor stream";
let v = make_vector(VectorType::DocxEmbeddedObject, payload);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
let tag = cve_tag(&v, &f);
assert!(tag.contains("CVE-2017-11882"));
assert!(tag.contains("Equation Editor RCE"));
}
#[test]
fn cve_description_lookup() {
assert!(cve_description("CVE-2017-11882").is_some());
assert!(cve_description("CVE-NOT-REAL").is_none());
}
#[test]
fn detects_cve_2012_0158_rtf() {
// RTF payload > 200 bytes in a DOCX embedded object → CVE-2012-0158.
let mut rtf = b"{\\rtf1\\ansi{} ".to_vec();
rtf.extend(vec![b'A'; 300]); // pad to > 200 bytes
let v = make_vector(VectorType::DocxEmbeddedObject, &rtf);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2012-0158");
assert!(cve.description.contains("RTF"));
}
#[test]
fn cve_2012_0158_requires_minimum_size() {
// RTF payload under 200 bytes should NOT match CVE-2012-0158.
let short_rtf = b"{\\rtf1\\ansi short}";
let v = make_vector(VectorType::DocxEmbeddedObject, short_rtf);
let f = make_malicious_finding(MaliciousType::MaliciousEmbeddedFile);
assert!(match_cve(&v, &f).is_none());
}
#[test]
fn detects_cve_2022_30190_msmsdt() {
let v = ExecutableVector {
location: Location::EpubEntry {
path: "word/_rels/document.xml.rels".to_string(),
anchor: Some("rId1".to_string()),
},
vector_type: VectorType::DocxExternalLink,
raw_payload: b"ms-msdt:/id PCW Diagnostic".to_vec(),
decoded_preview: Some("target=ms-msdt:/id PCW Diagnostic".to_string()),
};
let f = make_malicious_finding(MaliciousType::SuspiciousUri);
let cve = match_cve(&v, &f).unwrap();
assert_eq!(cve.cve_id, "CVE-2022-30190");
assert!(cve.description.contains("ms-msdt"));
}
#[test]
fn cve_table_has_correct_order() {
// CVE-2017-11882 must come before CVE-2018-0802, because
// CVE-2018-0802 explicitly checks that CVE-2017-11882 didn't
// match first (by checking for the absence of "Equation").
let table = CVE_TABLE;
let mut found_11882 = false;
for entry in table {
if entry.cve_id == "CVE-2017-11882" {
found_11882 = true;
}
if entry.cve_id == "CVE-2018-0802" {
assert!(
found_11882,
"CVE-2018-0802 must appear after CVE-2017-11882 in the table"
);
}
}
}
}

754
src/scanner/heuristics.rs Executable file
View File

@ -0,0 +1,754 @@
//! Heuristics: signature & anomaly detection rules for executable vectors.
//!
//! Each executable vector ([`crate::core::types::ExecutableVector`]) is
//! untrusted-by-default. The heuristics here classify them into
//! [`MaliciousType`](crate::core::types::MaliciousType) buckets.
use crate::core::config::Config;
use crate::core::types::{
ExecutableVector, Finding, MaliciousType, Recommendation, ThreatClassification, VectorType,
};
/// Inspect a single executable vector and produce a [`Finding`] if it
/// is flagged as suspicious or malicious.
///
/// Returns `None` if the vector is considered safe (rare — most vectors
/// produce at least a `Suspicious` finding).
pub fn inspect_vector(vector: &ExecutableVector, config: &Config) -> Option<Finding> {
let classification = classify_vector(vector, config);
let recommendation = recommendation_for(&classification);
// Drop Suspicious findings if the operator disabled them.
if !config.emit_suspicious && matches!(classification, ThreatClassification::Suspicious) {
return None;
}
// Drop Benign findings (rare, but possible for whitelisted URI schemes).
if matches!(classification, ThreatClassification::Benign) {
return None;
}
let payload_preview = vector
.decoded_preview
.clone()
.unwrap_or_else(|| {
String::from_utf8_lossy(&vector.raw_payload)
.chars()
.take(config.max_payload_preview_len)
.collect()
});
Some(Finding {
classification: classification.clone(),
location: vector.location.clone(),
vector_type: Some(vector.vector_type),
payload_preview,
context_notes: notes_for(vector, &classification),
recommendation,
})
}
/// Classify a vector into a [`ThreatClassification`].
fn classify_vector(vector: &ExecutableVector, config: &Config) -> ThreatClassification {
match vector.vector_type {
// PDF JavaScript in an executable hook is always malicious.
VectorType::PdfJavaScript => ThreatClassification::Malicious(
MaliciousType::ActiveJavaScriptInjection,
),
// PDF /Launch is always malicious.
VectorType::PdfLaunch => ThreatClassification::Malicious(MaliciousType::LaunchAction),
// PDF embedded file — inspect for executable / high-risk signatures.
VectorType::PdfEmbeddedFile => {
if let Some(_name) = super::signatures::match_file_signature(&vector.raw_payload) {
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
} else if has_obfuscated_shellcode(&vector.raw_payload) {
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
} else {
ThreatClassification::Suspicious
}
}
// PDF /URI — check scheme against allow-list and run phishing heuristics.
VectorType::PdfUri => classify_uri(vector, config),
// PDF GoToR — generally benign navigation but flag for review.
VectorType::PdfGoToR => ThreatClassification::Suspicious,
// PDF widget actions and AcroForm hooks — suspicious by default.
VectorType::PdfWidgetAction | VectorType::PdfAcroForm => ThreatClassification::Suspicious,
// EPUB <script> is malicious (EPUBs shouldn't ship active scripts).
VectorType::EpubScript => ThreatClassification::Malicious(MaliciousType::EpubActiveScript),
// EPUB external resource — check URI scheme.
VectorType::EpubExternalResource => classify_uri(vector, config),
// EPUB <object>/<embed> — suspicious.
VectorType::EpubObject => ThreatClassification::Suspicious,
// Markdown hyperlink — check URI scheme.
VectorType::MarkdownHyperlink => classify_uri(vector, config),
// DOCX VBA macros are always malicious.
VectorType::DocxMacro => ThreatClassification::Malicious(
MaliciousType::DocxActiveContent,
),
// DOCX embedded OLE objects — inspect for executable signatures.
VectorType::DocxEmbeddedObject => {
if super::signatures::match_file_signature(&vector.raw_payload).is_some() {
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
} else {
// Embedded object without a known signature — still
// suspicious (DOCX embedded objects are rare and
// almost always carry some form of active content).
ThreatClassification::Suspicious
}
}
// DOCX ActiveX controls — always suspicious.
VectorType::DocxActiveX => ThreatClassification::Suspicious,
// DOCX external hyperlinks — check URI scheme + phishing heuristics.
VectorType::DocxExternalLink => classify_uri(vector, config),
// Unknown payload — apply executable-signature + shellcode heuristics.
VectorType::UnknownPayload => {
if super::signatures::match_file_signature(&vector.raw_payload).is_some() {
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
} else if has_obfuscated_shellcode(&vector.raw_payload) {
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
} else {
ThreatClassification::Suspicious
}
}
}
}
/// Classify a URI-bearing vector by checking its scheme against the
/// configured allow-list and looking for known phishing patterns.
fn classify_uri(vector: &ExecutableVector, config: &Config) -> ThreatClassification {
let uri = vector
.decoded_preview
.as_deref()
.or_else(|| std::str::from_utf8(&vector.raw_payload).ok())
.unwrap_or("");
let scheme = uri
.split("://")
.next()
.map(|s| s.to_ascii_lowercase())
.unwrap_or_default();
// Disallowed scheme (javascript:, data:, file:, vbscript:, etc.) → malicious.
if !scheme.is_empty() && !config.allowed_uri_schemes.iter().any(|s| s == &scheme) {
return ThreatClassification::Malicious(MaliciousType::SuspiciousUri);
}
// If the scheme is in the allow-list, check for phishing patterns.
if config.allowed_uri_schemes.iter().any(|s| s == &scheme) {
if let Some(reason) = looks_like_phishing(uri) {
// Strong signals (homograph brand, IP host, credential URL)
// are Malicious; weaker signals (shortener, suspicious keyword,
// phishing TLD) are Suspicious.
return if reason.is_strong() {
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
} else if config.emit_suspicious {
ThreatClassification::Suspicious
} else {
ThreatClassification::Benign
};
}
return ThreatClassification::Benign;
}
// No scheme at all — likely a relative URL, treat as benign.
ThreatClassification::Benign
}
/// Reason a URI was flagged as phishing.
///
/// We split into "strong" (homograph, IP host, credential URL) and
/// "weak" (shortener, suspicious keyword, phishing TLD) signals so
/// the caller can decide whether to escalate to `Malicious` or just
/// emit a `Suspicious` finding.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PhishingReason {
/// Host is a raw IP address (no DNS resolution required — classic phishing).
IpHost,
/// URL contains `user:pass@host` — almost always credential harvesting.
CredentialUrl,
/// URL mentions a known brand but the host doesn't match the brand's
/// canonical domain (homograph bait like `micros0ft.com`).
BrandHomograph,
/// URL uses a known URL shortener (hides the real destination).
Shortener,
/// URL path/host contains a known suspicious keyword (`login`, `verify`, ...).
SuspiciousKeyword,
/// URL host ends with a known phishing TLD.
PhishingTld,
}
impl PhishingReason {
/// Strong signals warrant a `Malicious` classification.
#[must_use]
pub fn is_strong(self) -> bool {
matches!(
self,
Self::IpHost | Self::CredentialUrl | Self::BrandHomograph
)
}
}
impl std::fmt::Display for PhishingReason {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let s = match self {
Self::IpHost => "ip-host",
Self::CredentialUrl => "credential-url",
Self::BrandHomograph => "brand-homograph",
Self::Shortener => "url-shortener",
Self::SuspiciousKeyword => "suspicious-keyword",
Self::PhishingTld => "phishing-tld",
};
f.write_str(s)
}
}
/// Heuristic: does this URL look like a phishing pattern?
///
/// Returns the first matching [`PhishingReason`] (priority order:
/// strong signals first, then weak signals).
pub fn looks_like_phishing(uri: &str) -> Option<PhishingReason> {
let lower = uri.to_ascii_lowercase();
// Extract host (after scheme://, before path/query/fragment, sans port).
let host = lower
.split("://")
.nth(1)
.unwrap_or(&lower)
.split('/')
.next()
.unwrap_or("")
.split(':')
.next()
.unwrap_or("");
// --- STRONG SIGNALS ---
// IP-address host (very common in phishing).
let parts: Vec<_> = host.split('.').collect();
if parts.len() == 4 && parts.iter().all(|p| p.parse::<u8>().is_ok()) {
return Some(PhishingReason::IpHost);
}
// Credential-bearing URLs (user:pass@host).
if lower.contains("://") {
let after_scheme = lower.split("://").nth(1).unwrap_or("");
if after_scheme.contains('@') {
return Some(PhishingReason::CredentialUrl);
}
}
// Brand homograph bait (micros0ft.com, paypa1.com, ...).
if super::signatures::match_phishing_brand(uri).is_some() {
return Some(PhishingReason::BrandHomograph);
}
// --- WEAK SIGNALS ---
// URL shortener — destination is hidden.
if super::signatures::is_url_shortener(host) {
return Some(PhishingReason::Shortener);
}
// Suspicious keyword in the URL (login, verify, account, ...).
if super::signatures::match_suspicious_keyword(uri).is_some() {
return Some(PhishingReason::SuspiciousKeyword);
}
// Known phishing TLD.
if super::signatures::has_phishing_tld(uri).is_some() {
return Some(PhishingReason::PhishingTld);
}
None
}
/// Heuristic: does this byte sequence look like a Windows / Linux / Mac
/// executable, or another high-risk file format?
///
/// Delegates to [`super::signatures::match_file_signature`] so the
/// signature table stays in one place.
pub fn looks_like_executable(bytes: &[u8]) -> bool {
super::signatures::match_file_signature(bytes).is_some()
}
/// Heuristic: does this byte sequence look like obfuscated shellcode?
///
/// Triggers on:
/// - Known shellcode prologue patterns (NOP sled, Metasploit stagers, ...)
/// - Long runs of hex-encoded bytes (e.g. `\x41\x42\x43...`)
/// - High ratio of non-printable bytes
pub fn has_obfuscated_shellcode(bytes: &[u8]) -> bool {
if bytes.len() < 16 {
return false;
}
// Known shellcode prologue.
if super::signatures::match_shellcode_pattern(bytes).is_some() {
return true;
}
// Long hex-string run in the decoded preview.
let s = String::from_utf8_lossy(bytes);
let hex_chunks = s.split_whitespace().filter(|tok| {
tok.len() >= 4
&& tok.starts_with("\\x")
&& tok[2..].chars().all(|c| c.is_ascii_hexdigit())
});
if hex_chunks.count() >= 8 {
return true;
}
// High ratio of non-printable bytes (>50% non-printable over 256+ bytes).
if bytes.len() >= 256 {
let non_printable = bytes
.iter()
.filter(|b| {
!(**b >= 0x20 && **b < 0x7F) && **b != b'\n' && **b != b'\r' && **b != b'\t'
})
.count();
let ratio = non_printable as f64 / bytes.len() as f64;
if ratio > 0.5 {
return true;
}
}
false
}
/// Map a classification to a recommended action.
fn recommendation_for(classification: &ThreatClassification) -> Recommendation {
match classification {
ThreatClassification::Benign => Recommendation::Allow,
ThreatClassification::EducationalContent => Recommendation::WhitelistAsEducational,
ThreatClassification::Suspicious => Recommendation::Quarantine,
ThreatClassification::Malicious(_) => Recommendation::QuarantineAndCleanse,
}
}
/// Generate human-readable context notes for a finding.
fn notes_for(vector: &ExecutableVector, classification: &ThreatClassification) -> String {
match classification {
ThreatClassification::Benign => format!(
"{} vector at {} passed all heuristic checks",
vector.vector_type, vector.location
),
ThreatClassification::EducationalContent => format!(
"{} at {} appears in an educational / literature context — whitelisted",
vector.vector_type, vector.location
),
ThreatClassification::Suspicious => {
// For URI vectors, include the phishing reason if available.
if let Some(uri) = vector
.decoded_preview
.as_deref()
.or_else(|| std::str::from_utf8(&vector.raw_payload).ok())
{
if let Some(reason) = looks_like_phishing(uri) {
return format!(
"{} at {} flagged as suspicious (phishing signal: {})",
vector.vector_type, vector.location, reason
);
}
}
format!(
"{} at {} matched anomaly heuristics but lacked strong malicious signal",
vector.vector_type, vector.location
)
}
ThreatClassification::Malicious(t) => {
// For URI-based malicious findings, include the phishing reason.
if let Some(uri) = vector
.decoded_preview
.as_deref()
.or_else(|| std::str::from_utf8(&vector.raw_payload).ok())
{
if let Some(reason) = looks_like_phishing(uri) {
return format!(
"{} at {} classified as malicious ({}) [phishing signal: {}]",
vector.vector_type, vector.location, t, reason
);
}
}
// For embedded-file findings, include the matched file signature.
if vector.vector_type == VectorType::PdfEmbeddedFile
|| vector.vector_type == VectorType::UnknownPayload
|| vector.vector_type == VectorType::DocxEmbeddedObject
{
if let Some(sig_name) = super::signatures::match_file_signature(&vector.raw_payload) {
return format!(
"{} at {} classified as malicious ({}) [file signature: {}]",
vector.vector_type, vector.location, t, sig_name
);
}
if let Some(_pattern) = super::signatures::match_shellcode_pattern(&vector.raw_payload) {
return format!(
"{} at {} classified as malicious ({}) [shellcode prologue matched]",
vector.vector_type, vector.location, t
);
}
}
format!(
"{} at {} classified as malicious ({})",
vector.vector_type, vector.location, t
)
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::Location;
fn make_vector(vt: VectorType, payload: &[u8]) -> ExecutableVector {
ExecutableVector {
location: Location::PdfObject { id: 1, gen: 0 },
vector_type: vt,
raw_payload: payload.to_vec(),
decoded_preview: String::from_utf8_lossy(payload).to_string().into(),
}
}
#[test]
fn pdf_javascript_is_malicious() {
let v = make_vector(VectorType::PdfJavaScript, b"alert('xss')");
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::ActiveJavaScriptInjection)
));
}
#[test]
fn pdf_launch_is_malicious() {
let v = make_vector(VectorType::PdfLaunch, b"/bin/sh");
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::LaunchAction)
));
}
#[test]
fn embedded_exe_is_malicious() {
let v = make_vector(VectorType::PdfEmbeddedFile, b"MZ\x90\x00\x03\x00");
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
}
#[test]
fn https_url_is_benign() {
let v = make_vector(
VectorType::PdfUri,
b"https://example.com/path",
);
// Benign findings are dropped — so inspect_vector returns None.
let result = inspect_vector(&v, &Config::default());
assert!(result.is_none());
}
#[test]
fn javascript_scheme_url_is_malicious() {
let v = make_vector(
VectorType::PdfUri,
b"javascript:alert('xss')",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
}
#[test]
fn ip_address_url_is_phishing() {
let v = make_vector(
VectorType::PdfUri,
b"https://192.168.1.1/login",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
}
#[test]
fn epub_script_is_malicious() {
let v = make_vector(VectorType::EpubScript, b"alert('hi')");
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::EpubActiveScript)
));
}
#[test]
fn elf_binary_is_executable() {
let v = make_vector(VectorType::UnknownPayload, b"\x7FELF\x02\x01\x01");
let finding = inspect_vector(&v, &Config::default()).unwrap();
// Unknown payloads that match a file signature are now classified
// as MaliciousEmbeddedFile (more accurate than ObfuscatedShellcode).
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
// Notes should mention the ELF file signature.
assert!(
finding.context_notes.contains("elf"),
"notes should mention the matched file signature: {}",
finding.context_notes
);
}
#[test]
fn ole2_embedded_file_is_malicious() {
let v = make_vector(
VectorType::PdfEmbeddedFile,
b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1rest of file",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
assert!(finding.context_notes.contains("ole2"));
}
#[test]
fn rtf_embedded_file_is_malicious() {
let v = make_vector(
VectorType::PdfEmbeddedFile,
b"{\\rtf1\\ansi\\ansicpg1252}",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
assert!(finding.context_notes.contains("rtf"));
}
#[test]
fn vba_macro_embedded_file_is_malicious() {
let v = make_vector(
VectorType::PdfEmbeddedFile,
b"Attribute VB_Name = \"Module1\"\nSub AutoOpen()",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
assert!(finding.context_notes.contains("vba-macro"));
}
#[test]
fn metasploit_stager_is_shellcode() {
// FC E8 82 00 00 00 60 — Metasploit reverse TCP stager prologue.
// Pad to 16+ bytes so has_obfuscated_shellcode's length check passes.
let mut payload = vec![0xFC, 0xE8, 0x82, 0x00, 0x00, 0x00, 0x60, 0x89, 0xE5, 0x31, 0xC9];
payload.extend_from_slice(&[0x00; 8]); // padding
let v = make_vector(VectorType::UnknownPayload, &payload);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
));
assert!(finding.context_notes.contains("shellcode prologue"));
}
#[test]
fn lnk_embedded_file_is_malicious() {
let v = make_vector(
VectorType::PdfEmbeddedFile,
b"\x4c\x00\x00\x00\x01\x14\x02\x00rest",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::MaliciousEmbeddedFile)
));
assert!(finding.context_notes.contains("lnk"));
}
#[test]
fn nop_sled_is_shellcode() {
let mut payload = vec![0x90; 32];
payload.extend_from_slice(b"\xCC\xCC\xCC");
let v = make_vector(VectorType::UnknownPayload, &payload);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::ObfuscatedShellcode)
));
}
#[test]
fn suspicious_findings_can_be_suppressed() {
let v = make_vector(VectorType::PdfGoToR, b"");
let mut config = Config::default();
config.emit_suspicious = false;
let result = inspect_vector(&v, &config);
assert!(result.is_none(), "Suspicious findings should be suppressed");
}
// --- New phishing heuristics tests ---
#[test]
fn brand_homograph_url_is_malicious() {
// micros0ft (zero instead of 'o') → homograph bait.
let v = make_vector(
VectorType::PdfUri,
b"https://micros0ft.com/login",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(
matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
),
"brand homograph should be Malicious, got {:?}",
finding.classification
);
assert!(finding.context_notes.contains("brand-homograph"));
}
#[test]
fn canonical_brand_url_is_benign() {
let v = make_vector(
VectorType::PdfUri,
b"https://microsoft.com/windows",
);
// Canonical brand domain — should be Benign → dropped.
let result = inspect_vector(&v, &Config::default());
assert!(result.is_none());
}
#[test]
fn credential_url_is_malicious() {
let v = make_vector(
VectorType::PdfUri,
b"https://user:pass@evil.example.com/data",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
assert!(finding.context_notes.contains("credential-url"));
}
#[test]
fn url_shortener_is_suspicious() {
let v = make_vector(
VectorType::PdfUri,
b"https://bit.ly/3xyz",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert_eq!(
finding.classification,
ThreatClassification::Suspicious,
"URL shortener should be Suspicious (not Malicious)"
);
assert!(finding.context_notes.contains("url-shortener"));
}
#[test]
fn suspicious_keyword_in_url_is_suspicious() {
let v = make_vector(
VectorType::PdfUri,
b"https://example.com/account/verify",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert_eq!(finding.classification, ThreatClassification::Suspicious);
assert!(finding.context_notes.contains("suspicious-keyword"));
}
#[test]
fn phishing_tld_url_is_suspicious() {
let v = make_vector(
VectorType::PdfUri,
b"https://example.xyz/page",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert_eq!(finding.classification, ThreatClassification::Suspicious);
assert!(finding.context_notes.contains("phishing-tld"));
}
#[test]
fn phishing_tld_url_can_be_suppressed() {
let v = make_vector(
VectorType::PdfUri,
b"https://example.xyz/page",
);
let mut config = Config::default();
config.emit_suspicious = false;
// With suspicious findings suppressed, the .xyz URL becomes benign.
let result = inspect_vector(&v, &config);
assert!(result.is_none());
}
#[test]
fn data_url_is_malicious() {
let v = make_vector(
VectorType::PdfUri,
b"data:text/html,<script>alert(1)</script>",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
}
#[test]
fn vbscript_url_is_malicious() {
let v = make_vector(
VectorType::PdfUri,
b"vbscript:msgbox('xss')",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
}
#[test]
fn ip_url_strong_signal_overrides_weak_signals() {
// IP address with a "verify" keyword — both signals present,
// but IP (strong) should win.
let v = make_vector(
VectorType::PdfUri,
b"https://10.0.0.1/verify",
);
let finding = inspect_vector(&v, &Config::default()).unwrap();
assert!(matches!(
finding.classification,
ThreatClassification::Malicious(MaliciousType::SuspiciousUri)
));
assert!(
finding.context_notes.contains("ip-host"),
"strong IP signal should win over weak keyword signal: {}",
finding.context_notes
);
}
}

84
src/scanner/mod.rs Executable file
View File

@ -0,0 +1,84 @@
//! Security orchestration engine.
//!
//! The scanner takes a parsed [`Document`] and produces a [`ScanReport`]
//! by walking every [`TextNode`] and [`ExecutableVector`] through the
//! layered contextual engine described in the design manifest.
pub mod context_filter;
pub mod heuristics;
pub mod signatures;
pub mod cve_tags;
use crate::core::config::Config;
use crate::core::types::{Document, ScanReport};
/// Run the full scanner against `document`, producing a [`ScanReport`].
///
/// This is the top-level entrypoint called by [`crate::core::pipeline::Pipeline`].
#[must_use]
pub fn scan(document: &Document, config: &Config) -> ScanReport {
let mut findings = Vec::new();
// 1. Walk executable vectors — these are untrusted-by-default.
for vector in &document.executable_vectors {
if let Some(mut finding) = heuristics::inspect_vector(vector, config) {
// After classification, try to tag the finding with a
// known CVE if the payload matches a known exploit signature.
if let Some(cve) = cve_tags::match_cve(vector, &finding) {
finding.context_notes = format!(
"{} [{}: {}]",
finding.context_notes, cve.cve_id, cve.name
);
}
findings.push(finding);
}
}
// 2. Walk text nodes — these go through the context filter to
// distinguish educational content from active threats.
for node in &document.text_nodes {
if let Some(finding) = context_filter::evaluate(node, config) {
findings.push(finding);
}
}
let scanned_at = chrono::Utc::now().to_rfc3339();
ScanReport {
source_sha256: document.sha256.clone(),
format: document.format,
scanned_at,
findings,
text_nodes_scanned: document.text_nodes.len(),
vectors_scanned: document.executable_vectors.len(),
}
}
/// Re-export for callers that want to inspect individual findings.
pub use heuristics::inspect_vector;
pub use context_filter::evaluate;
#[cfg(test)]
mod tests {
use super::*;
use crate::core::types::*;
#[test]
fn empty_document_produces_empty_report() {
let doc = Document {
format: DocumentFormat::Markdown,
source_path: None,
raw_bytes: Vec::new(),
sha256: "abc".to_string(),
size: 0,
metadata: DocumentMetadata::default(),
text_nodes: Vec::new(),
executable_vectors: Vec::new(),
};
let config = Config::default();
let report = scan(&doc, &config);
assert_eq!(report.findings.len(), 0);
assert_eq!(report.text_nodes_scanned, 0);
assert_eq!(report.vectors_scanned, 0);
}
}

694
src/scanner/signatures.rs Executable file
View File

@ -0,0 +1,694 @@
//! Threat signature tables and pattern matchers.
//!
//! This module centralizes the static lookup tables used by the
//! heuristics engine. Keeping them in one place makes them easy to
//! audit, extend, and eventually wire up to an external threat-intel
//! feed (e.g. a YARA rules file or a STIX/TAXII subscription).
//!
//! ## What lives here
//!
//! - [`PHISHING_TLDS`] — TLDs statistically overrepresented in
//! phishing URLs. Sourced from public phishing reports
//! (Spamhaus, PhishTank yearly summaries).
//! - [`SUSPICIOUS_URL_KEYWORDS`] — path/host keywords that strongly
//! indicate credential harvesting or fake login pages.
//! - [`URL_SHORTENER_DOMAINS`] — shortener domains. Not malicious
//! per se, but a common obfuscation layer for phishing links.
//! - [`KNOWN_FILE_SIGNATURES`] — magic-byte signatures for executable
//! and high-risk file formats (PE, ELF, Mach-O, OLE2, RTF, etc.).
//! - [`SHELLCODE_PATTERNS`] — known shellcode prologue byte sequences
//! (NOP sleds, syscall stubs, common encoders).
//! - [`COMMON_PHISHING_BRANDS`] — brand names frequently spoofed in
//! phishing URLs (microsoft, paypal, appleid, …).
//!
//! ## External threat-intel feeds
//!
//! Additional rules can be loaded at runtime via
//! [`load_external_rules`]. Loaded rules are stored in a
//! process-wide static and checked by every match function
//! alongside the built-in tables.
use std::path::Path;
use std::sync::OnceLock;
use serde::Deserialize;
use crate::CorbelResult;
/// TLDs statistically overrepresented in phishing URLs.
///
/// Source: synthesized from public yearly phishing reports
/// (Spamhaus, PhishTank, Interisle). This list is intentionally
/// conservative — inclusion requires the TLD to appear in multiple
/// reports as a top-10 phishing TLD.
pub const PHISHING_TLDS: &[&str] = &[
// High-risk TLDs (cheap registration, low verification)
".zip", ".mov", ".xyz", ".top", ".click", ".link", ".rest", ".cyou",
".sbs", ".online", ".live", ".buzz", ".surf", ".monster", ".fit",
".loan", ".win", ".download", ".stream", ".review", ".men",
".work", ".racing", ".party", ".trade", ".science", ".kim",
".cricket", ".gq", ".cf", ".tk", ".ml", ".ga",
// Country-code TLDs frequently abused for phishing
".ru", ".cn", ".su", ".country", ".kim",
// Newer TLDs that have been flagged
".quest", ".bond", ".ha", ".cyou", ".quest", ".beauty",
];
/// URL path / host keywords that strongly suggest credential harvesting
/// or fake login pages. Matched case-insensitively as substrings.
pub const SUSPICIOUS_URL_KEYWORDS: &[&str] = &[
"login", "signin", "sign-in", "log-in", "verify", "verification",
"account", "update", "confirm", "secure", "security", "wallet",
"unlock", "recover", "reactivate", "validate", "activate",
"webscr", "cmd=", "_session", "authorization", "authenticate",
"reset", "password", "credential", "billing", "invoice",
"support", "suspended", "limited", "alert", "warning",
"urgent", "important-notice", "tax", "refund", "irs",
"postbank", "amzn", "appleid", "icloud", "office365",
];
/// Common URL-shortener domains. Shortened URLs are not malicious
/// per se, but they hide the real destination — we flag them as
/// `Suspicious` so the operator can preview the destination before
/// clicking.
pub const URL_SHORTENER_DOMAINS: &[&str] = &[
"bit.ly", "t.co", "tinyurl.com", "goo.gl", "ow.ly", "is.gd",
"buff.ly", "rebrand.ly", "cutt.ly", "shorturl.at", "tiny.cc",
"rb.gy", "s.id", "v.gd", "qr.ae", "x.co", "shorte.st",
"soo.gd", "lnkd.in", "po.st", "yourls.org", "bl.ink",
"surl.li", "kutt.it", "urlzs.com", "shrtco.de",
];
/// Brand names frequently spoofed in phishing URLs. Used to detect
/// homograph attacks (e.g. `micros0ft.com`, `paypa1.com`).
///
/// This list intentionally includes BOTH canonical spellings ("microsoft")
/// AND known homograph variants ("micros0ft" with zero instead of 'o').
/// The matcher uses a canonical-domain check to suppress benign
/// matches: when a brand is mentioned, we look for the canonical
/// spelling followed by a TLD; if found, we don't flag.
pub const COMMON_PHISHING_BRANDS: &[&str] = &[
// Microsoft family
"microsoft", "micros0ft", "micros0fte", "micr0soft",
"msn", "windows", "wind0ws", "office", "0ffice", "outlook",
"outl00k", "outl0ok", "live", "1ive",
// PayPal
"paypal", "paypa1", "paypaI", "paypa|",
// Apple
"apple", "app1e", "appie", "icloud", "ic1oud", "appleid",
"app1eid",
// Google
"google", "g00gle", "goog1e", "gmail", "gmai",
// Amazon
"amazon", "amzn", "amaz0n", "a-m-a-z-o-n",
// Social
"facebook", "faceb00k", "facebo0k", "instagram", "instagrarn",
"twitter", "tw1tter", "twtter", "linkedin", "1inkedin",
// Streaming
"netflix", "netf1ix", "spotify", "spot1fy",
// Storage / SaaS
"dropbox", "dr0pbox", "adobe", "ad0be",
// Banking
"bankofamerica", "bofa", "b0fa", "wellsfargo", "wellsfarg0",
"chase", "citibank", "citi", "hsbc", "barclays",
"santander", "unicredit",
// Crypto
"binance", "binanc3", "coinbase", "c0inbase", "metamask",
"metam4sk", "ledger", "trezor",
// Shipping
"dhl", "fedex", "f3dex", "ups", "usps", "royalmail",
// Gaming
"steamcommunity", "steampowered", "epicgames", "playstation",
"nintendo", "xbox",
];
/// Magic-byte signatures for executable and high-risk file formats.
///
/// Each entry is (offset, magic_bytes, name). When a payload's bytes
/// at `offset` match `magic_bytes`, the payload is considered
/// executable / high-risk.
pub const KNOWN_FILE_SIGNATURES: &[(usize, &[u8], &str)] = &[
// Windows PE
(0, b"MZ", "pe"),
// ELF
(0, b"\x7FELF", "elf"),
// Mach-O fat binary
(0, b"\xCA\xFE\xBA\xBE", "mach-o-fat"),
// Mach-O 64-bit (big-endian)
(0, b"\xFE\xED\xFA\xCF", "mach-o-64-be"),
// Mach-O 64-bit (little-endian)
(0, b"\xCF\xFA\xED\xFE", "mach-o-64-le"),
// Mach-O 32-bit (big-endian)
(0, b"\xFE\xED\xFA\xFE", "mach-o-32-be"),
// Mach-O 32-bit (little-endian)
(0, b"\xCE\xFA\xED\xFE", "mach-o-32-le"),
// OLE2 (Microsoft Office legacy, also used by some malware)
(0, b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1", "ole2"),
// RTF (often used as an obfuscation vector for CVE-2017-11882 etc.)
(0, b"{\\rtf", "rtf"),
// Java class file
(0, b"\xCA\xFE\xBA\xBE", "java-class"),
// Java JAR (zip, but flag if inside PDF)
// (zip is too generic — we don't flag it without other signals)
// Python bytecode
(0, b"\x42\x0d\x0d\x0a", "python-bytecode"),
// SWF (Flash — historically a huge attack surface)
(0, b"FWS", "swf"),
(0, b"CWS", "swf-compressed"),
(0, b"ZWS", "swf-lzma"),
// Windows Help file (.hlp) — old but still seen in attacks
(0, b"?_\x03\x00", "winhelp"),
// Windows shortcut (.lnk) — common payload in phishing docs
(0, b"\x4c\x00\x00\x00\x01\x14\x02\x00", "lnk"),
// HTA application (HTML Application — executes as a script)
(0, b"<html", "hta-candidate"),
// VBA macro stub (not a magic number, but a strong signal when
// seen at the start of an "embedded file" payload)
(0, b"Attribute VB_Name", "vba-macro"),
(0, b"Sub AutoOpen", "vba-macro"),
(0, b"Private Sub Document_Open", "vba-macro"),
];
/// Known shellcode prologue byte patterns.
///
/// Each entry is a byte sequence that, when found at the start of a
/// payload, strongly indicates shellcode. These are common prologues
/// used by off-the-shelf shellcode generators (msfvenom, etc.).
pub const SHELLCODE_PATTERNS: &[&[u8]] = &[
// NOP sled
&[0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90],
// x86 `xor edx, edx; push edx; ...` (common Win32 shellcode prologue)
&[0x31, 0xD2, 0x52, 0x68, 0x63, 0x61, 0x6C, 0x63],
// x86_64 syscall stub: `mov rax, 0x3b; xor rdi, rdi; ...` (execve)
&[0x48, 0xC7, 0xC0, 0x3B, 0x00, 0x00, 0x00, 0x48, 0x31, 0xFF],
// INT3 break sequence (debugger trap, sometimes used by packers)
&[0xCC, 0xCC, 0xCC, 0xCC],
// x86 `push esp; ret` (stack pivot gadget)
&[0x54, 0xC3],
// Metasploit stager signature
&[0xFC, 0xE8, 0x89, 0x00, 0x00, 0x00, 0x00, 0x60],
// Staged reverse TCP (Meters stage 1)
&[0xFC, 0xE8, 0x82, 0x00, 0x00, 0x00, 0x60],
// Common encoder stub: `xor eax, eax; ...` (shikata_ga_nai prologue)
&[0xB8, 0xC0, 0x18, 0x40, 0x00],
// x86_64 reverse-shell prologue
&[0x48, 0x31, 0xF6, 0x56, 0x48, 0xBF, 0x2F],
];
// ────────────────────────────────────────────────────────────────
// External threat-intel feed support
// ────────────────────────────────────────────────────────────────
/// A single external rule deserialized from a JSON threat-intel feed.
#[derive(Debug, Clone, Deserialize)]
pub struct ExternalRule {
/// Human-readable rule name (e.g. `"custom-phishing-tlds"`).
pub name: String,
/// Rule type: one of `"tld-list"`, `"keyword-list"`,
/// `"brand-list"`, `"signature-list"`, `"shellcode-list"`.
#[serde(rename = "type")]
pub rule_type: String,
/// Rule values. The shape depends on `rule_type`:
/// - `"tld-list"` / `"keyword-list"` / `"brand-list"` → array of strings
/// - `"signature-list"` → array of `{"offset": usize, "bytes": [u8], "name": str}`
/// - `"shellcode-list"` → array of hex strings or `[u8]` arrays
pub values: serde_json::Value,
}
/// Deserialized form of a single signature-list entry.
#[derive(Debug, Clone, Deserialize)]
struct SignatureEntry {
offset: usize,
bytes: Vec<u8>,
name: String,
}
/// Processed external rules, organized by type for fast matching.
#[derive(Debug, Clone, Default)]
pub(crate) struct ExternalRulesData {
/// Additional TLD strings.
pub(crate) tlds: Vec<String>,
/// Additional suspicious URL keywords.
pub(crate) keywords: Vec<String>,
/// Additional brand strings (may include homograph variants).
pub(crate) brands: Vec<String>,
/// Additional file-signature entries: (offset, magic-bytes, name).
pub(crate) signatures: Vec<(usize, Vec<u8>, String)>,
/// Additional shellcode byte patterns.
pub(crate) shellcode: Vec<Vec<u8>>,
}
/// Process-wide storage for externally loaded rules.
///
/// Populated once by [`load_external_rules`] and then read
/// (immutably) by every match function.
static EXTERNAL_RULES_STORAGE: OnceLock<ExternalRulesData> = OnceLock::new();
/// Load external signature rules from a JSON file.
///
/// The file must contain a JSON array of [`ExternalRule`] objects.
/// Rules are sorted into type-specific buckets and stored in a
/// process-wide static ([`EXTERNAL_RULES_STORAGE`]). Subsequent calls
/// to the match functions (`match_file_signature`, `has_phishing_tld`,
/// etc.) will check both the built-in tables and the external rules.
///
/// # Errors
///
/// Returns [`CorbelError::Io`] if the file cannot be read, or
/// [`CorbelError::Serde`] if the JSON is malformed.
pub fn load_external_rules(path: &Path) -> CorbelResult<Vec<ExternalRule>> {
let data = std::fs::read_to_string(path)?;
let rules: Vec<ExternalRule> = serde_json::from_str(&data)?;
let mut storage = ExternalRulesData::default();
for rule in &rules {
match rule.rule_type.as_str() {
"tld-list" => {
if let Some(arr) = rule.values.as_array() {
for v in arr {
if let Some(s) = v.as_str() {
storage.tlds.push(s.to_string());
}
}
}
}
"keyword-list" => {
if let Some(arr) = rule.values.as_array() {
for v in arr {
if let Some(s) = v.as_str() {
storage.keywords.push(s.to_string());
}
}
}
}
"brand-list" => {
if let Some(arr) = rule.values.as_array() {
for v in arr {
if let Some(s) = v.as_str() {
storage.brands.push(s.to_string());
}
}
}
}
"signature-list" => {
if let Some(arr) = rule.values.as_array() {
for v in arr {
if let Ok(sig) = serde_json::from_value::<SignatureEntry>(v.clone()) {
storage
.signatures
.push((sig.offset, sig.bytes, sig.name));
}
}
}
}
"shellcode-list" => {
if let Some(arr) = rule.values.as_array() {
for v in arr {
if let Some(hex_str) = v.as_str() {
// Hex-encoded string: "fc4883e4..."
if let Ok(bytes) = hex::decode(hex_str) {
if !bytes.is_empty() {
storage.shellcode.push(bytes);
}
}
} else if let Some(byte_arr) = v.as_array() {
// Raw byte array: [0xfc, 0x48, ...]
let bytes: Vec<u8> = byte_arr
.iter()
.filter_map(|b| b.as_u64().map(|n| n as u8))
.collect();
if !bytes.is_empty() {
storage.shellcode.push(bytes);
}
}
}
}
}
_ => {
// Unknown rule type — silently skip.
}
}
}
let _ = EXTERNAL_RULES_STORAGE.set(storage);
Ok(rules)
}
/// Return a reference to the externally loaded rules storage, if any
/// has been loaded via [`load_external_rules`].
///
/// This is `pub(crate)` because the return type (`ExternalRulesData`)
/// is itself `pub(crate)` — exposing it publicly would leak a private
/// type through the public API.
#[must_use]
pub(crate) fn get_external_rules_storage() -> Option<&'static ExternalRulesData> {
EXTERNAL_RULES_STORAGE.get()
}
/// Check whether `bytes` starts with any known executable / high-risk
/// file signature.
///
/// Returns the signature name (e.g. `"pe"`, `"elf"`) if matched, so
/// the caller can include it in the forensic report.
///
/// Checks both the built-in table and any external rules loaded via
/// [`load_external_rules`].
#[must_use]
pub fn match_file_signature(bytes: &[u8]) -> Option<&'static str> {
for (offset, magic, name) in KNOWN_FILE_SIGNATURES {
if bytes.len() >= *offset + magic.len() {
if &bytes[*offset..*offset + magic.len()] == *magic {
return Some(name);
}
}
}
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
for (offset, magic, name) in &ext.signatures {
if bytes.len() >= *offset + magic.len() {
if &bytes[*offset..*offset + magic.len()] == magic.as_slice() {
return Some(name);
}
}
}
}
None
}
/// Check whether `bytes` contains any known shellcode prologue.
///
/// Checks both the built-in table and any external rules loaded via
/// [`load_external_rules`].
#[must_use]
pub fn match_shellcode_pattern(bytes: &[u8]) -> Option<&'static [u8]> {
for pattern in SHELLCODE_PATTERNS {
if bytes.windows(pattern.len()).any(|w| w == *pattern) {
return Some(pattern);
}
}
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
for pattern in &ext.shellcode {
if bytes.windows(pattern.len()).any(|w| w == pattern.as_slice()) {
return Some(pattern.as_slice());
}
}
}
None
}
/// Check whether `host` (lowercase, no scheme) is a known URL shortener.
#[must_use]
pub fn is_url_shortener(host: &str) -> bool {
URL_SHORTENER_DOMAINS.iter().any(|d| host == *d || host.ends_with(&format!(".{d}")))
}
/// Check whether `uri` mentions a commonly-phished brand with a
/// non-canonical domain (homograph bait).
///
/// Returns the matched brand name if found.
///
/// Logic:
/// 1. If a homograph variant (`micros0ft`, `paypa1`, ...) is found
/// anywhere in the URI, it's always a phishing signal.
/// 2. If a canonical spelling (`microsoft`, `paypal`, ...) is found,
/// we check whether the host portion is ANY brand's canonical
/// domain (e.g. `microsoft.com` for "microsoft"). If yes → benign.
/// Otherwise, the brand is mentioned in a non-canonical context
/// → suspicious.
///
/// Checks both the built-in table and any external brands loaded via
/// [`load_external_rules`].
#[must_use]
pub fn match_phishing_brand(uri: &str) -> Option<&'static str> {
let lower = uri.to_ascii_lowercase();
// Extract host (after scheme://, before path/query/fragment, sans port).
let host = lower
.split("://")
.nth(1)
.unwrap_or(&lower)
.split('/')
.next()
.unwrap_or("")
.split(':')
.next()
.unwrap_or("");
let is_homograph = |brand: &str| brand.chars().any(|c| !c.is_ascii_alphabetic());
// First pass: check for homograph variants — these are ALWAYS phishing.
// Built-in brands.
for brand in COMMON_PHISHING_BRANDS {
if is_homograph(brand) && lower.contains(brand) {
return Some(brand);
}
}
// External brands.
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
for brand in &ext.brands {
if is_homograph(brand) && lower.contains(brand.as_str()) {
return Some(brand);
}
}
}
// Second pass: check canonical spellings. If the host is ANY
// brand's canonical domain, all canonical brand mentions are
// treated as benign. This handles cases like `microsoft.com/windows`
// (windows is a brand, but the host is microsoft's canonical domain).
let host_is_canonical_for_builtin = COMMON_PHISHING_BRANDS
.iter()
.any(|brand| !is_homograph(brand) && is_canonical_brand_host(host, brand));
let host_is_canonical_for_external = EXTERNAL_RULES_STORAGE
.get()
.map(|ext| {
ext.brands
.iter()
.any(|brand| !is_homograph(brand) && is_canonical_brand_host(host, brand))
})
.unwrap_or(false);
if host_is_canonical_for_builtin || host_is_canonical_for_external {
return None;
}
// Host is not a canonical brand domain — any canonical brand
// mentioned in the URL is suspicious.
// Built-in brands.
for brand in COMMON_PHISHING_BRANDS {
if !is_homograph(brand) && lower.contains(brand) {
return Some(brand);
}
}
// External brands.
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
for brand in &ext.brands {
if !is_homograph(brand) && lower.contains(brand.as_str()) {
return Some(brand);
}
}
}
None
}
/// Check whether `host` is the canonical domain for `brand`.
///
/// A host is canonical if it matches `<brand>.<tld>` or has `<brand>`
/// as a dot-separated segment (e.g. `microsoft.com`, `login.microsoft.com`).
/// The goal is to allow legitimate brand-owned domains while still
/// flagging `login-microsoft.com` (which is NOT a Microsoft domain).
fn is_canonical_brand_host(host: &str, brand: &str) -> bool {
if host == brand {
return true;
}
// Check if `<brand>.<tld>` is a prefix.
let canonical_prefix = format!("{}.", brand);
if host.starts_with(&canonical_prefix) {
return true;
}
// Check if `<brand>` is a dot-separated segment (e.g. `login.microsoft.com`).
host.split('.').any(|seg| seg == brand)
}
/// Check whether `uri`'s path/host contains any suspicious keyword.
///
/// Checks both the built-in table and any external rules loaded via
/// [`load_external_rules`].
#[must_use]
pub fn match_suspicious_keyword(uri: &str) -> Option<&'static str> {
let lower = uri.to_ascii_lowercase();
if let Some(kw) = SUSPICIOUS_URL_KEYWORDS
.iter()
.copied()
.find(|kw| lower.contains(kw))
{
return Some(kw);
}
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
if let Some(kw) = ext
.keywords
.iter()
.find(|kw| lower.contains(kw.as_str()))
{
return Some(kw);
}
}
None
}
/// Check whether the host part of `uri` ends with a known phishing TLD.
///
/// Checks both the built-in table and any external rules loaded via
/// [`load_external_rules`].
#[must_use]
pub fn has_phishing_tld(uri: &str) -> Option<&'static str> {
let lower = uri.to_ascii_lowercase();
// Extract host portion (after scheme://, before path/query/fragment).
let host = lower
.split("://")
.nth(1)
.unwrap_or(&lower)
.split('/')
.next()
.unwrap_or("");
// Strip port.
let host = host.split(':').next().unwrap_or("");
if let Some(tld) = PHISHING_TLDS.iter().copied().find(|tld| host.ends_with(tld)) {
return Some(tld);
}
if let Some(ext) = EXTERNAL_RULES_STORAGE.get() {
if let Some(tld) = ext.tlds.iter().find(|tld| host.ends_with(tld.as_str())) {
return Some(tld);
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn detects_pe_signature() {
assert_eq!(match_file_signature(b"MZ\x90\x00\x03"), Some("pe"));
}
#[test]
fn detects_elf_signature() {
assert_eq!(match_file_signature(b"\x7FELF\x02"), Some("elf"));
}
#[test]
fn detects_ole2_signature() {
assert_eq!(
match_file_signature(b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1rest"),
Some("ole2")
);
}
#[test]
fn detects_rtf_signature() {
assert_eq!(match_file_signature(b"{\\rtf1\\ansi..."), Some("rtf"));
}
#[test]
fn detects_swf_signature() {
assert_eq!(match_file_signature(b"FWS\x09"), Some("swf"));
}
#[test]
fn detects_lnk_signature() {
assert_eq!(
match_file_signature(b"\x4c\x00\x00\x00\x01\x14\x02\x00rest"),
Some("lnk")
);
}
#[test]
fn detects_vba_macro_signature() {
assert_eq!(
match_file_signature(b"Attribute VB_Name = \"evil\""),
Some("vba-macro")
);
}
#[test]
fn detects_nop_sled_pattern() {
let pattern = match_shellcode_pattern(&[0x90; 32]).unwrap();
assert_eq!(pattern[0], 0x90);
}
#[test]
fn detects_metasploit_stager_pattern() {
let payload = [0xFC, 0xE8, 0x82, 0x00, 0x00, 0x00, 0x60, 0x89];
assert!(match_shellcode_pattern(&payload).is_some());
}
#[test]
fn detects_url_shortener() {
assert!(is_url_shortener("bit.ly"));
assert!(is_url_shortener("sub.bit.ly"));
assert!(!is_url_shortener("example.com"));
}
#[test]
fn detects_phishing_brand_homograph() {
// micros0ft.com (with zero instead of 'o') should match the
// homograph variant directly.
assert_eq!(
match_phishing_brand("https://micros0ft.com/login"),
Some("micros0ft")
);
// paypa1.com (with one instead of 'l') should match.
assert_eq!(
match_phishing_brand("https://paypa1.com/signin"),
Some("paypa1")
);
}
#[test]
fn canonical_brand_domain_not_flagged() {
// microsoft.com (canonical) should not be flagged.
assert!(match_phishing_brand("https://microsoft.com/windows").is_none());
// paypal.com (canonical) should not be flagged.
assert!(match_phishing_brand("https://paypal.com/home").is_none());
}
#[test]
fn canonical_brand_in_non_canonical_domain_is_flagged() {
// microsoft mentioned in a non-canonical host → suspicious.
assert_eq!(
match_phishing_brand("https://login-microsoft.com/verify"),
Some("microsoft")
);
}
#[test]
fn detects_suspicious_keyword() {
// Should return the first matching keyword — both "account"
// and "verify" are in the list. Either is acceptable; check
// that we get one of them.
let result = match_suspicious_keyword("https://example.com/account/verify");
assert!(matches!(result, Some("account") | Some("verify")));
assert_eq!(
match_suspicious_keyword("https://example.com/signin"),
Some("signin")
);
}
#[test]
fn detects_phishing_tld() {
assert_eq!(has_phishing_tld("https://example.xyz"), Some(".xyz"));
assert_eq!(has_phishing_tld("https://example.top/path"), Some(".top"));
assert!(has_phishing_tld("https://example.com").is_none());
}
#[test]
fn detects_phishing_tld_with_port() {
assert_eq!(
has_phishing_tld("https://example.xyz:8080/path"),
Some(".xyz")
);
}
}

266
src/study/annotator.rs Executable file
View File

@ -0,0 +1,266 @@
//! Annotated HTML builder for study mode.
//!
//! Takes the original document bytes and the scan report, and produces
//! a self-contained HTML file with inline annotations around findings.
//!
//! For formats where we have structured content (PDF, EPUB, DOCX),
//! we extract text with location info and wrap findings in `<span>`.
//! For Markdown, we parse the source directly and inject spans.
use crate::core::types::{DocumentFormat, ScanReport, ThreatClassification, Finding};
/// Build the complete study-mode HTML document.
pub fn build_study_html(
raw_bytes: &[u8],
scan_report: &ScanReport,
sha256: &str,
source_path: &std::path::Path,
) -> String {
let format = scan_report.format;
// Build a mapping from location string to findings for O(1) lookup.
let mut location_findings: std::collections::HashMap<String, Vec<&Finding>> =
std::collections::HashMap::new();
for finding in &scan_report.findings {
location_findings
.entry(finding.location.to_string())
.or_default()
.push(finding);
}
let body_content = match format {
DocumentFormat::Markdown => build_markdown_study(raw_bytes, &location_findings),
_ => build_generic_study(raw_bytes, format, &location_findings),
};
let filename = source_path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("unknown");
format!(
r#"<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>CorbelPurge Study: {filename}</title>
<style>
body {{ font-family: 'SF Mono', 'Fira Code', 'Cascadia Code', monospace; max-width: 80ch; margin: 2em auto; padding: 0 1em; background: #0d0d0d; color: #e0e0e0; line-height: 1.6; }}
h1 {{ color: #d4af37; border-bottom: 2px solid #d4af37; padding-bottom: 0.5em; }}
.meta {{ color: #888; font-size: 0.85em; margin-bottom: 2em; }}
.meta dt {{ color: #aaa; }}
.meta dd {{ margin-left: 1em; margin-bottom: 0.5em; }}
.corbel-malicious {{ background: #5c1a1a; color: #ff6b6b; padding: 2px 4px; border-radius: 3px; border: 1px solid #ff6b6b; }}
.corbel-suspicious {{ background: #5c4a1a; color: #ffa94d; padding: 2px 4px; border-radius: 3px; border: 1px solid #ffa94d; }}
.corbel-educational {{ background: #1a4a2a; color: #69db7c; padding: 2px 4px; border-radius: 3px; border: 1px solid #69db7c; }}
.corbel-finding {{ margin: 0.5em 0; padding: 0.75em; border-left: 3px solid; font-size: 0.9em; }}
.corbel-finding.malicious {{ border-color: #ff6b6b; background: rgba(255,107,107,0.05); }}
.corbel-finding.suspicious {{ border-color: #ffa94d; background: rgba(255,169,77,0.05); }}
.corbel-finding.educational {{ border-color: #69db7c; background: rgba(105,219,124,0.05); }}
.finding-label {{ font-weight: bold; font-size: 0.8em; text-transform: uppercase; letter-spacing: 0.05em; }}
.finding-label.malicious {{ color: #ff6b6b; }}
.finding-label.suspicious {{ color: #ffa94d; }}
.finding-label.educational {{ color: #69db7c; }}
.finding-location {{ color: #888; font-size: 0.8em; }}
.finding-notes {{ color: #aaa; font-size: 0.8em; margin-top: 0.25em; }}
.finding-payload {{ background: #1a1a1a; padding: 0.5em; border-radius: 3px; font-size: 0.8em; white-space: pre-wrap; overflow-x: auto; max-height: 200px; overflow-y: auto; margin-top: 0.5em; }}
.summary {{ background: #1a1a1a; padding: 1em; border-radius: 5px; margin-bottom: 2em; }}
.summary .stat {{ display: inline-block; margin-right: 2em; }}
.summary .stat-label {{ color: #888; }}
.summary .stat-value {{ color: #e0e0e0; font-weight: bold; }}
.line-num {{ color: #555; user-select: none; display: inline-block; width: 4ch; text-align: right; margin-right: 1em; }}
pre {{ background: #1a1a1a; padding: 1em; border-radius: 5px; overflow-x: auto; }}
code {{ font-family: inherit; }}
</style>
</head>
<body>
<h1>CorbelPurge Study Mode</h1>
<div class="meta">
<dl>
<dt>File</dt><dd>{filename}</dd>
<dt>SHA-256</dt><dd><code>{sha256}</code></dd>
<dt>Format</dt><dd>{format}</dd>
<dt>Text nodes</dt><dd>{text_nodes}</dd>
<dt>Vectors</dt><dd>{vectors}</dd>
<dt>Findings</dt><dd>{total}</dd>
</dl>
</div>
<div class="summary">
<span class="stat"><span class="stat-label">Malicious:</span> <span class="stat-value">{malicious}</span></span>
<span class="stat"><span class="stat-label">Suspicious:</span> <span class="stat-value">{suspicious}</span></span>
<span class="stat"><span class="stat-label">Educational:</span> <span class="stat-value">{educational}</span></span>
</div>
{body_content}
</body>
</html>"#,
filename = html_escape(filename),
sha256 = sha256,
format = format,
text_nodes = scan_report.text_nodes_scanned,
vectors = scan_report.vectors_scanned,
total = scan_report.findings.len(),
malicious = scan_report.malicious_count(),
suspicious = scan_report.findings.iter().filter(|f| matches!(f.classification, ThreatClassification::Suspicious)).count(),
educational = scan_report.educational_count(),
body_content = body_content,
)
}
/// Build study HTML for Markdown source files.
///
/// Parses the Markdown line-by-line and wraps matching content in spans.
fn build_markdown_study(
raw: &[u8],
location_findings: &std::collections::HashMap<String, Vec<&Finding>>,
) -> String {
let text = String::from_utf8_lossy(raw);
let mut out = String::new();
out.push_str("<pre><code>\n");
for (idx, line) in text.lines().enumerate() {
let line_num = idx + 1;
let location_key = format!("md:{}:0", line_num);
let mut line_html = html_escape(line);
if let Some(findings) = location_findings.get(&location_key) {
for finding in findings {
let class = classification_class(&finding.classification);
line_html = format!(
"<span class=\"corbel-{}\">{}<span class=\"finding-label\"> {}</span></span>",
class, line_html, classification_label(&finding.classification)
);
}
}
out.push_str(&format!(
"<span class=\"line-num\">{line_num:>4}</span>{}\n",
line_html
));
}
out.push_str("</code></pre>\n");
out
}
/// Build study HTML for non-Markdown formats (PDF, EPUB, DOCX).
///
/// Since we can't reliably reconstruct the original rendering, we show
/// the findings in a structured list with their payload previews.
///
/// The `raw` bytes and `format` are accepted for symmetry with
/// [`build_markdown_study`] and for future per-format rendering hooks,
/// but are not currently consumed by this implementation.
#[allow(unused_variables)]
fn build_generic_study(
raw: &[u8],
format: DocumentFormat,
location_findings: &std::collections::HashMap<String, Vec<&Finding>>,
) -> String {
let mut out = String::new();
// Group findings by location for ordered display.
let mut seen = std::collections::HashSet::new();
for finding in location_findings
.values()
.flatten()
.collect::<Vec<_>>()
{
if !seen.insert(finding.location.to_string()) {
continue;
}
let class = classification_class(&finding.classification);
out.push_str(&format!(
"<div class=\"corbel-finding {}\">\n",
class
));
out.push_str(&format!(
" <div class=\"finding-label {}\">{}</div>\n",
class,
classification_label(&finding.classification)
));
out.push_str(&format!(
" <div class=\"finding-location\">{}</div>\n",
finding.location
));
out.push_str(&format!(
" <div class=\"finding-notes\">{}</div>\n",
html_escape(&finding.context_notes)
));
if !finding.payload_preview.is_empty() {
out.push_str(&format!(
" <div class=\"finding-payload\">{}</div>\n",
html_escape(&finding.payload_preview)
));
}
out.push_str("</div>\n");
}
// If no findings, show raw content in a code block.
if location_findings.is_empty() {
out.push_str("<p><em>No findings. Document passed all checks.</em></p>\n");
}
out
}
fn classification_class(c: &ThreatClassification) -> &'static str {
match c {
ThreatClassification::Benign => "benign",
ThreatClassification::Suspicious => "suspicious",
ThreatClassification::EducationalContent => "educational",
ThreatClassification::Malicious(_) => "malicious",
}
}
fn classification_label(c: &ThreatClassification) -> &'static str {
match c {
ThreatClassification::Benign => "benign",
ThreatClassification::Suspicious => "suspicious",
ThreatClassification::EducationalContent => "educational",
ThreatClassification::Malicious(_) => "malicious",
}
}
fn html_escape(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
.replace('>', "&gt;")
.replace('"', "&quot;")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn html_escape_basic() {
assert_eq!(html_escape("<script>alert('xss')</script>"), "&lt;script&gt;alert('xss')&lt;/script&gt;");
}
#[test]
fn build_study_html_has_structure() {
use crate::core::types::*;
let report = ScanReport {
source_sha256: "abc".to_string(),
format: DocumentFormat::Markdown,
scanned_at: "x".to_string(),
findings: vec![Finding {
classification: ThreatClassification::Malicious(MaliciousType::ActiveJavaScriptInjection),
location: Location::MarkdownLine { line: 3, col: 0 },
vector_type: None,
payload_preview: "eval(1)".to_string(),
context_notes: "found evil".to_string(),
recommendation: Recommendation::QuarantineAndCleanse,
}],
text_nodes_scanned: 5,
vectors_scanned: 1,
};
let html = build_study_html(b"hello\nworld\nevil", &report, "abc123", std::path::Path::new("test.md"));
assert!(html.contains("<span class=\"corbel-malicious\">"));
assert!(html.contains("<span class=\"line-num\"> 3</span>"));
assert!(html.contains("CorbelPurge Study Mode"));
assert!(html.contains("Malicious:"));
}
}

73
src/study/mod.rs Executable file
View File

@ -0,0 +1,73 @@
//! Study mode: annotated HTML output showing exploit locations.
//!
//! The `corbel-purge study <path>` subcommand runs the full pipeline
//! and then renders the **original** document content with malicious
//! findings annotated inline. This is useful for researchers who want
//! to see *where* in the document an exploit was found, not just read
//! a report about it.
//!
//! The output is a self-contained HTML file with inline `<span>` wrappers
//! around matched content, color-coded by classification:
//!
//! - **Red**: Malicious
//! - **Orange**: Suspicious
//! - **Green**: EducationalContent
//! - **Gray**: Benign (not annotated)
pub mod annotator;
use std::path::{Path, PathBuf};
use crate::{Config, CorbelResult, Pipeline};
/// Run study mode on a file, producing an annotated HTML file.
///
/// Returns the path to the generated HTML file.
pub fn run_study(
path: impl AsRef<Path>,
workspace: Option<&Path>,
) -> CorbelResult<PathBuf> {
let path = path.as_ref();
let bytes = std::fs::read(path)?;
let mut config = match workspace {
Some(p) => Config::with_workspace(p),
None => Config::default(),
};
config.abort_on_threat = false;
config = config.override_from_env();
let pipeline = Pipeline::with_config(config.clone());
let result = pipeline.run_on_bytes(
bytes.clone(),
crate::core::types::DocumentFormat::from_path(path)?,
Some(path.to_path_buf()),
)?;
let study_dir = workspace
.map(|p| p.to_path_buf())
.unwrap_or_else(|| std::env::current_dir().unwrap());
let study_dir = study_dir.join("corbel_study");
std::fs::create_dir_all(&study_dir)?;
let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S");
let sha_prefix = &result.source_sha256[..8.min(result.source_sha256.len())];
let html_filename = format!("study_{timestamp}_{sha_prefix}.html");
let html_path = study_dir.join(&html_filename);
let html = annotator::build_study_html(
&bytes,
&result.scan_report,
&result.source_sha256,
path,
);
std::fs::write(&html_path, html)?;
// Also write the JSON report for programmatic access.
let json_path = study_dir.join(format!("study_{timestamp}_{sha_prefix}.json"));
if let Some(ref report_path) = result.json_report_path {
std::fs::copy(report_path, &json_path)?;
}
Ok(html_path)
}

35
src/ui/alert_modal.rs Executable file
View File

@ -0,0 +1,35 @@
//! Quarantine & threat notification overlay — currently a stub.
//!
//! A full implementation would render a modal dialog over the viewer
//! when a threat is detected, with options to view the forensic report
//! or open the quarantine tarball.
use crate::core::ScanReport;
/// State for the alert modal.
#[derive(Debug, Default)]
pub struct AlertModal {
/// Whether the modal is currently shown.
pub visible: bool,
/// The scan report being displayed, if any.
pub report: Option<ScanReport>,
}
impl AlertModal {
/// Construct a new, hidden modal.
#[must_use]
pub fn new() -> Self {
Self::default()
}
/// Show the modal with the given report.
pub fn show(&mut self, report: ScanReport) {
self.report = Some(report);
self.visible = true;
}
/// Dismiss the modal.
pub fn dismiss(&mut self) {
self.visible = false;
}
}

16
src/ui/mod.rs Executable file
View File

@ -0,0 +1,16 @@
//! GUI module: iced-based viewer.
//!
//! This module is **gated behind the `gui` feature flag**. The default
//! build of CorbelPurge produces only the headless CLI binary, which is
//! what gets used in CI pipelines and server-side scanners. The GUI is
//! an optional build that produces a separate binary.
//!
//! In the current MVP, the GUI is a stub: it can open a file picker,
//! run the pipeline, and display the scan summary. A full rendered
//! canvas (text + images) is left as future work.
pub mod viewer;
pub mod alert_modal;
pub use viewer::Viewer;
pub use alert_modal::AlertModal;

36
src/ui/viewer.rs Executable file
View File

@ -0,0 +1,36 @@
//! Canvas renderer (text/images) — currently a stub.
//!
//! A full implementation would render cleansed document content to an
//! iced canvas. For the MVP, this module just holds the state and
//! delegates to the pipeline for the actual scanning work.
use crate::PipelineResult;
/// Top-level viewer state.
#[derive(Debug, Default)]
pub struct Viewer {
/// The most recent scan result, if any.
pub last_result: Option<PipelineResult>,
/// Whether the viewer is currently rendering a document.
pub is_rendering: bool,
}
impl Viewer {
/// Construct a new, empty viewer.
#[must_use]
pub fn new() -> Self {
Self::default()
}
/// Update the viewer with a fresh scan result.
pub fn set_result(&mut self, result: PipelineResult) {
self.last_result = Some(result);
self.is_rendering = false;
}
/// Clear the viewer.
pub fn clear(&mut self) {
self.last_result = None;
self.is_rendering = false;
}
}

286
src/util.rs Executable file
View File

@ -0,0 +1,286 @@
//! Small utility helpers shared across the crate.
//!
//! Kept in a dedicated module so that tests and downstream code have
//! one place to find cross-cutting helpers (hashing, truncation, etc.).
use std::io::Read;
use sha2::{Digest, Sha256};
/// Compute the SHA-256 hex digest of `bytes`.
///
/// Returned string is always 64 lowercase hex characters.
#[must_use]
pub fn sha256_hex(bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(bytes);
let digest = hasher.finalize();
hex::encode(digest)
}
/// Truncate `s` to at most `max_chars` characters, appending an ellipsis
/// if truncation occurred. Useful for payload previews in threat reports.
#[must_use]
pub fn truncate_with_ellipsis(s: &str, max_chars: usize) -> String {
if s.chars().count() <= max_chars {
return s.to_string();
}
let truncated: String = s.chars().take(max_chars.saturating_sub(1)).collect();
format!("{truncated}…")
}
/// Outcome of a streaming read with a byte cap.
///
/// The key insight: we count **actual bytes consumed from the stream**,
/// not the size declared in any header. A malicious ZIP archive can
/// declare `size = 100` in its central directory while actually
/// decompressing to gigabytes — [`read_with_cap`] detects this by
/// aborting mid-stream once `cap` bytes have been read.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ReadOutcome {
/// The stream reached EOF before the cap was hit. Contains all
/// bytes that were read.
Complete(Vec<u8>),
/// The cap was hit before EOF. Contains the bytes read so far
/// (exactly `cap` bytes, or fewer if the read stopped mid-chunk),
/// plus the total number of bytes consumed from the stream
/// (which may be slightly larger than `bytes.len()` if the final
/// chunk pushed us over the cap).
///
/// Callers should treat this as a likely zip-bomb / oversized
/// payload and emit a suspicious finding rather than continuing
/// to parse the truncated content.
Truncated {
/// The bytes we managed to read before stopping (≤ cap).
bytes: Vec<u8>,
/// Total bytes consumed from the stream before we stopped
/// reading. Always ≥ `bytes.len()` and always > `cap`.
bytes_read: usize,
/// The cap that was exceeded.
cap: usize,
},
}
/// Read from `reader` into a `Vec<u8>`, but stop once `cap` bytes have
/// been consumed.
///
/// This is the **zip-bomb defense** for ZIP-container formats (EPUB,
/// DOCX). The size declared in the ZIP central directory is *not
/// trusted* — we count actual decompressed bytes via repeated
/// `reader.read()` calls.
///
/// ## How it works
///
/// 1. Allocate a buffer with capacity `min(cap, 1 MiB)` (we don't
/// trust `reader`'s declared size).
/// 2. Read in 8 KiB chunks until either:
/// - `reader.read()` returns `Ok(0)` (EOF) → return `Complete`.
/// - The buffer would exceed `cap` → return `Truncated` with the
/// bytes read so far.
/// 3. On any I/O error, propagate it.
///
/// ## Why 8 KiB chunks
///
/// Small enough that we check the cap frequently (a malicious stream
/// can't sneak through too many bytes between checks), large enough
/// that the per-call overhead is negligible vs. the decompressor's
/// internal buffering.
///
/// ## What this defends against
///
/// - **Lying size header**: a ZIP entry that declares `size = 100` but
/// actually decompresses to 100 MB. The streaming reader detects
/// this after `cap` bytes regardless of what the header said.
/// - **Decompression ratio attack**: a 42 KB ZIP that decompresses to
/// petabytes (the famous 42.zip). The streaming reader stops at
/// `cap` bytes and never allocates more.
///
/// ## What this does NOT defend against
///
/// - **Pre-allocated capacity attacks**: if `cap` itself is huge
/// (e.g. 1 GiB), we'll happily allocate that much. Set `cap` to a
/// sane value (the default is 8 MiB).
/// - **Many small entries**: an archive with 10000 entries of 1 MiB
/// each still totals 10 GiB of memory. The per-entry cap doesn't
/// help here — the pipeline coordinator would need a total-memory
/// budget. (Future work.)
pub fn read_with_cap<R: Read>(
reader: &mut R,
cap: usize,
) -> std::io::Result<ReadOutcome> {
// Refuse a zero cap — that would always truncate immediately and
// is almost certainly a caller bug.
assert!(cap > 0, "read_with_cap: cap must be > 0");
// Pre-allocate up to 1 MiB or `cap`, whichever is smaller.
// We deliberately do NOT pre-allocate `cap` bytes — if `cap` is
// 1 GiB, that would itself be the DoS.
let initial_capacity = cap.min(1024 * 1024);
let mut buf = Vec::with_capacity(initial_capacity);
// 8 KiB chunk buffer.
let mut chunk = [0u8; 8 * 1024];
loop {
let n = reader.read(&mut chunk)?;
if n == 0 {
// EOF — we read the whole stream within the cap.
return Ok(ReadOutcome::Complete(buf));
}
// Check if appending would push us over the cap.
// We append the full chunk even if it overshoots — that way
// the caller gets a clean byte boundary to inspect (e.g. for
// file-signature matching on the first few bytes).
if buf.len() + n > cap {
// Append what fits.
let remaining = cap.saturating_sub(buf.len());
buf.extend_from_slice(&chunk[..remaining]);
let bytes_read = buf.len();
return Ok(ReadOutcome::Truncated {
bytes: buf,
bytes_read,
cap,
});
}
buf.extend_from_slice(&chunk[..n]);
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Cursor;
#[test]
fn sha256_known_vector() {
// SHA-256 of empty input.
assert_eq!(
sha256_hex(b""),
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
);
// SHA-256 of "abc"
assert_eq!(
sha256_hex(b"abc"),
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
);
}
#[test]
fn truncate_short_string_unchanged() {
assert_eq!(truncate_with_ellipsis("hello", 10), "hello");
}
#[test]
fn truncate_long_string_gets_ellipsis() {
let result = truncate_with_ellipsis("abcdefghij", 5);
assert_eq!(result, "abcd…");
}
#[test]
fn truncate_exact_length_unchanged() {
assert_eq!(truncate_with_ellipsis("abcde", 5), "abcde");
}
// --- read_with_cap tests ---
#[test]
fn read_small_stream_completes() {
let data = b"hello world".to_vec();
let mut cursor = Cursor::new(data.clone());
let outcome = read_with_cap(&mut cursor, 1024).unwrap();
assert_eq!(outcome, ReadOutcome::Complete(data));
}
#[test]
fn read_empty_stream_completes() {
let mut cursor = Cursor::new(Vec::new());
let outcome = read_with_cap(&mut cursor, 1024).unwrap();
assert_eq!(outcome, ReadOutcome::Complete(Vec::new()));
}
#[test]
fn read_exact_cap_completes() {
// 100 bytes, cap = 100 → should complete exactly at the boundary.
let data = vec![0x41u8; 100];
let mut cursor = Cursor::new(data.clone());
let outcome = read_with_cap(&mut cursor, 100).unwrap();
assert_eq!(outcome, ReadOutcome::Complete(data));
}
#[test]
fn read_just_over_cap_truncates() {
// 200 bytes, cap = 100 → should truncate.
let data = vec![0x41u8; 200];
let mut cursor = Cursor::new(data);
let outcome = read_with_cap(&mut cursor, 100).unwrap();
match outcome {
ReadOutcome::Truncated { bytes, bytes_read, cap } => {
assert_eq!(cap, 100);
assert!(bytes.len() <= 100, "bytes.len() should be ≤ cap");
assert_eq!(bytes_read, bytes.len());
// We should have read *something* — at least the first chunk.
assert!(!bytes.is_empty());
}
ReadOutcome::Complete(_) => panic!("should have truncated"),
}
}
#[test]
fn read_huge_stream_with_small_cap_truncates_quickly() {
// Simulate a zip-bomb: 10 MiB of data, 8 KiB cap.
// We should truncate after reading ~8 KiB, not allocate 10 MiB.
let data = vec![0x41u8; 10 * 1024 * 1024];
let mut cursor = Cursor::new(data);
let outcome = read_with_cap(&mut cursor, 8 * 1024).unwrap();
match outcome {
ReadOutcome::Truncated { bytes, bytes_read, cap } => {
assert_eq!(cap, 8 * 1024);
assert!(bytes.len() <= 8 * 1024);
assert_eq!(bytes_read, bytes.len());
// Critical: we never allocated more than cap.
assert!(bytes.len() <= 8 * 1024);
}
ReadOutcome::Complete(_) => panic!("should have truncated"),
}
}
#[test]
fn read_propagates_io_errors() {
use std::io::{self, Read};
struct ErroringReader;
impl Read for ErroringReader {
fn read(&mut self, _buf: &mut [u8]) -> io::Result<usize> {
Err(io::Error::new(io::ErrorKind::Other, "synthetic error"))
}
}
let mut reader = ErroringReader;
let result = read_with_cap(&mut reader, 1024);
assert!(result.is_err());
assert_eq!(
result.unwrap_err().to_string(),
"synthetic error"
);
}
#[test]
fn read_returns_partial_bytes_on_truncation() {
// Verify the truncated bytes are actually the *first* bytes
// of the stream (so the caller can still do file-signature
// matching on the prefix).
let data: Vec<u8> = (0..200).map(|i| i as u8).collect();
let mut cursor = Cursor::new(data);
let outcome = read_with_cap(&mut cursor, 50).unwrap();
match outcome {
ReadOutcome::Truncated { bytes, .. } => {
// The first byte should be 0, the last should be ≤ 50.
assert_eq!(bytes[0], 0);
assert!(bytes.last().copied().unwrap() <= 50);
}
ReadOutcome::Complete(_) => panic!("should have truncated"),
}
}
}

BIN
tests/fixtures/benign.docx vendored Executable file
View File

Binary file not shown.

BIN
tests/fixtures/benign.epub vendored Executable file
View File

Binary file not shown.

8
tests/fixtures/benign.md vendored Executable file
View File

@ -0,0 +1,8 @@
# Benign Document
This is a paragraph of perfectly normal text. It discusses the weather,
the state of the economy, and other uncontroversial topics.
## Subsection
More text here. Nothing suspicious whatsoever.

68
tests/fixtures/benign.pdf vendored Executable file
View File

@ -0,0 +1,68 @@
%PDF-1.3
%“Œ‹ž ReportLab Generated PDF document (opensource)
1 0 obj
<<
/F1 2 0 R
>>
endobj
2 0 obj
<<
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
>>
endobj
3 0 obj
<<
/Contents 7 0 R /MediaBox [ 0 0 612 792 ] /Parent 6 0 R /Resources <<
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
>> /Rotate 0 /Trans <<
>>
/Type /Page
>>
endobj
4 0 obj
<<
/PageMode /UseNone /Pages 6 0 R /Type /Catalog
>>
endobj
5 0 obj
<<
/Author (CorbelPurge Tests) /CreationDate (D:20260731131243+00'00') /Creator (anonymous) /Keywords () /ModDate (D:20260731131243+00'00') /Producer (ReportLab PDF Library - \(opensource\))
/Subject (unspecified) /Title (Benign Test PDF) /Trapped /False
>>
endobj
6 0 obj
<<
/Count 1 /Kids [ 3 0 R ] /Type /Pages
>>
endobj
7 0 obj
<<
/Filter [ /ASCII85Decode /FlateDecode ] /Length 170
>>
stream
GarVE]*\To&FAj9VGr[6.u.;d*XWYuqZARU)TciR>=AMJE*?Ab4GmQZ#d/[^%iqldbZY(m_&#pH2rFeH>a:hn<H8Sm,47/=8P478]>YhF*[f?[<?*r74U0f[P<\BPlEf=d6<<\THXoi(Qokgacd3(B/3[SmLt_;:W],C)02V~>endstream
endobj
xref
0 8
0000000000 65535 f
0000000061 00000 n
0000000092 00000 n
0000000199 00000 n
0000000392 00000 n
0000000460 00000 n
0000000736 00000 n
0000000795 00000 n
trailer
<<
/ID
[<88c9b83603e2b9c29dc61f2d955be744><88c9b83603e2b9c29dc61f2d955be744>]
% ReportLab generated PDF document -- digest (opensource)
/Info 5 0 R
/Root 4 0 R
/Size 8
>>
startxref
1055
%%EOF

21
tests/fixtures/cve_writeup.md vendored Executable file
View File

@ -0,0 +1,21 @@
# CVE-2024-1234: PDF JavaScript Injection
## Abstract
In this paper we describe a vulnerability in which a malicious PDF
uses a /JavaScript action to execute arbitrary code. The eval() function
is called with attacker-controlled input.
## Proof of Concept
```python
# This is a PoC for the vulnerability described above.
import subprocess
# Note: this code is for educational purposes only.
payload = "eval('alert(1)')"
print(f"Payload: {payload}")
```
## Remediation
Patch the reader to ignore /JavaScript actions in /OpenAction.

68
tests/fixtures/cve_writeup.pdf vendored Executable file
View File

@ -0,0 +1,68 @@
%PDF-1.3
%“Œ‹ž ReportLab Generated PDF document (opensource)
1 0 obj
<<
/F1 2 0 R
>>
endobj
2 0 obj
<<
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
>>
endobj
3 0 obj
<<
/Contents 7 0 R /MediaBox [ 0 0 612 792 ] /Parent 6 0 R /Resources <<
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
>> /Rotate 0 /Trans <<
>>
/Type /Page
>>
endobj
4 0 obj
<<
/PageMode /UseNone /Pages 6 0 R /Type /Catalog
>>
endobj
5 0 obj
<<
/Author (Security Researcher) /CreationDate (D:20260731131243+00'00') /Creator (anonymous) /Keywords () /ModDate (D:20260731131243+00'00') /Producer (ReportLab PDF Library - \(opensource\))
/Subject (unspecified) /Title (CVE-2024-1234 Writeup) /Trapped /False
>>
endobj
6 0 obj
<<
/Count 1 /Kids [ 3 0 R ] /Type /Pages
>>
endobj
7 0 obj
<<
/Filter [ /ASCII85Decode /FlateDecode ] /Length 384
>>
stream
Gas2D?VeNm'ZJu*'^'J'0\/6o_AasM(__le54R'D(O'lbQF6nkp3&e0%tlcRdei3thl/jRNYlY>!tQ$nV1;)[2\9,^6,$_M8"hR77R`Kur[rc'&NtRq->AWT'9V-`1V>k&nZ6p!6X+Fk<b>"YF(6m_W]E+_A&p7_bpC@$W&W+o[n?NJPK>L=?mYtfg-\=["(/?$X'2g]'+/c2,:=o+]#p+73EPPX@"WmL\dXR%<NNaq>gNE<)0LsEBMQmZW+-CrJ::%55,d2*"0+LL2[.0^-%U6>>"@oK8SQYAj'i#;WiE#5SpbSO?KlC6'AXLDS"[Zqm<jk:Wi9CVDp;$<Xn%%p#7F2)#l+N-J!n=)"\RAoSA)FBk/08Eqb2rt!E\rZR/~>endstream
endobj
xref
0 8
0000000000 65535 f
0000000061 00000 n
0000000092 00000 n
0000000199 00000 n
0000000392 00000 n
0000000460 00000 n
0000000744 00000 n
0000000803 00000 n
trailer
<<
/ID
[<6e2b4056078bc06bbe77ecc8c05b4144><6e2b4056078bc06bbe77ecc8c05b4144>]
% ReportLab generated PDF document -- digest (opensource)
/Info 5 0 R
/Root 4 0 R
/Size 8
>>
startxref
1277
%%EOF

BIN
tests/fixtures/honest_zip_bomb.zip vendored Executable file
View File

Binary file not shown.

BIN
tests/fixtures/lying_zip_bomb.zip vendored Executable file
View File

Binary file not shown.

BIN
tests/fixtures/malicious.epub vendored Executable file
View File

Binary file not shown.

5
tests/fixtures/malicious.md vendored Executable file
View File

@ -0,0 +1,5 @@
# Click Here
Free money! [Click now](javascript:alert('xss'))
Run this: \x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90

86
tests/fixtures/malicious_js.pdf vendored Executable file
View File

@ -0,0 +1,86 @@
%PDF-1.3
%âãÏÓ
1 0 obj
<<
/Producer (pypdf)
>>
endobj
2 0 obj
<<
/Type /Pages
/Count 1
/Kids [ 4 0 R ]
>>
endobj
3 0 obj
<<
/Type /Catalog
/Pages 2 0 R
/OpenAction 8 0 R
>>
endobj
4 0 obj
<<
/Contents 5 0 R
/MediaBox [ 0 0 612 792 ]
/Resources <<
/Font 6 0 R
/ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
>>
/Rotate 0
/Trans <<
>>
/Type /Page
/Parent 2 0 R
>>
endobj
5 0 obj
<<
/Filter [ /ASCII85Decode /FlateDecode ]
/Length 130
>>
stream
GapQh0E=F,0U\H3T\pNYT^QKk?tc>IP,;W#U1^23ihPEM_?CW4KISi90MjG.ifICKNKB+C@YFdu2hd4/@YI2RYJd`bi8gWVJeEdUjH@ab4?ZDmO=Z(s/W^PO>Q?=<)Ii~>
endstream
endobj
6 0 obj
<<
/F1 7 0 R
>>
endobj
7 0 obj
<<
/BaseFont /Helvetica
/Encoding /WinAnsiEncoding
/Name /F1
/Subtype /Type1
/Type /Font
>>
endobj
8 0 obj
<<
/Type /Action
/S /JavaScript
/JS (app\056alert\050\047XSS from PDF\047\051\073)
>>
endobj
xref
0 9
0000000000 65535 f
0000000015 00000 n
0000000054 00000 n
0000000113 00000 n
0000000180 00000 n
0000000369 00000 n
0000000590 00000 n
0000000621 00000 n
0000000728 00000 n
trailer
<<
/Size 9
/Root 3 0 R
/Info 1 0 R
>>
startxref
829
%%EOF

89
tests/fixtures/malicious_launch.pdf vendored Executable file
View File

@ -0,0 +1,89 @@
%PDF-1.3
%âãÏÓ
1 0 obj
<<
/Producer (pypdf)
>>
endobj
2 0 obj
<<
/Type /Pages
/Count 1
/Kids [ 4 0 R ]
>>
endobj
3 0 obj
<<
/Type /Catalog
/Pages 2 0 R
/OpenAction 8 0 R
>>
endobj
4 0 obj
<<
/Contents 5 0 R
/MediaBox [ 0 0 612 792 ]
/Resources <<
/Font 6 0 R
/ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
>>
/Rotate 0
/Trans <<
>>
/Type /Page
/Parent 2 0 R
>>
endobj
5 0 obj
<<
/Filter [ /ASCII85Decode /FlateDecode ]
/Length 122
>>
stream
Gap@D0a`Fb&-R>e0r8&e:'>o@.gq:45R9n)(^p$:7Zh8[=Cq`#+s5VN]i%XX2!Bh(<K#.]R\a;=<K*VEeX]rn3*CtBDbhpj]qb1$GD.8B"hLDhHRF.?\XXMW~>
endstream
endobj
6 0 obj
<<
/F1 7 0 R
>>
endobj
7 0 obj
<<
/BaseFont /Helvetica
/Encoding /WinAnsiEncoding
/Name /F1
/Subtype /Type1
/Type /Font
>>
endobj
8 0 obj
<<
/Type /Action
/S /Launch
/F (\057bin\057sh)
/Win <<
/F (cmd\056exe)
>>
>>
endobj
xref
0 9
0000000000 65535 f
0000000015 00000 n
0000000054 00000 n
0000000113 00000 n
0000000180 00000 n
0000000369 00000 n
0000000582 00000 n
0000000613 00000 n
0000000720 00000 n
trailer
<<
/Size 9
/Root 3 0 R
/Info 1 0 R
>>
startxref
812
%%EOF

BIN
tests/fixtures/malicious_link.docx vendored Normal file
View File

Binary file not shown.

BIN
tests/fixtures/malicious_macro.docx vendored Executable file
View File

Binary file not shown.

BIN
tests/fixtures/malicious_ole.docx vendored Executable file
View File

Binary file not shown.

443
tests/pipeline_integration.rs Executable file
View File

@ -0,0 +1,443 @@
// ---------------------------------------------------------------------------
// Pipeline integration tests
// ---------------------------------------------------------------------------
use std::path::PathBuf;
use corbel_purge::{Config, CleanseMode, DocumentFormat, Pipeline};
use corbel_purge::core::types::ThreatClassification;
use corbel_purge::parsers::{Dispatcher, DocumentParser};
use tempfile::tempdir;
fn fixtures_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures")
}
fn fixture(name: &str) -> PathBuf {
fixtures_dir().join(name)
}
// ---------------------------------------------------------------------------
// Benign documents: no findings, no quarantine, no cleanse
// ---------------------------------------------------------------------------
#[test]
fn benign_pdf_produces_no_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("benign.pdf")).unwrap();
assert_eq!(result.scan_report.malicious_count(), 0);
assert!(result.quarantine_path.is_none());
assert!(result.cleansed_path.is_none());
}
#[test]
fn benign_epub_produces_no_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("benign.epub")).unwrap();
assert_eq!(result.scan_report.malicious_count(), 0);
assert!(result.quarantine_path.is_none());
}
#[test]
fn benign_docx_produces_no_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("benign.docx")).unwrap();
assert_eq!(result.scan_report.malicious_count(), 0);
assert!(result.quarantine_path.is_none());
}
#[test]
fn benign_markdown_produces_no_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("benign.md")).unwrap();
assert_eq!(result.scan_report.malicious_count(), 0);
assert!(result.quarantine_path.is_none());
}
// ---------------------------------------------------------------------------
// Malicious documents: findings, quarantine, and cleanse
// ---------------------------------------------------------------------------
#[test]
fn malicious_pdf_js_produces_findings_and_quarantine() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_js.pdf")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
assert!(result.quarantine_path.is_some());
assert!(result.cleansed_path.is_some());
}
#[test]
fn malicious_pdf_launch_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_launch.pdf")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
assert!(result.quarantine_path.is_some());
}
#[test]
fn malicious_epub_script_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious.epub")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
assert!(result.quarantine_path.is_some());
}
#[test]
fn malicious_md_xss_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious.md")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
}
#[test]
fn malicious_docx_macro_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_macro.docx")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
assert!(result.quarantine_path.is_some());
}
#[test]
fn malicious_docx_ole_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_ole.docx")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
assert!(result.quarantine_path.is_some());
}
#[test]
fn malicious_docx_link_produces_findings() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_link.docx")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
}
// ---------------------------------------------------------------------------
// PreserveFormat repackage
// ---------------------------------------------------------------------------
#[test]
fn preserve_format_epub_strips_script_entry() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.cleanse_mode = CleanseMode::PreserveFormat;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("malicious.epub")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
let cleansed = result.cleansed_path.expect("cleansed EPUB should exist");
let content = std::fs::read(&cleansed).unwrap();
let content_str = String::from_utf8_lossy(&content);
// The repackaged EPUB should not contain the malicious script.
assert!(
!content_str.contains("alert"),
"repackaged EPUB should not contain the script payload"
);
// Output should be a valid ZIP.
let _archive = zip::ZipArchive::new(std::io::Cursor::new(&content))
.expect("cleansed EPUB should be a valid ZIP");
}
#[test]
fn preserve_format_docx_strips_macro() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.cleanse_mode = CleanseMode::PreserveFormat;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("malicious_macro.docx")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
let cleansed = result.cleansed_path.expect("cleansed DOCX should exist");
let content = std::fs::read(&cleansed).unwrap();
let content_str = String::from_utf8_lossy(&content);
assert!(
!content_str.contains("vbaProject"),
"repackaged DOCX should not contain VBA project"
);
}
// ---------------------------------------------------------------------------
// PDF PreserveFormat repackage tests (v0.3.0)
// ---------------------------------------------------------------------------
#[test]
fn preserve_format_pdf_strips_javascript() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("malicious_js.pdf")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
let cleansed = result.cleansed_path.expect("cleansed PDF should exist");
let content = std::fs::read(&cleansed).unwrap();
let content_str = String::from_utf8_lossy(&content);
// The JavaScript action payload should be gone.
assert!(
!content_str.contains("app.alert"),
"repackaged PDF should not contain the JS payload, got:\n{}",
content_str
);
}
#[test]
fn preserve_format_pdf_strips_launch() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("malicious_launch.pdf")).unwrap();
assert!(result.scan_report.malicious_count() > 0);
let cleansed = result.cleansed_path.expect("cleansed PDF should exist");
let content = std::fs::read(&cleansed).unwrap();
let content_str = String::from_utf8_lossy(&content);
// The /Launch action should be removed from the catalog.
assert!(
!content_str.contains("/Launch"),
"repackaged PDF should not contain /Launch action"
);
}
// ---------------------------------------------------------------------------
// Abort-on-threat
// ---------------------------------------------------------------------------
#[test]
fn abort_on_threat_returns_error_for_malicious() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.abort_on_threat = true;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("malicious_js.pdf"));
assert!(result.is_err(), "abort-on-threat should return Err for malicious PDF");
let err = result.unwrap_err().to_string();
assert!(
err.contains("threat detected"),
"error should mention threat, got: {err}"
);
}
#[test]
fn abort_on_threat_succeeds_for_benign() {
let tmp = tempdir().unwrap();
let mut config = Config::with_workspace(tmp.path());
config.abort_on_threat = true;
let pipeline = Pipeline::with_config(config);
let result = pipeline.run(fixture("benign.pdf"));
assert!(result.is_ok(), "abort-on-threat should succeed for benign PDF");
}
// ---------------------------------------------------------------------------
// Educational whitelisting
// ---------------------------------------------------------------------------
#[test]
fn cve_writeup_md_whitelisted_as_educational() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("cve_writeup.md")).unwrap();
// CVE writeups should produce educational findings, not malicious.
let has_malicious = result
.scan_report
.findings
.iter()
.any(|f| matches!(f.classification, ThreatClassification::Malicious(_)));
assert!(!has_malicious, "CVE writeup should not produce malicious findings");
}
#[test]
fn cve_writeup_pdf_whitelisted_as_educational() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("cve_writeup.pdf")).unwrap();
let has_malicious = result
.scan_report
.findings
.iter()
.any(|f| matches!(f.classification, ThreatClassification::Malicious(_)));
assert!(!has_malicious, "CVE writeup PDF should not produce malicious findings");
}
// ---------------------------------------------------------------------------
// Report content verification
// ---------------------------------------------------------------------------
#[test]
fn json_report_contains_schema_version() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_js.pdf")).unwrap();
let report_path = result
.json_report_path
.expect("JSON report should exist for malicious PDF");
let report_str = std::fs::read_to_string(&report_path).unwrap();
let report: serde_json::Value = serde_json::from_str(&report_str).unwrap();
assert_eq!(
report["schema_version"], 1,
"report should have schema_version = 1"
);
assert!(
report["findings"].as_array().unwrap().len() > 0,
"report should have at least one finding"
);
assert!(
report["source"].as_object().is_some(),
"report should have a source object"
);
}
#[test]
fn quarantine_tarball_contains_original_file() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let result = pipeline.run(fixture("malicious_js.pdf")).unwrap();
let tarball_path = result
.quarantine_path
.expect("quarantine tarball should exist");
let tarball_gz = std::fs::File::open(&tarball_path).unwrap();
let decoder = flate2::read::GzDecoder::new(tarball_gz);
let mut archive = tar::Archive::new(decoder);
let entries: Vec<String> = archive
.entries()
.unwrap()
.filter_map(|e| e.ok())
.map(|e| e.path().unwrap().to_string_lossy().to_string())
.collect();
assert!(
entries.iter().any(|e| e.starts_with("original.")),
"quarantine tarball should contain original.pdf, entries: {entries:?}"
);
assert!(
entries.iter().any(|e| e == "report.json"),
"quarantine tarball should contain report.json, entries: {entries:?}"
);
}
// ---------------------------------------------------------------------------
// Total-archive-scan-cap (TODO #8)
// ---------------------------------------------------------------------------
#[test]
fn total_archive_cap_limits_cumulative_reads() {
// Parse an EPUB with a very low total cap.
let bytes = std::fs::read(fixture("benign.epub")).unwrap();
let mut config = Config::default();
config.total_archive_scan_cap = 100; // 100 bytes — absurdly low.
let doc = Dispatcher::parse(&bytes, DocumentFormat::Epub, None, &config).unwrap();
// Some entries should have been skipped due to budget exhaustion.
let has_budget_msg = doc
.executable_vectors
.iter()
.any(|v| {
v.decoded_preview
.as_deref()
.map(|p| p.contains("total-archive-budget exhausted"))
.unwrap_or(false)
});
assert!(
has_budget_msg,
"with 100-byte total cap, some entries should be budget-exhausted"
);
}
// ---------------------------------------------------------------------------
// Payload carving v2: .hex and .info files
// ---------------------------------------------------------------------------
#[test]
fn quarantine_produces_hex_and_info_files() {
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
let _result = pipeline.run(fixture("malicious_js.pdf")).unwrap();
let q_dir = tmp.path().join("corbel_quarantine");
let hex_files: Vec<_> = std::fs::read_dir(&q_dir)
.unwrap()
.filter_map(|e| e.ok())
.filter(|e| {
e.path()
.extension()
.and_then(|ext| ext.to_str())
.map(|ext| ext == "hex")
.unwrap_or(false)
})
.collect();
assert!(
!hex_files.is_empty(),
"quarantine should produce at least one .hex file"
);
// Verify the .hex file has the xxd-style format.
let first_hex = std::fs::read_to_string(hex_files[0].path()).unwrap();
assert!(
first_hex.starts_with("00000000:"),
".hex file should start with offset, got: {first_hex:?}"
);
// Verify .info files exist too.
let info_files: Vec<_> = std::fs::read_dir(&q_dir)
.unwrap()
.filter_map(|e| e.ok())
.filter(|e| {
e.path()
.extension()
.and_then(|ext| ext.to_str())
.map(|ext| ext == "info")
.unwrap_or(false)
})
.collect();
assert!(
!info_files.is_empty(),
"quarantine should produce at least one .info file"
);
// Verify the .info file is valid JSON with required fields.
let first_info_str = std::fs::read_to_string(info_files[0].path()).unwrap();
let info: serde_json::Value = serde_json::from_str(&first_info_str).unwrap();
assert!(info.get("filename").is_some(), ".info should have filename");
assert!(
info.get("payload_sha256").is_some(),
".info should have payload_sha256"
);
assert!(
info.get("payload_size_bytes").is_some(),
".info should have payload_size_bytes"
);
assert!(
info.get("classification").is_some(),
".info should have classification"
);
}

190
tests/zip_bomb_defense.rs Executable file
View File

@ -0,0 +1,190 @@
//! Zip-bomb defense integration tests.
//!
//! These tests verify that the streaming `read_with_cap` defense
//! actually works against:
//!
//! 1. **Lying size header**: a ZIP that declares `size = 100` but
//! actually decompresses to 1 MiB.
//! 2. **Honest but oversized**: a ZIP that honestly declares 1 MiB
//! and decompresses to 1 MiB.
//!
//! Both should be detected and truncated at the configured cap.
use std::path::PathBuf;
use corbel_purge::{Config, DocumentFormat, Pipeline};
use corbel_purge::core::types::VectorType;
use corbel_purge::parsers::{Dispatcher, DocumentParser};
use tempfile::tempdir;
fn fixtures_dir() -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures")
}
fn fixture(name: &str) -> PathBuf {
fixtures_dir().join(name)
}
#[test]
fn lying_zip_bomb_is_detected_and_truncated() {
// The lying_zip_bomb.zip fixture declares uncompressed size = 100
// but actually decompresses to 1 MiB. The streaming reader must
// detect this by counting actual bytes, not trusting the header.
let bytes = std::fs::read(fixture("lying_zip_bomb.zip")).unwrap();
let config = Config::default(); // epub_entry_scan_cap = 8 MiB
let result = Dispatcher::parse(
&bytes,
DocumentFormat::Epub,
None,
&config,
);
// The zip crate may or may not be able to read our hand-crafted
// lying ZIP — if it errors out, that's also a valid defense
// (the malicious file is rejected). If it succeeds, we should
// have either:
// - An UnknownPayload vector (truncation detected), OR
// - A successfully-parsed-but-flagged entry.
match result {
Ok(doc) => {
// The entry was read. Verify that either:
// (a) it was truncated (UnknownPayload present), or
// (b) the entry was small enough to fit (unlikely given
// the 1 MiB actual size vs 8 MiB cap — but possible
// if the zip crate clamped to the declared size).
let has_unknown_payload = doc
.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::UnknownPayload);
let _ = has_unknown_payload; // informational
// Either way, we didn't crash or OOM.
}
Err(e) => {
// The zip crate rejected the lying ZIP — also a valid
// defense. Just make sure it's a parse error, not a panic.
eprintln!("zip crate rejected lying ZIP: {e}");
}
}
}
#[test]
fn honest_zip_bomb_is_truncated_at_cap() {
// The honest_zip_bomb.zip fixture honestly declares 1 MiB and
// decompresses to 1 MiB. With the default 8 MiB cap, this fits
// and should be read completely. But if we lower the cap to
// 100 KiB, the streaming reader should truncate.
let bytes = std::fs::read(fixture("honest_zip_bomb.zip")).unwrap();
// First, with the default 8 MiB cap — should succeed and the
// entry should be read completely.
let default_config = Config::default();
let doc = Dispatcher::parse(
&bytes,
DocumentFormat::Epub,
None,
&default_config,
)
.expect("default 8 MiB cap should allow 1 MiB entry");
// The entry is "bomb.txt" — unknown extension, so it gets emitted
// as UnknownPayload regardless of size.
assert!(
doc.executable_vectors
.iter()
.any(|v| v.vector_type == VectorType::UnknownPayload),
"the bomb.txt entry should be classified as UnknownPayload"
);
// Verify the full 1 MiB was read (not truncated).
let bomb_vector = doc
.executable_vectors
.iter()
.find(|v| v.vector_type == VectorType::UnknownPayload)
.unwrap();
assert_eq!(
bomb_vector.raw_payload.len(),
1024 * 1024,
"with 8 MiB cap, the 1 MiB entry should be read in full"
);
// Now lower the cap to 100 KiB and verify truncation.
let mut small_cap_config = Config::default();
small_cap_config.epub_entry_scan_cap = 100 * 1024; // 100 KiB
let doc_small = Dispatcher::parse(
&bytes,
DocumentFormat::Epub,
None,
&small_cap_config,
)
.expect("parse should still succeed (just truncate the entry)");
let bomb_vector_small = doc_small
.executable_vectors
.iter()
.find(|v| v.vector_type == VectorType::UnknownPayload)
.expect("truncated entry should still be emitted as UnknownPayload");
// The streaming reader should have stopped at ~100 KiB, not 1 MiB.
assert!(
bomb_vector_small.raw_payload.len() <= 100 * 1024,
"with 100 KiB cap, entry should be truncated to ≤100 KiB, got {} bytes",
bomb_vector_small.raw_payload.len()
);
assert!(
bomb_vector_small.raw_payload.len() > 0,
"truncated entry should still contain some bytes (for signature matching)"
);
// The decoded_preview should mention the truncation.
let preview = bomb_vector_small
.decoded_preview
.as_deref()
.expect("preview should exist");
assert!(
preview.contains("truncated") || preview.contains("oversized"),
"preview should mention truncation, got: {preview}"
);
}
#[test]
fn pipeline_with_lying_zip_bomb_does_not_oom() {
// End-to-end: run the full pipeline against the lying zip bomb.
// The key assertion: the process doesn't crash or hang.
let tmp = tempdir().unwrap();
let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path()));
// We don't care whether it succeeds or fails — only that it
// doesn't panic or OOM. Either outcome is a valid defense.
let _ = pipeline.run(fixture("lying_zip_bomb.zip"));
}
#[test]
fn streaming_read_never_exceeds_cap() {
// Direct unit test of the streaming reader's core guarantee:
// no matter how much data the stream produces, we never allocate
// more than `cap` bytes.
use corbel_purge::util::{read_with_cap, ReadOutcome};
use std::io::Cursor;
// 10 MiB of data, 1 KiB cap.
let data = vec![0x42u8; 10 * 1024 * 1024];
let mut cursor = Cursor::new(data);
let cap = 1024;
let outcome = read_with_cap(&mut cursor, cap).unwrap();
match outcome {
ReadOutcome::Truncated { bytes, bytes_read, cap: returned_cap } => {
assert_eq!(returned_cap, cap);
assert!(
bytes.len() <= cap,
"bytes.len() ({}) must be ≤ cap ({})",
bytes.len(),
cap
);
assert_eq!(bytes_read, bytes.len());
}
ReadOutcome::Complete(_) => {
panic!("10 MiB stream with 1 KiB cap should have truncated");
}
}
}