commit 7992a48d109431d1748172aa300fb1810f61de91 Author: Jeremy Anderson Date: Wed Aug 12 16:52:57 2026 -0400 CorbelPurge parses documents into a unified intermediate representation (Document struct), runs a layered contextual scanner that distinguishes educational security literature from active malicious injections, and produces cleansed derivatives with all executable content stripped. Malicious payloads are carved into quarantine tarballs with full forensic reports.' diff --git a/BLOG.md b/BLOG.md new file mode 100644 index 0000000..c74e587 --- /dev/null +++ b/BLOG.md @@ -0,0 +1,83 @@ +# Announcing CorbelPurge v0.4.2 + +*A defensive document research and cleaning tool for security analysts.* + +## Why This Exists + +Security researchers have a problem. When a suspicious PDF lands in your inbox or a weaponized DOCX turns up in a malware corpus, your options for safely inspecting it are limited. Open it in Adobe Reader and you might trigger the payload. Open it in LibreOffice and the macro runs. Send it to VirusTotal and you get a score but not an understanding. + +CorbelPurge is built to fill that gap. It is a strict-Rust document sanitizer that never executes embedded scripts, macros, or active content -- but still gives you a full understanding of what is inside the file, then carves the malicious bits into a quarantine tarball with forensic reports and leaves behind a cleansed derivative you can actually read. + +## What CorbelPurge Does + +- **Parses** PDF, EPUB, DOCX, and Markdown files into a unified intermediate representation (`Document` struct with `TextNode` and `ExecutableVector` items) +- **Scans** with a two-pass contextual engine: executable vectors go through `heuristics::classify_vector()` against file-signature, shellcode, phishing, and URI-allowlist tables; text nodes go through `context_filter::evaluate()` which distinguishes weaponized content from educational security literature (CVE writeups, code blocks, academic language) +- **Tags** known exploits automatically -- 11 entries in the built-in CVE table (CVE-2017-11882, CVE-2018-4990, CVE-2010-0188, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199, CVE-2012-0158, CVE-2015-2545, CVE-2021-40444, CVE-2022-30190, and EPUB-SCRIPT-INJECTION) +- **Quarantines** extracted payloads into a compressed `quarantine__.tar.gz` containing the original file, a JSON report, a Markdown report, and one `.bin` per carved payload -- each paired with a `.hex` annotated hex dump and a `.info` JSON metadata file +- **Cleanses** documents so you can read them safely, either as Markdown (default) or in the original format via `--preserve-format` (EPUB entries stripped from the ZIP, DOCX macros and embeddings removed, PDF objects deleted via lopdf) +- **Studies** documents in place via `corbel-purge study `, which renders the original to a single annotated HTML file with malicious regions wrapped in inline `` tags color-coded by classification +- **Defends** against zip-bomb attacks with streaming byte-counting (`util::read_with_cap()`) that counts actual decompressed bytes rather than trusting ZIP central-directory size headers, plus a cumulative `total_archive_scan_cap` (default 256 MiB) across all entries in a multi-entry archive + +## What Is New in v0.4.2 + +### Glyphfix (carried from v0.4.1) + +The iced 0.13 GUI previously rendered 11 Unicode glyphs as tofu boxes because iced's default embedded font (a subset of DejaVu Sans) does not cover them. Every problematic glyph was swapped for an ASCII equivalent that is guaranteed to render: shield emoji became `[+]`, block characters became `#` and `-`, the warning sign became `!`, and so on. No new dependencies, no embedded fonts, no binary-size regression. + +### About / License overlay (carried from v0.4.1) + +The ABOUT / LICENSE button in the GUI footer is now wired up. Clicking it opens a floating info panel anchored to the top-right corner -- dark panel, gold border, drop shadow, close button in the header, structured metadata (title, version, description, author, website, license, tech stack, copyright). A 55%-opacity black backdrop dims the underlying UI while the panel is open. + +### What else changed + +- **Version bumped** to 0.4.2 +- **Docs** (`README.md`, `MANIFEST.md`, `TODO.md`, `FIX-NOTES-glyphfix.md`, the `pdf_parser.rs` header comment) all updated to drop pdf-render references +- **Test suite** unchanged: 154 tests still pass (127 unit + 23 integration + 4 zip-bomb defense) + +## What Came Before + +- **v0.4.1:** Glyphfix patch, About / License overlay wired up +- **v0.3.0:** Study mode (`corbel-purge study`), payload carving v2 (`.hex` + `.info` files alongside `.bin`), EPUB/Markdown cleansed-document viewer in the GUI, external threat-intel feeds (`--rules` and `--cve-db` flags + `ExternalRulesData` static storage), expanded CVE table (4 new entries: CVE-2012-0158, CVE-2015-2545, CVE-2021-40444, CVE-2022-30190), total-memory budget for multi-entry archives, PDF repackage dangling-reference cleanup, OPF manifest cleanup after EPUB repackage, GUI CVE badges parsed from `Finding.context_notes` +- **v0.2.0:** 4-format parsing (PDF via `lopdf`, EPUB via `zip`, Markdown via `pulldown-cmark`, DOCX via `zip` + XML regex), unified intermediate representation, layered contextual scanner, CVE tagging, quarantine packaging, document cleansing (Markdown + PreserveFormat modes), zip-bomb defense, iced 0.13 GUI dashboard + +## Who Is This For? + +- **Incident responders** who need to safely inspect suspicious attachments +- **Threat researchers** studying document-based exploits from malware corpora +- **Security teams** who want CI/CD scanning for incoming documents (`--abort-on-threat` exits 2 on any malicious finding) +- **Academics** analyzing CVE write-ups and exploit techniques (the context filter whitelists educational content so a CVE writeup that quotes shellcode does not get flagged) +- **Anyone** who receives a document from an untrusted source and wants to read it without risking code execution + +## What CorbelPurge Is Not + +CorbelPurge does not create, generate, weaponize, or distribute exploits. It does not execute embedded scripts, macros, or active content. It is not an offensive security tool. It is a defensive research instrument. + +## Get Started + +```bash +tar xzf corbel-purge-0.4.2.tar.gz +cd corbel-purge-0.4.2 +cargo build --release +./target/release/corbel-purge scan path/to/suspicious.pdf +``` + +See [QUICKSTART.md](QUICKSTART.md) for the full step-by-step guide, including +the `study` subcommand, PreserveFormat mode, CI integration, external +threat-intel feeds, and the optional iced GUI build. + +## Roadmap Preview + +The highest-priority items currently on the deferred list: + +- **Bundled historical exploit corpus** -- too risky to ship; would bloat the repo +- **Sandbox hardening / fuzzing the tool itself** -- the tool is not the target per the threat model +- **Encrypted quarantine tarballs** -- unnecessary for the research use case; disk encryption is the operator's responsibility +- **Full OOXML schema validation** -- the current coarse regex extraction in `docx_parser.rs` is sufficient for the threat model +- **Magic-byte sniffing** -- `DocumentFormat::from_path()` uses extension only; a mismatched extension causes a parse error (safe failure mode) +- **Full syntect syntax highlighting** -- the cleansed-document viewer uses a lightweight dark-panel approach; `syntect` would add a heavy dependency to a security-critical crate + +See [TODO.md](TODO.md) for the full development roadmap. + +## License + +GNU General Public License v3.0-or-later. Free software for research, security analysis, and academic study. See [LICENSE](LICENSE) for full terms. diff --git a/Cargo.toml b/Cargo.toml new file mode 100755 index 0000000..f794c84 --- /dev/null +++ b/Cargo.toml @@ -0,0 +1,71 @@ +[package] +name = "corbel-purge" +version = "0.4.2" +edition = "2021" +description = "Strict Rust secure document sanitizer & threat neutralizer for PDF, EPUB, Markdown, and DOCX" +license = "GPL-3.0-or-later" +authors = ["Jeremy Anderson"] +repository = "https://git.dcos.net/dcosnet/corbel" +homepage = "https://git.dcos.net/dcosnet/corbel" + +[[bin]] +name = "corbel-purge" +path = "src/main.rs" + +[[bin]] +name = "corbel-purge-gui" +path = "src/bin/gui.rs" +required-features = ["gui"] + +[lib] +name = "corbel_purge" +path = "src/lib.rs" + +[features] +default = [] +# Enable the iced GUI binary. When disabled, the crate compiles as a headless +# CLI + library only β€” useful for CI and server-side scanning pipelines. +gui = ["dep:iced", "dep:rfd", "dep:tokio"] + +[dependencies] +# --- PDF structure inspection --- +lopdf = "0.34" + +# --- EPUB (ZIP container) inspection --- +# Using `zip` directly gives us byte-level control over the EPUB container, +# which the security scanner needs (we have to inspect raw stream bytes +# inside each ZIP entry, not just rendered XHTML). +zip = "2" + +# --- Markdown parsing --- +pulldown-cmark = { version = "0.12", default-features = false } + +# --- Quarantine packaging --- +tar = "0.4" +flate2 = "1" + +# --- Serialization & hashing --- +serde = { version = "1", features = ["derive"] } +serde_json = "1" +sha2 = "0.10" +hex = "0.4" + +# --- Misc utilities --- +thiserror = "2" +regex = "1" +chrono = { version = "0.4", features = ["serde"] } + +# --- Optional GUI (iced) --- +iced = { version = "0.13", optional = true, features = ["tokio", "debug"] } +rfd = { version = "0.15", optional = true } +tokio = { version = "1", optional = true, features = ["rt-multi-thread", "macros", "fs"] } + +[dev-dependencies] +tempfile = "3" +pretty_assertions = "1" + +[profile.release] +opt-level = 3 +lto = "thin" +codegen-units = 1 +strip = "symbols" diff --git a/FIX-NOTES-glyphfix.md b/FIX-NOTES-glyphfix.md new file mode 100644 index 0000000..b4a97c8 --- /dev/null +++ b/FIX-NOTES-glyphfix.md @@ -0,0 +1,133 @@ +# Patch notes β€” 0.4.1-glyphfix + about-overlay + +## Patch 1: Glyphfix + +The GUI (`src/bin/gui.rs`) used 11 Unicode glyphs that are not present in +iced 0.13's default embedded font (a subset of DejaVu Sans). Every missing +glyph rendered as a tofu box (β–‘) on screen. + +Affected locations (visible in the original screenshot): + +| Glyph | Code point | Where it appeared | +|-------|------------|-------------------| +| πŸ›‘ | U+1F6E1 | Brand icon, before "CORBELPURGE" | +| β–΄ β–Ύ | U+25B4 / U+25BE | Chevrons before PATHS / OPTIONS / CONSOLE | +| β–ˆ | U+2588 | CONSOLE section label + gauge filled cells | +| β–‘ | U+2591 | Gauge empty cells (the 16-box string) | +| ✨ | U+2728 | CLEANED stat | +| πŸ“„ | U+1F4C4 | COPIED stat | +| ⚠ | U+26A0 | ERRORS stat | +| ⏻ | U+23FB | START PROCESSING button | +| 🧹 | U+1F9F9 | CLEAR LOG button | +| β„Ή | U+2139 | ABOUT / LICENSE button | +| β€’ | U+2022 | Findings bullet (only visible with findings) | + +### Note on the `pdf-render` glyphs + +An earlier version of this patch also documented three glyphs that only +appeared when the optional `pdf-render` feature was enabled (πŸ“ on the +IN/OUT browse buttons, β—€ β–Ά on PDF prev/next page buttons, πŸ–± on the +"view original PDF" button). The `pdf-render` feature was removed in +v0.4.2 β€” CorbelPurge scans four formats (PDF/EPUB/MD/DOCX) but only +PDF had a visual renderer, which was an inconsistency that wasn't worth +the pdfium dynamic-library dependency. Those three rows are kept out of +the table above since the code paths no longer exist. + +### Fix + +Swapped every problematic glyph for an ASCII equivalent that is guaranteed +to render in iced's default font. No new dependencies, no embedded fonts, +no binary-size regression. + +| Old | New | Rationale | +|---------------|------|-----------| +| πŸ›‘ | `[+]` | "protected" badge feel | +| β–΄ (open) | `-` | CLI-standard "expanded" marker | +| β–Ύ (closed) | `+` | CLI-standard "collapsed" marker | +| β–ˆ (filled) | `#` | block character everyone has | +| β–‘ (empty) | `-` | clean empty-track look | +| ✨ | `*` | clean/done marker | +| πŸ“„ | `>` | "copied out" arrow | +| ⚠ | `!` | universal warning | +| ⏻ | `>` | "start" arrow | +| 🧹 | `x` | universal clear/delete | +| β„Ή | `i` | universal info | +| β€’ | `*` | bullet, safe everywhere | +| πŸ“ | `...` | standard "browse" indicator | + +### Alternative (not applied) + +If you want prettier icons later, embed the `iced_fonts` crate +(https://crates.io/crates/iced_fonts) which bundles Noto Sans Symbols 2 +and Bootstrap/Material icon fonts. Then restore the original glyphs and +apply `.font(iced_fonts::REQUIRED_FONT)` to each `text()` call. This adds +~600 KB to the binary but gives you proper iconography. + +--- + +## Patch 2: About / License overlay + +### Problem + +The ABOUT / LICENSE button in the footer was wired up to nothing β€” it had +no `.on_press(...)` handler, so clicking it did nothing. + +### Fix + +Added a floating info panel overlay, modelled on the ferret about-panel +screenshot. Visual style: + +- Dark panel background (`colors::panel()`) +- Gold border (`colors::gold()`, 1.5px, 6px rounded corners) +- Drop shadow (offset 4px down, 12px blur, 60% opacity) +- Close (X) button in the top-right corner of the panel header +- Structured content: + - Title "CORBELPURGE" + version (from `CARGO_PKG_VERSION`) + - Two-line description (from `Cargo.toml` description) + - Metadata rows: Author / Website / License (last two from + `CARGO_PKG_REPOSITORY` and `CARGO_PKG_LICENSE`) + - Thin separator + - Footer: tech stack + copyright + +### Behavior + +- Clicking ABOUT / LICENSE toggles the overlay open/closed. +- Clicking the X button (or pressing ABOUT / LICENSE again) closes it. +- When open, the main UI is dimmed (55% opacity black backdrop). +- No click-outside-to-close β€” see Implementation notes below. + +### Implementation notes + +iced 0.13 has no `Stack` widget (true non-modal overlays landed in +0.14). To emulate the ferret "floating panel over content" look without +adding a dependency, we use a full-window dim backdrop with the panel +positioned in the top-right via a `row + Space::Fill` layout. + +Click-outside-to-close was considered but dropped: wrapping the panel +in a no-op `Button` would make every label inside the panel close the +modal on click (since iced 0.13 buttons don't stop event propagation to +their parent). ESC-key handling requires a keyboard subscription, which +is a separate feature. + +### Future enhancements + +1. **Upgrade to iced 0.14+** to get the `Stack` widget, enabling true + non-modal floating panels with click-outside-to-close. +2. **Add a keyboard subscription** for the Escape key to close the + overlay without needing to click the X button. +3. **Embed `iced_fonts`** and restore the original emoji glyphs for a + richer visual style. +4. **Make the website URL clickable** β€” currently it's just colored + text. iced 0.13 doesn't have a native hyperlink widget, but you + could shell out to `xdg-open` via a button. + +### Files changed + +- `src/bin/gui.rs`: + - Added `about_open: bool` to `CorbelGui` state + - Added `ToggleAbout` and `CloseAbout` Message variants + update handlers + - Wired ABOUT / LICENSE button's `on_press` + - Added `build_about_panel()` and `build_about_overlay()` functions + - Added `about_dim_style()`, `about_panel_style()`, `about_close_btn_style()` style helpers + - Modified `view()` to render the overlay when `about_open` is true +- No changes to `Cargo.toml` β€” no new dependencies added diff --git a/MANIFEST.md b/MANIFEST.md new file mode 100755 index 0000000..28dae67 --- /dev/null +++ b/MANIFEST.md @@ -0,0 +1,438 @@ +# Project Manifest: CorbelPurge (v0.3.0) + +## Strict Rust Secure Document Sanitizer & Threat Neutralizer + +**Author:** Jeremy Anderson β€” https://git.dcos.net/dcosnet/corbel +**License:** GPL-3.0-or-later + +**Status:** v0.2.0 MVP shipped. v0.3.0 planning: study mode, payload carving v2, +scanner extensibility, GUI navigation improvements. + +--- + +## 1. Executive Summary + +CorbelPurge is a high-assurance document sanitizer written in strict Rust +(`#![forbid(unsafe_code)]`) for **PDF, EPUB, Markdown, and DOCX** formats. +It parses documents into a unified intermediate representation, runs a +layered contextual scanner that distinguishes educational security +literature from active malicious injections, then produces cleansed +derivatives with all executable content stripped. Malicious payloads are +carved into quarantine tarballs with full forensic reports. + +**Threat model:** CorbelPurge is a researcher's tool for studying old +exploits, not a target itself. The scanner safely ingests adversarial +samples (zip bombs, malicious docs from malware corpora) without +choking, so the researcher can study them. The tool itself is assumed +to not be the target. + +--- + +## 2. Core Specifications & Non-Goals + +### Core Features + +- **Multi-Format Support:** Native parsing pipelines for PDF (`lopdf`), + EPUB (`zip`), Markdown (`pulldown-cmark`), DOCX (`zip` + XML regex). +- **Unified Intermediate Representation:** All parsers emit the same + `Document` struct (`src/core/types.rs`). Downstream modules never +touch format-specific structures. +- **Automated Threat Scanning:** Two-pass engine β€” executable vectors + go through `heuristics::classify_vector()`, text nodes go through + `context_filter::evaluate()`, then findings are annotated by + `cve_tags::match_cve()`. +- **Context-Aware Heuristics:** Differentiation between static text + content (including code blocks discussing vulnerabilities) and + executable/malicious injection vectors. Academic markers (CVE IDs, + "remediation", code-block context) whitelist educational content. +- **Weaponization Detection:** Long hex-encoded runs (16+ `\xNN`), + 64+ char base64 blobs, NOP sleds, Metasploit stager prologues are + flagged as malicious regardless of context. +- **CVE Tagging:** 7 known exploits (CVE-2010-0188, CVE-2018-4990, + CVE-2017-11882, CVE-2018-0802, CVE-2017-8570, CVE-2017-0199, + EPUB-SCRIPT-INJECTION) with byte-signature and vector-type matching. +- **Forensic Extraction & Reporting:** JSON + Markdown reports with + SHA-256, vector location, classification, CVE tag, context notes. + Payload carving into standalone `.bin` files inside quarantine tarballs. +- **Secure Quarantine:** Compressed `quarantine__.tar.gz` + containing original file, report, and extracted payloads. +- **Document Cleansing (two modes):** + - *Markdown mode* (default): produces a safe Markdown derivative via + `cleanse::sanitizer::sanitize()`. A Markdown file cannot carry + executable content by definition. + - *PreserveFormat mode* (`--preserve-format`): repackages the document + in its original format (PDF/EPUB/DOCX) with malicious entries + stripped via `cleanse::repackage::repackage()`. +- **Zip-Bomb Defense:** `util::read_with_cap()` counts actual decompressed + bytes rather than trusting ZIP central directory size headers. + Configurable via `Config::epub_entry_scan_cap` (default 8 MiB). +- **Optional GUI:** iced 0.13 dashboard (`src/bin/gui.rs`) with dark + theme, collapsible panels, console log, sidebar stats, file picker + (`rfd::AsyncFileDialog`). +- **No-arg-parsing-crates:** The CLI (`src/main.rs`) parses its own + arguments to keep the dependency tree minimal β€” a defensive measure + for security-critical software. + +### Explicit Non-Goals + +- No document editing, annotation writing, form-filling, or signature generation. +- No execution of embedded active scripting (PDF JS, EPUB scripts, VBA macros). +- No sandbox hardening or fuzzing of the tool itself (it is not the target). +- No bundled historical exploit corpus (too risky to ship). +- No magic-byte sniffing for format detection (extension-only is the safe + failure mode β€” a mismatched extension causes a parse error). + +--- + +## 3. System Architecture & Tech Stack + +CorbelPurge uses safe Rust primitives, relying on memory-safe parsers to +avoid buffer overflow vulnerabilities common in legacy C/C++ document +viewers. The `#![forbid(unsafe_code)]` crate attribute is enforced at +compile time. + +### Dependency Table + +| Purpose | Crate | Version | +|---------|-------|---------| +| PDF structure inspection | `lopdf` | 0.34 | +| EPUB / DOCX containers | `zip` | 2 | +| Markdown parsing | `pulldown-cmark` | 0.12 | +| Quarantine packaging | `tar` + `flate2` | 0.4 / 1 | +| Serialization | `serde` + `serde_json` | 1 | +| Hashing | `sha2` + `hex` | 0.10 / 0.4 | +| Datetime | `chrono` | 0.4 | +| Context filter | `regex` | 1 | +| Error handling | `thiserror` | 2 | +| GUI (optional) | `iced` | 0.13 | +| File picker (optional) | `rfd` | 0.15 | +| Async runtime (optional) | `tokio` | 1 | +| Test utilities | `tempfile` + `pretty_assertions` | 3 / 1 | + +### Feature Flags + +- `default` β€” headless CLI + library only. No GUI dependencies. +- `gui` β€” adds the iced GUI binary (`corbel-purge-gui`) and depends + on `iced`, `rfd`, `tokio`. + +--- + +## 4. Unified Intermediate Representation + +Every parser emits the same [`Document`](src/core/types.rs) struct. +The scanner, quarantine, and cleanse modules consume only this UIR β€” +they never touch format-specific structures. Adding a new format +requires writing exactly one new `DocumentParser` impl and adding a +match arm in `Dispatcher::parse()` (`src/parsers/mod.rs`). + +Key types defined in `src/core/types.rs`: + +- **`DocumentFormat`** β€” `Pdf`, `Epub`, `Markdown`, `Docx`. Detected by + file extension via `DocumentFormat::from_path()` in `pipeline.rs`. +- **`Location`** β€” per-format source location (PDF object/stream IDs, + EPUB ZIP entry paths, Markdown line:col). Used in findings to report + exactly where a threat was found. +- **`TextContext`** β€” semantic context of extracted text (`Paragraph`, + `Heading`, `CodeBlock`, `CodeSpan`, `Hyperlink`, `BlockQuote`, + `Metadata`, `ExecutableHook`). The heart of the context-aware scanner. +- **`TextNode`** β€” extracted text with location + context. Emitted by + parsers for all rendered/static content. +- **`ExecutableVector`** β€” a potentially dangerous element (JS action, + embedded file, script tag, OLE object, external link) with raw bytes + and optional decoded preview. +- **`VectorType`** β€” 15 variants covering PDF (JS, Launch, URI, GoToR, + EmbeddedFile, WidgetAction, AcroForm), EPUB (Script, ExternalResource, + Object), Markdown (Hyperlink), DOCX (Macro, ExternalLink, + EmbeddedObject, ActiveX), plus UnknownPayload. +- **`Document`** β€” the top-level IR: format, raw bytes, SHA-256, + metadata, text nodes, executable vectors. +- **`ThreatClassification`** β€” `Benign`, `Suspicious`, `EducationalContent`, + `Malicious(MaliciousType)`. +- **`MaliciousType`** β€” 8 specific categories (ActiveJavaScriptInjection, + LaunchAction, MaliciousEmbeddedFile, ObfuscatedShellcode, + SuspiciousUri, EpubActiveScript, DocxActiveContent, Other). +- **`Finding`** β€” classification, location, vector type, payload preview, + context notes, recommendation. +- **`Recommendation`** β€” `Allow`, `WhitelistAsEducational`, `Quarantine`, + `QuarantineAndCleanse`. +- **`ScanReport`** β€” aggregate of all findings with counts and metadata. +- **`CorbelError`** β€” exhaustive error enum (Io, Serde, PdfParse, + EpubParse, MarkdownParse, UnknownFormat, ThreatDetected, Quarantine, + Cleanse, Internal). + +--- + +## 5. Security Scanning Pipeline + +The pipeline is orchestrated by `Pipeline::run()` in `src/core/pipeline.rs`. +Given an input path, it: reads bytes, detects format, dispatches to the +correct parser, runs the scanner, then conditionally quarantines and +cleanses. + +### Step 0: Weaponization Check + +`context_filter::has_weaponization_indicators()` runs first on every text +node. Pure obfuscation indicators (16+ `\xNN` runs, 64+ char base64 +blobs, 2+ shell commands in non-code context) are flagged as +`ObfuscatedShellcode` regardless of surrounding context. Code blocks +and block quotes get an exception β€” weaponized-looking content there is +almost certainly a research writeup. + +### Step 1: Structural vs. Textual Isolation + +The scanner in `scanner::scan()` walks two separate lists: + +- **Executable Vectors** (untrusted by default) β€” dispatched to + `heuristics::inspect_vector()` which calls `classify_vector()`. + This function matches against `signatures.rs` tables (file signatures, + shellcode patterns, phishing TLDs, brand homographs, URL shorteners, + suspicious keywords) and the `Config::allowed_uri_schemes` whitelist. +- **Static Text Nodes** (context-dependent) β€” dispatched to + `context_filter::evaluate()` which looks for suspicious signatures + (`/JavaScript`, `eval(`, `shellcode`, `exploit`, etc.) and then + checks whether the surrounding context is educational. + +### Step 2: Context-Aware Filtering + +`context_filter::looks_educational()` checks for: + +- **Structural signals:** code blocks, code spans, block quotes β€” always + educational. +- **Lexical signals:** presence of academic markers ("CVE-", + "vulnerability", "remediation", "patch", "mitigation", "advisory", + "for example", "proof of concept", etc.) + +**Decision:** A signature in an `ExecutableHook` context is flagged as +malicious immediately. A signature in a code block describing a CVE +with remediation steps is whitelisted as `EducationalContent`. + +### Step 3: CVE Tagging + +After classification, `scanner::scan()` calls `cve_tags::match_cve()` +for each finding. If a CVE matches, the ID and name are appended to the +finding's `context_notes` field. The forensic reporter (`reporter.rs`) +includes the full CVE description. + +| CVE ID | Name | Detection Signal | +|--------|------|------------------| +| CVE-2017-11882 | Equation Editor RCE | DOCX OLE + "Equation" in payload | +| CVE-2018-0802 | Equation Editor RCE (variant) | DOCX OLE + PE signature, no "Equation" | +| CVE-2017-8570 | Office RTF OLE Object RCE | DOCX OLE + RTF magic bytes | +| CVE-2017-0199 | Office OLE2Link RCE | DOCX external link + `.hta` target | +| CVE-2018-4990 | Adobe Reader JS RCE | PDF JS action >500 bytes | +| CVE-2010-0188 | PDF LibTiff Buffer Overflow | PDF embedded file + TIFF magic | +| EPUB-SCRIPT-INJECTION | EPUB Active Script Injection | EPUB ` + +""", + ) + return path + + +def main(): + paths = [ + make_benign_md(), + make_cve_writeup_md(), + make_malicious_md(), + make_benign_epub(), + make_malicious_epub(), + ] + for p in paths: + print(f" wrote {p} ({p.stat().st_size} bytes)") + + +if __name__ == "__main__": + main() diff --git a/scripts/gen_zip_bomb_fixtures.py b/scripts/gen_zip_bomb_fixtures.py new file mode 100755 index 0000000..226234e --- /dev/null +++ b/scripts/gen_zip_bomb_fixtures.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Generate a "lying" zip-bomb test fixture for CorbelPurge. + +This crafts a ZIP archive where the central directory declares a small +uncompressed size (100 bytes) but the actual decompressed content is +much larger (1 MiB). This simulates a malicious archive that tries to +bypass size-header-based caps. + +The ZIP format is hand-crafted (not via the `zip` library) so we can +lie about the size. The structure is: + + [Local File Header][file data][Central Directory][End of Central Dir] + +Each file header has both a "compressed size" and "uncompressed size" +field. We set the central directory's "uncompressed size" to 100, but +write 1 MiB of actual data. A naive reader that trusts the header +would only allocate 100 bytes; a streaming reader counts actual bytes +and detects the lie. +""" + +import struct +import zlib +from pathlib import Path + +FIXTURES_DIR = Path(__file__).parent.parent / "tests" / "fixtures" +FIXTURES_DIR.mkdir(parents=True, exist_ok=True) + + +def make_lying_zip_bomb(): + """Create a ZIP where the declared uncompressed size is 100 bytes + but the actual decompressed content is 1 MiB. + + The ZIP is structurally valid (decompressors can read it) but the + central directory lies about the size. CorbelPurge's streaming + `read_with_cap` should detect this by counting actual bytes. + """ + # The actual content: 1 MiB of 'A' characters. + actual_content = b"A" * (1024 * 1024) + # Compress it with DEFLATE. + compressed = zlib.compress(actual_content, 9) + + # The "lie": declare the uncompressed size as 100 bytes. + declared_uncompressed_size = 100 + declared_compressed_size = len(compressed) # we don't lie about this + + # CRC32 of the actual content (the decompressor will compute this + # and we need to match it for the CRC check to pass). + crc = zlib.crc32(actual_content) & 0xFFFFFFFF + + # --- Local File Header --- + local_header = struct.pack( + " Color { Color::from([0.051, 0.051, 0.051]) } + pub fn panel() -> Color { Color::from([0.10, 0.10, 0.10 ]) } + pub fn panel_hi() -> Color { Color::from([0.13, 0.13, 0.13 ]) } + pub fn header_bg() -> Color { Color::from([0.07, 0.07, 0.07 ]) } + pub fn footer_bg() -> Color { Color::from([0.07, 0.07, 0.07 ]) } + pub fn gold() -> Color { Color::from([0.83, 0.69, 0.22 ]) } + pub fn white() -> Color { Color::from([0.92, 0.92, 0.92 ]) } + pub fn dim() -> Color { Color::from([0.55, 0.55, 0.58 ]) } + pub fn green() -> Color { Color::from([0.35, 0.85, 0.45 ]) } + pub fn red() -> Color { Color::from([0.90, 0.30, 0.30 ]) } + pub fn yellow() -> Color { Color::from([0.95, 0.75, 0.25 ]) } + pub fn teal() -> Color { Color::from([0.20, 0.65, 0.60 ]) } + pub fn input_bg() -> Color { Color::from([0.12, 0.12, 0.12 ]) } + pub fn border_dim() -> Color { Color::from([0.22, 0.22, 0.22 ]) } + pub fn console_bg() -> Color { Color::from([0.06, 0.06, 0.06 ]) } +} + +// --------------------------------------------------------------------------- +// Style helpers (iced 0.13 closure-based API) +// --------------------------------------------------------------------------- + +fn color_style(c: Color) -> impl Fn(&Theme) -> TextStyle { + move |_t: &Theme| TextStyle { color: Some(c) } +} + +fn panel_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::panel().into()), + border: Border { color: colors::border_dim(), width: 1.0, radius: 4.0.into() }, + ..iced::widget::container::Style::default() + } +} + +fn header_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::header_bg().into()), + ..iced::widget::container::Style::default() + } +} + +fn footer_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::footer_bg().into()), + border: Border { color: colors::border_dim(), width: 1.0, radius: 0.0.into() }, + ..iced::widget::container::Style::default() + } +} + +fn console_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::console_bg().into()), + border: Border { color: colors::border_dim(), width: 1.0, radius: 4.0.into() }, + ..iced::widget::container::Style::default() + } +} + +fn section_hdr_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::panel_hi().into()), + ..iced::widget::container::Style::default() + } +} + +/// iced 0.13 button style closures must accept `(&Theme, button::Status)`. +fn gold_btn_style() -> impl Fn(&Theme, iced::widget::button::Status) -> ButtonStyle { + move |_t: &Theme, _s: iced::widget::button::Status| ButtonStyle { + background: Some(colors::gold().into()), + text_color: Color::BLACK, + border: Border { color: colors::gold(), width: 0.0, radius: 4.0.into() }, + ..ButtonStyle::default() + } +} + +fn dim_btn_style() -> impl Fn(&Theme, iced::widget::button::Status) -> ButtonStyle { + move |_t: &Theme, _s: iced::widget::button::Status| ButtonStyle { + background: Some(colors::panel_hi().into()), + text_color: colors::dim(), + border: Border { color: colors::border_dim(), width: 1.0, radius: 4.0.into() }, + ..ButtonStyle::default() + } +} + +fn teal_btn_style() -> impl Fn(&Theme, iced::widget::button::Status) -> ButtonStyle { + move |_t: &Theme, _s: iced::widget::button::Status| ButtonStyle { + background: Some(colors::teal().into()), + text_color: Color::BLACK, + border: Border { color: colors::teal(), width: 0.0, radius: 4.0.into() }, + ..ButtonStyle::default() + } +} + +fn green_badge_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::green().into()), + border: Border { color: colors::green(), width: 1.0, radius: 3.0.into() }, + ..iced::widget::container::Style::default() + } +} + +fn separator_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::border_dim().into()), + ..iced::widget::container::Style::default() + } +} + +/// Shorthand: coloured text. +/// +/// `size` is `f32` to match iced 0.13's `Text::size` signature. +fn t(content: impl ToString, size: f32, color: Color) -> iced::widget::Text<'static> { + text(content.to_string()).size(size).style(color_style(color)) +} + +/// Shorthand: f32 padding (iced 0.13 Padding fields are f32). +fn pad(t: f32, r: f32, b: f32, l: f32) -> Padding { + Padding { top: t, right: r, bottom: b, left: l } +} + +// --------------------------------------------------------------------------- +// Application +// --------------------------------------------------------------------------- + +fn main() -> iced::Result { + iced::application("CorbelPurge", update, view) + .theme(|_| Theme::Dark) + .window_size((1100.0, 750.0)) + .run() +} + +// --------------------------------------------------------------------------- +// State +// --------------------------------------------------------------------------- + +#[derive(Debug, Default)] +struct CorbelGui { + input_path: String, + output_path: String, + preserve_format: bool, + strip_metadata: bool, + recursive: bool, + abort_on_threat: bool, + paths_open: bool, + options_open: bool, + console_open: bool, + about_open: bool, + console_lines: Vec, + is_processing: bool, + last_result: Option, + cleansed_content: Option, + error: Option, + processed: u32, + cleaned: u32, + copied: u32, + errors: u32, +} + +#[derive(Debug, Clone)] +struct ConsoleLine { + timestamp: String, + tag: String, + message: String, + is_error: bool, +} + +// --------------------------------------------------------------------------- +// Messages +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone)] +enum Message { + InputPathChanged(String), + OutputPathChanged(String), + BrowseInput, + BrowseOutput, + PreserveFormatToggled(bool), + StripMetadataToggled(bool), + RecursiveToggled(bool), + AbortOnThreatToggled(bool), + TogglePaths, + ToggleOptions, + ToggleConsole, + ToggleAbout, + CloseAbout, + StartProcessing, + StopProcessing, + ClearLog, + FilePickedInput(Option), + FilePickedOutput(Option), + ScanCompleted(Result), + OpenQuarantineFolder, +} + +// --------------------------------------------------------------------------- +// Update +// --------------------------------------------------------------------------- + +fn update(state: &mut CorbelGui, msg: Message) -> Task { + match msg { + Message::InputPathChanged(s) => { state.input_path = s; Task::none() } + Message::OutputPathChanged(s) => { state.output_path = s; Task::none() } + Message::BrowseInput => Task::perform(pick_file(), Message::FilePickedInput), + Message::BrowseOutput => Task::perform(pick_folder(), Message::FilePickedOutput), + Message::FilePickedInput(p) => { if let Some(p) = p { state.input_path = p.display().to_string(); } Task::none() } + Message::FilePickedOutput(p) => { if let Some(p) = p { state.output_path = p.display().to_string(); } Task::none() } + Message::PreserveFormatToggled(b) => { state.preserve_format = b; Task::none() } + Message::StripMetadataToggled(b) => { state.strip_metadata = b; Task::none() } + Message::RecursiveToggled(b) => { state.recursive = b; Task::none() } + Message::AbortOnThreatToggled(b) => { state.abort_on_threat = b; Task::none() } + Message::TogglePaths => { state.paths_open = !state.paths_open; Task::none() } + Message::ToggleOptions => { state.options_open = !state.options_open; Task::none() } + Message::ToggleConsole => { state.console_open = !state.console_open; Task::none() } + Message::ToggleAbout => { state.about_open = !state.about_open; Task::none() } + Message::CloseAbout => { state.about_open = false; Task::none() } + Message::ClearLog => { state.console_lines.clear(); state.processed = 0; state.cleaned = 0; state.copied = 0; state.errors = 0; Task::none() } + Message::StopProcessing => { state.is_processing = false; Task::none() } + + Message::StartProcessing => { + if state.input_path.is_empty() { + state.console_lines.push(ConsoleLine { + timestamp: now_hms(), + tag: "ERROR".into(), + message: "No input path specified.".into(), + is_error: true, + }); + state.errors += 1; + return Task::none(); + } + state.is_processing = true; + state.error = None; + let path = PathBuf::from(&state.input_path); + let ws = if state.output_path.is_empty() { None } else { Some(PathBuf::from(&state.output_path)) }; + let abort = state.abort_on_threat; + let pf = state.preserve_format; + state.console_lines.push(ConsoleLine { + timestamp: now_hms(), + tag: "INFO".into(), + message: format!("corbel-purge v{} starting...", env!("CARGO_PKG_VERSION")), + is_error: false, + }); + Task::perform(run_pipeline(path, ws, abort, pf), Message::ScanCompleted) + } + + Message::ScanCompleted(result) => { + state.is_processing = false; + state.processed += 1; + match result { + Ok(pr) => { + let malicious = pr.scan_report.malicious_count(); + let total = pr.scan_report.findings.len(); + let src = pr.source_path.as_ref() + .map(|p| p.display().to_string()) + .unwrap_or_else(|| "".into()); + + if malicious > 0 { + state.console_lines.push(ConsoleLine { + timestamp: now_hms(), + tag: "ERROR".into(), + message: format!("{} β€” {} finding(s) β€” cleaned", src, total), + is_error: true, + }); + state.cleaned += 1; + state.errors += 1; + } else { + state.console_lines.push(ConsoleLine { + timestamp: now_hms(), + tag: "OK".into(), + message: format!("{} clean β€” copied", src), + is_error: false, + }); + state.copied += 1; + } + + if let Some(cp) = &pr.cleansed_path { + match std::fs::read_to_string(cp) { + Ok(c) => state.cleansed_content = Some(c), + Err(e) => state.error = Some(format!("read cleansed: {e}")), + } + } + state.last_result = Some(pr); + } + Err(e) => { + state.console_lines.push(ConsoleLine { + timestamp: now_hms(), + tag: "ERROR".into(), + message: e.clone(), + is_error: true, + }); + state.errors += 1; + state.error = Some(e); + } + } + Task::none() + } + + Message::OpenQuarantineFolder => { + if let Some(r) = &state.last_result { + if let Some(q) = &r.quarantine_path { + if let Some(parent) = q.parent() { open_dir(parent); } + } + } + Task::none() + } + } +} + +// --------------------------------------------------------------------------- +// View +// --------------------------------------------------------------------------- + +fn view(state: &CorbelGui) -> Element<'_, Message> { + let header = build_header(); + let body = build_body(state); + let footer = build_footer(state); + let main = column![header, body, footer].spacing(0); + + if state.about_open { + // About overlay takes over the view: dim backdrop + floating panel + // anchored top-right. The main UI is hidden behind the dim layer + // (iced 0.13 has no Stack widget for true non-modal overlays). + build_about_overlay() + } else { + main.into() + } +} + +/// 48 px header. +fn build_header<'a>() -> Element<'a, Message> { + let brand = row![ + t("[+]", 16.0, colors::gold()), + t("CORBELPURGE", 16.0, colors::gold()), + t(format!(" v{}", env!("CARGO_PKG_VERSION")), 12.0, colors::dim()), + t(" // defensive document research & cleaning", 12.0, colors::dim()), + ].spacing(6.0); + + let license_badge = container(t("GPL-3.0", 11.0, Color::BLACK)) + .padding(pad(2.0, 8.0, 2.0, 8.0)) + .style(green_badge_style()); + + let header_row = row![brand, row![].width(Length::Fill), license_badge] + .align_y(Alignment::Center) + .padding(pad(12.0, 16.0, 12.0, 16.0)); + + container(header_row) + .width(Length::Fill) + .height(48) + .center_y(Length::Shrink) + .style(header_style()) + .into() +} + +/// Main body: left panel + right sidebar. +fn build_body(state: &CorbelGui) -> Element<'_, Message> { + let left = build_left_panel(state); + let right = build_right_sidebar(state); + row![left, right].spacing(0).height(Length::Fill).into() +} + +/// Collapsible left panel with PATHS / OPTIONS / CONSOLE. +fn build_left_panel(state: &CorbelGui) -> Element<'_, Message> { + let paths_section = collapsible_section( + "PATHS", + state.paths_open, + Message::TogglePaths, + build_paths_content(state), + ); + + let options_section = collapsible_section( + "OPTIONS", + state.options_open, + Message::ToggleOptions, + build_options_content(state), + ); + + let console_section = section_console(state); + + let left_col = column![paths_section, options_section, console_section] + .spacing(4.0) + .width(Length::Fill); + + container(left_col) + .width(Length::Fill) + .height(Length::Fill) + .padding(pad(6.0, 6.0, 6.0, 6.0)) + .into() +} + +fn build_paths_content(state: &CorbelGui) -> Element<'_, Message> { + let in_row = row![ + t("IN", 12.0, colors::gold()), + row![].width(Length::Fill), + button(t("...", 14.0, colors::dim())) + .style(dim_btn_style()) + .padding(pad(4.0, 4.0, 4.0, 4.0)) + .on_press(Message::BrowseInput), + ] + .spacing(8.0) + .align_y(Alignment::Center); + + let in_input = text_input("/path/to/input", &state.input_path) + .on_input(Message::InputPathChanged) + .padding(pad(6.0, 6.0, 6.0, 6.0)) + .size(12); + + let out_row = row![ + t("OUT", 12.0, colors::gold()), + row![].width(Length::Fill), + button(t("...", 14.0, colors::dim())) + .style(dim_btn_style()) + .padding(pad(4.0, 4.0, 4.0, 4.0)) + .on_press(Message::BrowseOutput), + ] + .spacing(8.0) + .align_y(Alignment::Center); + + let out_input = text_input("/path/to/output (workspace)", &state.output_path) + .on_input(Message::OutputPathChanged) + .padding(pad(6.0, 6.0, 6.0, 6.0)) + .size(12); + + column![in_row, in_input, out_row, out_input] + .spacing(6.0) + .padding(pad(8.0, 10.0, 8.0, 10.0)) + .into() +} + +fn build_options_content(state: &CorbelGui) -> Element<'_, Message> { + let t1 = toggler(state.preserve_format) + .label("PRESERVE FORMAT".to_string()) + .on_toggle(Message::PreserveFormatToggled); + let t2 = toggler(state.strip_metadata) + .label("STRIP METADATA".to_string()) + .on_toggle(Message::StripMetadataToggled); + let t3 = toggler(state.recursive) + .label("RECURSIVE".to_string()) + .on_toggle(Message::RecursiveToggled); + let t4 = toggler(state.abort_on_threat) + .label("ABORT ON THREAT".to_string()) + .on_toggle(Message::AbortOnThreatToggled); + column![t1, t2, t3, t4] + .spacing(8.0) + .padding(pad(8.0, 10.0, 8.0, 10.0)) + .into() +} + +/// Console section (always visible, larger). +fn section_console(state: &CorbelGui) -> Element<'_, Message> { + let chevron = if state.console_open { "-" } else { "+" }; + let hdr = container( + button( + row![ + t(format!("{} # CONSOLE", chevron), 12.0, colors::gold()), + ] + .spacing(4.0) + .align_y(Alignment::Center), + ) + .style(dim_btn_style()) + .padding(pad(4.0, 8.0, 4.0, 8.0)) + .on_press(Message::ToggleConsole), + ) + .width(Length::Fill) + .style(section_hdr_style()) + .padding(pad(0.0, 0.0, 0.0, 0.0)); + + let console_body: Element<'_, Message> = if state.console_open { + let mut lines_col = column![].spacing(2.0); + for line in &state.console_lines { + let tag_color = if line.is_error { colors::red() } else { colors::green() }; + lines_col = lines_col.push( + row![ + t(&line.timestamp, 11.0, colors::dim()), + t(" ", 11.0, colors::dim()), + t(format!("[{}]", line.tag), 11.0, tag_color), + t(format!(" {}", line.message), 11.0, colors::white()), + ].spacing(0.0), + ); + } + if state.console_lines.is_empty() { + lines_col = lines_col.push( + t("# ready.", 12.0, colors::dim()), + ); + } + scrollable(lines_col) + .height(Length::Fill) + .width(Length::Fill) + .into() + } else { + column![].into() + }; + + column![hdr, console_body] + .spacing(0.0) + .width(Length::Fill) + .height(Length::FillPortion(3)) + .into() +} + +/// 280 px right sidebar. +fn build_right_sidebar(state: &CorbelGui) -> Element<'_, Message> { + let total = state.processed; + let gauge_text = build_gauge(total); + let processed_label = t(format!("{} PROCESSED", total), 11.0, colors::gold()); + + let stats = column![ + stat_row("*", "CLEANED", state.cleaned, colors::gold()), + stat_row(">", "COPIED", state.copied, colors::green()), + stat_row("!", "ERRORS", state.errors, colors::red()), + ] + .spacing(10.0) + .padding(pad(12.0, 14.0, 12.0, 14.0)); + + let in_display = if state.input_path.is_empty() { "(none)".to_string() } else { state.input_path.clone() }; + let out_display = if state.output_path.is_empty() { "(cwd)".to_string() } else { state.output_path.clone() }; + let mode = if state.preserve_format { "PreserveFormat" } else { "Markdown" }; + + let run_config = column![ + t("RUN CONFIG", 11.0, colors::gold()), + t(format!("IN: {}", in_display), 10.0, colors::dim()), + t(format!("OUT: {}", out_display), 10.0, colors::dim()), + t(format!("MODE: {}", mode), 10.0, colors::dim()), + ] + .spacing(4.0) + .padding(pad(8.0, 14.0, 12.0, 14.0)); + + let findings_block = if let Some(r) = &state.last_result { + let mut fcol = column![t("LAST SCAN", 11.0, colors::gold())].spacing(4.0); + for finding in &r.scan_report.findings { + let class_str = format!("{:?}", finding.classification); + let color = match finding.classification { + ThreatClassification::Malicious(_) => colors::red(), + ThreatClassification::Suspicious => colors::yellow(), + ThreatClassification::EducationalContent => colors::teal(), + ThreatClassification::Benign => colors::dim(), + }; + fcol = fcol.push(row![ + t("*", 10.0, color), + t(class_str, 9.0, color), + ].spacing(4.0)); + fcol = fcol.push(t(format!(" {}", finding.location), 9.0, colors::dim())); + // TODO #12: CVE badges β€” parse CVE tags from context_notes. + let cve_tags = extract_cve_tags(&finding.context_notes); + for tag in &cve_tags { + fcol = fcol.push( + container(t(tag, 8.0, Color::WHITE)) + .padding(pad(1.0, 4.0, 1.0, 4.0)) + .style(cve_badge_style()), + ); + fcol = fcol.push(t(" ", 6.0, colors::dim())); + } + } + if r.scan_report.findings.is_empty() { + fcol = fcol.push(t("(no findings)", 10.0, colors::dim())); + } + container(fcol).padding(pad(8.0, 14.0, 8.0, 14.0)).width(Length::Fill) + } else { + container(column![]).width(Length::Fill) + }; + + let sidebar_col = column![ + column![gauge_text, processed_label] + .spacing(4.0) + .align_x(Alignment::Center) + .padding(pad(16.0, 0.0, 8.0, 0.0)), + container("").height(1).width(Length::Fill).style(separator_style()), + stats, + container("").height(1).width(Length::Fill).style(separator_style()), + run_config, + container("").height(1).width(Length::Fill).style(separator_style()), + findings_block, + // TODO #4: EPUB/Markdown cleansed document viewer. + build_cleansed_viewer(state), + ] + .spacing(0.0); + + container(scrollable(sidebar_col)) + .width(280) + .height(Length::Fill) + .style(panel_style()) + .into() +} + +/// 56 px footer. +fn build_footer(state: &CorbelGui) -> Element<'_, Message> { + let start_btn = button( + row![ + t(">", 14.0, Color::BLACK), + t("START PROCESSING", 12.0, Color::BLACK), + ] + .spacing(6.0) + .align_y(Alignment::Center), + ) + .style(gold_btn_style()) + .padding(pad(8.0, 16.0, 8.0, 16.0)) + .on_press_maybe(if state.is_processing { None } else { Some(Message::StartProcessing) }); + + let stop_btn = button(t("STOP", 12.0, colors::dim())) + .style(dim_btn_style()) + .padding(pad(8.0, 16.0, 8.0, 16.0)) + .on_press_maybe(if state.is_processing { Some(Message::StopProcessing) } else { None }); + + let clear_btn = button( + row![ + t("x", 13.0, Color::BLACK), + t("CLEAR LOG", 12.0, Color::BLACK), + ] + .spacing(6.0) + .align_y(Alignment::Center), + ) + .style(teal_btn_style()) + .padding(pad(8.0, 16.0, 8.0, 16.0)) + .on_press(Message::ClearLog); + + let about_btn = button( + row![ + t("i", 14.0, colors::dim()), + t("ABOUT / LICENSE", 12.0, colors::dim()), + ] + .spacing(6.0) + .align_y(Alignment::Center), + ) + .style(dim_btn_style()) + .padding(pad(8.0, 16.0, 8.0, 16.0)) + .on_press(Message::ToggleAbout); + + let buttons = row![start_btn, stop_btn, clear_btn, about_btn] + .spacing(8.0) + .align_y(Alignment::Center); + + let status_left = t( + format!("CorbelPurge v{} - Defensive Document Research & Cleaning Tool", env!("CARGO_PKG_VERSION")), + 10.0, + colors::dim(), + ); + let status_right = t("GPL-3.0-or-later | Jeremy Anderson", 10.0, colors::dim()); + + let status_row = row![status_left, row![].width(Length::Fill), status_right] + .align_y(Alignment::Center); + + let footer_col = column![buttons, status_row] + .spacing(4.0) + .padding(pad(6.0, 16.0, 6.0, 16.0)); + + container(footer_col) + .width(Length::Fill) + .height(56) + .center_y(Length::Shrink) + .style(footer_style()) + .into() +} + +// --------------------------------------------------------------------------- +// Reusable widget builders +// --------------------------------------------------------------------------- + +fn collapsible_section<'a>( + label: &str, + is_open: bool, + toggle_msg: Message, + content: Element<'a, Message>, +) -> Element<'a, Message> { + let chevron = if is_open { "-" } else { "+" }; + let display_label = format!("{} {}", chevron, label); + + let hdr = container( + button(t(&display_label, 12.0, colors::gold())) + .style(dim_btn_style()) + .padding(pad(4.0, 8.0, 4.0, 8.0)) + .on_press(toggle_msg), + ) + .width(Length::Fill) + .style(section_hdr_style()) + .padding(pad(0.0, 0.0, 0.0, 0.0)); + + if is_open { + column![hdr, content].spacing(0.0).width(Length::Fill).into() + } else { + column![hdr].width(Length::Fill).into() + } +} + +fn stat_row(icon: &str, label: &str, count: u32, color: Color) -> Element<'static, Message> { + row![ + t(icon, 16.0, color), + t(label, 11.0, colors::dim()), + row![].width(Length::Fill), + t(count.to_string(), 18.0, color), + ] + .spacing(6.0) + .align_y(Alignment::Center) + .into() +} + +// --------------------------------------------------------------------------- +// EPUB / Markdown cleansed document viewer (TODO #4 + #5) +// --------------------------------------------------------------------------- + +fn build_cleansed_viewer(state: &CorbelGui) -> Element<'_, Message> { + let content = match &state.cleansed_content { + Some(c) => c, + None => return container(column![]).into(), + }; + + // Simple syntax highlighting for fenced code blocks (TODO #5). + // We highlight code block backgrounds with a darker panel. + let mut highlighted = String::new(); + let mut in_code_block = false; + for line in content.lines() { + if line.starts_with("```") { + in_code_block = !in_code_block; + // Keep the fence line as-is. + highlighted.push_str(line); + highlighted.push('\n'); + continue; + } + if in_code_block { + // Prefix code lines with a marker that the console-style + // dark background already provides visual distinction. + highlighted.push_str(line); + highlighted.push('\n'); + } else { + // Regular Markdown text β€” pass through. + highlighted.push_str(line); + highlighted.push('\n'); + } + } + + let viewer_content = scrollable( + column![ + t("CLEANSED DOCUMENT", 11.0, colors::gold()), + Space::with_height(4), + t(highlighted, 11.0, colors::white()), + ] + .spacing(2.0) + ) + .height(Length::FillPortion(4)); + + container(viewer_content) + .padding(pad(8.0, 10.0, 8.0, 10.0)) + .width(Length::Fill) + .into() +} + +// --------------------------------------------------------------------------- +// About / License overlay +// --------------------------------------------------------------------------- +// +// Floating info panel anchored to the top-right corner, modelled on the +// ferret about-panel screenshot. Visual style: dark panel, gold border, +// drop shadow, close (X) button in the top-right, structured metadata +// (title, version, description, author, website, license, tech stack, +// copyright). +// +// Implementation note: iced 0.13 has no Stack widget (true non-modal +// overlays landed in 0.14), so we emulate the floating-panel feel with +// a full-window dim backdrop. The panel sits in the top-right corner via +// a row + Space::Fill layout. Close is via the X button in the panel +// header; click-outside-to-close and ESC handling are left as future +// enhancements (would require upgrading to iced 0.14+ or adding a +// keyboard subscription). + +fn build_about_panel() -> Element<'static, Message> { + // --- Header row: title + version ... [X] --- + let title_row = row![ + t("CORBELPURGE", 18.0, colors::gold()), + t(format!(" v{}", env!("CARGO_PKG_VERSION")), 11.0, colors::dim()), + row![].width(Length::Fill), + button(t("X", 11.0, colors::gold())) + .style(about_close_btn_style()) + .padding(pad(2.0, 6.0, 2.0, 6.0)) + .on_press(Message::CloseAbout), + ] + .spacing(6.0) + .align_y(Alignment::Center); + + // --- Description --- + let description = t( + "Strict Rust document sanitizer & threat neutralizer\nfor PDF, EPUB, Markdown, and DOCX.", + 11.0, + colors::white(), + ); + + // --- Metadata rows --- + let meta_row = |label: &str, value: &str, value_color: Color| -> Element<'static, Message> { + row![ + t(format!("{}:", label), 10.0, colors::dim()), + t(value.to_string(), 10.0, value_color), + ] + .spacing(6.0) + .align_y(Alignment::Center) + .into() + }; + + let author_row = meta_row("Author", "Jeremy Anderson", colors::white()); + let website_row = meta_row("Website", env!("CARGO_PKG_REPOSITORY"), colors::gold()); + let license_row = meta_row("License", env!("CARGO_PKG_LICENSE"), colors::white()); + + // --- Separator (thin horizontal rule) --- + let separator = container(Space::new(Length::Fill, 1.0)) + .width(Length::Fill) + .style(separator_style()); + + // --- Footer: tech stack + copyright --- + let tech_stack = t( + "Built with Rust, iced 0.13, lopdf, pulldown-cmark, and zip.", + 9.0, + colors::dim(), + ); + let copyright = t( + "Copyright (c) 2026 Jeremy Anderson.", + 9.0, + colors::dim(), + ); + + // --- Assemble panel --- + let panel_body = column![ + title_row, + Space::with_height(8), + description, + Space::with_height(12), + author_row, + website_row, + license_row, + Space::with_height(10), + separator, + Space::with_height(8), + tech_stack, + copyright, + ] + .spacing(2.0) + .width(320); + + container(panel_body) + .style(about_panel_style()) + .padding(pad(16.0, 18.0, 16.0, 18.0)) + .width(320) + .into() +} + +/// Full-window overlay: dim backdrop + About panel anchored to the top-right +/// corner. +/// +/// Click-outside-to-close is sacrificed here because iced 0.13 lacks a true +/// Stack widget (added in 0.14) and wrapping the panel in a no-op button +/// would make the panel's own labels close the modal on click. The X button +/// in the panel header remains the close affordance; ESC handling can be +/// added later via a keyboard subscription. +fn build_about_overlay() -> Element<'static, Message> { + let layout = row![ + Space::with_width(Length::Fill), + column![ + Space::with_height(20.0), + build_about_panel(), + Space::with_height(Length::Fill), + ] + .width(320) + .height(Length::Fill), + Space::with_width(20.0), + ] + .align_y(Alignment::Start) + .height(Length::Fill) + .width(Length::Fill); + + container(layout) + .width(Length::Fill) + .height(Length::Fill) + .style(about_dim_style()) + .into() +} + +fn build_gauge(processed: u32) -> Element<'static, Message> { + let bar_width = 16usize; + let filled = ((processed as usize).min(20) * bar_width) / 20; + let empty = bar_width - filled; + let bar_str = format!("{}{}", "#".repeat(filled), "-".repeat(empty)); + column![ + t(processed.to_string(), 36.0, colors::gold()), + t(bar_str, 10.0, colors::gold()), + ] + .align_x(Alignment::Center) + .spacing(4.0) + .into() +} + +// --------------------------------------------------------------------------- +// Background tasks +// --------------------------------------------------------------------------- + +async fn pick_file() -> Option { + rfd::AsyncFileDialog::new() + .add_filter("Documents", &["pdf", "epub", "md", "markdown", "docx"]) + .set_title("Select input file") + .pick_file() + .await + .map(|h| h.path().to_path_buf()) +} + +async fn pick_folder() -> Option { + rfd::AsyncFileDialog::new() + .set_title("Select output directory") + .pick_folder() + .await + .map(|h| h.path().to_path_buf()) +} + +async fn run_pipeline( + path: PathBuf, + workspace: Option, + abort_on_threat: bool, + preserve_format: bool, +) -> Result { + let mut config = match workspace { + Some(p) => Config::with_workspace(p), + None => Config::default(), + }; + config.abort_on_threat = abort_on_threat; + if preserve_format { + config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat; + } + let pipeline = Pipeline::with_config(config); + tokio::task::spawn_blocking(move || pipeline.run(&path)) + .await + .map_err(|e| format!("task join error: {e}"))? + .map_err(|e| e.to_string()) +} + +// --------------------------------------------------------------------------- +// CVE badge helper (TODO #12) +// --------------------------------------------------------------------------- + +/// Extract CVE tags (e.g. `[CVE-2017-11882: Equation Editor RCE]`) +/// from a context_notes string. +fn extract_cve_tags(notes: &str) -> Vec { + let mut tags = Vec::new(); + let mut search_from = 0; + while let Some(start) = notes[search_from..].find('[') { + let rest = ¬es[search_from + start + 1..]; + if let Some(end) = rest.find(']') { + let tag = rest[..end].trim(); + if tag.starts_with("CVE-") || tag.starts_with("EPUB-") { + // Truncate long descriptions to just the CVE ID. + let short = tag.split(':').next().unwrap_or(tag); + tags.push(short.to_string()); + } + search_from += start + 1 + end + 1; + } else { + break; + } + } + tags +} + +/// Style for CVE badge containers. +fn cve_badge_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::teal().into()), + border: Border { color: colors::teal(), width: 1.0, radius: 3.0.into() }, + ..iced::widget::container::Style::default() + } +} + +/// Dimmed backdrop behind the About overlay. Subtle alpha so the underlying +/// UI is still partially visible β€” matches the "floating panel" feel of the +/// ferret reference without requiring iced's Stack widget (added in 0.14). +fn about_dim_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(Color::from_rgba(0.0, 0.0, 0.0, 0.55).into()), + ..iced::widget::container::Style::default() + } +} + +/// Style for the About panel itself: dark panel background, gold border, +/// rounded corners β€” mirrors the ferret about-panel aesthetic. +fn about_panel_style() -> impl Fn(&Theme) -> iced::widget::container::Style { + |_t: &Theme| iced::widget::container::Style { + background: Some(colors::panel().into()), + border: Border { color: colors::gold(), width: 1.5, radius: 6.0.into() }, + shadow: iced::Shadow { + color: Color::from_rgba(0.0, 0.0, 0.0, 0.6), + offset: iced::Vector::new(0.0, 4.0), + blur_radius: 12.0, + }, + ..iced::widget::container::Style::default() + } +} + +/// Style for the small close (X) button in the About panel header. +fn about_close_btn_style() -> impl Fn(&Theme, iced::widget::button::Status) -> ButtonStyle { + move |_t: &Theme, _s: iced::widget::button::Status| ButtonStyle { + background: Some(colors::panel_hi().into()), + text_color: colors::gold(), + border: Border { color: colors::gold(), width: 1.0, radius: 3.0.into() }, + ..ButtonStyle::default() + } +} + +// Note: a `about_backdrop_btn_style` (transparent, click-outside-to-close +// backdrop) was originally drafted for this overlay but is omitted because +// iced 0.13 cannot layer a button under a sibling widget without making +// the sibling's own clicks bubble up. Click-outside-to-close can be added +// later by upgrading to iced 0.14+ (which has a Stack widget) or by +// adding a keyboard subscription for the Escape key. + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +fn now_hms() -> String { + use std::time::{SystemTime, UNIX_EPOCH}; + let secs = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_secs(); + let h = (secs / 3600) % 24; + let m = (secs / 60) % 60; + let s = secs % 60; + format!("{:02}:{:02}:{:02}", h, m, s) +} + +fn open_dir(path: &std::path::Path) { + #[cfg(target_os = "linux")] { let _ = std::process::Command::new("xdg-open").arg(path).spawn(); } + #[cfg(target_os = "macos")] { let _ = std::process::Command::new("open").arg(path).spawn(); } + #[cfg(target_os = "windows")] { let _ = std::process::Command::new("explorer").arg(path).spawn(); } +} \ No newline at end of file diff --git a/src/cleanse/mod.rs b/src/cleanse/mod.rs new file mode 100755 index 0000000..50cb8ba --- /dev/null +++ b/src/cleanse/mod.rs @@ -0,0 +1,37 @@ +//! Cleansed-document builder: produces a sanitized derivative of the +//! original document containing only validated clean content. +//! +//! Two modes are supported (selected via [`crate::core::config::CleanseMode`]): +//! +//! - **Markdown** (default): produces a safe Markdown file. See [`sanitizer`]. +//! - **PreserveFormat**: repackages the document in its original format +//! (PDF/EPUB/DOCX) with malicious entries stripped. See [`repackage`]. + +pub mod repackage; +pub mod sanitizer; + +pub use repackage::repackage; +pub use sanitizer::sanitize; + +use std::path::PathBuf; + +use crate::core::config::{CleanseMode, Config}; +use crate::core::types::{Document, ScanReport}; +use crate::CorbelResult; + +/// Top-level cleanse dispatcher. Called by the pipeline when +/// [`ScanReport::overall_recommendation`] returns +/// [`crate::core::types::Recommendation::QuarantineAndCleanse`]. +/// +/// Dispatches to [`sanitize`] (Markdown) or [`repackage`] (PreserveFormat) +/// based on `config.cleanse_mode`. +pub fn cleanse( + document: &Document, + scan_report: &ScanReport, + config: &Config, +) -> CorbelResult { + match config.cleanse_mode { + CleanseMode::Markdown => sanitize(document, scan_report, config), + CleanseMode::PreserveFormat => repackage(document, scan_report, config), + } +} diff --git a/src/cleanse/repackage.rs b/src/cleanse/repackage.rs new file mode 100755 index 0000000..8f08c1d --- /dev/null +++ b/src/cleanse/repackage.rs @@ -0,0 +1,1173 @@ +//! Format-preserving repackage: rebuilds the document in its original +//! format with malicious entries stripped. +//! +//! Unlike the [`super::sanitizer`] (which always produces Markdown), +//! this module produces a structurally-similar derivative of the +//! original document β€” useful when the document itself is the artifact +//! of interest (e.g. a book the researcher wants to actually read +//! after cleaning). +//! +//! ## What gets stripped +//! +//! - **EPUB**: entries whose paths match a malicious finding's location +//! (typically `` blocks (entirely, including content) +/// - `` blocks +/// - `...` blocks +/// - `...` tags (self-closing or paired) +/// - `src="http(s)://..."` and `href="http(s)://..."` attributes +/// (replaced with `src="#"` / `href="#"` to keep the XHTML valid) +fn strip_xhtml_vectors(xhtml: &str) -> String { + let mut out = xhtml.to_string(); + + // Remove blocks (case-insensitive). + out = remove_tag_pair(&out, "script"); + // Remove blocks. + out = remove_tag_pair(&out, "iframe"); + // Remove ... blocks. + out = remove_tag_pair(&out, "object"); + // Remove tags (self-closing or open). + out = remove_self_closing_tag(&out, "embed"); + + // Neutralize external URLs in src= and href= attributes. + // We replace the URL with "#" so the attribute is still valid + // XHTML but doesn't point anywhere. + out = neutralize_external_urls(&out, "src"); + out = neutralize_external_urls(&out, "href"); + + out +} + +/// Remove `...` pairs (including content), case-insensitively. +fn remove_tag_pair(xml: &str, tag: &str) -> String { + let mut out = String::new(); + let lower = xml.to_ascii_lowercase(); + let open_tag = format!("<{tag}"); + let close_tag = format!(""); + let mut search_from = 0; + + while search_from < xml.len() { + let Some(open_pos) = lower[search_from..].find(&open_tag) else { + out.push_str(&xml[search_from..]); + break; + }; + let abs_open = search_from + open_pos; + // Copy everything before the opening tag. + out.push_str(&xml[search_from..abs_open]); + + // Find the closing tag. + let rest_lower = &lower[abs_open..]; + let Some(close_pos) = rest_lower.find(&close_tag) else { + // No closing tag β€” drop the rest to be safe. + break; + }; + let abs_close = abs_open + close_pos + close_tag.len(); + // Skip the entire block (don't copy it). + search_from = abs_close; + } + + out +} + +/// Remove self-closing `` or open `` tags. +fn remove_self_closing_tag(xml: &str, tag: &str) -> String { + let mut out = String::new(); + let lower = xml.to_ascii_lowercase(); + let open_tag = format!("<{tag}"); + let mut search_from = 0; + + while search_from < xml.len() { + let Some(open_pos) = lower[search_from..].find(&open_tag) else { + out.push_str(&xml[search_from..]); + break; + }; + let abs_open = search_from + open_pos; + out.push_str(&xml[search_from..abs_open]); + + // Find the end of this tag (>). + let rest = &xml[abs_open..]; + let Some(tag_end) = rest.find('>') else { + break; + }; + let abs_end = abs_open + tag_end + 1; + // Skip the tag. + search_from = abs_end; + } + + out +} + +/// Replace `attr="http(s)://..."` with `attr="#"` in the XML string. +fn neutralize_external_urls(xml: &str, attr: &str) -> String { + let mut out = xml.to_string(); + let lower = out.to_ascii_lowercase(); + let needle = format!("{attr}=\"http"); + let mut search_from = 0; + + while let Some(pos) = lower[search_from..].find(&needle) { + let abs_pos = search_from + pos; + // Find the closing quote. + let rest = &out[abs_pos..]; + let Some(end_quote) = rest.find('"').and_then(|p| rest[p + 1..].find('"')) else { + break; + }; + let abs_end = abs_pos + rest.find('"').unwrap() + 1 + end_quote + 1; + // Replace the URL value with "#". + let before = &out[..abs_pos]; + let after = &out[abs_end..]; + out = format!("{before}{attr}=\"#\"{after}"); + // The lower-case copy is now stale β€” recompute. (Inefficient + // but correct; repackage is not perf-critical.) + // Advance search_from past the replacement. + search_from = abs_pos + attr.len() + 4; // length of `=":"#` + } + + out +} + +/// Repackage a DOCX: same as `repackage_zip`, but also rewrites +/// `word/_rels/document.xml.rels` to drop `TargetMode="External"` +/// relationships (which point at malicious URLs). +fn repackage_docx(document: &Document, scan_report: &ScanReport) -> CorbelResult> { + let strip_paths = malicious_zip_paths(scan_report); + + // First, do the standard ZIP repackage. + let mut bytes = repackage_zip(document, scan_report, &strip_paths)?; + + // Then post-process: rewrite the rels file to drop external links. + // We do this in-place on the repackaged bytes. + bytes = rewrite_docx_rels(&bytes)?; + + Ok(bytes) +} + +/// Read the repackaged DOCX, find `word/_rels/document.xml.rels`, +/// and rewrite it to drop `TargetMode="External"` relationships. +fn rewrite_docx_rels(bytes: &[u8]) -> CorbelResult> { + let cursor = std::io::Cursor::new(bytes.to_vec()); + let mut archive = zip::ZipArchive::new(cursor) + .map_err(|e| CorbelError::Cleanse(format!("rels rewrite: zip open failed: {e}")))?; + + // Find the rels entry. + let rels_index = archive + .file_names() + .position(|n| n.eq_ignore_ascii_case("word/_rels/document.xml.rels")); + + let Some(rels_index) = rels_index else { + // No rels file β€” nothing to rewrite. + return Ok(bytes.to_vec()); + }; + + // Read the rels entry. + let mut rels_xml = String::new(); + { + let mut entry = archive + .by_index(rels_index) + .map_err(|e| CorbelError::Cleanse(format!("rels read failed: {e}")))?; + entry + .read_to_string(&mut rels_xml) + .map_err(|e| CorbelError::Cleanse(format!("rels read failed: {e}")))?; + } + + // Drop entries. + let cleaned_rels = drop_external_relationships(&rels_xml); + + // Rebuild the ZIP with the cleaned rels entry. + let mut output = Vec::new(); + { + let mut writer = zip::ZipWriter::new(std::io::Cursor::new(&mut output)); + let opts = zip::write::SimpleFileOptions::default() + .compression_method(zip::CompressionMethod::Deflated); + + for i in 0..archive.len() { + let mut entry = archive + .by_index(i) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: entry {i} failed: {e}")))?; + let name = entry.name().to_string(); + let entry_opts = opts + .compression_method(entry.compression()) + .last_modified_time(entry.last_modified().unwrap_or_default()); + + if i == rels_index { + writer + .start_file(&name, entry_opts) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: start_file failed: {e}")))?; + writer + .write_all(cleaned_rels.as_bytes()) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: write failed: {e}")))?; + } else { + let mut buf = Vec::new(); + entry + .read_to_end(&mut buf) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: entry {i} read failed: {e}")))?; + writer + .start_file(&name, entry_opts) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: start_file failed: {e}")))?; + writer + .write_all(&buf) + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: write failed: {e}")))?; + } + } + + writer + .finish() + .map_err(|e| CorbelError::Cleanse(format!("rels rebuild: finish failed: {e}")))?; + } + + Ok(output) +} + +/// Drop `` elements with `TargetMode="External"` from +/// the rels XML. Coarse regex-free approach: split on ` String { + let mut out = String::new(); + let mut search_from = 0; + + while let Some(rel_start) = xml[search_from..].find(" element. + let rest = &xml[abs_start..]; + let tag_end = match rest.find("/>").map(|p| abs_start + p + 2).or_else(|| { + rest.find('>').map(|p| abs_start + p + 1) + }) { + Some(p) => p, + None => { + // Malformed β€” copy the rest and bail. + out.push_str(rest); + return out; + } + }; + + let opening_tag = &xml[abs_start..tag_end]; + if !opening_tag.contains(r#"TargetMode="External""#) { + // Keep this relationship. + out.push_str(opening_tag); + } + // If it was external, we skip it (don't copy). + + search_from = tag_end; + } + + // Copy any trailing content. + out.push_str(&xml[search_from..]); + out +} + +/// Repackage a PDF: use lopdf to delete objects whose IDs match +/// malicious findings, then save with rebuilt xref. +fn repackage_pdf(document: &Document, scan_report: &ScanReport) -> CorbelResult> { + let mut pdf = lopdf::Document::load_mem(&document.raw_bytes) + .map_err(|e| CorbelError::Cleanse(format!("pdf load failed: {e}")))?; + + // Collect the set of object IDs to delete. lopdf's ObjectId is + // `(u32, u16)` β€” a tuple, not a struct. + let mut objs_to_delete: Vec = Vec::new(); + for finding in &scan_report.findings { + if !matches!( + finding.classification, + ThreatClassification::Malicious(_) + ) { + continue; + } + match &finding.location { + Location::PdfObject { id, gen } => { + objs_to_delete.push((*id, *gen as u16)); + } + Location::PdfStream { id, .. } => { + objs_to_delete.push((*id, 0)); + } + _ => {} + } + } + + // Delete each offending object. lopdf's `delete_object` removes + // it from the objects table; the xref is rebuilt on save. + for obj_id in &objs_to_delete { + let _ = pdf.delete_object(*obj_id); + } + + // Also walk the catalog and remove /OpenAction, /AA, /Names + // references that might point at deleted objects. This is + // best-effort β€” lopdf doesn't automatically clean up dangling + // references, but a dangling reference to a deleted object is + // a parse error in the viewer, not a security issue. + if let Ok(root_dict) = pdf.catalog_mut() { + let _ = root_dict.remove(b"OpenAction"); + let _ = root_dict.remove(b"AA"); + let _ = root_dict.remove(b"Names"); + let _ = root_dict.remove(b"AcroForm"); + } + + // Sweep remaining objects for dangling references to deleted IDs. + // A dangling reference is any indirect or direct reference + // (e.g. in a stream, array, or dictionary value) that contains + // the object ID of a deleted object. We nullify these references + // by replacing the reference value with null. + sweep_dangling_references(&mut pdf, &objs_to_delete); + + // Save to a new byte buffer via `save_to` (which takes &mut Write). + let mut output = Vec::new(); + pdf.save_to(&mut output) + .map_err(|e| CorbelError::Cleanse(format!("pdf save failed: {e}")))?; + Ok(output) +} + +/// Sweep all remaining objects for dangling references to deleted object IDs. +/// +/// After `lopdf::Document::delete_objects()` removes objects from +/// the object table, other objects may still contain indirect or +/// direct references (e.g. in streams, arrays, or dictionary values) that +/// point to the deleted IDs. A PDF reader encountering these will +/// typically log warnings or fail to render the page. +/// +/// This function walks every remaining object and attempts to nullify +/// any reference to a deleted ID. It operates on the serialized +/// object data since lopdf doesn't expose a structured object model. +/// +/// The approach is conservative: we replace byte sequences that look +/// like object references (e.g. `42 0 R` or `42 0 R` inside an +/// indirect reference array) with `0 0 R`. This is a best-effort +/// heuristic β€” false positives are possible but the worst case is +/// nullifying a valid numeric literal that happens to match the pattern. +fn sweep_dangling_references( + pdf: &mut lopdf::Document, + deleted_ids: &[(u32, u16)], +) { + let deleted_id_set: std::collections::HashSet = + deleted_ids.iter().map(|(id, _)| *id).collect(); + + let ids: Vec = pdf.objects.iter().map(|(id, _)| *id).collect(); + + for obj_id in ids { + if deleted_id_set.contains(&obj_id.0) { + continue; // Already deleted. + } + if let Ok(obj) = pdf.get_object_mut(obj_id) { + if let Ok(data) = obj.as_stream_mut() { + // Stream content: look for N N R patterns. + // + // `Stream::content` is a `Vec` (not an Option/Result), + // so we mutate it in place via `&mut`. + nullify_ref_in_bytes(&mut data.content, &deleted_id_set); + } + } + } +} + +/// Nullify object references (`N 0 R` patterns) in a byte buffer. +/// +/// We look for sequences matching the pattern: one or more ASCII digits, +/// a space, a zero (generation number), a space, the letter 'R'. +/// If the object ID (first number) matches a deleted ID, we replace +/// the entire reference with `0 0 R`. +/// +/// This is a best-effort heuristic β€” false positives are possible but +/// the worst case is nullifying a valid numeric literal. +fn nullify_ref_in_bytes(buf: &mut Vec, deleted_ids: &std::collections::HashSet) { + let original = std::mem::take(buf); + let len = original.len(); + let mut i = 0; + + while i < len { + // Look for the pattern: digits space digits space R + if original[i].is_ascii_digit() { + // Try to parse a potential "N 0 R" reference. + let num_start = i; + let mut num = 0u32; + while i < len && original[i].is_ascii_digit() { + num = num.saturating_mul(10).saturating_add((original[i] - b'0') as u32); + i += 1; + } + if i >= len || original[i] != b' ' { + // Not a reference pattern β€” keep the bytes. + buf.extend_from_slice(&original[num_start..i]); + continue; + } + i += 1; // skip space + if i >= len || !original[i].is_ascii_digit() { + buf.extend_from_slice(&original[num_start..i]); + continue; + } + // Skip the generation number (we don't need it). + while i < len && original[i].is_ascii_digit() { + i += 1; + } + if i >= len || original[i] != b' ' { + buf.extend_from_slice(&original[num_start..i]); + continue; + } + i += 1; // skip space + if i >= len || original[i] != b'R' { + buf.extend_from_slice(&original[num_start..i]); + continue; + } + i += 1; // skip 'R' + + // We have a complete "N G R" pattern. If N is a deleted ID, + // replace with "0 0 R". + if deleted_ids.contains(&num) { + buf.extend_from_slice(b"0 0 R"); + } else { + buf.extend_from_slice(&original[num_start..i]); + } + } else { + buf.push(original[i]); + i += 1; + } + } +} + +/// Parse a byte slice as a decimal number. +/// +/// Currently unused β€” kept as a utility for future structured object +/// reference parsing. The current `nullify_ref_in_bytes` implementation +/// parses inline because it needs to track byte offsets for replacement. +#[allow(dead_code)] +fn parse_id_slice(bytes: &[u8]) -> u32 { + let mut n: u32 = 0; + for &b in bytes { + if b.is_ascii_digit() { + n = n.saturating_mul(10).saturating_add((b - b'0') as u32); + } else { + break; + } + } + n +} +#[cfg(test)] +mod tests { + use super::*; + use crate::core::types::*; + use std::io::Write; + use tempfile::tempdir; + use crate::CleanseMode; + + fn make_zip(entries: &[(&str, &[u8])]) -> Vec { + let mut buf = Vec::new(); + { + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(&mut buf)); + let opts = zip::write::SimpleFileOptions::default() + .compression_method(zip::CompressionMethod::Stored); + for (name, data) in entries { + zip.start_file(name, opts).unwrap(); + zip.write_all(data).unwrap(); + } + zip.finish().unwrap(); + } + buf + } + + fn make_doc_with_format(format: DocumentFormat, raw_bytes: Vec) -> Document { + Document { + format, + source_path: None, + raw_bytes, + sha256: "abcdef".to_string(), + size: 0, + metadata: DocumentMetadata::default(), + text_nodes: Vec::new(), + executable_vectors: Vec::new(), + } + } + + fn make_scan_with_malicious_at(loc: Location) -> ScanReport { + ScanReport { + source_sha256: "abcdef".to_string(), + format: DocumentFormat::Epub, + scanned_at: "x".to_string(), + findings: vec![Finding { + classification: ThreatClassification::Malicious( + MaliciousType::EpubActiveScript, + ), + location: loc, + vector_type: None, + payload_preview: "x".to_string(), + context_notes: "x".to_string(), + recommendation: Recommendation::QuarantineAndCleanse, + }], + text_nodes_scanned: 0, + vectors_scanned: 0, + } + } + + #[test] + fn repackage_epub_strips_malicious_entry() { + let bytes = make_zip(&[ + ("mimetype", b"application/epub+zip"), + ("OEBPS/ch1.xhtml", b"

safe

"), + ("OEBPS/evil.xhtml", b""), + ]); + let doc = make_doc_with_format(DocumentFormat::Epub, bytes); + let scan = make_scan_with_malicious_at(Location::EpubEntry { + path: "OEBPS/evil.xhtml".to_string(), + anchor: None, + }); + let tmp = tempdir().unwrap(); + let mut config = Config::default(); + config.cleanse_dir = tmp.path().to_path_buf(); + config.cleanse_mode = CleanseMode::PreserveFormat; + + let path = repackage(&doc, &scan, &config).unwrap(); + let repackaged = std::fs::read(&path).unwrap(); + + // The repackaged EPUB should not contain "alert(1)". + assert!( + !String::from_utf8_lossy(&repackaged).contains("alert(1)"), + "repackaged EPUB should not contain the malicious script" + ); + // It should still contain the safe entry. + assert!( + String::from_utf8_lossy(&repackaged).contains("safe"), + "repackaged EPUB should contain the safe entry" + ); + // Output should be an .epub file. + assert!(path.to_string_lossy().ends_with(".epub")); + } + + #[test] + fn repackage_docx_strips_macro_and_rewrites_rels() { + let rels_xml = br#" + + + +"#; + let bytes = make_zip(&[ + ("word/document.xml", b""), + ("word/vbaProject.xml", b"Sub AutoOpen()"), + ("word/_rels/document.xml.rels", rels_xml), + ]); + let doc = make_doc_with_format(DocumentFormat::Docx, bytes); + let scan = make_scan_with_malicious_at(Location::EpubEntry { + path: "word/vbaProject.xml".to_string(), + anchor: None, + }); + let tmp = tempdir().unwrap(); + let mut config = Config::default(); + config.cleanse_dir = tmp.path().to_path_buf(); + config.cleanse_mode = CleanseMode::PreserveFormat; + + let path = repackage(&doc, &scan, &config).unwrap(); + let repackaged = std::fs::read(&path).unwrap(); + let repackaged_str = String::from_utf8_lossy(&repackaged); + + // VBA macro should be gone. + assert!(!repackaged_str.contains("vbaProject")); + assert!(!repackaged_str.contains("AutoOpen")); + // Internal relationship should be preserved. + assert!(repackaged_str.contains("styles.xml")); + // External relationship should be gone. + assert!( + !repackaged_str.contains("TargetMode=\"External\""), + "external relationships should be stripped" + ); + assert!(!repackaged_str.contains("evil.example.com")); + } + + #[test] + fn repackage_markdown_falls_back_to_sanitizer() { + let doc = make_doc_with_format( + DocumentFormat::Markdown, + b"# Title\n\nsafe text\n".to_vec(), + ); + let scan = ScanReport { + source_sha256: "x".to_string(), + format: DocumentFormat::Markdown, + scanned_at: "x".to_string(), + findings: vec![], + text_nodes_scanned: 0, + vectors_scanned: 0, + }; + let tmp = tempdir().unwrap(); + let mut config = Config::default(); + config.cleanse_dir = tmp.path().to_path_buf(); + config.cleanse_mode = CleanseMode::PreserveFormat; + + let path = repackage(&doc, &scan, &config).unwrap(); + assert!(path.to_string_lossy().ends_with(".md")); + } + + #[test] + fn drop_external_relationships_keeps_internal() { + let xml = r#" + + + + +"#; + let cleaned = drop_external_relationships(xml); + assert!(cleaned.contains("styles.xml")); + assert!(cleaned.contains("settings.xml")); + assert!(!cleaned.contains("evil.com")); + assert!(!cleaned.contains("TargetMode=\"External\"")); + } + + #[test] + fn malicious_zip_paths_collects_only_malicious() { + let scan = ScanReport { + source_sha256: "x".to_string(), + format: DocumentFormat::Epub, + scanned_at: "x".to_string(), + findings: vec![ + // XHTML file with malicious content β€” should NOT be in + // strip_paths (it gets cleaned inline by strip_xhtml_vectors). + Finding { + classification: ThreatClassification::Malicious( + MaliciousType::EpubActiveScript, + ), + location: Location::EpubEntry { + path: "evil.xhtml".to_string(), + anchor: None, + }, + vector_type: None, + payload_preview: "x".to_string(), + context_notes: "x".to_string(), + recommendation: Recommendation::QuarantineAndCleanse, + }, + // Non-XHTML malicious file (VBA macro) β€” SHOULD be in + // strip_paths. + Finding { + classification: ThreatClassification::Malicious( + MaliciousType::DocxActiveContent, + ), + location: Location::EpubEntry { + path: "word/vbaProject.xml".to_string(), + anchor: None, + }, + vector_type: None, + payload_preview: "x".to_string(), + context_notes: "x".to_string(), + recommendation: Recommendation::QuarantineAndCleanse, + }, + // Educational finding β€” should NOT be in strip_paths. + Finding { + classification: ThreatClassification::EducationalContent, + location: Location::EpubEntry { + path: "safe.xhtml".to_string(), + anchor: None, + }, + vector_type: None, + payload_preview: "x".to_string(), + context_notes: "x".to_string(), + recommendation: Recommendation::WhitelistAsEducational, + }, + ], + text_nodes_scanned: 0, + vectors_scanned: 0, + }; + let paths = malicious_zip_paths(&scan); + // XHTML files are cleaned inline, not stripped. + assert!( + !paths.contains("evil.xhtml"), + "XHTML files should not be in strip_paths (cleaned inline instead)" + ); + // Non-XHTML malicious files ARE stripped. + assert!( + paths.contains("word/vbaProject.xml"), + "non-XHTML malicious files should be in strip_paths" + ); + // Educational findings are never stripped. + assert!(!paths.contains("safe.xhtml")); + } + + #[test] + fn strip_xhtml_removes_script_tags() { + let xhtml = r#" +

Hello.

+ +

World.

+"#; + let cleaned = strip_xhtml_vectors(xhtml); + assert!(cleaned.contains("Hello.")); + assert!(cleaned.contains("World.")); + assert!(!cleaned.contains("script")); + assert!(!cleaned.contains("alert")); + } + + #[test] + fn strip_xhtml_neutralizes_external_urls() { + let xhtml = r#" + +click +"#; + let cleaned = strip_xhtml_vectors(xhtml); + assert!(!cleaned.contains("evil.example.com")); + assert!(cleaned.contains("src=\"#\"")); + assert!(cleaned.contains("href=\"#\"")); + } + + #[test] + fn strip_xhtml_removes_iframes() { + let xhtml = r#" +

before

+ +

after

+"#; + let cleaned = strip_xhtml_vectors(xhtml); + assert!(cleaned.contains("before")); + assert!(cleaned.contains("after")); + assert!(!cleaned.contains("iframe")); + }} diff --git a/src/cleanse/sanitizer.rs b/src/cleanse/sanitizer.rs new file mode 100755 index 0000000..c352a00 --- /dev/null +++ b/src/cleanse/sanitizer.rs @@ -0,0 +1,286 @@ +//! Structural re-serializer: generates a brand-new, sanitized document +//! containing only the verified-clean text content from the source. +//! +//! The cleansed output is always Markdown, regardless of the source +//! format. This is intentional: +//! +//! 1. Markdown cannot carry executable content β€” there is no +//! ` + if let Some(content_start) = xhtml[abs_start..].find('>') { + let after_open = abs_start + content_start + 1; + if let Some(end) = xhtml[after_open..].find("") { + let script_body = &xhtml[after_open..after_open + end]; + vectors.push(ExecutableVector { + location: Location::EpubEntry { + path: entry_name.to_string(), + anchor: Some("script".to_string()), + }, + vector_type: VectorType::EpubScript, + raw_payload: script_body.as_bytes().to_vec(), + decoded_preview: Some(script_body.chars().take(2048).collect()), + }); + search_from = after_open + end + 9; // length of "" + continue; + } + } + // Malformed +"#; + let bytes = make_minimal_epub(&[("OEBPS/ch1.xhtml", xhtml)]); + let doc = EpubParser::parse(&bytes, DocumentFormat::Epub, None, &Config::default()).unwrap(); + assert!( + doc.executable_vectors + .iter() + .any(|v| v.vector_type == VectorType::EpubScript), + "should have flagged the ", + ); + let finding = inspect_vector(&v, &Config::default()).unwrap(); + assert!(matches!( + finding.classification, + ThreatClassification::Malicious(MaliciousType::SuspiciousUri) + )); + } + + #[test] + fn vbscript_url_is_malicious() { + let v = make_vector( + VectorType::PdfUri, + b"vbscript:msgbox('xss')", + ); + let finding = inspect_vector(&v, &Config::default()).unwrap(); + assert!(matches!( + finding.classification, + ThreatClassification::Malicious(MaliciousType::SuspiciousUri) + )); + } + + #[test] + fn ip_url_strong_signal_overrides_weak_signals() { + // IP address with a "verify" keyword β€” both signals present, + // but IP (strong) should win. + let v = make_vector( + VectorType::PdfUri, + b"https://10.0.0.1/verify", + ); + let finding = inspect_vector(&v, &Config::default()).unwrap(); + assert!(matches!( + finding.classification, + ThreatClassification::Malicious(MaliciousType::SuspiciousUri) + )); + assert!( + finding.context_notes.contains("ip-host"), + "strong IP signal should win over weak keyword signal: {}", + finding.context_notes + ); + } +} diff --git a/src/scanner/mod.rs b/src/scanner/mod.rs new file mode 100755 index 0000000..d2583dd --- /dev/null +++ b/src/scanner/mod.rs @@ -0,0 +1,84 @@ +//! Security orchestration engine. +//! +//! The scanner takes a parsed [`Document`] and produces a [`ScanReport`] +//! by walking every [`TextNode`] and [`ExecutableVector`] through the +//! layered contextual engine described in the design manifest. + +pub mod context_filter; +pub mod heuristics; +pub mod signatures; +pub mod cve_tags; + +use crate::core::config::Config; +use crate::core::types::{Document, ScanReport}; + +/// Run the full scanner against `document`, producing a [`ScanReport`]. +/// +/// This is the top-level entrypoint called by [`crate::core::pipeline::Pipeline`]. +#[must_use] +pub fn scan(document: &Document, config: &Config) -> ScanReport { + let mut findings = Vec::new(); + + // 1. Walk executable vectors β€” these are untrusted-by-default. + for vector in &document.executable_vectors { + if let Some(mut finding) = heuristics::inspect_vector(vector, config) { + // After classification, try to tag the finding with a + // known CVE if the payload matches a known exploit signature. + if let Some(cve) = cve_tags::match_cve(vector, &finding) { + finding.context_notes = format!( + "{} [{}: {}]", + finding.context_notes, cve.cve_id, cve.name + ); + } + findings.push(finding); + } + } + + // 2. Walk text nodes β€” these go through the context filter to + // distinguish educational content from active threats. + for node in &document.text_nodes { + if let Some(finding) = context_filter::evaluate(node, config) { + findings.push(finding); + } + } + + let scanned_at = chrono::Utc::now().to_rfc3339(); + + ScanReport { + source_sha256: document.sha256.clone(), + format: document.format, + scanned_at, + findings, + text_nodes_scanned: document.text_nodes.len(), + vectors_scanned: document.executable_vectors.len(), + } +} + +/// Re-export for callers that want to inspect individual findings. +pub use heuristics::inspect_vector; +pub use context_filter::evaluate; + +#[cfg(test)] +mod tests { + use super::*; + use crate::core::types::*; + + #[test] + fn empty_document_produces_empty_report() { + let doc = Document { + format: DocumentFormat::Markdown, + source_path: None, + raw_bytes: Vec::new(), + sha256: "abc".to_string(), + size: 0, + metadata: DocumentMetadata::default(), + text_nodes: Vec::new(), + executable_vectors: Vec::new(), + }; + let config = Config::default(); + let report = scan(&doc, &config); + assert_eq!(report.findings.len(), 0); + assert_eq!(report.text_nodes_scanned, 0); + assert_eq!(report.vectors_scanned, 0); + } +} diff --git a/src/scanner/signatures.rs b/src/scanner/signatures.rs new file mode 100755 index 0000000..c49d19d --- /dev/null +++ b/src/scanner/signatures.rs @@ -0,0 +1,694 @@ +//! Threat signature tables and pattern matchers. +//! +//! This module centralizes the static lookup tables used by the +//! heuristics engine. Keeping them in one place makes them easy to +//! audit, extend, and eventually wire up to an external threat-intel +//! feed (e.g. a YARA rules file or a STIX/TAXII subscription). +//! +//! ## What lives here +//! +//! - [`PHISHING_TLDS`] β€” TLDs statistically overrepresented in +//! phishing URLs. Sourced from public phishing reports +//! (Spamhaus, PhishTank yearly summaries). +//! - [`SUSPICIOUS_URL_KEYWORDS`] β€” path/host keywords that strongly +//! indicate credential harvesting or fake login pages. +//! - [`URL_SHORTENER_DOMAINS`] β€” shortener domains. Not malicious +//! per se, but a common obfuscation layer for phishing links. +//! - [`KNOWN_FILE_SIGNATURES`] β€” magic-byte signatures for executable +//! and high-risk file formats (PE, ELF, Mach-O, OLE2, RTF, etc.). +//! - [`SHELLCODE_PATTERNS`] β€” known shellcode prologue byte sequences +//! (NOP sleds, syscall stubs, common encoders). +//! - [`COMMON_PHISHING_BRANDS`] β€” brand names frequently spoofed in +//! phishing URLs (microsoft, paypal, appleid, …). +//! +//! ## External threat-intel feeds +//! +//! Additional rules can be loaded at runtime via +//! [`load_external_rules`]. Loaded rules are stored in a +//! process-wide static and checked by every match function +//! alongside the built-in tables. + +use std::path::Path; +use std::sync::OnceLock; + +use serde::Deserialize; + +use crate::CorbelResult; + +/// TLDs statistically overrepresented in phishing URLs. +/// +/// Source: synthesized from public yearly phishing reports +/// (Spamhaus, PhishTank, Interisle). This list is intentionally +/// conservative β€” inclusion requires the TLD to appear in multiple +/// reports as a top-10 phishing TLD. +pub const PHISHING_TLDS: &[&str] = &[ + // High-risk TLDs (cheap registration, low verification) + ".zip", ".mov", ".xyz", ".top", ".click", ".link", ".rest", ".cyou", + ".sbs", ".online", ".live", ".buzz", ".surf", ".monster", ".fit", + ".loan", ".win", ".download", ".stream", ".review", ".men", + ".work", ".racing", ".party", ".trade", ".science", ".kim", + ".cricket", ".gq", ".cf", ".tk", ".ml", ".ga", + // Country-code TLDs frequently abused for phishing + ".ru", ".cn", ".su", ".country", ".kim", + // Newer TLDs that have been flagged + ".quest", ".bond", ".ha", ".cyou", ".quest", ".beauty", +]; + +/// URL path / host keywords that strongly suggest credential harvesting +/// or fake login pages. Matched case-insensitively as substrings. +pub const SUSPICIOUS_URL_KEYWORDS: &[&str] = &[ + "login", "signin", "sign-in", "log-in", "verify", "verification", + "account", "update", "confirm", "secure", "security", "wallet", + "unlock", "recover", "reactivate", "validate", "activate", + "webscr", "cmd=", "_session", "authorization", "authenticate", + "reset", "password", "credential", "billing", "invoice", + "support", "suspended", "limited", "alert", "warning", + "urgent", "important-notice", "tax", "refund", "irs", + "postbank", "amzn", "appleid", "icloud", "office365", +]; + +/// Common URL-shortener domains. Shortened URLs are not malicious +/// per se, but they hide the real destination β€” we flag them as +/// `Suspicious` so the operator can preview the destination before +/// clicking. +pub const URL_SHORTENER_DOMAINS: &[&str] = &[ + "bit.ly", "t.co", "tinyurl.com", "goo.gl", "ow.ly", "is.gd", + "buff.ly", "rebrand.ly", "cutt.ly", "shorturl.at", "tiny.cc", + "rb.gy", "s.id", "v.gd", "qr.ae", "x.co", "shorte.st", + "soo.gd", "lnkd.in", "po.st", "yourls.org", "bl.ink", + "surl.li", "kutt.it", "urlzs.com", "shrtco.de", +]; + +/// Brand names frequently spoofed in phishing URLs. Used to detect +/// homograph attacks (e.g. `micros0ft.com`, `paypa1.com`). +/// +/// This list intentionally includes BOTH canonical spellings ("microsoft") +/// AND known homograph variants ("micros0ft" with zero instead of 'o'). +/// The matcher uses a canonical-domain check to suppress benign +/// matches: when a brand is mentioned, we look for the canonical +/// spelling followed by a TLD; if found, we don't flag. +pub const COMMON_PHISHING_BRANDS: &[&str] = &[ + // Microsoft family + "microsoft", "micros0ft", "micros0fte", "micr0soft", + "msn", "windows", "wind0ws", "office", "0ffice", "outlook", + "outl00k", "outl0ok", "live", "1ive", + // PayPal + "paypal", "paypa1", "paypaI", "paypa|", + // Apple + "apple", "app1e", "appie", "icloud", "ic1oud", "appleid", + "app1eid", + // Google + "google", "g00gle", "goog1e", "gmail", "gmai", + // Amazon + "amazon", "amzn", "amaz0n", "a-m-a-z-o-n", + // Social + "facebook", "faceb00k", "facebo0k", "instagram", "instagrarn", + "twitter", "tw1tter", "twtter", "linkedin", "1inkedin", + // Streaming + "netflix", "netf1ix", "spotify", "spot1fy", + // Storage / SaaS + "dropbox", "dr0pbox", "adobe", "ad0be", + // Banking + "bankofamerica", "bofa", "b0fa", "wellsfargo", "wellsfarg0", + "chase", "citibank", "citi", "hsbc", "barclays", + "santander", "unicredit", + // Crypto + "binance", "binanc3", "coinbase", "c0inbase", "metamask", + "metam4sk", "ledger", "trezor", + // Shipping + "dhl", "fedex", "f3dex", "ups", "usps", "royalmail", + // Gaming + "steamcommunity", "steampowered", "epicgames", "playstation", + "nintendo", "xbox", +]; + +/// Magic-byte signatures for executable and high-risk file formats. +/// +/// Each entry is (offset, magic_bytes, name). When a payload's bytes +/// at `offset` match `magic_bytes`, the payload is considered +/// executable / high-risk. +pub const KNOWN_FILE_SIGNATURES: &[(usize, &[u8], &str)] = &[ + // Windows PE + (0, b"MZ", "pe"), + // ELF + (0, b"\x7FELF", "elf"), + // Mach-O fat binary + (0, b"\xCA\xFE\xBA\xBE", "mach-o-fat"), + // Mach-O 64-bit (big-endian) + (0, b"\xFE\xED\xFA\xCF", "mach-o-64-be"), + // Mach-O 64-bit (little-endian) + (0, b"\xCF\xFA\xED\xFE", "mach-o-64-le"), + // Mach-O 32-bit (big-endian) + (0, b"\xFE\xED\xFA\xFE", "mach-o-32-be"), + // Mach-O 32-bit (little-endian) + (0, b"\xCE\xFA\xED\xFE", "mach-o-32-le"), + // OLE2 (Microsoft Office legacy, also used by some malware) + (0, b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1", "ole2"), + // RTF (often used as an obfuscation vector for CVE-2017-11882 etc.) + (0, b"{\\rtf", "rtf"), + // Java class file + (0, b"\xCA\xFE\xBA\xBE", "java-class"), + // Java JAR (zip, but flag if inside PDF) + // (zip is too generic β€” we don't flag it without other signals) + // Python bytecode + (0, b"\x42\x0d\x0d\x0a", "python-bytecode"), + // SWF (Flash β€” historically a huge attack surface) + (0, b"FWS", "swf"), + (0, b"CWS", "swf-compressed"), + (0, b"ZWS", "swf-lzma"), + // Windows Help file (.hlp) β€” old but still seen in attacks + (0, b"?_\x03\x00", "winhelp"), + // Windows shortcut (.lnk) β€” common payload in phishing docs + (0, b"\x4c\x00\x00\x00\x01\x14\x02\x00", "lnk"), + // HTA application (HTML Application β€” executes as a script) + (0, b", + name: String, +} + +/// Processed external rules, organized by type for fast matching. +#[derive(Debug, Clone, Default)] +pub(crate) struct ExternalRulesData { + /// Additional TLD strings. + pub(crate) tlds: Vec, + /// Additional suspicious URL keywords. + pub(crate) keywords: Vec, + /// Additional brand strings (may include homograph variants). + pub(crate) brands: Vec, + /// Additional file-signature entries: (offset, magic-bytes, name). + pub(crate) signatures: Vec<(usize, Vec, String)>, + /// Additional shellcode byte patterns. + pub(crate) shellcode: Vec>, +} + +/// Process-wide storage for externally loaded rules. +/// +/// Populated once by [`load_external_rules`] and then read +/// (immutably) by every match function. +static EXTERNAL_RULES_STORAGE: OnceLock = OnceLock::new(); + +/// Load external signature rules from a JSON file. +/// +/// The file must contain a JSON array of [`ExternalRule`] objects. +/// Rules are sorted into type-specific buckets and stored in a +/// process-wide static ([`EXTERNAL_RULES_STORAGE`]). Subsequent calls +/// to the match functions (`match_file_signature`, `has_phishing_tld`, +/// etc.) will check both the built-in tables and the external rules. +/// +/// # Errors +/// +/// Returns [`CorbelError::Io`] if the file cannot be read, or +/// [`CorbelError::Serde`] if the JSON is malformed. +pub fn load_external_rules(path: &Path) -> CorbelResult> { + let data = std::fs::read_to_string(path)?; + let rules: Vec = serde_json::from_str(&data)?; + + let mut storage = ExternalRulesData::default(); + + for rule in &rules { + match rule.rule_type.as_str() { + "tld-list" => { + if let Some(arr) = rule.values.as_array() { + for v in arr { + if let Some(s) = v.as_str() { + storage.tlds.push(s.to_string()); + } + } + } + } + "keyword-list" => { + if let Some(arr) = rule.values.as_array() { + for v in arr { + if let Some(s) = v.as_str() { + storage.keywords.push(s.to_string()); + } + } + } + } + "brand-list" => { + if let Some(arr) = rule.values.as_array() { + for v in arr { + if let Some(s) = v.as_str() { + storage.brands.push(s.to_string()); + } + } + } + } + "signature-list" => { + if let Some(arr) = rule.values.as_array() { + for v in arr { + if let Ok(sig) = serde_json::from_value::(v.clone()) { + storage + .signatures + .push((sig.offset, sig.bytes, sig.name)); + } + } + } + } + "shellcode-list" => { + if let Some(arr) = rule.values.as_array() { + for v in arr { + if let Some(hex_str) = v.as_str() { + // Hex-encoded string: "fc4883e4..." + if let Ok(bytes) = hex::decode(hex_str) { + if !bytes.is_empty() { + storage.shellcode.push(bytes); + } + } + } else if let Some(byte_arr) = v.as_array() { + // Raw byte array: [0xfc, 0x48, ...] + let bytes: Vec = byte_arr + .iter() + .filter_map(|b| b.as_u64().map(|n| n as u8)) + .collect(); + if !bytes.is_empty() { + storage.shellcode.push(bytes); + } + } + } + } + } + _ => { + // Unknown rule type β€” silently skip. + } + } + } + + let _ = EXTERNAL_RULES_STORAGE.set(storage); + Ok(rules) +} + +/// Return a reference to the externally loaded rules storage, if any +/// has been loaded via [`load_external_rules`]. +/// +/// This is `pub(crate)` because the return type (`ExternalRulesData`) +/// is itself `pub(crate)` β€” exposing it publicly would leak a private +/// type through the public API. +#[must_use] +pub(crate) fn get_external_rules_storage() -> Option<&'static ExternalRulesData> { + EXTERNAL_RULES_STORAGE.get() +} + +/// Check whether `bytes` starts with any known executable / high-risk +/// file signature. +/// +/// Returns the signature name (e.g. `"pe"`, `"elf"`) if matched, so +/// the caller can include it in the forensic report. +/// +/// Checks both the built-in table and any external rules loaded via +/// [`load_external_rules`]. +#[must_use] +pub fn match_file_signature(bytes: &[u8]) -> Option<&'static str> { + for (offset, magic, name) in KNOWN_FILE_SIGNATURES { + if bytes.len() >= *offset + magic.len() { + if &bytes[*offset..*offset + magic.len()] == *magic { + return Some(name); + } + } + } + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + for (offset, magic, name) in &ext.signatures { + if bytes.len() >= *offset + magic.len() { + if &bytes[*offset..*offset + magic.len()] == magic.as_slice() { + return Some(name); + } + } + } + } + None +} + +/// Check whether `bytes` contains any known shellcode prologue. +/// +/// Checks both the built-in table and any external rules loaded via +/// [`load_external_rules`]. +#[must_use] +pub fn match_shellcode_pattern(bytes: &[u8]) -> Option<&'static [u8]> { + for pattern in SHELLCODE_PATTERNS { + if bytes.windows(pattern.len()).any(|w| w == *pattern) { + return Some(pattern); + } + } + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + for pattern in &ext.shellcode { + if bytes.windows(pattern.len()).any(|w| w == pattern.as_slice()) { + return Some(pattern.as_slice()); + } + } + } + None +} + +/// Check whether `host` (lowercase, no scheme) is a known URL shortener. +#[must_use] +pub fn is_url_shortener(host: &str) -> bool { + URL_SHORTENER_DOMAINS.iter().any(|d| host == *d || host.ends_with(&format!(".{d}"))) +} + +/// Check whether `uri` mentions a commonly-phished brand with a +/// non-canonical domain (homograph bait). +/// +/// Returns the matched brand name if found. +/// +/// Logic: +/// 1. If a homograph variant (`micros0ft`, `paypa1`, ...) is found +/// anywhere in the URI, it's always a phishing signal. +/// 2. If a canonical spelling (`microsoft`, `paypal`, ...) is found, +/// we check whether the host portion is ANY brand's canonical +/// domain (e.g. `microsoft.com` for "microsoft"). If yes β†’ benign. +/// Otherwise, the brand is mentioned in a non-canonical context +/// β†’ suspicious. +/// +/// Checks both the built-in table and any external brands loaded via +/// [`load_external_rules`]. +#[must_use] +pub fn match_phishing_brand(uri: &str) -> Option<&'static str> { + let lower = uri.to_ascii_lowercase(); + + // Extract host (after scheme://, before path/query/fragment, sans port). + let host = lower + .split("://") + .nth(1) + .unwrap_or(&lower) + .split('/') + .next() + .unwrap_or("") + .split(':') + .next() + .unwrap_or(""); + + let is_homograph = |brand: &str| brand.chars().any(|c| !c.is_ascii_alphabetic()); + + // First pass: check for homograph variants β€” these are ALWAYS phishing. + // Built-in brands. + for brand in COMMON_PHISHING_BRANDS { + if is_homograph(brand) && lower.contains(brand) { + return Some(brand); + } + } + // External brands. + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + for brand in &ext.brands { + if is_homograph(brand) && lower.contains(brand.as_str()) { + return Some(brand); + } + } + } + + // Second pass: check canonical spellings. If the host is ANY + // brand's canonical domain, all canonical brand mentions are + // treated as benign. This handles cases like `microsoft.com/windows` + // (windows is a brand, but the host is microsoft's canonical domain). + let host_is_canonical_for_builtin = COMMON_PHISHING_BRANDS + .iter() + .any(|brand| !is_homograph(brand) && is_canonical_brand_host(host, brand)); + let host_is_canonical_for_external = EXTERNAL_RULES_STORAGE + .get() + .map(|ext| { + ext.brands + .iter() + .any(|brand| !is_homograph(brand) && is_canonical_brand_host(host, brand)) + }) + .unwrap_or(false); + + if host_is_canonical_for_builtin || host_is_canonical_for_external { + return None; + } + + // Host is not a canonical brand domain β€” any canonical brand + // mentioned in the URL is suspicious. + // Built-in brands. + for brand in COMMON_PHISHING_BRANDS { + if !is_homograph(brand) && lower.contains(brand) { + return Some(brand); + } + } + // External brands. + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + for brand in &ext.brands { + if !is_homograph(brand) && lower.contains(brand.as_str()) { + return Some(brand); + } + } + } + + None +} + +/// Check whether `host` is the canonical domain for `brand`. +/// +/// A host is canonical if it matches `.` or has `` +/// as a dot-separated segment (e.g. `microsoft.com`, `login.microsoft.com`). +/// The goal is to allow legitimate brand-owned domains while still +/// flagging `login-microsoft.com` (which is NOT a Microsoft domain). +fn is_canonical_brand_host(host: &str, brand: &str) -> bool { + if host == brand { + return true; + } + // Check if `.` is a prefix. + let canonical_prefix = format!("{}.", brand); + if host.starts_with(&canonical_prefix) { + return true; + } + // Check if `` is a dot-separated segment (e.g. `login.microsoft.com`). + host.split('.').any(|seg| seg == brand) +} + +/// Check whether `uri`'s path/host contains any suspicious keyword. +/// +/// Checks both the built-in table and any external rules loaded via +/// [`load_external_rules`]. +#[must_use] +pub fn match_suspicious_keyword(uri: &str) -> Option<&'static str> { + let lower = uri.to_ascii_lowercase(); + if let Some(kw) = SUSPICIOUS_URL_KEYWORDS + .iter() + .copied() + .find(|kw| lower.contains(kw)) + { + return Some(kw); + } + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + if let Some(kw) = ext + .keywords + .iter() + .find(|kw| lower.contains(kw.as_str())) + { + return Some(kw); + } + } + None +} + +/// Check whether the host part of `uri` ends with a known phishing TLD. +/// +/// Checks both the built-in table and any external rules loaded via +/// [`load_external_rules`]. +#[must_use] +pub fn has_phishing_tld(uri: &str) -> Option<&'static str> { + let lower = uri.to_ascii_lowercase(); + // Extract host portion (after scheme://, before path/query/fragment). + let host = lower + .split("://") + .nth(1) + .unwrap_or(&lower) + .split('/') + .next() + .unwrap_or(""); + // Strip port. + let host = host.split(':').next().unwrap_or(""); + if let Some(tld) = PHISHING_TLDS.iter().copied().find(|tld| host.ends_with(tld)) { + return Some(tld); + } + if let Some(ext) = EXTERNAL_RULES_STORAGE.get() { + if let Some(tld) = ext.tlds.iter().find(|tld| host.ends_with(tld.as_str())) { + return Some(tld); + } + } + None +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn detects_pe_signature() { + assert_eq!(match_file_signature(b"MZ\x90\x00\x03"), Some("pe")); + } + + #[test] + fn detects_elf_signature() { + assert_eq!(match_file_signature(b"\x7FELF\x02"), Some("elf")); + } + + #[test] + fn detects_ole2_signature() { + assert_eq!( + match_file_signature(b"\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1rest"), + Some("ole2") + ); + } + + #[test] + fn detects_rtf_signature() { + assert_eq!(match_file_signature(b"{\\rtf1\\ansi..."), Some("rtf")); + } + + #[test] + fn detects_swf_signature() { + assert_eq!(match_file_signature(b"FWS\x09"), Some("swf")); + } + + #[test] + fn detects_lnk_signature() { + assert_eq!( + match_file_signature(b"\x4c\x00\x00\x00\x01\x14\x02\x00rest"), + Some("lnk") + ); + } + + #[test] + fn detects_vba_macro_signature() { + assert_eq!( + match_file_signature(b"Attribute VB_Name = \"evil\""), + Some("vba-macro") + ); + } + + #[test] + fn detects_nop_sled_pattern() { + let pattern = match_shellcode_pattern(&[0x90; 32]).unwrap(); + assert_eq!(pattern[0], 0x90); + } + + #[test] + fn detects_metasploit_stager_pattern() { + let payload = [0xFC, 0xE8, 0x82, 0x00, 0x00, 0x00, 0x60, 0x89]; + assert!(match_shellcode_pattern(&payload).is_some()); + } + + #[test] + fn detects_url_shortener() { + assert!(is_url_shortener("bit.ly")); + assert!(is_url_shortener("sub.bit.ly")); + assert!(!is_url_shortener("example.com")); + } + + #[test] + fn detects_phishing_brand_homograph() { + // micros0ft.com (with zero instead of 'o') should match the + // homograph variant directly. + assert_eq!( + match_phishing_brand("https://micros0ft.com/login"), + Some("micros0ft") + ); + // paypa1.com (with one instead of 'l') should match. + assert_eq!( + match_phishing_brand("https://paypa1.com/signin"), + Some("paypa1") + ); + } + + #[test] + fn canonical_brand_domain_not_flagged() { + // microsoft.com (canonical) should not be flagged. + assert!(match_phishing_brand("https://microsoft.com/windows").is_none()); + // paypal.com (canonical) should not be flagged. + assert!(match_phishing_brand("https://paypal.com/home").is_none()); + } + + #[test] + fn canonical_brand_in_non_canonical_domain_is_flagged() { + // microsoft mentioned in a non-canonical host β†’ suspicious. + assert_eq!( + match_phishing_brand("https://login-microsoft.com/verify"), + Some("microsoft") + ); + } + + #[test] + fn detects_suspicious_keyword() { + // Should return the first matching keyword β€” both "account" + // and "verify" are in the list. Either is acceptable; check + // that we get one of them. + let result = match_suspicious_keyword("https://example.com/account/verify"); + assert!(matches!(result, Some("account") | Some("verify"))); + assert_eq!( + match_suspicious_keyword("https://example.com/signin"), + Some("signin") + ); + } + + #[test] + fn detects_phishing_tld() { + assert_eq!(has_phishing_tld("https://example.xyz"), Some(".xyz")); + assert_eq!(has_phishing_tld("https://example.top/path"), Some(".top")); + assert!(has_phishing_tld("https://example.com").is_none()); + } + + #[test] + fn detects_phishing_tld_with_port() { + assert_eq!( + has_phishing_tld("https://example.xyz:8080/path"), + Some(".xyz") + ); + } +} diff --git a/src/study/annotator.rs b/src/study/annotator.rs new file mode 100755 index 0000000..7df34b0 --- /dev/null +++ b/src/study/annotator.rs @@ -0,0 +1,266 @@ +//! Annotated HTML builder for study mode. +//! +//! Takes the original document bytes and the scan report, and produces +//! a self-contained HTML file with inline annotations around findings. +//! +//! For formats where we have structured content (PDF, EPUB, DOCX), +//! we extract text with location info and wrap findings in ``. +//! For Markdown, we parse the source directly and inject spans. + +use crate::core::types::{DocumentFormat, ScanReport, ThreatClassification, Finding}; + +/// Build the complete study-mode HTML document. +pub fn build_study_html( + raw_bytes: &[u8], + scan_report: &ScanReport, + sha256: &str, + source_path: &std::path::Path, +) -> String { + let format = scan_report.format; + + // Build a mapping from location string to findings for O(1) lookup. + let mut location_findings: std::collections::HashMap> = + std::collections::HashMap::new(); + for finding in &scan_report.findings { + location_findings + .entry(finding.location.to_string()) + .or_default() + .push(finding); + } + + let body_content = match format { + DocumentFormat::Markdown => build_markdown_study(raw_bytes, &location_findings), + _ => build_generic_study(raw_bytes, format, &location_findings), + }; + + let filename = source_path + .file_name() + .and_then(|n| n.to_str()) + .unwrap_or("unknown"); + + format!( + r#" + + + + +CorbelPurge Study: {filename} + + + +

CorbelPurge Study Mode

+
+
+
File
{filename}
+
SHA-256
{sha256}
+
Format
{format}
+
Text nodes
{text_nodes}
+
Vectors
{vectors}
+
Findings
{total}
+
+
+
+ Malicious: {malicious} + Suspicious: {suspicious} + Educational: {educational} +
+{body_content} + +"#, + filename = html_escape(filename), + sha256 = sha256, + format = format, + text_nodes = scan_report.text_nodes_scanned, + vectors = scan_report.vectors_scanned, + total = scan_report.findings.len(), + malicious = scan_report.malicious_count(), + suspicious = scan_report.findings.iter().filter(|f| matches!(f.classification, ThreatClassification::Suspicious)).count(), + educational = scan_report.educational_count(), + body_content = body_content, + ) +} + +/// Build study HTML for Markdown source files. +/// +/// Parses the Markdown line-by-line and wraps matching content in spans. +fn build_markdown_study( + raw: &[u8], + location_findings: &std::collections::HashMap>, +) -> String { + let text = String::from_utf8_lossy(raw); + let mut out = String::new(); + out.push_str("
\n");
+
+    for (idx, line) in text.lines().enumerate() {
+        let line_num = idx + 1;
+        let location_key = format!("md:{}:0", line_num);
+
+        let mut line_html = html_escape(line);
+
+        if let Some(findings) = location_findings.get(&location_key) {
+            for finding in findings {
+                let class = classification_class(&finding.classification);
+                line_html = format!(
+                    "{} {}",
+                    class, line_html, classification_label(&finding.classification)
+                );
+            }
+        }
+
+        out.push_str(&format!(
+            "{line_num:>4}{}\n",
+            line_html
+        ));
+    }
+
+    out.push_str("
\n"); + out +} + +/// Build study HTML for non-Markdown formats (PDF, EPUB, DOCX). +/// +/// Since we can't reliably reconstruct the original rendering, we show +/// the findings in a structured list with their payload previews. +/// +/// The `raw` bytes and `format` are accepted for symmetry with +/// [`build_markdown_study`] and for future per-format rendering hooks, +/// but are not currently consumed by this implementation. +#[allow(unused_variables)] +fn build_generic_study( + raw: &[u8], + format: DocumentFormat, + location_findings: &std::collections::HashMap>, +) -> String { + let mut out = String::new(); + + // Group findings by location for ordered display. + let mut seen = std::collections::HashSet::new(); + for finding in location_findings + .values() + .flatten() + .collect::>() + { + if !seen.insert(finding.location.to_string()) { + continue; + } + let class = classification_class(&finding.classification); + out.push_str(&format!( + "
\n", + class + )); + out.push_str(&format!( + "
{}
\n", + class, + classification_label(&finding.classification) + )); + out.push_str(&format!( + "
{}
\n", + finding.location + )); + out.push_str(&format!( + "
{}
\n", + html_escape(&finding.context_notes) + )); + if !finding.payload_preview.is_empty() { + out.push_str(&format!( + "
{}
\n", + html_escape(&finding.payload_preview) + )); + } + out.push_str("
\n"); + } + + // If no findings, show raw content in a code block. + if location_findings.is_empty() { + out.push_str("

No findings. Document passed all checks.

\n"); + } + + out +} + +fn classification_class(c: &ThreatClassification) -> &'static str { + match c { + ThreatClassification::Benign => "benign", + ThreatClassification::Suspicious => "suspicious", + ThreatClassification::EducationalContent => "educational", + ThreatClassification::Malicious(_) => "malicious", + } +} + +fn classification_label(c: &ThreatClassification) -> &'static str { + match c { + ThreatClassification::Benign => "benign", + ThreatClassification::Suspicious => "suspicious", + ThreatClassification::EducationalContent => "educational", + ThreatClassification::Malicious(_) => "malicious", + } +} + +fn html_escape(s: &str) -> String { + s.replace('&', "&") + .replace('<', "<") + .replace('>', ">") + .replace('"', """) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn html_escape_basic() { + assert_eq!(html_escape(""), "<script>alert('xss')</script>"); + } + + #[test] + fn build_study_html_has_structure() { + use crate::core::types::*; + let report = ScanReport { + source_sha256: "abc".to_string(), + format: DocumentFormat::Markdown, + scanned_at: "x".to_string(), + findings: vec![Finding { + classification: ThreatClassification::Malicious(MaliciousType::ActiveJavaScriptInjection), + location: Location::MarkdownLine { line: 3, col: 0 }, + vector_type: None, + payload_preview: "eval(1)".to_string(), + context_notes: "found evil".to_string(), + recommendation: Recommendation::QuarantineAndCleanse, + }], + text_nodes_scanned: 5, + vectors_scanned: 1, + }; + let html = build_study_html(b"hello\nworld\nevil", &report, "abc123", std::path::Path::new("test.md")); + assert!(html.contains("")); + assert!(html.contains(" 3")); + assert!(html.contains("CorbelPurge Study Mode")); + assert!(html.contains("Malicious:")); + } +} \ No newline at end of file diff --git a/src/study/mod.rs b/src/study/mod.rs new file mode 100755 index 0000000..85ed4cf --- /dev/null +++ b/src/study/mod.rs @@ -0,0 +1,73 @@ +//! Study mode: annotated HTML output showing exploit locations. +//! +//! The `corbel-purge study ` subcommand runs the full pipeline +//! and then renders the **original** document content with malicious +//! findings annotated inline. This is useful for researchers who want +//! to see *where* in the document an exploit was found, not just read +//! a report about it. +//! +//! The output is a self-contained HTML file with inline `` wrappers +//! around matched content, color-coded by classification: +//! +//! - **Red**: Malicious +//! - **Orange**: Suspicious +//! - **Green**: EducationalContent +//! - **Gray**: Benign (not annotated) + +pub mod annotator; + +use std::path::{Path, PathBuf}; + +use crate::{Config, CorbelResult, Pipeline}; + +/// Run study mode on a file, producing an annotated HTML file. +/// +/// Returns the path to the generated HTML file. +pub fn run_study( + path: impl AsRef, + workspace: Option<&Path>, +) -> CorbelResult { + let path = path.as_ref(); + let bytes = std::fs::read(path)?; + + let mut config = match workspace { + Some(p) => Config::with_workspace(p), + None => Config::default(), + }; + config.abort_on_threat = false; + config = config.override_from_env(); + + let pipeline = Pipeline::with_config(config.clone()); + let result = pipeline.run_on_bytes( + bytes.clone(), + crate::core::types::DocumentFormat::from_path(path)?, + Some(path.to_path_buf()), + )?; + + let study_dir = workspace + .map(|p| p.to_path_buf()) + .unwrap_or_else(|| std::env::current_dir().unwrap()); + let study_dir = study_dir.join("corbel_study"); + std::fs::create_dir_all(&study_dir)?; + + let timestamp = chrono::Utc::now().format("%Y%m%dT%H%M%S"); + let sha_prefix = &result.source_sha256[..8.min(result.source_sha256.len())]; + let html_filename = format!("study_{timestamp}_{sha_prefix}.html"); + let html_path = study_dir.join(&html_filename); + + let html = annotator::build_study_html( + &bytes, + &result.scan_report, + &result.source_sha256, + path, + ); + std::fs::write(&html_path, html)?; + + // Also write the JSON report for programmatic access. + let json_path = study_dir.join(format!("study_{timestamp}_{sha_prefix}.json")); + if let Some(ref report_path) = result.json_report_path { + std::fs::copy(report_path, &json_path)?; + } + + Ok(html_path) +} diff --git a/src/ui/alert_modal.rs b/src/ui/alert_modal.rs new file mode 100755 index 0000000..f8b9401 --- /dev/null +++ b/src/ui/alert_modal.rs @@ -0,0 +1,35 @@ +//! Quarantine & threat notification overlay β€” currently a stub. +//! +//! A full implementation would render a modal dialog over the viewer +//! when a threat is detected, with options to view the forensic report +//! or open the quarantine tarball. + +use crate::core::ScanReport; + +/// State for the alert modal. +#[derive(Debug, Default)] +pub struct AlertModal { + /// Whether the modal is currently shown. + pub visible: bool, + /// The scan report being displayed, if any. + pub report: Option, +} + +impl AlertModal { + /// Construct a new, hidden modal. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Show the modal with the given report. + pub fn show(&mut self, report: ScanReport) { + self.report = Some(report); + self.visible = true; + } + + /// Dismiss the modal. + pub fn dismiss(&mut self) { + self.visible = false; + } +} diff --git a/src/ui/mod.rs b/src/ui/mod.rs new file mode 100755 index 0000000..ca4b2a9 --- /dev/null +++ b/src/ui/mod.rs @@ -0,0 +1,16 @@ +//! GUI module: iced-based viewer. +//! +//! This module is **gated behind the `gui` feature flag**. The default +//! build of CorbelPurge produces only the headless CLI binary, which is +//! what gets used in CI pipelines and server-side scanners. The GUI is +//! an optional build that produces a separate binary. +//! +//! In the current MVP, the GUI is a stub: it can open a file picker, +//! run the pipeline, and display the scan summary. A full rendered +//! canvas (text + images) is left as future work. + +pub mod viewer; +pub mod alert_modal; + +pub use viewer::Viewer; +pub use alert_modal::AlertModal; diff --git a/src/ui/viewer.rs b/src/ui/viewer.rs new file mode 100755 index 0000000..a69c754 --- /dev/null +++ b/src/ui/viewer.rs @@ -0,0 +1,36 @@ +//! Canvas renderer (text/images) β€” currently a stub. +//! +//! A full implementation would render cleansed document content to an +//! iced canvas. For the MVP, this module just holds the state and +//! delegates to the pipeline for the actual scanning work. + +use crate::PipelineResult; + +/// Top-level viewer state. +#[derive(Debug, Default)] +pub struct Viewer { + /// The most recent scan result, if any. + pub last_result: Option, + /// Whether the viewer is currently rendering a document. + pub is_rendering: bool, +} + +impl Viewer { + /// Construct a new, empty viewer. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Update the viewer with a fresh scan result. + pub fn set_result(&mut self, result: PipelineResult) { + self.last_result = Some(result); + self.is_rendering = false; + } + + /// Clear the viewer. + pub fn clear(&mut self) { + self.last_result = None; + self.is_rendering = false; + } +} diff --git a/src/util.rs b/src/util.rs new file mode 100755 index 0000000..20dc800 --- /dev/null +++ b/src/util.rs @@ -0,0 +1,286 @@ +//! Small utility helpers shared across the crate. +//! +//! Kept in a dedicated module so that tests and downstream code have +//! one place to find cross-cutting helpers (hashing, truncation, etc.). + +use std::io::Read; + +use sha2::{Digest, Sha256}; + +/// Compute the SHA-256 hex digest of `bytes`. +/// +/// Returned string is always 64 lowercase hex characters. +#[must_use] +pub fn sha256_hex(bytes: &[u8]) -> String { + let mut hasher = Sha256::new(); + hasher.update(bytes); + let digest = hasher.finalize(); + hex::encode(digest) +} + +/// Truncate `s` to at most `max_chars` characters, appending an ellipsis +/// if truncation occurred. Useful for payload previews in threat reports. +#[must_use] +pub fn truncate_with_ellipsis(s: &str, max_chars: usize) -> String { + if s.chars().count() <= max_chars { + return s.to_string(); + } + let truncated: String = s.chars().take(max_chars.saturating_sub(1)).collect(); + format!("{truncated}…") +} + +/// Outcome of a streaming read with a byte cap. +/// +/// The key insight: we count **actual bytes consumed from the stream**, +/// not the size declared in any header. A malicious ZIP archive can +/// declare `size = 100` in its central directory while actually +/// decompressing to gigabytes β€” [`read_with_cap`] detects this by +/// aborting mid-stream once `cap` bytes have been read. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ReadOutcome { + /// The stream reached EOF before the cap was hit. Contains all + /// bytes that were read. + Complete(Vec), + /// The cap was hit before EOF. Contains the bytes read so far + /// (exactly `cap` bytes, or fewer if the read stopped mid-chunk), + /// plus the total number of bytes consumed from the stream + /// (which may be slightly larger than `bytes.len()` if the final + /// chunk pushed us over the cap). + /// + /// Callers should treat this as a likely zip-bomb / oversized + /// payload and emit a suspicious finding rather than continuing + /// to parse the truncated content. + Truncated { + /// The bytes we managed to read before stopping (≀ cap). + bytes: Vec, + /// Total bytes consumed from the stream before we stopped + /// reading. Always β‰₯ `bytes.len()` and always > `cap`. + bytes_read: usize, + /// The cap that was exceeded. + cap: usize, + }, +} + +/// Read from `reader` into a `Vec`, but stop once `cap` bytes have +/// been consumed. +/// +/// This is the **zip-bomb defense** for ZIP-container formats (EPUB, +/// DOCX). The size declared in the ZIP central directory is *not +/// trusted* β€” we count actual decompressed bytes via repeated +/// `reader.read()` calls. +/// +/// ## How it works +/// +/// 1. Allocate a buffer with capacity `min(cap, 1 MiB)` (we don't +/// trust `reader`'s declared size). +/// 2. Read in 8 KiB chunks until either: +/// - `reader.read()` returns `Ok(0)` (EOF) β†’ return `Complete`. +/// - The buffer would exceed `cap` β†’ return `Truncated` with the +/// bytes read so far. +/// 3. On any I/O error, propagate it. +/// +/// ## Why 8 KiB chunks +/// +/// Small enough that we check the cap frequently (a malicious stream +/// can't sneak through too many bytes between checks), large enough +/// that the per-call overhead is negligible vs. the decompressor's +/// internal buffering. +/// +/// ## What this defends against +/// +/// - **Lying size header**: a ZIP entry that declares `size = 100` but +/// actually decompresses to 100 MB. The streaming reader detects +/// this after `cap` bytes regardless of what the header said. +/// - **Decompression ratio attack**: a 42 KB ZIP that decompresses to +/// petabytes (the famous 42.zip). The streaming reader stops at +/// `cap` bytes and never allocates more. +/// +/// ## What this does NOT defend against +/// +/// - **Pre-allocated capacity attacks**: if `cap` itself is huge +/// (e.g. 1 GiB), we'll happily allocate that much. Set `cap` to a +/// sane value (the default is 8 MiB). +/// - **Many small entries**: an archive with 10000 entries of 1 MiB +/// each still totals 10 GiB of memory. The per-entry cap doesn't +/// help here β€” the pipeline coordinator would need a total-memory +/// budget. (Future work.) +pub fn read_with_cap( + reader: &mut R, + cap: usize, +) -> std::io::Result { + // Refuse a zero cap β€” that would always truncate immediately and + // is almost certainly a caller bug. + assert!(cap > 0, "read_with_cap: cap must be > 0"); + + // Pre-allocate up to 1 MiB or `cap`, whichever is smaller. + // We deliberately do NOT pre-allocate `cap` bytes β€” if `cap` is + // 1 GiB, that would itself be the DoS. + let initial_capacity = cap.min(1024 * 1024); + let mut buf = Vec::with_capacity(initial_capacity); + + // 8 KiB chunk buffer. + let mut chunk = [0u8; 8 * 1024]; + + loop { + let n = reader.read(&mut chunk)?; + if n == 0 { + // EOF β€” we read the whole stream within the cap. + return Ok(ReadOutcome::Complete(buf)); + } + + // Check if appending would push us over the cap. + // We append the full chunk even if it overshoots β€” that way + // the caller gets a clean byte boundary to inspect (e.g. for + // file-signature matching on the first few bytes). + if buf.len() + n > cap { + // Append what fits. + let remaining = cap.saturating_sub(buf.len()); + buf.extend_from_slice(&chunk[..remaining]); + let bytes_read = buf.len(); + return Ok(ReadOutcome::Truncated { + bytes: buf, + bytes_read, + cap, + }); + } + + buf.extend_from_slice(&chunk[..n]); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Cursor; + + #[test] + fn sha256_known_vector() { + // SHA-256 of empty input. + assert_eq!( + sha256_hex(b""), + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + ); + // SHA-256 of "abc" + assert_eq!( + sha256_hex(b"abc"), + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad" + ); + } + + #[test] + fn truncate_short_string_unchanged() { + assert_eq!(truncate_with_ellipsis("hello", 10), "hello"); + } + + #[test] + fn truncate_long_string_gets_ellipsis() { + let result = truncate_with_ellipsis("abcdefghij", 5); + assert_eq!(result, "abcd…"); + } + + #[test] + fn truncate_exact_length_unchanged() { + assert_eq!(truncate_with_ellipsis("abcde", 5), "abcde"); + } + + // --- read_with_cap tests --- + + #[test] + fn read_small_stream_completes() { + let data = b"hello world".to_vec(); + let mut cursor = Cursor::new(data.clone()); + let outcome = read_with_cap(&mut cursor, 1024).unwrap(); + assert_eq!(outcome, ReadOutcome::Complete(data)); + } + + #[test] + fn read_empty_stream_completes() { + let mut cursor = Cursor::new(Vec::new()); + let outcome = read_with_cap(&mut cursor, 1024).unwrap(); + assert_eq!(outcome, ReadOutcome::Complete(Vec::new())); + } + + #[test] + fn read_exact_cap_completes() { + // 100 bytes, cap = 100 β†’ should complete exactly at the boundary. + let data = vec![0x41u8; 100]; + let mut cursor = Cursor::new(data.clone()); + let outcome = read_with_cap(&mut cursor, 100).unwrap(); + assert_eq!(outcome, ReadOutcome::Complete(data)); + } + + #[test] + fn read_just_over_cap_truncates() { + // 200 bytes, cap = 100 β†’ should truncate. + let data = vec![0x41u8; 200]; + let mut cursor = Cursor::new(data); + let outcome = read_with_cap(&mut cursor, 100).unwrap(); + match outcome { + ReadOutcome::Truncated { bytes, bytes_read, cap } => { + assert_eq!(cap, 100); + assert!(bytes.len() <= 100, "bytes.len() should be ≀ cap"); + assert_eq!(bytes_read, bytes.len()); + // We should have read *something* β€” at least the first chunk. + assert!(!bytes.is_empty()); + } + ReadOutcome::Complete(_) => panic!("should have truncated"), + } + } + + #[test] + fn read_huge_stream_with_small_cap_truncates_quickly() { + // Simulate a zip-bomb: 10 MiB of data, 8 KiB cap. + // We should truncate after reading ~8 KiB, not allocate 10 MiB. + let data = vec![0x41u8; 10 * 1024 * 1024]; + let mut cursor = Cursor::new(data); + let outcome = read_with_cap(&mut cursor, 8 * 1024).unwrap(); + match outcome { + ReadOutcome::Truncated { bytes, bytes_read, cap } => { + assert_eq!(cap, 8 * 1024); + assert!(bytes.len() <= 8 * 1024); + assert_eq!(bytes_read, bytes.len()); + // Critical: we never allocated more than cap. + assert!(bytes.len() <= 8 * 1024); + } + ReadOutcome::Complete(_) => panic!("should have truncated"), + } + } + + #[test] + fn read_propagates_io_errors() { + use std::io::{self, Read}; + + struct ErroringReader; + impl Read for ErroringReader { + fn read(&mut self, _buf: &mut [u8]) -> io::Result { + Err(io::Error::new(io::ErrorKind::Other, "synthetic error")) + } + } + + let mut reader = ErroringReader; + let result = read_with_cap(&mut reader, 1024); + assert!(result.is_err()); + assert_eq!( + result.unwrap_err().to_string(), + "synthetic error" + ); + } + + #[test] + fn read_returns_partial_bytes_on_truncation() { + // Verify the truncated bytes are actually the *first* bytes + // of the stream (so the caller can still do file-signature + // matching on the prefix). + let data: Vec = (0..200).map(|i| i as u8).collect(); + let mut cursor = Cursor::new(data); + let outcome = read_with_cap(&mut cursor, 50).unwrap(); + match outcome { + ReadOutcome::Truncated { bytes, .. } => { + // The first byte should be 0, the last should be ≀ 50. + assert_eq!(bytes[0], 0); + assert!(bytes.last().copied().unwrap() <= 50); + } + ReadOutcome::Complete(_) => panic!("should have truncated"), + } + } +} diff --git a/tests/fixtures/benign.docx b/tests/fixtures/benign.docx new file mode 100755 index 0000000..3de9436 Binary files /dev/null and b/tests/fixtures/benign.docx differ diff --git a/tests/fixtures/benign.epub b/tests/fixtures/benign.epub new file mode 100755 index 0000000..1d802b2 Binary files /dev/null and b/tests/fixtures/benign.epub differ diff --git a/tests/fixtures/benign.md b/tests/fixtures/benign.md new file mode 100755 index 0000000..6723fa9 --- /dev/null +++ b/tests/fixtures/benign.md @@ -0,0 +1,8 @@ +# Benign Document + +This is a paragraph of perfectly normal text. It discusses the weather, +the state of the economy, and other uncontroversial topics. + +## Subsection + +More text here. Nothing suspicious whatsoever. diff --git a/tests/fixtures/benign.pdf b/tests/fixtures/benign.pdf new file mode 100755 index 0000000..8e383cc --- /dev/null +++ b/tests/fixtures/benign.pdf @@ -0,0 +1,68 @@ +%PDF-1.3 +%“Œ‹ž ReportLab Generated PDF document (opensource) +1 0 obj +<< +/F1 2 0 R +>> +endobj +2 0 obj +<< +/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font +>> +endobj +3 0 obj +<< +/Contents 7 0 R /MediaBox [ 0 0 612 792 ] /Parent 6 0 R /Resources << +/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] +>> /Rotate 0 /Trans << + +>> + /Type /Page +>> +endobj +4 0 obj +<< +/PageMode /UseNone /Pages 6 0 R /Type /Catalog +>> +endobj +5 0 obj +<< +/Author (CorbelPurge Tests) /CreationDate (D:20260731131243+00'00') /Creator (anonymous) /Keywords () /ModDate (D:20260731131243+00'00') /Producer (ReportLab PDF Library - \(opensource\)) + /Subject (unspecified) /Title (Benign Test PDF) /Trapped /False +>> +endobj +6 0 obj +<< +/Count 1 /Kids [ 3 0 R ] /Type /Pages +>> +endobj +7 0 obj +<< +/Filter [ /ASCII85Decode /FlateDecode ] /Length 170 +>> +stream +GarVE]*\To&FAj9VGr[6.u.;d*XWYuqZARU)TciR>=AMJE*?Ab4GmQZ#d/[^%iqldbZY(m_&#pH2rFeH>a:hnYhF*[f?[endstream +endobj +xref +0 8 +0000000000 65535 f +0000000061 00000 n +0000000092 00000 n +0000000199 00000 n +0000000392 00000 n +0000000460 00000 n +0000000736 00000 n +0000000795 00000 n +trailer +<< +/ID +[<88c9b83603e2b9c29dc61f2d955be744><88c9b83603e2b9c29dc61f2d955be744>] +% ReportLab generated PDF document -- digest (opensource) + +/Info 5 0 R +/Root 4 0 R +/Size 8 +>> +startxref +1055 +%%EOF diff --git a/tests/fixtures/cve_writeup.md b/tests/fixtures/cve_writeup.md new file mode 100755 index 0000000..a974fef --- /dev/null +++ b/tests/fixtures/cve_writeup.md @@ -0,0 +1,21 @@ +# CVE-2024-1234: PDF JavaScript Injection + +## Abstract + +In this paper we describe a vulnerability in which a malicious PDF +uses a /JavaScript action to execute arbitrary code. The eval() function +is called with attacker-controlled input. + +## Proof of Concept + +```python +# This is a PoC for the vulnerability described above. +import subprocess +# Note: this code is for educational purposes only. +payload = "eval('alert(1)')" +print(f"Payload: {payload}") +``` + +## Remediation + +Patch the reader to ignore /JavaScript actions in /OpenAction. diff --git a/tests/fixtures/cve_writeup.pdf b/tests/fixtures/cve_writeup.pdf new file mode 100755 index 0000000..d10a096 --- /dev/null +++ b/tests/fixtures/cve_writeup.pdf @@ -0,0 +1,68 @@ +%PDF-1.3 +%“Œ‹ž ReportLab Generated PDF document (opensource) +1 0 obj +<< +/F1 2 0 R +>> +endobj +2 0 obj +<< +/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font +>> +endobj +3 0 obj +<< +/Contents 7 0 R /MediaBox [ 0 0 612 792 ] /Parent 6 0 R /Resources << +/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] +>> /Rotate 0 /Trans << + +>> + /Type /Page +>> +endobj +4 0 obj +<< +/PageMode /UseNone /Pages 6 0 R /Type /Catalog +>> +endobj +5 0 obj +<< +/Author (Security Researcher) /CreationDate (D:20260731131243+00'00') /Creator (anonymous) /Keywords () /ModDate (D:20260731131243+00'00') /Producer (ReportLab PDF Library - \(opensource\)) + /Subject (unspecified) /Title (CVE-2024-1234 Writeup) /Trapped /False +>> +endobj +6 0 obj +<< +/Count 1 /Kids [ 3 0 R ] /Type /Pages +>> +endobj +7 0 obj +<< +/Filter [ /ASCII85Decode /FlateDecode ] /Length 384 +>> +stream +Gas2D?VeNm'ZJu*'^'J'0\/6o_AasM(__le54R'D(O'lbQF6nkp3&e0%tlcRdei3thl/jRNYlY>!tQ$nV1;)[2\9,^6,$_M8"hR77R`Kur[rc'&NtRq->AWT'9V-`1V>k&nZ6p!6X+Fk"YF(6m_W]E+_A&p7_bpC@$W&W+o[n?NJPK>L=?mYtfg-\=["(/?$X'2g]'+/c2,:=o+]#p+73EPPX@"WmL\dXR%gNE<)0LsEBMQmZW+-CrJ::%55,d2*"0+LL2[.0^-%U6>>"@oK8SQYAj'i#;WiE#5SpbSO?KlC6'AXLDS"[Zqmendstream +endobj +xref +0 8 +0000000000 65535 f +0000000061 00000 n +0000000092 00000 n +0000000199 00000 n +0000000392 00000 n +0000000460 00000 n +0000000744 00000 n +0000000803 00000 n +trailer +<< +/ID +[<6e2b4056078bc06bbe77ecc8c05b4144><6e2b4056078bc06bbe77ecc8c05b4144>] +% ReportLab generated PDF document -- digest (opensource) + +/Info 5 0 R +/Root 4 0 R +/Size 8 +>> +startxref +1277 +%%EOF diff --git a/tests/fixtures/honest_zip_bomb.zip b/tests/fixtures/honest_zip_bomb.zip new file mode 100755 index 0000000..be591d6 Binary files /dev/null and b/tests/fixtures/honest_zip_bomb.zip differ diff --git a/tests/fixtures/lying_zip_bomb.zip b/tests/fixtures/lying_zip_bomb.zip new file mode 100755 index 0000000..19c742d Binary files /dev/null and b/tests/fixtures/lying_zip_bomb.zip differ diff --git a/tests/fixtures/malicious.epub b/tests/fixtures/malicious.epub new file mode 100755 index 0000000..247d319 Binary files /dev/null and b/tests/fixtures/malicious.epub differ diff --git a/tests/fixtures/malicious.md b/tests/fixtures/malicious.md new file mode 100755 index 0000000..8fd6631 --- /dev/null +++ b/tests/fixtures/malicious.md @@ -0,0 +1,5 @@ +# Click Here + +Free money! [Click now](javascript:alert('xss')) + +Run this: \x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90 diff --git a/tests/fixtures/malicious_js.pdf b/tests/fixtures/malicious_js.pdf new file mode 100755 index 0000000..8035456 --- /dev/null +++ b/tests/fixtures/malicious_js.pdf @@ -0,0 +1,86 @@ +%PDF-1.3 +%βγΟΣ +1 0 obj +<< +/Producer (pypdf) +>> +endobj +2 0 obj +<< +/Type /Pages +/Count 1 +/Kids [ 4 0 R ] +>> +endobj +3 0 obj +<< +/Type /Catalog +/Pages 2 0 R +/OpenAction 8 0 R +>> +endobj +4 0 obj +<< +/Contents 5 0 R +/MediaBox [ 0 0 612 792 ] +/Resources << +/Font 6 0 R +/ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] +>> +/Rotate 0 +/Trans << +>> +/Type /Page +/Parent 2 0 R +>> +endobj +5 0 obj +<< +/Filter [ /ASCII85Decode /FlateDecode ] +/Length 130 +>> +stream +GapQh0E=F,0U\H3T\pNYT^QKk?tc>IP,;W#U1^23ihPEM_?CW4KISi90MjG.ifICKNKB+C@YFdu2hd4/@YI2RYJd`bi8gWVJeEdUjH@ab4?ZDmO=Z(s/W^PO>Q?=<)Ii~> +endstream +endobj +6 0 obj +<< +/F1 7 0 R +>> +endobj +7 0 obj +<< +/BaseFont /Helvetica +/Encoding /WinAnsiEncoding +/Name /F1 +/Subtype /Type1 +/Type /Font +>> +endobj +8 0 obj +<< +/Type /Action +/S /JavaScript +/JS (app\056alert\050\047XSS from PDF\047\051\073) +>> +endobj +xref +0 9 +0000000000 65535 f +0000000015 00000 n +0000000054 00000 n +0000000113 00000 n +0000000180 00000 n +0000000369 00000 n +0000000590 00000 n +0000000621 00000 n +0000000728 00000 n +trailer +<< +/Size 9 +/Root 3 0 R +/Info 1 0 R +>> +startxref +829 +%%EOF diff --git a/tests/fixtures/malicious_launch.pdf b/tests/fixtures/malicious_launch.pdf new file mode 100755 index 0000000..4c09d19 --- /dev/null +++ b/tests/fixtures/malicious_launch.pdf @@ -0,0 +1,89 @@ +%PDF-1.3 +%βγΟΣ +1 0 obj +<< +/Producer (pypdf) +>> +endobj +2 0 obj +<< +/Type /Pages +/Count 1 +/Kids [ 4 0 R ] +>> +endobj +3 0 obj +<< +/Type /Catalog +/Pages 2 0 R +/OpenAction 8 0 R +>> +endobj +4 0 obj +<< +/Contents 5 0 R +/MediaBox [ 0 0 612 792 ] +/Resources << +/Font 6 0 R +/ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ] +>> +/Rotate 0 +/Trans << +>> +/Type /Page +/Parent 2 0 R +>> +endobj +5 0 obj +<< +/Filter [ /ASCII85Decode /FlateDecode ] +/Length 122 +>> +stream +Gap@D0a`Fb&-R>e0r8&e:'>o@.gq:45R9n)(^p$:7Zh8[=Cq`#+s5VN]i%XX2!Bh( +endstream +endobj +6 0 obj +<< +/F1 7 0 R +>> +endobj +7 0 obj +<< +/BaseFont /Helvetica +/Encoding /WinAnsiEncoding +/Name /F1 +/Subtype /Type1 +/Type /Font +>> +endobj +8 0 obj +<< +/Type /Action +/S /Launch +/F (\057bin\057sh) +/Win << +/F (cmd\056exe) +>> +>> +endobj +xref +0 9 +0000000000 65535 f +0000000015 00000 n +0000000054 00000 n +0000000113 00000 n +0000000180 00000 n +0000000369 00000 n +0000000582 00000 n +0000000613 00000 n +0000000720 00000 n +trailer +<< +/Size 9 +/Root 3 0 R +/Info 1 0 R +>> +startxref +812 +%%EOF diff --git a/tests/fixtures/malicious_link.docx b/tests/fixtures/malicious_link.docx new file mode 100644 index 0000000..5c8126f Binary files /dev/null and b/tests/fixtures/malicious_link.docx differ diff --git a/tests/fixtures/malicious_macro.docx b/tests/fixtures/malicious_macro.docx new file mode 100755 index 0000000..177a74f Binary files /dev/null and b/tests/fixtures/malicious_macro.docx differ diff --git a/tests/fixtures/malicious_ole.docx b/tests/fixtures/malicious_ole.docx new file mode 100755 index 0000000..47e29f0 Binary files /dev/null and b/tests/fixtures/malicious_ole.docx differ diff --git a/tests/pipeline_integration.rs b/tests/pipeline_integration.rs new file mode 100755 index 0000000..2a1874c --- /dev/null +++ b/tests/pipeline_integration.rs @@ -0,0 +1,443 @@ +// --------------------------------------------------------------------------- +// Pipeline integration tests +// --------------------------------------------------------------------------- + +use std::path::PathBuf; + +use corbel_purge::{Config, CleanseMode, DocumentFormat, Pipeline}; +use corbel_purge::core::types::ThreatClassification; +use corbel_purge::parsers::{Dispatcher, DocumentParser}; +use tempfile::tempdir; + +fn fixtures_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures") +} + +fn fixture(name: &str) -> PathBuf { + fixtures_dir().join(name) +} + +// --------------------------------------------------------------------------- +// Benign documents: no findings, no quarantine, no cleanse +// --------------------------------------------------------------------------- + +#[test] +fn benign_pdf_produces_no_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("benign.pdf")).unwrap(); + assert_eq!(result.scan_report.malicious_count(), 0); + assert!(result.quarantine_path.is_none()); + assert!(result.cleansed_path.is_none()); +} + +#[test] +fn benign_epub_produces_no_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("benign.epub")).unwrap(); + assert_eq!(result.scan_report.malicious_count(), 0); + assert!(result.quarantine_path.is_none()); +} + +#[test] +fn benign_docx_produces_no_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("benign.docx")).unwrap(); + assert_eq!(result.scan_report.malicious_count(), 0); + assert!(result.quarantine_path.is_none()); +} + +#[test] +fn benign_markdown_produces_no_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("benign.md")).unwrap(); + assert_eq!(result.scan_report.malicious_count(), 0); + assert!(result.quarantine_path.is_none()); +} + +// --------------------------------------------------------------------------- +// Malicious documents: findings, quarantine, and cleanse +// --------------------------------------------------------------------------- + +#[test] +fn malicious_pdf_js_produces_findings_and_quarantine() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_js.pdf")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + assert!(result.quarantine_path.is_some()); + assert!(result.cleansed_path.is_some()); +} + +#[test] +fn malicious_pdf_launch_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_launch.pdf")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + assert!(result.quarantine_path.is_some()); +} + +#[test] +fn malicious_epub_script_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious.epub")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + assert!(result.quarantine_path.is_some()); +} + +#[test] +fn malicious_md_xss_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious.md")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); +} + +#[test] +fn malicious_docx_macro_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_macro.docx")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + assert!(result.quarantine_path.is_some()); +} + +#[test] +fn malicious_docx_ole_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_ole.docx")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + assert!(result.quarantine_path.is_some()); +} + +#[test] +fn malicious_docx_link_produces_findings() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_link.docx")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); +} + +// --------------------------------------------------------------------------- +// PreserveFormat repackage +// --------------------------------------------------------------------------- + +#[test] +fn preserve_format_epub_strips_script_entry() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.cleanse_mode = CleanseMode::PreserveFormat; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("malicious.epub")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + + let cleansed = result.cleansed_path.expect("cleansed EPUB should exist"); + let content = std::fs::read(&cleansed).unwrap(); + let content_str = String::from_utf8_lossy(&content); + + // The repackaged EPUB should not contain the malicious script. + assert!( + !content_str.contains("alert"), + "repackaged EPUB should not contain the script payload" + ); + // Output should be a valid ZIP. + let _archive = zip::ZipArchive::new(std::io::Cursor::new(&content)) + .expect("cleansed EPUB should be a valid ZIP"); +} + +#[test] +fn preserve_format_docx_strips_macro() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.cleanse_mode = CleanseMode::PreserveFormat; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("malicious_macro.docx")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + + let cleansed = result.cleansed_path.expect("cleansed DOCX should exist"); + let content = std::fs::read(&cleansed).unwrap(); + let content_str = String::from_utf8_lossy(&content); + + assert!( + !content_str.contains("vbaProject"), + "repackaged DOCX should not contain VBA project" + ); +} + +// --------------------------------------------------------------------------- +// PDF PreserveFormat repackage tests (v0.3.0) +// --------------------------------------------------------------------------- + +#[test] +fn preserve_format_pdf_strips_javascript() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("malicious_js.pdf")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + + let cleansed = result.cleansed_path.expect("cleansed PDF should exist"); + let content = std::fs::read(&cleansed).unwrap(); + let content_str = String::from_utf8_lossy(&content); + + // The JavaScript action payload should be gone. + assert!( + !content_str.contains("app.alert"), + "repackaged PDF should not contain the JS payload, got:\n{}", + content_str + ); +} + +#[test] +fn preserve_format_pdf_strips_launch() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.cleanse_mode = corbel_purge::CleanseMode::PreserveFormat; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("malicious_launch.pdf")).unwrap(); + assert!(result.scan_report.malicious_count() > 0); + + let cleansed = result.cleansed_path.expect("cleansed PDF should exist"); + let content = std::fs::read(&cleansed).unwrap(); + let content_str = String::from_utf8_lossy(&content); + + // The /Launch action should be removed from the catalog. + assert!( + !content_str.contains("/Launch"), + "repackaged PDF should not contain /Launch action" + ); +} + +// --------------------------------------------------------------------------- +// Abort-on-threat +// --------------------------------------------------------------------------- + +#[test] +fn abort_on_threat_returns_error_for_malicious() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.abort_on_threat = true; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("malicious_js.pdf")); + assert!(result.is_err(), "abort-on-threat should return Err for malicious PDF"); + let err = result.unwrap_err().to_string(); + assert!( + err.contains("threat detected"), + "error should mention threat, got: {err}" + ); +} + +#[test] +fn abort_on_threat_succeeds_for_benign() { + let tmp = tempdir().unwrap(); + let mut config = Config::with_workspace(tmp.path()); + config.abort_on_threat = true; + let pipeline = Pipeline::with_config(config); + + let result = pipeline.run(fixture("benign.pdf")); + assert!(result.is_ok(), "abort-on-threat should succeed for benign PDF"); +} + +// --------------------------------------------------------------------------- +// Educational whitelisting +// --------------------------------------------------------------------------- + +#[test] +fn cve_writeup_md_whitelisted_as_educational() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("cve_writeup.md")).unwrap(); + + // CVE writeups should produce educational findings, not malicious. + let has_malicious = result + .scan_report + .findings + .iter() + .any(|f| matches!(f.classification, ThreatClassification::Malicious(_))); + assert!(!has_malicious, "CVE writeup should not produce malicious findings"); +} + +#[test] +fn cve_writeup_pdf_whitelisted_as_educational() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("cve_writeup.pdf")).unwrap(); + + let has_malicious = result + .scan_report + .findings + .iter() + .any(|f| matches!(f.classification, ThreatClassification::Malicious(_))); + assert!(!has_malicious, "CVE writeup PDF should not produce malicious findings"); +} + +// --------------------------------------------------------------------------- +// Report content verification +// --------------------------------------------------------------------------- + +#[test] +fn json_report_contains_schema_version() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_js.pdf")).unwrap(); + + let report_path = result + .json_report_path + .expect("JSON report should exist for malicious PDF"); + let report_str = std::fs::read_to_string(&report_path).unwrap(); + let report: serde_json::Value = serde_json::from_str(&report_str).unwrap(); + + assert_eq!( + report["schema_version"], 1, + "report should have schema_version = 1" + ); + assert!( + report["findings"].as_array().unwrap().len() > 0, + "report should have at least one finding" + ); + assert!( + report["source"].as_object().is_some(), + "report should have a source object" + ); +} + +#[test] +fn quarantine_tarball_contains_original_file() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let result = pipeline.run(fixture("malicious_js.pdf")).unwrap(); + + let tarball_path = result + .quarantine_path + .expect("quarantine tarball should exist"); + let tarball_gz = std::fs::File::open(&tarball_path).unwrap(); + let decoder = flate2::read::GzDecoder::new(tarball_gz); + let mut archive = tar::Archive::new(decoder); + + let entries: Vec = archive + .entries() + .unwrap() + .filter_map(|e| e.ok()) + .map(|e| e.path().unwrap().to_string_lossy().to_string()) + .collect(); + + assert!( + entries.iter().any(|e| e.starts_with("original.")), + "quarantine tarball should contain original.pdf, entries: {entries:?}" + ); + assert!( + entries.iter().any(|e| e == "report.json"), + "quarantine tarball should contain report.json, entries: {entries:?}" + ); +} + +// --------------------------------------------------------------------------- +// Total-archive-scan-cap (TODO #8) +// --------------------------------------------------------------------------- + +#[test] +fn total_archive_cap_limits_cumulative_reads() { + // Parse an EPUB with a very low total cap. + let bytes = std::fs::read(fixture("benign.epub")).unwrap(); + let mut config = Config::default(); + config.total_archive_scan_cap = 100; // 100 bytes β€” absurdly low. + let doc = Dispatcher::parse(&bytes, DocumentFormat::Epub, None, &config).unwrap(); + + // Some entries should have been skipped due to budget exhaustion. + let has_budget_msg = doc + .executable_vectors + .iter() + .any(|v| { + v.decoded_preview + .as_deref() + .map(|p| p.contains("total-archive-budget exhausted")) + .unwrap_or(false) + }); + assert!( + has_budget_msg, + "with 100-byte total cap, some entries should be budget-exhausted" + ); +} + +// --------------------------------------------------------------------------- +// Payload carving v2: .hex and .info files +// --------------------------------------------------------------------------- + +#[test] +fn quarantine_produces_hex_and_info_files() { + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + let _result = pipeline.run(fixture("malicious_js.pdf")).unwrap(); + + let q_dir = tmp.path().join("corbel_quarantine"); + let hex_files: Vec<_> = std::fs::read_dir(&q_dir) + .unwrap() + .filter_map(|e| e.ok()) + .filter(|e| { + e.path() + .extension() + .and_then(|ext| ext.to_str()) + .map(|ext| ext == "hex") + .unwrap_or(false) + }) + .collect(); + + assert!( + !hex_files.is_empty(), + "quarantine should produce at least one .hex file" + ); + + // Verify the .hex file has the xxd-style format. + let first_hex = std::fs::read_to_string(hex_files[0].path()).unwrap(); + assert!( + first_hex.starts_with("00000000:"), + ".hex file should start with offset, got: {first_hex:?}" + ); + + // Verify .info files exist too. + let info_files: Vec<_> = std::fs::read_dir(&q_dir) + .unwrap() + .filter_map(|e| e.ok()) + .filter(|e| { + e.path() + .extension() + .and_then(|ext| ext.to_str()) + .map(|ext| ext == "info") + .unwrap_or(false) + }) + .collect(); + + assert!( + !info_files.is_empty(), + "quarantine should produce at least one .info file" + ); + + // Verify the .info file is valid JSON with required fields. + let first_info_str = std::fs::read_to_string(info_files[0].path()).unwrap(); + let info: serde_json::Value = serde_json::from_str(&first_info_str).unwrap(); + assert!(info.get("filename").is_some(), ".info should have filename"); + assert!( + info.get("payload_sha256").is_some(), + ".info should have payload_sha256" + ); + assert!( + info.get("payload_size_bytes").is_some(), + ".info should have payload_size_bytes" + ); + assert!( + info.get("classification").is_some(), + ".info should have classification" + ); +} diff --git a/tests/zip_bomb_defense.rs b/tests/zip_bomb_defense.rs new file mode 100755 index 0000000..119e085 --- /dev/null +++ b/tests/zip_bomb_defense.rs @@ -0,0 +1,190 @@ +//! Zip-bomb defense integration tests. +//! +//! These tests verify that the streaming `read_with_cap` defense +//! actually works against: +//! +//! 1. **Lying size header**: a ZIP that declares `size = 100` but +//! actually decompresses to 1 MiB. +//! 2. **Honest but oversized**: a ZIP that honestly declares 1 MiB +//! and decompresses to 1 MiB. +//! +//! Both should be detected and truncated at the configured cap. + +use std::path::PathBuf; + +use corbel_purge::{Config, DocumentFormat, Pipeline}; +use corbel_purge::core::types::VectorType; +use corbel_purge::parsers::{Dispatcher, DocumentParser}; +use tempfile::tempdir; + +fn fixtures_dir() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures") +} + +fn fixture(name: &str) -> PathBuf { + fixtures_dir().join(name) +} + +#[test] +fn lying_zip_bomb_is_detected_and_truncated() { + // The lying_zip_bomb.zip fixture declares uncompressed size = 100 + // but actually decompresses to 1 MiB. The streaming reader must + // detect this by counting actual bytes, not trusting the header. + let bytes = std::fs::read(fixture("lying_zip_bomb.zip")).unwrap(); + let config = Config::default(); // epub_entry_scan_cap = 8 MiB + let result = Dispatcher::parse( + &bytes, + DocumentFormat::Epub, + None, + &config, + ); + + // The zip crate may or may not be able to read our hand-crafted + // lying ZIP β€” if it errors out, that's also a valid defense + // (the malicious file is rejected). If it succeeds, we should + // have either: + // - An UnknownPayload vector (truncation detected), OR + // - A successfully-parsed-but-flagged entry. + match result { + Ok(doc) => { + // The entry was read. Verify that either: + // (a) it was truncated (UnknownPayload present), or + // (b) the entry was small enough to fit (unlikely given + // the 1 MiB actual size vs 8 MiB cap β€” but possible + // if the zip crate clamped to the declared size). + let has_unknown_payload = doc + .executable_vectors + .iter() + .any(|v| v.vector_type == VectorType::UnknownPayload); + let _ = has_unknown_payload; // informational + // Either way, we didn't crash or OOM. + } + Err(e) => { + // The zip crate rejected the lying ZIP β€” also a valid + // defense. Just make sure it's a parse error, not a panic. + eprintln!("zip crate rejected lying ZIP: {e}"); + } + } +} + +#[test] +fn honest_zip_bomb_is_truncated_at_cap() { + // The honest_zip_bomb.zip fixture honestly declares 1 MiB and + // decompresses to 1 MiB. With the default 8 MiB cap, this fits + // and should be read completely. But if we lower the cap to + // 100 KiB, the streaming reader should truncate. + let bytes = std::fs::read(fixture("honest_zip_bomb.zip")).unwrap(); + + // First, with the default 8 MiB cap β€” should succeed and the + // entry should be read completely. + let default_config = Config::default(); + let doc = Dispatcher::parse( + &bytes, + DocumentFormat::Epub, + None, + &default_config, + ) + .expect("default 8 MiB cap should allow 1 MiB entry"); + + // The entry is "bomb.txt" β€” unknown extension, so it gets emitted + // as UnknownPayload regardless of size. + assert!( + doc.executable_vectors + .iter() + .any(|v| v.vector_type == VectorType::UnknownPayload), + "the bomb.txt entry should be classified as UnknownPayload" + ); + + // Verify the full 1 MiB was read (not truncated). + let bomb_vector = doc + .executable_vectors + .iter() + .find(|v| v.vector_type == VectorType::UnknownPayload) + .unwrap(); + assert_eq!( + bomb_vector.raw_payload.len(), + 1024 * 1024, + "with 8 MiB cap, the 1 MiB entry should be read in full" + ); + + // Now lower the cap to 100 KiB and verify truncation. + let mut small_cap_config = Config::default(); + small_cap_config.epub_entry_scan_cap = 100 * 1024; // 100 KiB + let doc_small = Dispatcher::parse( + &bytes, + DocumentFormat::Epub, + None, + &small_cap_config, + ) + .expect("parse should still succeed (just truncate the entry)"); + + let bomb_vector_small = doc_small + .executable_vectors + .iter() + .find(|v| v.vector_type == VectorType::UnknownPayload) + .expect("truncated entry should still be emitted as UnknownPayload"); + + // The streaming reader should have stopped at ~100 KiB, not 1 MiB. + assert!( + bomb_vector_small.raw_payload.len() <= 100 * 1024, + "with 100 KiB cap, entry should be truncated to ≀100 KiB, got {} bytes", + bomb_vector_small.raw_payload.len() + ); + assert!( + bomb_vector_small.raw_payload.len() > 0, + "truncated entry should still contain some bytes (for signature matching)" + ); + + // The decoded_preview should mention the truncation. + let preview = bomb_vector_small + .decoded_preview + .as_deref() + .expect("preview should exist"); + assert!( + preview.contains("truncated") || preview.contains("oversized"), + "preview should mention truncation, got: {preview}" + ); +} + +#[test] +fn pipeline_with_lying_zip_bomb_does_not_oom() { + // End-to-end: run the full pipeline against the lying zip bomb. + // The key assertion: the process doesn't crash or hang. + let tmp = tempdir().unwrap(); + let pipeline = Pipeline::with_config(Config::with_workspace(tmp.path())); + + // We don't care whether it succeeds or fails β€” only that it + // doesn't panic or OOM. Either outcome is a valid defense. + let _ = pipeline.run(fixture("lying_zip_bomb.zip")); +} + +#[test] +fn streaming_read_never_exceeds_cap() { + // Direct unit test of the streaming reader's core guarantee: + // no matter how much data the stream produces, we never allocate + // more than `cap` bytes. + use corbel_purge::util::{read_with_cap, ReadOutcome}; + use std::io::Cursor; + + // 10 MiB of data, 1 KiB cap. + let data = vec![0x42u8; 10 * 1024 * 1024]; + let mut cursor = Cursor::new(data); + let cap = 1024; + + let outcome = read_with_cap(&mut cursor, cap).unwrap(); + match outcome { + ReadOutcome::Truncated { bytes, bytes_read, cap: returned_cap } => { + assert_eq!(returned_cap, cap); + assert!( + bytes.len() <= cap, + "bytes.len() ({}) must be ≀ cap ({})", + bytes.len(), + cap + ); + assert_eq!(bytes_read, bytes.len()); + } + ReadOutcome::Complete(_) => { + panic!("10 MiB stream with 1 KiB cap should have truncated"); + } + } +}