# AutoIngest — design notes **Jeremy Anderson — [dcos.net](https://dcos.net) — info@dcos.net** A camera photo importer for ecommerce workflows. Version 3.0. --- ## The problem A friend sells vintage reel-to-reel film on eBay. His workflow: photograph items with a camera, move the photos to a PC, attach them to listings in the browser. Every photo must be a JPG. His cameras shoot a mix of JPG, RAW, and HEIC. Photos land on an SD card and must arrive as JPGs in a date-stamped folder. He is not a technical person. He inserts the card and expects it to work. The original tool was Python — PySide6, Pillow, rawpy, pillow-heif, cairosvg, pywin32. Six packages with native C extensions that must compile or match prebuilt wheels for the target Windows version. On a non-technical person's PC, that is a liability. ## The decision: C# / .NET 8 The choice came down to one question: what can we hand the seller as a single file that works? | Requirement | Python | C# / .NET 8 | |-------------|--------|-------------| | Deploy as single exe | No (interpreter + pip packages) | Yes (`PublishSingleFile`) | | Native image formats | 4+ fragile native libraries | One NuGet package (Magick.NET) | | Drive detection | pywin32 (another native dep) | P/Invoke to kernel32 (zero deps) | | GUI | PySide6 (large native dep) | WinForms (ships with .NET) | | Runtime on target PC | Python 3.x + pip + packages | .NET 8 Desktop Runtime (Win11 default) | | Self-contained option | PyInstaller (fragile, 200+ MB) | `dotnet publish --self-contained` (~150–200 MB, reliable) | C# wins on every axis that matters for deployment to a non-technical end user. ## Architecture Three layers, one NuGet dependency, no reverse references: ``` App/ UI layer (namespace AutoIngest.App) Program.cs Single-instance mutex, entry point MainForm.cs Tray, toast, coordinator wiring, Options menu, first-run hook MainForm.Designer.cs Toast popup layout (WinForms Designer) SettingsForm.cs Settings: device registry + retention + access tracking WelcomeForm.cs First-run welcome / branding wizard BrandingForm.cs Branding editor opened from the tray BrandingEditorPanel.cs Shared branding fields + live preview TrayIconFactory.cs Tray icon as a micro brand mark AutostartManager.cs Self-install + Startup shortcut (IShellLinkW COM) Theme.cs Shared dark-theme palette Core/ Config/models layer (namespace AutoIngest.Core) ConfigManager.cs JSON config persistence DeviceRegistry.cs In-memory registry of trusted devices Engine/ Import/conversion layer (namespace AutoIngest.Engine) IDeviceSource.cs Source-of-devices interface DeviceIdentity.cs Connected-device DTO IPhotoProvider.cs Enumerate + remove source photos SDCardMonitor.cs Mass-storage source: kernel32, volume-serial ids MtpDeviceSource.cs Phone source: Shell.Application COM on an STA thread DeviceImporter.cs Shared photo-discovery + move/convert pipeline ImportCoordinator.cs Polling loop, registry gate, retention sweep, event fan-out FolderRetentionPolicy.cs Enumerate dated folders + sweep aged ones to Recycle Bin AccessTimeTracker.cs Detects/enables NTFS last-access-time tracking BrandingRenderer.cs Burns text/logo watermarks into a photo ImageConverter.cs Magick.NET wrapper: orient → brand → strip → copyright → JPG ``` Each class owns one responsibility. The original `SDCardMonitor` did everything (detect, find, import, convert); v3 splits detection (`SDCardMonitor`, `MtpDeviceSource`), importing (`DeviceImporter`), coordination (`ImportCoordinator`), trust (`DeviceRegistry`), and persistence (`ConfigManager`) into separate classes. No god objects. ### Drive detection No WMI. No COM initialization for mass storage. Direct P/Invoke to `kernel32.GetDriveType()` and `kernel32.GetVolumeInformationW()`. The 4-second polling loop checks every drive letter, accepts removable drives and fixed drives that look like cameras (DCIM folder or camera-related volume label), and surfaces each as a `DeviceIdentity` keyed by volume serial. ### Phone (MTP) detection Phones never appear as drive letters — they expose MTP/PTP. `MtpDeviceSource` walks the shell "This PC" namespace via `Shell.Application` COM for portable devices, runs on a dedicated STA thread (shell COM is STA-happiest), and reads each device's PTP serial for the registry fingerprint. Every COM call is wrapped: a locked phone, dead battery, or flaky driver degrades to a log line, never a crash. Phones are strictly best-effort; the SD/USB path is independent. ### Import pipeline 1. Discover photos on the device (mass storage: DCIM → Pictures → root → 3-level scan; phone: shell namespace walk). 2. Create `~/Pictures/YYYY-MM-DD/`. 3. JPG/JPEG: copy through, verify size, reclaim source. 4. Everything else: convert via Magick.NET (sRGB, alpha off, progressive optimized JPG, native resolution), re-decode the output to confirm validity, reclaim source. 5. Same-day collisions: size then SHA-256. Byte-identical skips; different content gets a `_1`, `_2` suffix (cap 9999, GUID fallback). 6. Log every file. The pipeline never touches the device directly. Both mass storage (where photos are files and removal is `File.Delete`) and MTP (where photos live behind a COM namespace and removal is a shell verb) feed the same `DeviceImporter` through `IPhotoProvider`. ### Verify-before-delete Every source reclamation follows the same contract: destination exists, destination is the expected size, and (for converts) the destination re-decodes as a valid image. Only then does `RemoveSource` run. The contract is identical for SD/USB and phones — critical because the phone-delete toggle makes phones destructive. ## Code quality conventions - **Targeted exception handling.** Catch blocks name specific types (`UnauthorizedAccessException`, `IOException`, `MagickException`, `JsonException`). Top-of-loop `catch (Exception)` guards exist where any failure must mean "skip this iteration" — the polling loop, per-file import, COM wrapping. - **Named constants over magic numbers.** Intervals (`CHECK_INTERVAL_MS = 4000`), suffix caps (`MAX_COLLISION_SUFFIX = 9999`), colors, skip-lists, retention presets — all named fields. - **Bounded loops.** The polling thread is a `_running`-flag-controlled `while (_running)` with a fixed sleep, not `while (true)`. Collision suffixes cap at 9999 before a GUID fallback. - **Step-down logic.** Guard clauses and early returns throughout (`ResolveDestPath`, `FindPhotos`, `ShowToast`, `BrandingRenderer.Apply`). The common case executes first; edge cases bail early. - **One responsibility per class.** Detection, importing, coordination, trust, persistence each live in their own type. - **Disposable cleanup.** `MainForm.Dispose` releases the tray bitmap, hide-timer, coordinator, HICON, and components. `MagickImage` is `using`-scoped. The MTP provider disposes its staging temp directory on import completion. - **Comments describe why, not what.** Every non-obvious decision carries a rationale comment. ## Device registry The original tool imported from any removable drive. That works for one seller with one SD card. The moment someone else plugs in a USB stick — or the seller plugs in a phone that is not theirs — every image on it moves off and converts. Unacceptable. The registry gates all imports. Each device carries a stable fingerprint, not a drive letter: - **SD/USB** uses the volume serial number from `GetVolumeInformationW`. The same call the detection code already made; v3 captures the serial instead of discarding it. - **Phones** use the PTP/MTP serial read from the shell namespace. - Devices with no serial fall back to a weak label-plus-size id, flagged in the UI. First time a device is seen, the user picks **Register & Import**, **Import Once**, or **Ignore**. Registered devices persist in config and import silently from then on. The registry is editable from **Settings… → Registered devices**. ## Auto-orient and metadata stripping Two conversion-time improvements that are pure wins for the workflow: - **Auto-orient.** Phones and some cameras set an EXIF orientation tag without rotating the pixels. The photo looks right in viewers that honor the tag and sideways in many browser upload previews. `image.AutoOrient()` bakes the correction into the pixels before write. - **Strip metadata.** Phone photos embed GPS. A listing photo that pinpoints the seller's house is a privacy hole. `image.Strip()` removes EXIF/XMP/etc. on export. Both default on. Both are tray toggles under **Options ▸**. Both apply to converted files only; JPGs that move straight through are byte-for-byte untouched. ## Branding Branding puts the seller's identity on every converted photo: a visible text watermark (store name / handle / URL), a logo watermark, and EXIF copyright fields. Off by default. Configured via the first-launch welcome wizard or the tray editor. The interesting part is the pipeline order. The naive approach — strip everything, or embed EXIF and hope strip stays off — either erases the seller's copyright or leaks the camera's metadata. Neither is acceptable. The resolved order in `ConvertToJpg`: 1. `AutoOrient` — bake orientation into pixels. 2. Color space / alpha. 3. `BrandingRenderer.Apply` — burn watermarks into pixels (survives everything downstream). 4. `Strip()` — wipe all metadata: camera model, GPS, software tags, everything. 5. Write EXIF copyright — Artist / Copyright / Description are the only metadata written back, so they are the only metadata that survives. 6. Encode as JPG. Step 4 strips the world. Step 5 writes the seller's stamp onto a clean slate. Camera details are gone; seller attribution remains. Watermarks in step 3 are pixels, so strip is irrelevant to them. ### Brand tray icon The tray icon is the seller's face on the product. Once branding is configured, the icon renders the logo (scaled to fit) or the first letter of the store name on a rounded tile. The mark renders at 64×64 (super-sampled) and the code computes the **median luminance** of its non-transparent pixels. Median, not average: a black-and-white logo averages to mid-gray, which turns "light or dark background?" into a coin flip. Median says "this mark has both; pick the background that makes at least one of them pop." Luminance ≥ 0.5 → dark background (`#1f1f1f`); < 0.5 → light background (`#f0f0f0`). Backgrounds are deliberately strong — not mid-tones — so contrast holds on either taskbar theme. The 64×64 composite downsamples to 16×16 with high-quality bicubic interpolation, which is why a glyph drawn at 64px and scaled down reads crisp. ## Phone delete Phones default to copy-and-leave. The phone original stays put; the user curates phone storage. **Options ▸ Delete from phone after import** opts into phone-side deletion after the local copy verifies — mirroring the move behavior SD/USB have always had. The delete invokes the shell delete verb on the phone-side object, then verifies the object is gone by re-resolving it. The outcome surfaces in the import log: "removed" only when the delete succeeded, "src kept" when the driver did not honor the verb. Phones only; SD/USB always move/convert as before. ## Retention Once photos accumulate, disk space becomes the next problem. Retention is the answer, but it must be safe — auto-deleting a seller's photos because of a misconfigured setting is worse than running out of space. Retention is off by default and strictly opt-in. The seller picks an age (1 week through 7 years), and on a daily sweep plus once at startup any dated import folder older than that age goes to the Recycle Bin — recoverable, not gone. Age is measured from the folder's most recent activity, not its creation date. A folder the seller keeps reopening in Explorer stays alive; a folder nothing has touched for six months recycles. Using the max activity timestamp across the folder's files means "any file touched → whole folder is in use," which matches how sellers think about a day's shoot. Two safety nets protect active imports: - **Today-folder-name exemption.** The folder matching today's date never recycles. - **Freshness guard.** Any folder whose most recent activity is within 10 minutes never recycles, independent of its name. This catches midnight rollover (an import that started at 23:58 writes to yesterday's folder, which would not match today's name but is clearly still active) and clock skew. ### The LastAccessTime reality Windows disables `LastAccessTime` updates by default since Vista for performance. A naive "delete folders whose last-access time is old" implementation deletes everything, because every file's last-access time looks old. AutoIngest handles this in three steps: 1. **Detect.** `AccessTimeTracker.IsEnabled()` runs `fsutil behavior query DisableLastAccess` and parses the result, including the Win10 1803+ system-managed mode. 2. **Fall back honestly.** When access tracking is off, retention uses `LastWriteTime` (import date). The Settings dialog states this plainly. 3. **Offer to enable.** A button launches an elevated `fsutil behavior set DisableLastAccess 0` (UAC prompt), with a note that a restart is required. ### No separate skip-list The filesystem timestamps are the access record. Any access resets a folder's clock and the next sweep re-evaluates from scratch. A parallel data structure would need reconciliation with real access events, would drift, and would lose history on reinstall. The sweep is stateless: each pass recomputes every folder's age. ## Self-install and autostart The seller wants AutoIngest to launch at login and live in a stable place rather than Downloads. The classic Win9x method — copy the exe to a system directory and drop a shortcut in the Startup folder — half works on Windows 11. **The Startup-folder shortcut: fully supported.** A `.lnk` in `%AppData%\Microsoft\Windows\Start Menu\Programs\Startup\` launches the target at every login. No admin, no UAC, no registry. Explorer honors it exactly as it did twenty-five years ago. **The system-root copy: wrong on Win11.** `C:\Windows\` is admin-owned and write-protected. It semantically signals "part of the OS." Defender scrutinizes system-directory binaries harder. A Windows reset can wipe it. Clinging to it means a UAC prompt at install time, breaking the zero-friction posture. The install target is `%LocalAppData%\Programs\AutoIngest\` — the modern per-user programs location VS Code and Discord use. Same properties that make the Startup folder viable (user-writable, no admin, per-user, survives reboots), none of the system-root costs. The `.lnk` itself is raw `IShellLinkW` + `IPersistFile` COM interop — the old-school way, no `IWshRuntimeLibrary` reference. About fifty lines of interface definitions and a `CoCreateInstance` call. The toggle's persisted state is the shortcut's existence: `IsEnabled` is `File.Exists(StartupLink)`. No config flag to drift out of sync with reality. The checkbox reads the filesystem every time the menu opens. ## Final word The Python version was 2900 lines. The C# version is a larger feature surface in a comparable line count, ships as one file, and carries zero runtime dependencies on the target PC in the self-contained build. The seller gets an exe on the desktop and it works. That is the goal. --- **Author:** Jeremy Anderson — [dcos.net](https://dcos.net) — info@dcos.net **License:** MIT (MIT License). See [LICENSE](LICENSE).