16 KiB

Raw Blame History

AutoIngest — design notes

Jeremy Anderson — dcos.net — info@dcos.net

A camera photo importer for ecommerce workflows. Version 3.0.

The problem

A friend sells vintage reel-to-reel film on eBay. His workflow: photograph items with a camera, move the photos to a PC, attach them to listings in the browser. Every photo must be a JPG.

His cameras shoot a mix of JPG, RAW, and HEIC. Photos land on an SD card and must arrive as JPGs in a date-stamped folder. He is not a technical person. He inserts the card and expects it to work.

The original tool was Python — PySide6, Pillow, rawpy, pillow-heif, cairosvg, pywin32. Six packages with native C extensions that must compile or match prebuilt wheels for the target Windows version. On a non-technical person's PC, that is a liability.

The decision: C# / .NET 8

The choice came down to one question: what can we hand the seller as a single file that works?

Requirement	Python	C# / .NET 8
Deploy as single exe	No (interpreter + pip packages)	Yes (`PublishSingleFile`)
Native image formats	4+ fragile native libraries	One NuGet package (Magick.NET)
Drive detection	pywin32 (another native dep)	P/Invoke to kernel32 (zero deps)
GUI	PySide6 (large native dep)	WinForms (ships with .NET)
Runtime on target PC	Python 3.x + pip + packages	.NET 8 Desktop Runtime (Win11 default)
Self-contained option	PyInstaller (fragile, 200+ MB)	`dotnet publish --self-contained` (~150–200 MB, reliable)

C# wins on every axis that matters for deployment to a non-technical end user.

Architecture

Three layers, one NuGet dependency, no reverse references:

App/                         UI layer (namespace AutoIngest.App)
  Program.cs                 Single-instance mutex, entry point
  MainForm.cs                Tray, toast, coordinator wiring, Options menu, first-run hook
  MainForm.Designer.cs       Toast popup layout (WinForms Designer)
  SettingsForm.cs            Settings: device registry + retention + access tracking
  WelcomeForm.cs             First-run welcome / branding wizard
  BrandingForm.cs            Branding editor opened from the tray
  BrandingEditorPanel.cs     Shared branding fields + live preview
  TrayIconFactory.cs         Tray icon as a micro brand mark
  AutostartManager.cs        Self-install + Startup shortcut (IShellLinkW COM)
  Theme.cs                   Shared dark-theme palette
Core/                        Config/models layer (namespace AutoIngest.Core)
  ConfigManager.cs           JSON config persistence
  DeviceRegistry.cs          In-memory registry of trusted devices
Engine/                      Import/conversion layer (namespace AutoIngest.Engine)
  IDeviceSource.cs           Source-of-devices interface
  DeviceIdentity.cs          Connected-device DTO
  IPhotoProvider.cs          Enumerate + remove source photos
  SDCardMonitor.cs           Mass-storage source: kernel32, volume-serial ids
  MtpDeviceSource.cs         Phone source: Shell.Application COM on an STA thread
  DeviceImporter.cs          Shared photo-discovery + move/convert pipeline
  ImportCoordinator.cs       Polling loop, registry gate, retention sweep, event fan-out
  FolderRetentionPolicy.cs   Enumerate dated folders + sweep aged ones to Recycle Bin
  AccessTimeTracker.cs       Detects/enables NTFS last-access-time tracking
  BrandingRenderer.cs        Burns text/logo watermarks into a photo
  ImageConverter.cs          Magick.NET wrapper: orient → brand → strip → copyright → JPG

Each class owns one responsibility. The original SDCardMonitor did everything (detect, find, import, convert); v3 splits detection (SDCardMonitor, MtpDeviceSource), importing (DeviceImporter), coordination (ImportCoordinator), trust (DeviceRegistry), and persistence (ConfigManager) into separate classes. No god objects.

Drive detection

No WMI. No COM initialization for mass storage. Direct P/Invoke to kernel32.GetDriveType() and kernel32.GetVolumeInformationW(). The 4-second polling loop checks every drive letter, accepts removable drives and fixed drives that look like cameras (DCIM folder or camera-related volume label), and surfaces each as a DeviceIdentity keyed by volume serial.

Phone (MTP) detection

Phones never appear as drive letters — they expose MTP/PTP. MtpDeviceSource walks the shell "This PC" namespace via Shell.Application COM for portable devices, runs on a dedicated STA thread (shell COM is STA-happiest), and reads each device's PTP serial for the registry fingerprint. Every COM call is wrapped: a locked phone, dead battery, or flaky driver degrades to a log line, never a crash. Phones are strictly best-effort; the SD/USB path is independent.

Import pipeline

Discover photos on the device (mass storage: DCIM → Pictures → root → 3-level scan; phone: shell namespace walk).
Create ~/Pictures/YYYY-MM-DD/.
JPG/JPEG: copy through, verify size, reclaim source.
Everything else: convert via Magick.NET (sRGB, alpha off, progressive optimized JPG, native resolution), re-decode the output to confirm validity, reclaim source.
Same-day collisions: size then SHA-256. Byte-identical skips; different content gets a _1, _2 suffix (cap 9999, GUID fallback).
Log every file.

The pipeline never touches the device directly. Both mass storage (where photos are files and removal is File.Delete) and MTP (where photos live behind a COM namespace and removal is a shell verb) feed the same DeviceImporter through IPhotoProvider.

Verify-before-delete

Every source reclamation follows the same contract: destination exists, destination is the expected size, and (for converts) the destination re-decodes as a valid image. Only then does RemoveSource run. The contract is identical for SD/USB and phones — critical because the phone-delete toggle makes phones destructive.

Code quality conventions

Targeted exception handling. Catch blocks name specific types (UnauthorizedAccessException, IOException, MagickException, JsonException). Top-of-loop catch (Exception) guards exist where any failure must mean "skip this iteration" — the polling loop, per-file import, COM wrapping.
Named constants over magic numbers. Intervals (CHECK_INTERVAL_MS = 4000), suffix caps (MAX_COLLISION_SUFFIX = 9999), colors, skip-lists, retention presets — all named fields.
Bounded loops. The polling thread is a _running-flag-controlled while (_running) with a fixed sleep, not while (true). Collision suffixes cap at 9999 before a GUID fallback.
Step-down logic. Guard clauses and early returns throughout (ResolveDestPath, FindPhotos, ShowToast, BrandingRenderer.Apply). The common case executes first; edge cases bail early.
One responsibility per class. Detection, importing, coordination, trust, persistence each live in their own type.
Disposable cleanup. MainForm.Dispose releases the tray bitmap, hide-timer, coordinator, HICON, and components. MagickImage is using-scoped. The MTP provider disposes its staging temp directory on import completion.
Comments describe why, not what. Every non-obvious decision carries a rationale comment.

Device registry

The original tool imported from any removable drive. That works for one seller with one SD card. The moment someone else plugs in a USB stick — or the seller plugs in a phone that is not theirs — every image on it moves off and converts. Unacceptable.

The registry gates all imports. Each device carries a stable fingerprint, not a drive letter:

SD/USB uses the volume serial number from GetVolumeInformationW. The same call the detection code already made; v3 captures the serial instead of discarding it.
Phones use the PTP/MTP serial read from the shell namespace.
Devices with no serial fall back to a weak label-plus-size id, flagged in the UI.

First time a device is seen, the user picks Register & Import, Import Once, or Ignore. Registered devices persist in config and import silently from then on. The registry is editable from Settings… → Registered devices.

Auto-orient and metadata stripping

Two conversion-time improvements that are pure wins for the workflow:

Auto-orient. Phones and some cameras set an EXIF orientation tag without rotating the pixels. The photo looks right in viewers that honor the tag and sideways in many browser upload previews. image.AutoOrient() bakes the correction into the pixels before write.
Strip metadata. Phone photos embed GPS. A listing photo that pinpoints the seller's house is a privacy hole. image.Strip() removes EXIF/XMP/etc. on export.

Both default on. Both are tray toggles under Options ▸. Both apply to converted files only; JPGs that move straight through are byte-for-byte untouched.

Branding

Branding puts the seller's identity on every converted photo: a visible text watermark (store name / handle / URL), a logo watermark, and EXIF copyright fields. Off by default. Configured via the first-launch welcome wizard or the tray editor.

The interesting part is the pipeline order. The naive approach — strip everything, or embed EXIF and hope strip stays off — either erases the seller's copyright or leaks the camera's metadata. Neither is acceptable.

The resolved order in ConvertToJpg:

AutoOrient — bake orientation into pixels.
Color space / alpha.
BrandingRenderer.Apply — burn watermarks into pixels (survives everything downstream).
Strip() — wipe all metadata: camera model, GPS, software tags, everything.
Write EXIF copyright — Artist / Copyright / Description are the only metadata written back, so they are the only metadata that survives.
Encode as JPG.

Step 4 strips the world. Step 5 writes the seller's stamp onto a clean slate. Camera details are gone; seller attribution remains. Watermarks in step 3 are pixels, so strip is irrelevant to them.

Brand tray icon

The tray icon is the seller's face on the product. Once branding is configured, the icon renders the logo (scaled to fit) or the first letter of the store name on a rounded tile.

The mark renders at 64×64 (super-sampled) and the code computes the median luminance of its non-transparent pixels. Median, not average: a black-and-white logo averages to mid-gray, which turns "light or dark background?" into a coin flip. Median says "this mark has both; pick the background that makes at least one of them pop." Luminance ≥ 0.5 → dark background (#1f1f1f); < 0.5 → light background (#f0f0f0). Backgrounds are deliberately strong — not mid-tones — so contrast holds on either taskbar theme. The 64×64 composite downsamples to 16×16 with high-quality bicubic interpolation, which is why a glyph drawn at 64px and scaled down reads crisp.

Phone delete

Phones default to copy-and-leave. The phone original stays put; the user curates phone storage. Options ▸ Delete from phone after import opts into phone-side deletion after the local copy verifies — mirroring the move behavior SD/USB have always had.

The delete invokes the shell delete verb on the phone-side object, then verifies the object is gone by re-resolving it. The outcome surfaces in the import log: "removed" only when the delete succeeded, "src kept" when the driver did not honor the verb. Phones only; SD/USB always move/convert as before.

Retention

Once photos accumulate, disk space becomes the next problem. Retention is the answer, but it must be safe — auto-deleting a seller's photos because of a misconfigured setting is worse than running out of space.

Retention is off by default and strictly opt-in. The seller picks an age (1 week through 7 years), and on a daily sweep plus once at startup any dated import folder older than that age goes to the Recycle Bin — recoverable, not gone.

Age is measured from the folder's most recent activity, not its creation date. A folder the seller keeps reopening in Explorer stays alive; a folder nothing has touched for six months recycles. Using the max activity timestamp across the folder's files means "any file touched → whole folder is in use," which matches how sellers think about a day's shoot.

Two safety nets protect active imports:

Today-folder-name exemption. The folder matching today's date never recycles.
Freshness guard. Any folder whose most recent activity is within 10 minutes never recycles, independent of its name. This catches midnight rollover (an import that started at 23:58 writes to yesterday's folder, which would not match today's name but is clearly still active) and clock skew.

The LastAccessTime reality

Windows disables LastAccessTime updates by default since Vista for performance. A naive "delete folders whose last-access time is old" implementation deletes everything, because every file's last-access time looks old. AutoIngest handles this in three steps:

Detect. AccessTimeTracker.IsEnabled() runs fsutil behavior query DisableLastAccess and parses the result, including the Win10 1803+ system-managed mode.
Fall back honestly. When access tracking is off, retention uses LastWriteTime (import date). The Settings dialog states this plainly.
Offer to enable. A button launches an elevated fsutil behavior set DisableLastAccess 0 (UAC prompt), with a note that a restart is required.

No separate skip-list

The filesystem timestamps are the access record. Any access resets a folder's clock and the next sweep re-evaluates from scratch. A parallel data structure would need reconciliation with real access events, would drift, and would lose history on reinstall. The sweep is stateless: each pass recomputes every folder's age.

Self-install and autostart

The seller wants AutoIngest to launch at login and live in a stable place rather than Downloads. The classic Win9x method — copy the exe to a system directory and drop a shortcut in the Startup folder — half works on Windows 11.

The Startup-folder shortcut: fully supported. A .lnk in %AppData%\Microsoft\Windows\Start Menu\Programs\Startup\ launches the target at every login. No admin, no UAC, no registry. Explorer honors it exactly as it did twenty-five years ago.

The system-root copy: wrong on Win11. C:\Windows\ is admin-owned and write-protected. It semantically signals "part of the OS." Defender scrutinizes system-directory binaries harder. A Windows reset can wipe it. Clinging to it means a UAC prompt at install time, breaking the zero-friction posture.

The install target is %LocalAppData%\Programs\AutoIngest\ — the modern per-user programs location VS Code and Discord use. Same properties that make the Startup folder viable (user-writable, no admin, per-user, survives reboots), none of the system-root costs.

The .lnk itself is raw IShellLinkW + IPersistFile COM interop — the old-school way, no IWshRuntimeLibrary reference. About fifty lines of interface definitions and a CoCreateInstance call.

The toggle's persisted state is the shortcut's existence: IsEnabled is File.Exists(StartupLink). No config flag to drift out of sync with reality. The checkbox reads the filesystem every time the menu opens.

Final word

The Python version was 2900 lines. The C# version is a larger feature surface in a comparable line count, ships as one file, and carries zero runtime dependencies on the target PC in the self-contained build. The seller gets an exe on the desktop and it works. That is the goal.

Author: Jeremy Anderson — dcos.net — info@dcos.net License: MIT (MIT License). See LICENSE.

16 KiB Raw Blame History Unescape Escape