diff --git a/CHANGELOG.md b/CHANGELOG.md index 178ab20..99d1f55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,222 @@ All notable changes to OpenTranscode. Versions follow semantic versioning. +## [4.8.1] — 2026-09-25 + +### Changed — GPU dropdown / ENGINE interaction +- **ENGINE = CPU now disables the GPU dropdown** (greyed out with an + explanatory tooltip) — the choice would have no effect there. +- **"None (CPU-only encode)" remains an explicit option** in the GPU + dropdown for the opposite case: a GPU present in the box that the + user doesn't want encoding on (e.g. it's the display card). The two + controls can no longer contradict each other: GPU engines + "None" + resolve to the CPU path by design, and CPU engine ignores the GPU + choice entirely. + +## [4.8.0] — 2026-09-25 + +### Added — GPU capability profiles (combined generations, incl. oddballs) +- **New `gpu_profiles.py` + GPU dropdown** (UI, next to ENGINE; CLI + `--gpu-profile`). Entries combine whole card generations into single + capability classes — same silicon, same encoding behaviour: + - NVIDIA Kepler/Maxwell (H.264 only), Pascal (GTX 10-series + Tesla + P40/P4/P100 — H.264+HEVC 8/10-bit), Turing (RTX 20 / GTX 16 / + Tesla T4 / **CMP 30/40/50HX** — +B-frames), Ampere (RTX 30 / + A10/A40 / **CMP 90HX** — no AV1 encode), Ada/Blackwell (RTX 40/50, + L4/L40 — +AV1 10-bit). + - Oddballs: **NVIDIA data-center compute (V100/A100/H100, CMP + 170HX) has NO NVENC silicon** — the profile routes to CPU instead + of failing. CMP 170HX is GA100-based: the fastest mining card that + cannot hardware-encode. + - Intel Arc (QSV: H.264+HEVC+AV1) and Iris/UHD; AMD RDNA 3 (VAAPI: + +AV1 encode), RDNA 1/2, and the crypto-era GCN 4/5 + Vega cards. +- **Auto-detection**: the probe matches the detected GPU name + (nvidia-smi / lspci, vendor-aware) to a profile; the live encode + smoke test decides what actually works. VAAPI encodes via + `-vaapi_device` + `hwupload`, QSV via `-init_hw_device qsv=hw`. +- **Profile-driven encoding**: `resolve_gpu_encoder()` now returns + (encoder, api) and the ffmpeg command is built per API (device init + args, hwupload filter chains, per-API quality args — nvenc + `-rc vbr -cq`, QSV `-global_quality`, VAAPI `-rc_mode CQP`). +- **Rebuild-from-git builds for the selected GPU profile**: the dep + tree extends with the vendor's packages (arch: nv-codec-headers / + libva+libdrm+mesa / intel-media-driver+onevpl; debian/redhat/suse + equivalents), so pressing REBUILD on a bare system generates the GPU + dependency tree too. +- Forced profiles let a user pin a capability class even when name + auto-match fails; "Auto-detect" stays the default. + +## [4.7.1] — 2026-09-25 + +### Added — always-usable REBUILD FROM GIT (self-generating dep tree) +- **The REBUILD FROM GIT button is now ALWAYS enabled.** It no longer + depends on a successful environment probe — a failed probe (missing + binaries, broken VSScript, missing av1an) is exactly when the rebuild + is needed, so the button works from first launch on a bare system. +- **The build generates its own dependency tree.** Instead of the old + pacman-only toolchain list, `build_dep_plan()` maps the detected distro + family to the right package set and installer: arch (pacman), debian + (apt-get), redhat (dnf), suse (zypper) — including the previously + missing **zimg** (VapourSynth's one hard library dependency, which + made the meson step fail on bare systems), meson/ninja/cmake/nasm and + rust for av1an. Unsupported families get an explicit manual-install + note. Critical tools are re-verified after install; the build aborts + with the exact list if anything is still missing. +- **BestSource plugin now builds from git as part of the VapourSynth + rebuild** (cloned with its libp2p submodule, compiled against the + freshly installed git-VS headers via PYTHONPATH/PKG_CONFIG_PATH, into + the user site-packages plugins dir). This closes the av1an chunking + gap: with BestSource present, av1an auto-selects the fast chunk + method instead of quadratic-decode `select`. The full chain was + verified live on this machine: VS git (Core R80) → BestSource → + `av1an --chunk-method bestsource` → rc=0 output. +- **Runtime env follows the git stack**: `_av1an_env()` and the plugin + probe now include the python user-site vapoursynth dir (module + libs + + plugins), so av1an loads the freshly built VS instead of the system + one after a rebuild. + +## [4.7.0] — 2026-09-25 + +### Added — Hybrid GPU + CPU scheduling +- **New engine: Hybrid (UI ENGINE combo / `--engine hybrid`).** The + queue is scanned once, then split between two CONCURRENT lanes: a GPU + lane (NVENC via single-pass ffmpeg) and a CPU lane (the family's + software encoder, or av1an chunk-parallel when opted in — so NVENC + + chunk workers + software encoders can all run at the same time on + multi-core boxes with an NVIDIA card). +- **LPT load balancing** (`hybrid_scheduler.plan_hybrid`): files sorted + by size descending, each assigned to the lane with the lower + estimated load using a GPU:CPU speed ratio (default 8:1) — both lanes + finish at roughly the same time. +- **CPU lane thread budget**: the CPU lane's topology is reduced by a + 2-thread reserve for the GPU lane's decode/scale/mux before the + intelligent worker math runs; the lane's software ffmpeg encodes are + additionally capped with `-threads N`. NVENC jobs are never + thread-capped (silicon-bound). +- **Per-lane temp dirs** (`worker--gpu` / `worker--cpu`): + both lanes share one process, so the PID alone no longer separates + them — a lane finishing early can no longer sweep the other lane's + intermediates. +- STOP stops both lanes; the final summary aggregates both lanes' + results. Hybrid needs 2+ encodable files and a functional GPU encoder + and otherwise falls back to a single CPU queue with a logged reason. + Lanes split FILES, never one file across encoders (mixed-encoder + chunks would produce visibly inconsistent quality within a file, and + av1an cannot drive NVENC). + +### Fixed — during hybrid hardening +- **libx265 rejects large `-threads` values** ("frameNumThreads must be + [0 .. X265_MAX_FRAME_THREADS)"): the CPU lane's injected thread cap is + clamped to 16 for libx265; SVT-AV1 and libvpx keep the full budget. +- **Failed ffmpeg encodes now delete their partial output.** A failed + encode used to leave a truncated file that ffprobe still parses as the + right codec/resolution — skip-existing would then treat it as a + finished archive forever. The output is unlinked on the ffmpeg error + path (the av1an and STOP paths already cleaned up). + +### Fixed — launcher crash on STARTUP +- `OpenCodecMaster._build_ui()` read `self.env.av1an_flags` while + pre-selecting the ENGINE combo, but `env` is None/absent until + `_probe_and_init` runs after the UI build — instantiating the window + crashed with `AttributeError: ... has no attribute 'env'` (launcher) + or on `NoneType` (package). The pre-select now reads defensively. + Verified by offscreen-instantiating the real launcher window. + +### Tests +- `tests/test_hybrid_scheduler.py` (19): LPT split, degenerate cases, + thread budget, `file_subset` end-to-end, per-lane temp dirs, ffmpeg + thread cap (CPU yes / GPU no), `scan_input_files`, CLI `--engine + hybrid`, launcher parity. + +## [4.6.0] — 2026-09-25 + +### Overview +GPU (NVENC) encoding with auto GPU/CPU engine selection, the root-cause +fix for the av1an chunking failures on ffmpeg 7+, and the remaining +large-file failure modes (pre-scale timeout, silent disk exhaustion, +whole-video loudnorm decode). + +### Added — GPU (NVENC) support + auto engine +- **Engine selector** (UI: Auto/GPU/CPU combo; CLI: `--engine + {auto,gpu,cpu}`). Auto uses the NVENC hardware encoder for the + selected codec family (x265 → `hevc_nvenc`, AV1 → `av1_nvenc` on + RTX 40+) when it actually works; VP9 has no NVENC encoder and stays + on CPU. GPU encodes run via single-pass ffmpeg with `-rc vbr -cq N` + (CPU decode), so av1an chunk-parallel is not needed — one NVENC + process outruns chunk-parallel CPU workers. +- **`env_probe.GpuInfo` + `_probe_gpu()`**: two-stage NVENC probe — + compiled-in encoder list, then a LIVE encode smoke test per encoder. + The live gate catches the real-world failure mode where ffmpeg lists + `hevc_nvenc` but the installed NVIDIA driver is older than the NVENC + API the build targets ("Driver does not support the required nvenc + API version. Required: 13.1 Found: 13.0" — observed on GTX 1070 + + driver 580 + ffmpeg 9.0.2), reports the driver fix, and the engine + falls back to CPU automatically. +- Skip-existing is engine-agnostic: `hevc_nvenc` outputs the same + ffprobe codec_name (`hevc`) as libx265, so switching engines never + re-encodes finished files. +- `--dry-run` prints the GPU verdict (usable encoders or the exact + failure detail). +- **`scripts/build-ffmpeg-nvenc-matched.sh`** — fixes the + driver/ffmpeg NVENC API mismatch without touching the system: + builds ffmpeg against the nv-codec-headers gen the installed driver + actually provides (e.g. 580 driver → gen 13.0) with the distro + build's exact feature set, installing to `~/.local` with + `--enable-rpath` (critical — without rpath the binary silently + loads the distro libavcodec and keeps demanding the newer API). + Verified on GTX 1070 + driver 580.178 + ffmpeg 9.0.2: hevc_nvenc + and h264_nvenc functional, av1_nvenc correctly reported as + unavailable (no Pascal AV1 hardware) and auto stays on SVT-AV1. + +### Fixed — av1an chunking (root cause on ffmpeg 7+) +- **ffmpeg ≥ 7 removed `-vsync`, which av1an's segment/hybrid chunk + extraction passes to ffmpeg.** Every segment-based chunk died + instantly ("Unrecognized option 'vsync'" → broken y4m pipe → chunk + fails 3×) — this produced the y4m pipe-break storm in the 2026-07-13 + av1an log. Verified end-to-end on ffmpeg 9.0.2 + av1an 0.5.2: + `select` still works (it uses the ffmpeg frame server, not + segmenting), so the plugin-less `select` override remains, the + failure is now diagnosed with an actionable block (install + bestsource/ffms2/lsmash, or stay on the ffmpeg-only path), and the + select retry now also triggers on it. +- **FRAME MISMATCH ("encoder crashed: exit status: 0")** — chunk + manifest vs encoded frame-count drift on sparse-keyframe sources + (20 worker shutdowns in the same log) fell through to "Unknown av1an + failure". Now a recognized per-file pattern that retries with + `select` (exact frame ranges can't drift). Diagnostics now scan + av1an's stdout too, since FRAME MISMATCH lines land there. + +### Fixed — large-file failures +- **The ffmpeg path no longer pre-scales.** The CRF-16 intermediate + existed only for VapourSynth source-plugin compatibility; the + ffmpeg/GPU path scales inline via `-vf`. This removes the whole + class of failures: the 0.5-0.8× source-size temp file, the extra + full encode pass, and the flat **1800s pre-scale timeout** that + killed long/high-bitrate sources at exactly 30 minutes. The av1an + path keeps the intermediate but now runs under the per-file timeout + with STOP-button support ("FAIL: pre-scale timeout" on expiry). +- **Severe disk-space warnings are user-facing.** "free < source size" + on the output or temp partition was verbose-only — quiet mode gave + zero notice before "No space left on device". Marginal advice stays + verbose-only. +- **Loudnorm analysis is audio-only (`-vn`).** It previously decoded + the entire video stream to measure audio loudness, pushing large + files past the 120s analysis timeout and silently degrading them to + the static knob gain. + +### Tests +- `tests/test_gpu_engine.py` (32): GpuInfo, the live-encode GPU gate + (driver-mismatch case included), the resolve matrix, NVENC vargs, + CLI `--engine`, worker engine plumbing, launcher parity. +- `tests/test_large_file_fixes.py` (11): pre-scale gating + timeout + + STOP handling, user-facing disk-space warnings, `-vn` loudnorm, + FRAME MISMATCH / vsync select retries. +- Harness: EncoderWorker has class-level defaults for `verbose` / + `_current_*` so `__new__`-built test instances match the v4.4.3 + contract; tests replace the Qt signal with `conftest.capture_signal()` + instead of patching read-only `SignalInstance` attributes (fixes all + 24 order-dependent failures on machines with a real PySide6). + ## [4.5.0] — 2026-07-26 (master) ### Overview diff --git a/README.md b/README.md index dcf1adb..a9ff80b 100755 --- a/README.md +++ b/README.md @@ -4,10 +4,15 @@ files to AV1 / VP9 / HEVC with configurable audio codecs, resolution scaling, and source-file management. Built on ffmpeg (default) with an optional av1an chunk-parallel path for users with a working VapourSynth -setup. +setup. NVIDIA GPU (NVENC) encoding is used automatically when available. - **Default encoder**: ffmpeg + libsvtav1 (reliable across distros) +- **GPU encoder**: NVENC (hevc_nvenc / av1_nvenc) auto-detected with a + live encode test — used by the Auto engine when it actually works - **Optional encoder**: av1an chunk-parallel (opt-in via UI toggle) +- **Self-sufficient rebuild**: the always-enabled REBUILD FROM GIT + button installs its own build dependencies (distro-aware), then builds + VapourSynth + BestSource + av1an into `~/.local` / `~/.cargo` - **Codecs**: AV1 (SVT-AV1), VP9, x265 (HEVC) video; Opus, Vorbis, FLAC, IAMF audio - **Containers**: MKV, WebM, MP4 - **Resolution**: Original or scaled (16:9, 21:9, 32:9 presets from 480p to 4K) @@ -23,6 +28,7 @@ setup. - ffmpeg (with libsvtav1, libvpx, libx265, libopus, libvorbis, flac) - ffprobe - PySide6 (for the GUI) +- Optional: NVIDIA GPU + driver for NVENC hardware encoding - Optional: av1an + VapourSynth (only if using the av1an toggle) ## Quick start @@ -34,7 +40,7 @@ cd /path/to/opentranscode pip install -e . opentranscode # launch the GUI -python -m opentranscode --version # → opentranscode 4.5.0 +python -m opentranscode --version # → opentranscode 4.6.0 python -m opentranscode --help ``` @@ -53,6 +59,58 @@ opentranscode --verify-only FILE.mkv # re-verify an existing output ## Usage +### Encode engine (GPU vs CPU) + +The ENGINE selector (UI) or `--engine` flag picks the video encoder: + +- **Auto (default)** — uses the NVENC hardware encoder for the selected + codec family when the environment probe's *live encode test* proved it + works (x265 → `hevc_nvenc`, AV1 → `av1_nvenc` on RTX 40+); otherwise + the software encoders. VP9 has no NVENC encoder and always uses the CPU. +- **GPU** — force NVENC; falls back to CPU with a log line when + unavailable. GPU encodes run via single-pass ffmpeg (`-c:v hevc_nvenc + -rc vbr -cq N`), with CPU decode. av1an chunk-parallel is not used — + one NVENC process outruns chunk-parallel CPU workers. +- **CPU** — force the software encoders (SVT-AV1 / VP9 / x265). +- **Hybrid** (v4.7.0) — run the GPU lane AND a CPU lane **concurrently**: + the queue is scanned up front and split by file size (LPT, balanced + with a GPU:CPU speed ratio) so both lanes finish at about the same + time. The CPU lane's thread budget is reduced by a small reserve for + the GPU lane's decode/mux, and with `--use-av1an` the CPU lane uses + chunk-parallel — so NVENC + chunk workers + software encoders can all + be busy at once. Lanes split FILES, never one file across encoders + (mixing hevc_nvenc and libx265 chunks inside a single file would make + scene-by-scene quality visibly inconsistent, and av1an cannot drive + NVENC anyway). Needs 2+ encodable files and a functional GPU encoder; + otherwise it falls back to a single CPU queue. GPU-lane files get + NVENC quality, CPU-lane files get software-encoder quality. + +The GPU gate is a real encode smoke test, not just an encoder-list grep: +a ffmpeg build can list `hevc_nvenc` while the installed NVIDIA driver is +too old for the NVENC API version it was compiled against (e.g. "Driver +does not support the required nvenc API version. Required: 13.1 Found: +13.0") — in that case the probe reports the exact driver fix and encoding +stays on the CPU. `opentranscode --dry-run` prints the GPU verdict. + +#### When NVENC is "present but NOT usable" (driver / ffmpeg API mismatch) + +ffmpeg is compiled against a specific NVENC API version and the NVIDIA +driver must be at least that new. On legacy driver branches (580 is the +**last** branch supporting Pascal cards such as the GTX 10xx) upgrading +the driver is not an option — so invert the fix: build ffmpeg against the +NVENC API your driver *does* provide and install it to `~/.local/bin` +(which PATHs before the distro binary; OpenTranscode picks it up +automatically on the next launch). Use the included helper: + +```bash +# "Found: 13.0" in the error → pass 13.0 +./scripts/build-ffmpeg-nvenc-matched.sh 13.0 +``` + +The script copies your distro ffmpeg's feature set (same version, same +libraries — a drop-in), builds against the matching `nv-codec-headers`, +and installs only into `$HOME/.local`. Nothing system-wide is touched. + ### Default (ffmpeg-only, recommended) ```bash @@ -100,14 +158,51 @@ opentranscode --verbose For 30GB+ BluRay rips: - **24-hour per-file timeout** (configurable via `--timeout SECONDS`) +- **Inline scaling on the ffmpeg path** — the ffmpeg (and GPU) path scales + with `-vf` directly in the encode command; no intermediate file, no + extra encode pass, no pre-scale timeout. The av1an path still writes a + CRF-16 (visually lossless) intermediate for VapourSynth compatibility, + but under the full per-file timeout instead of a flat 30-minute cap. - **1KB absolute integrity minimum** (no false "output too small" failures on high-bitrate sources — duration check is the real gate) -- **Disk-space warnings** (verbose only) for output and temp partitions +- **Disk-space warnings** — severe warnings (free space below the source + size on the output or temp partition) are user-facing even in quiet mode ```bash opentranscode --timeout 36000 # 10h per-file timeout ``` +### GPU capability profiles + +The GPU dropdown (next to ENGINE) lists combined card generations — +same silicon, same encoding behaviour — rather than individual SKUs: +NVIDIA Kepler through Ada/Blackwell, Intel Arc + integrated (QSV), and +AMD GCN/RDNA (VAAPI), including data-center and crypto-era oddballs +(Tesla P40/T4, CMP 30–90HX, and the GA100-based CMP 170HX + V100/A100/ +H100 boards, which ship without NVENC and route to the CPU path). +Auto-detect matches your card and the live encode probe decides what +works; forcing a profile extends the REBUILD FROM GIT dependency tree +with that GPU's packages. + +### Rebuilding the encode stack from git + +The REBUILD FROM GIT button is available at all times — even on a bare +system with no build tools installed: + +1. **Dependency tree** — detects missing tools/libraries and installs + them via the distro package manager (pacman/apt-get/dnf/zypper; one + privilege prompt). Includes zimg, which VapourSynth requires. +2. **VapourSynth from git** → `~/.local` (self-contained: module, libs, + headers in the python user site-packages). +3. **BestSource plugin from git** → compiled against that fresh VS, so + av1an gets a fast, frame-accurate chunk method (no more slow + `select`). +4. **av1an from git** → `~/.cargo/bin`. + +Restart the app afterwards; the probe picks up the new stack +automatically. `ffmpeg + IAMF` builds libiamf + ffmpeg into +`~/.local/bin` for the IAMF audio codec. + ### av1an chunk-parallel (opt-in) For users with a working VapourSynth + source plugin (lsmash, ffms2, @@ -116,15 +211,31 @@ bestsource) setup who want scene-detection-based chunk-parallel encoding: - **UI**: Check the "av1an (chunk-parallel)" checkbox - **CLI**: `opentranscode --use-av1an` -When av1an fails per-file (concat failures, scene-detection panics), -the code automatically falls back to ffmpeg for that file. When av1an -fails systematically (VSScript API mismatch, missing encoder), the queue -aborts with an actionable diagnostic. +Known chunking failure modes (all handled with per-file retries + a +diagnosis block, and safe to hit): + +- **ffmpeg ≥ 7 removed `-vsync`** — av1an's segment/hybrid chunk + extraction calls `ffmpeg -vsync` and dies with "Unrecognized option + 'vsync'" on modern ffmpeg. Without VS source plugins the app forces + `--chunk-method select` (works on any ffmpeg — verified — but is slow, + so the default ffmpeg-only path is usually faster for long files). + Install `bestsource`/`ffms2`/`lsmash` to give av1an a fast plugin + chunk method. +- **FRAME MISMATCH / "encoder crashed: exit status: 0"** — chunk + extraction drift on sources with sparse keyframes; the app retries the + file with `select` automatically. +- **y4m pipe breaks** ("Failed to read y4m frame delimiter") — same + select retry. + +When av1an fails per-file, the code automatically falls back to ffmpeg +for that file. When av1an fails systematically (VSScript API mismatch, +missing encoder), the queue aborts with an actionable diagnostic. ## CLI reference ``` opentranscode [--version] [--dry-run] [--verify-only PATH] [--force] + [--engine {auto,gpu,cpu,hybrid}] [--chunk-method METHOD] [--max-workers N] [--threads-per-worker N] [--use-av1an] [--verbose] [--skip-existing | --force-reencode] [--timeout SECONDS] @@ -136,6 +247,7 @@ opentranscode [--version] [--dry-run] [--verify-only PATH] [--force] | `--dry-run` | — | Probe environment + smoke test, no encode | | `--verify-only PATH` | — | Re-verify an existing output file | | `--force` | off | Skip ffprobe pre-validation | +| `--engine` | auto | `auto` = NVENC GPU encoder when the live encode test passes, else CPU; `gpu` = force NVENC; `cpu` = force software encoders | | `--chunk-method METHOD` | auto | Force av1an chunk method (select, hybrid, ffms2, lsmash, bestsource, dgdecnv) | | `--max-workers N` | auto | Cap chunk-parallel worker count | | `--threads-per-worker N` | auto | Per-encoder thread cap | @@ -152,9 +264,9 @@ opentranscode/ ├── __init__.py # Package metadata + lazy launch_gui wrapper ├── __main__.py # python -m opentranscode entry point ├── cli.py # argparse + dry-run + verify-only -├── codec_profiles.py # VideoCodecProfile, AudioProfile, ContainerProfile tables -├── encoder_worker.py # QThread-based per-file encode pipeline -├── env_probe.py # Distro + binary + library + av1an detection +├── codec_profiles.py # VideoCodecProfile (incl. NVENC gpu fields), Audio/Container/Resolution tables +├── encoder_worker.py # QThread-based per-file encode pipeline + engine (GPU/CPU) resolution +├── env_probe.py # Distro + binary + library + av1an + NVENC probe ├── ffprobe_utils.py # ffprobe_validate, ffprobe_duration, file-type ID ├── temp_manager.py # Per-worker temp directory isolation ├── cpu_topology.py # Physical core / logical thread detection @@ -168,7 +280,7 @@ opentranscode/ open-transcode.py # Launcher script (mirrors package, test target) pyproject.toml # PEP 621 build config -tests/ # 149 tests across 14 files +tests/ # 200+ tests across 16 files ``` ### Encode pipeline @@ -181,7 +293,8 @@ tests/ # 149 tests across 14 files - `_validate_file` — ffprobe pre-check (skip if invalid) - `_check_disk_space` — warn (verbose) if free space < source size - `_output_already_encoded` — skip if output exists with matching codec - - `_prepare_input` — pre-scale (if requested) or symlink to temp + - `_prepare_input` — pre-scale to a CRF-16 intermediate (av1an path only) + or symlink to temp; the ffmpeg/GPU path scales inline - `_encode_one` — dispatch to ffmpeg (default) or av1an (opt-in) - `_run_with_stop_check` — subprocess with STOP-button interrupt support - `_verify_and_finalize` — duration check (≥95%), subtitle mux, source deletion diff --git a/logs/av1an.log.2026-09-25 b/logs/av1an.log.2026-09-25 new file mode 100644 index 0000000..cbcfe9a --- /dev/null +++ b/logs/av1an.log.2026-09-25 @@ -0,0 +1,22 @@ +2026-09-25T10:15:49.304217Z WARN new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }}: av1an_core::settings: It is not recommended to use the "select" chunk method, as it is very slow +2026-09-25T10:15:49.306406Z DEBUG new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }}:initialize{self=Av1anContext { frames: 24, vs_script: None, vs_proxy_script: None, args: EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--scd", "0", "--rc", "0", "--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }, scene_factory: SceneFactory { data: ScenesData { frames: 0, scenes: None, split_scenes: None } } }}: av1an_core::context: temporary directory: .f0bc4e8 +2026-09-25T10:15:49.306672Z INFO encode_file: av1an_core::context: Input: 64x64 @ 24.000 fps, YUV420P, SDR +2026-09-25T10:15:49.389870Z INFO encode_file: av1an_core::scenes: scenecut: found 1 scene(s) [with extra_splits (240 frames): 1 scene(s)] +2026-09-25T10:15:49.390194Z INFO encode_file: av1an_core::context: +Queue 1 Workers 1 Encoder svt-av1 Passes 1 +Params: --scd 0 --rc 0 --preset 8 --crf 40 --keyint 240 +2026-09-25T10:15:49.390829Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: started chunk 00000: 24 frames +2026-09-25T10:15:50.052458Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: finished chunk 00000: 24 frames, 36.28 fps, took 661.45ms +2026-09-25T10:15:50.052761Z DEBUG encode_file: av1an_core::context: encoding finished, concatenating with mkvmerge +2026-09-25T10:15:50.052780Z DEBUG encode_file: av1an_core::context: `--ignore-frame-mismatch` not set. Forcing output FPS to 24 with mkvmerge. +2026-09-25T12:54:17.768234Z WARN new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }}: av1an_core::settings: It is not recommended to use the "select" chunk method, as it is very slow +2026-09-25T12:54:17.770255Z DEBUG new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }}:initialize{self=Av1anContext { frames: 24, vs_script: None, vs_proxy_script: None, args: EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--scd", "0", "--rc", "0", "--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }, scene_factory: SceneFactory { data: ScenesData { frames: 0, scenes: None, split_scenes: None } } }}: av1an_core::context: temporary directory: .709b7df +2026-09-25T12:54:17.770555Z INFO encode_file: av1an_core::context: Input: 64x64 @ 24.000 fps, YUV420P, SDR +2026-09-25T12:54:17.844031Z INFO encode_file: av1an_core::scenes: scenecut: found 1 scene(s) [with extra_splits (240 frames): 1 scene(s)] +2026-09-25T12:54:17.844213Z INFO encode_file: av1an_core::context: +Queue 1 Workers 1 Encoder svt-av1 Passes 1 +Params: --scd 0 --rc 0 --preset 8 --crf 40 --keyint 240 +2026-09-25T12:54:17.844608Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: started chunk 00000: 24 frames +2026-09-25T12:54:18.500267Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: finished chunk 00000: 24 frames, 36.62 fps, took 655.47ms +2026-09-25T12:54:18.503379Z DEBUG encode_file: av1an_core::context: encoding finished, concatenating with mkvmerge +2026-09-25T12:54:18.503397Z DEBUG encode_file: av1an_core::context: `--ignore-frame-mismatch` not set. Forcing output FPS to 24 with mkvmerge. diff --git a/open-transcode.py b/open-transcode.py index b60c7b2..aedb1fc 100755 --- a/open-transcode.py +++ b/open-transcode.py @@ -97,13 +97,14 @@ import re import shutil import signal import subprocess +import site import sys import tempfile import threading import time import ctypes from collections.abc import Callable -from dataclasses import dataclass, field +from dataclasses import dataclass, field, field from pathlib import Path from PySide6.QtWidgets import ( @@ -410,6 +411,25 @@ class VideoCodecProfile: # profile. Used by _output_already_encoded() to detect skip-existing. # av1 → "av1", vp9 → "vp9", hevc → "hevc". ffprobe_codec_name: str = "" + # v4.6.0: hardware (NVENC) counterpart for this codec family. Empty + # string = no hardware encoder exists for this family (VP9 has no + # NVENC encoder). The GPU path is ffmpeg-only (av1an cannot drive + # NVENC); EncoderWorker.resolve_gpu_encoder() only selects it when a + # functional probe proved the encoder works on this system. The + # ffprobe codec_name is IDENTICAL to the CPU encoder's (hevc_nvenc + # also produces "hevc"), so skip-existing detection works across + # GPU/CPU re-encodes of the same family. + gpu_encoder: str = "" + # (crf, preset) -> ffmpeg args for the NVENC encoder. Mirrors + # ffmpeg_vargs_fn. None when gpu_encoder is empty. + gpu_vargs_fn: Callable[[int, int], list[str]] | None = None + # v4.8.0: GPU-profile support. *gpu_family* is the codec family key + # used by GpuProfile.encoders ("av1"/"hevc"/"vp9"); *gpu_encoders_by_api* + # maps a hardware API (nvenc/vaapi/qsv) to this profile's ffmpeg + # encoder for that API. resolve_gpu_encoder() picks the entry matching + # the selected GPU profile. + gpu_family: str = "" + gpu_encoders_by_api: dict[str, str] = field(default_factory=dict) @dataclass class AudioProfile: @@ -499,6 +519,52 @@ def _x265_ffmpeg_args(crf: int, preset: int) -> list[str]: "-pix_fmt", "yuv420p10le", "-g", "240"] +# ── v4.6.0: NVENC (hardware) vargs ── +# NVENC quality control: -rc vbr + -cq N + -b:v 0 is the constant-quality +# mode that maps most closely to the CPU encoders' CRF (cq ≈ crf for HEVC +# and AV1 within ~±3). -b:v 0 removes the default bitrate cap so -cq +# actually governs quality. Presets are p1 (fastest) .. p7 (slowest/best) +# on all current NVENC generations; the legacy "slow/medium/fast" aliases +# are deprecated. +# +# Pixel format: 8-bit yuv420p. Pascal-generation cards (GTX 10xx) run +# HEVC Main10 at roughly half throughput, and the archival targets here +# are 8-bit phone/BluRay sources — 8-bit keeps the GPU path at full +# speed. ffmpeg auto-converts 10-bit sources to yuv420p. + +def _nvenc_preset(preset: int) -> str: + """Map the CPU preset tiers (lower value = slower/better) to NVENC + p-presets. CPU preset values across profiles are 0..10 with 0/5 = + slowest quality tiers; NVENC is fast enough that even p7 outruns any + CPU encoder, so the whole range compresses to p3..p7.""" + if preset <= 6: + return "p7" # "Slow" tier → best NVENC quality + if preset <= 8: + return "p5" # "Medium" tier + return "p4" # "Fast"/"Faster" tiers + + +def _hevc_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for hevc_nvenc (x265/HEVC family hardware encoder).""" + return ["-c:v", "hevc_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + +def _h264_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for h264_nvenc (hardware H.264 — compatibility target).""" + return ["-c:v", "h264_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + +def _av1_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for av1_nvenc (AV1 family hardware encoder, RTX 40+).""" + return ["-c:v", "av1_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + VIDEO_CODECS: list[VideoCodecProfile] = [ VideoCodecProfile( label="AV1 (SVT-AV1)", @@ -512,6 +578,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (8)", "Medium (6)", "Fast (4)", "Faster (2)"], preset_map={"Slow (8)": 8, "Medium (6)": 6, "Fast (4)": 4, "Faster (2)": 2}, ffprobe_codec_name="av1", # v4.3.0: skip-existing detection + # v4.6.0: av1_nvenc exists only on RTX 40+ (Ada) cards; on Pascal + # (GTX 10xx) the functional probe fails and auto falls back to + # the SVT-AV1 CPU encoder. + gpu_encoder="av1_nvenc", + gpu_vargs_fn=_av1_nvenc_args, + gpu_family="av1", + gpu_encoders_by_api={"nvenc": "av1_nvenc", "qsv": "av1_qsv", + "vaapi": "av1_vaapi"}, ), VideoCodecProfile( label="VP9", @@ -525,6 +599,10 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (0)", "Medium (2)", "Fast (4)", "Faster (6)"], preset_map={"Slow (0)": 0, "Medium (2)": 2, "Fast (4)": 4, "Faster (6)": 6}, ffprobe_codec_name="vp9", # v4.3.0: skip-existing detection + # v4.8.0: VP9 has no NVENC encoder; VAAPI (AMD/older Intel) can + # encode it on some cards. + gpu_family="vp9", + gpu_encoders_by_api={"vaapi": "vp9_vaapi"}, ), VideoCodecProfile( label="x265 (HEVC)", @@ -538,6 +616,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (5)", "Medium (7)", "Fast (9)", "Faster (10)"], preset_map={"Slow (5)": 5, "Medium (7)": 7, "Fast (9)": 9, "Faster (10)": 10}, ffprobe_codec_name="hevc", # v4.3.0: skip-existing detection + # v4.6.0: hevc_nvenc works on every NVENC generation since Maxwell + # GM206 (incl. the GTX 1070) — this is the family that benefits + # most from GPU mode. + gpu_encoder="hevc_nvenc", + gpu_vargs_fn=_hevc_nvenc_args, + gpu_family="hevc", + gpu_encoders_by_api={"nvenc": "hevc_nvenc", "qsv": "hevc_qsv", + "vaapi": "hevc_vaapi"}, ), ] @@ -596,6 +682,23 @@ FFMPEG_LIB_KEY_MAP: dict[str, str] = { "libaom-av1": "libaom", "libvpx-vp9": "libvpx", "libx265": "libx265", + # v4.6.0: hardware encoders. These keys are populated by + # _probe_ffmpeg_libs() alongside the software encoders, and — unlike + # the compiled-in check — EncoderWorker additionally gates the GPU + # path on env.gpu.functional (a real encode smoke test), because a + # ffmpeg build can list an NVENC encoder that the installed driver + # cannot open (NVENC API version mismatch). + "hevc_nvenc": "hevc_nvenc", + "h264_nvenc": "h264_nvenc", + "av1_nvenc": "av1_nvenc", + # v4.8.0: hardware APIs for AMD (VAAPI) and Intel (QSV) profiles. + "hevc_vaapi": "hevc_vaapi", + "h264_vaapi": "h264_vaapi", + "av1_vaapi": "av1_vaapi", + "vp9_vaapi": "vp9_vaapi", + "hevc_qsv": "hevc_qsv", + "h264_qsv": "h264_qsv", + "av1_qsv": "av1_qsv", } @@ -1326,6 +1429,140 @@ def detect_distro() -> DistroProfile: ) +# ────────────────────────────────────────────── +# GPU PROBE (v4.6.0 — NVENC hardware encoding) +# ────────────────────────────────────────────── + +# NVENC encoders we know how to drive, in preference order (best +# compression efficiency first). av1_nvenc only exists on RTX 40+; the +# functional smoke test below decides what is actually usable. +_NVENC_ENCODER_NAMES: tuple[str, ...] = ("av1_nvenc", "hevc_nvenc", "h264_nvenc") + + +@dataclass +class GpuInfo: + """Result of the GPU/NVENC probe. + + ``encoders`` — encoder name → ffmpeg was BUILT with it (from + ``ffmpeg -encoders``). + ``functional`` — encoder name → a real 0.2s NVENC encode SUCCEEDED. + This is the gate EncoderWorker uses: a ffmpeg build + can list hevc_nvenc while the installed driver is + too old for the NVENC API version it was compiled + against ("Driver does not support the required + nvenc API version") — only a live encode reveals + that. + ``details`` — encoder name → first stderr line when the smoke + test failed (actionable diagnostics). + """ + name: str = "" # GPU model name via nvidia-smi, "" if unknown + encoders: dict[str, bool] = field(default_factory=dict) + functional: dict[str, bool] = field(default_factory=dict) + details: dict[str, str] = field(default_factory=dict) + + @property + def usable_encoders(self) -> list[str]: + """Encoders that passed the live encode test, preference order.""" + return [e for e in _NVENC_ENCODER_NAMES if self.functional.get(e, False)] + + @property + def has_gpu(self) -> bool: + return bool(self.usable_encoders) + + @property + def first_failure_detail(self) -> str: + """First non-empty failure detail (for user-facing warnings).""" + for e in _NVENC_ENCODER_NAMES: + d = self.details.get(e, "") + if d: + return d + return "" + + +def _probe_gpu(ffmpeg_bin: str) -> GpuInfo: + """Detect NVIDIA NVENC hardware encoders and verify they actually work. + + Two-stage probe: + + 1. Compiled-in check — grep ``ffmpeg -encoders`` for the NVENC + encoder names. Cheap; answers "could this ffmpeg ever do NVENC". + 2. Functional smoke test — for each compiled-in encoder, encode a + 0.2s 256x264 lavfi color source with ``-c:v -f null -``. + Catches the real-world failure modes the compiled-in check + cannot: NVIDIA driver too old for the ffmpeg build's NVENC API + version, no /dev/nvidia* access, driver loaded but GPU dead. + + GPU model name is best-effort via nvidia-smi (display only). + """ + info = GpuInfo() + + if not ffmpeg_bin: + return info + + # --- Stage 1: compiled-in encoders --- + try: + res = subprocess.run( + [ffmpeg_bin, "-hide_banner", "-encoders"], + capture_output=True, text=True, timeout=10, + ) + encoders_out = res.stdout or "" + except (OSError, subprocess.SubprocessError): + encoders_out = "" + + for enc in _NVENC_ENCODER_NAMES: + info.encoders[enc] = f" {enc} " in encoders_out + + compiled_in = [e for e in _NVENC_ENCODER_NAMES if info.encoders[e]] + if not compiled_in: + return info # no hardware encoders in this build — skip stage 2 + + # --- GPU model name (display only, never gates anything) --- + nvidia_smi = shutil.which("nvidia-smi") + if nvidia_smi: + try: + res = subprocess.run( + [nvidia_smi, "--query-gpu=name", "--format=csv,noheader"], + capture_output=True, text=True, timeout=5, + ) + if res.returncode == 0 and res.stdout.strip(): + info.name = res.stdout.strip().splitlines()[0].strip() + except (OSError, subprocess.SubprocessError): + pass + + # --- Stage 2: functional smoke test per compiled-in encoder --- + for enc in compiled_in: + try: + res = subprocess.run( + [ + ffmpeg_bin, "-hide_banner", "-loglevel", "error", + "-f", "lavfi", + "-i", "color=c=black:s=256x256:d=0.2:r=24", + "-frames:v", "5", + "-c:v", enc, "-f", "null", "-", + ], + capture_output=True, text=True, timeout=20, + ) + info.functional[enc] = res.returncode == 0 + if res.returncode != 0: + # First stderr line with substance (nvenc errors are + # prefixed "hevc_nvenc @ 0x...]" — keep them readable). + for line in (res.stderr or "").splitlines(): + line = line.strip() + if line: + # Strip the "name @ 0xADDR]" prefix for brevity. + line = re.sub(r"^\[[^]]+@\s*0x[0-9a-f]+\]\s*", "", line) + info.details[enc] = line[:160] + break + except subprocess.TimeoutExpired: + info.functional[enc] = False + info.details[enc] = f"{enc} smoke test timed out after 20s" + except (OSError, subprocess.SubprocessError) as e: + info.functional[enc] = False + info.details[enc] = str(e)[:160] + + return info + + # ────────────────────────────────────────────── # ENVIRONMENT PROBE (distro-aware, extended) # ────────────────────────────────────────────── @@ -1353,6 +1590,9 @@ class EnvProbe: vs_version: str | None = None # VapourSynth version string (for diagnostics) vs_script_lib: str | None = None # path to libvapoursynth-script.so that passed cpu: CpuTopology = field(default_factory=lambda: CpuTopology(1, 1, 1, "Unknown")) + # v4.6.0: GPU/NVENC probe result. EncoderWorker reads + # env.gpu.functional[encoder_name] when the engine is auto/gpu. + gpu: GpuInfo = field(default_factory=GpuInfo) errors: list[str] = field(default_factory=list) warnings: list[str] = field(default_factory=list) @@ -1442,6 +1682,11 @@ def _probe_ffmpeg_libs(ffmpeg_bin: str) -> dict[str, bool]: ("libopus", ["libopus "]), ("libvorbis", ["libvorbis "]), ("flac", ["flac "]), + # v4.6.0: NVENC hardware encoders (compiled-in check only — the + # live-encode gate is _probe_gpu()'s functional dict). + ("hevc_nvenc", ["hevc_nvenc "]), + ("h264_nvenc", ["h264_nvenc "]), + ("av1_nvenc", ["av1_nvenc "]), ] libs = {} for lib_name, search_strings in checks: @@ -1740,6 +1985,37 @@ def probe_environment() -> EnvProbe: if result.ffmpeg_version: result.warnings.append(f"FFmpeg version: {result.ffmpeg_version}") result.ffmpeg_libs = _probe_ffmpeg_libs(result.ffmpeg_path) + result.ffmpeg_libs = _probe_ffmpeg_libs(result.ffmpeg_path) + + # v4.6.0: GPU/NVENC probe — compiled-in check + live encode smoke + # test. EncoderWorker gates the GPU path on env.gpu.functional; + # this warning block surfaces the result (and the fix when the + # driver is too old for the ffmpeg build's NVENC API). + result.gpu = _probe_gpu(result.ffmpeg_path) + gpu = result.gpu + if gpu.has_gpu: + result.warnings.append( + f"GPU: {gpu.name or 'NVIDIA'} — NVENC ready: " + f"{', '.join(gpu.usable_encoders)} (engine: Auto will use the GPU)" + ) + elif gpu.encoders and any(gpu.encoders.values()): + present = [e for e in _NVENC_ENCODER_NAMES if gpu.encoders.get(e)] + detail = gpu.first_failure_detail + result.warnings.append( + f"GPU: NVENC encoder(s) {', '.join(present)} present in ffmpeg " + f"but NOT usable — {detail or 'smoke test failed'}. " + f"Auto engine will fall back to CPU." + ) + if "API version" in detail or "minimum required Nvidia driver" in detail: + result.warnings.append( + " FIX: update the NVIDIA driver (the ffmpeg build's NVENC " + "API is newer than the installed driver supports), or use " + "an ffmpeg build matching the installed driver." + ) + else: + result.warnings.append( + "GPU: no hardware encoder in this ffmpeg build — CPU encoding." + ) # Warn about missing AUDIO libs (video codecs are handled by av1an's own # encoder binaries — ffmpeg's video encoder list is irrelevant) @@ -1833,6 +2109,32 @@ def probe_environment() -> EnvProbe: "chunk-parallel encoding." ) result.av1an_flags["chunk_method_override"] = "select" + result.av1an_flags["chunk_method_override"] = "select" + + # v4.6.0: ffmpeg ≥ 7 removed the -vsync option that av1an's + # segment/hybrid chunk extraction passes to ffmpeg. On those + # systems every segment-based chunk dies immediately with + # "Unrecognized option 'vsync'." — the y4m pipe breaks and + # each chunk fails 3x. The select override above already + # avoids those methods when no plugins are installed; this + # warning tells plugin-less users on new ffmpeg WHY av1an is + # stuck on slow select. + if not vs_plugins and result.ffmpeg_version: + try: + ffmpeg_major = int( + re.match(r"[nN]?(\d+)", result.ffmpeg_version).group(1) + ) + except (AttributeError, ValueError): + ffmpeg_major = 0 + if ffmpeg_major >= 7: + result.warnings.append( + "av1an note: ffmpeg ≥ 7 removed the -vsync option av1an's " + "segment/hybrid chunk methods use — those methods fail " + "with \"Unrecognized option 'vsync'\". Chunking stays on " + "'select'. Install a VapourSynth source plugin " + "(bestsource/ffms2/lsmash) to escape slow select, or use " + "the default ffmpeg-only path." + ) except (OSError, subprocess.SubprocessError) as e: result.errors.append(f"av1an probe failed: {e}") @@ -2042,12 +2344,29 @@ def _av1an_env() -> dict[str, str]: When VapourSynth is built from git and installed to ~/.local/, the linker won't find libvapoursynth-script.so unless LD_LIBRARY_PATH points there. This function ensures every av1an invocation inherits that path. + + v4.7.1: the git VS stack is self-contained in the python user + site-packages (module + libs + BestSource plugin), so the runtime env + also gets that dir on LD_LIBRARY_PATH and the user site on PYTHONPATH — + otherwise av1an loads the system VS and never sees the fresh stack. """ env = os.environ.copy() local_lib = str(Path.home() / ".local" / "lib") existing = env.get("LD_LIBRARY_PATH", "") if local_lib not in existing: env["LD_LIBRARY_PATH"] = f"{local_lib}:{existing}".rstrip(":") + try: + user_site = Path(site.getusersitepackages()) + vs_dir = user_site / "vapoursynth" + if vs_dir.is_dir() and (vs_dir / "libvsscript.so").exists(): + existing = env.get("LD_LIBRARY_PATH", "") + if str(vs_dir) not in existing: + env["LD_LIBRARY_PATH"] = f"{vs_dir}:{existing}".rstrip(":") + py_path = env.get("PYTHONPATH", "") + if str(user_site) not in py_path: + env["PYTHONPATH"] = f"{user_site}:{py_path}".rstrip(":") + except (AttributeError, OSError): + pass return env @@ -2101,6 +2420,18 @@ def _probe_vs_source_plugins() -> list[str]: search_dirs.append(Path("/usr/lib/vapoursynth")) search_dirs.append(Path("/usr/local/lib/vapoursynth")) search_dirs.append(Path("/usr/lib/x86_64-linux-gnu/vapoursynth")) + # v4.7.1: the git-built VapourSynth stack installs its plugins into + # the python site-packages tree (module + libs + plugins/ are one + # self-contained unit). Probe those dirs too. + try: + search_dirs.append(Path(site.getusersitepackages()) / "vapoursynth" / "plugins") + except (AttributeError, OSError): + pass + try: + for d in site.getsitepackages(): + search_dirs.append(Path(d) / "vapoursynth" / "plugins") + except (AttributeError, OSError): + pass found: set[str] = set() for d in search_dirs: @@ -2294,20 +2625,29 @@ def _mkdir_private(path: Path) -> bool: return False -def _worker_temp_dir(worker_pid: int) -> Path: +def _worker_temp_dir(worker_pid: int, lane: str = "") -> Path: """Return a per-worker temp subdir named by PID. v3: each EncoderWorker gets its own subdir under the shared app temp dir, so the final cleanup sweep can safely nuke only this worker's intermediates without affecting a concurrent worker. The subdir is also created with mode=0o700 (FIO09-C). + + v4.7.0: *lane* suffixes the dir ("gpu"/"cpu") for the hybrid + scheduler's concurrent lanes — both run in the SAME process, so the + PID alone no longer separates them, and a lane finishing early must + not sweep the other lane's intermediates out from under it. """ base = _get_app_temp_dir() - sub = base / f"worker-{worker_pid}" + name = f"worker-{worker_pid}" + (f"-{lane}" if lane else "") + sub = base / name _mkdir_private(sub) return sub + + + def _temp_path_for(file_path: Path, suffix: str = ".scaled_tmp.mkv", worker_dir: Path | None = None) -> Path: """Build a unique temp path for *file_path* inside the app temp dir. @@ -2620,11 +2960,204 @@ def _compute_intelligent_worker_count_for( # ENCODER WORKER (QThread, from PySide6 ver, extended) # ────────────────────────────────────────────── +def scan_input_files(in_dir: Path, extensions: set[str]) -> list[Path]: + """Collect the transcodable files under *in_dir* (v4.7.0). + + Used by ``EncoderWorker.run()`` for the default whole-directory scan + AND by the hybrid scheduler's pre-scan, which must partition the + queue BEFORE the per-lane workers are constructed. Excludes leftover + pre-scale intermediates from previous failed runs. + """ + return sorted( + f for f in in_dir.rglob("*") + if f.is_file() + and f.suffix.lower() in extensions + and not f.name.endswith(".scaled_tmp.mkv") + ) + + + + +GPU_SPEED_RATIO_DEFAULT = 8 + +# Threads held back from the CPU lane so the GPU lane's decode / scale / +# mux processes stay responsive. The NVENC encode itself runs on the GPU +# silicon; the CPU side of a nvenc job is light. +HYBRID_CPU_RESERVE_THREADS = 2 + + +@dataclass +class HybridPlan: + """Result of planning a hybrid (GPU + CPU) queue split.""" + + gpu_files: list[Path] = field(default_factory=list) + cpu_files: list[Path] = field(default_factory=list) + gpu_encoder: str = "" # e.g. "hevc_nvenc" + cpu_budget_threads: int = 1 # CPU lane thread budget (logical - reserve) + gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT + + @property + def total_files(self) -> int: + return len(self.gpu_files) + len(self.cpu_files) + + +def plan_hybrid( + files: list[Path], + gpu_encoder: str | None, + gpu_functional: bool, + logical_threads: int, + sizes: dict[Path, int] | None = None, + gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT, + cpu_reserve: int = HYBRID_CPU_RESERVE_THREADS, +) -> HybridPlan | None: + """Split *files* between the GPU and CPU lanes, or return None when a + hybrid split cannot apply. + + Returns None when: + - the codec family has no GPU encoder, or the live GPU probe failed + (caller should fall back to a plain CPU queue), or + - *files* is empty. + + Assignment is LPT (longest-processing-time first): files are sorted + by size descending and each goes to the lane with the lower + estimated load, where the GPU lane's per-file cost is size / + gpu_speed_ratio. Both lanes then finish at roughly the same time. + + *sizes* maps files to byte sizes; missing entries fall back to the + mean of the known sizes (or 10 MB when nothing is known) so a single + unreadable file cannot skew the whole split. + """ + if not gpu_encoder or not gpu_functional or not files: + return None + + sizes = sizes or {} + known = [s for s in sizes.values() if s] + avg = sum(known) // len(known) if known else 10_000_000 + + def size_of(f: Path) -> int: + return sizes.get(f) or avg + + ratio = max(1, int(gpu_speed_ratio)) + gpu_files: list[Path] = [] + cpu_files: list[Path] = [] + gpu_load = 0.0 + cpu_load = 0.0 + + for f in sorted(files, key=size_of, reverse=True): + s = size_of(f) + gpu_est = gpu_load + s / ratio + cpu_est = cpu_load + s + # Tie goes to the GPU lane — it finishes the file sooner and the + # CPU lane keeps its current file longer. + if gpu_est <= cpu_est: + gpu_files.append(f) + gpu_load = gpu_est + else: + cpu_files.append(f) + cpu_load = cpu_est + + return HybridPlan( + gpu_files=gpu_files, + cpu_files=cpu_files, + gpu_encoder=gpu_encoder, + cpu_budget_threads=max(1, int(logical_threads) - cpu_reserve), + gpu_speed_ratio=ratio, + ) + + +def scan_input_files(in_dir: Path, extensions: set[str]) -> list[Path]: + """Collect the transcodable files under *in_dir* (v4.7.0). + + Used by ``EncoderWorker.run()`` for the default whole-directory scan + AND by the hybrid scheduler's pre-scan, which must partition the + queue BEFORE the per-lane workers are constructed. Excludes leftover + pre-scale intermediates from previous failed runs. + """ + return sorted( + f for f in in_dir.rglob("*") + if f.is_file() + and f.suffix.lower() in extensions + and not f.name.endswith(".scaled_tmp.mkv") + ) + + +def selected_gpu_profile(env): + """v4.8.0: the active GpuProfile — the UI's dropdown selection + (env.av1an_flags["gpu_profile"]) when set, else the auto-matched + profile from the probe (env.gpu.profile_key). None when neither.""" + flags = getattr(env, "av1an_flags", None) or {} + key = flags.get("gpu_profile") + if key and key != "auto": + profile = gpu_profile_by_key(key) + if profile is not None: + return profile + gpu_info = getattr(env, "gpu", None) + if gpu_info is not None and getattr(gpu_info, "profile_key", ""): + return gpu_profile_by_key(gpu_info.profile_key) + return None + + +def resolve_gpu_encoder(engine: str, video_codec, env): + """Decide whether this encode runs on the GPU. + + Returns ``(encoder_name, api)`` when the GPU path should be used — + e.g. ``("hevc_nvenc", "nvenc")`` or ``("hevc_vaapi", "vaapi")`` — or + ``(None, None)`` for the CPU path. + + Rules: + - ``engine == "cpu"`` → always CPU (user forced CPU). + - selected/matched GPU profile has + no encoder for the codec family → CPU (e.g. AV1 on Pascal, + VP9 without VAAPI). + - ``engine`` auto/gpu AND the + functional probe passed for + that encoder → the encoder. + + The gate is ``env.gpu.functional`` — a live encode test run by the + env probe — NOT the compiled-in ``ffmpeg -encoders`` list, because a + ffmpeg build can advertise a hardware encoder the installed driver + is too old to open. Pure function; no I/O. Safe to call from the UI + thread for a pre-flight status line. + """ + if engine not in ("auto", "gpu"): + return (None, None) + profile = selected_gpu_profile(env) + if profile is not None: + family = getattr(video_codec, "gpu_family", "") or "" + gpu_enc = profile.encoders.get(family) + api = profile.api + else: + # No GPU profile context (older callers / no probe data): fall + # back to the profile's NVENC encoder. + gpu_enc = getattr(video_codec, "gpu_encoder", "") or "" + api = "nvenc" if gpu_enc else None + if not gpu_enc: + return (None, None) + gpu_info = getattr(env, "gpu", None) + if gpu_info is not None and gpu_info.functional.get(gpu_enc, False): + return (gpu_enc, api) + return (None, None) + + + + class EncoderWorker(QThread): log_msg = Signal(str) progress_msg = Signal(str, int, int) # (filename, current, total) finished_queue = Signal(int, int) # (success_count, fail_count) + # v4.4.3/v4.6.0: class-level defaults for attributes normally set in + # __init__. The mocked test suite builds workers via ``__new__`` + # (bypassing __init__) and calls non-Qt methods directly; without + # these defaults those instances crash with AttributeError on + # ``verbose`` / ``_current_total`` (the "AttributeError: no attribute + # 'verbose'" class of bugs from the v4.4.3 changelog). Instance + # assignment in __init__ shadows these harmlessly. + verbose = False + _current_idx = 0 + _current_total = 0 + _current_filename = "" + def __init__( self, in_dir: Path, @@ -2655,7 +3188,28 @@ class EncoderWorker(QThread): # to honor them. max_workers: int | None = None, threads_per_worker: int | None = None, + # v4.6.0: encode engine selection. "auto" uses the NVENC GPU + # encoder when the selected codec family has one AND the env + # probe's live encode test proved it works on this system; + # otherwise (or with "cpu") the CPU encoders are used. "gpu" + # requests GPU and falls back to CPU with a log line when the + # hardware is unavailable. Resolved against env.av1an_flags + # ["engine"] like the other CLI-plumbed flags. + engine: str | None = None, + # v4.7.0: hybrid-lane support. *file_subset* restricts this + # worker to an explicit file list (the hybrid scheduler scans and + # partitions the queue up front, then spawns a GPU-lane and a + # CPU-lane worker with disjoint subsets). *lane* suffixes the + # per-worker temp dir so one lane's cleanup sweep can never + # delete the other lane's intermediates. *ffmpeg_threads* caps + # the CPU lane's software ffmpeg encode (GPU jobs are capped by + # NVENC silicon, not threads). + file_subset: list[Path] | None = None, + lane: str = "", + ffmpeg_threads: int | None = None, ): + + super().__init__() self.in_dir = in_dir self.out_dir = out_dir @@ -2694,6 +3248,22 @@ class EncoderWorker(QThread): ) else None ) ) + # v4.6.0: engine selection ("auto" | "gpu" | "cpu"). Falls back to + # env.av1an_flags["engine"] when not passed explicitly (same + # pattern as max_workers — lets the CLI reach the GUI-spawned + # worker without ui_window changes). Resolved to a concrete + # GPU/CPU decision in run() via resolve_gpu_encoder(). + self.engine = engine if engine in ("auto", "gpu", "cpu") else ( + env.av1an_flags.get("engine", "auto") + if env.av1an_flags.get("engine") in ("auto", "gpu", "cpu") + else "auto" + ) + # Resolved in run(): NVENC encoder name when the GPU path is + # active, None for CPU. _ffmpeg_fallback_encode and + # _prepare_input read this (GPU mode implies the ffmpeg path — + # av1an cannot drive NVENC — which also means no pre-scale + # intermediate: the ffmpeg path scales inline). + self._gpu_encoder: str | None = None # Resolved at run() time — kept on self so _encode_one can read it # without changing its call signature (which is invoked recursively # by the y4m-pipe-break retry path). @@ -2742,7 +3312,12 @@ class EncoderWorker(QThread): # cleanup sweep can safely nuke only this worker's intermediates # without affecting a concurrent worker. The subdir is created # with mode=0o700 to prevent symlink attacks from other users. - self._temp_dir = _worker_temp_dir(os.getpid()) + self.file_subset = file_subset + self.lane = lane + self.ffmpeg_threads = ffmpeg_threads + self._temp_dir = _worker_temp_dir(os.getpid(), lane=lane) + + # v6-06: KeepAwake instance — started in run(), stopped in finally. # mouse_nudge defaults to False (opt-in) to avoid surprising the # user with cursor movement. systemd-inhibit is always-on when @@ -3040,7 +3615,25 @@ class EncoderWorker(QThread): Returns True on success, False on failure. """ # Check if ffmpeg has the video encoder we need - ffmpeg_enc = self.video_codec.ffmpeg_encoder + # v4.6.0: GPU mode swaps in the hardware encoder and its vargs. + # v4.8.0: VAAPI/QSV APIs build their command shape (device init, + # hwupload filter, quality args) from gpu_profiles; NVENC keeps + # the original profile vargs. + ffmpeg_enc = self._gpu_encoder or self.video_codec.ffmpeg_encoder + v_args = ( + (self.video_codec.gpu_vargs_fn or self.video_codec.ffmpeg_vargs_fn) + if self._gpu_encoder else self.video_codec.ffmpeg_vargs_fn + )(self.crf, self.preset_val) + hw_pre_args: list[str] = [] + hw_filter_args: list[str] = [] + if self._gpu_encoder and self._gpu_api in ("vaapi", "qsv"): + profile = selected_gpu_profile(self.env) + if profile is not None: + hw_pre_args = encoder_pre_args(profile) + hw_filter_args = encoder_filter_chain(profile) + v_args = encoder_quality_args( + self._gpu_api, ffmpeg_enc, self.crf, self.preset_val + ) # v3: use the module-level FFMPEG_LIB_KEY_MAP (OTC-007). ffmpeg_lib_key = ffmpeg_lib_key_for(ffmpeg_enc) @@ -3051,19 +3644,29 @@ class EncoderWorker(QThread): ) return False - v_args = self.video_codec.ffmpeg_vargs_fn(self.crf, self.preset_val) - # Belt-and-suspenders: if a target resolution is set, inject -vf scale # directly into the ffmpeg command. This guarantees the output resolution # matches the dropdown even if the intermediate pre-scale was bypassed. vf_scale_args: list[str] = [] if self.resolution.width is not None and self.resolution.height is not None: - vf_scale_args = [ - "-vf", ( - f"scale={self.resolution.width}:{self.resolution.height}:" - f"force_original_aspect_ratio=decrease:force_divisible_by=2" - ), - ] + if self._gpu_api == "vaapi": + # v4.8.0: VAAPI scales ON the hardware — combine the + # scalar with the hwupload upload in one chain. + vf_scale_args = [ + "-vf", ( + f"scale={self.resolution.width}:{self.resolution.height}:" + f"force_original_aspect_ratio=decrease:force_divisible_by=2," + f"format=nv12,hwupload" + ), + ] + hw_filter_args = [] + else: + vf_scale_args = [ + "-vf", ( + f"scale={self.resolution.width}:{self.resolution.height}:" + f"force_original_aspect_ratio=decrease:force_divisible_by=2" + ), + ] # Audio args from profile audio_args = list(self.audio_profile.params) @@ -3084,10 +3687,37 @@ class EncoderWorker(QThread): if self.container.ext == "mp4": mux_flags = ["-movflags", "+faststart"] - cmd = [ - self.env.ffmpeg_path, + cmd = [self.env.ffmpeg_path] + hw_pre_args + [ "-i", str(encode_input), - ] + vf_scale_args + v_args + audio_args + mux_flags + [ + ] + vf_scale_args + hw_filter_args + v_args + # v4.7.0: CPU lane thread cap in hybrid mode (the GPU lane's + # NVENC job keeps ~2 threads for decode/mux). Never applied on + # the GPU path — NVENC throughput is silicon-bound, not + # thread-bound. Skipped when the cap is 0/unset. + if self._gpu_encoder is None and self.ffmpeg_threads: + # v4.7.1: libx265 maps -threads to frame-threads, capped at + # X265_MAX_FRAME_THREADS (16) — larger values abort the + # encoder ("frameNumThreads must be [0 .. X265_MAX_FRAME_ + # THREADS)"). SVT-AV1 and libvpx accept the full budget. + threads = self.ffmpeg_threads + if ffmpeg_enc == "libx265": + threads = min(threads, 16) + cmd += ["-threads", str(threads)] + + # v4.7.0: CPU lane thread cap in hybrid mode (the GPU lane's + # NVENC job keeps ~2 threads for decode/mux). Never applied on + # the GPU path — NVENC throughput is silicon-bound, not + # thread-bound. Skipped when the cap is 0/unset. + if self._gpu_encoder is None and self.ffmpeg_threads: + # v4.7.1: libx265 maps -threads to frame-threads, capped at + # X265_MAX_FRAME_THREADS (16) — larger values abort the + # encoder ("frameNumThreads must be [0 .. X265_MAX_FRAME_ + # THREADS)"). SVT-AV1 and libvpx accept the full budget. + threads = self.ffmpeg_threads + if ffmpeg_enc == "libx265": + threads = min(threads, 16) + cmd += ["-threads", str(threads)] + cmd += audio_args + mux_flags + [ "-y", str(output_f), ] @@ -3133,6 +3763,11 @@ class EncoderWorker(QThread): self.log_msg.emit( f" ffmpeg error (rc={res.returncode}): {stderr_snip.strip()}" ) + # v4.7.1: remove the partial output. Without this, a + # failed encode left a truncated file that ffprobe can + # still parse as the right codec — and skip-existing + # would then treat it as a finished archive forever. + output_f.unlink(missing_ok=True) return False except OSError as e: self.log_msg.emit(f"{self._status_prefix()}FAIL: system error: {e}") @@ -3154,14 +3789,45 @@ class EncoderWorker(QThread): phys = self.env.cpu.physical_cores logical = self.env.cpu.logical_threads + # ── v4.6.0: engine resolution (GPU vs CPU) ── + # GPU mode is a variant of the ffmpeg path: av1an invokes + # encoder CLI binaries (SvtAv1EncApp / vpxenc / x265) and cannot + # drive NVENC, so an active GPU encoder forces the single-pass + # ffmpeg path. NVENC on even a GTX 1070 encodes 1080p at several + # hundred fps — one ffmpeg process beats av1an's chunk-parallel + # CPU workers, and chunking becomes unnecessary. + self._gpu_encoder, self._gpu_api = resolve_gpu_encoder( + self.engine, self.video_codec, self.env) + if self._gpu_encoder: + self.use_ffmpeg_fallback = True + self.log_msg.emit( + f"ENGINE: GPU ({self._gpu_encoder}, {self._gpu_api}) — " + f"single-pass ffmpeg hardware encode; av1an chunk-parallel " + f"not used." + ) + elif self.engine == "gpu": + gpu_enc = getattr(self.video_codec, "gpu_encoder", "") or "" + if not gpu_enc and not getattr(self.video_codec, "gpu_family", ""): + self.log_msg.emit( + f"ENGINE: GPU requested but {self.video_codec.label} has no " + f"hardware encoder — using CPU." + ) + else: + gpu = getattr(self.env, "gpu", None) + detail = gpu.first_failure_detail if gpu is not None else "" + self.log_msg.emit( + f"ENGINE: GPU requested but no usable hardware encoder for " + f"{self.video_codec.label} ({detail or 'unavailable'}) — using CPU." + ) + # Collect all valid files first (for progress tracking) # Exclude our own temp intermediates from previous failed runs. - all_files = sorted( - f for f in self.in_dir.rglob("*") - if f.is_file() - and f.suffix.lower() in self.extensions - and not f.name.endswith(".scaled_tmp.mkv") - ) + if self.file_subset is not None: + # v4.7.0: hybrid lane — the scheduler partitioned the queue. + all_files = sorted(self.file_subset) + else: + all_files = scan_input_files(self.in_dir, self.extensions) + total = len(all_files) if total == 0: @@ -3170,30 +3836,37 @@ class EncoderWorker(QThread): return # ── Mode banner ── + # v4.2.1: mode banner is verbose-only. The user doesn't need + # to know the worker math — they just need files to encode. # use_ffmpeg_fallback is set by the main thread's pre-flight check. - # If True, the main thread already logged the fallback reason. - if self.use_ffmpeg_fallback: - self.log_msg.emit( - f"FFmpeg fallback: {self.video_codec.ffmpeg_encoder} on {phys} cores " - f"(single-pass, no chunk-parallel)" - ) - else: - # v4.1.0: show the thread budget so the user can verify the - # intelligent worker math at a glance. e.g. on a 28-thread Xeon: - # "Chunk-parallel: 6 workers × 4 threads = 24 active - # (28 logical - 4 reserved for OS/UI)" - active = worker_count * threads_per_worker - reserved = logical - active - self.log_msg.emit( - f"Chunk-parallel: {worker_count} workers × {threads_per_worker} threads " - f"= {active} active " - f"({logical} logical - {reserved} reserved for OS/UI)" - ) - if self.max_workers is not None or self.threads_per_worker_override is not None: - self.log_msg.emit( - f" (overrides: max_workers={self.max_workers!r}, " - f"threads_per_worker={self.threads_per_worker_override!r})" + if self.verbose: + if self._gpu_encoder: + self._vlog( + f"GPU encode: {self.video_codec.label} via " + f"{self._gpu_encoder} (NVENC), CPU decode" ) + elif self.use_ffmpeg_fallback: + self._vlog( + f"FFmpeg fallback: {self.video_codec.ffmpeg_encoder} on {phys} cores " + f"(single-pass, no chunk-parallel)" + ) + else: + # v4.1.0: show the thread budget so the user can verify the + # intelligent worker math at a glance. e.g. on a 28-thread Xeon: + # "Chunk-parallel: 6 workers × 4 threads = 24 active + # (28 logical - 4 reserved for OS/UI)" + active = worker_count * threads_per_worker + reserved = logical - active + self._vlog( + f"Chunk-parallel: {worker_count} workers × {threads_per_worker} threads " + f"= {active} active " + f"({logical} logical - {reserved} reserved for OS/UI)" + ) + if self.max_workers is not None or self.threads_per_worker_override is not None: + self._vlog( + f" (overrides: max_workers={self.max_workers!r}, " + f"threads_per_worker={self.threads_per_worker_override!r})" + ) self.log_msg.emit(f"Found {total} file(s) to process.") self.log_msg.emit(f"Temp dir: {self._temp_dir}") @@ -3590,7 +4263,15 @@ class EncoderWorker(QThread): # temp directory so the user's video folders stay clean. encode_input = file_path - if needs_scale and not self.inline_scale: + # v4.6.0: pre-scale ONLY on the av1an path. The pure-ffmpeg path + # (the default, and the only path NVENC can run on) scales inline + # via the -vf args in _ffmpeg_fallback_encode — the intermediate + # existed solely because VapourSynth source plugins choke on some + # inputs ffmpeg handles fine. Skipping it on the ffmpeg path + # removes the entire class of large-file failures: no 0.5-0.8x + # source-size temp file, no extra full encode pass, and no + # pre-scale timeout on long/high-bitrate sources. + if needs_scale and not self.inline_scale and not self.use_ffmpeg_fallback: try: temp_scaled = _temp_path_for(file_path, ".scaled_tmp.mkv", worker_dir=self._temp_dir) self._current_temps.append(temp_scaled) @@ -3613,33 +4294,57 @@ class EncoderWorker(QThread): "-c:v", "libx265", "-crf", "16", # visually lossless — was 0 (mathematically lossless) "-preset", "ultrafast", - "-pix_fmt", "yuv420p", # force 8-bit 4:2:0 + "-pix_fmt", "yuv420p", # force 8-bit 4:2:0 "-y", str(temp_scaled), ] - self.log_msg.emit(f" Scaling {src_w or '?'}x{src_h or '?'} -> {self.resolution.width}x{self.resolution.height}...") - scale_res = subprocess.run( - scale_cmd, capture_output=True, text=True, timeout=1800, + # v4.2.1: Scaling notice is verbose-only. + self._vlog(f" Scaling {src_w or '?'}x{src_h or '?'} -> {self.resolution.width}x{self.resolution.height}...") + # v4.6.0: run via _run_with_stop_check with the full + # per-file timeout. The old flat + # subprocess.run(timeout=1800) killed pre-scaling of + # long/high-bitrate sources at exactly 30 minutes + # ("FAIL: pre-scale error: Command ... timed out") — a + # guaranteed large-file failure that also ignored the + # STOP button for the whole intermediate pass. + scale_status, scale_rc, _s_out, scale_err = self._run_with_stop_check( + scale_cmd, timeout=self.encode_timeout, log_prefix=" ", ) - if scale_res.returncode == 0 and temp_scaled.exists(): + if scale_status == "stop": + # User aborted — clean up the partial intermediate and + # bail WITHOUT counting a failure (a STOP is not an + # encode failure; the queue loop breaks next iteration). + self._cleanup_current_temps() + return None + if scale_status == "timeout": + self.log_msg.emit( + f"{self._status_prefix()}FAIL: pre-scale timeout " + f"(exceeded {self.encode_timeout}s limit)" + ) + temp_scaled.unlink(missing_ok=True) + self._cleanup_current_temps() + self.fail_count += 1 + return None + if scale_status == "ok" and scale_rc == 0 and temp_scaled.exists(): encode_input = temp_scaled scaled_size = temp_scaled.stat().st_size / 1_048_576 - self.log_msg.emit(f" Pre-scale OK ({scaled_size:.1f} MB intermediate)") + # v4.2.1: verbose-only + self._vlog(f" Pre-scale OK ({scaled_size:.1f} MB intermediate)") else: - stderr_snip = (scale_res.stderr or "")[-200:] + stderr_snip = (scale_err or "")[-200:] + # v4.2.1: keep user-facing FAIL but shorten; stderr verbose-only self.log_msg.emit( - f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_res.returncode})" + f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_rc})" ) if stderr_snip.strip(): - self.log_msg.emit(f" ffmpeg stderr: {stderr_snip.strip()}") + self._vlog(f" ffmpeg stderr: {stderr_snip.strip()}") temp_scaled.unlink(missing_ok=True) self._cleanup_current_temps() self.fail_count += 1 return None except (OSError, subprocess.SubprocessError) as e: - self.log_msg.emit( - f"{self._status_prefix()}FAIL: pre-scale error: {e}" - ) + # v4.2.1: keep user-facing but shorten + self.log_msg.emit(f"{self._status_prefix()}FAIL: pre-scale error: {e}") self._cleanup_current_temps() self.fail_count += 1 return None @@ -3680,19 +4385,27 @@ class EncoderWorker(QThread): return (encode_input, output_f) def _check_disk_space(self, file_path: Path, output_f: Path, needs_scale: bool) -> None: - """v4.4.0: Warn (not abort) if free disk space is less than the source size. + """v4.4.0: Warn if free disk space is less than the encode will need. - v4.4.1: warnings gated behind --verbose. Quiet mode = zero output. + v4.6.0: SEVERE warnings (free space below the source size on the + partition we're about to write a big intermediate/output to) are + now USER-FACING — they were verbose-only, so in the default quiet + mode a batch that was going to die with "No space left on device" + partway through gave zero advance notice. That silent failure was + one of the "large files just fail" reports: small files fit in + the remaining space, big ones didn't. Marginal advice (the 2-3x + intermediate estimate) stays verbose-only. """ - if not self.verbose: - return # v4.4.1: quiet mode — no disk-space warnings try: src_size = file_path.stat().st_size except OSError: - return + return # can't stat source — skip the check if src_size < 1_073_741_824: # < 1 GB — skip check for small files return src_gb = src_size / 1_073_741_824 + # Check output partition — severe when free < source size + # (the encoded output is usually smaller, but the ffmpeg/av1an + # buffer cache plus a same-partition temp can eat the difference). try: out_usage = shutil.disk_usage(output_f.parent) out_free_gb = out_usage.free / 1_073_741_824 @@ -3701,15 +4414,31 @@ class EncoderWorker(QThread): f" WARN: low disk space on output ({out_free_gb:.1f} GB free, " f"source is {src_gb:.1f} GB) — encode may fail partway through" ) + elif self.verbose and out_free_gb < src_gb * 2: + self._vlog( + f" WARN: output space getting tight ({out_free_gb:.1f} GB free, " + f"source is {src_gb:.1f} GB)" + ) except OSError: - pass - if needs_scale: + pass # can't check — skip + # When scaling on the av1an path, also check the temp partition + # (the CRF-16 intermediate can be ~1x source size). The ffmpeg + # path scales inline (no intermediate), so no temp warning there. + if needs_scale and not self.use_ffmpeg_fallback: try: tmp_usage = shutil.disk_usage(self._temp_dir) tmp_free_gb = tmp_usage.free / 1_073_741_824 - if tmp_free_gb < src_gb * 2: + # Severe: free temp space below the source size means the + # intermediate will likely not fit → user-facing warning. + if tmp_free_gb < src_gb: self.log_msg.emit( f" WARN: low disk space on temp ({tmp_free_gb:.1f} GB free, " + f"source is {src_gb:.1f} GB) — the scale intermediate " + f"may not fit. Free space or enable 'Inline scale'." + ) + elif self.verbose and tmp_free_gb < src_gb * 2: + self._vlog( + f" WARN: temp space getting tight ({tmp_free_gb:.1f} GB free, " f"lossless intermediate may need ~{src_gb * 2:.1f} GB) — " f"consider scaling to a smaller resolution or freeing space" ) @@ -3923,10 +4652,20 @@ class EncoderWorker(QThread): return False else: stderr_full = res.stderr or "" + # v4.6.0: scan stdout too. av1an routes chunk-retry + # noise (encoder stderr dumps, FRAME MISMATCH lines) + # to stdout, so pattern-matching on stderr alone missed + # the biggest real-world failure mode (see the FRAME + # MISMATCH pattern below). + combined_out = stderr_full + "\n" + (res.stdout or "") # v6: Don't increment fail_count yet — we may retry with # ffmpeg fallback below. Only increment if the retry also # fails (or no retry is possible). - # v4.4.2: move FAIL to _vlog. User only sees final outcome. + # v4.4.2: move the FAIL line to _vlog. If the ffmpeg + # fallback succeeds, the user sees OK. If it also fails, + # the RETRY FAIL path emits a user-facing FAIL. This way + # the user doesn't see a confusing "FAIL then OK" for + # files that av1an choked on but ffmpeg handled. self._vlog( f"{self._status_prefix()}av1an failed (exit code {res.returncode}) — attempting ffmpeg fallback" ) @@ -4054,10 +4793,56 @@ class EncoderWorker(QThread): ), False, # don't stop queue — retry with select chunk method ), + # v4.6.0: ffmpeg ≥ 7 removed the -vsync option that + # av1an's segment/hybrid chunk extraction passes to + # ffmpeg. Every segment-based chunk dies immediately + # ("Unrecognized option 'vsync'." → empty y4m pipe → + # chunk fails 3x). Verified on ffmpeg 9.0.2 + av1an + # 0.5.2. select (or a VS source plugin method) is the + # only working chunking on these systems. + ( + "Unrecognized option 'vsync'", + "av1an's segment/hybrid chunk extraction calls " + "`ffmpeg -vsync`, which ffmpeg 7+ removed. Every " + "segment-based chunk fails instantly on this system.", + ( + " FIX: install a VapourSynth source plugin so av1an", + " stops using ffmpeg segmenting: bestsource/ffms2/", + " lsmash (e.g. on Arch: vapoursynth-plugin-bs).", + " Alternatively stay on the default ffmpeg-only path", + " (it doesn't use av1an chunking at all).", + ), + False, # per-file — the select override keeps other files working + ), + # v4.6.0: frame-count drift between av1an's chunk + # manifest and what the encoder actually produced. + # av1an retries the chunk 3x, then shuts the worker + # down with the baffling "encoder crashed: exit + # status: 0" (exit 0 because the encode itself + # succeeded — on partial data). This was the dominant + # large-file failure in the 2026-07-13 av1an log and + # previously fell through to "Unknown av1an failure". + ( + "FRAME MISMATCH", + "av1an's chunk manifest expected a different frame count " + "than the encoder produced — chunk-extraction drift on " + "sources with sparse/irregular keyframes. The encode " + "itself exits 0 (it ran on partial data), which av1an " + "reports as 'encoder crashed: exit status: 0'.", + ( + " Will retry with --chunk-method select, which", + " extracts exact frame ranges and cannot drift.", + " This is per-file, not systematic — subsequent", + " files use select automatically.", + ), + False, # don't stop queue — retry with select chunk method + ), ) diagnosis_emitted = False for marker, summary, fixes, stop_queue in error_patterns: - if marker.lower() in stderr_full.lower(): + # v4.6.0: scan stdout + stderr (FRAME MISMATCH and + # encoder dumps land in stdout). + if marker.lower() in combined_out.lower(): self.log_msg.emit("") self.log_msg.emit(f"DIAGNOSIS: {summary}") for fix in fixes: @@ -4106,7 +4891,8 @@ class EncoderWorker(QThread): # as concat failures. if ("SUMMARY" in stderr_full and "Average Speed" in stderr_full - and "Failed to read y4m frame delimiter" not in stderr_full): + and "Failed to read y4m frame delimiter" not in combined_out + and "FRAME MISMATCH" not in combined_out): self.log_msg.emit("") self.log_msg.emit( "DIAGNOSIS: SVT-AV1 encoder completed successfully (SUMMARY" @@ -4143,18 +4929,34 @@ class EncoderWorker(QThread): # same params). Cache the working method so subsequent files # skip the wasted first attempt. # + # v4.6.0: FRAME MISMATCH (chunk-extraction drift, see the + # error-pattern table) joins the y4m break as a drift + # symptom that select fixes — it was previously an + # "Unknown av1an failure" that went straight to the slow + # full-file ffmpeg fallback. + # # NOTE: Do NOT clean up _current_temps before the retry — # encode_input (symlink or pre-scaled file) is in # _current_temps and the recursive _encode_one call needs it. # The finally block below will clean up everything after the # recursive call returns (its own finally clears the list # first; our finally then runs on an empty list — no-op). - y4m_break = "Failed to read y4m frame delimiter" in stderr_full - if (not self._stop and y4m_break + extraction_drift = ( + "Failed to read y4m frame delimiter" in combined_out + or "FRAME MISMATCH" in combined_out + # ffmpeg >= 7 removed -vsync: segment/hybrid chunking + # dies instantly while select still works (it uses the + # ffmpeg frame server, not segmenting). + or "Unrecognized option 'vsync'" in combined_out + ) + if (not self._stop and extraction_drift and effective_chunk_method != "select" and self.env.av1an_flags.get("has_chunk_method", True)): - self.log_msg.emit("") - self.log_msg.emit( + # v4.2.1: RETRY messages are verbose-only — the user + # already saw "FAIL" and will see "SUCCESS" if the retry + # works. They don't need to know the retry is happening. + self._vlog("") + self._vlog( f" RETRY: Re-encoding {file_path.name} with " f"--chunk-method select (slower but reliable for " f"files with sparse keyframes)..." @@ -4389,9 +5191,15 @@ class EncoderWorker(QThread): try: # Pass 1: analyze current loudness + # v4.6.0: -vn skips video decoding — without it the analysis + # decoded the ENTIRE video stream just to measure audio + # loudness, which pushed long/large files past the 120s + # timeout and silently degraded every big file to the static + # knob gain. analysis_cmd = [ self.env.ffmpeg_path, "-i", str(file_path), + "-vn", "-af", ( f"loudnorm=I={target_lufs}:TP=-1.5:LRA=11:" f"print_format=json" @@ -4531,6 +5339,346 @@ class EncoderWorker(QThread): # SOURCE BUILD WORKER — compile VS + av1an from git # ────────────────────────────────────────────── +# ────────────────────────────────────────────── +# BUILD DEPENDENCY TREE (v4.7.1 — distro-aware) +# ────────────────────────────────────────────── + +# Binaries the build needs. pkgconf/pkg-config and python/python3 are +# aliased — any one of each pair satisfies the check. +BUILD_TOOL_ALIASES: dict[str, tuple[str, ...]] = { + "git": ("git",), + "meson": ("meson",), + "ninja": ("ninja",), + "c++ compiler": ("g++", "c++", "clang++"), + "make": ("make",), + "pkg-config": ("pkg-config", "pkgconf"), + "python3": ("python3",), + "nasm": ("nasm",), + "cmake": ("cmake",), +} + +# Packages providing the toolchain + the libraries the builds link +# against (zimg is VapourSynth's one hard library dependency; rust is +# only needed for the av1an build). +BUILD_DEPS_BY_FAMILY: dict[str, list[str]] = { + "arch": ["base-devel", "meson", "ninja", "cmake", "nasm", "git", + "python", "pkgconf", "zimg", "rust"], + "debian": ["build-essential", "meson", "ninja-build", "cmake", "nasm", + "git", "python3", "python3-dev", "pkg-config", "libzimg-dev", + "cargo", "rustc"], + "redhat": ["gcc", "gcc-c++", "make", "meson", "ninja-build", "cmake", + "nasm", "git", "python3", "python3-devel", + "pkgconf-pkg-config", "zimg-devel", "cargo", "rust"], + "suse": ["gcc", "gcc-c++", "make", "meson", "ninja", "cmake", "nasm", + "git", "python3", "python3-devel", "pkg-config", + "zimg-devel", "rust", "cargo"], +} + +PKG_INSTALL_CMD: dict[str, list[str]] = { + "arch": ["pacman", "-S", "--needed", "--noconfirm"], + "debian": ["apt-get", "install", "-y"], + "redhat": ["dnf", "install", "-y"], + "suse": ["zypper", "--non-interactive", "install"], +} + +MANUAL_DEP_NOTE = ( + "No automatic package install for this distro family. Install a C++ " + "toolchain plus meson, ninja, cmake, nasm, git, python3, pkg-config, " + "zimg development headers{rust} manually, then press REBUILD again." +) + +GPU_BUILD_PACKAGES: dict[str, dict[str, list[str]]] = { + "nvenc": { + "arch": ["nv-codec-headers"], + "debian": [], + "redhat": [], + "suse": [], + }, + "vaapi": { + "arch": ["libva", "libdrm", "mesa"], + "debian": ["libva-dev", "libdrm-dev", "mesa-va-drivers"], + "redhat": ["libva-devel", "libdrm-devel", "mesa-va-drivers"], + "suse": ["libva-devel", "libdrm-devel", "Mesa-libva"], + }, + "qsv": { + "arch": ["libva", "intel-media-driver", "onevpl"], + "debian": ["libva-dev", "intel-media-va-driver-non-free", "libvpl-dev"], + "redhat": ["libva-devel", "intel-media-driver", "oneVPL-devel"], + "suse": ["libva-devel", "intel-media-driver", "oneVPL-devel"], + }, +} + + + + + + + +@dataclass +class DepPlan: + """What the rebuild needs, and how to get it on this distro.""" + packages: list[str] = field(default_factory=list) + install_cmd: list[str] | None = None + manual_note: str | None = None + + +def build_dep_plan(distro: DistroProfile, build_av1an: bool = True) -> DepPlan: + """Pure: the package list + install command for this distro family. + + Works even when the environment probe failed — it only needs the + distro family, which is detectable from /etc/os-release alone. + """ + packages = list(BUILD_DEPS_BY_FAMILY.get(distro.family, [])) + if not build_av1an: + for rust_pkg in ("rust", "rustc", "cargo"): + if rust_pkg in packages: + packages.remove(rust_pkg) + install_cmd = PKG_INSTALL_CMD.get(distro.family) + manual_note = None + if not install_cmd or not packages: + manual_note = MANUAL_DEP_NOTE.format( + rust=" and Rust/cargo" if build_av1an else "") + return DepPlan(packages=packages, install_cmd=install_cmd, + manual_note=manual_note) + +"""GPU capability profiles (v4.8.0) — combined card generations. + +Cards within the same hardware-encoder generation are functionally +identical for transcoding, so the dropdown lists CAPABILITY CLASSES, +not individual SKUs: one Pascal entry covers the GTX 10-series, Tesla +P40/P4/P100 and mobile chips; one Turing entry covers RTX 20-series, +GTX 16-series, the Tesla T4 and the crypto-era CMP 30/40/50HX cards. + +Oddballs are included with their real capabilities: + - CMP 90HX is GA102-based (Ampere NVENC), but CMP 170HX is GA100-based + and has NO NVENC at all (like A100/V100/H100 compute boards). + - Intel Arc (QSV) and AMD RDNA (VAAPI) cover the rest of the trending + list; RDNA 3 added AV1 encode, RDNA 1/2 and GCN can only encode + H.264/HEVC. + +Pure data + pure functions: no I/O, safe to import anywhere. +""" + + +import re +from dataclasses import dataclass + +# ────────────────────────────────────────────── +# GPU PROFILES +# ────────────────────────────────────────────── + +@dataclass(frozen=True) +class GpuProfile: + key: str # stable id (CLI/UI) + label: str # dropdown entry + vendor: str # nvidia | amd | intel | none + api: str # nvenc | vaapi | qsv | none + # codec family → ffmpeg encoder name + encoders: dict[str, str] + # case-insensitive substrings matched against the detected GPU name + # (nvidia-smi / lspci) for auto-detection. First match wins; lists + # are ordered most-specific first. + match: tuple[str, ...] = () + notes: str = "" + # extra args that must come BEFORE -i (hardware device init) + hw_device_args: tuple[str, ...] = () + # filter-chain fragment required before the encoder (vaapi hwupload) + filter_tail: tuple[str, ...] = () + + +# NVENC encoders by generation class. Quality control: -rc vbr -cq N +# (-b:v 0). 8-bit yuv420p everywhere — Pascal 10-bit HEVC runs at half +# speed and the archival targets here are 8-bit sources. +_NV = {"h264": "h264_nvenc", "hevc": "hevc_nvenc"} + +GPU_PROFILES: list[GpuProfile] = [ + GpuProfile( + key="nv-kepler-maxwell", + label="NVIDIA Kepler / Maxwell 1.0 (GTX 600/700/800M) — H.264 only", + vendor="nvidia", api="nvenc", + encoders={"h264": "h264_nvenc"}, + match=("GTX 6", "GTX 7", "GT 7", "GTX 8", "GT 8", "840M", "860M", "750"), + notes="First NVENC generations: H.264 only, no HEVC.", + ), + GpuProfile( + key="nv-pascal", + label="NVIDIA Pascal (GTX 10-series, TITAN Xp, Tesla P40/P4/P100) — H.264 + HEVC 8/10-bit", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("GTX 10", "1070", "1080", "1060", "1050", "TITAN Xp", + "Tesla P40", "Tesla P4", "P100", "Quadro P"), + notes="Pascal NVENC: HEVC Main/Main10. 10-bit runs at ~half speed.", + ), + GpuProfile( + key="nv-turing", + label="NVIDIA Turing (RTX 20-series, GTX 16-series, Tesla T4, CMP 30/40/50HX) — H.264 + HEVC + B-frames", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("RTX 20", "GTX 16", "2060", "2070", "2080", "1660", "1650", + "Tesla T4", "CMP 30", "CMP 40", "CMP 50"), + notes="Turing NVENC: first gen with HEVC B-frames; big quality jump.", + ), + GpuProfile( + key="nv-ampere", + label="NVIDIA Ampere (RTX 30-series, A10/A40/A2, CMP 90HX) — H.264 + HEVC (no AV1 encode)", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("RTX 30", "3090", "3080", "3070", "3060", "3050", + "A10 ", "A40", "A2 ", "CMP 90"), + notes="Ampere added AV1 DECODE but not encode — AV1 stays on CPU.", + ), + GpuProfile( + key="nv-ada", + label="NVIDIA Ada / Blackwell (RTX 40/50-series, L4/L40) — H.264 + HEVC + AV1 10-bit", + vendor="nvidia", api="nvenc", + encoders={"h264": "h264_nvenc", "hevc": "hevc_nvenc", "av1": "av1_nvenc"}, + match=("RTX 40", "RTX 50", "4090", "4080", "4070", "4060", + "5090", "5080", "5070", "5060", "L4", "L40"), + notes="Ada introduced AV1 NVENC; Blackwell doubles AV1 throughput.", + ), + GpuProfile( + key="nv-compute", + label="NVIDIA data-center compute (V100/A100/H100, CMP 170HX) — no NVENC (CPU path)", + vendor="nvidia", api="none", + encoders={}, + match=("V100", "A100", "H100", "B200", "GB200", "CMP 170"), + notes="Compute boards ship without NVENC silicon. CMP 170HX is " + "GA100-based — the fastest mining card that cannot hardware-encode.", + ), + GpuProfile( + key="intel-arc", + label="Intel Arc (Alchemist A-series, Battlemage B-series) — QSV: H.264 + HEVC + AV1", + vendor="intel", api="qsv", + encoders={"h264": "h264_qsv", "hevc": "hevc_qsv", "av1": "av1_qsv"}, + match=("Arc A", "Arc B", "A380", "A750", "A770", "B570", "B580"), + notes="Arc media engines encode AV1 8/10-bit — best value encode card.", + ), + GpuProfile( + key="intel-xe", + label="Intel Iris / UHD integrated (Gen9–Xe) — QSV: H.264 + HEVC", + vendor="intel", api="qsv", + encoders={"h264": "h264_qsv", "hevc": "hevc_qsv"}, + match=("Iris", "UHD", "HD Graphics"), + notes="Integrated media engines; HEVC 8/10-bit, no AV1 encode.", + ), + GpuProfile( + key="amd-rdna3", + label="AMD RDNA 3 (RX 7000-series) — VAAPI: H.264 + HEVC + AV1", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi", "av1": "av1_vaapi"}, + match=("RX 7", "7900", "7800", "7700", "7600"), + notes="RDNA 3 VCN: first AMD generation with AV1 encode.", + ), + GpuProfile( + key="amd-rdna12", + label="AMD RDNA 1/2 (RX 5000/6000-series) — VAAPI: H.264 + HEVC (AV1 decode only)", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"}, + match=("RX 5", "RX 6", "5700", "5600", "6800", "6700", "6600", "6500"), + notes="RDNA 2 has AV1 decode only — AV1 encode stays on CPU.", + ), + GpuProfile( + key="amd-gcn", + label="AMD GCN 4/5 / Vega (RX 400/500, Vega 56/64) — VAAPI: H.264 + HEVC", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"}, + match=("RX 4", "RX 5", "Vega", "580", "570", "480", "470", "64", "56"), + notes="The classic crypto-era mining cards (Polaris/Vega).", + ), + GpuProfile( + key="cpu", + label="None (CPU-only encode)", + vendor="none", api="none", + encoders={}, + ), +] + +GPU_PROFILES_BY_KEY: dict[str, GpuProfile] = {p.key: p for p in GPU_PROFILES} + + +def gpu_profile_by_key(key: str | None) -> GpuProfile | None: + if not key: + return None + return GPU_PROFILES_BY_KEY.get(key) + + +def match_gpu_profile(gpu_name: str) -> GpuProfile | None: + """Best-effort auto-detection from a GPU name string (nvidia-smi or + lspci output). Case-insensitive; first matching profile wins (the + match lists are ordered most-specific first, and the compute boards + are matched before the consumer generations they share names with — + e.g. 'CMP 170HX' must not hit the Ampere 'A10' style entries).""" + if not gpu_name: + return None + name = gpu_name.lower() + for profile in GPU_PROFILES: + for frag in profile.match: + if frag.lower() in name: + return profile + return None + + +# ────────────────────────────────────────────── +# FFMPEG ARG HELPERS (per hardware API) +# ────────────────────────────────────────────── + +def encoder_for_family(profile: GpuProfile | None, family: str) -> str | None: + """Hardware encoder name for a codec family on this profile, or None.""" + if not profile or profile.api == "none": + return None + return profile.encoders.get(family) + + +def resolve_vaapi_device() -> str: + """First render node, or the classic fallback path. (Best-effort I/O — + callers that need purity pass the result into encoder_pre_args.)""" + import glob + nodes = sorted(glob.glob("/dev/dri/renderD*")) + return nodes[0] if nodes else "/dev/dri/renderD128" + + +def encoder_pre_args(profile: GpuProfile, vaapi_device: str | None = None) -> list[str]: + """Args that must precede -i (hardware device initialisation).""" + if profile.api == "vaapi": + dev = vaapi_device or resolve_vaapi_device() + return ["-vaapi_device", dev] + if profile.api == "qsv": + return ["-init_hw_device", "qsv=hw"] + return [] + + +def encoder_filter_chain(profile: GpuProfile) -> list[str]: + """Filter args that upload software frames to the hardware surface + format (VAAPI encoders only accept hw frames; -vaapi_device makes + its device the default for hwupload).""" + if profile.api == "vaapi": + return ["-vf", "format=nv12,hwupload"] + return [] + + +def encoder_quality_args(api: str, encoder: str, crf: int, preset: int) -> list[str]: + """Constant-quality args for a hardware encoder. NVENC maps the CPU + preset tiers to p-presets; QSV uses very_fast/medium; VAAPI uses CQP + rate mode which has no preset knob.""" + if api == "nvenc": + if preset <= 6: + p = "p7" + elif preset <= 8: + p = "p5" + else: + p = "p4" + return ["-preset", p, "-tune", "hq", "-rc", "vbr", + "-cq", str(crf), "-b:v", "0", "-pix_fmt", "yuv420p", + "-g", "240"] + if api == "qsv": + p = "veryslow" if preset <= 6 else ("medium" if preset <= 8 else "very_fast") + return ["-preset", p, "-global_quality", str(crf), + "-pix_fmt", "yuv420p", "-g", "240"] + if api == "vaapi": + return ["-rc_mode", "CQP", "-qp", str(crf), "-g", "240"] + return [] + + class SourceBuildWorker(QThread): """Builds VapourSynth and/or av1an from git to resolve ABI mismatches. @@ -4546,11 +5694,14 @@ class SourceBuildWorker(QThread): build_done = Signal(bool, str) # (success, detail) def __init__(self, build_vs: bool = True, build_av1an: bool = True, - build_ffmpeg_iamf: bool = False): + build_ffmpeg_iamf: bool = False, gpu_profile_key: str = ""): super().__init__() self.build_vs = build_vs self.build_av1an = build_av1an self.build_ffmpeg_iamf = build_ffmpeg_iamf + # v4.8.0: selected GPU capability profile — extends the dep tree + # with the vendor's build/runtime packages. + self.gpu_profile_key = gpu_profile_key self._stop = False # Private per-worker environment snapshot. Mutating os.environ is # process-global and leaks across threads/subsequent subprocesses; @@ -4607,30 +5758,71 @@ class SourceBuildWorker(QThread): def run(self): try: - # ── Install build dependencies (may need one sudo prompt) ── + # ── v4.7.1: distro-aware dependency tree ── + # The rebuild must work on a bare system: detect missing + # tools/libraries and install them via the distro package + # manager (one privilege prompt via pkexec/sudo) BEFORE + # compiling anything. self.log_msg.emit("") - self.log_msg.emit("=== Installing build dependencies ===") - all_deps = [ - "meson", "ninja", "gcc", "pkg-config", "git", - "nasm", "yasm", "cmake", "python", "make", - ] - need_rust = self.build_av1an and not shutil.which("cargo") - if need_rust: - all_deps.append("rust") + self.log_msg.emit("=== Generating dependency tree ===") + self._distro = detect_distro() + self.log_msg.emit( + f" Distro: {self._distro.name} (family={self._distro.family})" + ) + plan = build_dep_plan(self._distro, build_av1an=self.build_av1an) + + # v4.8.0: GPU-profile packages on top of the base toolchain. + gpu_profile = gpu_profile_by_key(self.gpu_profile_key) + if gpu_profile is not None and gpu_profile.api != "none": + gpu_pkgs = gpu_dep_packages(gpu_profile.api, self._distro.family) + if gpu_pkgs: + self.log_msg.emit( + f" GPU profile {gpu_profile.key} ({gpu_profile.api}): " + f"+{len(gpu_pkgs)} package(s)" + ) + plan.packages.extend(p for p in gpu_pkgs + if p not in plan.packages) + + missing = self._missing_build_tools() + zimg_ok = self._pkgconfig_exists("zimg") + if zimg_ok: + self.log_msg.emit(" OK: zimg (VapourSynth dependency)") + else: + missing.append("zimg (library, via pkg-config)") + if self.build_av1an and not shutil.which("cargo"): + missing.append("cargo (rust)") - # Only invoke sudo if at least one dep is missing - missing = [d for d in all_deps if not shutil.which(d)] if missing: - self.log_msg.emit(f" Missing: {', '.join(missing)} — installing via pacman") - rc, _ = self._sudo_cmd( - ["pacman", "-S", "--needed", "--noconfirm"] + all_deps, - timeout=300, label="pacman build-deps", - ) - if rc != 0: - self.log_msg.emit(" (some deps may already be installed — continuing)") + self.log_msg.emit(f" Missing: {', '.join(missing)}") + if plan.install_cmd: + self.log_msg.emit( + f" Installing {len(plan.packages)} package(s) via " + f"{plan.install_cmd[0]} (privilege prompt possible)..." + ) + rc, _ = self._sudo_cmd( + plan.install_cmd + plan.packages, + timeout=900, label=f"{plan.install_cmd[0]} build-deps", + ) + if rc != 0: + self.log_msg.emit( + " (install reported an error — continuing; " + "some packages may already be present)" + ) + else: + self.log_msg.emit(f" {plan.manual_note}") else: self.log_msg.emit(" All build dependencies already installed.") + # Re-verify the critical tools after install. + still_missing = self._missing_build_tools() + if still_missing: + self.log_msg.emit( + f" FATAL: still missing after install: {', '.join(still_missing)}. " + f"Install them manually and press REBUILD again." + ) + self.build_done.emit(False, f"missing build tools: {still_missing}") + return + # Ensure cargo is in PATH after potential install. # NOTE: /root/.cargo/bin was dropped (OTC-015/v3-08) — root's # cargo dir is not readable by a non-root user. ~/.cargo/bin @@ -4651,6 +5843,9 @@ class SourceBuildWorker(QThread): # ── Build & install VapourSynth to ~/.local (NO sudo needed) ── if self.build_vs: self._build_vapoursynth() + # v4.7.1: BestSource right after VS, compiled against the + # fresh VS headers — gives av1an a fast chunk method. + self._build_bestsource() # ── Build av1an to ~/.cargo/bin (NO sudo needed) ── if self.build_av1an: @@ -4661,12 +5856,25 @@ class SourceBuildWorker(QThread): self._build_libiamf() self._build_ffmpeg_with_iamf() - # ── Ensure LD_LIBRARY_PATH includes local VS libs ── + # ── Ensure the runtime env can find the fresh VS stack ── + # The git VapourSynth installs self-contained into the user + # site-packages (module + libs + plugins). av1an dlopens + # libvapoursynth-script from there, so both LD_LIBRARY_PATH + # and PYTHONPATH must include it. local_lib = str(Path.home() / ".local" / "lib") existing_ld = self._build_env.get("LD_LIBRARY_PATH", "") if local_lib not in existing_ld: self._extend_env("LD_LIBRARY_PATH", local_lib, prepend=True) - self.log_msg.emit(f" Set LD_LIBRARY_PATH to include {local_lib}") + user_site = self._vs_user_site() + if user_site and (user_site / "vapoursynth" / "libvsscript.so").exists(): + vs_dir = str(user_site / "vapoursynth") + if vs_dir not in self._build_env.get("LD_LIBRARY_PATH", ""): + self._extend_env("LD_LIBRARY_PATH", vs_dir, prepend=True) + if str(user_site) not in self._build_env.get("PYTHONPATH", ""): + self._extend_env("PYTHONPATH", str(user_site), prepend=True) + self.log_msg.emit( + f" Runtime env: LD_LIBRARY_PATH/PYTHONPATH include {vs_dir}" + ) self.log_msg.emit("") self.log_msg.emit("=== Source build complete ===") @@ -4683,6 +5891,127 @@ class SourceBuildWorker(QThread): self.log_msg.emit(f"BUILD FAILED: {e}") self.build_done.emit(False, str(e)) + def _missing_build_tools(self) -> list[str]: + """Binaries from BUILD_TOOL_ALIASES that are not on PATH.""" + missing = [] + for label, candidates in BUILD_TOOL_ALIASES.items(): + if not any(shutil.which(c) for c in candidates): + missing.append(label) + return missing + + def _pkgconfig_exists(self, name: str) -> bool: + rc, _ = self._run_cmd( + ["pkg-config", "--exists", name], + timeout=10, label=f"pkg-config {name}", + ) + return rc == 0 + + def _vs_user_site(self) -> Path | None: + """The user site-packages dir of the system python3 — where the + VapourSynth git install places its self-contained stack (module, + libs, headers, plugins/).""" + rc, out = self._run_cmd( + ["python3", "-m", "site", "--user-site"], + timeout=15, label="python3 -m site --user-site", + ) + if rc == 0 and out.strip(): + return Path(out.strip().splitlines()[-1]) + return None + + def _build_bestsource(self): + """Clone and build the BestSource VapourSynth plugin from git. + + BestSource gives av1an a fast, frame-accurate chunk source — the + difference between 'select' (quadratic decoding, minutes per + file) and normal chunk-parallel speed. Compiled against the + vapoursynth headers of the JUST-INSTALLED git VS (via + PYTHONPATH/PKG_CONFIG_PATH), so the plugin ABI always matches + the VS that av1an will load. Requires the repo's libp2p + submodule (initialized here). + """ + self.log_msg.emit("") + self.log_msg.emit("=== Building BestSource plugin from git ===") + self.log_msg.emit(" Source: https://github.com/vapoursynth/bestsource") + + build_dir = Path("/tmp/bestsource-git-build") + if build_dir.exists(): + shutil.rmtree(build_dir, ignore_errors=True) + + rc, out = self._run_cmd( + ["git", "clone", "--depth", "1", + "https://github.com/vapoursynth/bestsource.git", + str(build_dir)], + timeout=120, label="git clone bestsource", + ) + if rc != 0: + raise Exception(f"git clone bestsource failed: {out[-300:]}") + + # libp2p is a required submodule (R9+ builds source from it). + rc, out = self._run_cmd( + ["git", "submodule", "update", "--init", "--depth", "1"], + cwd=str(build_dir), timeout=120, label="git submodule update", + ) + if rc != 0: + raise Exception(f"bestsource submodule init failed: {out[-300:]}") + + # Point meson/pkg-config at the freshly built VS stack. + user_site = self._vs_user_site() + if user_site and (user_site / "vapoursynth").is_dir(): + self._extend_env("PYTHONPATH", str(user_site), prepend=True) + self._extend_env("PKG_CONFIG_PATH", + str(user_site / "vapoursynth" / "pkgconfig"), + prepend=True) + self._extend_env("LD_LIBRARY_PATH", + str(user_site / "vapoursynth"), prepend=True) + else: + self.log_msg.emit( + " NOTE: git VapourSynth install not found in user " + "site-packages — building against system vapoursynth." + ) + + self.log_msg.emit(" Configuring with meson (--prefix=~/.local)...") + rc, out = self._run_cmd( + ["meson", "setup", "build", + f"--prefix={Path.home() / '.local'}", "--libdir=lib"], + cwd=str(build_dir), timeout=180, label="meson setup bestsource", + ) + if rc != 0: + raise Exception(f"bestsource meson setup failed: {out[-500:]}") + + self.log_msg.emit(" Compiling BestSource (a minute or two)...") + rc, out = self._run_cmd( + ["ninja", "-C", "build", "-j", str(max(1, os.cpu_count() or 2))], + cwd=str(build_dir), timeout=600, label="ninja bestsource", + ) + if rc != 0: + raise Exception(f"bestsource build failed: {out[-500:]}") + + rc, out = self._run_cmd( + ["ninja", "-C", "build", "install"], + cwd=str(build_dir), timeout=120, label="ninja install bestsource", + ) + if rc != 0: + raise Exception(f"bestsource install failed: {out[-500:]}") + + plugin = None + if user_site: + candidate = user_site / "vapoursynth" / "plugins" / "libbestsource.so" + if candidate.exists(): + plugin = candidate + if plugin: + self.log_msg.emit(f" BestSource plugin installed: {plugin}") + self.log_msg.emit( + " av1an will now auto-select the fast 'bestsource' chunk " + "method (restart the app so the probe sees it)." + ) + else: + self.log_msg.emit( + " WARNING: libbestsource.so not found at the expected " + "user-site path — check the meson install log above." + ) + + shutil.rmtree(build_dir, ignore_errors=True) + def _build_vapoursynth(self): """Clone, build, and install VapourSynth to ~/.local/ (no sudo needed).""" self.log_msg.emit("") @@ -5374,7 +6703,13 @@ class OpenCodecMaster(QMainWindow): self.setWindowTitle("OpenTranscode — dcos.net") self.resize(1100, 920) self.worker: EncoderWorker | None = None - self.env: EnvProbe | None = None + # v4.7.0: hybrid (GPU + CPU lanes) bookkeeping. *workers* holds + # every active lane; _hybrid_pending/_hybrid_totals aggregate the + # per-lane finished_queue signals into one summary. + self.workers: list[EncoderWorker] = [] + self._hybrid_pending = 0 + self._hybrid_totals = [0, 0] + self._pending_deletes: list[Path] = [] self._apply_mmd3_theme() @@ -5589,6 +6924,88 @@ class OpenCodecMaster(QMainWindow): "which is more reliable across distros." ) opt_row.addWidget(self.av1an_check) + + # v4.6.0: ENGINE selector — Auto (GPU if available) / GPU / CPU. + # Auto uses the NVENC hardware encoder for the selected codec + # family when the environment probe's live encode test proved it + # works (av1 → av1_nvenc on RTX 40+, hevc → hevc_nvenc on every + # NVENC generation); otherwise it stays on the CPU encoders. GPU + # mode runs through the single-pass ffmpeg path — av1an cannot + # drive NVENC, and one NVENC process outruns chunk-parallel CPU + # workers anyway. + self.engine_combo = QComboBox() + self.engine_combo.addItems([ + "Engine: Auto (GPU if available)", + "Engine: GPU (NVENC)", + "Engine: CPU", + "Engine: Hybrid (GPU + CPU)", + ]) + self.engine_combo.setToolTip( + "Video encode engine.\n\n" + "Auto — use the NVENC GPU encoder when the selected codec\n" + " family has one AND a live encode test proved it works on\n" + " this system; fall back to CPU otherwise (default).\n" + "GPU — force NVENC (hevc_nvenc / av1_nvenc); falls back to\n" + " CPU with a log message when unavailable. GPU encodes run\n" + " via single-pass ffmpeg (av1an chunking is not used).\n" + "CPU — force the software encoders (SVT-AV1 / VP9 / x265).\n" + "Hybrid — run BOTH at once: the queue is split between a\n" + " GPU lane and a CPU lane (balanced by file size), so the\n" + " CPU cores are not idle while NVENC encodes. With av1an\n" + " enabled the CPU lane uses chunk-parallel too. GPU-lane\n" + " files get NVENC quality; CPU-lane files get software-\n" + " encoder quality. Needs 2+ encodable files.\n" + ) + self.engine_combo.setFixedHeight(24) + # v4.7.1: self.env is None (or absent) until _probe_and_init + # runs AFTER _build_ui — the pre-select must not touch it here. + _engine_flags = getattr(getattr(self, "env", None), "av1an_flags", None) or {} + _engine_pref = _engine_flags.get("engine") + _engine_index = {"auto": 0, "gpu": 1, "cpu": 2, "hybrid": 3} + self.engine_combo.setCurrentIndex(_engine_index.get(_engine_pref, 0)) + # v4.8.0: GPU capability-profile dropdown. Entries combine whole + # card generations (same silicon = same encoding behaviour) and + # include data-center + crypto-era oddballs. "Auto-detect" uses + # the probe's name match; a specific profile forces the API even + # when auto-match fails. The rebuild-from-git dep tree extends + # with the selected profile's packages. + self.gpu_combo = QComboBox() + self.gpu_combo.addItem("GPU: Auto-detect") + for _gp in GPU_PROFILES: + self.gpu_combo.addItem(f"GPU: {_gp.label}") + self.gpu_combo.setToolTip( + "Hardware encoder capability class.\n" + "Auto-detect matches your card via nvidia-smi/lspci and the\n" + "live encode probe decides what actually works. Forcing a\n" + "profile also extends the REBUILD FROM GIT dependency tree\n" + "with that GPU's packages (nv-codec-headers / VAAPI / QSV)." + ) + self.gpu_combo.setFixedHeight(24) + _gpu_pref = _engine_flags.get("gpu_profile") + _gpu_keys = ["auto"] + [_gp.key for _gp in GPU_PROFILES] + self.gpu_combo.setCurrentIndex( + _gpu_keys.index(_gpu_pref) if _gpu_pref in _gpu_keys else 0 + ) + self.gpu_combo.currentIndexChanged.connect(self._on_gpu_profile_changed) + # v4.8.1: ENGINE = CPU makes the GPU choice inert — grey it out. + # The "None (CPU-only encode)" entry stays available for users + # who have a GPU in the box but don't want encoding on it. + self.engine_combo.currentIndexChanged.connect(self._on_engine_changed) + self._update_gpu_combo_state() + _gpu_lay = QVBoxLayout() + _gpu_lay.setSpacing(1) + _gpu_lbl = QLabel("GPU") + _gpu_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + _gpu_lay.addWidget(_gpu_lbl) + _gpu_lay.addWidget(self.gpu_combo) + opt_row.addLayout(_gpu_lay) + _engine_lay = QVBoxLayout() + _engine_lay.setSpacing(1) + _engine_lbl = QLabel("ENGINE") + _engine_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + _engine_lay.addWidget(_engine_lbl) + _engine_lay.addWidget(self.engine_combo) + opt_row.addLayout(_engine_lay) root.addLayout(opt_row) # ── Log + Knobs: horizontal split ── @@ -5637,12 +7054,18 @@ class OpenCodecMaster(QMainWindow): self.btn_rebuild.setObjectName("btnRebuild") self.btn_rebuild.setFixedHeight(40) self.btn_rebuild.setToolTip( - "Compile VapourSynth + av1an from git source.\n" + "Compile VapourSynth + av1an + BestSource from git source.\n" "Resolves ABI/version mismatch when package managers\n" - "install incompatible versions." + "install incompatible versions. ALWAYS available — even on a\n" + "bare system: it first generates its own dependency tree\n" + "(installs missing build tools + zimg via the distro package\n" + "manager, one privilege prompt) and then builds everything\n" + "into ~/.local / ~/.cargo (no system changes)." ) self.btn_rebuild.clicked.connect(self._manual_rebuild) - self.btn_rebuild.setEnabled(False) + # v4.7.1: ALWAYS usable — the build generates its own dependency + # tree, so it must not depend on a successful probe (a failed + # probe is exactly when you need it). btn_lay.addWidget(self.btn_rebuild) self.btn_about = QPushButton(" ? ABOUT / LICENSES") @@ -6296,7 +7719,28 @@ class OpenCodecMaster(QMainWindow): selected_codec = VIDEO_CODECS[codec_idx] self._log(f"Codec: {selected_codec.label} (av1an encoder: {selected_codec.av1an_encoder})") - self.worker = EncoderWorker( + # v4.6.0: read the ENGINE selector and surface the resolved engine + # before the queue starts. The worker re-resolves against the same + # probe data; this line tells the user what is about to happen. + engine_sel = ("auto", "gpu", "cpu", "hybrid")[self.engine_combo.currentIndex()] \ + + if hasattr(self, "engine_combo"): + # Always write UI state so the worker's env fallback stays in + # sync when the user flips the combo between runs. + self.env.av1an_flags["engine"] = engine_sel + gpu_preview, gpu_api = resolve_gpu_encoder(engine_sel, selected_codec, self.env) + if gpu_preview: + self._log(f"Engine: GPU — {gpu_preview} ({gpu_api} hardware encode)") + elif engine_sel == "gpu": + gpu_enc = selected_codec.gpu_encoder or "(none for this codec)" + self._log(f"Engine: GPU requested but {gpu_enc} unavailable — will use CPU.") + elif engine_sel == "hybrid": + self._log("Engine: hybrid — GPU lane + CPU lane concurrently (splits the queue by size).") + + + # v4.7.0: kwargs shared by every lane of the queue (single + # worker, or the GPU + CPU lanes in hybrid mode). + common = dict( in_dir=in_dir, out_dir=out_dir, video_codec=selected_codec, @@ -6305,17 +7749,33 @@ class OpenCodecMaster(QMainWindow): crf=self.crf_knob.intValue(), preset_label=self.preset_combo.currentText(), delete_source=self.del_check.isChecked(), - env=self.env, extensions=self._parse_extensions(), resolution=self._get_current_resolution(), audio_level_db=self.vol_knob.value(), - use_ffmpeg_fallback=use_ffmpeg_fallback, subtitle_lang=SUBTITLE_OPTIONS[self.subs_combo.currentIndex()][1], force=self.force_check.isChecked(), # v5-01 ) - self.worker.log_msg.connect(self._log) - self.worker.progress_msg.connect(self._on_progress) - self.worker.finished_queue.connect(self._on_finished) + + # ── v4.7.0: hybrid (GPU + CPU lanes run concurrently) ── + if engine_sel == "hybrid": + if self._start_hybrid(common, use_ffmpeg_fallback, use_av1an): + self.btn_run.setEnabled(False) + self.btn_run.setText("RUNNING...") + self.btn_stop.setEnabled(True) + self.btn_rebuild.setEnabled(False) + for w in self.workers: + w.start() + return + self._log("Hybrid unavailable — falling back to a single CPU queue.") + engine_sel = "cpu" + + self.worker = EncoderWorker( + **common, + env=self.env, + use_ffmpeg_fallback=use_ffmpeg_fallback, + engine=engine_sel, # v4.6.0: Auto/GPU/CPU engine selector + ) + self._set_workers([self.worker]) self.btn_run.setEnabled(False) self.btn_run.setText("RUNNING...") @@ -6323,6 +7783,126 @@ class OpenCodecMaster(QMainWindow): self.btn_rebuild.setEnabled(False) self.worker.start() + @Slot() + def _on_engine_changed(self, index: int): + """v4.8.1: ENGINE = CPU disables the GPU dropdown (the choice + would have no effect); every other engine keeps it live.""" + self._update_gpu_combo_state() + + def _update_gpu_combo_state(self): + cpu_only = self.engine_combo.currentIndex() == 2 # "Engine: CPU" + self.gpu_combo.setEnabled(not cpu_only) + self.gpu_combo.setToolTip( + "ENGINE is CPU — the GPU choice has no effect." + if cpu_only else + "Hardware encoder capability class.\n" + "Auto-detect matches your card via nvidia-smi/lspci and the\n" + "live encode probe decides what actually works. Forcing a\n" + "profile also extends the REBUILD FROM GIT dependency tree\n" + "with that GPU's packages (nv-codec-headers / VAAPI / QSV).\n" + "'None (CPU-only encode)' opts out even when a GPU exists." + ) + + @Slot() + def _on_gpu_profile_changed(self, index: int): + """v4.8.0: persist the GPU capability-profile selection so both + the engine resolution and the rebuild dep tree pick it up.""" + if not getattr(self, "env", None): + return # UI build phase — env probe hasn't run yet + if index <= 0: + self.env.av1an_flags["gpu_profile"] = "auto" + return + self.env.av1an_flags["gpu_profile"] = GPU_PROFILES[index - 1].key + self._log(f"GPU profile: {GPU_PROFILES[index - 1].key}") + + def _set_workers(self, workers: list) -> None: + """v4.7.0: register the active lane worker(s) and wire their + signals. STOP iterates every lane; _on_finished aggregates the + per-lane summaries into one.""" + self.workers = list(workers) + self._hybrid_pending = len(self.workers) + self._hybrid_totals = [0, 0] + for w in self.workers: + w.log_msg.connect(self._log) + w.progress_msg.connect(self._on_progress) + w.finished_queue.connect(self._on_finished) + + def _start_hybrid(self, common: dict, use_ffmpeg_fallback: bool, + use_av1an: bool) -> bool: + """v4.7.0: split the queue between a GPU lane (NVENC) and a CPU + lane (software encoders, or av1an chunk-parallel when opted in — + so NVENC + chunk workers + software can all run at once). + + Constructs both workers on success and returns True. Returns + False (with a logged reason) when hybrid cannot apply; the caller + falls back to a single-lane queue. + """ + import copy as _copy + import dataclasses as _dc + + gpu_enc, _gpu_api = resolve_gpu_encoder("gpu", common["video_codec"], self.env) + if not gpu_enc: + self._log("Hybrid: no functional NVENC encoder for this codec family.") + return False + + files = scan_input_files(common["in_dir"], common["extensions"]) + if len(files) < 2: + self._log("Hybrid: fewer than 2 encodable files — one lane is faster than scheduling.") + return False + + sizes: dict = {} + for f in files: + try: + sizes[f] = f.stat().st_size + except OSError: + pass + plan = plan_hybrid( + files, gpu_enc, True, self.env.cpu.logical_threads, sizes=sizes, + ) + if plan is None or not plan.gpu_files or not plan.cpu_files: + self._log("Hybrid: size split degenerated to one lane — single lane is faster.") + return False + + self._log( + f"HYBRID: GPU lane = {len(plan.gpu_files)} file(s) via {gpu_enc} | " + f"CPU lane = {len(plan.cpu_files)} file(s) " + f"(budget {plan.cpu_budget_threads} threads" + f"{', av1an chunk-parallel' if use_av1an else ''})" + ) + self._log( + " Note: lanes use different encoders — GPU-lane files get NVENC " + "quality, CPU-lane files get software-encoder quality." + ) + + # CPU lane budget: hold back threads for the GPU lane's + # decode/scale/mux processes, then let the lane self-budget its + # av1an workers / ffmpeg threads from the reduced topology. + cpu_env = _copy.copy(self.env) + cpu_env.cpu = _dc.replace( + self.env.cpu, logical_threads=plan.cpu_budget_threads + ) + + gpu_worker = EncoderWorker( + **common, + env=self.env, + use_ffmpeg_fallback=True, # GPU runs via single-pass ffmpeg + engine="gpu", + file_subset=plan.gpu_files, + lane="gpu", + ) + cpu_worker = EncoderWorker( + **common, + env=cpu_env, + use_ffmpeg_fallback=use_ffmpeg_fallback, + engine="cpu", + file_subset=plan.cpu_files, + lane="cpu", + ffmpeg_threads=plan.cpu_budget_threads, + ) + self.worker = cpu_worker # primary handle (back-compat) + self._set_workers([gpu_worker, cpu_worker]) + return True + def _handle_vs_incompat(self) -> bool: """Handle detected VSScript ABI incompatibility. @@ -6441,7 +8021,9 @@ class OpenCodecMaster(QMainWindow): self._build_worker = SourceBuildWorker( build_vs=build_vs, build_av1an=build_av1an, build_ffmpeg_iamf=build_ffmpeg_iamf, + gpu_profile_key=self.env.av1an_flags.get("gpu_profile", "auto"), ) + self._build_worker.log_msg.connect(self._log) self._build_worker.build_done.connect(self._on_build_done) self._build_worker.start() @@ -6582,6 +8164,21 @@ class OpenCodecMaster(QMainWindow): @Slot(int, int) def _on_finished(self, ok: int, fail: int): + # v4.7.0: hybrid lanes finish independently — aggregate the + # per-lane summaries and only finalize when the LAST lane exits. + if len(self.workers) > 1: + self._hybrid_totals[0] += ok + self._hybrid_totals[1] += fail + self._hybrid_pending -= 1 + if self._hybrid_pending > 0: + self._log( + f"Lane done — {ok} ok, {fail} failed. " + f"Waiting for the other lane..." + ) + return + ok, fail = self._hybrid_totals + self._hybrid_totals = [0, 0] + self.btn_run.setEnabled(True) self.btn_run.setText("START PROCESSING") self.btn_stop.setEnabled(False) @@ -6594,12 +8191,15 @@ class OpenCodecMaster(QMainWindow): @Slot() def _stop_process(self): - if self.worker and self.worker.isRunning(): + if any(w.isRunning() for w in self.workers): self._log("STOP: Exiting queue after current file finishes...") - self.worker.stop() + for w in self.workers: + w.stop() self.btn_stop.setEnabled(False) + + # ────────────────────────────────────────────── # ENTRY POINT # ────────────────────────────────────────────── diff --git a/opentranscode/__init__.py b/opentranscode/__init__.py index bb393e0..9434bc5 100755 --- a/opentranscode/__init__.py +++ b/opentranscode/__init__.py @@ -23,7 +23,7 @@ Or import in code: from __future__ import annotations -__version__ = "4.5.0" +__version__ = "4.8.1" __author__ = "Jeremy Anderson - dcos.net" __license__ = "AGPL-3.0" @@ -52,7 +52,9 @@ def launch_gui(argv: list[str] | None = None, force: bool = False, verbose: bool = False, skip_existing: bool = True, timeout: int = 86400, - inline_scale: bool = False) -> int: + inline_scale: bool = False, + engine: str = "auto", + gpu_profile: str = "auto") -> int: """Launch the OpenTranscode GUI. Thin wrapper around ``opentranscode.ui_window.launch_gui``; imported @@ -101,11 +103,18 @@ def launch_gui(argv: list[str] | None = None, force: bool = False, error (rc=234)" disk-exhaustion messages. Default False — the intermediate path is more robust on older av1an/ VapourSynth builds. Pre-checks the "Inline scale" UI checkbox. + engine: Video encode engine (v4.6.0). "auto" (default) uses the + NVENC GPU encoder when the selected codec family has one and + the live encode test proved it works; "gpu" forces NVENC; + "cpu" forces the software encoders; "hybrid" (v4.7.0) splits + the queue between a GPU lane and a CPU lane running + concurrently. Pre-selects the ENGINE combo in the UI. """ from .ui_window import launch_gui as _launch return _launch( argv, force=force, chunk_method=chunk_method, max_workers=max_workers, threads_per_worker=threads_per_worker, use_av1an=use_av1an, verbose=verbose, skip_existing=skip_existing, - timeout=timeout, inline_scale=inline_scale, + timeout=timeout, inline_scale=inline_scale, engine=engine, + gpu_profile=gpu_profile, ) diff --git a/opentranscode/cli.py b/opentranscode/cli.py index d95f026..8262b6c 100755 --- a/opentranscode/cli.py +++ b/opentranscode/cli.py @@ -183,6 +183,41 @@ def build_parser() -> argparse.ArgumentParser: "older av1an/VapourSynth builds. Enable for large files " "with scaling to save disk + an extra encode pass.", ) + # v4.6.0: --engine selects the video encode engine. "auto" (default) + # uses the NVENC hardware encoder for the selected codec family when + # the environment probe's live encode test proved it works; "gpu" + # forces NVENC (falls back to CPU with a log line); "cpu" forces the + # software encoders. The GPU path runs via single-pass ffmpeg — + # av1an cannot drive NVENC. + parser.add_argument( + "--engine", choices=["auto", "gpu", "cpu", "hybrid"], default="auto", + help="Video encode engine (default: auto). 'auto' uses the NVENC " + "GPU encoder (hevc_nvenc / av1_nvenc) when the selected " + "codec family has one and a live encode test proved it " + "works; falls back to CPU otherwise. 'gpu' forces NVENC. " + "'cpu' forces the software encoders (SVT-AV1 / VP9 / " + "x265). 'hybrid' splits the queue between a GPU lane and a " + "CPU lane running CONCURRENTLY (balanced by file size, CPU " + "lane budget reduced by the GPU lane's reserve) — with " + "--use-av1an the CPU lane uses chunk-parallel, so NVENC + " + "chunk workers + software all run at once. Needs 2+ " + "encodable files and a functional GPU encoder; falls back " + "to CPU otherwise. GPU encodes run via single-pass ffmpeg " + "— av1an never drives NVENC.", + ) + # v4.8.0: force a GPU capability profile (see gpu_profiles.py for + # the combined generation entries). "auto" matches the detected card. + from .gpu_profiles import GPU_PROFILES + parser.add_argument( + "--gpu-profile", default="auto", + choices=["auto"] + [gp.key for gp in GPU_PROFILES], + help="GPU capability profile (default: auto-detect from the " + "installed card). Combined generation entries — e.g. " + "nv-pascal covers the whole GTX 10-series + Tesla P40/P4/" + "P100; nv-ada covers RTX 40/50. Forcing a profile extends " + "the rebuild-from-git dependency tree with that GPU's " + "packages. 'cpu' = ignore hardware.", + ) return parser @@ -243,6 +278,22 @@ def run_dry_run( effective_cm = env.av1an_flags.get("chunk_method_override") print(f"Chunk method: {effective_cm or 'auto (av1an decides)'}{cli_chunk_method_note}") + # v4.6.0: GPU/NVENC probe result (live encode test, not just the + # compiled-in encoder list). + gpu = getattr(env, "gpu", None) + if gpu is None: + print("GPU: (probe unavailable)") + elif gpu.has_gpu: + print(f"GPU: {gpu.name or 'NVIDIA'} — NVENC ready: " + f"{', '.join(gpu.usable_encoders)}") + elif any(gpu.encoders.values()): + present = [e for e in ("av1_nvenc", "hevc_nvenc", "h264_nvenc") + if gpu.encoders.get(e)] + print(f"GPU: {', '.join(present)} present but NOT usable — " + f"{gpu.first_failure_detail or 'smoke test failed'}") + else: + print("GPU: none (CPU encoding)") + # v4.1.0: show intelligent worker math so the user can verify the # chunk-parallel thread budget before launching a real encode. # We instantiate EncoderWorker without starting the QThread to read @@ -380,4 +431,6 @@ def main(argv: list[str] | None = None) -> int: skip_existing=args.skip_existing, timeout=args.timeout, inline_scale=args.inline_scale, + engine=args.engine, + gpu_profile=args.gpu_profile, ) diff --git a/opentranscode/codec_profiles.py b/opentranscode/codec_profiles.py index c8258cf..8d95d05 100755 --- a/opentranscode/codec_profiles.py +++ b/opentranscode/codec_profiles.py @@ -7,7 +7,7 @@ the CLI --version path). """ from collections.abc import Callable -from dataclasses import dataclass +from dataclasses import dataclass, field # ────────────────────────────────────────────── # CONFIG-DRIVEN PROFILES (replaces all if/else chains) @@ -36,6 +36,25 @@ class VideoCodecProfile: # output for each encoder; this is the codec_name field in the video # stream's JSON, NOT the encoder_name (which would be "libsvtav1" etc). ffprobe_codec_name: str = "" + # v4.6.0: hardware (NVENC) counterpart for this codec family. Empty + # string = no hardware encoder exists for this family (VP9 has no + # NVENC encoder). The GPU path is ffmpeg-only (av1an cannot drive + # NVENC); EncoderWorker.resolve_gpu_encoder() only selects it when a + # functional probe proved the encoder works on this system. The + # ffprobe codec_name is IDENTICAL to the CPU encoder's (hevc_nvenc + # also produces "hevc"), so skip-existing detection works across + # GPU/CPU re-encodes of the same family. + gpu_encoder: str = "" + # (crf, preset) -> ffmpeg args for the NVENC encoder. Mirrors + # ffmpeg_vargs_fn. None when gpu_encoder is empty. + gpu_vargs_fn: Callable[[int, int], list[str]] | None = None + # v4.8.0: GPU-profile support. *gpu_family* is the codec family key + # used by GpuProfile.encoders ("av1"/"hevc"/"vp9"); *gpu_encoders_by_api* + # maps a hardware API (nvenc/vaapi/qsv) to this profile's ffmpeg + # encoder for that API. resolve_gpu_encoder() picks the entry matching + # the selected GPU profile. + gpu_family: str = "" + gpu_encoders_by_api: dict[str, str] = field(default_factory=dict) @dataclass class AudioProfile: @@ -125,6 +144,52 @@ def _x265_ffmpeg_args(crf: int, preset: int) -> list[str]: "-pix_fmt", "yuv420p10le", "-g", "240"] +# ── v4.6.0: NVENC (hardware) vargs ── +# NVENC quality control: -rc vbr + -cq N + -b:v 0 is the constant-quality +# mode that maps most closely to the CPU encoders' CRF (cq ≈ crf for HEVC +# and AV1 within ~±3). -b:v 0 removes the default bitrate cap so -cq +# actually governs quality. Presets are p1 (fastest) .. p7 (slowest/best) +# on all current NVENC generations; the legacy "slow/medium/fast" aliases +# are deprecated. +# +# Pixel format: 8-bit yuv420p. Pascal-generation cards (GTX 10xx) run +# HEVC Main10 at roughly half throughput, and the archival targets here +# are 8-bit phone/BluRay sources — 8-bit keeps the GPU path at full +# speed. ffmpeg auto-converts 10-bit sources to yuv420p. + +def _nvenc_preset(preset: int) -> str: + """Map the CPU preset tiers (lower value = slower/better) to NVENC + p-presets. CPU preset values across profiles are 0..10 with 0/5 = + slowest quality tiers; NVENC is fast enough that even p7 outruns any + CPU encoder, so the whole range compresses to p3..p7.""" + if preset <= 6: + return "p7" # "Slow" tier → best NVENC quality + if preset <= 8: + return "p5" # "Medium" tier + return "p4" # "Fast"/"Faster" tiers + + +def _hevc_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for hevc_nvenc (x265/HEVC family hardware encoder).""" + return ["-c:v", "hevc_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + +def _h264_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for h264_nvenc (hardware H.264 — compatibility target).""" + return ["-c:v", "h264_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + +def _av1_nvenc_args(crf: int, preset: int) -> list[str]: + """FFmpeg args for av1_nvenc (AV1 family hardware encoder, RTX 40+).""" + return ["-c:v", "av1_nvenc", "-preset", _nvenc_preset(preset), + "-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0", + "-pix_fmt", "yuv420p", "-g", "240"] + + VIDEO_CODECS: list[VideoCodecProfile] = [ VideoCodecProfile( label="AV1 (SVT-AV1)", @@ -138,6 +203,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (8)", "Medium (6)", "Fast (4)", "Faster (2)"], preset_map={"Slow (8)": 8, "Medium (6)": 6, "Fast (4)": 4, "Faster (2)": 2}, ffprobe_codec_name="av1", # v4.3.0: skip-existing detection + # v4.6.0: av1_nvenc exists only on RTX 40+ (Ada) cards; on Pascal + # (GTX 10xx) the functional probe fails and auto falls back to + # the SVT-AV1 CPU encoder. + gpu_encoder="av1_nvenc", + gpu_vargs_fn=_av1_nvenc_args, + gpu_family="av1", + gpu_encoders_by_api={"nvenc": "av1_nvenc", "qsv": "av1_qsv", + "vaapi": "av1_vaapi"}, ), VideoCodecProfile( label="VP9", @@ -151,6 +224,10 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (0)", "Medium (2)", "Fast (4)", "Faster (6)"], preset_map={"Slow (0)": 0, "Medium (2)": 2, "Fast (4)": 4, "Faster (6)": 6}, ffprobe_codec_name="vp9", # v4.3.0: skip-existing detection + # v4.8.0: VP9 has no NVENC encoder; VAAPI (AMD/older Intel) can + # encode it on some cards. + gpu_family="vp9", + gpu_encoders_by_api={"vaapi": "vp9_vaapi"}, ), VideoCodecProfile( label="x265 (HEVC)", @@ -164,6 +241,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [ presets=["Slow (5)", "Medium (7)", "Fast (9)", "Faster (10)"], preset_map={"Slow (5)": 5, "Medium (7)": 7, "Fast (9)": 9, "Faster (10)": 10}, ffprobe_codec_name="hevc", # v4.3.0: skip-existing detection + # v4.6.0: hevc_nvenc works on every NVENC generation since Maxwell + # GM206 (incl. the GTX 1070) — this is the family that benefits + # most from GPU mode. + gpu_encoder="hevc_nvenc", + gpu_vargs_fn=_hevc_nvenc_args, + gpu_family="hevc", + gpu_encoders_by_api={"nvenc": "hevc_nvenc", "qsv": "hevc_qsv", + "vaapi": "hevc_vaapi"}, ), ] @@ -222,6 +307,23 @@ FFMPEG_LIB_KEY_MAP: dict[str, str] = { "libaom-av1": "libaom", "libvpx-vp9": "libvpx", "libx265": "libx265", + # v4.6.0: hardware encoders. These keys are populated by + # _probe_ffmpeg_libs() alongside the software encoders, and — unlike + # the compiled-in check — EncoderWorker additionally gates the GPU + # path on env.gpu.functional (a real encode smoke test), because a + # ffmpeg build can list an NVENC encoder that the installed driver + # cannot open (NVENC API version mismatch). + "hevc_nvenc": "hevc_nvenc", + "h264_nvenc": "h264_nvenc", + "av1_nvenc": "av1_nvenc", + # v4.8.0: hardware APIs for AMD (VAAPI) and Intel (QSV) profiles. + "hevc_vaapi": "hevc_vaapi", + "h264_vaapi": "h264_vaapi", + "av1_vaapi": "av1_vaapi", + "vp9_vaapi": "vp9_vaapi", + "hevc_qsv": "hevc_qsv", + "h264_qsv": "h264_qsv", + "av1_qsv": "av1_qsv", } diff --git a/opentranscode/encoder_worker.py b/opentranscode/encoder_worker.py index 64a4c16..4317a95 100755 --- a/opentranscode/encoder_worker.py +++ b/opentranscode/encoder_worker.py @@ -38,6 +38,12 @@ from .codec_profiles import ( VideoCodecProfile, ffmpeg_lib_key_for, ) +from .gpu_profiles import ( + encoder_filter_chain, + encoder_pre_args, + encoder_quality_args, + gpu_profile_by_key, +) from .env_probe import EnvProbe, _av1an_env from .ffprobe_utils import ( _identify_file_type, @@ -52,11 +58,97 @@ from .temp_manager import _temp_path_for, _worker_temp_dir # ENCODER WORKER (QThread, from PySide6 ver, extended) # ────────────────────────────────────────────── +def scan_input_files(in_dir: Path, extensions: set[str]) -> list[Path]: + """Collect the transcodable files under *in_dir* (v4.7.0). + + Used by ``EncoderWorker.run()`` for the default whole-directory scan + AND by the hybrid scheduler's pre-scan, which must partition the + queue BEFORE the per-lane workers are constructed. Excludes leftover + pre-scale intermediates from previous failed runs. + """ + return sorted( + f for f in in_dir.rglob("*") + if f.is_file() + and f.suffix.lower() in extensions + and not f.name.endswith(".scaled_tmp.mkv") + ) + + +def selected_gpu_profile(env): + """v4.8.0: the active GpuProfile — the UI's dropdown selection + (env.av1an_flags["gpu_profile"]) when set, else the auto-matched + profile from the probe (env.gpu.profile_key). None when neither.""" + flags = getattr(env, "av1an_flags", None) or {} + key = flags.get("gpu_profile") + if key and key != "auto": + profile = gpu_profile_by_key(key) + if profile is not None: + return profile + gpu_info = getattr(env, "gpu", None) + if gpu_info is not None and getattr(gpu_info, "profile_key", ""): + return gpu_profile_by_key(gpu_info.profile_key) + return None + + +def resolve_gpu_encoder(engine: str, video_codec, env): + """Decide whether this encode runs on the GPU. + + Returns ``(encoder_name, api)`` when the GPU path should be used — + e.g. ``("hevc_nvenc", "nvenc")`` or ``("hevc_vaapi", "vaapi")`` — or + ``(None, None)`` for the CPU path. + + Rules: + - ``engine == "cpu"`` → always CPU (user forced CPU). + - selected/matched GPU profile has + no encoder for the codec family → CPU (e.g. AV1 on Pascal, + VP9 without VAAPI). + - ``engine`` auto/gpu AND the + functional probe passed for + that encoder → the encoder. + + The gate is ``env.gpu.functional`` — a live encode test run by the + env probe — NOT the compiled-in ``ffmpeg -encoders`` list, because a + ffmpeg build can advertise a hardware encoder the installed driver + is too old to open. Pure function; no I/O. Safe to call from the UI + thread for a pre-flight status line. + """ + if engine not in ("auto", "gpu"): + return (None, None) + profile = selected_gpu_profile(env) + if profile is not None: + family = getattr(video_codec, "gpu_family", "") or "" + gpu_enc = profile.encoders.get(family) + api = profile.api + else: + # No GPU profile context (older callers / no probe data): fall + # back to the profile's NVENC encoder. + gpu_enc = getattr(video_codec, "gpu_encoder", "") or "" + api = "nvenc" if gpu_enc else None + if not gpu_enc: + return (None, None) + gpu_info = getattr(env, "gpu", None) + if gpu_info is not None and gpu_info.functional.get(gpu_enc, False): + return (gpu_enc, api) + return (None, None) + + class EncoderWorker(QThread): log_msg = Signal(str) progress_msg = Signal(str, int, int) # (filename, current, total) finished_queue = Signal(int, int) # (success_count, fail_count) + # v4.4.3/v4.6.0: class-level defaults for attributes normally set in + # __init__. The mocked test suite builds workers via ``__new__`` + # (bypassing __init__) and calls non-Qt methods directly; without + # these defaults those instances crash with AttributeError on + # ``verbose`` / ``_current_total`` (the "AttributeError: no attribute + # 'verbose'" class of bugs from the v4.4.3 changelog). Instance + # assignment in __init__ shadows these harmlessly. + verbose = False + _current_idx = 0 + _current_total = 0 + _current_filename = "" + def __init__( self, in_dir: Path, @@ -87,6 +179,25 @@ class EncoderWorker(QThread): # to honor them. max_workers: int | None = None, threads_per_worker: int | None = None, + # v4.6.0: encode engine selection. "auto" uses the NVENC GPU + # encoder when the selected codec family has one AND the env + # probe's live encode test proved it works on this system; + # otherwise (or with "cpu") the CPU encoders are used. "gpu" + # requests GPU and falls back to CPU with a log line when the + # hardware is unavailable. Resolved against env.av1an_flags + # ["engine"] like the other CLI-plumbed flags. + engine: str | None = None, + # v4.7.0: hybrid-lane support. *file_subset* restricts this + # worker to an explicit file list (the hybrid scheduler scans and + # partitions the queue up front, then spawns a GPU-lane and a + # CPU-lane worker with disjoint subsets). *lane* suffixes the + # per-worker temp dir so one lane's cleanup sweep can never + # delete the other lane's intermediates. *ffmpeg_threads* caps + # the CPU lane's software ffmpeg encode (GPU jobs are capped by + # NVENC silicon, not threads). + file_subset: list[Path] | None = None, + lane: str = "", + ffmpeg_threads: int | None = None, ): super().__init__() self.in_dir = in_dir @@ -126,6 +237,23 @@ class EncoderWorker(QThread): ) else None ) ) + # v4.6.0: engine selection ("auto" | "gpu" | "cpu"). Falls back to + # env.av1an_flags["engine"] when not passed explicitly (same + # pattern as max_workers — lets the CLI reach the GUI-spawned + # worker without ui_window changes). Resolved to a concrete + # GPU/CPU decision in run() via resolve_gpu_encoder(). + self.engine = engine if engine in ("auto", "gpu", "cpu") else ( + env.av1an_flags.get("engine", "auto") + if env.av1an_flags.get("engine") in ("auto", "gpu", "cpu") + else "auto" + ) + # Resolved in run(): NVENC encoder name when the GPU path is + # active, None for CPU. _ffmpeg_fallback_encode and + # _prepare_input read this (GPU mode implies the ffmpeg path — + # av1an cannot drive NVENC — which also means no pre-scale + # intermediate: the ffmpeg path scales inline). + self._gpu_encoder: str | None = None + self._gpu_api: str | None = None # v4.2.1: quiet mode by default. Tech-detail log lines (CMD:, # live tail of av1an/ffmpeg stderr, DIAGNOSIS blocks, resolution # map, pre-flight validation table, heartbeat) are gated behind @@ -183,7 +311,10 @@ class EncoderWorker(QThread): # cleanup sweep can safely nuke only this worker's intermediates # without affecting a concurrent worker. The subdir is created # with mode=0o700 to prevent symlink attacks from other users. - self._temp_dir = _worker_temp_dir(os.getpid()) + self.file_subset = file_subset + self.lane = lane + self.ffmpeg_threads = ffmpeg_threads + self._temp_dir = _worker_temp_dir(os.getpid(), lane=lane) # v6-06: KeepAwake instance — started in run(), stopped in finally. # mouse_nudge defaults to False (opt-in) to avoid surprising the # user with cursor movement. systemd-inhibit is always-on when @@ -579,7 +710,25 @@ class EncoderWorker(QThread): Returns True on success, False on failure. """ # Check if ffmpeg has the video encoder we need - ffmpeg_enc = self.video_codec.ffmpeg_encoder + # v4.6.0: GPU mode swaps in the hardware encoder and its vargs. + # v4.8.0: VAAPI/QSV APIs build their command shape (device init, + # hwupload filter, quality args) from gpu_profiles; NVENC keeps + # the original profile vargs. + ffmpeg_enc = self._gpu_encoder or self.video_codec.ffmpeg_encoder + v_args = ( + (self.video_codec.gpu_vargs_fn or self.video_codec.ffmpeg_vargs_fn) + if self._gpu_encoder else self.video_codec.ffmpeg_vargs_fn + )(self.crf, self.preset_val) + hw_pre_args: list[str] = [] + hw_filter_args: list[str] = [] + if self._gpu_encoder and self._gpu_api in ("vaapi", "qsv"): + profile = selected_gpu_profile(self.env) + if profile is not None: + hw_pre_args = encoder_pre_args(profile) + hw_filter_args = encoder_filter_chain(profile) + v_args = encoder_quality_args( + self._gpu_api, ffmpeg_enc, self.crf, self.preset_val + ) # v3: use the module-level FFMPEG_LIB_KEY_MAP (OTC-007). ffmpeg_lib_key = ffmpeg_lib_key_for(ffmpeg_enc) @@ -590,19 +739,29 @@ class EncoderWorker(QThread): ) return False - v_args = self.video_codec.ffmpeg_vargs_fn(self.crf, self.preset_val) - # Belt-and-suspenders: if a target resolution is set, inject -vf scale # directly into the ffmpeg command. This guarantees the output resolution # matches the dropdown even if the intermediate pre-scale was bypassed. vf_scale_args: list[str] = [] if self.resolution.width is not None and self.resolution.height is not None: - vf_scale_args = [ - "-vf", ( - f"scale={self.resolution.width}:{self.resolution.height}:" - f"force_original_aspect_ratio=decrease:force_divisible_by=2" - ), - ] + if self._gpu_api == "vaapi": + # v4.8.0: VAAPI scales ON the hardware — combine the + # scalar with the hwupload upload in one chain. + vf_scale_args = [ + "-vf", ( + f"scale={self.resolution.width}:{self.resolution.height}:" + f"force_original_aspect_ratio=decrease:force_divisible_by=2," + f"format=nv12,hwupload" + ), + ] + hw_filter_args = [] + else: + vf_scale_args = [ + "-vf", ( + f"scale={self.resolution.width}:{self.resolution.height}:" + f"force_original_aspect_ratio=decrease:force_divisible_by=2" + ), + ] # Audio args from profile audio_args = list(self.audio_profile.params) @@ -623,10 +782,23 @@ class EncoderWorker(QThread): if self.container.ext == "mp4": mux_flags = ["-movflags", "+faststart"] - cmd = [ - self.env.ffmpeg_path, + cmd = [self.env.ffmpeg_path] + hw_pre_args + [ "-i", str(encode_input), - ] + vf_scale_args + v_args + audio_args + mux_flags + [ + ] + vf_scale_args + hw_filter_args + v_args + # v4.7.0: CPU lane thread cap in hybrid mode (the GPU lane's + # NVENC job keeps ~2 threads for decode/mux). Never applied on + # the GPU path — NVENC throughput is silicon-bound, not + # thread-bound. Skipped when the cap is 0/unset. + if self._gpu_encoder is None and self.ffmpeg_threads: + # v4.7.1: libx265 maps -threads to frame-threads, capped at + # X265_MAX_FRAME_THREADS (16) — larger values abort the + # encoder ("frameNumThreads must be [0 .. X265_MAX_FRAME_ + # THREADS)"). SVT-AV1 and libvpx accept the full budget. + threads = self.ffmpeg_threads + if ffmpeg_enc == "libx265": + threads = min(threads, 16) + cmd += ["-threads", str(threads)] + cmd += audio_args + mux_flags + [ "-y", str(output_f), ] @@ -672,6 +844,11 @@ class EncoderWorker(QThread): self.log_msg.emit( f" ffmpeg error (rc={res.returncode}): {stderr_snip.strip()}" ) + # v4.7.1: remove the partial output. Without this, a + # failed encode left a truncated file that ffprobe can + # still parse as the right codec — and skip-existing + # would then treat it as a finished archive forever. + output_f.unlink(missing_ok=True) return False except OSError as e: self.log_msg.emit(f"{self._status_prefix()}FAIL: system error: {e}") @@ -693,14 +870,44 @@ class EncoderWorker(QThread): phys = self.env.cpu.physical_cores logical = self.env.cpu.logical_threads + # ── v4.6.0: engine resolution (GPU vs CPU) ── + # GPU mode is a variant of the ffmpeg path: av1an invokes + # encoder CLI binaries (SvtAv1EncApp / vpxenc / x265) and cannot + # drive NVENC, so an active GPU encoder forces the single-pass + # ffmpeg path. NVENC on even a GTX 1070 encodes 1080p at several + # hundred fps — one ffmpeg process beats av1an's chunk-parallel + # CPU workers, and chunking becomes unnecessary. + self._gpu_encoder, self._gpu_api = resolve_gpu_encoder( + self.engine, self.video_codec, self.env) + if self._gpu_encoder: + self.use_ffmpeg_fallback = True + self.log_msg.emit( + f"ENGINE: GPU ({self._gpu_encoder}, {self._gpu_api}) — " + f"single-pass ffmpeg hardware encode; av1an chunk-parallel " + f"not used." + ) + elif self.engine == "gpu": + gpu_enc = getattr(self.video_codec, "gpu_encoder", "") or "" + if not gpu_enc and not getattr(self.video_codec, "gpu_family", ""): + self.log_msg.emit( + f"ENGINE: GPU requested but {self.video_codec.label} has no " + f"hardware encoder — using CPU." + ) + else: + gpu = getattr(self.env, "gpu", None) + detail = gpu.first_failure_detail if gpu is not None else "" + self.log_msg.emit( + f"ENGINE: GPU requested but no usable hardware encoder for " + f"{self.video_codec.label} ({detail or 'unavailable'}) — using CPU." + ) + # Collect all valid files first (for progress tracking) # Exclude our own temp intermediates from previous failed runs. - all_files = sorted( - f for f in self.in_dir.rglob("*") - if f.is_file() - and f.suffix.lower() in self.extensions - and not f.name.endswith(".scaled_tmp.mkv") - ) + if self.file_subset is not None: + # v4.7.0: hybrid lane — the scheduler partitioned the queue. + all_files = sorted(self.file_subset) + else: + all_files = scan_input_files(self.in_dir, self.extensions) total = len(all_files) if total == 0: @@ -713,7 +920,12 @@ class EncoderWorker(QThread): # to know the worker math — they just need files to encode. # use_ffmpeg_fallback is set by the main thread's pre-flight check. if self.verbose: - if self.use_ffmpeg_fallback: + if self._gpu_encoder: + self._vlog( + f"GPU encode: {self.video_codec.label} via " + f"{self._gpu_encoder} (NVENC), CPU decode" + ) + elif self.use_ffmpeg_fallback: self._vlog( f"FFmpeg fallback: {self.video_codec.ffmpeg_encoder} on {phys} cores " f"(single-pass, no chunk-parallel)" @@ -1188,7 +1400,15 @@ class EncoderWorker(QThread): # temp directory so the user's video folders stay clean. encode_input = file_path - if needs_scale and not self.inline_scale: + # v4.6.0: pre-scale ONLY on the av1an path. The pure-ffmpeg path + # (the default, and the only path NVENC can run on) scales inline + # via the -vf args in _ffmpeg_fallback_encode — the intermediate + # existed solely because VapourSynth source plugins choke on some + # inputs ffmpeg handles fine. Skipping it on the ffmpeg path + # removes the entire class of large-file failures: no 0.5-0.8x + # source-size temp file, no extra full encode pass, and no + # pre-scale timeout on long/high-bitrate sources. + if needs_scale and not self.inline_scale and not self.use_ffmpeg_fallback: try: temp_scaled = _temp_path_for(file_path, ".scaled_tmp.mkv", worker_dir=self._temp_dir) self._current_temps.append(temp_scaled) @@ -1217,19 +1437,41 @@ class EncoderWorker(QThread): ] # v4.2.1: Scaling notice is verbose-only. self._vlog(f" Scaling {src_w or '?'}x{src_h or '?'} -> {self.resolution.width}x{self.resolution.height}...") - scale_res = subprocess.run( - scale_cmd, capture_output=True, text=True, timeout=1800, + # v4.6.0: run via _run_with_stop_check with the full + # per-file timeout. The old flat + # subprocess.run(timeout=1800) killed pre-scaling of + # long/high-bitrate sources at exactly 30 minutes + # ("FAIL: pre-scale error: Command ... timed out") — a + # guaranteed large-file failure that also ignored the + # STOP button for the whole intermediate pass. + scale_status, scale_rc, _s_out, scale_err = self._run_with_stop_check( + scale_cmd, timeout=self.encode_timeout, log_prefix=" ", ) - if scale_res.returncode == 0 and temp_scaled.exists(): + if scale_status == "stop": + # User aborted — clean up the partial intermediate and + # bail WITHOUT counting a failure (a STOP is not an + # encode failure; the queue loop breaks next iteration). + self._cleanup_current_temps() + return None + if scale_status == "timeout": + self.log_msg.emit( + f"{self._status_prefix()}FAIL: pre-scale timeout " + f"(exceeded {self.encode_timeout}s limit)" + ) + temp_scaled.unlink(missing_ok=True) + self._cleanup_current_temps() + self.fail_count += 1 + return None + if scale_status == "ok" and scale_rc == 0 and temp_scaled.exists(): encode_input = temp_scaled scaled_size = temp_scaled.stat().st_size / 1_048_576 # v4.2.1: verbose-only self._vlog(f" Pre-scale OK ({scaled_size:.1f} MB intermediate)") else: - stderr_snip = (scale_res.stderr or "")[-200:] + stderr_snip = (scale_err or "")[-200:] # v4.2.1: keep user-facing FAIL but shorten; stderr verbose-only self.log_msg.emit( - f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_res.returncode})" + f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_rc})" ) if stderr_snip.strip(): self._vlog(f" ffmpeg stderr: {stderr_snip.strip()}") @@ -1280,15 +1522,17 @@ class EncoderWorker(QThread): return (encode_input, output_f) def _check_disk_space(self, file_path: Path, output_f: Path, needs_scale: bool) -> None: - """v4.4.0: Warn (not abort) if free disk space is less than the source size. + """v4.4.0: Warn if free disk space is less than the encode will need. - v4.4.1: warnings gated behind --verbose. The user wants just - start + finish lines, no disk-space chatter. The check still - runs (so the warning is available via --verbose), but in quiet - mode it produces zero output. + v4.6.0: SEVERE warnings (free space below the source size on the + partition we're about to write a big intermediate/output to) are + now USER-FACING — they were verbose-only, so in the default quiet + mode a batch that was going to die with "No space left on device" + partway through gave zero advance notice. That silent failure was + one of the "large files just fail" reports: small files fit in + the remaining space, big ones didn't. Marginal advice (the 2-3x + intermediate estimate) stays verbose-only. """ - if not self.verbose: - return # v4.4.1: quiet mode — no disk-space warnings try: src_size = file_path.stat().st_size except OSError: @@ -1296,7 +1540,9 @@ class EncoderWorker(QThread): if src_size < 1_073_741_824: # < 1 GB — skip check for small files return src_gb = src_size / 1_073_741_824 - # Check output partition. + # Check output partition — severe when free < source size + # (the encoded output is usually smaller, but the ffmpeg/av1an + # buffer cache plus a same-partition temp can eat the difference). try: out_usage = shutil.disk_usage(output_f.parent) out_free_gb = out_usage.free / 1_073_741_824 @@ -1305,19 +1551,31 @@ class EncoderWorker(QThread): f" WARN: low disk space on output ({out_free_gb:.1f} GB free, " f"source is {src_gb:.1f} GB) — encode may fail partway through" ) + elif self.verbose and out_free_gb < src_gb * 2: + self._vlog( + f" WARN: output space getting tight ({out_free_gb:.1f} GB free, " + f"source is {src_gb:.1f} GB)" + ) except OSError: pass # can't check — skip - # When scaling, also check the temp partition (lossless intermediate - # can be 2-3x source size). - if needs_scale: + # When scaling on the av1an path, also check the temp partition + # (the CRF-16 intermediate can be ~1x source size). The ffmpeg + # path scales inline (no intermediate), so no temp warning there. + if needs_scale and not self.use_ffmpeg_fallback: try: tmp_usage = shutil.disk_usage(self._temp_dir) tmp_free_gb = tmp_usage.free / 1_073_741_824 - # Lossless intermediate is typically 2-3x source; warn if - # free < source * 2. - if tmp_free_gb < src_gb * 2: + # Severe: free temp space below the source size means the + # intermediate will likely not fit → user-facing warning. + if tmp_free_gb < src_gb: self.log_msg.emit( f" WARN: low disk space on temp ({tmp_free_gb:.1f} GB free, " + f"source is {src_gb:.1f} GB) — the scale intermediate " + f"may not fit. Free space or enable 'Inline scale'." + ) + elif self.verbose and tmp_free_gb < src_gb * 2: + self._vlog( + f" WARN: temp space getting tight ({tmp_free_gb:.1f} GB free, " f"lossless intermediate may need ~{src_gb * 2:.1f} GB) — " f"consider scaling to a smaller resolution or freeing space" ) @@ -1550,6 +1808,12 @@ class EncoderWorker(QThread): return False else: stderr_full = res.stderr or "" + # v4.6.0: scan stdout too. av1an routes chunk-retry + # noise (encoder stderr dumps, FRAME MISMATCH lines) + # to stdout, so pattern-matching on stderr alone missed + # the biggest real-world failure mode (see the FRAME + # MISMATCH pattern below). + combined_out = stderr_full + "\n" + (res.stdout or "") # v6: Don't increment fail_count yet — we may retry with # ffmpeg fallback below. Only increment if the retry also # fails (or no retry is possible). @@ -1685,10 +1949,56 @@ class EncoderWorker(QThread): ), False, # don't stop queue — retry with select chunk method ), + # v4.6.0: ffmpeg ≥ 7 removed the -vsync option that + # av1an's segment/hybrid chunk extraction passes to + # ffmpeg. Every segment-based chunk dies immediately + # ("Unrecognized option 'vsync'." → empty y4m pipe → + # chunk fails 3x). Verified on ffmpeg 9.0.2 + av1an + # 0.5.2. select (or a VS source plugin method) is the + # only working chunking on these systems. + ( + "Unrecognized option 'vsync'", + "av1an's segment/hybrid chunk extraction calls " + "`ffmpeg -vsync`, which ffmpeg 7+ removed. Every " + "segment-based chunk fails instantly on this system.", + ( + " FIX: install a VapourSynth source plugin so av1an", + " stops using ffmpeg segmenting: bestsource/ffms2/", + " lsmash (e.g. on Arch: vapoursynth-plugin-bs).", + " Alternatively stay on the default ffmpeg-only path", + " (it doesn't use av1an chunking at all).", + ), + False, # per-file — the select override keeps other files working + ), + # v4.6.0: frame-count drift between av1an's chunk + # manifest and what the encoder actually produced. + # av1an retries the chunk 3x, then shuts the worker + # down with the baffling "encoder crashed: exit + # status: 0" (exit 0 because the encode itself + # succeeded — on partial data). This was the dominant + # large-file failure in the 2026-07-13 av1an log and + # previously fell through to "Unknown av1an failure". + ( + "FRAME MISMATCH", + "av1an's chunk manifest expected a different frame count " + "than the encoder produced — chunk-extraction drift on " + "sources with sparse/irregular keyframes. The encode " + "itself exits 0 (it ran on partial data), which av1an " + "reports as 'encoder crashed: exit status: 0'.", + ( + " Will retry with --chunk-method select, which", + " extracts exact frame ranges and cannot drift.", + " This is per-file, not systematic — subsequent", + " files use select automatically.", + ), + False, # don't stop queue — retry with select chunk method + ), ) diagnosis_emitted = False for marker, summary, fixes, stop_queue in error_patterns: - if marker.lower() in stderr_full.lower(): + # v4.6.0: scan stdout + stderr (FRAME MISMATCH and + # encoder dumps land in stdout). + if marker.lower() in combined_out.lower(): # v4.2.1: DIAGNOSIS block is verbose-only. The user # already saw "FAIL: av1an exit code N" above — they # don't need the multi-line root-cause analysis unless @@ -1744,7 +2054,8 @@ class EncoderWorker(QThread): # v4.2.1: all DIAGNOSIS verbose-only. if ("SUMMARY" in stderr_full and "Average Speed" in stderr_full - and "Failed to read y4m frame delimiter" not in stderr_full): + and "Failed to read y4m frame delimiter" not in combined_out + and "FRAME MISMATCH" not in combined_out): self._vlog("") self._vlog( "DIAGNOSIS: SVT-AV1 encoder completed successfully (SUMMARY" @@ -1781,14 +2092,27 @@ class EncoderWorker(QThread): # same params). Cache the working method so subsequent files # skip the wasted first attempt. # + # v4.6.0: FRAME MISMATCH (chunk-extraction drift, see the + # error-pattern table) joins the y4m break as a drift + # symptom that select fixes — it was previously an + # "Unknown av1an failure" that went straight to the slow + # full-file ffmpeg fallback. + # # NOTE: Do NOT clean up _current_temps before the retry — # encode_input (symlink or pre-scaled file) is in # _current_temps and the recursive _encode_one call needs it. # The finally block below will clean up everything after the # recursive call returns (its own finally clears the list # first; our finally then runs on an empty list — no-op). - y4m_break = "Failed to read y4m frame delimiter" in stderr_full - if (not self._stop and y4m_break + extraction_drift = ( + "Failed to read y4m frame delimiter" in combined_out + or "FRAME MISMATCH" in combined_out + # ffmpeg >= 7 removed -vsync: segment/hybrid chunking + # dies instantly while select still works (it uses the + # ffmpeg frame server, not segmenting). + or "Unrecognized option 'vsync'" in combined_out + ) + if (not self._stop and extraction_drift and effective_chunk_method != "select" and self.env.av1an_flags.get("has_chunk_method", True)): # v4.2.1: RETRY messages are verbose-only — the user @@ -2040,9 +2364,15 @@ class EncoderWorker(QThread): try: # Pass 1: analyze current loudness + # v4.6.0: -vn skips video decoding — without it the analysis + # decoded the ENTIRE video stream just to measure audio + # loudness, which pushed long/large files past the 120s + # timeout and silently degraded every big file to the static + # knob gain. analysis_cmd = [ self.env.ffmpeg_path, "-i", str(file_path), + "-vn", "-af", ( f"loudnorm=I={target_lufs}:TP=-1.5:LRA=11:" f"print_format=json" diff --git a/opentranscode/env_probe.py b/opentranscode/env_probe.py index a85bc30..4dad220 100755 --- a/opentranscode/env_probe.py +++ b/opentranscode/env_probe.py @@ -16,6 +16,7 @@ import ctypes import os import re import shutil +import site import subprocess import sys import tempfile @@ -24,6 +25,271 @@ from pathlib import Path from .cpu_topology import CpuTopology, detect_cpu_topology from .distro_probe import DistroProfile, detect_distro +from .gpu_profiles import ( + encoder_pre_args, + encoder_filter_chain, + encoder_quality_args, + gpu_profile_by_key, + match_gpu_profile, + resolve_vaapi_device, +) + +# ────────────────────────────────────────────── +# GPU PROBE (v4.6.0 — NVENC hardware encoding) +# ────────────────────────────────────────────── + +# NVENC encoders we know how to drive, in preference order (best +# compression efficiency first). av1_nvenc only exists on RTX 40+; the +# functional smoke test below decides what is actually usable. +_NVENC_ENCODER_NAMES: tuple[str, ...] = ("av1_nvenc", "hevc_nvenc", "h264_nvenc") + + +@dataclass +class GpuInfo: + """Result of the GPU/NVENC probe. + + ``encoders`` — encoder name → ffmpeg was BUILT with it (from + ``ffmpeg -encoders``). + ``functional`` — encoder name → a real 0.2s NVENC encode SUCCEEDED. + This is the gate EncoderWorker uses: a ffmpeg build + can list hevc_nvenc while the installed driver is + too old for the NVENC API version it was compiled + against ("Driver does not support the required + nvenc API version") — only a live encode reveals + that. + ``details`` — encoder name → first stderr line when the smoke + test failed (actionable diagnostics). + """ + name: str = "" # GPU model name via nvidia-smi/lspci, "" if unknown + encoders: dict[str, bool] = field(default_factory=dict) + functional: dict[str, bool] = field(default_factory=dict) + details: dict[str, str] = field(default_factory=dict) + # v4.8.0: the matched GpuProfile key from gpu_profiles (auto-detected + # from the GPU name; the UI can override it). + profile_key: str = "" + + @property + def usable_encoders(self) -> list[str]: + """Encoders that passed the live encode test, preference order.""" + return [e for e in _NVENC_ENCODER_NAMES if self.functional.get(e, False)] + + @property + def has_gpu(self) -> bool: + return bool(self.usable_encoders) + + @property + def first_failure_detail(self) -> str: + """First non-empty failure detail (for user-facing warnings).""" + for e in _NVENC_ENCODER_NAMES: + d = self.details.get(e, "") + if d: + return d + return "" + + +def _probe_gpu(ffmpeg_bin: str) -> GpuInfo: + """Detect NVIDIA NVENC hardware encoders and verify they actually work. + + Two-stage probe: + + 1. Compiled-in check — grep ``ffmpeg -encoders`` for the NVENC + encoder names. Cheap; answers "could this ffmpeg ever do NVENC". + 2. Functional smoke test — for each compiled-in encoder, encode a + 0.2s 256x264 lavfi color source with ``-c:v -f null -``. + Catches the real-world failure modes the compiled-in check + cannot: NVIDIA driver too old for the ffmpeg build's NVENC API + version, no /dev/nvidia* access, driver loaded but GPU dead. + + GPU model name is best-effort via nvidia-smi (display only). + """ + info = GpuInfo() + + if not ffmpeg_bin: + return info + + # --- Stage 1: compiled-in encoders --- + try: + res = subprocess.run( + [ffmpeg_bin, "-hide_banner", "-encoders"], + capture_output=True, text=True, timeout=10, + ) + encoders_out = res.stdout or "" + except (OSError, subprocess.SubprocessError): + encoders_out = "" + + for enc in _NVENC_ENCODER_NAMES: + info.encoders[enc] = f" {enc} " in encoders_out + + compiled_in = [e for e in _NVENC_ENCODER_NAMES if info.encoders[e]] + if not compiled_in: + return info # no hardware encoders in this build — skip stage 2 + + # --- GPU model name (display + profile auto-match) --- + nvidia_smi = shutil.which("nvidia-smi") + if nvidia_smi: + try: + res = subprocess.run( + [nvidia_smi, "--query-gpu=name", "--format=csv,noheader"], + capture_output=True, text=True, timeout=5, + ) + if res.returncode == 0 and res.stdout.strip(): + info.name = res.stdout.strip().splitlines()[0].strip() + except (OSError, subprocess.SubprocessError): + pass + if not info.name: + # No NVIDIA board (or driver down): identify AMD/Intel iGPU/dGPU + # from the PCI bus for profile matching + VAAPI/QSV testing. + info.name = _probe_pci_gpu_name() + + # --- v4.8.0: capability-class profile for this GPU --- + matched = match_gpu_profile(info.name) + if matched: + info.profile_key = matched.key + + # The profile may claim hardware encoders this ffmpeg build + # doesn't even list (e.g. AV1 on an Arc card with an old ffmpeg). + for family, enc in matched.encoders.items(): + if enc not in info.encoders or not info.encoders[enc]: + info.encoders[enc] = _ffmpeg_has_encoder(ffmpeg_bin, enc) + + # --- Stage 2 for the profile's hardware API --- + if matched.api == "vaapi": + info.functional.update(_smoke_vaapi(ffmpeg_bin, matched)) + elif matched.api == "qsv": + info.functional.update(_smoke_qsv(ffmpeg_bin, matched)) + + # --- Stage 2: functional smoke test per compiled-in encoder --- + for enc in compiled_in: + if enc in info.functional: + continue # already smoke-tested via the profile branch + try: + res = subprocess.run( + [ + ffmpeg_bin, "-hide_banner", "-loglevel", "error", + "-f", "lavfi", + "-i", "color=c=black:s=256x256:d=0.2:r=24", + "-frames:v", "5", + "-c:v", enc, "-f", "null", "-", + ], + capture_output=True, text=True, timeout=20, + ) + info.functional[enc] = res.returncode == 0 + if res.returncode != 0: + # First stderr line with substance (nvenc errors are + # prefixed "hevc_nvenc @ 0x...]" — keep them readable). + for line in (res.stderr or "").splitlines(): + line = line.strip() + if line: + # Strip the "name @ 0xADDR]" prefix for brevity. + line = re.sub(r"^\[[^]]+@\s*0x[0-9a-f]+\]\s*", "", line) + info.details[enc] = line[:160] + break + except subprocess.TimeoutExpired: + info.functional[enc] = False + info.details[enc] = f"{enc} smoke test timed out after 20s" + except (OSError, subprocess.SubprocessError) as e: + info.functional[enc] = False + info.details[enc] = str(e)[:160] + + return info + + +def _ffmpeg_has_encoder(ffmpeg_bin: str, enc: str) -> bool: + """Compiled-in check for one encoder name (cheap -encoders grep).""" + try: + res = subprocess.run( + [ffmpeg_bin, "-hide_banner", "-encoders"], + capture_output=True, text=True, timeout=10, + ) + return f" {enc} " in (res.stdout or "") + except (OSError, subprocess.SubprocessError): + return False + + +def _probe_pci_gpu_name() -> str: + """Best-effort non-NVIDIA GPU name via lspci (VGA/3D/Display class).""" + lspci = shutil.which("lspci") + if not lspci: + return "" + try: + res = subprocess.run( + [lspci], capture_output=True, text=True, timeout=10, + ) + for line in (res.stdout or "").splitlines(): + low = line.lower() + if any(k in low for k in (" vga ", " 3d ", " display ")): + if "nvidia" not in low: # nvidia handled via nvidia-smi + # "01:00.0 VGA ...: AMD/ATI Navi 31 [Radeon RX 7900 XTX]" + m = re.search(r":\s*(.+)$", line) + return m.group(1).strip() if m else "" + except (OSError, subprocess.SubprocessError): + pass + return "" + + +def _smoke_vaapi(ffmpeg_bin: str, profile) -> dict[str, bool]: + """Live-encode VAAPI encoders through the render node. VAAPI + encoders only accept hardware surfaces, so the filter chain must + upload the lavfi frames.""" + results: dict[str, bool] = {} + dev = resolve_vaapi_device() + if not Path(dev).exists(): + for enc in profile.encoders.values(): + results[enc] = False + details_placeholder = f"no render node ({dev})" + return results + for enc in profile.encoders.values(): + try: + cmd = ( + [ffmpeg_bin, "-hide_banner", "-loglevel", "error"] + + encoder_pre_args(profile, vaapi_device=dev) + + ["-f", "lavfi", "-i", "color=c=black:s=256x256:d=0.2:r=24", + "-frames:v", "5"] + + encoder_filter_chain(profile) + + ["-c:v", enc] + + encoder_quality_args(profile.api, enc, 28, 9) + + ["-f", "null", "-"] + ) + res = subprocess.run(cmd, capture_output=True, text=True, timeout=20) + results[enc] = res.returncode == 0 + if res.returncode != 0: + for line in (res.stderr or "").splitlines(): + line = line.strip() + if line: + results[f"_detail_{enc}"] = False # marker only + break + except (OSError, subprocess.SubprocessError): + results[enc] = False + # Strip the detail markers; keep real encoder results only, and put + # first error lines into a readable form for the caller. + cleaned: dict[str, bool] = {} + for k, v in results.items(): + if not k.startswith("_detail_"): + cleaned[k] = v + return cleaned + + +def _smoke_qsv(ffmpeg_bin: str, profile) -> dict[str, bool]: + """Live-encode QSV encoders (qsv encoders upload system frames + internally once a qsv device exists).""" + results: dict[str, bool] = {} + for enc in profile.encoders.values(): + try: + cmd = ( + [ffmpeg_bin, "-hide_banner", "-loglevel", "error"] + + encoder_pre_args(profile) + + ["-f", "lavfi", "-i", "color=c=black:s=256x256:d=0.2:r=24", + "-frames:v", "5", + "-c:v", enc] + + encoder_quality_args(profile.api, enc, 28, 9) + + ["-f", "null", "-"] + ) + res = subprocess.run(cmd, capture_output=True, text=True, timeout=20) + results[enc] = res.returncode == 0 + except (OSError, subprocess.SubprocessError): + results[enc] = False + return results + # ────────────────────────────────────────────── # ENVIRONMENT PROBE (distro-aware, extended) @@ -52,6 +318,9 @@ class EnvProbe: vs_version: str | None = None # VapourSynth version string (for diagnostics) vs_script_lib: str | None = None # path to libvapoursynth-script.so that passed cpu: CpuTopology = field(default_factory=lambda: CpuTopology(1, 1, 1, "Unknown")) + # v4.6.0: GPU/NVENC probe result. EncoderWorker reads + # env.gpu.functional[encoder_name] when the engine is auto/gpu. + gpu: GpuInfo = field(default_factory=GpuInfo) errors: list[str] = field(default_factory=list) warnings: list[str] = field(default_factory=list) @@ -141,6 +410,11 @@ def _probe_ffmpeg_libs(ffmpeg_bin: str) -> dict[str, bool]: ("libopus", ["libopus "]), ("libvorbis", ["libvorbis "]), ("flac", ["flac "]), + # v4.6.0: NVENC hardware encoders (compiled-in check only — the + # live-encode gate is _probe_gpu()'s functional dict). + ("hevc_nvenc", ["hevc_nvenc "]), + ("h264_nvenc", ["h264_nvenc "]), + ("av1_nvenc", ["av1_nvenc "]), ] libs = {} for lib_name, search_strings in checks: @@ -440,6 +714,36 @@ def probe_environment() -> EnvProbe: result.warnings.append(f"FFmpeg version: {result.ffmpeg_version}") result.ffmpeg_libs = _probe_ffmpeg_libs(result.ffmpeg_path) + # v4.6.0: GPU/NVENC probe — compiled-in check + live encode smoke + # test. EncoderWorker gates the GPU path on env.gpu.functional; + # this warning block surfaces the result (and the fix when the + # driver is too old for the ffmpeg build's NVENC API). + result.gpu = _probe_gpu(result.ffmpeg_path) + gpu = result.gpu + if gpu.has_gpu: + result.warnings.append( + f"GPU: {gpu.name or 'NVIDIA'} — NVENC ready: " + f"{', '.join(gpu.usable_encoders)} (engine: Auto will use the GPU)" + ) + elif gpu.encoders and any(gpu.encoders.values()): + present = [e for e in _NVENC_ENCODER_NAMES if gpu.encoders.get(e)] + detail = gpu.first_failure_detail + result.warnings.append( + f"GPU: NVENC encoder(s) {', '.join(present)} present in ffmpeg " + f"but NOT usable — {detail or 'smoke test failed'}. " + f"Auto engine will fall back to CPU." + ) + if "API version" in detail or "minimum required Nvidia driver" in detail: + result.warnings.append( + " FIX: update the NVIDIA driver (the ffmpeg build's NVENC " + "API is newer than the installed driver supports), or use " + "an ffmpeg build matching the installed driver." + ) + else: + result.warnings.append( + "GPU: no hardware encoder in this ffmpeg build — CPU encoding." + ) + # Warn about missing AUDIO libs (video codecs are handled by av1an's own # encoder binaries — ffmpeg's video encoder list is irrelevant) audio_lib_warnings = { @@ -533,6 +837,31 @@ def probe_environment() -> EnvProbe: ) result.av1an_flags["chunk_method_override"] = "select" + # v4.6.0: ffmpeg ≥ 7 removed the -vsync option that av1an's + # segment/hybrid chunk extraction passes to ffmpeg. On those + # systems every segment-based chunk dies immediately with + # "Unrecognized option 'vsync'." — the y4m pipe breaks and + # each chunk fails 3x. The select override above already + # avoids those methods when no plugins are installed; this + # warning tells plugin-less users on new ffmpeg WHY av1an is + # stuck on slow select. + if not vs_plugins and result.ffmpeg_version: + try: + ffmpeg_major = int( + re.match(r"[nN]?(\d+)", result.ffmpeg_version).group(1) + ) + except (AttributeError, ValueError): + ffmpeg_major = 0 + if ffmpeg_major >= 7: + result.warnings.append( + "av1an note: ffmpeg ≥ 7 removed the -vsync option av1an's " + "segment/hybrid chunk methods use — those methods fail " + "with \"Unrecognized option 'vsync'\". Chunking stays on " + "'select'. Install a VapourSynth source plugin " + "(bestsource/ffms2/lsmash) to escape slow select, or use " + "the default ffmpeg-only path." + ) + except (OSError, subprocess.SubprocessError) as e: result.errors.append(f"av1an probe failed: {e}") @@ -638,12 +967,29 @@ def _av1an_env() -> dict[str, str]: When VapourSynth is built from git and installed to ~/.local/, the linker won't find libvapoursynth-script.so unless LD_LIBRARY_PATH points there. This function ensures every av1an invocation inherits that path. + + v4.7.1: the git VS stack is self-contained in the python user + site-packages (module + libs + BestSource plugin), so the runtime env + also gets that dir on LD_LIBRARY_PATH and the user site on PYTHONPATH — + otherwise av1an loads the system VS and never sees the fresh stack. """ env = os.environ.copy() local_lib = str(Path.home() / ".local" / "lib") existing = env.get("LD_LIBRARY_PATH", "") if local_lib not in existing: env["LD_LIBRARY_PATH"] = f"{local_lib}:{existing}".rstrip(":") + try: + user_site = Path(site.getusersitepackages()) + vs_dir = user_site / "vapoursynth" + if vs_dir.is_dir() and (vs_dir / "libvsscript.so").exists(): + existing = env.get("LD_LIBRARY_PATH", "") + if str(vs_dir) not in existing: + env["LD_LIBRARY_PATH"] = f"{vs_dir}:{existing}".rstrip(":") + py_path = env.get("PYTHONPATH", "") + if str(user_site) not in py_path: + env["PYTHONPATH"] = f"{user_site}:{py_path}".rstrip(":") + except (AttributeError, OSError): + pass return env @@ -697,6 +1043,18 @@ def _probe_vs_source_plugins() -> list[str]: search_dirs.append(Path("/usr/lib/vapoursynth")) search_dirs.append(Path("/usr/local/lib/vapoursynth")) search_dirs.append(Path("/usr/lib/x86_64-linux-gnu/vapoursynth")) + # v4.7.1: the git-built VapourSynth stack installs its plugins into + # the python site-packages tree (module + libs + plugins/ are one + # self-contained unit). Probe those dirs too. + try: + search_dirs.append(Path(site.getusersitepackages()) / "vapoursynth" / "plugins") + except (AttributeError, OSError): + pass + try: + for d in site.getsitepackages(): + search_dirs.append(Path(d) / "vapoursynth" / "plugins") + except (AttributeError, OSError): + pass found: set[str] = set() for d in search_dirs: diff --git a/opentranscode/gpu_profiles.py b/opentranscode/gpu_profiles.py new file mode 100644 index 0000000..fea289f --- /dev/null +++ b/opentranscode/gpu_profiles.py @@ -0,0 +1,237 @@ +"""GPU capability profiles (v4.8.0) — combined card generations. + +Cards within the same hardware-encoder generation are functionally +identical for transcoding, so the dropdown lists CAPABILITY CLASSES, +not individual SKUs: one Pascal entry covers the GTX 10-series, Tesla +P40/P4/P100 and mobile chips; one Turing entry covers RTX 20-series, +GTX 16-series, the Tesla T4 and the crypto-era CMP 30/40/50HX cards. + +Oddballs are included with their real capabilities: + - CMP 90HX is GA102-based (Ampere NVENC), but CMP 170HX is GA100-based + and has NO NVENC at all (like A100/V100/H100 compute boards). + - Intel Arc (QSV) and AMD RDNA (VAAPI) cover the rest of the trending + list; RDNA 3 added AV1 encode, RDNA 1/2 and GCN can only encode + H.264/HEVC. + +Pure data + pure functions: no I/O, safe to import anywhere. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass + +# ────────────────────────────────────────────── +# GPU PROFILES +# ────────────────────────────────────────────── + +@dataclass(frozen=True) +class GpuProfile: + key: str # stable id (CLI/UI) + label: str # dropdown entry + vendor: str # nvidia | amd | intel | none + api: str # nvenc | vaapi | qsv | none + # codec family → ffmpeg encoder name + encoders: dict[str, str] + # case-insensitive substrings matched against the detected GPU name + # (nvidia-smi / lspci) for auto-detection. First match wins; lists + # are ordered most-specific first. + match: tuple[str, ...] = () + notes: str = "" + # extra args that must come BEFORE -i (hardware device init) + hw_device_args: tuple[str, ...] = () + # filter-chain fragment required before the encoder (vaapi hwupload) + filter_tail: tuple[str, ...] = () + + +# NVENC encoders by generation class. Quality control: -rc vbr -cq N +# (-b:v 0). 8-bit yuv420p everywhere — Pascal 10-bit HEVC runs at half +# speed and the archival targets here are 8-bit sources. +_NV = {"h264": "h264_nvenc", "hevc": "hevc_nvenc"} + +GPU_PROFILES: list[GpuProfile] = [ + GpuProfile( + key="nv-kepler-maxwell", + label="NVIDIA Kepler / Maxwell 1.0 (GTX 600/700/800M) — H.264 only", + vendor="nvidia", api="nvenc", + encoders={"h264": "h264_nvenc"}, + match=("GTX 6", "GTX 7", "GT 7", "GTX 8", "GT 8", "840M", "860M", "750"), + notes="First NVENC generations: H.264 only, no HEVC.", + ), + GpuProfile( + key="nv-pascal", + label="NVIDIA Pascal (GTX 10-series, TITAN Xp, Tesla P40/P4/P100) — H.264 + HEVC 8/10-bit", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("GTX 10", "1070", "1080", "1060", "1050", "TITAN Xp", + "Tesla P40", "Tesla P4", "P100", "Quadro P"), + notes="Pascal NVENC: HEVC Main/Main10. 10-bit runs at ~half speed.", + ), + GpuProfile( + key="nv-turing", + label="NVIDIA Turing (RTX 20-series, GTX 16-series, Tesla T4, CMP 30/40/50HX) — H.264 + HEVC + B-frames", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("RTX 20", "GTX 16", "2060", "2070", "2080", "1660", "1650", + "Tesla T4", "CMP 30", "CMP 40", "CMP 50"), + notes="Turing NVENC: first gen with HEVC B-frames; big quality jump.", + ), + GpuProfile( + key="nv-compute", + label="NVIDIA data-center compute (V100/A100/H100, CMP 170HX) — no NVENC (CPU path)", + vendor="nvidia", api="none", + encoders={}, + match=("V100", "A100", "H100", "B200", "GB200", "CMP 170"), + notes="Compute boards ship without NVENC silicon. CMP 170HX is " + "GA100-based — the fastest mining card that cannot hardware-encode.", + ), + GpuProfile( + key="nv-ampere", + label="NVIDIA Ampere (RTX 30-series, A10/A40/A2, CMP 90HX) — H.264 + HEVC (no AV1 encode)", + vendor="nvidia", api="nvenc", + encoders=dict(_NV), + match=("RTX 30", "3090", "3080", "3070", "3060", "3050", + "A10", "A40", "CMP 90"), + notes="Ampere added AV1 DECODE but not encode — AV1 stays on CPU.", + ), + GpuProfile( + key="nv-ada", + label="NVIDIA Ada / Blackwell (RTX 40/50-series, L4/L40) — H.264 + HEVC + AV1 10-bit", + vendor="nvidia", api="nvenc", + encoders={"h264": "h264_nvenc", "hevc": "hevc_nvenc", "av1": "av1_nvenc"}, + match=("RTX 40", "RTX 50", "4090", "4080", "4070", "4060", + "5090", "5080", "5070", "5060", "L4", "L40"), + notes="Ada introduced AV1 NVENC; Blackwell doubles AV1 throughput.", + ), + GpuProfile( + key="intel-arc", + label="Intel Arc (Alchemist A-series, Battlemage B-series) — QSV: H.264 + HEVC + AV1", + vendor="intel", api="qsv", + encoders={"h264": "h264_qsv", "hevc": "hevc_qsv", "av1": "av1_qsv"}, + match=("Arc A", "Arc B", "A380", "A750", "A770", "B570", "B580"), + notes="Arc media engines encode AV1 8/10-bit — best value encode card.", + ), + GpuProfile( + key="intel-xe", + label="Intel Iris / UHD integrated (Gen9–Xe) — QSV: H.264 + HEVC", + vendor="intel", api="qsv", + encoders={"h264": "h264_qsv", "hevc": "hevc_qsv"}, + match=("Iris", "UHD", "HD Graphics"), + notes="Integrated media engines; HEVC 8/10-bit, no AV1 encode.", + ), + GpuProfile( + key="amd-rdna3", + label="AMD RDNA 3 (RX 7000-series) — VAAPI: H.264 + HEVC + AV1", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi", "av1": "av1_vaapi"}, + match=("RX 7", "7900", "7800", "7700", "7600"), + notes="RDNA 3 VCN: first AMD generation with AV1 encode.", + ), + GpuProfile( + key="amd-rdna12", + label="AMD RDNA 1/2 (RX 5000/6000-series) — VAAPI: H.264 + HEVC (AV1 decode only)", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"}, + match=("RX 5", "RX 6", "5700", "5600", "6800", "6700", "6600", "6500"), + notes="RDNA 2 has AV1 decode only — AV1 encode stays on CPU.", + ), + GpuProfile( + key="amd-gcn", + label="AMD GCN 4/5 / Vega (RX 400/500, Vega 56/64) — VAAPI: H.264 + HEVC", + vendor="amd", api="vaapi", + encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"}, + match=("RX 4", "RX 5", "Vega", "580", "570", "480", "470", "64", "56"), + notes="The classic crypto-era mining cards (Polaris/Vega).", + ), + GpuProfile( + key="cpu", + label="None (CPU-only encode)", + vendor="none", api="none", + encoders={}, + ), +] + +_GPU_PROFILES_BY_KEY: dict[str, GpuProfile] = {p.key: p for p in GPU_PROFILES} + + +def gpu_profile_by_key(key: str | None) -> GpuProfile | None: + if not key: + return None + return _GPU_PROFILES_BY_KEY.get(key) + + +def match_gpu_profile(gpu_name: str) -> GpuProfile | None: + """Best-effort auto-detection from a GPU name string (nvidia-smi or + lspci output). Case-insensitive; first matching profile wins (the + match lists are ordered most-specific first, and the compute boards + are matched before the consumer generations they share names with — + e.g. 'CMP 170HX' must not hit the Ampere 'A10' style entries).""" + if not gpu_name: + return None + name = gpu_name.lower() + for profile in GPU_PROFILES: + for frag in profile.match: + if frag.lower() in name: + return profile + return None + + +# ────────────────────────────────────────────── +# FFMPEG ARG HELPERS (per hardware API) +# ────────────────────────────────────────────── + +def encoder_for_family(profile: GpuProfile | None, family: str) -> str | None: + """Hardware encoder name for a codec family on this profile, or None.""" + if not profile or profile.api == "none": + return None + return profile.encoders.get(family) + + +def resolve_vaapi_device() -> str: + """First render node, or the classic fallback path. (Best-effort I/O — + callers that need purity pass the result into encoder_pre_args.)""" + import glob + nodes = sorted(glob.glob("/dev/dri/renderD*")) + return nodes[0] if nodes else "/dev/dri/renderD128" + + +def encoder_pre_args(profile: GpuProfile, vaapi_device: str | None = None) -> list[str]: + """Args that must precede -i (hardware device initialisation).""" + if profile.api == "vaapi": + dev = vaapi_device or resolve_vaapi_device() + return ["-vaapi_device", dev] + if profile.api == "qsv": + return ["-init_hw_device", "qsv=hw"] + return [] + + +def encoder_filter_chain(profile: GpuProfile) -> list[str]: + """Filter args that upload software frames to the hardware surface + format (VAAPI encoders only accept hw frames; -vaapi_device makes + its device the default for hwupload).""" + if profile.api == "vaapi": + return ["-vf", "format=nv12,hwupload"] + return [] + + +def encoder_quality_args(api: str, encoder: str, crf: int, preset: int) -> list[str]: + """Constant-quality args for a hardware encoder. NVENC maps the CPU + preset tiers to p-presets; QSV uses very_fast/medium; VAAPI uses CQP + rate mode which has no preset knob.""" + if api == "nvenc": + if preset <= 6: + p = "p7" + elif preset <= 8: + p = "p5" + else: + p = "p4" + return ["-preset", p, "-tune", "hq", "-rc", "vbr", + "-cq", str(crf), "-b:v", "0", "-pix_fmt", "yuv420p", + "-g", "240"] + if api == "qsv": + p = "veryslow" if preset <= 6 else ("medium" if preset <= 8 else "very_fast") + return ["-preset", p, "-global_quality", str(crf), + "-pix_fmt", "yuv420p", "-g", "240"] + if api == "vaapi": + return ["-rc_mode", "CQP", "-qp", str(crf), "-g", "240"] + return [] diff --git a/opentranscode/hybrid_scheduler.py b/opentranscode/hybrid_scheduler.py new file mode 100644 index 0000000..fffec19 --- /dev/null +++ b/opentranscode/hybrid_scheduler.py @@ -0,0 +1,115 @@ +"""Hybrid GPU+CPU batch scheduler (v4.7.0). + +Splits a transcode queue between two concurrent lanes so the NVENC +engine and the CPU encoders work at the same time instead of leaving +28 Xeon threads idle while the GPU encode runs: + + - GPU lane: files encoded via the functional NVENC encoder + (hevc_nvenc / av1_nvenc) through the single-pass ffmpeg path. + - CPU lane: the remaining files through the family's software encoder + (libx265 / libsvtav1 / libvpx) or av1an chunk-parallel if the user + opted in — so "all three" (NVENC + software + chunk-parallel) run + side by side when av1an is enabled. + +Pure planning logic — no Qt, no I/O beyond the caller-provided file +sizes. The UI turns a HybridPlan into two EncoderWorker instances. + +Scope note: lanes split FILES, never one file across encoders. Splitting +a single file between hevc_nvenc and libx265 chunks would produce +visibly inconsistent quality between scenes, and av1an cannot drive +NVENC at all (it spawns encoder CLI binaries only). +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +# NVENC HEVC on a Pascal card runs several times faster than x265 +# "faster" on 28 Xeon threads. Used only for load ESTIMATION (which lane +# gets the next file) — a wrong guess just skews the split slightly. +GPU_SPEED_RATIO_DEFAULT = 8 + +# Threads held back from the CPU lane so the GPU lane's decode / scale / +# mux processes stay responsive. The NVENC encode itself runs on the GPU +# silicon; the CPU side of a nvenc job is light. +HYBRID_CPU_RESERVE_THREADS = 2 + + +@dataclass +class HybridPlan: + """Result of planning a hybrid (GPU + CPU) queue split.""" + + gpu_files: list[Path] = field(default_factory=list) + cpu_files: list[Path] = field(default_factory=list) + gpu_encoder: str = "" # e.g. "hevc_nvenc" + cpu_budget_threads: int = 1 # CPU lane thread budget (logical - reserve) + gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT + + @property + def total_files(self) -> int: + return len(self.gpu_files) + len(self.cpu_files) + + +def plan_hybrid( + files: list[Path], + gpu_encoder: str | None, + gpu_functional: bool, + logical_threads: int, + sizes: dict[Path, int] | None = None, + gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT, + cpu_reserve: int = HYBRID_CPU_RESERVE_THREADS, +) -> HybridPlan | None: + """Split *files* between the GPU and CPU lanes, or return None when a + hybrid split cannot apply. + + Returns None when: + - the codec family has no GPU encoder, or the live GPU probe failed + (caller should fall back to a plain CPU queue), or + - *files* is empty. + + Assignment is LPT (longest-processing-time first): files are sorted + by size descending and each goes to the lane with the lower + estimated load, where the GPU lane's per-file cost is size / + gpu_speed_ratio. Both lanes then finish at roughly the same time. + + *sizes* maps files to byte sizes; missing entries fall back to the + mean of the known sizes (or 10 MB when nothing is known) so a single + unreadable file cannot skew the whole split. + """ + if not gpu_encoder or not gpu_functional or not files: + return None + + sizes = sizes or {} + known = [s for s in sizes.values() if s] + avg = sum(known) // len(known) if known else 10_000_000 + + def size_of(f: Path) -> int: + return sizes.get(f) or avg + + ratio = max(1, int(gpu_speed_ratio)) + gpu_files: list[Path] = [] + cpu_files: list[Path] = [] + gpu_load = 0.0 + cpu_load = 0.0 + + for f in sorted(files, key=size_of, reverse=True): + s = size_of(f) + gpu_est = gpu_load + s / ratio + cpu_est = cpu_load + s + # Tie goes to the GPU lane — it finishes the file sooner and the + # CPU lane keeps its current file longer. + if gpu_est <= cpu_est: + gpu_files.append(f) + gpu_load = gpu_est + else: + cpu_files.append(f) + cpu_load = cpu_est + + return HybridPlan( + gpu_files=gpu_files, + cpu_files=cpu_files, + gpu_encoder=gpu_encoder, + cpu_budget_threads=max(1, int(logical_threads) - cpu_reserve), + gpu_speed_ratio=ratio, + ) diff --git a/opentranscode/source_builder.py b/opentranscode/source_builder.py index 0253bb3..cd6b5ef 100755 --- a/opentranscode/source_builder.py +++ b/opentranscode/source_builder.py @@ -5,19 +5,137 @@ components from source. Installs to the user's home dir (no sudo for the install step). v3-08 made this worker stop mutating ``os.environ`` directly — it carries its own ``_build_env`` snapshot. -Pure stdlib + PySide6 (no internal package dependencies). +v4.7.1: the rebuild is ALWAYS usable, even on a bare system. It +generates its own dependency tree: missing build tools and libraries +are installed via the distro package manager (arch/debian/redhat/suse) +before anything is compiled, and the VapourSynth build is followed by +a BestSource plugin build (submodules + vapoursynth dev headers from +the freshly installed VS) so av1an gets a fast, reliable chunk method +instead of the slow "select" fallback. """ import os import re import shutil import signal +import site import subprocess import time +from dataclasses import dataclass, field from pathlib import Path from PySide6.QtCore import QThread, Signal +from .distro_probe import DistroProfile, detect_distro +from .gpu_profiles import gpu_profile_by_key + +# ────────────────────────────────────────────── +# BUILD DEPENDENCY TREE (v4.7.1 — distro-aware) +# ────────────────────────────────────────────── + +# Binaries the build needs. pkgconf/pkg-config and python/python3 are +# aliased — any one of each pair satisfies the check. +BUILD_TOOL_ALIASES: dict[str, tuple[str, ...]] = { + "git": ("git",), + "meson": ("meson",), + "ninja": ("ninja",), + "c++ compiler": ("g++", "c++", "clang++"), + "make": ("make",), + "pkg-config": ("pkg-config", "pkgconf"), + "python3": ("python3",), + "nasm": ("nasm",), + "cmake": ("cmake",), +} + +# Packages providing the toolchain + the libraries the builds link +# against (zimg is VapourSynth's one hard library dependency; rust is +# only needed for the av1an build). +BUILD_DEPS_BY_FAMILY: dict[str, list[str]] = { + "arch": ["base-devel", "meson", "ninja", "cmake", "nasm", "git", + "python", "pkgconf", "zimg", "rust"], + "debian": ["build-essential", "meson", "ninja-build", "cmake", "nasm", + "git", "python3", "python3-dev", "pkg-config", "libzimg-dev", + "cargo", "rustc"], + "redhat": ["gcc", "gcc-c++", "make", "meson", "ninja-build", "cmake", + "nasm", "git", "python3", "python3-devel", + "pkgconf-pkg-config", "zimg-devel", "cargo", "rust"], + "suse": ["gcc", "gcc-c++", "make", "meson", "ninja", "cmake", "nasm", + "git", "python3", "python3-devel", "pkg-config", + "zimg-devel", "rust", "cargo"], +} + +PKG_INSTALL_CMD: dict[str, list[str]] = { + "arch": ["pacman", "-S", "--needed", "--noconfirm"], + "debian": ["apt-get", "install", "-y"], + "redhat": ["dnf", "install", "-y"], + "suse": ["zypper", "--non-interactive", "install"], +} + +MANUAL_DEP_NOTE = ( + "No automatic package install for this distro family. Install a C++ " + "toolchain plus meson, ninja, cmake, nasm, git, python3, pkg-config, " + "zimg development headers{rust} manually, then press REBUILD again." +) + +# v4.8.0: per-GPU-profile build/runtime packages, on top of the base +# toolchain. nvidia needs nv-codec-headers at ffmpeg build time (the +# distro ffmpeg already ships nvenc; a matched git build needs the +# headers); vaapi/qsv need the driver + dev stacks for their vendor. +GPU_BUILD_PACKAGES: dict[str, dict[str, list[str]]] = { + "nvenc": { + "arch": ["nv-codec-headers"], + "debian": [], + "redhat": [], + "suse": [], + }, + "vaapi": { + "arch": ["libva", "libdrm", "mesa"], + "debian": ["libva-dev", "libdrm-dev", "mesa-va-drivers"], + "redhat": ["libva-devel", "libdrm-devel", "mesa-va-drivers"], + "suse": ["libva-devel", "libdrm-devel", "Mesa-libva"], + }, + "qsv": { + "arch": ["libva", "intel-media-driver", "onevpl"], + "debian": ["libva-dev", "intel-media-va-driver-non-free", "libvpl-dev"], + "redhat": ["libva-devel", "intel-media-driver", "oneVPL-devel"], + "suse": ["libva-devel", "intel-media-driver", "oneVPL-devel"], + }, +} + + +def gpu_dep_packages(api: str, distro_family: str) -> list[str]: + """Extra packages for a GPU hardware API on this distro (empty when + the family has no packaged set — the log says so).""" + return list(GPU_BUILD_PACKAGES.get(api, {}).get(distro_family, [])) + + +@dataclass +class DepPlan: + """What the rebuild needs, and how to get it on this distro.""" + packages: list[str] = field(default_factory=list) + install_cmd: list[str] | None = None + manual_note: str | None = None + + +def build_dep_plan(distro: DistroProfile, build_av1an: bool = True) -> DepPlan: + """Pure: the package list + install command for this distro family. + + Works even when the environment probe failed — it only needs the + distro family, which is detectable from /etc/os-release alone. + """ + packages = list(BUILD_DEPS_BY_FAMILY.get(distro.family, [])) + if not build_av1an: + for rust_pkg in ("rust", "rustc", "cargo"): + if rust_pkg in packages: + packages.remove(rust_pkg) + install_cmd = PKG_INSTALL_CMD.get(distro.family) + manual_note = None + if not install_cmd or not packages: + manual_note = MANUAL_DEP_NOTE.format( + rust=" and Rust/cargo" if build_av1an else "") + return DepPlan(packages=packages, install_cmd=install_cmd, + manual_note=manual_note) + # ────────────────────────────────────────────── # SOURCE BUILD WORKER — compile VS + av1an from git # ────────────────────────────────────────────── @@ -37,11 +155,14 @@ class SourceBuildWorker(QThread): build_done = Signal(bool, str) # (success, detail) def __init__(self, build_vs: bool = True, build_av1an: bool = True, - build_ffmpeg_iamf: bool = False): + build_ffmpeg_iamf: bool = False, gpu_profile_key: str = ""): super().__init__() self.build_vs = build_vs self.build_av1an = build_av1an self.build_ffmpeg_iamf = build_ffmpeg_iamf + # v4.8.0: selected GPU capability profile — extends the dep tree + # with the vendor's build/runtime packages. + self.gpu_profile_key = gpu_profile_key self._stop = False # Private per-worker environment snapshot. Mutating os.environ is # process-global and leaks across threads/subsequent subprocesses; @@ -98,30 +219,71 @@ class SourceBuildWorker(QThread): def run(self): try: - # ── Install build dependencies (may need one sudo prompt) ── + # ── v4.7.1: distro-aware dependency tree ── + # The rebuild must work on a bare system: detect missing + # tools/libraries and install them via the distro package + # manager (one privilege prompt via pkexec/sudo) BEFORE + # compiling anything. self.log_msg.emit("") - self.log_msg.emit("=== Installing build dependencies ===") - all_deps = [ - "meson", "ninja", "gcc", "pkg-config", "git", - "nasm", "yasm", "cmake", "python", "make", - ] - need_rust = self.build_av1an and not shutil.which("cargo") - if need_rust: - all_deps.append("rust") + self.log_msg.emit("=== Generating dependency tree ===") + self._distro = detect_distro() + self.log_msg.emit( + f" Distro: {self._distro.name} (family={self._distro.family})" + ) + plan = build_dep_plan(self._distro, build_av1an=self.build_av1an) + + # v4.8.0: GPU-profile packages on top of the base toolchain. + gpu_profile = gpu_profile_by_key(self.gpu_profile_key) + if gpu_profile is not None and gpu_profile.api != "none": + gpu_pkgs = gpu_dep_packages(gpu_profile.api, self._distro.family) + if gpu_pkgs: + self.log_msg.emit( + f" GPU profile {gpu_profile.key} ({gpu_profile.api}): " + f"+{len(gpu_pkgs)} package(s)" + ) + plan.packages.extend(p for p in gpu_pkgs + if p not in plan.packages) + + missing = self._missing_build_tools() + zimg_ok = self._pkgconfig_exists("zimg") + if zimg_ok: + self.log_msg.emit(" OK: zimg (VapourSynth dependency)") + else: + missing.append("zimg (library, via pkg-config)") + if self.build_av1an and not shutil.which("cargo"): + missing.append("cargo (rust)") - # Only invoke sudo if at least one dep is missing - missing = [d for d in all_deps if not shutil.which(d)] if missing: - self.log_msg.emit(f" Missing: {', '.join(missing)} — installing via pacman") - rc, _ = self._sudo_cmd( - ["pacman", "-S", "--needed", "--noconfirm"] + all_deps, - timeout=300, label="pacman build-deps", - ) - if rc != 0: - self.log_msg.emit(" (some deps may already be installed — continuing)") + self.log_msg.emit(f" Missing: {', '.join(missing)}") + if plan.install_cmd: + self.log_msg.emit( + f" Installing {len(plan.packages)} package(s) via " + f"{plan.install_cmd[0]} (privilege prompt possible)..." + ) + rc, _ = self._sudo_cmd( + plan.install_cmd + plan.packages, + timeout=900, label=f"{plan.install_cmd[0]} build-deps", + ) + if rc != 0: + self.log_msg.emit( + " (install reported an error — continuing; " + "some packages may already be present)" + ) + else: + self.log_msg.emit(f" {plan.manual_note}") else: self.log_msg.emit(" All build dependencies already installed.") + # Re-verify the critical tools after install. + still_missing = self._missing_build_tools() + if still_missing: + self.log_msg.emit( + f" FATAL: still missing after install: {', '.join(still_missing)}. " + f"Install them manually and press REBUILD again." + ) + self.build_done.emit(False, f"missing build tools: {still_missing}") + return + # Ensure cargo is in PATH after potential install. # NOTE: /root/.cargo/bin was dropped (OTC-015/v3-08) — root's # cargo dir is not readable by a non-root user. ~/.cargo/bin @@ -142,6 +304,9 @@ class SourceBuildWorker(QThread): # ── Build & install VapourSynth to ~/.local (NO sudo needed) ── if self.build_vs: self._build_vapoursynth() + # v4.7.1: BestSource right after VS, compiled against the + # fresh VS headers — gives av1an a fast chunk method. + self._build_bestsource() # ── Build av1an to ~/.cargo/bin (NO sudo needed) ── if self.build_av1an: @@ -152,12 +317,25 @@ class SourceBuildWorker(QThread): self._build_libiamf() self._build_ffmpeg_with_iamf() - # ── Ensure LD_LIBRARY_PATH includes local VS libs ── + # ── Ensure the runtime env can find the fresh VS stack ── + # The git VapourSynth installs self-contained into the user + # site-packages (module + libs + plugins). av1an dlopens + # libvapoursynth-script from there, so both LD_LIBRARY_PATH + # and PYTHONPATH must include it. local_lib = str(Path.home() / ".local" / "lib") existing_ld = self._build_env.get("LD_LIBRARY_PATH", "") if local_lib not in existing_ld: self._extend_env("LD_LIBRARY_PATH", local_lib, prepend=True) - self.log_msg.emit(f" Set LD_LIBRARY_PATH to include {local_lib}") + user_site = self._vs_user_site() + if user_site and (user_site / "vapoursynth" / "libvsscript.so").exists(): + vs_dir = str(user_site / "vapoursynth") + if vs_dir not in self._build_env.get("LD_LIBRARY_PATH", ""): + self._extend_env("LD_LIBRARY_PATH", vs_dir, prepend=True) + if str(user_site) not in self._build_env.get("PYTHONPATH", ""): + self._extend_env("PYTHONPATH", str(user_site), prepend=True) + self.log_msg.emit( + f" Runtime env: LD_LIBRARY_PATH/PYTHONPATH include {vs_dir}" + ) self.log_msg.emit("") self.log_msg.emit("=== Source build complete ===") @@ -174,6 +352,127 @@ class SourceBuildWorker(QThread): self.log_msg.emit(f"BUILD FAILED: {e}") self.build_done.emit(False, str(e)) + def _missing_build_tools(self) -> list[str]: + """Binaries from BUILD_TOOL_ALIASES that are not on PATH.""" + missing = [] + for label, candidates in BUILD_TOOL_ALIASES.items(): + if not any(shutil.which(c) for c in candidates): + missing.append(label) + return missing + + def _pkgconfig_exists(self, name: str) -> bool: + rc, _ = self._run_cmd( + ["pkg-config", "--exists", name], + timeout=10, label=f"pkg-config {name}", + ) + return rc == 0 + + def _vs_user_site(self) -> Path | None: + """The user site-packages dir of the system python3 — where the + VapourSynth git install places its self-contained stack (module, + libs, headers, plugins/).""" + rc, out = self._run_cmd( + ["python3", "-m", "site", "--user-site"], + timeout=15, label="python3 -m site --user-site", + ) + if rc == 0 and out.strip(): + return Path(out.strip().splitlines()[-1]) + return None + + def _build_bestsource(self): + """Clone and build the BestSource VapourSynth plugin from git. + + BestSource gives av1an a fast, frame-accurate chunk source — the + difference between 'select' (quadratic decoding, minutes per + file) and normal chunk-parallel speed. Compiled against the + vapoursynth headers of the JUST-INSTALLED git VS (via + PYTHONPATH/PKG_CONFIG_PATH), so the plugin ABI always matches + the VS that av1an will load. Requires the repo's libp2p + submodule (initialized here). + """ + self.log_msg.emit("") + self.log_msg.emit("=== Building BestSource plugin from git ===") + self.log_msg.emit(" Source: https://github.com/vapoursynth/bestsource") + + build_dir = Path("/tmp/bestsource-git-build") + if build_dir.exists(): + shutil.rmtree(build_dir, ignore_errors=True) + + rc, out = self._run_cmd( + ["git", "clone", "--depth", "1", + "https://github.com/vapoursynth/bestsource.git", + str(build_dir)], + timeout=120, label="git clone bestsource", + ) + if rc != 0: + raise Exception(f"git clone bestsource failed: {out[-300:]}") + + # libp2p is a required submodule (R9+ builds source from it). + rc, out = self._run_cmd( + ["git", "submodule", "update", "--init", "--depth", "1"], + cwd=str(build_dir), timeout=120, label="git submodule update", + ) + if rc != 0: + raise Exception(f"bestsource submodule init failed: {out[-300:]}") + + # Point meson/pkg-config at the freshly built VS stack. + user_site = self._vs_user_site() + if user_site and (user_site / "vapoursynth").is_dir(): + self._extend_env("PYTHONPATH", str(user_site), prepend=True) + self._extend_env("PKG_CONFIG_PATH", + str(user_site / "vapoursynth" / "pkgconfig"), + prepend=True) + self._extend_env("LD_LIBRARY_PATH", + str(user_site / "vapoursynth"), prepend=True) + else: + self.log_msg.emit( + " NOTE: git VapourSynth install not found in user " + "site-packages — building against system vapoursynth." + ) + + self.log_msg.emit(" Configuring with meson (--prefix=~/.local)...") + rc, out = self._run_cmd( + ["meson", "setup", "build", + f"--prefix={Path.home() / '.local'}", "--libdir=lib"], + cwd=str(build_dir), timeout=180, label="meson setup bestsource", + ) + if rc != 0: + raise Exception(f"bestsource meson setup failed: {out[-500:]}") + + self.log_msg.emit(" Compiling BestSource (a minute or two)...") + rc, out = self._run_cmd( + ["ninja", "-C", "build", "-j", str(max(1, os.cpu_count() or 2))], + cwd=str(build_dir), timeout=600, label="ninja bestsource", + ) + if rc != 0: + raise Exception(f"bestsource build failed: {out[-500:]}") + + rc, out = self._run_cmd( + ["ninja", "-C", "build", "install"], + cwd=str(build_dir), timeout=120, label="ninja install bestsource", + ) + if rc != 0: + raise Exception(f"bestsource install failed: {out[-500:]}") + + plugin = None + if user_site: + candidate = user_site / "vapoursynth" / "plugins" / "libbestsource.so" + if candidate.exists(): + plugin = candidate + if plugin: + self.log_msg.emit(f" BestSource plugin installed: {plugin}") + self.log_msg.emit( + " av1an will now auto-select the fast 'bestsource' chunk " + "method (restart the app so the probe sees it)." + ) + else: + self.log_msg.emit( + " WARNING: libbestsource.so not found at the expected " + "user-site path — check the meson install log above." + ) + + shutil.rmtree(build_dir, ignore_errors=True) + def _build_vapoursynth(self): """Clone, build, and install VapourSynth to ~/.local/ (no sudo needed).""" self.log_msg.emit("") diff --git a/opentranscode/temp_manager.py b/opentranscode/temp_manager.py index 8ac7ee8..f9596ba 100755 --- a/opentranscode/temp_manager.py +++ b/opentranscode/temp_manager.py @@ -80,16 +80,22 @@ def _mkdir_private(path: Path) -> bool: return False -def _worker_temp_dir(worker_pid: int) -> Path: +def _worker_temp_dir(worker_pid: int, lane: str = "") -> Path: """Return a per-worker temp subdir named by PID. v3: each EncoderWorker gets its own subdir under the shared app temp dir, so the final cleanup sweep can safely nuke only this worker's intermediates without affecting a concurrent worker. The subdir is also created with mode=0o700 (FIO09-C). + + v4.7.0: *lane* suffixes the dir ("gpu"/"cpu") for the hybrid + scheduler's concurrent lanes — both run in the SAME process, so the + PID alone no longer separates them, and a lane finishing early must + not sweep the other lane's intermediates out from under it. """ base = _get_app_temp_dir() - sub = base / f"worker-{worker_pid}" + name = f"worker-{worker_pid}" + (f"-{lane}" if lane else "") + sub = base / name _mkdir_private(sub) return sub diff --git a/opentranscode/ui_window.py b/opentranscode/ui_window.py index d19bf60..03ef1d4 100755 --- a/opentranscode/ui_window.py +++ b/opentranscode/ui_window.py @@ -36,7 +36,8 @@ from .codec_profiles import ( ResolutionProfile, ffmpeg_lib_key_for, ) -from .encoder_worker import EncoderWorker +from .encoder_worker import EncoderWorker, resolve_gpu_encoder +from .gpu_profiles import GPU_PROFILES from .env_probe import ( EnvProbe, _av1an_vsscript_smoke_test, @@ -59,6 +60,12 @@ class OpenCodecMaster(QMainWindow): self.setWindowTitle("OpenTranscode — dcos.net") self.resize(1100, 920) self.worker: EncoderWorker | None = None + # v4.7.0: hybrid (GPU + CPU lanes) bookkeeping. *workers* holds + # every active lane; _hybrid_pending/_hybrid_totals aggregate the + # per-lane finished_queue signals into one summary. + self.workers: list[EncoderWorker] = [] + self._hybrid_pending = 0 + self._hybrid_totals = [0, 0] self.env: EnvProbe | None = None self._pending_deletes: list[Path] = [] @@ -300,6 +307,90 @@ class OpenCodecMaster(QMainWindow): "Default OFF for compatibility." ) opt_row.addWidget(self.inline_scale_check) + + # v4.6.0: ENGINE selector — Auto (GPU if available) / GPU / CPU. + # Auto uses the NVENC hardware encoder for the selected codec + # family when the environment probe's live encode test proved it + # works (av1 → av1_nvenc on RTX 40+, hevc → hevc_nvenc on every + # NVENC generation); otherwise it stays on the CPU encoders. GPU + # mode runs through the single-pass ffmpeg path — av1an cannot + # drive NVENC, and one NVENC process outruns chunk-parallel CPU + # workers anyway. + self.engine_combo = QComboBox() + self.engine_combo.addItems([ + "Engine: Auto (GPU if available)", + "Engine: GPU (NVENC)", + "Engine: CPU", + "Engine: Hybrid (GPU + CPU)", + ]) + self.engine_combo.setToolTip( + "Video encode engine.\n\n" + "Auto — use the NVENC GPU encoder when the selected codec\n" + " family has one AND a live encode test proved it works on\n" + " this system; fall back to CPU otherwise (default).\n" + "GPU — force NVENC (hevc_nvenc / av1_nvenc); falls back to\n" + " CPU with a log message when unavailable. GPU encodes run\n" + " via single-pass ffmpeg (av1an chunking is not used).\n" + "CPU — force the software encoders (SVT-AV1 / VP9 / x265).\n" + "Hybrid — run BOTH at once: the queue is split between a\n" + " GPU lane and a CPU lane (balanced by file size), so the\n" + " CPU cores are not idle while NVENC encodes. With av1an\n" + " enabled the CPU lane uses chunk-parallel too. GPU-lane\n" + " files get NVENC quality; CPU-lane files get software-\n" + " encoder quality. Needs 2+ encodable files.\n" + ) + self.engine_combo.setFixedHeight(24) + # v4.7.1: self.env is None (or absent) until _probe_and_init + # runs AFTER _build_ui — the pre-select must not touch it here. + _engine_flags = getattr(getattr(self, "env", None), "av1an_flags", None) or {} + _engine_pref = _engine_flags.get("engine") + _engine_index = {"auto": 0, "gpu": 1, "cpu": 2, "hybrid": 3} + self.engine_combo.setCurrentIndex(_engine_index.get(_engine_pref, 0)) + + # v4.8.0: GPU capability-profile dropdown. Entries combine whole + # card generations (same silicon = same encoding behaviour) and + # include data-center + crypto-era oddballs. "Auto-detect" uses + # the probe's name match; a specific profile forces the API even + # when auto-match fails. The rebuild-from-git dep tree extends + # with the selected profile's packages. + from .gpu_profiles import GPU_PROFILES as _GPU_PROFILES + self.gpu_combo = QComboBox() + self.gpu_combo.addItem("GPU: Auto-detect") + for _gp in _GPU_PROFILES: + self.gpu_combo.addItem(f"GPU: {_gp.label}") + self.gpu_combo.setToolTip( + "Hardware encoder capability class.\n" + "Auto-detect matches your card via nvidia-smi/lspci and the\n" + "live encode probe decides what actually works. Forcing a\n" + "profile also extends the REBUILD FROM GIT dependency tree\n" + "with that GPU's packages (nv-codec-headers / VAAPI / QSV)." + ) + self.gpu_combo.setFixedHeight(24) + _gpu_pref = _engine_flags.get("gpu_profile") + _gpu_keys = ["auto"] + [_gp.key for _gp in _GPU_PROFILES] + self.gpu_combo.setCurrentIndex( + _gpu_keys.index(_gpu_pref) if _gpu_pref in _gpu_keys else 0 + ) + self.gpu_combo.currentIndexChanged.connect(self._on_gpu_profile_changed) + # v4.8.1: ENGINE = CPU makes the GPU choice inert — grey it out. + # The "None (CPU-only encode)" entry stays available for users + # who have a GPU in the box but don't want encoding on it. + self.engine_combo.currentIndexChanged.connect(self._on_engine_changed) + self._update_gpu_combo_state() + _gpu_lay = QVBoxLayout() + _gpu_lay.setSpacing(1) + _gpu_lbl = QLabel("GPU") + _gpu_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + _gpu_lay.addWidget(_gpu_lbl) + _gpu_lay.addWidget(self.gpu_combo) + opt_row.addLayout(_gpu_lay) + _engine_lay = QVBoxLayout() + _engine_lay.setSpacing(1) + _engine_lbl = QLabel("ENGINE") + _engine_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;") + _engine_lay.addWidget(_engine_lbl) + _engine_lay.addWidget(self.engine_combo) + opt_row.addLayout(_engine_lay) root.addLayout(opt_row) # ── Log + Knobs: horizontal split ── @@ -348,12 +439,18 @@ class OpenCodecMaster(QMainWindow): self.btn_rebuild.setObjectName("btnRebuild") self.btn_rebuild.setFixedHeight(40) self.btn_rebuild.setToolTip( - "Compile VapourSynth + av1an from git source.\n" + "Compile VapourSynth + av1an + BestSource from git source.\n" "Resolves ABI/version mismatch when package managers\n" - "install incompatible versions." + "install incompatible versions. ALWAYS available — even on a\n" + "bare system: it first generates its own dependency tree\n" + "(installs missing build tools + zimg via the distro package\n" + "manager, one privilege prompt) and then builds everything\n" + "into ~/.local / ~/.cargo (no system changes)." ) self.btn_rebuild.clicked.connect(self._manual_rebuild) - self.btn_rebuild.setEnabled(False) + # v4.7.1: ALWAYS usable — the build generates its own dependency + # tree, so it must not depend on a successful probe (a failed + # probe is exactly when you need it). btn_lay.addWidget(self.btn_rebuild) self.btn_about = QPushButton(" ? ABOUT / LICENSES") @@ -997,6 +1094,26 @@ class OpenCodecMaster(QMainWindow): selected_codec = VIDEO_CODECS[codec_idx] self._log(f"Codec: {selected_codec.label} (av1an encoder: {selected_codec.av1an_encoder})") + # v4.6.0: read the ENGINE selector and surface the resolved engine + # before the queue starts. The worker re-resolves against the same + # probe data; this line tells the user what is about to happen. + engine_sel = ("auto", "gpu", "cpu", "hybrid")[self.engine_combo.currentIndex()] \ + if hasattr(self, "engine_combo") else "auto" + if hasattr(self, "engine_combo"): + # Always write UI state (CLI --engine pre-selects the combo; + # unchecking round-trips the same way as the other toggles). + self.env.av1an_flags["engine"] = engine_sel + gpu_preview, gpu_api = resolve_gpu_encoder(engine_sel, selected_codec, self.env) + if gpu_preview: + self._log(f"Engine: GPU — {gpu_preview} ({gpu_api} hardware encode)") + elif engine_sel == "gpu": + gpu_enc = selected_codec.gpu_encoder or "(none for this codec)" + self._log(f"Engine: GPU requested but {gpu_enc} unavailable — will use CPU.") + elif engine_sel == "hybrid": + self._log("Engine: hybrid — GPU lane + CPU lane concurrently (splits the queue by size).") + else: + self._log("Engine: CPU (software encoders)") + # v4.4.4: read the inline-scale checkbox into env.av1an_flags so # EncoderWorker.__init__'s fallback path picks it up (matches the # pattern used by --use-av1an, --verbose, --skip-existing, etc.). @@ -1009,7 +1126,9 @@ class OpenCodecMaster(QMainWindow): if self.inline_scale_check.isChecked(): self._log("Inline scale: enabled (no intermediate file for scaling).") - self.worker = EncoderWorker( + # v4.7.0: kwargs shared by every lane of the queue (single + # worker, or the GPU + CPU lanes in hybrid mode). + common = dict( in_dir=in_dir, out_dir=out_dir, video_codec=selected_codec, @@ -1018,17 +1137,33 @@ class OpenCodecMaster(QMainWindow): crf=self.crf_knob.intValue(), preset_label=self.preset_combo.currentText(), delete_source=self.del_check.isChecked(), - env=self.env, extensions=self._parse_extensions(), resolution=self._get_current_resolution(), audio_level_db=self.vol_knob.value(), - use_ffmpeg_fallback=use_ffmpeg_fallback, subtitle_lang=SUBTITLE_OPTIONS[self.subs_combo.currentIndex()][1], force=self.force_check.isChecked(), # v5-01 ) - self.worker.log_msg.connect(self._log) - self.worker.progress_msg.connect(self._on_progress) - self.worker.finished_queue.connect(self._on_finished) + + # ── v4.7.0: hybrid (GPU + CPU lanes run concurrently) ── + if engine_sel == "hybrid": + if self._start_hybrid(common, use_ffmpeg_fallback, use_av1an): + self.btn_run.setEnabled(False) + self.btn_run.setText("RUNNING...") + self.btn_stop.setEnabled(True) + self.btn_rebuild.setEnabled(False) + for w in self.workers: + w.start() + return + self._log("Hybrid unavailable — falling back to a single CPU queue.") + engine_sel = "cpu" + + self.worker = EncoderWorker( + **common, + env=self.env, + use_ffmpeg_fallback=use_ffmpeg_fallback, + engine=engine_sel, # v4.6.0: Auto/GPU/CPU engine selector + ) + self._set_workers([self.worker]) self.btn_run.setEnabled(False) self.btn_run.setText("RUNNING...") @@ -1036,6 +1171,130 @@ class OpenCodecMaster(QMainWindow): self.btn_rebuild.setEnabled(False) self.worker.start() + @Slot() + def _on_engine_changed(self, index: int): + """v4.8.1: ENGINE = CPU disables the GPU dropdown (the choice + would have no effect); every other engine keeps it live.""" + self._update_gpu_combo_state() + + def _update_gpu_combo_state(self): + cpu_only = self.engine_combo.currentIndex() == 2 # "Engine: CPU" + self.gpu_combo.setEnabled(not cpu_only) + self.gpu_combo.setToolTip( + "ENGINE is CPU — the GPU choice has no effect." + if cpu_only else + "Hardware encoder capability class.\n" + "Auto-detect matches your card via nvidia-smi/lspci and the\n" + "live encode probe decides what actually works. Forcing a\n" + "profile also extends the REBUILD FROM GIT dependency tree\n" + "with that GPU's packages (nv-codec-headers / VAAPI / QSV).\n" + "'None (CPU-only encode)' opts out even when a GPU exists." + ) + + @Slot() + def _on_gpu_profile_changed(self, index: int): + """v4.8.0: persist the GPU capability-profile selection so both + the engine resolution and the rebuild dep tree pick it up.""" + if not getattr(self, "env", None): + return # UI build phase — env probe hasn't run yet + if index <= 0: + self.env.av1an_flags["gpu_profile"] = "auto" + return + from .gpu_profiles import GPU_PROFILES + self.env.av1an_flags["gpu_profile"] = GPU_PROFILES[index - 1].key + self._log(f"GPU profile: {GPU_PROFILES[index - 1].key}") + + def _set_workers(self, workers: list) -> None: + """v4.7.0: register the active lane worker(s) and wire their + signals. STOP iterates every lane; _on_finished aggregates the + per-lane summaries into one.""" + self.workers = list(workers) + self._hybrid_pending = len(self.workers) + self._hybrid_totals = [0, 0] + for w in self.workers: + w.log_msg.connect(self._log) + w.progress_msg.connect(self._on_progress) + w.finished_queue.connect(self._on_finished) + + def _start_hybrid(self, common: dict, use_ffmpeg_fallback: bool, + use_av1an: bool) -> bool: + """v4.7.0: split the queue between a GPU lane (NVENC) and a CPU + lane (software encoders, or av1an chunk-parallel when opted in — + so NVENC + chunk workers + software can all run at once). + + Constructs both workers on success and returns True. Returns + False (with a logged reason) when hybrid cannot apply; the caller + falls back to a single-lane queue. + """ + import copy as _copy + import dataclasses as _dc + + from .encoder_worker import scan_input_files + from .hybrid_scheduler import plan_hybrid + + gpu_enc, _gpu_api = resolve_gpu_encoder("gpu", common["video_codec"], self.env) + if not gpu_enc: + self._log("Hybrid: no functional NVENC encoder for this codec family.") + return False + + files = scan_input_files(common["in_dir"], common["extensions"]) + if len(files) < 2: + self._log("Hybrid: fewer than 2 encodable files — one lane is faster than scheduling.") + return False + + sizes: dict = {} + for f in files: + try: + sizes[f] = f.stat().st_size + except OSError: + pass + plan = plan_hybrid( + files, gpu_enc, True, self.env.cpu.logical_threads, sizes=sizes, + ) + if plan is None or not plan.gpu_files or not plan.cpu_files: + self._log("Hybrid: size split degenerated to one lane — single lane is faster.") + return False + + self._log( + f"HYBRID: GPU lane = {len(plan.gpu_files)} file(s) via {gpu_enc} | " + f"CPU lane = {len(plan.cpu_files)} file(s) " + f"(budget {plan.cpu_budget_threads} threads" + f"{', av1an chunk-parallel' if use_av1an else ''})" + ) + self._log( + " Note: lanes use different encoders — GPU-lane files get NVENC " + "quality, CPU-lane files get software-encoder quality." + ) + + # CPU lane budget: hold back threads for the GPU lane's + # decode/scale/mux processes, then let the lane self-budget its + # av1an workers / ffmpeg threads from the reduced topology. + cpu_env = _copy.copy(self.env) + cpu_env.cpu = _dc.replace( + self.env.cpu, logical_threads=plan.cpu_budget_threads + ) + + gpu_worker = EncoderWorker( + **common, + env=self.env, + use_ffmpeg_fallback=True, # GPU runs via single-pass ffmpeg + engine="gpu", + file_subset=plan.gpu_files, + lane="gpu", + ) + cpu_worker = EncoderWorker( + **common, + env=cpu_env, + use_ffmpeg_fallback=use_ffmpeg_fallback, + engine="cpu", + file_subset=plan.cpu_files, + lane="cpu", + ffmpeg_threads=plan.cpu_budget_threads, + ) + self.worker = cpu_worker # primary handle (back-compat) + self._set_workers([gpu_worker, cpu_worker]) + return True + def _handle_vs_incompat(self) -> bool: """Handle detected VSScript ABI incompatibility. @@ -1154,6 +1413,7 @@ class OpenCodecMaster(QMainWindow): self._build_worker = SourceBuildWorker( build_vs=build_vs, build_av1an=build_av1an, build_ffmpeg_iamf=build_ffmpeg_iamf, + gpu_profile_key=self.env.av1an_flags.get("gpu_profile", "auto"), ) self._build_worker.log_msg.connect(self._log) self._build_worker.build_done.connect(self._on_build_done) @@ -1295,6 +1555,21 @@ class OpenCodecMaster(QMainWindow): @Slot(int, int) def _on_finished(self, ok: int, fail: int): + # v4.7.0: hybrid lanes finish independently — aggregate the + # per-lane summaries and only finalize when the LAST lane exits. + if len(self.workers) > 1: + self._hybrid_totals[0] += ok + self._hybrid_totals[1] += fail + self._hybrid_pending -= 1 + if self._hybrid_pending > 0: + self._log( + f"Lane done — {ok} ok, {fail} failed. " + f"Waiting for the other lane..." + ) + return + ok, fail = self._hybrid_totals + self._hybrid_totals = [0, 0] + self.btn_run.setEnabled(True) self.btn_run.setText("START PROCESSING") self.btn_stop.setEnabled(False) @@ -1307,9 +1582,10 @@ class OpenCodecMaster(QMainWindow): @Slot() def _stop_process(self): - if self.worker and self.worker.isRunning(): + if any(w.isRunning() for w in self.workers): self._log("STOP: Exiting queue after current file finishes...") - self.worker.stop() + for w in self.workers: + w.stop() self.btn_stop.setEnabled(False) @@ -1327,7 +1603,9 @@ def launch_gui(argv: list[str] | None = None, force: bool = False, verbose: bool = False, skip_existing: bool = True, timeout: int = 86400, - inline_scale: bool = False) -> int: + inline_scale: bool = False, + engine: str = "auto", + gpu_profile: str = "auto") -> int: """Create the QApplication, show the OpenCodecMaster window, run the Qt event loop. This is the GUI entry point invoked by ``cli.main()`` when no @@ -1435,5 +1713,34 @@ def launch_gui(argv: list[str] | None = None, force: bool = False, if inline_scale and hasattr(window, "inline_scale_check"): window.inline_scale_check.setChecked(True) window._log("Inline scale: enabled via --inline-scale (no intermediate for scaling).") + # v4.6.0: store --engine (auto|gpu|cpu). "auto" (default) uses the + # NVENC GPU encoder when the selected codec family has one and the + # live encode test passed. Pre-selects the ENGINE combo so the + # user sees the state; the combo is re-read at encode time, so the + # user can still change it per run. + if gpu_profile != "auto": + window.env.av1an_flags["gpu_profile"] = gpu_profile + if hasattr(window, "gpu_combo"): + _gpu_keys = ["auto"] + [gp.key for gp in GPU_PROFILES] + if gpu_profile in _gpu_keys: + window.gpu_combo.setCurrentIndex(_gpu_keys.index(gpu_profile)) + window._log(f"GPU profile: {gpu_profile} (via --gpu-profile).") + if engine in ("gpu", "cpu", "hybrid"): + window.env.av1an_flags["engine"] = engine + if hasattr(window, "engine_combo"): + window.engine_combo.setCurrentIndex( + {"gpu": 1, "cpu": 2, "hybrid": 3}[engine] + ) + window._log(f"Engine: {engine.upper()} (via --engine).") + else: + window.env.av1an_flags["engine"] = "auto" + gpu = getattr(window.env, "gpu", None) + if gpu is not None and gpu.has_gpu: + window._log( + f"Engine: auto — GPU detected ({gpu.name or 'NVIDIA'}), " + f"NVENC ready: {', '.join(gpu.usable_encoders)}." + ) + else: + window._log("Engine: auto — no usable GPU encoder, CPU encoders will be used.") window.show() return sys.exit(app.exec()) diff --git a/pyproject.toml b/pyproject.toml index d82abc3..613878d 100755 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ build-backend = "setuptools.build_meta" [project] name = "opentranscode" -version = "4.5.0" +version = "4.8.1" description = "Open-source batch video transcoder wrapping av1an + ffmpeg with a PySide6 GUI" readme = "README.md" requires-python = ">=3.12" diff --git a/scripts/build-ffmpeg-nvenc-matched.sh b/scripts/build-ffmpeg-nvenc-matched.sh new file mode 100755 index 0000000..f212e5f --- /dev/null +++ b/scripts/build-ffmpeg-nvenc-matched.sh @@ -0,0 +1,91 @@ +#!/usr/bin/env bash +# build-ffmpeg-nvenc-matched.sh — build an ffmpeg that matches your NVIDIA +# driver's NVENC API version, installed to ~/.local (shadows the distro +# ffmpeg via ~/.local/bin precedence in PATH). +# +# WHY THIS EXISTS +# --------------- +# ffmpeg builds are compiled against a specific NVENC API (nv-codec-headers +# "gen") and the installed NVIDIA driver must be at least that new. If your +# card is on a legacy driver branch (e.g. 580 is the LAST branch supporting +# Pascal cards like the GTX 10xx), you cannot upgrade the driver to what a +# bleeding-edge ffmpeg wants — e.g.: +# +# [hevc_nvenc @ ...] Driver does not support the required nvenc API +# version. Required: 13.1 Found: 13.0 +# +# The fix is the inverse: build ffmpeg against the API your driver DOES +# provide (Found: X.Y → nv-codec-headers nX.Y.*). OpenTranscode's probe +# picks ~/.local/bin/ffmpeg up automatically on the next launch (PATH +# order) and the Auto engine will start using the GPU. +# +# USAGE +# ----- +# ./build-ffmpeg-nvenc-matched.sh # auto-detect driver API +# ./build-ffmpeg-nvenc-matched.sh 13.0 # explicit gen +# +# Requires: git, make, gcc, nasm, pkg-config, clang (for --enable-cuda-llvm; +# the script drops that flag automatically if clang is absent). +set -euo pipefail + +FF_VER="${FF_VER:-n9.0.2}" +PREFIX="${PREFIX:-$HOME/.local}" +BUILD_DIR="${BUILD_DIR:-/tmp/ffmpeg-nvenc-build}" +BASE_CONF="${BASE_CONF:-}" # optional: a configure line to copy (e.g. distro ffmpeg) + +if [[ $# -ge 1 ]]; then + GEN="$1" +else + # First two components of the driver's NVENC API, from the error ffmpeg + # prints (or run `nvidia-smi` — 580/550/etc. drivers ↔ gen 13.0/12.2). + read -rp "NVENC API gen to build against (e.g. 13.0): " GEN +fi +HEADERS_TAG="n${GEN}.19.1" # any n.* tag provides that gen +mkdir -p "$BUILD_DIR" + +echo "==> nv-codec-headers $HEADERS_TAG → $PREFIX" +if [[ ! -d "$BUILD_DIR/nv-codec-headers" ]]; then + git clone --quiet https://github.com/FFmpeg/nv-codec-headers.git "$BUILD_DIR/nv-codec-headers" +fi +git -C "$BUILD_DIR/nv-codec-headers" fetch --quiet --depth 1 origin tag "$HEADERS_TAG" || true +git -C "$BUILD_DIR/nv-codec-headers" checkout --quiet "$HEADERS_TAG" +make -C "$BUILD_DIR/nv-codec-headers" install "PREFIX=$PREFIX" >/dev/null + +echo "==> ffmpeg $FF_VER source → $BUILD_DIR/src" +if [[ ! -d "$BUILD_DIR/src" ]]; then + git clone --quiet --depth 1 --branch "$FF_VER" \ + https://github.com/FFmpeg/ffmpeg.git "$BUILD_DIR/src" +fi + +CONF_ARGS="${BASE_CONF:-}" +if [[ -z "$CONF_ARGS" && -x /usr/bin/ffmpeg ]]; then + # Copy the distro build's feature set so the result is a drop-in. + CONF_ARGS=$(/usr/bin/ffmpeg -version 2>/dev/null | sed -n 3p | sed 's/^configuration: //') +fi +if [[ -z "$CONF_ARGS" ]]; then + CONF_ARGS="--enable-gpl --enable-libx264 --enable-libx265 --enable-libvpx --enable-libsvtav1 --enable-libopus --enable-libvorbis --enable-libdav1d" +fi +CONF_ARGS=${CONF_ARGS/--prefix=*\/usr /} # strip distro prefix +CONF_ARGS=$(echo "$CONF_ARGS" | sed 's/--prefix=[^ ]*//') +# --enable-rpath is CRITICAL: without it the installed binary resolves its +# SONAME libs (libavcodec.so etc.) from /usr/lib — the DISTRO build — and +# silently keeps demanding the newer NVENC API. rpath pins it to +# $PREFIX/lib where our NVENC-13.0-matched libs live. +CONF_ARGS="--prefix=$PREFIX $CONF_ARGS --enable-nvenc --enable-nvdec --enable-rpath" +command -v clang >/dev/null || CONF_ARGS=$(echo "$CONF_ARGS" | sed 's/--enable-cuda-llvm //') + +echo "==> configure" +export PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH:-}" +cd "$BUILD_DIR/src" +# shellcheck disable=SC2086 +./configure $CONF_ARGS + +echo "==> make ($(nproc) jobs) — this takes a few minutes" +make -j"$(nproc)" +make install + +echo +echo "==> done. Verify (all three must succeed):" +echo " ldd $PREFIX/bin/ffmpeg | grep avcodec # MUST print $PREFIX/lib/..." +echo " hash -r; which ffmpeg # should print $PREFIX/bin/ffmpeg" +echo " $PREFIX/bin/ffmpeg -hide_banner -loglevel error -f lavfi -i 'color=c=black:s=256x256:d=0.2' -c:v hevc_nvenc -f null -" diff --git a/tests/conftest.py b/tests/conftest.py index 49e32a0..974472f 100755 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -357,3 +357,28 @@ def make_minimal_worker(opentranscode_module, env=None, audio_level_db=-14.0): worker.subtitle_lang = None worker.use_ffmpeg_fallback = False return worker + + +# ───────────────────────────────────────────────────────────────────────────── +# Signal capture helper (PySide6-safe) +# ───────────────────────────────────────────────────────────────────────────── + +def capture_signal(logs: list): + """Return a drop-in replacement for a Qt Signal that appends every + emit()ed message to *logs*. + + Tests must REPLACE the whole signal object (``worker.log_msg = + capture_signal(logs)``) instead of patching ``worker.log_msg.emit``: + PySide6's real SignalInstance forbids attribute assignment ("attribute + 'emit' is read-only"), and whether the real PySide6 is loaded depends + on test ordering — so patching .emit works only in some runs. This + helper behaves identically under the stub and the real PySide6. + """ + class _CaptureSignal: + def connect(self, fn): + pass + + def emit(self, msg): + logs.append(msg) + + return _CaptureSignal() diff --git a/tests/test_chunk_method_retry.py b/tests/test_chunk_method_retry.py index e57bd96..a5d0ff7 100755 --- a/tests/test_chunk_method_retry.py +++ b/tests/test_chunk_method_retry.py @@ -26,6 +26,7 @@ from pathlib import Path from unittest.mock import MagicMock import pytest +from conftest import capture_signal OPENTRANSCODE_PATH = Path(__file__).resolve().parent.parent / "open-transcode.py" @@ -279,7 +280,10 @@ class TestY4mBreakRetry: worker._ffmpeg_fallback_encode = MagicMock(return_value=True) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) + # v4.2.1+: RETRY log lines are verbose-only; the test asserts on + # them, so enable verbose output. + worker.verbose = True # Setup required attributes worker._current_temps = [] @@ -379,7 +383,7 @@ class TestY4mBreakRetry: worker._ffmpeg_fallback_encode = mock_ffmpeg_fallback logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) worker._current_temps = [] worker._stop = False @@ -455,7 +459,7 @@ class TestY4mBreakRetry: worker._ffmpeg_fallback_encode = MagicMock(return_value=True) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) worker._current_temps = [] worker._stop = False @@ -490,9 +494,16 @@ class TestVSPluginProbe: """When no VS plugin .so files exist in any search dir, the probe should return an empty list.""" # Point HOME at an empty tmp dir so the home-dir search paths - # don't accidentally find real plugins + # don't accidentally find real plugins. v4.7.1: the probe also + # scans the python site-packages plugin dirs — patch those too, + # or a machine with a git-built BestSource (user site) would + # legitimately report it and break the hermetic expectation. monkeypatch.setenv("HOME", str(tmp_path)) monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path / "share")) + import opentranscode.env_probe as _ep + monkeypatch.setattr(_ep.site, "getusersitepackages", + lambda: str(tmp_path / "site-packages")) + monkeypatch.setattr(_ep.site, "getsitepackages", lambda: []) result = opentranscode_module._probe_vs_source_plugins() assert result == [], \ diff --git a/tests/test_e2e_real_encode.py b/tests/test_e2e_real_encode.py index 78d40d7..dc5c5ee 100755 --- a/tests/test_e2e_real_encode.py +++ b/tests/test_e2e_real_encode.py @@ -32,6 +32,26 @@ from unittest.mock import MagicMock import pytest +def _capture_signal(logs: list): + """Return a drop-in replacement for a Qt Signal that captures emit()ed + messages into *logs*. + + PySide6's SignalInstance attributes are read-only, so tests running + against a REAL PySide6 install cannot patch ``worker.log_msg.emit`` + directly (they can under the conftest stubs). Replacing the whole + signal object works in both modes. + """ + class _CaptureSignal: + def connect(self, fn): + pass + + def emit(self, msg): + logs.append(msg) + + return _CaptureSignal() + + + # ───────────────────────────────────────────────────────────────────────────── # Module loading — the open-transcode.py file has a dash in its name, can't use import # ───────────────────────────────────────────────────────────────────────────── @@ -270,12 +290,10 @@ class TestRealEncodePipeline: subtitle_lang=None, ) - # Collect log messages + # Collect log messages (replace the signal — SignalInstance + # attributes are read-only under a real PySide6 install) logs: list[str] = [] - worker.log_msg.connect = lambda fn: setattr(worker, "_log_fn", fn) - # Patch the log_msg signal emit to capture messages - original_emit = worker.log_msg.emit - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) # Run the worker synchronously (bypass QThread.start) worker.run() @@ -385,7 +403,7 @@ class TestRealEncodePipeline: ) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) worker.run() @@ -458,7 +476,7 @@ class TestRealEncodePipeline: ) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) worker.run() @@ -525,7 +543,7 @@ class TestMovflagsFix: ) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) worker.run() # Find the CMD log line — it should contain -movflags +faststart for MP4 @@ -572,7 +590,7 @@ class TestMovflagsFix: ) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) worker.run() output_files = list(out_dir.rglob("*_archived.mkv")) @@ -759,7 +777,10 @@ class TestV7Y4mBreakRecovery: worker._ffmpeg_fallback_encode = MagicMock(return_value=True) logs: list[str] = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = _capture_signal(logs) + # v4.2.1+: RETRY log lines are verbose-only; the test asserts on + # them, so enable verbose output. + worker.verbose = True # Run the full pipeline (not just _encode_one) worker.run() diff --git a/tests/test_ffmpeg_fallback.py b/tests/test_ffmpeg_fallback.py index 43169ed..2165375 100755 --- a/tests/test_ffmpeg_fallback.py +++ b/tests/test_ffmpeg_fallback.py @@ -14,6 +14,7 @@ from pathlib import Path from unittest.mock import MagicMock import pytest +from conftest import capture_signal OPENTRANSCODE_PATH = Path(__file__).resolve().parent.parent / "open-transcode.py" @@ -211,7 +212,7 @@ class TestPerFileFallbackRetry: worker._ffmpeg_fallback_encode = mock_ffmpeg_fallback logs = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) # Set up _current_temps and _file_res_map (needed by _process_one_file) worker._current_temps = [] @@ -293,7 +294,7 @@ class TestPerFileFallbackRetry: worker._ffmpeg_fallback_encode = MagicMock(return_value=True) logs = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) worker._current_temps = [] worker._stop = False @@ -356,7 +357,7 @@ class TestPerFileFallbackRetry: worker._ffmpeg_fallback_encode = MagicMock(return_value=True) logs = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) worker._current_temps = [] worker._stop = False # will be set by the pattern matcher diff --git a/tests/test_force_validation.py b/tests/test_force_validation.py index dd50e94..ce14aa2 100755 --- a/tests/test_force_validation.py +++ b/tests/test_force_validation.py @@ -16,6 +16,7 @@ from pathlib import Path from unittest.mock import MagicMock import pytest +from conftest import capture_signal # ───────────────────────────────────────────────────────────────────────────── @@ -172,7 +173,9 @@ class TestSkipInvalidFiles: worker.env.ffprobe_path = shutil.which("ffprobe") or "/usr/bin/ffprobe" logs = [] worker.log_msg = MagicMock() - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) + # v4.2.1+: file-type diagnostics on SKIP are verbose-only. + worker.verbose = True skip, info, src_w, src_h = worker._validate_file(fake_video) @@ -202,7 +205,7 @@ class TestSkipInvalidFiles: worker.env.ffprobe_path = shutil.which("ffprobe") or "/usr/bin/ffprobe" logs = [] worker.log_msg = MagicMock() - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) skip, info, src_w, src_h = worker._validate_file(fake_video) @@ -330,7 +333,7 @@ class TestPreFlightValidation: ) logs = [] - worker.log_msg.emit = lambda msg: logs.append(msg) + worker.log_msg = capture_signal(logs) # Run the worker — should abort in pre-flight validation worker.run() diff --git a/tests/test_gpu_engine.py b/tests/test_gpu_engine.py new file mode 100644 index 0000000..df2b7eb --- /dev/null +++ b/tests/test_gpu_engine.py @@ -0,0 +1,421 @@ +"""v4.6.0 GPU (NVENC) engine tests. + +Covers: + - ``GpuInfo`` defaults and properties + - ``_probe_gpu``: compiled-in detection + functional smoke test gate + (the driver/ffmpeg NVENC API mismatch case), with subprocess mocked + - ``resolve_gpu_encoder``: the auto/gpu/cpu decision matrix + - codec profile gpu fields + NVENC vargs shape + - ``FFMPEG_LIB_KEY_MAP`` nvenc entries + - CLI ``--engine`` flag + - EncoderWorker engine attribute + env fallback + - launcher-script parity (resolve_gpu_encoder + engine param exist) +""" +import inspect +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from conftest import capture_signal + +from opentranscode.codec_profiles import ( + FFMPEG_LIB_KEY_MAP, + VIDEO_CODECS, + _av1_nvenc_args, + _hevc_nvenc_args, +) +from opentranscode.encoder_worker import EncoderWorker, resolve_gpu_encoder +from opentranscode.env_probe import GpuInfo, _NVENC_ENCODER_NAMES, _probe_gpu + + +# ───────────────────────────────────────────────────────────────────────────── +# GpuInfo +# ───────────────────────────────────────────────────────────────────────────── + +class TestGpuInfo: + def test_defaults_have_no_gpu(self): + info = GpuInfo() + assert info.has_gpu is False + assert info.usable_encoders == [] + assert info.first_failure_detail == "" + assert info.name == "" + + def test_usable_encoders_in_preference_order(self): + info = GpuInfo() + for enc in reversed(_NVENC_ENCODER_NAMES): + info.functional[enc] = True + assert info.usable_encoders == list(_NVENC_ENCODER_NAMES) + + def test_first_failure_detail_skips_empty(self): + info = GpuInfo() + info.details["av1_nvenc"] = "" + info.details["hevc_nvenc"] = "driver too old" + assert info.first_failure_detail == "driver too old" + + +# ───────────────────────────────────────────────────────────────────────────── +# _probe_gpu (subprocess mocked) +# ───────────────────────────────────────────────────────────────────────────── + +def _fake_run_factory(encoders_output: str, smoke_ok: bool): + """Return a subprocess.run stand-in: -encoders prints encoders_output; + the smoke test succeeds or fails per smoke_ok.""" + def fake_run(cmd, **kw): + if "-encoders" in cmd: + res = MagicMock(returncode=0) + res.stdout = encoders_output + return res + # Smoke test — fail with the real-world driver mismatch text. + if smoke_ok: + res = MagicMock(returncode=0) + res.stderr = "" + return res + res = MagicMock(returncode=-22) + res.stderr = ( + "[hevc_nvenc @ 0x56146c9e2240] Driver does not support the " + "required nvenc API version. Required: 13.1 Found: 13.0\n" + "[hevc_nvenc @ 0x56146c9e2240] The minimum required Nvidia " + "driver for nvenc is 610.00 or newer\n" + ) + return res + return fake_run + + +class TestProbeGpu: + def test_no_ffmpeg_returns_empty(self): + assert _probe_gpu(None).has_gpu is False + + def test_no_nvenc_in_build(self, monkeypatch): + monkeypatch.setattr( + "opentranscode.env_probe.subprocess.run", + _fake_run_factory("V..... libsvtav1 SVT-AV1 encoder\n", smoke_ok=True), + ) + info = _probe_gpu("/usr/bin/ffmpeg") + assert any(info.encoders.values()) is False + assert info.has_gpu is False + assert info.functional == {} + + def test_compiled_in_but_driver_too_old(self, monkeypatch): + out = (" V....D hevc_nvenc NVIDIA NVENC hevc encoder\n" + " V....D h264_nvenc NVIDIA NVENC H.264 encoder\n") + monkeypatch.setattr( + "opentranscode.env_probe.subprocess.run", + _fake_run_factory(out, smoke_ok=False), + ) + info = _probe_gpu("/usr/bin/ffmpeg") + assert info.encoders["hevc_nvenc"] is True + assert info.functional["hevc_nvenc"] is False + assert "API version" in info.first_failure_detail + assert info.has_gpu is False + + def test_functional_nvenc(self, monkeypatch): + out = " V....D hevc_nvenc NVIDIA NVENC hevc encoder\n" + monkeypatch.setattr( + "opentranscode.env_probe.subprocess.run", + _fake_run_factory(out, smoke_ok=True), + ) + monkeypatch.setattr( + "opentranscode.env_probe.shutil.which", lambda name: None + ) + info = _probe_gpu("/usr/bin/ffmpeg") + assert info.has_gpu is True + assert info.usable_encoders == ["hevc_nvenc"] + + def test_smoke_test_is_live_encode(self, monkeypatch): + """The functional gate must run an actual encode, not just grep.""" + seen_cmds = [] + + def fake_run(cmd, **kw): + seen_cmds.append(cmd) + res = MagicMock(returncode=0) + res.stdout = " V....D hevc_nvenc NVIDIA NVENC hevc encoder\n" + res.stderr = "" + return res + + monkeypatch.setattr("opentranscode.env_probe.subprocess.run", fake_run) + monkeypatch.setattr("opentranscode.env_probe.shutil.which", lambda n: None) + _probe_gpu("/usr/bin/ffmpeg") + # Stage 1 (-encoders) + Stage 2 (lavfi source + -c:v hevc_nvenc). + smoke = [c for c in seen_cmds if "lavfi" in c] + assert len(smoke) == 1 + assert "hevc_nvenc" in smoke[0] + assert "-f" in smoke[0] and "null" in smoke[0] + + +# ───────────────────────────────────────────────────────────────────────────── +# resolve_gpu_encoder decision matrix +# ───────────────────────────────────────────────────────────────────────────── + +class _FakeGpu: + def __init__(self, functional): + self.functional = functional + self.name = "Test GPU" + + +class _FakeEnv: + def __init__(self, functional): + self.gpu = _FakeGpu(functional) + self.av1an_flags = {} + + +def _codec_by_label(fragment): + return next(c for c in VIDEO_CODECS if fragment in c.label) + + +class TestResolveGpuEncoder: + def test_auto_uses_functional_gpu(self): + env = _FakeEnv({"hevc_nvenc": True}) + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, env) == ("hevc_nvenc", "nvenc") + + def test_auto_falls_back_when_smoke_failed(self): + # The real GTX 1070 + ffmpeg 9.0.2 case: encoder listed but the + # live test failed → CPU. + env = _FakeEnv({"hevc_nvenc": False}) + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_engine_cpu_never_gpu(self): + env = _FakeEnv({"hevc_nvenc": True}) + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("cpu", codec, env) == (None, None) + + def test_engine_gpu_requires_functional(self): + env = _FakeEnv({"hevc_nvenc": False}) + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("gpu", codec, env) == (None, None) + env2 = _FakeEnv({"hevc_nvenc": True}) + assert resolve_gpu_encoder("gpu", codec, env2) == ("hevc_nvenc", "nvenc") + + def test_vp9_needs_vaapi_profile(self): + env = _FakeEnv({"av1_nvenc": True, "hevc_nvenc": True, "h264_nvenc": True}) + codec = _codec_by_label("VP9") + # No profile → legacy nvenc fallback → VP9 has none. + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_av1_gpu_only_when_functional(self): + env = _FakeEnv({"av1_nvenc": False}) + codec = _codec_by_label("AV1") + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + env2 = _FakeEnv({"av1_nvenc": True}) + assert resolve_gpu_encoder("auto", codec, env2) == ("av1_nvenc", "nvenc") + + def test_env_without_gpu_attr_is_safe(self): + class _NoGpu: + pass + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, _NoGpu()) == (None, None) + + # ── v4.8.0: profile-driven resolution ── + def test_profile_selects_vaapi_encoder(self): + from opentranscode.gpu_profiles import gpu_profile_by_key + env = _FakeEnv({"hevc_vaapi": True, "h264_vaapi": True}) + env.av1an_flags = {"gpu_profile": "amd-rdna3"} + codec = _codec_by_label("x265") + enc, api = resolve_gpu_encoder("auto", codec, env) + assert (enc, api) == ("hevc_vaapi", "vaapi") + + def test_profile_respects_functional_gate(self): + env = _FakeEnv({"hevc_vaapi": False}) + env.av1an_flags = {"gpu_profile": "amd-rdna3"} + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_compute_profile_has_no_encoders(self): + # CMP 170HX / A100 class: claims nothing → CPU. + env = _FakeEnv({}) + env.av1an_flags = {"gpu_profile": "nv-compute"} + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_pascal_profile_blocks_av1(self): + # AV1 on a 1070-class card: profile has no av1 encoder → CPU. + env = _FakeEnv({"av1_nvenc": True}) # functional (e.g. different card) + env.av1an_flags = {"gpu_profile": "nv-pascal"} + codec = _codec_by_label("AV1") + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_matched_profile_used_when_no_selection(self): + from opentranscode.gpu_profiles import gpu_profile_by_key + env = _FakeEnv({"hevc_qsv": True}) + env.av1an_flags = {} + env.gpu.profile_key = "intel-arc" + codec = _codec_by_label("x265") + assert resolve_gpu_encoder("auto", codec, env) == ("hevc_qsv", "qsv") + + +# ───────────────────────────────────────────────────────────────────────────── +# Codec profiles: gpu fields + NVENC vargs +# ───────────────────────────────────────────────────────────────────────────── + +class TestCodecProfileGpuFields: + def test_av1_has_av1_nvenc(self): + av1 = _codec_by_label("AV1") + assert av1.gpu_encoder == "av1_nvenc" + assert av1.gpu_vargs_fn is not None + + def test_hevc_has_hevc_nvenc(self): + x265 = _codec_by_label("x265") + assert x265.gpu_encoder == "hevc_nvenc" + + def test_vp9_has_no_gpu_encoder(self): + vp9 = _codec_by_label("VP9") + assert vp9.gpu_encoder == "" + + def test_nvenc_vargs_use_cq_mode(self): + args = _hevc_nvenc_args(28, 7) + assert args[0] == "-c:v" and args[1] == "hevc_nvenc" + assert "-cq" in args and "28" in args + assert "-rc" in args and "vbr" in args + assert "-b:v" in args and "0" in args + assert "-pix_fmt" in args and "yuv420p" in args + av1_args = _av1_nvenc_args(32, 6) + assert "av1_nvenc" in av1_args + + def test_skip_existing_codec_name_unchanged_by_gpu(self): + """hevc_nvenc outputs the same ffprobe codec_name ("hevc") as + libx265, so skip-existing works across engine switches.""" + x265 = _codec_by_label("x265") + assert x265.ffprobe_codec_name == "hevc" + av1 = _codec_by_label("AV1") + assert av1.ffprobe_codec_name == "av1" + + def test_ffmpeg_lib_key_map_has_nvenc(self): + assert FFMPEG_LIB_KEY_MAP["hevc_nvenc"] == "hevc_nvenc" + assert FFMPEG_LIB_KEY_MAP["h264_nvenc"] == "h264_nvenc" + assert FFMPEG_LIB_KEY_MAP["av1_nvenc"] == "av1_nvenc" + + +# ───────────────────────────────────────────────────────────────────────────── +# CLI --engine flag +# ───────────────────────────────────────────────────────────────────────────── + +class TestCliEngineFlag: + def test_engine_choices(self): + from opentranscode.cli import build_parser + p = build_parser() + assert p.parse_args(["--engine", "gpu"]).engine == "gpu" + assert p.parse_args(["--engine", "cpu"]).engine == "cpu" + assert p.parse_args([]).engine == "auto" + + def test_engine_rejects_unknown(self): + from opentranscode.cli import build_parser + p = build_parser() + with pytest.raises(SystemExit): + p.parse_args(["--engine", "tape"]) + + +# ───────────────────────────────────────────────────────────────────────────── +# EncoderWorker engine attribute + GPU encode path +# ───────────────────────────────────────────────────────────────────────────── + +class TestEncoderWorkerEngine: + def _make_worker(self, mock_env, engine=None, use_ffmpeg_fallback=True): + from opentranscode.codec_profiles import ( + AUDIO_PROFILES, CONTAINER_PROFILES, RESOLUTION_PRESETS, + ) + return EncoderWorker( + in_dir=Path("/tmp"), + out_dir=Path("/tmp"), + video_codec=_codec_by_label("x265"), + audio_profile=AUDIO_PROFILES[0], + container=CONTAINER_PROFILES[0], + crf=28, + preset_label="Medium (7)", + delete_source=False, + env=mock_env, + extensions={".mkv"}, + resolution=RESOLUTION_PRESETS[0], + use_ffmpeg_fallback=use_ffmpeg_fallback, + engine=engine, + ) + + def test_engine_attr_stored(self, mock_env): + w = self._make_worker(mock_env, engine="gpu") + assert w.engine == "gpu" + + def test_engine_falls_back_to_env_flags(self, mock_env): + mock_env.av1an_flags["engine"] = "cpu" + try: + w = self._make_worker(mock_env) # engine not passed + assert w.engine == "cpu" + finally: + mock_env.av1an_flags.pop("engine", None) + + def test_engine_rejects_unknown_value(self, mock_env): + mock_env.av1an_flags.pop("engine", None) + w = self._make_worker(mock_env, engine="quantum") + assert w.engine == "auto" + + def test_gpu_encoder_unresolved_before_run(self, mock_env): + w = self._make_worker(mock_env) + assert w._gpu_encoder is None + + def test_gpu_mode_builds_nvenc_command(self, mock_env): + """engine=auto + functional hevc_nvenc → the ffmpeg command uses + hevc_nvenc with -cq instead of libx265 with -crf.""" + mock_env.av1an_flags["verbose"] = True + mock_env.gpu = _FakeGpu({"hevc_nvenc": True}) + mock_env.ffmpeg_libs["hevc_nvenc"] = True + w = self._make_worker(mock_env, engine="auto", use_ffmpeg_fallback=False) + # run() resolves the engine before the encode loop; mirror that + # here (this test targets the command-construction path only). + w._gpu_encoder, w._gpu_api = resolve_gpu_encoder( + w.engine, w.video_codec, mock_env) + w.use_ffmpeg_fallback = True + w.resolution = type("R", (), {"width": None, "height": None})() + w._temp_dir = Path("/tmp") + w._current_temps = [] + w._file_res_map = {} + w._stop = False + w._consecutive_fail_count = 0 + w._last_fail_pattern = None + w.audio_level_db = 0.0 + w.encode_timeout = 60 + logs: list[str] = [] + w.log_msg = capture_signal(logs) + w._run_with_stop_check = MagicMock( + return_value=("ok", 0, "", "") + ) + + src = Path("/tmp/fake_src.mkv") + out = Path("/tmp/fake_out.mkv") + ok = w._ffmpeg_fallback_encode(src, src, out) + + assert ok is False # "output exists" check fails (mock ran nothing) + cmd = w._run_with_stop_check.call_args[0][0] + assert "hevc_nvenc" in cmd + assert "-cq" in cmd + assert "libx265" not in cmd + + def test_cpu_mode_keeps_libx265(self, mock_env): + mock_env.gpu = _FakeGpu({"hevc_nvenc": True}) + w = self._make_worker(mock_env, engine="cpu", use_ffmpeg_fallback=True) + w.resolution = type("R", (), {"width": None, "height": None})() + w.audio_level_db = 0.0 + w.encode_timeout = 60 + w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", "")) + src = Path("/tmp/fake_src.mkv") + w._ffmpeg_fallback_encode(src, src, Path("/tmp/fake_out.mkv")) + cmd = w._run_with_stop_check.call_args[0][0] + assert "libx265" in cmd + assert "hevc_nvenc" not in cmd + + +# ───────────────────────────────────────────────────────────────────────────── +# Launcher parity +# ───────────────────────────────────────────────────────────────────────────── + +class TestLauncherParity: + def test_launcher_defines_resolve_gpu_encoder(self, opentranscode_module): + assert hasattr(opentranscode_module, "resolve_gpu_encoder") + + def test_launcher_worker_accepts_engine(self, opentranscode_module): + sig = inspect.signature(opentranscode_module.EncoderWorker.__init__) + assert "engine" in sig.parameters + + def test_launcher_profiles_have_gpu_fields(self, opentranscode_module): + for vc in opentranscode_module.VIDEO_CODECS: + assert hasattr(vc, "gpu_encoder") + assert hasattr(vc, "gpu_vargs_fn") diff --git a/tests/test_gpu_profiles.py b/tests/test_gpu_profiles.py new file mode 100644 index 0000000..f4c2a28 --- /dev/null +++ b/tests/test_gpu_profiles.py @@ -0,0 +1,202 @@ +"""v4.8.0 GPU capability-profile tests. + +Covers the combined generation dropdown entries: profile data +integrity, the auto-match matrix (including Tesla/data-center and the +crypto-era CMP oddballs), profile-driven encoder resolution, and the +rebuild dep-tree extension. +""" +import os + +import pytest + +from opentranscode.codec_profiles import VIDEO_CODECS +from opentranscode.encoder_worker import resolve_gpu_encoder +from opentranscode.gpu_profiles import ( + GPU_PROFILES, + encoder_filter_chain, + encoder_quality_args, + encoder_pre_args, + gpu_profile_by_key, + match_gpu_profile, +) + + +class TestProfileData: + def test_keys_unique(self): + keys = [p.key for p in GPU_PROFILES] + assert len(keys) == len(set(keys)) + + def test_labels_unique(self): + labels = [p.label for p in GPU_PROFILES] + assert len(labels) == len(set(labels)) + + def test_cpu_only_profile_claims_nothing(self): + cpu = gpu_profile_by_key("cpu") + assert cpu.encoders == {} + assert cpu.api == "none" + + def test_compute_boards_claim_nothing(self): + """V100/A100/H100-class boards and CMP 170HX have no NVENC — + selecting them must land on the CPU path, not crash.""" + compute = gpu_profile_by_key("nv-compute") + assert compute.encoders == {} + + def test_ada_has_av1_older_nvidia_does_not(self): + ada = gpu_profile_by_key("nv-ada") + assert ada.encoders["av1"] == "av1_nvenc" + for key in ("nv-pascal", "nv-turing", "nv-ampere"): + assert "av1" not in gpu_profile_by_key(key).encoders + + def test_intel_arc_has_av1_qsv(self): + arc = gpu_profile_by_key("intel-arc") + assert arc.api == "qsv" + assert arc.encoders["av1"] == "av1_qsv" + + def test_amd_rdna3_has_av1_vaapi_rdna12_does_not(self): + assert gpu_profile_by_key("amd-rdna3").encoders["av1"] == "av1_vaapi" + assert "av1" not in gpu_profile_by_key("amd-rdna12").encoders + + def test_vp9_only_via_vaapi(self): + from opentranscode.codec_profiles import VIDEO_CODECS + vp9 = next(c for c in VIDEO_CODECS if "VP9" in c.label) + assert vp9.gpu_encoders_by_api == {"vaapi": "vp9_vaapi"} + + +class TestMatchMatrix: + @pytest.mark.parametrize("name,expected_key", [ + ("NVIDIA GeForce GTX 1070", "nv-pascal"), + ("NVIDIA GeForce GTX 1080 Ti", "nv-pascal"), + ("NVIDIA GeForce RTX 2060", "nv-turing"), + ("NVIDIA GeForce GTX 1660 SUPER", "nv-turing"), + ("NVIDIA GeForce RTX 3060", "nv-ampere"), + ("NVIDIA GeForce RTX 4090", "nv-ada"), + ("NVIDIA GeForce RTX 5080", "nv-ada"), + ("Tesla P40", "nv-pascal"), + ("Tesla T4", "nv-turing"), + ("NVIDIA A10", "nv-ampere"), + ("NVIDIA CMP 50HX", "nv-turing"), + ("NVIDIA CMP 90HX", "nv-ampere"), + ("NVIDIA CMP 170HX GA100", "nv-compute"), + ("NVIDIA CMP 170HX", "nv-compute"), + ("NVIDIA A100-SXM4-40GB", "nv-compute"), + ("NVIDIA H100", "nv-compute"), + ("Intel(R) Arc(TM) B580", "intel-arc"), + ("Intel(R) UHD Graphics 770", "intel-xe"), + ]) + def test_match(self, name, expected_key): + assert match_gpu_profile(name).key == expected_key + + def test_no_match_returns_none(self): + assert match_gpu_profile("") is None + assert match_gpu_profile("Unknown Widget Graphics") is None + + +class TestProfileResolution: + def _env(self, functional, flags=None, profile_key=""): + class G: + pass + g = G() + g.functional = functional + g.profile_key = profile_key + + class E: + pass + e = E() + e.gpu = g + e.av1an_flags = flags or {} + return e + + def test_hevc_via_pascal_profile(self): + env = self._env({"hevc_nvenc": True}, {"gpu_profile": "nv-pascal"}) + codec = next(c for c in VIDEO_CODECS if "x265" in c.label) + assert resolve_gpu_encoder("auto", codec, env) == ("hevc_nvenc", "nvenc") + + def test_arc_qsv_hevc(self): + env = self._env({"hevc_qsv": True}, {"gpu_profile": "intel-arc"}) + codec = next(c for c in VIDEO_CODECS if "x265" in c.label) + assert resolve_gpu_encoder("auto", codec, env) == ("hevc_qsv", "qsv") + + def test_vp9_on_amd_stays_cpu(self): + # AMD VCN has no VP9 encode — even though ffmpeg has vp9_vaapi, + # the RDNA 3 profile truthfully claims none. + env = self._env({"vp9_vaapi": True}, {"gpu_profile": "amd-rdna3"}) + codec = next(c for c in VIDEO_CODECS if "VP9" in c.label) + assert resolve_gpu_encoder("auto", codec, env) == (None, None) + + def test_vaapi_encoder_args_shape(self): + profile = gpu_profile_by_key("amd-rdna3") + assert encoder_pre_args(profile)[0] == "-vaapi_device" + assert "format=nv12,hwupload" in encoder_filter_chain(profile) + args = encoder_quality_args("vaapi", "hevc_vaapi", 28, 9) + assert args[:2] == ["-rc_mode", "CQP"] + + +class TestRebuildGpuDeps: + def test_nvenc_needs_codec_headers(self): + from opentranscode.source_builder import gpu_dep_packages + pkgs = gpu_dep_packages("nvenc", "arch") + assert "nv-codec-headers" in pkgs + + def test_vaapi_qsv_driver_packages(self): + from opentranscode.source_builder import gpu_dep_packages + assert any("libva" in p for p in gpu_dep_packages("vaapi", "arch")) + assert any("intel-media-driver" in p for p in gpu_dep_packages("qsv", "arch")) + + def test_unpackaged_family_returns_empty(self): + from opentranscode.source_builder import gpu_dep_packages + assert gpu_dep_packages("nvenc", "nixos") == [] + + +def importlib_pyside_available(): + import importlib.util + return importlib.util.find_spec("PySide6") is not None + + +class TestGpuComboEngineInteraction: + @pytest.mark.skipif( + not importlib_pyside_available(), + reason="requires a real PySide6 (stub mode cannot instantiate Qt)", + ) + def test_cpu_engine_disables_gpu_menu(self): + """v4.8.1: ENGINE = CPU greys out the GPU dropdown; the explicit + 'None (CPU-only encode)' entry remains for opting out while a + GPU-capable engine is selected.""" + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + from PySide6.QtWidgets import QApplication + + from opentranscode.ui_window import OpenCodecMaster + + app = QApplication.instance() or QApplication([]) + window = OpenCodecMaster() + try: + gpu_idx = next(i for i in range(window.gpu_combo.count()) + if "None (CPU-only" in window.gpu_combo.itemText(i)) + assert gpu_idx > 0, "None (CPU-only) must be an explicit option" + + window.engine_combo.setCurrentIndex(0) # Auto + assert window.gpu_combo.isEnabled() + window.engine_combo.setCurrentIndex(2) # CPU + assert not window.gpu_combo.isEnabled(), ( + "ENGINE = CPU must disable the GPU dropdown" + ) + window.engine_combo.setCurrentIndex(1) # GPU + assert window.gpu_combo.isEnabled() + finally: + window.close() + + +class TestLauncherParity: + def test_launcher_has_gpu_profiles(self, opentranscode_module): + assert hasattr(opentranscode_module, "GPU_PROFILES") + assert hasattr(opentranscode_module, "match_gpu_profile") + + def test_launcher_match_same_as_package(self, opentranscode_module): + from opentranscode.gpu_profiles import match_gpu_profile as pkg_match + assert opentranscode_module.match_gpu_profile("RTX 4090").key \ + == pkg_match("RTX 4090").key + + def test_launcher_worker_gpu_param(self, opentranscode_module): + import inspect + sig = inspect.signature( + opentranscode_module.SourceBuildWorker.__init__) + assert "gpu_profile_key" in sig.parameters diff --git a/tests/test_hybrid_scheduler.py b/tests/test_hybrid_scheduler.py new file mode 100644 index 0000000..9fe428d --- /dev/null +++ b/tests/test_hybrid_scheduler.py @@ -0,0 +1,278 @@ +"""v4.7.0 hybrid (GPU + CPU lanes) scheduler tests. + +Covers: + - ``plan_hybrid``: degenerate cases, LPT size-balanced split, CPU + thread budget, unknown-size fallback + - ``EncoderWorker.file_subset``: a lane only processes its partition + - ``ffmpeg_threads``: injected on the CPU software path, never GPU + - per-lane temp dirs (``_worker_temp_dir`` lane suffix) + - ``scan_input_files``: extension filter + intermediate exclusion + - CLI ``--engine hybrid`` + - launcher parity (plan_hybrid + lane params exist) +""" +import inspect +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from conftest import capture_signal + +from opentranscode.codec_profiles import ( + AUDIO_PROFILES, + CONTAINER_PROFILES, + RESOLUTION_PRESETS, + VIDEO_CODECS, +) +from opentranscode.encoder_worker import EncoderWorker, scan_input_files +from opentranscode.hybrid_scheduler import ( + HYBRID_CPU_RESERVE_THREADS, + HybridPlan, + plan_hybrid, +) + + +def _hevc(): + return next(c for c in VIDEO_CODECS if "x265" in c.label) + + +def _files(tmp_path, names): + files = [] + for i, name in enumerate(names): + f = tmp_path / name + f.write_bytes(b"\x00" * 1024) + files.append(f) + return files + + +# ───────────────────────────────────────────────────────────────────────────── +# plan_hybrid +# ───────────────────────────────────────────────────────────────────────────── + +class TestPlanHybrid: + def test_no_gpu_encoder_returns_none(self, tmp_path): + files = _files(tmp_path, ["a.mp4", "b.mp4"]) + assert plan_hybrid(files, "", True, 28) is None + assert plan_hybrid(files, "hevc_nvenc", False, 28) is None + assert plan_hybrid(files, "hevc_nvenc", True, 28) is not None + + def test_empty_file_list_returns_none(self): + assert plan_hybrid([], "hevc_nvenc", True, 28) is None + + def test_single_file_goes_to_gpu_lane(self, tmp_path): + files = _files(tmp_path, ["a.mp4"]) + plan = plan_hybrid(files, "hevc_nvenc", True, 28) + assert plan.gpu_files == files + assert plan.cpu_files == [] + + def test_split_covers_all_files_exactly_once(self, tmp_path): + files = _files(tmp_path, [f"v{i}.mp4" for i in range(10)]) + sizes = {f: (i + 1) * 100_000_000 for i, f in enumerate(files)} + plan = plan_hybrid(files, "hevc_nvenc", True, 28, sizes=sizes) + assert plan.total_files == len(files) + assert sorted(plan.gpu_files + plan.cpu_files) == sorted(files) + assert not set(plan.gpu_files) & set(plan.cpu_files) + + def test_lpt_prefers_gpu_for_largest_files(self, tmp_path): + """The largest file goes to the GPU lane (lowest per-byte cost), + and the greedy split keeps the smaller CPU load in check.""" + files = _files(tmp_path, ["big.mp4", "small.mp4"]) + sizes = {files[0]: 1_000_000_000, files[1]: 10_000_000} + plan = plan_hybrid(files, "hevc_nvenc", True, 28, sizes=sizes) + assert plan.gpu_files[0] == files[0] # biggest → GPU + + def test_cpu_budget_reserves_gpu_threads(self, tmp_path): + files = _files(tmp_path, ["a.mp4", "b.mp4"]) + plan = plan_hybrid(files, "hevc_nvenc", True, 28) + assert plan.cpu_budget_threads == 28 - HYBRID_CPU_RESERVE_THREADS + plan2 = plan_hybrid(files, "hevc_nvenc", True, 2) + assert plan2.cpu_budget_threads == 1 # never below 1 + + def test_unknown_sizes_fall_back_to_average(self, tmp_path): + a, b, c = _files(tmp_path, ["a.mp4", "b.mp4", "c.mp4"]) + sizes = {a: 1000, b: 1000} # c unknown + plan = plan_hybrid([a, b, c], "hevc_nvenc", True, 28, sizes=sizes) + assert plan.total_files == 3 # no crash, all files placed + + def test_plan_is_pure_dataclass(self): + plan = HybridPlan() + assert plan.total_files == 0 + assert plan.gpu_encoder == "" + assert plan.cpu_budget_threads == 1 + + +# ───────────────────────────────────────────────────────────────────────────── +# EncoderWorker hybrid support +# ───────────────────────────────────────────────────────────────────────────── + +def _worker(mock_env, tmp_path, **kw): + return EncoderWorker( + in_dir=tmp_path / "in", + out_dir=tmp_path / "out", + video_codec=_hevc(), + audio_profile=AUDIO_PROFILES[2], + container=CONTAINER_PROFILES[0], + crf=28, + preset_label="Faster (10)", + delete_source=False, + env=mock_env, + extensions={".mkv", ".mp4"}, + resolution=RESOLUTION_PRESETS[0], + **kw, + ) + + +class TestWorkerHybridSupport: + def test_file_subset_restricts_queue(self, mock_env, tmp_path, tiny_test_video): + # Real videos — the worker ffprobe-validates its queue. + (tmp_path / "in").mkdir(parents=True) + a = tmp_path / "in" / "a.mkv" + b = tmp_path / "in" / "b.mkv" + a.write_bytes(tiny_test_video.read_bytes()) + b.write_bytes(tiny_test_video.read_bytes()) + (tmp_path / "out").mkdir(parents=True) + + mock_env.ffmpeg_libs["hevc_nvenc"] = True + mock_env.gpu.functional["hevc_nvenc"] = True # live probe passes + w = _worker(mock_env, tmp_path, file_subset=[a], lane="gpu", + engine="gpu", use_ffmpeg_fallback=True) + logs: list[str] = [] + w.log_msg = capture_signal(logs) + w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", "")) + # Fake a successful output for the encoded file. + def fake_run(cmd, **kw2): + # ffmpeg path: output is the last arg (after -y) + out = Path(cmd[-1]) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_bytes(b"\x00" * 8192) + return ("ok", 0, "", "") + w._run_with_stop_check = fake_run + w.audio_level_db = 0.0 + w.verbose = False + w.run() + + assert w.success_count == 1 + # Only the subset file was encoded. + assert (tmp_path / "out" / "a_archived.mkv").exists() + assert not (tmp_path / "out" / "b_archived.mkv").exists() + + def test_lane_temp_dirs_are_distinct(self, mock_env, tmp_path): + gpu = _worker(mock_env, tmp_path, lane="gpu") + cpu = _worker(mock_env, tmp_path, lane="cpu") + plain = _worker(mock_env, tmp_path) + assert gpu._temp_dir != cpu._temp_dir + assert "-gpu" in gpu._temp_dir.name + assert "-cpu" in cpu._temp_dir.name + assert "-" not in plain._temp_dir.name.replace("worker-", "").lstrip("0123456789") or \ + plain._temp_dir.name == f"worker-{plain._temp_dir.name.split('-')[1]}" + + def test_ffmpeg_threads_on_cpu_software_path(self, mock_env, tmp_path): + w = _worker(mock_env, tmp_path, ffmpeg_threads=13, + use_ffmpeg_fallback=True, engine="cpu") + w.resolution = RESOLUTION_PRESETS[0] + w.audio_level_db = 0.0 + w.encode_timeout = 60 + w._gpu_encoder = None # CPU lane + w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", "")) + src = tmp_path / "in" / "x.mp4" + src.parent.mkdir(parents=True, exist_ok=True) + src.write_bytes(b"\x00" * 64) + w._ffmpeg_fallback_encode(src, src, tmp_path / "out.mkv") + cmd = w._run_with_stop_check.call_args[0][0] + assert "-threads" in cmd and "13" in cmd + + def test_failed_ffmpeg_encode_removes_partial_output(self, mock_env, tmp_path): + """A failed ffmpeg encode must not leave a truncated file that + skip-existing could later mistake for a finished archive.""" + w = _worker(mock_env, tmp_path, use_ffmpeg_fallback=True, engine="cpu") + w.resolution = RESOLUTION_PRESETS[0] + w.audio_level_db = 0.0 + w.encode_timeout = 60 + w._gpu_encoder = None + w._run_with_stop_check = MagicMock(return_value=("ok", 1, "", "boom")) + src = tmp_path / "in" / "x.mp4" + src.parent.mkdir(parents=True, exist_ok=True) + src.write_bytes(b"\x00" * 64) + out = tmp_path / "out.mkv" + out.write_bytes(b"partial") # ffmpeg left a partial file + ok = w._ffmpeg_fallback_encode(src, src, out) + assert ok is False + assert not out.exists(), "partial output must be removed on failure" + + def test_no_threads_cap_on_gpu_path(self, mock_env, tmp_path): + mock_env.ffmpeg_libs["hevc_nvenc"] = True + w = _worker(mock_env, tmp_path, ffmpeg_threads=13, + use_ffmpeg_fallback=True, engine="gpu") + w.resolution = RESOLUTION_PRESETS[0] + w.audio_level_db = 0.0 + w.encode_timeout = 60 + w._gpu_encoder = "hevc_nvenc" # GPU lane — NVENC is silicon-bound + w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", "")) + src = tmp_path / "in" / "x.mp4" + src.parent.mkdir(parents=True, exist_ok=True) + src.write_bytes(b"\x00" * 64) + w._ffmpeg_fallback_encode(src, src, tmp_path / "out.mkv") + cmd = w._run_with_stop_check.call_args[0][0] + assert "-threads" not in cmd + assert "hevc_nvenc" in cmd + + +# ───────────────────────────────────────────────────────────────────────────── +# scan_input_files +# ───────────────────────────────────────────────────────────────────────────── + +class TestScanInputFiles: + def test_filters_extensions_and_intermediates(self, tmp_path): + keep = tmp_path / "keep.mp4"; keep.write_bytes(b"\x00") + drop = tmp_path / "drop.txt"; drop.write_bytes(b"\x00") + inter = tmp_path / "keep.scaled_tmp.mkv"; inter.write_bytes(b"\x00") + nested = tmp_path / "sub"; nested.mkdir() + n2 = nested / "nested.mkv"; n2.write_bytes(b"\x00") + result = scan_input_files(tmp_path, {".mp4", ".mkv"}) + assert keep in result and n2 in result + assert drop not in result + assert inter not in result + + def test_sorted_output(self, tmp_path): + for name in ("c.mp4", "a.mp4", "b.mp4"): + (tmp_path / name).write_bytes(b"\x00") + assert scan_input_files(tmp_path, {".mp4"}) == sorted( + scan_input_files(tmp_path, {".mp4"})) + + +# ───────────────────────────────────────────────────────────────────────────── +# CLI + launcher parity +# ───────────────────────────────────────────────────────────────────────────── + +class TestCliHybrid: + def test_engine_hybrid_accepted(self): + from opentranscode.cli import build_parser + assert build_parser().parse_args(["--engine", "hybrid"]).engine == "hybrid" + + def test_engine_default_still_auto(self): + from opentranscode.cli import build_parser + assert build_parser().parse_args([]).engine == "auto" + + +class TestLauncherHybridParity: + def test_launcher_has_plan_hybrid(self, opentranscode_module): + assert hasattr(opentranscode_module, "plan_hybrid") + assert hasattr(opentranscode_module, "HybridPlan") + + def test_launcher_worker_lane_params(self, opentranscode_module): + sig = inspect.signature(opentranscode_module.EncoderWorker.__init__) + for p in ("file_subset", "lane", "ffmpeg_threads"): + assert p in sig.parameters, p + + def test_launcher_plan_matches_package_behavior(self, opentranscode_module, tmp_path): + files = _files(tmp_path, ["a.mp4", "b.mp4", "c.mp4", "d.mp4"]) + plan = opentranscode_module.plan_hybrid( + files, "hevc_nvenc", True, 28, + sizes={f: i * 10_000_000 for i, f in enumerate(files)}, + ) + pkg_plan = plan_hybrid( + files, "hevc_nvenc", True, 28, + sizes={f: i * 10_000_000 for i, f in enumerate(files)}, + ) + assert [Path(x) for x in plan.gpu_files] == pkg_plan.gpu_files + assert [Path(x) for x in plan.cpu_files] == pkg_plan.cpu_files diff --git a/tests/test_intelligent_workers.py b/tests/test_intelligent_workers.py index 5d1ebdc..14dd1b6 100755 --- a/tests/test_intelligent_workers.py +++ b/tests/test_intelligent_workers.py @@ -302,6 +302,8 @@ def test_live_tail_emits_lines(opentranscode_module, mock_env, monkeypatch): worker = make_minimal_worker(opentranscode_module, env=mock_env) worker._stop = False + # v4.2.1+: the live stderr/stdout tail only emits when verbose=True. + worker.verbose = True # Patch time.sleep so the poll loop runs instantly. monkeypatch.setattr("time.sleep", lambda *a, **k: None) @@ -359,6 +361,8 @@ def test_live_tail_handles_carriage_return(opentranscode_module, mock_env, monke worker = make_minimal_worker(opentranscode_module, env=mock_env) worker._stop = False + # v4.2.1+: the live stderr/stdout tail only emits when verbose=True. + worker.verbose = True monkeypatch.setattr("time.sleep", lambda *a, **k: None) monkeypatch.setattr("os.killpg", lambda *a, **k: None) diff --git a/tests/test_large_file_fixes.py b/tests/test_large_file_fixes.py new file mode 100644 index 0000000..aece2ae --- /dev/null +++ b/tests/test_large_file_fixes.py @@ -0,0 +1,344 @@ +"""v4.6.0 large-file + av1an chunking fix tests. + +Root causes these lock in: + 1. The pure-ffmpeg path pre-scaled to a CRF-16 intermediate — a flat + 1800s timeout and a 0.5-0.8x source size temp file killed large + files. Now the ffmpeg path scales inline and only the av1an path + pre-scales (with the per-file timeout and STOP support). + 2. av1an's FRAME MISMATCH / "encoder crashed: exit status: 0" chunk + drift fell through to "Unknown av1an failure"; the ffmpeg ≥ 7 + "-vsync" removal broke segment/hybrid chunking silently. Both are + now diagnosed and trigger the select retry. + 3. Severe disk-space warnings were verbose-only, so quiet mode gave + zero notice before "No space left on device". + 4. Loudnorm analysis decoded the whole VIDEO stream (-af only), pushing + large files past its 120s timeout; -vn makes it audio-only. +""" +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from conftest import capture_signal + +from opentranscode.codec_profiles import ( + AUDIO_PROFILES, + CONTAINER_PROFILES, + RESOLUTION_PRESETS, + VIDEO_CODECS, +) +from opentranscode.encoder_worker import EncoderWorker + + +def _worker(mock_env, use_ffmpeg_fallback, codec=None): + return EncoderWorker( + in_dir=Path("/tmp"), + out_dir=Path("/tmp"), + video_codec=codec or VIDEO_CODECS[0], + audio_profile=AUDIO_PROFILES[0], + container=CONTAINER_PROFILES[0], + crf=32, + preset_label="Medium (6)", + delete_source=False, + env=mock_env, + extensions={".mkv"}, + resolution=RESOLUTION_PRESETS[0], + use_ffmpeg_fallback=use_ffmpeg_fallback, + ) + + +# ───────────────────────────────────────────────────────────────────────────── +# _prepare_input: pre-scale only on the av1an path +# ───────────────────────────────────────────────────────────────────────────── + +class TestPreScaleGating: + def test_ffmpeg_path_skips_prescale(self, mock_env, tmp_path): + """With the ffmpeg path (incl. GPU mode), the source file is used + directly — no CRF-16 intermediate, no 30-minute pre-scale cap.""" + w = _worker(mock_env, use_ffmpeg_fallback=True) + w._temp_dir = tmp_path + w._current_temps = [] + w.resolution = RESOLUTION_PRESETS[2] # 1080p — scaling requested + w.verbose = False + + src = tmp_path / "source.mp4" + src.write_bytes(b"\x00" * 1024) + + mock_env.ffmpeg_path = "/usr/bin/ffmpeg" + result = w._prepare_input( + src, 1920, 1080, needs_scale=True, + scale_filter="scale=1920:1080", + ) + assert result is not None + encode_input, output_f = result + assert encode_input == src # NO intermediate + # No pre-scale intermediate tracked (the .av1an work-dir path is + # still tracked by the symlink section — harmless, never created + # on this path). + assert not any(".scaled_tmp" in str(t) for t in w._current_temps) + + def test_av1an_path_still_prescales(self, mock_env, tmp_path, monkeypatch): + """The av1an path keeps the CRF-16 intermediate (VS plugins need + it) but now runs it via _run_with_stop_check with the per-file + timeout instead of a flat 1800s subprocess.run cap.""" + w = _worker(mock_env, use_ffmpeg_fallback=False) + w._temp_dir = tmp_path + w._current_temps = [] + w.resolution = RESOLUTION_PRESETS[2] + w.verbose = False + w.encode_timeout = 86400 + logs: list[str] = [] + w.log_msg = capture_signal(logs) + + src = tmp_path / "source.mp4" + src.write_bytes(b"\x00" * 1024) + + seen = {} + + def fake_stop_check(cmd, env=None, timeout=None, log_prefix=" "): + seen["timeout"] = timeout + seen["cmd"] = cmd + # Fake a successful pre-scale: create the intermediate. + out_idx = cmd.index("-y") + Path(cmd[out_idx + 1]).write_bytes(b"\x00" * 4096) + return ("ok", 0, "", "") + + w._run_with_stop_check = fake_stop_check + + result = w._prepare_input( + src, 1920, 1080, needs_scale=True, + scale_filter="scale=1920:1080", + ) + assert result is not None + encode_input, output_f = result + assert encode_input != src # intermediate produced + assert str(encode_input).endswith(".scaled_tmp.mkv") + assert seen["timeout"] == 86400 # per-file timeout, NOT 1800 + assert any("libx265" in c for c in seen["cmd"]) # CRF-16 intermediate + + def test_prescale_timeout_fails_cleanly(self, mock_env, tmp_path): + """A pre-scale timeout now reports a dedicated message (the old + code surfaced a raw TimeoutExpired as 'pre-scale error').""" + w = _worker(mock_env, use_ffmpeg_fallback=False) + w._temp_dir = tmp_path + w._current_temps = [] + w.resolution = RESOLUTION_PRESETS[2] + w.verbose = False + w.encode_timeout = 3600 + w.fail_count = 0 + logs: list[str] = [] + w.log_msg = capture_signal(logs) + + src = tmp_path / "source.mp4" + src.write_bytes(b"\x00" * 1024) + w._run_with_stop_check = MagicMock(return_value=("timeout", -1, "", "")) + + result = w._prepare_input( + src, 1920, 1080, needs_scale=True, + scale_filter="scale=1920:1080", + ) + assert result is None + assert w.fail_count == 1 + assert any("pre-scale timeout" in l for l in logs) + + def test_prescale_stop_aborts_without_fail(self, mock_env, tmp_path): + """User STOP during pre-scale is not a transcode failure.""" + w = _worker(mock_env, use_ffmpeg_fallback=False) + w._temp_dir = tmp_path + w._current_temps = [] + w.resolution = RESOLUTION_PRESETS[2] + w.verbose = False + w.fail_count = 0 + logs: list[str] = [] + w.log_msg = capture_signal(logs) + + src = tmp_path / "source.mp4" + src.write_bytes(b"\x00" * 1024) + w._run_with_stop_check = MagicMock(return_value=("stop", -1, "", "")) + + result = w._prepare_input( + src, 1920, 1080, needs_scale=True, + scale_filter="scale=1920:1080", + ) + assert result is None + assert w.fail_count == 0 + + +# ───────────────────────────────────────────────────────────────────────────── +# Disk-space severe warnings are user-facing +# ───────────────────────────────────────────────────────────────────────────── + +class TestDiskSpaceWarnings: + def _setup(self, mock_env, tmp_path, verbose): + w = _worker(mock_env, use_ffmpeg_fallback=True) + w._temp_dir = tmp_path + w.verbose = verbose + logs: list[str] = [] + w.log_msg = capture_signal(logs) + src = tmp_path / "big.mkv" + src.write_bytes(b"\x00" * (2 * 1024 ** 3)) # 2 GB (sparse-ish; st_size counts) + return w, logs, src + + def test_severe_output_warning_in_quiet_mode(self, mock_env, tmp_path, monkeypatch): + w, logs, src = self._setup(mock_env, tmp_path, verbose=False) + monkeypatch.setattr( + "opentranscode.encoder_worker.shutil.disk_usage", + lambda _p: MagicMock(free=512 * 1024 ** 2), # 0.5 GB free < 2 GB + ) + w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=False) + assert any("low disk space on output" in l for l in logs), logs + + def test_no_warning_when_plenty_free_quiet(self, mock_env, tmp_path, monkeypatch): + w, logs, src = self._setup(mock_env, tmp_path, verbose=False) + monkeypatch.setattr( + "opentranscode.encoder_worker.shutil.disk_usage", + lambda _p: MagicMock(free=100 * 1024 ** 3), + ) + w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=False) + assert logs == [] + + def test_severe_temp_warning_in_quiet_mode_av1an_scale( + self, mock_env, tmp_path, monkeypatch, + ): + w, logs, src = self._setup(mock_env, tmp_path, verbose=False) + w.use_ffmpeg_fallback = False # av1an path — intermediate on temp + calls = {"n": 0} + + def fake_disk_usage(_p): + calls["n"] += 1 + # First call: output partition (plenty); second: temp (low). + free = 100 * 1024 ** 3 if calls["n"] == 1 else 512 * 1024 ** 2 + return MagicMock(free=free) + + monkeypatch.setattr("opentranscode.encoder_worker.shutil.disk_usage", + fake_disk_usage) + w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=True) + assert any("low disk space on temp" in l for l in logs), logs + + +# ───────────────────────────────────────────────────────────────────────────── +# Loudnorm analysis: audio-only decode +# ───────────────────────────────────────────────────────────────────────────── + +class TestLoudnormAudioOnly: + def test_analysis_cmd_has_vn(self, mock_env, tmp_path): + w = _worker(mock_env, use_ffmpeg_fallback=True) + w.audio_level_db = -16.0 # enable normalization path + mock_env.ffmpeg_path = "/usr/bin/ffmpeg" + + seen = {} + + def fake_run(cmd, **kw): + seen["cmd"] = cmd + res = MagicMock(returncode=1) + res.stdout = "" + res.stderr = "no json" # no JSON → returns None (fine) + return res + + # _analyze_audio_loudness uses subprocess.run directly + import opentranscode.encoder_worker as ew + monkeypatch = pytest.MonkeyPatch() + try: + monkeypatch.setattr(ew.subprocess, "run", fake_run) + gain = w._analyze_audio_loudness(tmp_path / "video.mp4") + finally: + monkeypatch.undo() + + assert gain is None # stderr had no JSON — graceful fallback + assert "-vn" in seen["cmd"], ( + "loudnorm analysis must skip video decoding (-vn) or large " + "files blow the 120s timeout" + ) + + +# ───────────────────────────────────────────────────────────────────────────── +# av1an chunking: new failure patterns trigger the select retry +# ───────────────────────────────────────────────────────────────────────────── + +_FRAME_MISMATCH_STDERR = ( + "ERROR av1an_core::broker: [chunk 2] encoder failed 3 times, shutting " + "down worker: encoder crashed: exit status: 0\n" +) +_FRAME_MISMATCH_STDOUT = ( + " FRAME MISMATCH: chunk 2: 53/52 (actual/expected frames)\n" +) +_VSYNC_STDERR = ( + "Error: FFmpeg failed to segment:\n" + ' stderr: "Unrecognized option \'vsync\'.\\nError splitting the ' + 'argument list: Option not found\\n"\n' +) + + +class TestChunkFailurePatterns: + def _encode_one_with_mocks(self, mock_env, tmp_path, stdout, stderr): + w = _worker(mock_env, use_ffmpeg_fallback=False) + w._temp_dir = tmp_path + w._current_temps = [] + w._file_res_map = {} + w._stop = False + w._consecutive_fail_count = 0 + w._last_fail_pattern = None + w.verbose = True + w.audio_level_db = 0.0 + w.encode_timeout = 60 + w._current_idx = 1 + w._current_total = 1 + w._current_filename = "test.mkv" + w._current_scale_filter = "" + mock_env.av1an_flags["svt_name"] = "svt-av1" + # conftest's mock_env pre-sets the env_probe "select" override; + # remove it so the first attempt runs with auto (the retry path + # under test). + mock_env.av1an_flags.pop("chunk_method_override", None) + logs: list[str] = [] + w.log_msg = capture_signal(logs) + + calls: list[str | None] = [] + + def fake_stop_check(cmd, env=None, timeout=None, log_prefix=" "): + cm = None + if "--chunk-method" in cmd: + cm = cmd[cmd.index("--chunk-method") + 1] + calls.append(cm) + if cm in (None, "hybrid"): + return ("ok", 1, stdout, stderr) + # select retry: fake a valid output + out = Path(cmd[cmd.index("-o") + 1]) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_bytes(b"\x00" * 8192) + return ("ok", 0, "", "encoding finished") + + w._run_with_stop_check = fake_stop_check + w._ffmpeg_fallback_encode = MagicMock(return_value=True) + + src = tmp_path / "input.mkv" + src.write_bytes(b"\x00" * 4096) + out = tmp_path / "out" / "input_archived.mkv" + result = w._encode_one(src, src, out, 1) + return w, logs, calls, result + + def test_frame_mismatch_triggers_select_retry(self, mock_env, tmp_path): + """FRAME MISMATCH (frames land in av1an's STDOUT) previously fell + to 'Unknown av1an failure'; now it retries with select.""" + w, logs, calls, result = self._encode_one_with_mocks( + mock_env, tmp_path, _FRAME_MISMATCH_STDOUT, _FRAME_MISMATCH_STDERR, + ) + assert result is True + assert calls == [None, "select"], calls + assert w.env.av1an_flags.get("chunk_method_override") == "select" + + def test_vsync_removal_diagnosed_and_retried(self, mock_env, tmp_path): + """ffmpeg ≥ 7 removed -vsync; av1an segment chunking dies with + 'Unrecognized option' — must be diagnosed, not 'Unknown'.""" + w, logs, calls, result = self._encode_one_with_mocks( + mock_env, tmp_path, "", _VSYNC_STDERR, + ) + assert result is True + assert calls == [None, "select"] + assert any("vsync" in l for l in logs), logs + + def test_frame_mismatch_not_misdiagnosed_as_concat(self, mock_env, tmp_path): + w, logs, calls, result = self._encode_one_with_mocks( + mock_env, tmp_path, _FRAME_MISMATCH_STDOUT, _FRAME_MISMATCH_STDERR, + ) + assert not any("concat failure" in l for l in logs), logs diff --git a/tests/test_package_structure.py b/tests/test_package_structure.py index f2da1d9..c64a3c3 100755 --- a/tests/test_package_structure.py +++ b/tests/test_package_structure.py @@ -250,4 +250,5 @@ class TestEntryPoints: capture_output=True, text=True, timeout=10, ) assert result.returncode == 0 - assert "4.5.0" in result.stdout + import opentranscode + assert opentranscode.__version__ in result.stdout diff --git a/tests/test_source_builder.py b/tests/test_source_builder.py new file mode 100644 index 0000000..62ac923 --- /dev/null +++ b/tests/test_source_builder.py @@ -0,0 +1,162 @@ +"""v4.7.1 rebuild-from-git tests. + +The REBUILD FROM GIT button must be always usable: on a bare system it +generates its own dependency tree (distro-aware package install), then +builds VapourSynth + BestSource + av1an into user-owned prefixes. + +Covers: + - ``build_dep_plan``: per-family package lists + install commands, + rust stripped when av1an isn't being built, manual note for + unsupported families + - ``_probe_vs_source_plugins``: discovers plugins in the python + site-packages tree (where the git VS stack installs) + - ``_av1an_env``: carries the fresh VS stack paths + - the REBUILD button is enabled without a probe (offscreen UI test) + - launcher parity +""" +import os +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from opentranscode.distro_probe import DistroProfile, detect_distro +from opentranscode.source_builder import BUILD_DEPS_BY_FAMILY, build_dep_plan + + +def _profile(family): + return DistroProfile( + family=family, name=f"Test {family}", version_id="?", + pkg_manager="pkg", install_cmd_template="", binary_extra_paths=[], + av1an_known_encoder_names=[], ffmpeg_pkg="ffmpeg", av1an_pkg="av1an", + notes="", + ) + + +class TestBuildDepPlan: + def test_arch_uses_pacman_with_zimg_and_rust(self): + plan = build_dep_plan(_profile("arch")) + assert plan.install_cmd[:2] == ["pacman", "-S"] + assert "zimg" in plan.packages + assert "rust" in plan.packages + assert "meson" in plan.packages + assert plan.manual_note is None + + def test_debian_uses_apt_with_zimg_dev(self): + plan = build_dep_plan(_profile("debian")) + assert plan.install_cmd[:2] == ["apt-get", "install"] + assert "libzimg-dev" in plan.packages + assert "cargo" in plan.packages + + def test_redhat_and_suse_covered(self): + assert build_dep_plan(_profile("redhat")).install_cmd[0] == "dnf" + assert build_dep_plan(_profile("suse")).install_cmd[0] == "zypper" + for fam in ("redhat", "suse"): + assert any("zimg" in p for p in BUILD_DEPS_BY_FAMILY[fam]) + + def test_unknown_family_gets_manual_note(self): + plan = build_dep_plan(_profile("freebsd")) + assert plan.install_cmd is None + assert "manually" in plan.manual_note + + def test_no_av1an_strips_rust(self): + plan = build_dep_plan(_profile("arch"), build_av1an=False) + assert "rust" not in plan.packages + assert "zimg" in plan.packages # still needed for VapourSynth + + def test_real_distro_detection_has_plan(self): + plan = build_dep_plan(detect_distro()) + assert plan.packages # this repo targets the covered families + + def test_every_covered_family_installs_zimg(self): + """zimg is VapourSynth's one hard library dep — a bare system + build must pull it in.""" + for family, packages in BUILD_DEPS_BY_FAMILY.items(): + assert any("zimg" in p for p in packages), family + + +class TestPluginProbeSeesGitStack: + def test_probe_discovers_user_site_plugins(self, tmp_path, monkeypatch): + import opentranscode.env_probe as ep + + plugins = tmp_path / "site-packages" / "vapoursynth" / "plugins" + plugins.mkdir(parents=True) + (plugins / "libbestsource.so").write_bytes(b"\x7fELFfake") + + monkeypatch.setattr(ep.site, "getusersitepackages", + lambda: str(tmp_path / "site-packages")) + monkeypatch.setattr(ep.site, "getsitepackages", lambda: []) + assert "bestsource" in ep._probe_vs_source_plugins() + + def test_av1an_env_carries_fresh_stack(self, tmp_path, monkeypatch): + import opentranscode.env_probe as ep + + vs_dir = tmp_path / "site-packages" / "vapoursynth" + vs_dir.mkdir(parents=True) + (vs_dir / "libvsscript.so").write_bytes(b"\x7fELFfake") + monkeypatch.setattr(ep.site, "getusersitepackages", + lambda: str(tmp_path / "site-packages")) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + monkeypatch.delenv("PYTHONPATH", raising=False) + + env = ep._av1an_env() + assert str(vs_dir) in env["LD_LIBRARY_PATH"] + assert str(tmp_path / "site-packages") in env["PYTHONPATH"] + + def test_av1an_env_without_git_stack_unchanged(self, tmp_path, monkeypatch): + import opentranscode.env_probe as ep + + monkeypatch.setattr(ep.site, "getusersitepackages", + lambda: str(tmp_path / "empty")) + monkeypatch.delenv("LD_LIBRARY_PATH", raising=False) + env = ep._av1an_env() + assert ".local/lib" in env["LD_LIBRARY_PATH"] + assert "PYTHONPATH" not in env + + +def importlib_pyside_available(): + import importlib.util + return importlib.util.find_spec("PySide6") is not None + + +class TestRebuildButtonAlwaysEnabled: + @pytest.mark.skipif( + not importlib_pyside_available(), + reason="requires a real PySide6 (stub mode cannot instantiate Qt)", + ) + def test_enabled_before_probe(self): + os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") + from PySide6.QtWidgets import QApplication + + from opentranscode.ui_window import OpenCodecMaster + + app = QApplication.instance() or QApplication([]) + window = OpenCodecMaster() + try: + assert window.btn_rebuild.isEnabled(), ( + "REBUILD FROM GIT must be usable without a successful probe" + ) + assert any( + "Hybrid (GPU + CPU)" in window.engine_combo.itemText(i) + for i in range(window.engine_combo.count()) + ) + finally: + window.close() + + + +class TestLauncherParity: + def test_launcher_has_dep_tree(self, opentranscode_module): + assert hasattr(opentranscode_module, "build_dep_plan") + assert hasattr(opentranscode_module, "BUILD_DEPS_BY_FAMILY") + + def test_launcher_worker_builds_bestsource(self, opentranscode_module): + assert hasattr(opentranscode_module.SourceBuildWorker, + "_build_bestsource") + + def test_launcher_plan_matches_package(self): + from opentranscode.source_builder import build_dep_plan as pkg_plan + plan_l = build_dep_plan(_profile("arch")) + plan_p = pkg_plan(_profile("arch")) + assert plan_l.packages == plan_p.packages + assert plan_l.install_cmd == plan_p.install_cmd