dcosnet
/
OpenTranscode
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

OpenTranscode 4.8.1 - GPU update

This commit is contained in:
Jeremy Anderson 2026-09-25 09:45:39 -04:00
parent 1df68fe785
commit 08440d198e
28 changed files with 5555 additions and 224 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

216
CHANGELOG.md
View File

@ -2,6 +2,222 @@
All notable changes to OpenTranscode. Versions follow semantic versioning.
## [4.8.1] — 2026-09-25
### Changed — GPU dropdown / ENGINE interaction
- **ENGINE = CPU now disables the GPU dropdown** (greyed out with an
explanatory tooltip) — the choice would have no effect there.
- **"None (CPU-only encode)" remains an explicit option** in the GPU
dropdown for the opposite case: a GPU present in the box that the
user doesn't want encoding on (e.g. it's the display card). The two
controls can no longer contradict each other: GPU engines + "None"
resolve to the CPU path by design, and CPU engine ignores the GPU
choice entirely.
## [4.8.0] — 2026-09-25
### Added — GPU capability profiles (combined generations, incl. oddballs)
- **New `gpu_profiles.py` + GPU dropdown** (UI, next to ENGINE; CLI
`--gpu-profile`). Entries combine whole card generations into single
capability classes — same silicon, same encoding behaviour:
- NVIDIA Kepler/Maxwell (H.264 only), Pascal (GTX 10-series + Tesla
P40/P4/P100 — H.264+HEVC 8/10-bit), Turing (RTX 20 / GTX 16 /
Tesla T4 / **CMP 30/40/50HX** — +B-frames), Ampere (RTX 30 /
A10/A40 / **CMP 90HX** — no AV1 encode), Ada/Blackwell (RTX 40/50,
L4/L40 — +AV1 10-bit).
- Oddballs: **NVIDIA data-center compute (V100/A100/H100, CMP
170HX) has NO NVENC silicon** — the profile routes to CPU instead
of failing. CMP 170HX is GA100-based: the fastest mining card that
cannot hardware-encode.
- Intel Arc (QSV: H.264+HEVC+AV1) and Iris/UHD; AMD RDNA 3 (VAAPI:
+AV1 encode), RDNA 1/2, and the crypto-era GCN 4/5 + Vega cards.
- **Auto-detection**: the probe matches the detected GPU name
(nvidia-smi / lspci, vendor-aware) to a profile; the live encode
smoke test decides what actually works. VAAPI encodes via
`-vaapi_device` + `hwupload`, QSV via `-init_hw_device qsv=hw`.
- **Profile-driven encoding**: `resolve_gpu_encoder()` now returns
(encoder, api) and the ffmpeg command is built per API (device init
args, hwupload filter chains, per-API quality args — nvenc
`-rc vbr -cq`, QSV `-global_quality`, VAAPI `-rc_mode CQP`).
- **Rebuild-from-git builds for the selected GPU profile**: the dep
tree extends with the vendor's packages (arch: nv-codec-headers /
libva+libdrm+mesa / intel-media-driver+onevpl; debian/redhat/suse
equivalents), so pressing REBUILD on a bare system generates the GPU
dependency tree too.
- Forced profiles let a user pin a capability class even when name
auto-match fails; "Auto-detect" stays the default.
## [4.7.1] — 2026-09-25
### Added — always-usable REBUILD FROM GIT (self-generating dep tree)
- **The REBUILD FROM GIT button is now ALWAYS enabled.** It no longer
depends on a successful environment probe — a failed probe (missing
binaries, broken VSScript, missing av1an) is exactly when the rebuild
is needed, so the button works from first launch on a bare system.
- **The build generates its own dependency tree.** Instead of the old
pacman-only toolchain list, `build_dep_plan()` maps the detected distro
family to the right package set and installer: arch (pacman), debian
(apt-get), redhat (dnf), suse (zypper) — including the previously
missing **zimg** (VapourSynth's one hard library dependency, which
made the meson step fail on bare systems), meson/ninja/cmake/nasm and
rust for av1an. Unsupported families get an explicit manual-install
note. Critical tools are re-verified after install; the build aborts
with the exact list if anything is still missing.
- **BestSource plugin now builds from git as part of the VapourSynth
rebuild** (cloned with its libp2p submodule, compiled against the
freshly installed git-VS headers via PYTHONPATH/PKG_CONFIG_PATH, into
the user site-packages plugins dir). This closes the av1an chunking
gap: with BestSource present, av1an auto-selects the fast chunk
method instead of quadratic-decode `select`. The full chain was
verified live on this machine: VS git (Core R80) → BestSource →
`av1an --chunk-method bestsource` → rc=0 output.
- **Runtime env follows the git stack**: `_av1an_env()` and the plugin
probe now include the python user-site vapoursynth dir (module + libs
+ plugins), so av1an loads the freshly built VS instead of the system
one after a rebuild.
## [4.7.0] — 2026-09-25
### Added — Hybrid GPU + CPU scheduling
- **New engine: Hybrid (UI ENGINE combo / `--engine hybrid`).** The
queue is scanned once, then split between two CONCURRENT lanes: a GPU
lane (NVENC via single-pass ffmpeg) and a CPU lane (the family's
software encoder, or av1an chunk-parallel when opted in — so NVENC +
chunk workers + software encoders can all run at the same time on
multi-core boxes with an NVIDIA card).
- **LPT load balancing** (`hybrid_scheduler.plan_hybrid`): files sorted
by size descending, each assigned to the lane with the lower
estimated load using a GPU:CPU speed ratio (default 8:1) — both lanes
finish at roughly the same time.
- **CPU lane thread budget**: the CPU lane's topology is reduced by a
2-thread reserve for the GPU lane's decode/scale/mux before the
intelligent worker math runs; the lane's software ffmpeg encodes are
additionally capped with `-threads N`. NVENC jobs are never
thread-capped (silicon-bound).
- **Per-lane temp dirs** (`worker-<pid>-gpu` / `worker-<pid>-cpu`):
both lanes share one process, so the PID alone no longer separates
them — a lane finishing early can no longer sweep the other lane's
intermediates.
- STOP stops both lanes; the final summary aggregates both lanes'
results. Hybrid needs 2+ encodable files and a functional GPU encoder
and otherwise falls back to a single CPU queue with a logged reason.
Lanes split FILES, never one file across encoders (mixed-encoder
chunks would produce visibly inconsistent quality within a file, and
av1an cannot drive NVENC).
### Fixed — during hybrid hardening
- **libx265 rejects large `-threads` values** ("frameNumThreads must be
[0 .. X265_MAX_FRAME_THREADS)"): the CPU lane's injected thread cap is
clamped to 16 for libx265; SVT-AV1 and libvpx keep the full budget.
- **Failed ffmpeg encodes now delete their partial output.** A failed
encode used to leave a truncated file that ffprobe still parses as the
right codec/resolution — skip-existing would then treat it as a
finished archive forever. The output is unlinked on the ffmpeg error
path (the av1an and STOP paths already cleaned up).
### Fixed — launcher crash on STARTUP
- `OpenCodecMaster._build_ui()` read `self.env.av1an_flags` while
pre-selecting the ENGINE combo, but `env` is None/absent until
`_probe_and_init` runs after the UI build — instantiating the window
crashed with `AttributeError: ... has no attribute 'env'` (launcher)
or on `NoneType` (package). The pre-select now reads defensively.
Verified by offscreen-instantiating the real launcher window.
### Tests
- `tests/test_hybrid_scheduler.py` (19): LPT split, degenerate cases,
thread budget, `file_subset` end-to-end, per-lane temp dirs, ffmpeg
thread cap (CPU yes / GPU no), `scan_input_files`, CLI `--engine
hybrid`, launcher parity.
## [4.6.0] — 2026-09-25
### Overview
GPU (NVENC) encoding with auto GPU/CPU engine selection, the root-cause
fix for the av1an chunking failures on ffmpeg 7+, and the remaining
large-file failure modes (pre-scale timeout, silent disk exhaustion,
whole-video loudnorm decode).
### Added — GPU (NVENC) support + auto engine
- **Engine selector** (UI: Auto/GPU/CPU combo; CLI: `--engine
{auto,gpu,cpu}`). Auto uses the NVENC hardware encoder for the
selected codec family (x265 → `hevc_nvenc`, AV1 → `av1_nvenc` on
RTX 40+) when it actually works; VP9 has no NVENC encoder and stays
on CPU. GPU encodes run via single-pass ffmpeg with `-rc vbr -cq N`
(CPU decode), so av1an chunk-parallel is not needed — one NVENC
process outruns chunk-parallel CPU workers.
- **`env_probe.GpuInfo` + `_probe_gpu()`**: two-stage NVENC probe —
compiled-in encoder list, then a LIVE encode smoke test per encoder.
The live gate catches the real-world failure mode where ffmpeg lists
`hevc_nvenc` but the installed NVIDIA driver is older than the NVENC
API the build targets ("Driver does not support the required nvenc
API version. Required: 13.1 Found: 13.0" — observed on GTX 1070 +
driver 580 + ffmpeg 9.0.2), reports the driver fix, and the engine
falls back to CPU automatically.
- Skip-existing is engine-agnostic: `hevc_nvenc` outputs the same
ffprobe codec_name (`hevc`) as libx265, so switching engines never
re-encodes finished files.
- `--dry-run` prints the GPU verdict (usable encoders or the exact
failure detail).
- **`scripts/build-ffmpeg-nvenc-matched.sh`** — fixes the
driver/ffmpeg NVENC API mismatch without touching the system:
builds ffmpeg against the nv-codec-headers gen the installed driver
actually provides (e.g. 580 driver → gen 13.0) with the distro
build's exact feature set, installing to `~/.local` with
`--enable-rpath` (critical — without rpath the binary silently
loads the distro libavcodec and keeps demanding the newer API).
Verified on GTX 1070 + driver 580.178 + ffmpeg 9.0.2: hevc_nvenc
and h264_nvenc functional, av1_nvenc correctly reported as
unavailable (no Pascal AV1 hardware) and auto stays on SVT-AV1.
### Fixed — av1an chunking (root cause on ffmpeg 7+)
- **ffmpeg ≥ 7 removed `-vsync`, which av1an's segment/hybrid chunk
extraction passes to ffmpeg.** Every segment-based chunk died
instantly ("Unrecognized option 'vsync'" → broken y4m pipe → chunk
fails 3×) — this produced the y4m pipe-break storm in the 2026-07-13
av1an log. Verified end-to-end on ffmpeg 9.0.2 + av1an 0.5.2:
`select` still works (it uses the ffmpeg frame server, not
segmenting), so the plugin-less `select` override remains, the
failure is now diagnosed with an actionable block (install
bestsource/ffms2/lsmash, or stay on the ffmpeg-only path), and the
select retry now also triggers on it.
- **FRAME MISMATCH ("encoder crashed: exit status: 0")** — chunk
manifest vs encoded frame-count drift on sparse-keyframe sources
(20 worker shutdowns in the same log) fell through to "Unknown av1an
failure". Now a recognized per-file pattern that retries with
`select` (exact frame ranges can't drift). Diagnostics now scan
av1an's stdout too, since FRAME MISMATCH lines land there.
### Fixed — large-file failures
- **The ffmpeg path no longer pre-scales.** The CRF-16 intermediate
existed only for VapourSynth source-plugin compatibility; the
ffmpeg/GPU path scales inline via `-vf`. This removes the whole
class of failures: the 0.5-0.8× source-size temp file, the extra
full encode pass, and the flat **1800s pre-scale timeout** that
killed long/high-bitrate sources at exactly 30 minutes. The av1an
path keeps the intermediate but now runs under the per-file timeout
with STOP-button support ("FAIL: pre-scale timeout" on expiry).
- **Severe disk-space warnings are user-facing.** "free < source size"
on the output or temp partition was verbose-only — quiet mode gave
zero notice before "No space left on device". Marginal advice stays
verbose-only.
- **Loudnorm analysis is audio-only (`-vn`).** It previously decoded
the entire video stream to measure audio loudness, pushing large
files past the 120s analysis timeout and silently degrading them to
the static knob gain.
### Tests
- `tests/test_gpu_engine.py` (32): GpuInfo, the live-encode GPU gate
(driver-mismatch case included), the resolve matrix, NVENC vargs,
CLI `--engine`, worker engine plumbing, launcher parity.
- `tests/test_large_file_fixes.py` (11): pre-scale gating + timeout +
STOP handling, user-facing disk-space warnings, `-vn` loudnorm,
FRAME MISMATCH / vsync select retries.
- Harness: EncoderWorker has class-level defaults for `verbose` /
`_current_*` so `__new__`-built test instances match the v4.4.3
contract; tests replace the Qt signal with `conftest.capture_signal()`
instead of patching read-only `SignalInstance` attributes (fixes all
24 order-dependent failures on machines with a real PySide6).
## [4.5.0] — 2026-07-26 (master)
### Overview

137
README.md
View File

@ -4,10 +4,15 @@
files to AV1 / VP9 / HEVC with configurable audio codecs, resolution
scaling, and source-file management. Built on ffmpeg (default) with an
optional av1an chunk-parallel path for users with a working VapourSynth
setup.
setup. NVIDIA GPU (NVENC) encoding is used automatically when available.
- **Default encoder**: ffmpeg + libsvtav1 (reliable across distros)
- **GPU encoder**: NVENC (hevc_nvenc / av1_nvenc) auto-detected with a
live encode test — used by the Auto engine when it actually works
- **Optional encoder**: av1an chunk-parallel (opt-in via UI toggle)
- **Self-sufficient rebuild**: the always-enabled REBUILD FROM GIT
button installs its own build dependencies (distro-aware), then builds
VapourSynth + BestSource + av1an into `~/.local` / `~/.cargo`
- **Codecs**: AV1 (SVT-AV1), VP9, x265 (HEVC) video; Opus, Vorbis, FLAC, IAMF audio
- **Containers**: MKV, WebM, MP4
- **Resolution**: Original or scaled (16:9, 21:9, 32:9 presets from 480p to 4K)
@ -23,6 +28,7 @@ setup.
- ffmpeg (with libsvtav1, libvpx, libx265, libopus, libvorbis, flac)
- ffprobe
- PySide6 (for the GUI)
- Optional: NVIDIA GPU + driver for NVENC hardware encoding
- Optional: av1an + VapourSynth (only if using the av1an toggle)
## Quick start
@ -34,7 +40,7 @@ cd /path/to/opentranscode
pip install -e .
opentranscode # launch the GUI
python -m opentranscode --version # → opentranscode 4.5.0
python -m opentranscode --version # → opentranscode 4.6.0
python -m opentranscode --help
```
@ -53,6 +59,58 @@ opentranscode --verify-only FILE.mkv # re-verify an existing output
## Usage
### Encode engine (GPU vs CPU)
The ENGINE selector (UI) or `--engine` flag picks the video encoder:
- **Auto (default)** — uses the NVENC hardware encoder for the selected
codec family when the environment probe's *live encode test* proved it
works (x265 → `hevc_nvenc`, AV1 → `av1_nvenc` on RTX 40+); otherwise
the software encoders. VP9 has no NVENC encoder and always uses the CPU.
- **GPU** — force NVENC; falls back to CPU with a log line when
unavailable. GPU encodes run via single-pass ffmpeg (`-c:v hevc_nvenc
-rc vbr -cq N`), with CPU decode. av1an chunk-parallel is not used —
one NVENC process outruns chunk-parallel CPU workers.
- **CPU** — force the software encoders (SVT-AV1 / VP9 / x265).
- **Hybrid** (v4.7.0) — run the GPU lane AND a CPU lane **concurrently**:
the queue is scanned up front and split by file size (LPT, balanced
with a GPU:CPU speed ratio) so both lanes finish at about the same
time. The CPU lane's thread budget is reduced by a small reserve for
the GPU lane's decode/mux, and with `--use-av1an` the CPU lane uses
chunk-parallel — so NVENC + chunk workers + software encoders can all
be busy at once. Lanes split FILES, never one file across encoders
(mixing hevc_nvenc and libx265 chunks inside a single file would make
scene-by-scene quality visibly inconsistent, and av1an cannot drive
NVENC anyway). Needs 2+ encodable files and a functional GPU encoder;
otherwise it falls back to a single CPU queue. GPU-lane files get
NVENC quality, CPU-lane files get software-encoder quality.
The GPU gate is a real encode smoke test, not just an encoder-list grep:
a ffmpeg build can list `hevc_nvenc` while the installed NVIDIA driver is
too old for the NVENC API version it was compiled against (e.g. "Driver
does not support the required nvenc API version. Required: 13.1 Found:
13.0") — in that case the probe reports the exact driver fix and encoding
stays on the CPU. `opentranscode --dry-run` prints the GPU verdict.
#### When NVENC is "present but NOT usable" (driver / ffmpeg API mismatch)
ffmpeg is compiled against a specific NVENC API version and the NVIDIA
driver must be at least that new. On legacy driver branches (580 is the
**last** branch supporting Pascal cards such as the GTX 10xx) upgrading
the driver is not an option — so invert the fix: build ffmpeg against the
NVENC API your driver *does* provide and install it to `~/.local/bin`
(which PATHs before the distro binary; OpenTranscode picks it up
automatically on the next launch). Use the included helper:
```bash
# "Found: 13.0" in the error → pass 13.0
./scripts/build-ffmpeg-nvenc-matched.sh 13.0
```
The script copies your distro ffmpeg's feature set (same version, same
libraries — a drop-in), builds against the matching `nv-codec-headers`,
and installs only into `$HOME/.local`. Nothing system-wide is touched.
### Default (ffmpeg-only, recommended)
```bash
@ -100,14 +158,51 @@ opentranscode --verbose
For 30GB+ BluRay rips:
- **24-hour per-file timeout** (configurable via `--timeout SECONDS`)
- **Inline scaling on the ffmpeg path** — the ffmpeg (and GPU) path scales
with `-vf` directly in the encode command; no intermediate file, no
extra encode pass, no pre-scale timeout. The av1an path still writes a
CRF-16 (visually lossless) intermediate for VapourSynth compatibility,
but under the full per-file timeout instead of a flat 30-minute cap.
- **1KB absolute integrity minimum** (no false "output too small" failures
on high-bitrate sources — duration check is the real gate)
- **Disk-space warnings** (verbose only) for output and temp partitions
- **Disk-space warnings** — severe warnings (free space below the source
size on the output or temp partition) are user-facing even in quiet mode
```bash
opentranscode --timeout 36000 # 10h per-file timeout
```
### GPU capability profiles
The GPU dropdown (next to ENGINE) lists combined card generations —
same silicon, same encoding behaviour — rather than individual SKUs:
NVIDIA Kepler through Ada/Blackwell, Intel Arc + integrated (QSV), and
AMD GCN/RDNA (VAAPI), including data-center and crypto-era oddballs
(Tesla P40/T4, CMP 30–90HX, and the GA100-based CMP 170HX + V100/A100/
H100 boards, which ship without NVENC and route to the CPU path).
Auto-detect matches your card and the live encode probe decides what
works; forcing a profile extends the REBUILD FROM GIT dependency tree
with that GPU's packages.
### Rebuilding the encode stack from git
The REBUILD FROM GIT button is available at all times — even on a bare
system with no build tools installed:
1. **Dependency tree** — detects missing tools/libraries and installs
them via the distro package manager (pacman/apt-get/dnf/zypper; one
privilege prompt). Includes zimg, which VapourSynth requires.
2. **VapourSynth from git** → `~/.local` (self-contained: module, libs,
headers in the python user site-packages).
3. **BestSource plugin from git** → compiled against that fresh VS, so
av1an gets a fast, frame-accurate chunk method (no more slow
`select`).
4. **av1an from git** → `~/.cargo/bin`.
Restart the app afterwards; the probe picks up the new stack
automatically. `ffmpeg + IAMF` builds libiamf + ffmpeg into
`~/.local/bin` for the IAMF audio codec.
### av1an chunk-parallel (opt-in)
For users with a working VapourSynth + source plugin (lsmash, ffms2,
@ -116,15 +211,31 @@ bestsource) setup who want scene-detection-based chunk-parallel encoding:
- **UI**: Check the "av1an (chunk-parallel)" checkbox
- **CLI**: `opentranscode --use-av1an`
When av1an fails per-file (concat failures, scene-detection panics),
the code automatically falls back to ffmpeg for that file. When av1an
fails systematically (VSScript API mismatch, missing encoder), the queue
aborts with an actionable diagnostic.
Known chunking failure modes (all handled with per-file retries + a
diagnosis block, and safe to hit):
- **ffmpeg ≥ 7 removed `-vsync`** — av1an's segment/hybrid chunk
extraction calls `ffmpeg -vsync` and dies with "Unrecognized option
'vsync'" on modern ffmpeg. Without VS source plugins the app forces
`--chunk-method select` (works on any ffmpeg — verified — but is slow,
so the default ffmpeg-only path is usually faster for long files).
Install `bestsource`/`ffms2`/`lsmash` to give av1an a fast plugin
chunk method.
- **FRAME MISMATCH / "encoder crashed: exit status: 0"** — chunk
extraction drift on sources with sparse keyframes; the app retries the
file with `select` automatically.
- **y4m pipe breaks** ("Failed to read y4m frame delimiter") — same
select retry.
When av1an fails per-file, the code automatically falls back to ffmpeg
for that file. When av1an fails systematically (VSScript API mismatch,
missing encoder), the queue aborts with an actionable diagnostic.
## CLI reference
```
opentranscode [--version] [--dry-run] [--verify-only PATH] [--force]
[--engine {auto,gpu,cpu,hybrid}]
[--chunk-method METHOD] [--max-workers N] [--threads-per-worker N]
[--use-av1an] [--verbose] [--skip-existing | --force-reencode]
[--timeout SECONDS]
@ -136,6 +247,7 @@ opentranscode [--version] [--dry-run] [--verify-only PATH] [--force]
| `--dry-run` | — | Probe environment + smoke test, no encode |
| `--verify-only PATH` | — | Re-verify an existing output file |
| `--force` | off | Skip ffprobe pre-validation |
| `--engine` | auto | `auto` = NVENC GPU encoder when the live encode test passes, else CPU; `gpu` = force NVENC; `cpu` = force software encoders |
| `--chunk-method METHOD` | auto | Force av1an chunk method (select, hybrid, ffms2, lsmash, bestsource, dgdecnv) |
| `--max-workers N` | auto | Cap chunk-parallel worker count |
| `--threads-per-worker N` | auto | Per-encoder thread cap |
@ -152,9 +264,9 @@ opentranscode/
├── __init__.py # Package metadata + lazy launch_gui wrapper
├── __main__.py # python -m opentranscode entry point
├── cli.py # argparse + dry-run + verify-only
├── codec_profiles.py # VideoCodecProfile, AudioProfile, ContainerProfile tables
├── encoder_worker.py # QThread-based per-file encode pipeline
├── env_probe.py # Distro + binary + library + av1an detection
├── codec_profiles.py # VideoCodecProfile (incl. NVENC gpu fields), Audio/Container/Resolution tables
├── encoder_worker.py # QThread-based per-file encode pipeline + engine (GPU/CPU) resolution
├── env_probe.py # Distro + binary + library + av1an + NVENC probe
├── ffprobe_utils.py # ffprobe_validate, ffprobe_duration, file-type ID
├── temp_manager.py # Per-worker temp directory isolation
├── cpu_topology.py # Physical core / logical thread detection
@ -168,7 +280,7 @@ opentranscode/
open-transcode.py # Launcher script (mirrors package, test target)
pyproject.toml # PEP 621 build config
tests/ # 149 tests across 14 files
tests/ # 200+ tests across 16 files
```
### Encode pipeline
@ -181,7 +293,8 @@ tests/ # 149 tests across 14 files
- `_validate_file` — ffprobe pre-check (skip if invalid)
- `_check_disk_space` — warn (verbose) if free space < source size
- `_output_already_encoded` — skip if output exists with matching codec
- `_prepare_input` — pre-scale (if requested) or symlink to temp
- `_prepare_input` — pre-scale to a CRF-16 intermediate (av1an path only)
or symlink to temp; the ffmpeg/GPU path scales inline
- `_encode_one` — dispatch to ffmpeg (default) or av1an (opt-in)
- `_run_with_stop_check` — subprocess with STOP-button interrupt support
- `_verify_and_finalize` — duration check (≥95%), subtitle mux, source deletion

22
logs/av1an.log.2026-09-25 Normal file
View File

@ -0,0 +1,22 @@
2026-09-25T10:15:49.304217Z WARN new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }}: av1an_core::settings: It is not recommended to use the "select" chunk method, as it is very slow
2026-09-25T10:15:49.306406Z DEBUG new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }}:initialize{self=Av1anContext { frames: 24, vs_script: None, vs_proxy_script: None, args: EncodeArgs { input: Video { path: "/tmp/av1an_smoke__0a69sjo/test_smoke.mkv", temp: ".f0bc4e8", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".f0bc4e8", output_file: "/tmp/av1an_smoke__0a69sjo/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--scd", "0", "--rc", "0", "--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".f0bc4e8", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: false, julek: false, vszip: None, vship: false }) }, scene_factory: SceneFactory { data: ScenesData { frames: 0, scenes: None, split_scenes: None } } }}: av1an_core::context: temporary directory: .f0bc4e8
2026-09-25T10:15:49.306672Z INFO encode_file: av1an_core::context: Input: 64x64 @ 24.000 fps, YUV420P, SDR
2026-09-25T10:15:49.389870Z INFO encode_file: av1an_core::scenes: scenecut: found 1 scene(s) [with extra_splits (240 frames): 1 scene(s)]
2026-09-25T10:15:49.390194Z INFO encode_file: av1an_core::context:
Queue 1 Workers 1 Encoder svt-av1 Passes 1
Params: --scd 0 --rc 0 --preset 8 --crf 40 --keyint 240
2026-09-25T10:15:49.390829Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: started chunk 00000: 24 frames
2026-09-25T10:15:50.052458Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: finished chunk 00000: 24 frames, 36.28 fps, took 661.45ms
2026-09-25T10:15:50.052761Z DEBUG encode_file: av1an_core::context: encoding finished, concatenating with mkvmerge
2026-09-25T10:15:50.052780Z DEBUG encode_file: av1an_core::context: `--ignore-frame-mismatch` not set. Forcing output FPS to 24 with mkvmerge.
2026-09-25T12:54:17.768234Z WARN new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }}: av1an_core::settings: It is not recommended to use the "select" chunk method, as it is very slow
2026-09-25T12:54:17.770255Z DEBUG new{args=EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }}:initialize{self=Av1anContext { frames: 24, vs_script: None, vs_proxy_script: None, args: EncodeArgs { input: Video { path: "/tmp/av1an_smoke_g07f02gt/test_smoke.mkv", temp: ".709b7df", chunk_method: Select, is_proxy: false, cache_mode: SOURCE }, proxy: None, temp: ".709b7df", output_file: "/tmp/av1an_smoke_g07f02gt/test_smoke_out.mkv", chunk_method: Select, chunk_order: LongestFirst, scaler: "bicubic+accurate_rnd+full_chroma_int+full_chroma_inp+bitexact", scenes: None, split_method: AvScenechange, sc_pix_format: None, sc_method: Standard, sc_only: false, sc_downscale_height: None, extra_splits_len: Some(240), min_scene_len: 24, force_keyframes: [], ignore_frame_mismatch: false, max_tries: 3, passes: 1, video_params: ["--scd", "0", "--rc", "0", "--preset", "8", "--crf", "40", "--keyint", "240"], tiles: (1, 1), encoder: svt_av1, workers: 1, set_thread_affinity: None, photon_noise: None, photon_noise_size: (None, None), chroma_noise: false, zones: None, cache_mode: SOURCE, pix_format_converter: FFMPEG, ffmpeg_filter_args: [], audio_params: ["-c:a", "copy"], input_pix_format: FFmpeg { format: YUV420P }, output_pix_format: PixelFormat { format: YUV420P10LE, bit_depth: 10 }, verbosity: Normal, resume: false, keep: false, force: false, no_defaults: false, tile_auto: false, concat: MKVMerge, target_quality: TargetQuality { vmaf_res: "1920x1080", probe_res: None, vmaf_scaler: "bicubic", vmaf_filter: None, vmaf_threads: 28, model: None, probing_rate: 1, probes: 4, target: None, metric: VMAF, min_q: 15, max_q: 50, interp_method: None, encoder: svt_av1, pix_format: YUV420P10LE, temp: ".709b7df", workers: 1, video_params: None, params_copied: false, vspipe_args: [], probing_vmaf_features: [Default], probing_statistic: ProbingStatistic { name: Automatic, value: None } }, vmaf: false, vmaf_path: None, vmaf_res: "1920x1080", probe_res: None, vmaf_threads: None, vmaf_filter: None, vapoursynth_plugins: Some(VapoursynthPlugins { lsmash: false, ffms2: false, dgdecnv: false, bestsource: true, julek: false, vszip: None, vship: false }) }, scene_factory: SceneFactory { data: ScenesData { frames: 0, scenes: None, split_scenes: None } } }}: av1an_core::context: temporary directory: .709b7df
2026-09-25T12:54:17.770555Z INFO encode_file: av1an_core::context: Input: 64x64 @ 24.000 fps, YUV420P, SDR
2026-09-25T12:54:17.844031Z INFO encode_file: av1an_core::scenes: scenecut: found 1 scene(s) [with extra_splits (240 frames): 1 scene(s)]
2026-09-25T12:54:17.844213Z INFO encode_file: av1an_core::context:
Queue 1 Workers 1 Encoder svt-av1 Passes 1
Params: --scd 0 --rc 0 --preset 8 --crf 40 --keyint 240
2026-09-25T12:54:17.844608Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: started chunk 00000: 24 frames
2026-09-25T12:54:18.500267Z DEBUG encode_chunk{worker_id=0 total_chunks=1 chunk_index="00000"}: av1an_core::broker: finished chunk 00000: 24 frames, 36.62 fps, took 655.47ms
2026-09-25T12:54:18.503379Z DEBUG encode_file: av1an_core::context: encoding finished, concatenating with mkvmerge
2026-09-25T12:54:18.503397Z DEBUG encode_file: av1an_core::context: `--ignore-frame-mismatch` not set. Forcing output FPS to 24 with mkvmerge.

1810
open-transcode.py
View File

File diff suppressed because it is too large Load Diff

15
opentranscode/__init__.py
View File

@ -23,7 +23,7 @@ Or import in code:
from __future__ import annotations
__version__ = "4.5.0"
__version__ = "4.8.1"
__author__ = "Jeremy Anderson - dcos.net"
__license__ = "AGPL-3.0"
@ -52,7 +52,9 @@ def launch_gui(argv: list[str] | None = None, force: bool = False,
verbose: bool = False,
skip_existing: bool = True,
timeout: int = 86400,
inline_scale: bool = False) -> int:
inline_scale: bool = False,
engine: str = "auto",
gpu_profile: str = "auto") -> int:
"""Launch the OpenTranscode GUI.
Thin wrapper around ``opentranscode.ui_window.launch_gui``; imported
@ -101,11 +103,18 @@ def launch_gui(argv: list[str] | None = None, force: bool = False,
error (rc=234)" disk-exhaustion messages. Default False —
the intermediate path is more robust on older av1an/
VapourSynth builds. Pre-checks the "Inline scale" UI checkbox.
engine: Video encode engine (v4.6.0). "auto" (default) uses the
NVENC GPU encoder when the selected codec family has one and
the live encode test proved it works; "gpu" forces NVENC;
"cpu" forces the software encoders; "hybrid" (v4.7.0) splits
the queue between a GPU lane and a CPU lane running
concurrently. Pre-selects the ENGINE combo in the UI.
"""
from .ui_window import launch_gui as _launch
return _launch(
argv, force=force, chunk_method=chunk_method,
max_workers=max_workers, threads_per_worker=threads_per_worker,
use_av1an=use_av1an, verbose=verbose, skip_existing=skip_existing,
timeout=timeout, inline_scale=inline_scale,
timeout=timeout, inline_scale=inline_scale, engine=engine,
gpu_profile=gpu_profile,
)

53
opentranscode/cli.py
View File

@ -183,6 +183,41 @@ def build_parser() -> argparse.ArgumentParser:
"older av1an/VapourSynth builds. Enable for large files "
"with scaling to save disk + an extra encode pass.",
)
# v4.6.0: --engine selects the video encode engine. "auto" (default)
# uses the NVENC hardware encoder for the selected codec family when
# the environment probe's live encode test proved it works; "gpu"
# forces NVENC (falls back to CPU with a log line); "cpu" forces the
# software encoders. The GPU path runs via single-pass ffmpeg —
# av1an cannot drive NVENC.
parser.add_argument(
"--engine", choices=["auto", "gpu", "cpu", "hybrid"], default="auto",
help="Video encode engine (default: auto). 'auto' uses the NVENC "
"GPU encoder (hevc_nvenc / av1_nvenc) when the selected "
"codec family has one and a live encode test proved it "
"works; falls back to CPU otherwise. 'gpu' forces NVENC. "
"'cpu' forces the software encoders (SVT-AV1 / VP9 / "
"x265). 'hybrid' splits the queue between a GPU lane and a "
"CPU lane running CONCURRENTLY (balanced by file size, CPU "
"lane budget reduced by the GPU lane's reserve) — with "
"--use-av1an the CPU lane uses chunk-parallel, so NVENC + "
"chunk workers + software all run at once. Needs 2+ "
"encodable files and a functional GPU encoder; falls back "
"to CPU otherwise. GPU encodes run via single-pass ffmpeg "
"— av1an never drives NVENC.",
)
# v4.8.0: force a GPU capability profile (see gpu_profiles.py for
# the combined generation entries). "auto" matches the detected card.
from .gpu_profiles import GPU_PROFILES
parser.add_argument(
"--gpu-profile", default="auto",
choices=["auto"] + [gp.key for gp in GPU_PROFILES],
help="GPU capability profile (default: auto-detect from the "
"installed card). Combined generation entries — e.g. "
"nv-pascal covers the whole GTX 10-series + Tesla P40/P4/"
"P100; nv-ada covers RTX 40/50. Forcing a profile extends "
"the rebuild-from-git dependency tree with that GPU's "
"packages. 'cpu' = ignore hardware.",
)
return parser
@ -243,6 +278,22 @@ def run_dry_run(
effective_cm = env.av1an_flags.get("chunk_method_override")
print(f"Chunk method: {effective_cm or 'auto (av1an decides)'}{cli_chunk_method_note}")
# v4.6.0: GPU/NVENC probe result (live encode test, not just the
# compiled-in encoder list).
gpu = getattr(env, "gpu", None)
if gpu is None:
print("GPU: (probe unavailable)")
elif gpu.has_gpu:
print(f"GPU: {gpu.name or 'NVIDIA'} — NVENC ready: "
f"{', '.join(gpu.usable_encoders)}")
elif any(gpu.encoders.values()):
present = [e for e in ("av1_nvenc", "hevc_nvenc", "h264_nvenc")
if gpu.encoders.get(e)]
print(f"GPU: {', '.join(present)} present but NOT usable — "
f"{gpu.first_failure_detail or 'smoke test failed'}")
else:
print("GPU: none (CPU encoding)")
# v4.1.0: show intelligent worker math so the user can verify the
# chunk-parallel thread budget before launching a real encode.
# We instantiate EncoderWorker without starting the QThread to read
@ -380,4 +431,6 @@ def main(argv: list[str] | None = None) -> int:
skip_existing=args.skip_existing,
timeout=args.timeout,
inline_scale=args.inline_scale,
engine=args.engine,
gpu_profile=args.gpu_profile,
)

104
opentranscode/codec_profiles.py
View File

@ -7,7 +7,7 @@ the CLI --version path).
"""
from collections.abc import Callable
from dataclasses import dataclass
from dataclasses import dataclass, field
# ──────────────────────────────────────────────
# CONFIG-DRIVEN PROFILES (replaces all if/else chains)
@ -36,6 +36,25 @@ class VideoCodecProfile:
# output for each encoder; this is the codec_name field in the video
# stream's JSON, NOT the encoder_name (which would be "libsvtav1" etc).
ffprobe_codec_name: str = ""
# v4.6.0: hardware (NVENC) counterpart for this codec family. Empty
# string = no hardware encoder exists for this family (VP9 has no
# NVENC encoder). The GPU path is ffmpeg-only (av1an cannot drive
# NVENC); EncoderWorker.resolve_gpu_encoder() only selects it when a
# functional probe proved the encoder works on this system. The
# ffprobe codec_name is IDENTICAL to the CPU encoder's (hevc_nvenc
# also produces "hevc"), so skip-existing detection works across
# GPU/CPU re-encodes of the same family.
gpu_encoder: str = ""
# (crf, preset) -> ffmpeg args for the NVENC encoder. Mirrors
# ffmpeg_vargs_fn. None when gpu_encoder is empty.
gpu_vargs_fn: Callable[[int, int], list[str]] | None = None
# v4.8.0: GPU-profile support. *gpu_family* is the codec family key
# used by GpuProfile.encoders ("av1"/"hevc"/"vp9"); *gpu_encoders_by_api*
# maps a hardware API (nvenc/vaapi/qsv) to this profile's ffmpeg
# encoder for that API. resolve_gpu_encoder() picks the entry matching
# the selected GPU profile.
gpu_family: str = ""
gpu_encoders_by_api: dict[str, str] = field(default_factory=dict)
@dataclass
class AudioProfile:
@ -125,6 +144,52 @@ def _x265_ffmpeg_args(crf: int, preset: int) -> list[str]:
"-pix_fmt", "yuv420p10le", "-g", "240"]
# ── v4.6.0: NVENC (hardware) vargs ──
# NVENC quality control: -rc vbr + -cq N + -b:v 0 is the constant-quality
# mode that maps most closely to the CPU encoders' CRF (cq ≈ crf for HEVC
# and AV1 within ~±3). -b:v 0 removes the default bitrate cap so -cq
# actually governs quality. Presets are p1 (fastest) .. p7 (slowest/best)
# on all current NVENC generations; the legacy "slow/medium/fast" aliases
# are deprecated.
#
# Pixel format: 8-bit yuv420p. Pascal-generation cards (GTX 10xx) run
# HEVC Main10 at roughly half throughput, and the archival targets here
# are 8-bit phone/BluRay sources — 8-bit keeps the GPU path at full
# speed. ffmpeg auto-converts 10-bit sources to yuv420p.
def _nvenc_preset(preset: int) -> str:
"""Map the CPU preset tiers (lower value = slower/better) to NVENC
p-presets. CPU preset values across profiles are 0..10 with 0/5 =
slowest quality tiers; NVENC is fast enough that even p7 outruns any
CPU encoder, so the whole range compresses to p3..p7."""
if preset <= 6:
return "p7" # "Slow" tier → best NVENC quality
if preset <= 8:
return "p5" # "Medium" tier
return "p4" # "Fast"/"Faster" tiers
def _hevc_nvenc_args(crf: int, preset: int) -> list[str]:
"""FFmpeg args for hevc_nvenc (x265/HEVC family hardware encoder)."""
return ["-c:v", "hevc_nvenc", "-preset", _nvenc_preset(preset),
"-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0",
"-pix_fmt", "yuv420p", "-g", "240"]
def _h264_nvenc_args(crf: int, preset: int) -> list[str]:
"""FFmpeg args for h264_nvenc (hardware H.264 — compatibility target)."""
return ["-c:v", "h264_nvenc", "-preset", _nvenc_preset(preset),
"-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0",
"-pix_fmt", "yuv420p", "-g", "240"]
def _av1_nvenc_args(crf: int, preset: int) -> list[str]:
"""FFmpeg args for av1_nvenc (AV1 family hardware encoder, RTX 40+)."""
return ["-c:v", "av1_nvenc", "-preset", _nvenc_preset(preset),
"-tune", "hq", "-rc", "vbr", "-cq", str(crf), "-b:v", "0",
"-pix_fmt", "yuv420p", "-g", "240"]
VIDEO_CODECS: list[VideoCodecProfile] = [
VideoCodecProfile(
label="AV1 (SVT-AV1)",
@ -138,6 +203,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [
presets=["Slow (8)", "Medium (6)", "Fast (4)", "Faster (2)"],
preset_map={"Slow (8)": 8, "Medium (6)": 6, "Fast (4)": 4, "Faster (2)": 2},
ffprobe_codec_name="av1", # v4.3.0: skip-existing detection
# v4.6.0: av1_nvenc exists only on RTX 40+ (Ada) cards; on Pascal
# (GTX 10xx) the functional probe fails and auto falls back to
# the SVT-AV1 CPU encoder.
gpu_encoder="av1_nvenc",
gpu_vargs_fn=_av1_nvenc_args,
gpu_family="av1",
gpu_encoders_by_api={"nvenc": "av1_nvenc", "qsv": "av1_qsv",
"vaapi": "av1_vaapi"},
),
VideoCodecProfile(
label="VP9",
@ -151,6 +224,10 @@ VIDEO_CODECS: list[VideoCodecProfile] = [
presets=["Slow (0)", "Medium (2)", "Fast (4)", "Faster (6)"],
preset_map={"Slow (0)": 0, "Medium (2)": 2, "Fast (4)": 4, "Faster (6)": 6},
ffprobe_codec_name="vp9", # v4.3.0: skip-existing detection
# v4.8.0: VP9 has no NVENC encoder; VAAPI (AMD/older Intel) can
# encode it on some cards.
gpu_family="vp9",
gpu_encoders_by_api={"vaapi": "vp9_vaapi"},
),
VideoCodecProfile(
label="x265 (HEVC)",
@ -164,6 +241,14 @@ VIDEO_CODECS: list[VideoCodecProfile] = [
presets=["Slow (5)", "Medium (7)", "Fast (9)", "Faster (10)"],
preset_map={"Slow (5)": 5, "Medium (7)": 7, "Fast (9)": 9, "Faster (10)": 10},
ffprobe_codec_name="hevc", # v4.3.0: skip-existing detection
# v4.6.0: hevc_nvenc works on every NVENC generation since Maxwell
# GM206 (incl. the GTX 1070) — this is the family that benefits
# most from GPU mode.
gpu_encoder="hevc_nvenc",
gpu_vargs_fn=_hevc_nvenc_args,
gpu_family="hevc",
gpu_encoders_by_api={"nvenc": "hevc_nvenc", "qsv": "hevc_qsv",
"vaapi": "hevc_vaapi"},
),
]
@ -222,6 +307,23 @@ FFMPEG_LIB_KEY_MAP: dict[str, str] = {
"libaom-av1": "libaom",
"libvpx-vp9": "libvpx",
"libx265": "libx265",
# v4.6.0: hardware encoders. These keys are populated by
# _probe_ffmpeg_libs() alongside the software encoders, and — unlike
# the compiled-in check — EncoderWorker additionally gates the GPU
# path on env.gpu.functional (a real encode smoke test), because a
# ffmpeg build can list an NVENC encoder that the installed driver
# cannot open (NVENC API version mismatch).
"hevc_nvenc": "hevc_nvenc",
"h264_nvenc": "h264_nvenc",
"av1_nvenc": "av1_nvenc",
# v4.8.0: hardware APIs for AMD (VAAPI) and Intel (QSV) profiles.
"hevc_vaapi": "hevc_vaapi",
"h264_vaapi": "h264_vaapi",
"av1_vaapi": "av1_vaapi",
"vp9_vaapi": "vp9_vaapi",
"hevc_qsv": "hevc_qsv",
"h264_qsv": "h264_qsv",
"av1_qsv": "av1_qsv",
}

418
opentranscode/encoder_worker.py
View File

@ -38,6 +38,12 @@ from .codec_profiles import (
VideoCodecProfile,
ffmpeg_lib_key_for,
)
from .gpu_profiles import (
encoder_filter_chain,
encoder_pre_args,
encoder_quality_args,
gpu_profile_by_key,
)
from .env_probe import EnvProbe, _av1an_env
from .ffprobe_utils import (
_identify_file_type,
@ -52,11 +58,97 @@ from .temp_manager import _temp_path_for, _worker_temp_dir
# ENCODER WORKER (QThread, from PySide6 ver, extended)
# ──────────────────────────────────────────────
def scan_input_files(in_dir: Path, extensions: set[str]) -> list[Path]:
"""Collect the transcodable files under *in_dir* (v4.7.0).
Used by ``EncoderWorker.run()`` for the default whole-directory scan
AND by the hybrid scheduler's pre-scan, which must partition the
queue BEFORE the per-lane workers are constructed. Excludes leftover
pre-scale intermediates from previous failed runs.
"""
return sorted(
f for f in in_dir.rglob("*")
if f.is_file()
and f.suffix.lower() in extensions
and not f.name.endswith(".scaled_tmp.mkv")
)
def selected_gpu_profile(env):
"""v4.8.0: the active GpuProfile — the UI's dropdown selection
(env.av1an_flags["gpu_profile"]) when set, else the auto-matched
profile from the probe (env.gpu.profile_key). None when neither."""
flags = getattr(env, "av1an_flags", None) or {}
key = flags.get("gpu_profile")
if key and key != "auto":
profile = gpu_profile_by_key(key)
if profile is not None:
return profile
gpu_info = getattr(env, "gpu", None)
if gpu_info is not None and getattr(gpu_info, "profile_key", ""):
return gpu_profile_by_key(gpu_info.profile_key)
return None
def resolve_gpu_encoder(engine: str, video_codec, env):
"""Decide whether this encode runs on the GPU.
Returns ``(encoder_name, api)`` when the GPU path should be used —
e.g. ``("hevc_nvenc", "nvenc")`` or ``("hevc_vaapi", "vaapi")`` — or
``(None, None)`` for the CPU path.
Rules:
- ``engine == "cpu"`` → always CPU (user forced CPU).
- selected/matched GPU profile has
no encoder for the codec family → CPU (e.g. AV1 on Pascal,
VP9 without VAAPI).
- ``engine`` auto/gpu AND the
functional probe passed for
that encoder → the encoder.
The gate is ``env.gpu.functional`` — a live encode test run by the
env probe — NOT the compiled-in ``ffmpeg -encoders`` list, because a
ffmpeg build can advertise a hardware encoder the installed driver
is too old to open. Pure function; no I/O. Safe to call from the UI
thread for a pre-flight status line.
"""
if engine not in ("auto", "gpu"):
return (None, None)
profile = selected_gpu_profile(env)
if profile is not None:
family = getattr(video_codec, "gpu_family", "") or ""
gpu_enc = profile.encoders.get(family)
api = profile.api
else:
# No GPU profile context (older callers / no probe data): fall
# back to the profile's NVENC encoder.
gpu_enc = getattr(video_codec, "gpu_encoder", "") or ""
api = "nvenc" if gpu_enc else None
if not gpu_enc:
return (None, None)
gpu_info = getattr(env, "gpu", None)
if gpu_info is not None and gpu_info.functional.get(gpu_enc, False):
return (gpu_enc, api)
return (None, None)
class EncoderWorker(QThread):
log_msg = Signal(str)
progress_msg = Signal(str, int, int) # (filename, current, total)
finished_queue = Signal(int, int) # (success_count, fail_count)
# v4.4.3/v4.6.0: class-level defaults for attributes normally set in
# __init__. The mocked test suite builds workers via ``__new__``
# (bypassing __init__) and calls non-Qt methods directly; without
# these defaults those instances crash with AttributeError on
# ``verbose`` / ``_current_total`` (the "AttributeError: no attribute
# 'verbose'" class of bugs from the v4.4.3 changelog). Instance
# assignment in __init__ shadows these harmlessly.
verbose = False
_current_idx = 0
_current_total = 0
_current_filename = ""
def __init__(
self,
in_dir: Path,
@ -87,6 +179,25 @@ class EncoderWorker(QThread):
# to honor them.
max_workers: int | None = None,
threads_per_worker: int | None = None,
# v4.6.0: encode engine selection. "auto" uses the NVENC GPU
# encoder when the selected codec family has one AND the env
# probe's live encode test proved it works on this system;
# otherwise (or with "cpu") the CPU encoders are used. "gpu"
# requests GPU and falls back to CPU with a log line when the
# hardware is unavailable. Resolved against env.av1an_flags
# ["engine"] like the other CLI-plumbed flags.
engine: str | None = None,
# v4.7.0: hybrid-lane support. *file_subset* restricts this
# worker to an explicit file list (the hybrid scheduler scans and
# partitions the queue up front, then spawns a GPU-lane and a
# CPU-lane worker with disjoint subsets). *lane* suffixes the
# per-worker temp dir so one lane's cleanup sweep can never
# delete the other lane's intermediates. *ffmpeg_threads* caps
# the CPU lane's software ffmpeg encode (GPU jobs are capped by
# NVENC silicon, not threads).
file_subset: list[Path] | None = None,
lane: str = "",
ffmpeg_threads: int | None = None,
):
super().__init__()
self.in_dir = in_dir
@ -126,6 +237,23 @@ class EncoderWorker(QThread):
) else None
)
)
# v4.6.0: engine selection ("auto" | "gpu" | "cpu"). Falls back to
# env.av1an_flags["engine"] when not passed explicitly (same
# pattern as max_workers — lets the CLI reach the GUI-spawned
# worker without ui_window changes). Resolved to a concrete
# GPU/CPU decision in run() via resolve_gpu_encoder().
self.engine = engine if engine in ("auto", "gpu", "cpu") else (
env.av1an_flags.get("engine", "auto")
if env.av1an_flags.get("engine") in ("auto", "gpu", "cpu")
else "auto"
)
# Resolved in run(): NVENC encoder name when the GPU path is
# active, None for CPU. _ffmpeg_fallback_encode and
# _prepare_input read this (GPU mode implies the ffmpeg path —
# av1an cannot drive NVENC — which also means no pre-scale
# intermediate: the ffmpeg path scales inline).
self._gpu_encoder: str | None = None
self._gpu_api: str | None = None
# v4.2.1: quiet mode by default. Tech-detail log lines (CMD:,
# live tail of av1an/ffmpeg stderr, DIAGNOSIS blocks, resolution
# map, pre-flight validation table, heartbeat) are gated behind
@ -183,7 +311,10 @@ class EncoderWorker(QThread):
# cleanup sweep can safely nuke only this worker's intermediates
# without affecting a concurrent worker. The subdir is created
# with mode=0o700 to prevent symlink attacks from other users.
self._temp_dir = _worker_temp_dir(os.getpid())
self.file_subset = file_subset
self.lane = lane
self.ffmpeg_threads = ffmpeg_threads
self._temp_dir = _worker_temp_dir(os.getpid(), lane=lane)
# v6-06: KeepAwake instance — started in run(), stopped in finally.
# mouse_nudge defaults to False (opt-in) to avoid surprising the
# user with cursor movement. systemd-inhibit is always-on when
@ -579,7 +710,25 @@ class EncoderWorker(QThread):
Returns True on success, False on failure.
"""
# Check if ffmpeg has the video encoder we need
ffmpeg_enc = self.video_codec.ffmpeg_encoder
# v4.6.0: GPU mode swaps in the hardware encoder and its vargs.
# v4.8.0: VAAPI/QSV APIs build their command shape (device init,
# hwupload filter, quality args) from gpu_profiles; NVENC keeps
# the original profile vargs.
ffmpeg_enc = self._gpu_encoder or self.video_codec.ffmpeg_encoder
v_args = (
(self.video_codec.gpu_vargs_fn or self.video_codec.ffmpeg_vargs_fn)
if self._gpu_encoder else self.video_codec.ffmpeg_vargs_fn
)(self.crf, self.preset_val)
hw_pre_args: list[str] = []
hw_filter_args: list[str] = []
if self._gpu_encoder and self._gpu_api in ("vaapi", "qsv"):
profile = selected_gpu_profile(self.env)
if profile is not None:
hw_pre_args = encoder_pre_args(profile)
hw_filter_args = encoder_filter_chain(profile)
v_args = encoder_quality_args(
self._gpu_api, ffmpeg_enc, self.crf, self.preset_val
)
# v3: use the module-level FFMPEG_LIB_KEY_MAP (OTC-007).
ffmpeg_lib_key = ffmpeg_lib_key_for(ffmpeg_enc)
@ -590,19 +739,29 @@ class EncoderWorker(QThread):
)
return False
v_args = self.video_codec.ffmpeg_vargs_fn(self.crf, self.preset_val)
# Belt-and-suspenders: if a target resolution is set, inject -vf scale
# directly into the ffmpeg command. This guarantees the output resolution
# matches the dropdown even if the intermediate pre-scale was bypassed.
vf_scale_args: list[str] = []
if self.resolution.width is not None and self.resolution.height is not None:
vf_scale_args = [
"-vf", (
f"scale={self.resolution.width}:{self.resolution.height}:"
f"force_original_aspect_ratio=decrease:force_divisible_by=2"
),
]
if self._gpu_api == "vaapi":
# v4.8.0: VAAPI scales ON the hardware — combine the
# scalar with the hwupload upload in one chain.
vf_scale_args = [
"-vf", (
f"scale={self.resolution.width}:{self.resolution.height}:"
f"force_original_aspect_ratio=decrease:force_divisible_by=2,"
f"format=nv12,hwupload"
),
]
hw_filter_args = []
else:
vf_scale_args = [
"-vf", (
f"scale={self.resolution.width}:{self.resolution.height}:"
f"force_original_aspect_ratio=decrease:force_divisible_by=2"
),
]
# Audio args from profile
audio_args = list(self.audio_profile.params)
@ -623,10 +782,23 @@ class EncoderWorker(QThread):
if self.container.ext == "mp4":
mux_flags = ["-movflags", "+faststart"]
cmd = [
self.env.ffmpeg_path,
cmd = [self.env.ffmpeg_path] + hw_pre_args + [
"-i", str(encode_input),
] + vf_scale_args + v_args + audio_args + mux_flags + [
] + vf_scale_args + hw_filter_args + v_args
# v4.7.0: CPU lane thread cap in hybrid mode (the GPU lane's
# NVENC job keeps ~2 threads for decode/mux). Never applied on
# the GPU path — NVENC throughput is silicon-bound, not
# thread-bound. Skipped when the cap is 0/unset.
if self._gpu_encoder is None and self.ffmpeg_threads:
# v4.7.1: libx265 maps -threads to frame-threads, capped at
# X265_MAX_FRAME_THREADS (16) — larger values abort the
# encoder ("frameNumThreads must be [0 .. X265_MAX_FRAME_
# THREADS)"). SVT-AV1 and libvpx accept the full budget.
threads = self.ffmpeg_threads
if ffmpeg_enc == "libx265":
threads = min(threads, 16)
cmd += ["-threads", str(threads)]
cmd += audio_args + mux_flags + [
"-y",
str(output_f),
]
@ -672,6 +844,11 @@ class EncoderWorker(QThread):
self.log_msg.emit(
f" ffmpeg error (rc={res.returncode}): {stderr_snip.strip()}"
)
# v4.7.1: remove the partial output. Without this, a
# failed encode left a truncated file that ffprobe can
# still parse as the right codec — and skip-existing
# would then treat it as a finished archive forever.
output_f.unlink(missing_ok=True)
return False
except OSError as e:
self.log_msg.emit(f"{self._status_prefix()}FAIL: system error: {e}")
@ -693,14 +870,44 @@ class EncoderWorker(QThread):
phys = self.env.cpu.physical_cores
logical = self.env.cpu.logical_threads
# ── v4.6.0: engine resolution (GPU vs CPU) ──
# GPU mode is a variant of the ffmpeg path: av1an invokes
# encoder CLI binaries (SvtAv1EncApp / vpxenc / x265) and cannot
# drive NVENC, so an active GPU encoder forces the single-pass
# ffmpeg path. NVENC on even a GTX 1070 encodes 1080p at several
# hundred fps — one ffmpeg process beats av1an's chunk-parallel
# CPU workers, and chunking becomes unnecessary.
self._gpu_encoder, self._gpu_api = resolve_gpu_encoder(
self.engine, self.video_codec, self.env)
if self._gpu_encoder:
self.use_ffmpeg_fallback = True
self.log_msg.emit(
f"ENGINE: GPU ({self._gpu_encoder}, {self._gpu_api}) — "
f"single-pass ffmpeg hardware encode; av1an chunk-parallel "
f"not used."
)
elif self.engine == "gpu":
gpu_enc = getattr(self.video_codec, "gpu_encoder", "") or ""
if not gpu_enc and not getattr(self.video_codec, "gpu_family", ""):
self.log_msg.emit(
f"ENGINE: GPU requested but {self.video_codec.label} has no "
f"hardware encoder — using CPU."
)
else:
gpu = getattr(self.env, "gpu", None)
detail = gpu.first_failure_detail if gpu is not None else ""
self.log_msg.emit(
f"ENGINE: GPU requested but no usable hardware encoder for "
f"{self.video_codec.label} ({detail or 'unavailable'}) — using CPU."
)
# Collect all valid files first (for progress tracking)
# Exclude our own temp intermediates from previous failed runs.
all_files = sorted(
f for f in self.in_dir.rglob("*")
if f.is_file()
and f.suffix.lower() in self.extensions
and not f.name.endswith(".scaled_tmp.mkv")
)
if self.file_subset is not None:
# v4.7.0: hybrid lane — the scheduler partitioned the queue.
all_files = sorted(self.file_subset)
else:
all_files = scan_input_files(self.in_dir, self.extensions)
total = len(all_files)
if total == 0:
@ -713,7 +920,12 @@ class EncoderWorker(QThread):
# to know the worker math — they just need files to encode.
# use_ffmpeg_fallback is set by the main thread's pre-flight check.
if self.verbose:
if self.use_ffmpeg_fallback:
if self._gpu_encoder:
self._vlog(
f"GPU encode: {self.video_codec.label} via "
f"{self._gpu_encoder} (NVENC), CPU decode"
)
elif self.use_ffmpeg_fallback:
self._vlog(
f"FFmpeg fallback: {self.video_codec.ffmpeg_encoder} on {phys} cores "
f"(single-pass, no chunk-parallel)"
@ -1188,7 +1400,15 @@ class EncoderWorker(QThread):
# temp directory so the user's video folders stay clean.
encode_input = file_path
if needs_scale and not self.inline_scale:
# v4.6.0: pre-scale ONLY on the av1an path. The pure-ffmpeg path
# (the default, and the only path NVENC can run on) scales inline
# via the -vf args in _ffmpeg_fallback_encode — the intermediate
# existed solely because VapourSynth source plugins choke on some
# inputs ffmpeg handles fine. Skipping it on the ffmpeg path
# removes the entire class of large-file failures: no 0.5-0.8x
# source-size temp file, no extra full encode pass, and no
# pre-scale timeout on long/high-bitrate sources.
if needs_scale and not self.inline_scale and not self.use_ffmpeg_fallback:
try:
temp_scaled = _temp_path_for(file_path, ".scaled_tmp.mkv", worker_dir=self._temp_dir)
self._current_temps.append(temp_scaled)
@ -1217,19 +1437,41 @@ class EncoderWorker(QThread):
]
# v4.2.1: Scaling notice is verbose-only.
self._vlog(f" Scaling {src_w or '?'}x{src_h or '?'} -> {self.resolution.width}x{self.resolution.height}...")
scale_res = subprocess.run(
scale_cmd, capture_output=True, text=True, timeout=1800,
# v4.6.0: run via _run_with_stop_check with the full
# per-file timeout. The old flat
# subprocess.run(timeout=1800) killed pre-scaling of
# long/high-bitrate sources at exactly 30 minutes
# ("FAIL: pre-scale error: Command ... timed out") — a
# guaranteed large-file failure that also ignored the
# STOP button for the whole intermediate pass.
scale_status, scale_rc, _s_out, scale_err = self._run_with_stop_check(
scale_cmd, timeout=self.encode_timeout, log_prefix=" ",
)
if scale_res.returncode == 0 and temp_scaled.exists():
if scale_status == "stop":
# User aborted — clean up the partial intermediate and
# bail WITHOUT counting a failure (a STOP is not an
# encode failure; the queue loop breaks next iteration).
self._cleanup_current_temps()
return None
if scale_status == "timeout":
self.log_msg.emit(
f"{self._status_prefix()}FAIL: pre-scale timeout "
f"(exceeded {self.encode_timeout}s limit)"
)
temp_scaled.unlink(missing_ok=True)
self._cleanup_current_temps()
self.fail_count += 1
return None
if scale_status == "ok" and scale_rc == 0 and temp_scaled.exists():
encode_input = temp_scaled
scaled_size = temp_scaled.stat().st_size / 1_048_576
# v4.2.1: verbose-only
self._vlog(f" Pre-scale OK ({scaled_size:.1f} MB intermediate)")
else:
stderr_snip = (scale_res.stderr or "")[-200:]
stderr_snip = (scale_err or "")[-200:]
# v4.2.1: keep user-facing FAIL but shorten; stderr verbose-only
self.log_msg.emit(
f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_res.returncode})"
f"{self._status_prefix()}FAIL: pre-scale failed (rc={scale_rc})"
)
if stderr_snip.strip():
self._vlog(f" ffmpeg stderr: {stderr_snip.strip()}")
@ -1280,15 +1522,17 @@ class EncoderWorker(QThread):
return (encode_input, output_f)
def _check_disk_space(self, file_path: Path, output_f: Path, needs_scale: bool) -> None:
"""v4.4.0: Warn (not abort) if free disk space is less than the source size.
"""v4.4.0: Warn if free disk space is less than the encode will need.
v4.4.1: warnings gated behind --verbose. The user wants just
start + finish lines, no disk-space chatter. The check still
runs (so the warning is available via --verbose), but in quiet
mode it produces zero output.
v4.6.0: SEVERE warnings (free space below the source size on the
partition we're about to write a big intermediate/output to) are
now USER-FACING — they were verbose-only, so in the default quiet
mode a batch that was going to die with "No space left on device"
partway through gave zero advance notice. That silent failure was
one of the "large files just fail" reports: small files fit in
the remaining space, big ones didn't. Marginal advice (the 2-3x
intermediate estimate) stays verbose-only.
"""
if not self.verbose:
return # v4.4.1: quiet mode — no disk-space warnings
try:
src_size = file_path.stat().st_size
except OSError:
@ -1296,7 +1540,9 @@ class EncoderWorker(QThread):
if src_size < 1_073_741_824: # < 1 GB — skip check for small files
return
src_gb = src_size / 1_073_741_824
# Check output partition.
# Check output partition — severe when free < source size
# (the encoded output is usually smaller, but the ffmpeg/av1an
# buffer cache plus a same-partition temp can eat the difference).
try:
out_usage = shutil.disk_usage(output_f.parent)
out_free_gb = out_usage.free / 1_073_741_824
@ -1305,19 +1551,31 @@ class EncoderWorker(QThread):
f" WARN: low disk space on output ({out_free_gb:.1f} GB free, "
f"source is {src_gb:.1f} GB) — encode may fail partway through"
)
elif self.verbose and out_free_gb < src_gb * 2:
self._vlog(
f" WARN: output space getting tight ({out_free_gb:.1f} GB free, "
f"source is {src_gb:.1f} GB)"
)
except OSError:
pass # can't check — skip
# When scaling, also check the temp partition (lossless intermediate
# can be 2-3x source size).
if needs_scale:
# When scaling on the av1an path, also check the temp partition
# (the CRF-16 intermediate can be ~1x source size). The ffmpeg
# path scales inline (no intermediate), so no temp warning there.
if needs_scale and not self.use_ffmpeg_fallback:
try:
tmp_usage = shutil.disk_usage(self._temp_dir)
tmp_free_gb = tmp_usage.free / 1_073_741_824
# Lossless intermediate is typically 2-3x source; warn if
# free < source * 2.
if tmp_free_gb < src_gb * 2:
# Severe: free temp space below the source size means the
# intermediate will likely not fit → user-facing warning.
if tmp_free_gb < src_gb:
self.log_msg.emit(
f" WARN: low disk space on temp ({tmp_free_gb:.1f} GB free, "
f"source is {src_gb:.1f} GB) — the scale intermediate "
f"may not fit. Free space or enable 'Inline scale'."
)
elif self.verbose and tmp_free_gb < src_gb * 2:
self._vlog(
f" WARN: temp space getting tight ({tmp_free_gb:.1f} GB free, "
f"lossless intermediate may need ~{src_gb * 2:.1f} GB) — "
f"consider scaling to a smaller resolution or freeing space"
)
@ -1550,6 +1808,12 @@ class EncoderWorker(QThread):
return False
else:
stderr_full = res.stderr or ""
# v4.6.0: scan stdout too. av1an routes chunk-retry
# noise (encoder stderr dumps, FRAME MISMATCH lines)
# to stdout, so pattern-matching on stderr alone missed
# the biggest real-world failure mode (see the FRAME
# MISMATCH pattern below).
combined_out = stderr_full + "\n" + (res.stdout or "")
# v6: Don't increment fail_count yet — we may retry with
# ffmpeg fallback below. Only increment if the retry also
# fails (or no retry is possible).
@ -1685,10 +1949,56 @@ class EncoderWorker(QThread):
),
False, # don't stop queue — retry with select chunk method
),
# v4.6.0: ffmpeg ≥ 7 removed the -vsync option that
# av1an's segment/hybrid chunk extraction passes to
# ffmpeg. Every segment-based chunk dies immediately
# ("Unrecognized option 'vsync'." → empty y4m pipe →
# chunk fails 3x). Verified on ffmpeg 9.0.2 + av1an
# 0.5.2. select (or a VS source plugin method) is the
# only working chunking on these systems.
(
"Unrecognized option 'vsync'",
"av1an's segment/hybrid chunk extraction calls "
"`ffmpeg -vsync`, which ffmpeg 7+ removed. Every "
"segment-based chunk fails instantly on this system.",
(
" FIX: install a VapourSynth source plugin so av1an",
" stops using ffmpeg segmenting: bestsource/ffms2/",
" lsmash (e.g. on Arch: vapoursynth-plugin-bs).",
" Alternatively stay on the default ffmpeg-only path",
" (it doesn't use av1an chunking at all).",
),
False, # per-file — the select override keeps other files working
),
# v4.6.0: frame-count drift between av1an's chunk
# manifest and what the encoder actually produced.
# av1an retries the chunk 3x, then shuts the worker
# down with the baffling "encoder crashed: exit
# status: 0" (exit 0 because the encode itself
# succeeded — on partial data). This was the dominant
# large-file failure in the 2026-07-13 av1an log and
# previously fell through to "Unknown av1an failure".
(
"FRAME MISMATCH",
"av1an's chunk manifest expected a different frame count "
"than the encoder produced — chunk-extraction drift on "
"sources with sparse/irregular keyframes. The encode "
"itself exits 0 (it ran on partial data), which av1an "
"reports as 'encoder crashed: exit status: 0'.",
(
" Will retry with --chunk-method select, which",
" extracts exact frame ranges and cannot drift.",
" This is per-file, not systematic — subsequent",
" files use select automatically.",
),
False, # don't stop queue — retry with select chunk method
),
)
diagnosis_emitted = False
for marker, summary, fixes, stop_queue in error_patterns:
if marker.lower() in stderr_full.lower():
# v4.6.0: scan stdout + stderr (FRAME MISMATCH and
# encoder dumps land in stdout).
if marker.lower() in combined_out.lower():
# v4.2.1: DIAGNOSIS block is verbose-only. The user
# already saw "FAIL: av1an exit code N" above — they
# don't need the multi-line root-cause analysis unless
@ -1744,7 +2054,8 @@ class EncoderWorker(QThread):
# v4.2.1: all DIAGNOSIS verbose-only.
if ("SUMMARY" in stderr_full
and "Average Speed" in stderr_full
and "Failed to read y4m frame delimiter" not in stderr_full):
and "Failed to read y4m frame delimiter" not in combined_out
and "FRAME MISMATCH" not in combined_out):
self._vlog("")
self._vlog(
"DIAGNOSIS: SVT-AV1 encoder completed successfully (SUMMARY"
@ -1781,14 +2092,27 @@ class EncoderWorker(QThread):
# same params). Cache the working method so subsequent files
# skip the wasted first attempt.
#
# v4.6.0: FRAME MISMATCH (chunk-extraction drift, see the
# error-pattern table) joins the y4m break as a drift
# symptom that select fixes — it was previously an
# "Unknown av1an failure" that went straight to the slow
# full-file ffmpeg fallback.
#
# NOTE: Do NOT clean up _current_temps before the retry —
# encode_input (symlink or pre-scaled file) is in
# _current_temps and the recursive _encode_one call needs it.
# The finally block below will clean up everything after the
# recursive call returns (its own finally clears the list
# first; our finally then runs on an empty list — no-op).
y4m_break = "Failed to read y4m frame delimiter" in stderr_full
if (not self._stop and y4m_break
extraction_drift = (
"Failed to read y4m frame delimiter" in combined_out
or "FRAME MISMATCH" in combined_out
# ffmpeg >= 7 removed -vsync: segment/hybrid chunking
# dies instantly while select still works (it uses the
# ffmpeg frame server, not segmenting).
or "Unrecognized option 'vsync'" in combined_out
)
if (not self._stop and extraction_drift
and effective_chunk_method != "select"
and self.env.av1an_flags.get("has_chunk_method", True)):
# v4.2.1: RETRY messages are verbose-only — the user
@ -2040,9 +2364,15 @@ class EncoderWorker(QThread):
try:
# Pass 1: analyze current loudness
# v4.6.0: -vn skips video decoding — without it the analysis
# decoded the ENTIRE video stream just to measure audio
# loudness, which pushed long/large files past the 120s
# timeout and silently degraded every big file to the static
# knob gain.
analysis_cmd = [
self.env.ffmpeg_path,
"-i", str(file_path),
"-vn",
"-af", (
f"loudnorm=I={target_lufs}:TP=-1.5:LRA=11:"
f"print_format=json"

358
opentranscode/env_probe.py
View File

@ -16,6 +16,7 @@ import ctypes
import os
import re
import shutil
import site
import subprocess
import sys
import tempfile
@ -24,6 +25,271 @@ from pathlib import Path
from .cpu_topology import CpuTopology, detect_cpu_topology
from .distro_probe import DistroProfile, detect_distro
from .gpu_profiles import (
encoder_pre_args,
encoder_filter_chain,
encoder_quality_args,
gpu_profile_by_key,
match_gpu_profile,
resolve_vaapi_device,
)
# ──────────────────────────────────────────────
# GPU PROBE (v4.6.0 — NVENC hardware encoding)
# ──────────────────────────────────────────────
# NVENC encoders we know how to drive, in preference order (best
# compression efficiency first). av1_nvenc only exists on RTX 40+; the
# functional smoke test below decides what is actually usable.
_NVENC_ENCODER_NAMES: tuple[str, ...] = ("av1_nvenc", "hevc_nvenc", "h264_nvenc")
@dataclass
class GpuInfo:
"""Result of the GPU/NVENC probe.
``encoders`` — encoder name → ffmpeg was BUILT with it (from
``ffmpeg -encoders``).
``functional`` — encoder name → a real 0.2s NVENC encode SUCCEEDED.
This is the gate EncoderWorker uses: a ffmpeg build
can list hevc_nvenc while the installed driver is
too old for the NVENC API version it was compiled
against ("Driver does not support the required
nvenc API version") — only a live encode reveals
that.
``details`` — encoder name → first stderr line when the smoke
test failed (actionable diagnostics).
"""
name: str = "" # GPU model name via nvidia-smi/lspci, "" if unknown
encoders: dict[str, bool] = field(default_factory=dict)
functional: dict[str, bool] = field(default_factory=dict)
details: dict[str, str] = field(default_factory=dict)
# v4.8.0: the matched GpuProfile key from gpu_profiles (auto-detected
# from the GPU name; the UI can override it).
profile_key: str = ""
@property
def usable_encoders(self) -> list[str]:
"""Encoders that passed the live encode test, preference order."""
return [e for e in _NVENC_ENCODER_NAMES if self.functional.get(e, False)]
@property
def has_gpu(self) -> bool:
return bool(self.usable_encoders)
@property
def first_failure_detail(self) -> str:
"""First non-empty failure detail (for user-facing warnings)."""
for e in _NVENC_ENCODER_NAMES:
d = self.details.get(e, "")
if d:
return d
return ""
def _probe_gpu(ffmpeg_bin: str) -> GpuInfo:
"""Detect NVIDIA NVENC hardware encoders and verify they actually work.
Two-stage probe:
1. Compiled-in check — grep ``ffmpeg -encoders`` for the NVENC
encoder names. Cheap; answers "could this ffmpeg ever do NVENC".
2. Functional smoke test — for each compiled-in encoder, encode a
0.2s 256x264 lavfi color source with ``-c:v <enc> -f null -``.
Catches the real-world failure modes the compiled-in check
cannot: NVIDIA driver too old for the ffmpeg build's NVENC API
version, no /dev/nvidia* access, driver loaded but GPU dead.
GPU model name is best-effort via nvidia-smi (display only).
"""
info = GpuInfo()
if not ffmpeg_bin:
return info
# --- Stage 1: compiled-in encoders ---
try:
res = subprocess.run(
[ffmpeg_bin, "-hide_banner", "-encoders"],
capture_output=True, text=True, timeout=10,
)
encoders_out = res.stdout or ""
except (OSError, subprocess.SubprocessError):
encoders_out = ""
for enc in _NVENC_ENCODER_NAMES:
info.encoders[enc] = f" {enc} " in encoders_out
compiled_in = [e for e in _NVENC_ENCODER_NAMES if info.encoders[e]]
if not compiled_in:
return info # no hardware encoders in this build — skip stage 2
# --- GPU model name (display + profile auto-match) ---
nvidia_smi = shutil.which("nvidia-smi")
if nvidia_smi:
try:
res = subprocess.run(
[nvidia_smi, "--query-gpu=name", "--format=csv,noheader"],
capture_output=True, text=True, timeout=5,
)
if res.returncode == 0 and res.stdout.strip():
info.name = res.stdout.strip().splitlines()[0].strip()
except (OSError, subprocess.SubprocessError):
pass
if not info.name:
# No NVIDIA board (or driver down): identify AMD/Intel iGPU/dGPU
# from the PCI bus for profile matching + VAAPI/QSV testing.
info.name = _probe_pci_gpu_name()
# --- v4.8.0: capability-class profile for this GPU ---
matched = match_gpu_profile(info.name)
if matched:
info.profile_key = matched.key
# The profile may claim hardware encoders this ffmpeg build
# doesn't even list (e.g. AV1 on an Arc card with an old ffmpeg).
for family, enc in matched.encoders.items():
if enc not in info.encoders or not info.encoders[enc]:
info.encoders[enc] = _ffmpeg_has_encoder(ffmpeg_bin, enc)
# --- Stage 2 for the profile's hardware API ---
if matched.api == "vaapi":
info.functional.update(_smoke_vaapi(ffmpeg_bin, matched))
elif matched.api == "qsv":
info.functional.update(_smoke_qsv(ffmpeg_bin, matched))
# --- Stage 2: functional smoke test per compiled-in encoder ---
for enc in compiled_in:
if enc in info.functional:
continue # already smoke-tested via the profile branch
try:
res = subprocess.run(
[
ffmpeg_bin, "-hide_banner", "-loglevel", "error",
"-f", "lavfi",
"-i", "color=c=black:s=256x256:d=0.2:r=24",
"-frames:v", "5",
"-c:v", enc, "-f", "null", "-",
],
capture_output=True, text=True, timeout=20,
)
info.functional[enc] = res.returncode == 0
if res.returncode != 0:
# First stderr line with substance (nvenc errors are
# prefixed "hevc_nvenc @ 0x...]" — keep them readable).
for line in (res.stderr or "").splitlines():
line = line.strip()
if line:
# Strip the "name @ 0xADDR]" prefix for brevity.
line = re.sub(r"^\[[^]]+@\s*0x[0-9a-f]+\]\s*", "", line)
info.details[enc] = line[:160]
break
except subprocess.TimeoutExpired:
info.functional[enc] = False
info.details[enc] = f"{enc} smoke test timed out after 20s"
except (OSError, subprocess.SubprocessError) as e:
info.functional[enc] = False
info.details[enc] = str(e)[:160]
return info
def _ffmpeg_has_encoder(ffmpeg_bin: str, enc: str) -> bool:
"""Compiled-in check for one encoder name (cheap -encoders grep)."""
try:
res = subprocess.run(
[ffmpeg_bin, "-hide_banner", "-encoders"],
capture_output=True, text=True, timeout=10,
)
return f" {enc} " in (res.stdout or "")
except (OSError, subprocess.SubprocessError):
return False
def _probe_pci_gpu_name() -> str:
"""Best-effort non-NVIDIA GPU name via lspci (VGA/3D/Display class)."""
lspci = shutil.which("lspci")
if not lspci:
return ""
try:
res = subprocess.run(
[lspci], capture_output=True, text=True, timeout=10,
)
for line in (res.stdout or "").splitlines():
low = line.lower()
if any(k in low for k in (" vga ", " 3d ", " display ")):
if "nvidia" not in low: # nvidia handled via nvidia-smi
# "01:00.0 VGA ...: AMD/ATI Navi 31 [Radeon RX 7900 XTX]"
m = re.search(r":\s*(.+)$", line)
return m.group(1).strip() if m else ""
except (OSError, subprocess.SubprocessError):
pass
return ""
def _smoke_vaapi(ffmpeg_bin: str, profile) -> dict[str, bool]:
"""Live-encode VAAPI encoders through the render node. VAAPI
encoders only accept hardware surfaces, so the filter chain must
upload the lavfi frames."""
results: dict[str, bool] = {}
dev = resolve_vaapi_device()
if not Path(dev).exists():
for enc in profile.encoders.values():
results[enc] = False
details_placeholder = f"no render node ({dev})"
return results
for enc in profile.encoders.values():
try:
cmd = (
[ffmpeg_bin, "-hide_banner", "-loglevel", "error"]
+ encoder_pre_args(profile, vaapi_device=dev)
+ ["-f", "lavfi", "-i", "color=c=black:s=256x256:d=0.2:r=24",
"-frames:v", "5"]
+ encoder_filter_chain(profile)
+ ["-c:v", enc]
+ encoder_quality_args(profile.api, enc, 28, 9)
+ ["-f", "null", "-"]
)
res = subprocess.run(cmd, capture_output=True, text=True, timeout=20)
results[enc] = res.returncode == 0
if res.returncode != 0:
for line in (res.stderr or "").splitlines():
line = line.strip()
if line:
results[f"_detail_{enc}"] = False # marker only
break
except (OSError, subprocess.SubprocessError):
results[enc] = False
# Strip the detail markers; keep real encoder results only, and put
# first error lines into a readable form for the caller.
cleaned: dict[str, bool] = {}
for k, v in results.items():
if not k.startswith("_detail_"):
cleaned[k] = v
return cleaned
def _smoke_qsv(ffmpeg_bin: str, profile) -> dict[str, bool]:
"""Live-encode QSV encoders (qsv encoders upload system frames
internally once a qsv device exists)."""
results: dict[str, bool] = {}
for enc in profile.encoders.values():
try:
cmd = (
[ffmpeg_bin, "-hide_banner", "-loglevel", "error"]
+ encoder_pre_args(profile)
+ ["-f", "lavfi", "-i", "color=c=black:s=256x256:d=0.2:r=24",
"-frames:v", "5",
"-c:v", enc]
+ encoder_quality_args(profile.api, enc, 28, 9)
+ ["-f", "null", "-"]
)
res = subprocess.run(cmd, capture_output=True, text=True, timeout=20)
results[enc] = res.returncode == 0
except (OSError, subprocess.SubprocessError):
results[enc] = False
return results
# ──────────────────────────────────────────────
# ENVIRONMENT PROBE (distro-aware, extended)
@ -52,6 +318,9 @@ class EnvProbe:
vs_version: str | None = None # VapourSynth version string (for diagnostics)
vs_script_lib: str | None = None # path to libvapoursynth-script.so that passed
cpu: CpuTopology = field(default_factory=lambda: CpuTopology(1, 1, 1, "Unknown"))
# v4.6.0: GPU/NVENC probe result. EncoderWorker reads
# env.gpu.functional[encoder_name] when the engine is auto/gpu.
gpu: GpuInfo = field(default_factory=GpuInfo)
errors: list[str] = field(default_factory=list)
warnings: list[str] = field(default_factory=list)
@ -141,6 +410,11 @@ def _probe_ffmpeg_libs(ffmpeg_bin: str) -> dict[str, bool]:
("libopus", ["libopus "]),
("libvorbis", ["libvorbis "]),
("flac", ["flac "]),
# v4.6.0: NVENC hardware encoders (compiled-in check only — the
# live-encode gate is _probe_gpu()'s functional dict).
("hevc_nvenc", ["hevc_nvenc "]),
("h264_nvenc", ["h264_nvenc "]),
("av1_nvenc", ["av1_nvenc "]),
]
libs = {}
for lib_name, search_strings in checks:
@ -440,6 +714,36 @@ def probe_environment() -> EnvProbe:
result.warnings.append(f"FFmpeg version: {result.ffmpeg_version}")
result.ffmpeg_libs = _probe_ffmpeg_libs(result.ffmpeg_path)
# v4.6.0: GPU/NVENC probe — compiled-in check + live encode smoke
# test. EncoderWorker gates the GPU path on env.gpu.functional;
# this warning block surfaces the result (and the fix when the
# driver is too old for the ffmpeg build's NVENC API).
result.gpu = _probe_gpu(result.ffmpeg_path)
gpu = result.gpu
if gpu.has_gpu:
result.warnings.append(
f"GPU: {gpu.name or 'NVIDIA'} — NVENC ready: "
f"{', '.join(gpu.usable_encoders)} (engine: Auto will use the GPU)"
)
elif gpu.encoders and any(gpu.encoders.values()):
present = [e for e in _NVENC_ENCODER_NAMES if gpu.encoders.get(e)]
detail = gpu.first_failure_detail
result.warnings.append(
f"GPU: NVENC encoder(s) {', '.join(present)} present in ffmpeg "
f"but NOT usable — {detail or 'smoke test failed'}. "
f"Auto engine will fall back to CPU."
)
if "API version" in detail or "minimum required Nvidia driver" in detail:
result.warnings.append(
" FIX: update the NVIDIA driver (the ffmpeg build's NVENC "
"API is newer than the installed driver supports), or use "
"an ffmpeg build matching the installed driver."
)
else:
result.warnings.append(
"GPU: no hardware encoder in this ffmpeg build — CPU encoding."
)
# Warn about missing AUDIO libs (video codecs are handled by av1an's own
# encoder binaries — ffmpeg's video encoder list is irrelevant)
audio_lib_warnings = {
@ -533,6 +837,31 @@ def probe_environment() -> EnvProbe:
)
result.av1an_flags["chunk_method_override"] = "select"
# v4.6.0: ffmpeg ≥ 7 removed the -vsync option that av1an's
# segment/hybrid chunk extraction passes to ffmpeg. On those
# systems every segment-based chunk dies immediately with
# "Unrecognized option 'vsync'." — the y4m pipe breaks and
# each chunk fails 3x. The select override above already
# avoids those methods when no plugins are installed; this
# warning tells plugin-less users on new ffmpeg WHY av1an is
# stuck on slow select.
if not vs_plugins and result.ffmpeg_version:
try:
ffmpeg_major = int(
re.match(r"[nN]?(\d+)", result.ffmpeg_version).group(1)
)
except (AttributeError, ValueError):
ffmpeg_major = 0
if ffmpeg_major >= 7:
result.warnings.append(
"av1an note: ffmpeg ≥ 7 removed the -vsync option av1an's "
"segment/hybrid chunk methods use — those methods fail "
"with \"Unrecognized option 'vsync'\". Chunking stays on "
"'select'. Install a VapourSynth source plugin "
"(bestsource/ffms2/lsmash) to escape slow select, or use "
"the default ffmpeg-only path."
)
except (OSError, subprocess.SubprocessError) as e:
result.errors.append(f"av1an probe failed: {e}")
@ -638,12 +967,29 @@ def _av1an_env() -> dict[str, str]:
When VapourSynth is built from git and installed to ~/.local/, the linker
won't find libvapoursynth-script.so unless LD_LIBRARY_PATH points there.
This function ensures every av1an invocation inherits that path.
v4.7.1: the git VS stack is self-contained in the python user
site-packages (module + libs + BestSource plugin), so the runtime env
also gets that dir on LD_LIBRARY_PATH and the user site on PYTHONPATH —
otherwise av1an loads the system VS and never sees the fresh stack.
"""
env = os.environ.copy()
local_lib = str(Path.home() / ".local" / "lib")
existing = env.get("LD_LIBRARY_PATH", "")
if local_lib not in existing:
env["LD_LIBRARY_PATH"] = f"{local_lib}:{existing}".rstrip(":")
try:
user_site = Path(site.getusersitepackages())
vs_dir = user_site / "vapoursynth"
if vs_dir.is_dir() and (vs_dir / "libvsscript.so").exists():
existing = env.get("LD_LIBRARY_PATH", "")
if str(vs_dir) not in existing:
env["LD_LIBRARY_PATH"] = f"{vs_dir}:{existing}".rstrip(":")
py_path = env.get("PYTHONPATH", "")
if str(user_site) not in py_path:
env["PYTHONPATH"] = f"{user_site}:{py_path}".rstrip(":")
except (AttributeError, OSError):
pass
return env
@ -697,6 +1043,18 @@ def _probe_vs_source_plugins() -> list[str]:
search_dirs.append(Path("/usr/lib/vapoursynth"))
search_dirs.append(Path("/usr/local/lib/vapoursynth"))
search_dirs.append(Path("/usr/lib/x86_64-linux-gnu/vapoursynth"))
# v4.7.1: the git-built VapourSynth stack installs its plugins into
# the python site-packages tree (module + libs + plugins/ are one
# self-contained unit). Probe those dirs too.
try:
search_dirs.append(Path(site.getusersitepackages()) / "vapoursynth" / "plugins")
except (AttributeError, OSError):
pass
try:
for d in site.getsitepackages():
search_dirs.append(Path(d) / "vapoursynth" / "plugins")
except (AttributeError, OSError):
pass
found: set[str] = set()
for d in search_dirs:

237
opentranscode/gpu_profiles.py Normal file
View File

@ -0,0 +1,237 @@
"""GPU capability profiles (v4.8.0) — combined card generations.
Cards within the same hardware-encoder generation are functionally
identical for transcoding, so the dropdown lists CAPABILITY CLASSES,
not individual SKUs: one Pascal entry covers the GTX 10-series, Tesla
P40/P4/P100 and mobile chips; one Turing entry covers RTX 20-series,
GTX 16-series, the Tesla T4 and the crypto-era CMP 30/40/50HX cards.
Oddballs are included with their real capabilities:
- CMP 90HX is GA102-based (Ampere NVENC), but CMP 170HX is GA100-based
and has NO NVENC at all (like A100/V100/H100 compute boards).
- Intel Arc (QSV) and AMD RDNA (VAAPI) cover the rest of the trending
list; RDNA 3 added AV1 encode, RDNA 1/2 and GCN can only encode
H.264/HEVC.
Pure data + pure functions: no I/O, safe to import anywhere.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
# ──────────────────────────────────────────────
# GPU PROFILES
# ──────────────────────────────────────────────
@dataclass(frozen=True)
class GpuProfile:
key: str # stable id (CLI/UI)
label: str # dropdown entry
vendor: str # nvidia | amd | intel | none
api: str # nvenc | vaapi | qsv | none
# codec family → ffmpeg encoder name
encoders: dict[str, str]
# case-insensitive substrings matched against the detected GPU name
# (nvidia-smi / lspci) for auto-detection. First match wins; lists
# are ordered most-specific first.
match: tuple[str, ...] = ()
notes: str = ""
# extra args that must come BEFORE -i (hardware device init)
hw_device_args: tuple[str, ...] = ()
# filter-chain fragment required before the encoder (vaapi hwupload)
filter_tail: tuple[str, ...] = ()
# NVENC encoders by generation class. Quality control: -rc vbr -cq N
# (-b:v 0). 8-bit yuv420p everywhere — Pascal 10-bit HEVC runs at half
# speed and the archival targets here are 8-bit sources.
_NV = {"h264": "h264_nvenc", "hevc": "hevc_nvenc"}
GPU_PROFILES: list[GpuProfile] = [
GpuProfile(
key="nv-kepler-maxwell",
label="NVIDIA Kepler / Maxwell 1.0 (GTX 600/700/800M) — H.264 only",
vendor="nvidia", api="nvenc",
encoders={"h264": "h264_nvenc"},
match=("GTX 6", "GTX 7", "GT 7", "GTX 8", "GT 8", "840M", "860M", "750"),
notes="First NVENC generations: H.264 only, no HEVC.",
),
GpuProfile(
key="nv-pascal",
label="NVIDIA Pascal (GTX 10-series, TITAN Xp, Tesla P40/P4/P100) — H.264 + HEVC 8/10-bit",
vendor="nvidia", api="nvenc",
encoders=dict(_NV),
match=("GTX 10", "1070", "1080", "1060", "1050", "TITAN Xp",
"Tesla P40", "Tesla P4", "P100", "Quadro P"),
notes="Pascal NVENC: HEVC Main/Main10. 10-bit runs at ~half speed.",
),
GpuProfile(
key="nv-turing",
label="NVIDIA Turing (RTX 20-series, GTX 16-series, Tesla T4, CMP 30/40/50HX) — H.264 + HEVC + B-frames",
vendor="nvidia", api="nvenc",
encoders=dict(_NV),
match=("RTX 20", "GTX 16", "2060", "2070", "2080", "1660", "1650",
"Tesla T4", "CMP 30", "CMP 40", "CMP 50"),
notes="Turing NVENC: first gen with HEVC B-frames; big quality jump.",
),
GpuProfile(
key="nv-compute",
label="NVIDIA data-center compute (V100/A100/H100, CMP 170HX) — no NVENC (CPU path)",
vendor="nvidia", api="none",
encoders={},
match=("V100", "A100", "H100", "B200", "GB200", "CMP 170"),
notes="Compute boards ship without NVENC silicon. CMP 170HX is "
"GA100-based — the fastest mining card that cannot hardware-encode.",
),
GpuProfile(
key="nv-ampere",
label="NVIDIA Ampere (RTX 30-series, A10/A40/A2, CMP 90HX) — H.264 + HEVC (no AV1 encode)",
vendor="nvidia", api="nvenc",
encoders=dict(_NV),
match=("RTX 30", "3090", "3080", "3070", "3060", "3050",
"A10", "A40", "CMP 90"),
notes="Ampere added AV1 DECODE but not encode — AV1 stays on CPU.",
),
GpuProfile(
key="nv-ada",
label="NVIDIA Ada / Blackwell (RTX 40/50-series, L4/L40) — H.264 + HEVC + AV1 10-bit",
vendor="nvidia", api="nvenc",
encoders={"h264": "h264_nvenc", "hevc": "hevc_nvenc", "av1": "av1_nvenc"},
match=("RTX 40", "RTX 50", "4090", "4080", "4070", "4060",
"5090", "5080", "5070", "5060", "L4", "L40"),
notes="Ada introduced AV1 NVENC; Blackwell doubles AV1 throughput.",
),
GpuProfile(
key="intel-arc",
label="Intel Arc (Alchemist A-series, Battlemage B-series) — QSV: H.264 + HEVC + AV1",
vendor="intel", api="qsv",
encoders={"h264": "h264_qsv", "hevc": "hevc_qsv", "av1": "av1_qsv"},
match=("Arc A", "Arc B", "A380", "A750", "A770", "B570", "B580"),
notes="Arc media engines encode AV1 8/10-bit — best value encode card.",
),
GpuProfile(
key="intel-xe",
label="Intel Iris / UHD integrated (Gen9–Xe) — QSV: H.264 + HEVC",
vendor="intel", api="qsv",
encoders={"h264": "h264_qsv", "hevc": "hevc_qsv"},
match=("Iris", "UHD", "HD Graphics"),
notes="Integrated media engines; HEVC 8/10-bit, no AV1 encode.",
),
GpuProfile(
key="amd-rdna3",
label="AMD RDNA 3 (RX 7000-series) — VAAPI: H.264 + HEVC + AV1",
vendor="amd", api="vaapi",
encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi", "av1": "av1_vaapi"},
match=("RX 7", "7900", "7800", "7700", "7600"),
notes="RDNA 3 VCN: first AMD generation with AV1 encode.",
),
GpuProfile(
key="amd-rdna12",
label="AMD RDNA 1/2 (RX 5000/6000-series) — VAAPI: H.264 + HEVC (AV1 decode only)",
vendor="amd", api="vaapi",
encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"},
match=("RX 5", "RX 6", "5700", "5600", "6800", "6700", "6600", "6500"),
notes="RDNA 2 has AV1 decode only — AV1 encode stays on CPU.",
),
GpuProfile(
key="amd-gcn",
label="AMD GCN 4/5 / Vega (RX 400/500, Vega 56/64) — VAAPI: H.264 + HEVC",
vendor="amd", api="vaapi",
encoders={"h264": "h264_vaapi", "hevc": "hevc_vaapi"},
match=("RX 4", "RX 5", "Vega", "580", "570", "480", "470", "64", "56"),
notes="The classic crypto-era mining cards (Polaris/Vega).",
),
GpuProfile(
key="cpu",
label="None (CPU-only encode)",
vendor="none", api="none",
encoders={},
),
]
_GPU_PROFILES_BY_KEY: dict[str, GpuProfile] = {p.key: p for p in GPU_PROFILES}
def gpu_profile_by_key(key: str | None) -> GpuProfile | None:
if not key:
return None
return _GPU_PROFILES_BY_KEY.get(key)
def match_gpu_profile(gpu_name: str) -> GpuProfile | None:
"""Best-effort auto-detection from a GPU name string (nvidia-smi or
lspci output). Case-insensitive; first matching profile wins (the
match lists are ordered most-specific first, and the compute boards
are matched before the consumer generations they share names with —
e.g. 'CMP 170HX' must not hit the Ampere 'A10' style entries)."""
if not gpu_name:
return None
name = gpu_name.lower()
for profile in GPU_PROFILES:
for frag in profile.match:
if frag.lower() in name:
return profile
return None
# ──────────────────────────────────────────────
# FFMPEG ARG HELPERS (per hardware API)
# ──────────────────────────────────────────────
def encoder_for_family(profile: GpuProfile | None, family: str) -> str | None:
"""Hardware encoder name for a codec family on this profile, or None."""
if not profile or profile.api == "none":
return None
return profile.encoders.get(family)
def resolve_vaapi_device() -> str:
"""First render node, or the classic fallback path. (Best-effort I/O —
callers that need purity pass the result into encoder_pre_args.)"""
import glob
nodes = sorted(glob.glob("/dev/dri/renderD*"))
return nodes[0] if nodes else "/dev/dri/renderD128"
def encoder_pre_args(profile: GpuProfile, vaapi_device: str | None = None) -> list[str]:
"""Args that must precede -i (hardware device initialisation)."""
if profile.api == "vaapi":
dev = vaapi_device or resolve_vaapi_device()
return ["-vaapi_device", dev]
if profile.api == "qsv":
return ["-init_hw_device", "qsv=hw"]
return []
def encoder_filter_chain(profile: GpuProfile) -> list[str]:
"""Filter args that upload software frames to the hardware surface
format (VAAPI encoders only accept hw frames; -vaapi_device makes
its device the default for hwupload)."""
if profile.api == "vaapi":
return ["-vf", "format=nv12,hwupload"]
return []
def encoder_quality_args(api: str, encoder: str, crf: int, preset: int) -> list[str]:
"""Constant-quality args for a hardware encoder. NVENC maps the CPU
preset tiers to p-presets; QSV uses very_fast/medium; VAAPI uses CQP
rate mode which has no preset knob."""
if api == "nvenc":
if preset <= 6:
p = "p7"
elif preset <= 8:
p = "p5"
else:
p = "p4"
return ["-preset", p, "-tune", "hq", "-rc", "vbr",
"-cq", str(crf), "-b:v", "0", "-pix_fmt", "yuv420p",
"-g", "240"]
if api == "qsv":
p = "veryslow" if preset <= 6 else ("medium" if preset <= 8 else "very_fast")
return ["-preset", p, "-global_quality", str(crf),
"-pix_fmt", "yuv420p", "-g", "240"]
if api == "vaapi":
return ["-rc_mode", "CQP", "-qp", str(crf), "-g", "240"]
return []

115
opentranscode/hybrid_scheduler.py Normal file
View File

@ -0,0 +1,115 @@
"""Hybrid GPU+CPU batch scheduler (v4.7.0).
Splits a transcode queue between two concurrent lanes so the NVENC
engine and the CPU encoders work at the same time instead of leaving
28 Xeon threads idle while the GPU encode runs:
- GPU lane: files encoded via the functional NVENC encoder
(hevc_nvenc / av1_nvenc) through the single-pass ffmpeg path.
- CPU lane: the remaining files through the family's software encoder
(libx265 / libsvtav1 / libvpx) or av1an chunk-parallel if the user
opted in — so "all three" (NVENC + software + chunk-parallel) run
side by side when av1an is enabled.
Pure planning logic — no Qt, no I/O beyond the caller-provided file
sizes. The UI turns a HybridPlan into two EncoderWorker instances.
Scope note: lanes split FILES, never one file across encoders. Splitting
a single file between hevc_nvenc and libx265 chunks would produce
visibly inconsistent quality between scenes, and av1an cannot drive
NVENC at all (it spawns encoder CLI binaries only).
"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
# NVENC HEVC on a Pascal card runs several times faster than x265
# "faster" on 28 Xeon threads. Used only for load ESTIMATION (which lane
# gets the next file) — a wrong guess just skews the split slightly.
GPU_SPEED_RATIO_DEFAULT = 8
# Threads held back from the CPU lane so the GPU lane's decode / scale /
# mux processes stay responsive. The NVENC encode itself runs on the GPU
# silicon; the CPU side of a nvenc job is light.
HYBRID_CPU_RESERVE_THREADS = 2
@dataclass
class HybridPlan:
"""Result of planning a hybrid (GPU + CPU) queue split."""
gpu_files: list[Path] = field(default_factory=list)
cpu_files: list[Path] = field(default_factory=list)
gpu_encoder: str = "" # e.g. "hevc_nvenc"
cpu_budget_threads: int = 1 # CPU lane thread budget (logical - reserve)
gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT
@property
def total_files(self) -> int:
return len(self.gpu_files) + len(self.cpu_files)
def plan_hybrid(
files: list[Path],
gpu_encoder: str | None,
gpu_functional: bool,
logical_threads: int,
sizes: dict[Path, int] | None = None,
gpu_speed_ratio: int = GPU_SPEED_RATIO_DEFAULT,
cpu_reserve: int = HYBRID_CPU_RESERVE_THREADS,
) -> HybridPlan | None:
"""Split *files* between the GPU and CPU lanes, or return None when a
hybrid split cannot apply.
Returns None when:
- the codec family has no GPU encoder, or the live GPU probe failed
(caller should fall back to a plain CPU queue), or
- *files* is empty.
Assignment is LPT (longest-processing-time first): files are sorted
by size descending and each goes to the lane with the lower
estimated load, where the GPU lane's per-file cost is size /
gpu_speed_ratio. Both lanes then finish at roughly the same time.
*sizes* maps files to byte sizes; missing entries fall back to the
mean of the known sizes (or 10 MB when nothing is known) so a single
unreadable file cannot skew the whole split.
"""
if not gpu_encoder or not gpu_functional or not files:
return None
sizes = sizes or {}
known = [s for s in sizes.values() if s]
avg = sum(known) // len(known) if known else 10_000_000
def size_of(f: Path) -> int:
return sizes.get(f) or avg
ratio = max(1, int(gpu_speed_ratio))
gpu_files: list[Path] = []
cpu_files: list[Path] = []
gpu_load = 0.0
cpu_load = 0.0
for f in sorted(files, key=size_of, reverse=True):
s = size_of(f)
gpu_est = gpu_load + s / ratio
cpu_est = cpu_load + s
# Tie goes to the GPU lane — it finishes the file sooner and the
# CPU lane keeps its current file longer.
if gpu_est <= cpu_est:
gpu_files.append(f)
gpu_load = gpu_est
else:
cpu_files.append(f)
cpu_load = cpu_est
return HybridPlan(
gpu_files=gpu_files,
cpu_files=cpu_files,
gpu_encoder=gpu_encoder,
cpu_budget_threads=max(1, int(logical_threads) - cpu_reserve),
gpu_speed_ratio=ratio,
)

343
opentranscode/source_builder.py
View File

@ -5,19 +5,137 @@ components from source. Installs to the user's home dir (no sudo for
the install step). v3-08 made this worker stop mutating
``os.environ`` directly — it carries its own ``_build_env`` snapshot.
Pure stdlib + PySide6 (no internal package dependencies).
v4.7.1: the rebuild is ALWAYS usable, even on a bare system. It
generates its own dependency tree: missing build tools and libraries
are installed via the distro package manager (arch/debian/redhat/suse)
before anything is compiled, and the VapourSynth build is followed by
a BestSource plugin build (submodules + vapoursynth dev headers from
the freshly installed VS) so av1an gets a fast, reliable chunk method
instead of the slow "select" fallback.
"""
import os
import re
import shutil
import signal
import site
import subprocess
import time
from dataclasses import dataclass, field
from pathlib import Path
from PySide6.QtCore import QThread, Signal
from .distro_probe import DistroProfile, detect_distro
from .gpu_profiles import gpu_profile_by_key
# ──────────────────────────────────────────────
# BUILD DEPENDENCY TREE (v4.7.1 — distro-aware)
# ──────────────────────────────────────────────
# Binaries the build needs. pkgconf/pkg-config and python/python3 are
# aliased — any one of each pair satisfies the check.
BUILD_TOOL_ALIASES: dict[str, tuple[str, ...]] = {
"git": ("git",),
"meson": ("meson",),
"ninja": ("ninja",),
"c++ compiler": ("g++", "c++", "clang++"),
"make": ("make",),
"pkg-config": ("pkg-config", "pkgconf"),
"python3": ("python3",),
"nasm": ("nasm",),
"cmake": ("cmake",),
}
# Packages providing the toolchain + the libraries the builds link
# against (zimg is VapourSynth's one hard library dependency; rust is
# only needed for the av1an build).
BUILD_DEPS_BY_FAMILY: dict[str, list[str]] = {
"arch": ["base-devel", "meson", "ninja", "cmake", "nasm", "git",
"python", "pkgconf", "zimg", "rust"],
"debian": ["build-essential", "meson", "ninja-build", "cmake", "nasm",
"git", "python3", "python3-dev", "pkg-config", "libzimg-dev",
"cargo", "rustc"],
"redhat": ["gcc", "gcc-c++", "make", "meson", "ninja-build", "cmake",
"nasm", "git", "python3", "python3-devel",
"pkgconf-pkg-config", "zimg-devel", "cargo", "rust"],
"suse": ["gcc", "gcc-c++", "make", "meson", "ninja", "cmake", "nasm",
"git", "python3", "python3-devel", "pkg-config",
"zimg-devel", "rust", "cargo"],
}
PKG_INSTALL_CMD: dict[str, list[str]] = {
"arch": ["pacman", "-S", "--needed", "--noconfirm"],
"debian": ["apt-get", "install", "-y"],
"redhat": ["dnf", "install", "-y"],
"suse": ["zypper", "--non-interactive", "install"],
}
MANUAL_DEP_NOTE = (
"No automatic package install for this distro family. Install a C++ "
"toolchain plus meson, ninja, cmake, nasm, git, python3, pkg-config, "
"zimg development headers{rust} manually, then press REBUILD again."
)
# v4.8.0: per-GPU-profile build/runtime packages, on top of the base
# toolchain. nvidia needs nv-codec-headers at ffmpeg build time (the
# distro ffmpeg already ships nvenc; a matched git build needs the
# headers); vaapi/qsv need the driver + dev stacks for their vendor.
GPU_BUILD_PACKAGES: dict[str, dict[str, list[str]]] = {
"nvenc": {
"arch": ["nv-codec-headers"],
"debian": [],
"redhat": [],
"suse": [],
},
"vaapi": {
"arch": ["libva", "libdrm", "mesa"],
"debian": ["libva-dev", "libdrm-dev", "mesa-va-drivers"],
"redhat": ["libva-devel", "libdrm-devel", "mesa-va-drivers"],
"suse": ["libva-devel", "libdrm-devel", "Mesa-libva"],
},
"qsv": {
"arch": ["libva", "intel-media-driver", "onevpl"],
"debian": ["libva-dev", "intel-media-va-driver-non-free", "libvpl-dev"],
"redhat": ["libva-devel", "intel-media-driver", "oneVPL-devel"],
"suse": ["libva-devel", "intel-media-driver", "oneVPL-devel"],
},
}
def gpu_dep_packages(api: str, distro_family: str) -> list[str]:
"""Extra packages for a GPU hardware API on this distro (empty when
the family has no packaged set — the log says so)."""
return list(GPU_BUILD_PACKAGES.get(api, {}).get(distro_family, []))
@dataclass
class DepPlan:
"""What the rebuild needs, and how to get it on this distro."""
packages: list[str] = field(default_factory=list)
install_cmd: list[str] | None = None
manual_note: str | None = None
def build_dep_plan(distro: DistroProfile, build_av1an: bool = True) -> DepPlan:
"""Pure: the package list + install command for this distro family.
Works even when the environment probe failed — it only needs the
distro family, which is detectable from /etc/os-release alone.
"""
packages = list(BUILD_DEPS_BY_FAMILY.get(distro.family, []))
if not build_av1an:
for rust_pkg in ("rust", "rustc", "cargo"):
if rust_pkg in packages:
packages.remove(rust_pkg)
install_cmd = PKG_INSTALL_CMD.get(distro.family)
manual_note = None
if not install_cmd or not packages:
manual_note = MANUAL_DEP_NOTE.format(
rust=" and Rust/cargo" if build_av1an else "")
return DepPlan(packages=packages, install_cmd=install_cmd,
manual_note=manual_note)
# ──────────────────────────────────────────────
# SOURCE BUILD WORKER — compile VS + av1an from git
# ──────────────────────────────────────────────
@ -37,11 +155,14 @@ class SourceBuildWorker(QThread):
build_done = Signal(bool, str) # (success, detail)
def __init__(self, build_vs: bool = True, build_av1an: bool = True,
build_ffmpeg_iamf: bool = False):
build_ffmpeg_iamf: bool = False, gpu_profile_key: str = ""):
super().__init__()
self.build_vs = build_vs
self.build_av1an = build_av1an
self.build_ffmpeg_iamf = build_ffmpeg_iamf
# v4.8.0: selected GPU capability profile — extends the dep tree
# with the vendor's build/runtime packages.
self.gpu_profile_key = gpu_profile_key
self._stop = False
# Private per-worker environment snapshot. Mutating os.environ is
# process-global and leaks across threads/subsequent subprocesses;
@ -98,30 +219,71 @@ class SourceBuildWorker(QThread):
def run(self):
try:
# ── Install build dependencies (may need one sudo prompt) ──
# ── v4.7.1: distro-aware dependency tree ──
# The rebuild must work on a bare system: detect missing
# tools/libraries and install them via the distro package
# manager (one privilege prompt via pkexec/sudo) BEFORE
# compiling anything.
self.log_msg.emit("")
self.log_msg.emit("=== Installing build dependencies ===")
all_deps = [
"meson", "ninja", "gcc", "pkg-config", "git",
"nasm", "yasm", "cmake", "python", "make",
]
need_rust = self.build_av1an and not shutil.which("cargo")
if need_rust:
all_deps.append("rust")
self.log_msg.emit("=== Generating dependency tree ===")
self._distro = detect_distro()
self.log_msg.emit(
f" Distro: {self._distro.name} (family={self._distro.family})"
)
plan = build_dep_plan(self._distro, build_av1an=self.build_av1an)
# v4.8.0: GPU-profile packages on top of the base toolchain.
gpu_profile = gpu_profile_by_key(self.gpu_profile_key)
if gpu_profile is not None and gpu_profile.api != "none":
gpu_pkgs = gpu_dep_packages(gpu_profile.api, self._distro.family)
if gpu_pkgs:
self.log_msg.emit(
f" GPU profile {gpu_profile.key} ({gpu_profile.api}): "
f"+{len(gpu_pkgs)} package(s)"
)
plan.packages.extend(p for p in gpu_pkgs
if p not in plan.packages)
missing = self._missing_build_tools()
zimg_ok = self._pkgconfig_exists("zimg")
if zimg_ok:
self.log_msg.emit(" OK: zimg (VapourSynth dependency)")
else:
missing.append("zimg (library, via pkg-config)")
if self.build_av1an and not shutil.which("cargo"):
missing.append("cargo (rust)")
# Only invoke sudo if at least one dep is missing
missing = [d for d in all_deps if not shutil.which(d)]
if missing:
self.log_msg.emit(f" Missing: {', '.join(missing)} — installing via pacman")
rc, _ = self._sudo_cmd(
["pacman", "-S", "--needed", "--noconfirm"] + all_deps,
timeout=300, label="pacman build-deps",
)
if rc != 0:
self.log_msg.emit(" (some deps may already be installed — continuing)")
self.log_msg.emit(f" Missing: {', '.join(missing)}")
if plan.install_cmd:
self.log_msg.emit(
f" Installing {len(plan.packages)} package(s) via "
f"{plan.install_cmd[0]} (privilege prompt possible)..."
)
rc, _ = self._sudo_cmd(
plan.install_cmd + plan.packages,
timeout=900, label=f"{plan.install_cmd[0]} build-deps",
)
if rc != 0:
self.log_msg.emit(
" (install reported an error — continuing; "
"some packages may already be present)"
)
else:
self.log_msg.emit(f" {plan.manual_note}")
else:
self.log_msg.emit(" All build dependencies already installed.")
# Re-verify the critical tools after install.
still_missing = self._missing_build_tools()
if still_missing:
self.log_msg.emit(
f" FATAL: still missing after install: {', '.join(still_missing)}. "
f"Install them manually and press REBUILD again."
)
self.build_done.emit(False, f"missing build tools: {still_missing}")
return
# Ensure cargo is in PATH after potential install.
# NOTE: /root/.cargo/bin was dropped (OTC-015/v3-08) — root's
# cargo dir is not readable by a non-root user. ~/.cargo/bin
@ -142,6 +304,9 @@ class SourceBuildWorker(QThread):
# ── Build & install VapourSynth to ~/.local (NO sudo needed) ──
if self.build_vs:
self._build_vapoursynth()
# v4.7.1: BestSource right after VS, compiled against the
# fresh VS headers — gives av1an a fast chunk method.
self._build_bestsource()
# ── Build av1an to ~/.cargo/bin (NO sudo needed) ──
if self.build_av1an:
@ -152,12 +317,25 @@ class SourceBuildWorker(QThread):
self._build_libiamf()
self._build_ffmpeg_with_iamf()
# ── Ensure LD_LIBRARY_PATH includes local VS libs ──
# ── Ensure the runtime env can find the fresh VS stack ──
# The git VapourSynth installs self-contained into the user
# site-packages (module + libs + plugins). av1an dlopens
# libvapoursynth-script from there, so both LD_LIBRARY_PATH
# and PYTHONPATH must include it.
local_lib = str(Path.home() / ".local" / "lib")
existing_ld = self._build_env.get("LD_LIBRARY_PATH", "")
if local_lib not in existing_ld:
self._extend_env("LD_LIBRARY_PATH", local_lib, prepend=True)
self.log_msg.emit(f" Set LD_LIBRARY_PATH to include {local_lib}")
user_site = self._vs_user_site()
if user_site and (user_site / "vapoursynth" / "libvsscript.so").exists():
vs_dir = str(user_site / "vapoursynth")
if vs_dir not in self._build_env.get("LD_LIBRARY_PATH", ""):
self._extend_env("LD_LIBRARY_PATH", vs_dir, prepend=True)
if str(user_site) not in self._build_env.get("PYTHONPATH", ""):
self._extend_env("PYTHONPATH", str(user_site), prepend=True)
self.log_msg.emit(
f" Runtime env: LD_LIBRARY_PATH/PYTHONPATH include {vs_dir}"
)
self.log_msg.emit("")
self.log_msg.emit("=== Source build complete ===")
@ -174,6 +352,127 @@ class SourceBuildWorker(QThread):
self.log_msg.emit(f"BUILD FAILED: {e}")
self.build_done.emit(False, str(e))
def _missing_build_tools(self) -> list[str]:
"""Binaries from BUILD_TOOL_ALIASES that are not on PATH."""
missing = []
for label, candidates in BUILD_TOOL_ALIASES.items():
if not any(shutil.which(c) for c in candidates):
missing.append(label)
return missing
def _pkgconfig_exists(self, name: str) -> bool:
rc, _ = self._run_cmd(
["pkg-config", "--exists", name],
timeout=10, label=f"pkg-config {name}",
)
return rc == 0
def _vs_user_site(self) -> Path | None:
"""The user site-packages dir of the system python3 — where the
VapourSynth git install places its self-contained stack (module,
libs, headers, plugins/)."""
rc, out = self._run_cmd(
["python3", "-m", "site", "--user-site"],
timeout=15, label="python3 -m site --user-site",
)
if rc == 0 and out.strip():
return Path(out.strip().splitlines()[-1])
return None
def _build_bestsource(self):
"""Clone and build the BestSource VapourSynth plugin from git.
BestSource gives av1an a fast, frame-accurate chunk source — the
difference between 'select' (quadratic decoding, minutes per
file) and normal chunk-parallel speed. Compiled against the
vapoursynth headers of the JUST-INSTALLED git VS (via
PYTHONPATH/PKG_CONFIG_PATH), so the plugin ABI always matches
the VS that av1an will load. Requires the repo's libp2p
submodule (initialized here).
"""
self.log_msg.emit("")
self.log_msg.emit("=== Building BestSource plugin from git ===")
self.log_msg.emit(" Source: https://github.com/vapoursynth/bestsource")
build_dir = Path("/tmp/bestsource-git-build")
if build_dir.exists():
shutil.rmtree(build_dir, ignore_errors=True)
rc, out = self._run_cmd(
["git", "clone", "--depth", "1",
"https://github.com/vapoursynth/bestsource.git",
str(build_dir)],
timeout=120, label="git clone bestsource",
)
if rc != 0:
raise Exception(f"git clone bestsource failed: {out[-300:]}")
# libp2p is a required submodule (R9+ builds source from it).
rc, out = self._run_cmd(
["git", "submodule", "update", "--init", "--depth", "1"],
cwd=str(build_dir), timeout=120, label="git submodule update",
)
if rc != 0:
raise Exception(f"bestsource submodule init failed: {out[-300:]}")
# Point meson/pkg-config at the freshly built VS stack.
user_site = self._vs_user_site()
if user_site and (user_site / "vapoursynth").is_dir():
self._extend_env("PYTHONPATH", str(user_site), prepend=True)
self._extend_env("PKG_CONFIG_PATH",
str(user_site / "vapoursynth" / "pkgconfig"),
prepend=True)
self._extend_env("LD_LIBRARY_PATH",
str(user_site / "vapoursynth"), prepend=True)
else:
self.log_msg.emit(
" NOTE: git VapourSynth install not found in user "
"site-packages — building against system vapoursynth."
)
self.log_msg.emit(" Configuring with meson (--prefix=~/.local)...")
rc, out = self._run_cmd(
["meson", "setup", "build",
f"--prefix={Path.home() / '.local'}", "--libdir=lib"],
cwd=str(build_dir), timeout=180, label="meson setup bestsource",
)
if rc != 0:
raise Exception(f"bestsource meson setup failed: {out[-500:]}")
self.log_msg.emit(" Compiling BestSource (a minute or two)...")
rc, out = self._run_cmd(
["ninja", "-C", "build", "-j", str(max(1, os.cpu_count() or 2))],
cwd=str(build_dir), timeout=600, label="ninja bestsource",
)
if rc != 0:
raise Exception(f"bestsource build failed: {out[-500:]}")
rc, out = self._run_cmd(
["ninja", "-C", "build", "install"],
cwd=str(build_dir), timeout=120, label="ninja install bestsource",
)
if rc != 0:
raise Exception(f"bestsource install failed: {out[-500:]}")
plugin = None
if user_site:
candidate = user_site / "vapoursynth" / "plugins" / "libbestsource.so"
if candidate.exists():
plugin = candidate
if plugin:
self.log_msg.emit(f" BestSource plugin installed: {plugin}")
self.log_msg.emit(
" av1an will now auto-select the fast 'bestsource' chunk "
"method (restart the app so the probe sees it)."
)
else:
self.log_msg.emit(
" WARNING: libbestsource.so not found at the expected "
"user-site path — check the meson install log above."
)
shutil.rmtree(build_dir, ignore_errors=True)
def _build_vapoursynth(self):
"""Clone, build, and install VapourSynth to ~/.local/ (no sudo needed)."""
self.log_msg.emit("")

10
opentranscode/temp_manager.py
View File

@ -80,16 +80,22 @@ def _mkdir_private(path: Path) -> bool:
return False
def _worker_temp_dir(worker_pid: int) -> Path:
def _worker_temp_dir(worker_pid: int, lane: str = "") -> Path:
"""Return a per-worker temp subdir named by PID.
v3: each EncoderWorker gets its own subdir under the shared app temp
dir, so the final cleanup sweep can safely nuke only this worker's
intermediates without affecting a concurrent worker. The subdir is
also created with mode=0o700 (FIO09-C).
v4.7.0: *lane* suffixes the dir ("gpu"/"cpu") for the hybrid
scheduler's concurrent lanes — both run in the SAME process, so the
PID alone no longer separates them, and a lane finishing early must
not sweep the other lane's intermediates out from under it.
"""
base = _get_app_temp_dir()
sub = base / f"worker-{worker_pid}"
name = f"worker-{worker_pid}" + (f"-{lane}" if lane else "")
sub = base / name
_mkdir_private(sub)
return sub

333
opentranscode/ui_window.py
View File

@ -36,7 +36,8 @@ from .codec_profiles import (
ResolutionProfile,
ffmpeg_lib_key_for,
)
from .encoder_worker import EncoderWorker
from .encoder_worker import EncoderWorker, resolve_gpu_encoder
from .gpu_profiles import GPU_PROFILES
from .env_probe import (
EnvProbe,
_av1an_vsscript_smoke_test,
@ -59,6 +60,12 @@ class OpenCodecMaster(QMainWindow):
self.setWindowTitle("OpenTranscode — dcos.net")
self.resize(1100, 920)
self.worker: EncoderWorker | None = None
# v4.7.0: hybrid (GPU + CPU lanes) bookkeeping. *workers* holds
# every active lane; _hybrid_pending/_hybrid_totals aggregate the
# per-lane finished_queue signals into one summary.
self.workers: list[EncoderWorker] = []
self._hybrid_pending = 0
self._hybrid_totals = [0, 0]
self.env: EnvProbe | None = None
self._pending_deletes: list[Path] = []
@ -300,6 +307,90 @@ class OpenCodecMaster(QMainWindow):
"Default OFF for compatibility."
)
opt_row.addWidget(self.inline_scale_check)
# v4.6.0: ENGINE selector — Auto (GPU if available) / GPU / CPU.
# Auto uses the NVENC hardware encoder for the selected codec
# family when the environment probe's live encode test proved it
# works (av1 → av1_nvenc on RTX 40+, hevc → hevc_nvenc on every
# NVENC generation); otherwise it stays on the CPU encoders. GPU
# mode runs through the single-pass ffmpeg path — av1an cannot
# drive NVENC, and one NVENC process outruns chunk-parallel CPU
# workers anyway.
self.engine_combo = QComboBox()
self.engine_combo.addItems([
"Engine: Auto (GPU if available)",
"Engine: GPU (NVENC)",
"Engine: CPU",
"Engine: Hybrid (GPU + CPU)",
])
self.engine_combo.setToolTip(
"Video encode engine.\n\n"
"Auto — use the NVENC GPU encoder when the selected codec\n"
" family has one AND a live encode test proved it works on\n"
" this system; fall back to CPU otherwise (default).\n"
"GPU — force NVENC (hevc_nvenc / av1_nvenc); falls back to\n"
" CPU with a log message when unavailable. GPU encodes run\n"
" via single-pass ffmpeg (av1an chunking is not used).\n"
"CPU — force the software encoders (SVT-AV1 / VP9 / x265).\n"
"Hybrid — run BOTH at once: the queue is split between a\n"
" GPU lane and a CPU lane (balanced by file size), so the\n"
" CPU cores are not idle while NVENC encodes. With av1an\n"
" enabled the CPU lane uses chunk-parallel too. GPU-lane\n"
" files get NVENC quality; CPU-lane files get software-\n"
" encoder quality. Needs 2+ encodable files.\n"
)
self.engine_combo.setFixedHeight(24)
# v4.7.1: self.env is None (or absent) until _probe_and_init
# runs AFTER _build_ui — the pre-select must not touch it here.
_engine_flags = getattr(getattr(self, "env", None), "av1an_flags", None) or {}
_engine_pref = _engine_flags.get("engine")
_engine_index = {"auto": 0, "gpu": 1, "cpu": 2, "hybrid": 3}
self.engine_combo.setCurrentIndex(_engine_index.get(_engine_pref, 0))
# v4.8.0: GPU capability-profile dropdown. Entries combine whole
# card generations (same silicon = same encoding behaviour) and
# include data-center + crypto-era oddballs. "Auto-detect" uses
# the probe's name match; a specific profile forces the API even
# when auto-match fails. The rebuild-from-git dep tree extends
# with the selected profile's packages.
from .gpu_profiles import GPU_PROFILES as _GPU_PROFILES
self.gpu_combo = QComboBox()
self.gpu_combo.addItem("GPU: Auto-detect")
for _gp in _GPU_PROFILES:
self.gpu_combo.addItem(f"GPU: {_gp.label}")
self.gpu_combo.setToolTip(
"Hardware encoder capability class.\n"
"Auto-detect matches your card via nvidia-smi/lspci and the\n"
"live encode probe decides what actually works. Forcing a\n"
"profile also extends the REBUILD FROM GIT dependency tree\n"
"with that GPU's packages (nv-codec-headers / VAAPI / QSV)."
)
self.gpu_combo.setFixedHeight(24)
_gpu_pref = _engine_flags.get("gpu_profile")
_gpu_keys = ["auto"] + [_gp.key for _gp in _GPU_PROFILES]
self.gpu_combo.setCurrentIndex(
_gpu_keys.index(_gpu_pref) if _gpu_pref in _gpu_keys else 0
)
self.gpu_combo.currentIndexChanged.connect(self._on_gpu_profile_changed)
# v4.8.1: ENGINE = CPU makes the GPU choice inert — grey it out.
# The "None (CPU-only encode)" entry stays available for users
# who have a GPU in the box but don't want encoding on it.
self.engine_combo.currentIndexChanged.connect(self._on_engine_changed)
self._update_gpu_combo_state()
_gpu_lay = QVBoxLayout()
_gpu_lay.setSpacing(1)
_gpu_lbl = QLabel("GPU")
_gpu_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;")
_gpu_lay.addWidget(_gpu_lbl)
_gpu_lay.addWidget(self.gpu_combo)
opt_row.addLayout(_gpu_lay)
_engine_lay = QVBoxLayout()
_engine_lay.setSpacing(1)
_engine_lbl = QLabel("ENGINE")
_engine_lbl.setStyleSheet("color: #666; font-size: 7px; letter-spacing: 1px;")
_engine_lay.addWidget(_engine_lbl)
_engine_lay.addWidget(self.engine_combo)
opt_row.addLayout(_engine_lay)
root.addLayout(opt_row)
# ── Log + Knobs: horizontal split ──
@ -348,12 +439,18 @@ class OpenCodecMaster(QMainWindow):
self.btn_rebuild.setObjectName("btnRebuild")
self.btn_rebuild.setFixedHeight(40)
self.btn_rebuild.setToolTip(
"Compile VapourSynth + av1an from git source.\n"
"Compile VapourSynth + av1an + BestSource from git source.\n"
"Resolves ABI/version mismatch when package managers\n"
"install incompatible versions."
"install incompatible versions. ALWAYS available — even on a\n"
"bare system: it first generates its own dependency tree\n"
"(installs missing build tools + zimg via the distro package\n"
"manager, one privilege prompt) and then builds everything\n"
"into ~/.local / ~/.cargo (no system changes)."
)
self.btn_rebuild.clicked.connect(self._manual_rebuild)
self.btn_rebuild.setEnabled(False)
# v4.7.1: ALWAYS usable — the build generates its own dependency
# tree, so it must not depend on a successful probe (a failed
# probe is exactly when you need it).
btn_lay.addWidget(self.btn_rebuild)
self.btn_about = QPushButton(" ? ABOUT / LICENSES")
@ -997,6 +1094,26 @@ class OpenCodecMaster(QMainWindow):
selected_codec = VIDEO_CODECS[codec_idx]
self._log(f"Codec: {selected_codec.label} (av1an encoder: {selected_codec.av1an_encoder})")
# v4.6.0: read the ENGINE selector and surface the resolved engine
# before the queue starts. The worker re-resolves against the same
# probe data; this line tells the user what is about to happen.
engine_sel = ("auto", "gpu", "cpu", "hybrid")[self.engine_combo.currentIndex()] \
if hasattr(self, "engine_combo") else "auto"
if hasattr(self, "engine_combo"):
# Always write UI state (CLI --engine pre-selects the combo;
# unchecking round-trips the same way as the other toggles).
self.env.av1an_flags["engine"] = engine_sel
gpu_preview, gpu_api = resolve_gpu_encoder(engine_sel, selected_codec, self.env)
if gpu_preview:
self._log(f"Engine: GPU — {gpu_preview} ({gpu_api} hardware encode)")
elif engine_sel == "gpu":
gpu_enc = selected_codec.gpu_encoder or "(none for this codec)"
self._log(f"Engine: GPU requested but {gpu_enc} unavailable — will use CPU.")
elif engine_sel == "hybrid":
self._log("Engine: hybrid — GPU lane + CPU lane concurrently (splits the queue by size).")
else:
self._log("Engine: CPU (software encoders)")
# v4.4.4: read the inline-scale checkbox into env.av1an_flags so
# EncoderWorker.__init__'s fallback path picks it up (matches the
# pattern used by --use-av1an, --verbose, --skip-existing, etc.).
@ -1009,7 +1126,9 @@ class OpenCodecMaster(QMainWindow):
if self.inline_scale_check.isChecked():
self._log("Inline scale: enabled (no intermediate file for scaling).")
self.worker = EncoderWorker(
# v4.7.0: kwargs shared by every lane of the queue (single
# worker, or the GPU + CPU lanes in hybrid mode).
common = dict(
in_dir=in_dir,
out_dir=out_dir,
video_codec=selected_codec,
@ -1018,17 +1137,33 @@ class OpenCodecMaster(QMainWindow):
crf=self.crf_knob.intValue(),
preset_label=self.preset_combo.currentText(),
delete_source=self.del_check.isChecked(),
env=self.env,
extensions=self._parse_extensions(),
resolution=self._get_current_resolution(),
audio_level_db=self.vol_knob.value(),
use_ffmpeg_fallback=use_ffmpeg_fallback,
subtitle_lang=SUBTITLE_OPTIONS[self.subs_combo.currentIndex()][1],
force=self.force_check.isChecked(), # v5-01
)
self.worker.log_msg.connect(self._log)
self.worker.progress_msg.connect(self._on_progress)
self.worker.finished_queue.connect(self._on_finished)
# ── v4.7.0: hybrid (GPU + CPU lanes run concurrently) ──
if engine_sel == "hybrid":
if self._start_hybrid(common, use_ffmpeg_fallback, use_av1an):
self.btn_run.setEnabled(False)
self.btn_run.setText("RUNNING...")
self.btn_stop.setEnabled(True)
self.btn_rebuild.setEnabled(False)
for w in self.workers:
w.start()
return
self._log("Hybrid unavailable — falling back to a single CPU queue.")
engine_sel = "cpu"
self.worker = EncoderWorker(
**common,
env=self.env,
use_ffmpeg_fallback=use_ffmpeg_fallback,
engine=engine_sel, # v4.6.0: Auto/GPU/CPU engine selector
)
self._set_workers([self.worker])
self.btn_run.setEnabled(False)
self.btn_run.setText("RUNNING...")
@ -1036,6 +1171,130 @@ class OpenCodecMaster(QMainWindow):
self.btn_rebuild.setEnabled(False)
self.worker.start()
@Slot()
def _on_engine_changed(self, index: int):
"""v4.8.1: ENGINE = CPU disables the GPU dropdown (the choice
would have no effect); every other engine keeps it live."""
self._update_gpu_combo_state()
def _update_gpu_combo_state(self):
cpu_only = self.engine_combo.currentIndex() == 2 # "Engine: CPU"
self.gpu_combo.setEnabled(not cpu_only)
self.gpu_combo.setToolTip(
"ENGINE is CPU — the GPU choice has no effect."
if cpu_only else
"Hardware encoder capability class.\n"
"Auto-detect matches your card via nvidia-smi/lspci and the\n"
"live encode probe decides what actually works. Forcing a\n"
"profile also extends the REBUILD FROM GIT dependency tree\n"
"with that GPU's packages (nv-codec-headers / VAAPI / QSV).\n"
"'None (CPU-only encode)' opts out even when a GPU exists."
)
@Slot()
def _on_gpu_profile_changed(self, index: int):
"""v4.8.0: persist the GPU capability-profile selection so both
the engine resolution and the rebuild dep tree pick it up."""
if not getattr(self, "env", None):
return # UI build phase — env probe hasn't run yet
if index <= 0:
self.env.av1an_flags["gpu_profile"] = "auto"
return
from .gpu_profiles import GPU_PROFILES
self.env.av1an_flags["gpu_profile"] = GPU_PROFILES[index - 1].key
self._log(f"GPU profile: {GPU_PROFILES[index - 1].key}")
def _set_workers(self, workers: list) -> None:
"""v4.7.0: register the active lane worker(s) and wire their
signals. STOP iterates every lane; _on_finished aggregates the
per-lane summaries into one."""
self.workers = list(workers)
self._hybrid_pending = len(self.workers)
self._hybrid_totals = [0, 0]
for w in self.workers:
w.log_msg.connect(self._log)
w.progress_msg.connect(self._on_progress)
w.finished_queue.connect(self._on_finished)
def _start_hybrid(self, common: dict, use_ffmpeg_fallback: bool,
use_av1an: bool) -> bool:
"""v4.7.0: split the queue between a GPU lane (NVENC) and a CPU
lane (software encoders, or av1an chunk-parallel when opted in —
so NVENC + chunk workers + software can all run at once).
Constructs both workers on success and returns True. Returns
False (with a logged reason) when hybrid cannot apply; the caller
falls back to a single-lane queue.
"""
import copy as _copy
import dataclasses as _dc
from .encoder_worker import scan_input_files
from .hybrid_scheduler import plan_hybrid
gpu_enc, _gpu_api = resolve_gpu_encoder("gpu", common["video_codec"], self.env)
if not gpu_enc:
self._log("Hybrid: no functional NVENC encoder for this codec family.")
return False
files = scan_input_files(common["in_dir"], common["extensions"])
if len(files) < 2:
self._log("Hybrid: fewer than 2 encodable files — one lane is faster than scheduling.")
return False
sizes: dict = {}
for f in files:
try:
sizes[f] = f.stat().st_size
except OSError:
pass
plan = plan_hybrid(
files, gpu_enc, True, self.env.cpu.logical_threads, sizes=sizes,
)
if plan is None or not plan.gpu_files or not plan.cpu_files:
self._log("Hybrid: size split degenerated to one lane — single lane is faster.")
return False
self._log(
f"HYBRID: GPU lane = {len(plan.gpu_files)} file(s) via {gpu_enc} | "
f"CPU lane = {len(plan.cpu_files)} file(s) "
f"(budget {plan.cpu_budget_threads} threads"
f"{', av1an chunk-parallel' if use_av1an else ''})"
)
self._log(
" Note: lanes use different encoders — GPU-lane files get NVENC "
"quality, CPU-lane files get software-encoder quality."
)
# CPU lane budget: hold back threads for the GPU lane's
# decode/scale/mux processes, then let the lane self-budget its
# av1an workers / ffmpeg threads from the reduced topology.
cpu_env = _copy.copy(self.env)
cpu_env.cpu = _dc.replace(
self.env.cpu, logical_threads=plan.cpu_budget_threads
)
gpu_worker = EncoderWorker(
**common,
env=self.env,
use_ffmpeg_fallback=True, # GPU runs via single-pass ffmpeg
engine="gpu",
file_subset=plan.gpu_files,
lane="gpu",
)
cpu_worker = EncoderWorker(
**common,
env=cpu_env,
use_ffmpeg_fallback=use_ffmpeg_fallback,
engine="cpu",
file_subset=plan.cpu_files,
lane="cpu",
ffmpeg_threads=plan.cpu_budget_threads,
)
self.worker = cpu_worker # primary handle (back-compat)
self._set_workers([gpu_worker, cpu_worker])
return True
def _handle_vs_incompat(self) -> bool:
"""Handle detected VSScript ABI incompatibility.
@ -1154,6 +1413,7 @@ class OpenCodecMaster(QMainWindow):
self._build_worker = SourceBuildWorker(
build_vs=build_vs, build_av1an=build_av1an,
build_ffmpeg_iamf=build_ffmpeg_iamf,
gpu_profile_key=self.env.av1an_flags.get("gpu_profile", "auto"),
)
self._build_worker.log_msg.connect(self._log)
self._build_worker.build_done.connect(self._on_build_done)
@ -1295,6 +1555,21 @@ class OpenCodecMaster(QMainWindow):
@Slot(int, int)
def _on_finished(self, ok: int, fail: int):
# v4.7.0: hybrid lanes finish independently — aggregate the
# per-lane summaries and only finalize when the LAST lane exits.
if len(self.workers) > 1:
self._hybrid_totals[0] += ok
self._hybrid_totals[1] += fail
self._hybrid_pending -= 1
if self._hybrid_pending > 0:
self._log(
f"Lane done — {ok} ok, {fail} failed. "
f"Waiting for the other lane..."
)
return
ok, fail = self._hybrid_totals
self._hybrid_totals = [0, 0]
self.btn_run.setEnabled(True)
self.btn_run.setText("START PROCESSING")
self.btn_stop.setEnabled(False)
@ -1307,9 +1582,10 @@ class OpenCodecMaster(QMainWindow):
@Slot()
def _stop_process(self):
if self.worker and self.worker.isRunning():
if any(w.isRunning() for w in self.workers):
self._log("STOP: Exiting queue after current file finishes...")
self.worker.stop()
for w in self.workers:
w.stop()
self.btn_stop.setEnabled(False)
@ -1327,7 +1603,9 @@ def launch_gui(argv: list[str] | None = None, force: bool = False,
verbose: bool = False,
skip_existing: bool = True,
timeout: int = 86400,
inline_scale: bool = False) -> int:
inline_scale: bool = False,
engine: str = "auto",
gpu_profile: str = "auto") -> int:
"""Create the QApplication, show the OpenCodecMaster window, run the Qt event loop.
This is the GUI entry point invoked by ``cli.main()`` when no
@ -1435,5 +1713,34 @@ def launch_gui(argv: list[str] | None = None, force: bool = False,
if inline_scale and hasattr(window, "inline_scale_check"):
window.inline_scale_check.setChecked(True)
window._log("Inline scale: enabled via --inline-scale (no intermediate for scaling).")
# v4.6.0: store --engine (auto|gpu|cpu). "auto" (default) uses the
# NVENC GPU encoder when the selected codec family has one and the
# live encode test passed. Pre-selects the ENGINE combo so the
# user sees the state; the combo is re-read at encode time, so the
# user can still change it per run.
if gpu_profile != "auto":
window.env.av1an_flags["gpu_profile"] = gpu_profile
if hasattr(window, "gpu_combo"):
_gpu_keys = ["auto"] + [gp.key for gp in GPU_PROFILES]
if gpu_profile in _gpu_keys:
window.gpu_combo.setCurrentIndex(_gpu_keys.index(gpu_profile))
window._log(f"GPU profile: {gpu_profile} (via --gpu-profile).")
if engine in ("gpu", "cpu", "hybrid"):
window.env.av1an_flags["engine"] = engine
if hasattr(window, "engine_combo"):
window.engine_combo.setCurrentIndex(
{"gpu": 1, "cpu": 2, "hybrid": 3}[engine]
)
window._log(f"Engine: {engine.upper()} (via --engine).")
else:
window.env.av1an_flags["engine"] = "auto"
gpu = getattr(window.env, "gpu", None)
if gpu is not None and gpu.has_gpu:
window._log(
f"Engine: auto — GPU detected ({gpu.name or 'NVIDIA'}), "
f"NVENC ready: {', '.join(gpu.usable_encoders)}."
)
else:
window._log("Engine: auto — no usable GPU encoder, CPU encoders will be used.")
window.show()
return sys.exit(app.exec())

2
pyproject.toml
View File

@ -18,7 +18,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "opentranscode"
version = "4.5.0"
version = "4.8.1"
description = "Open-source batch video transcoder wrapping av1an + ffmpeg with a PySide6 GUI"
readme = "README.md"
requires-python = ">=3.12"

91
scripts/build-ffmpeg-nvenc-matched.sh Executable file
View File

@ -0,0 +1,91 @@
#!/usr/bin/env bash
# build-ffmpeg-nvenc-matched.sh — build an ffmpeg that matches your NVIDIA
# driver's NVENC API version, installed to ~/.local (shadows the distro
# ffmpeg via ~/.local/bin precedence in PATH).
#
# WHY THIS EXISTS
# ---------------
# ffmpeg builds are compiled against a specific NVENC API (nv-codec-headers
# "gen") and the installed NVIDIA driver must be at least that new. If your
# card is on a legacy driver branch (e.g. 580 is the LAST branch supporting
# Pascal cards like the GTX 10xx), you cannot upgrade the driver to what a
# bleeding-edge ffmpeg wants — e.g.:
#
# [hevc_nvenc @ ...] Driver does not support the required nvenc API
# version. Required: 13.1 Found: 13.0
#
# The fix is the inverse: build ffmpeg against the API your driver DOES
# provide (Found: X.Y → nv-codec-headers nX.Y.*). OpenTranscode's probe
# picks ~/.local/bin/ffmpeg up automatically on the next launch (PATH
# order) and the Auto engine will start using the GPU.
#
# USAGE
# -----
# ./build-ffmpeg-nvenc-matched.sh # auto-detect driver API
# ./build-ffmpeg-nvenc-matched.sh 13.0 # explicit gen
#
# Requires: git, make, gcc, nasm, pkg-config, clang (for --enable-cuda-llvm;
# the script drops that flag automatically if clang is absent).
set -euo pipefail
FF_VER="${FF_VER:-n9.0.2}"
PREFIX="${PREFIX:-$HOME/.local}"
BUILD_DIR="${BUILD_DIR:-/tmp/ffmpeg-nvenc-build}"
BASE_CONF="${BASE_CONF:-}" # optional: a configure line to copy (e.g. distro ffmpeg)
if [[ $# -ge 1 ]]; then
GEN="$1"
else
# First two components of the driver's NVENC API, from the error ffmpeg
# prints (or run `nvidia-smi` — 580/550/etc. drivers ↔ gen 13.0/12.2).
read -rp "NVENC API gen to build against (e.g. 13.0): " GEN
fi
HEADERS_TAG="n${GEN}.19.1" # any n<GEN>.* tag provides that gen
mkdir -p "$BUILD_DIR"
echo "==> nv-codec-headers $HEADERS_TAG → $PREFIX"
if [[ ! -d "$BUILD_DIR/nv-codec-headers" ]]; then
git clone --quiet https://github.com/FFmpeg/nv-codec-headers.git "$BUILD_DIR/nv-codec-headers"
fi
git -C "$BUILD_DIR/nv-codec-headers" fetch --quiet --depth 1 origin tag "$HEADERS_TAG" || true
git -C "$BUILD_DIR/nv-codec-headers" checkout --quiet "$HEADERS_TAG"
make -C "$BUILD_DIR/nv-codec-headers" install "PREFIX=$PREFIX" >/dev/null
echo "==> ffmpeg $FF_VER source → $BUILD_DIR/src"
if [[ ! -d "$BUILD_DIR/src" ]]; then
git clone --quiet --depth 1 --branch "$FF_VER" \
https://github.com/FFmpeg/ffmpeg.git "$BUILD_DIR/src"
fi
CONF_ARGS="${BASE_CONF:-}"
if [[ -z "$CONF_ARGS" && -x /usr/bin/ffmpeg ]]; then
# Copy the distro build's feature set so the result is a drop-in.
CONF_ARGS=$(/usr/bin/ffmpeg -version 2>/dev/null | sed -n 3p | sed 's/^configuration: //')
fi
if [[ -z "$CONF_ARGS" ]]; then
CONF_ARGS="--enable-gpl --enable-libx264 --enable-libx265 --enable-libvpx --enable-libsvtav1 --enable-libopus --enable-libvorbis --enable-libdav1d"
fi
CONF_ARGS=${CONF_ARGS/--prefix=*\/usr /} # strip distro prefix
CONF_ARGS=$(echo "$CONF_ARGS" | sed 's/--prefix=[^ ]*//')
# --enable-rpath is CRITICAL: without it the installed binary resolves its
# SONAME libs (libavcodec.so etc.) from /usr/lib — the DISTRO build — and
# silently keeps demanding the newer NVENC API. rpath pins it to
# $PREFIX/lib where our NVENC-13.0-matched libs live.
CONF_ARGS="--prefix=$PREFIX $CONF_ARGS --enable-nvenc --enable-nvdec --enable-rpath"
command -v clang >/dev/null || CONF_ARGS=$(echo "$CONF_ARGS" | sed 's/--enable-cuda-llvm //')
echo "==> configure"
export PKG_CONFIG_PATH="$PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
cd "$BUILD_DIR/src"
# shellcheck disable=SC2086
./configure $CONF_ARGS
echo "==> make ($(nproc) jobs) — this takes a few minutes"
make -j"$(nproc)"
make install
echo
echo "==> done. Verify (all three must succeed):"
echo " ldd $PREFIX/bin/ffmpeg | grep avcodec # MUST print $PREFIX/lib/..."
echo " hash -r; which ffmpeg # should print $PREFIX/bin/ffmpeg"
echo " $PREFIX/bin/ffmpeg -hide_banner -loglevel error -f lavfi -i 'color=c=black:s=256x256:d=0.2' -c:v hevc_nvenc -f null -"

25
tests/conftest.py
View File

@ -357,3 +357,28 @@ def make_minimal_worker(opentranscode_module, env=None, audio_level_db=-14.0):
worker.subtitle_lang = None
worker.use_ffmpeg_fallback = False
return worker
# ─────────────────────────────────────────────────────────────────────────────
# Signal capture helper (PySide6-safe)
# ─────────────────────────────────────────────────────────────────────────────
def capture_signal(logs: list):
"""Return a drop-in replacement for a Qt Signal that appends every
emit()ed message to *logs*.
Tests must REPLACE the whole signal object (``worker.log_msg =
capture_signal(logs)``) instead of patching ``worker.log_msg.emit``:
PySide6's real SignalInstance forbids attribute assignment ("attribute
'emit' is read-only"), and whether the real PySide6 is loaded depends
on test ordering — so patching .emit works only in some runs. This
helper behaves identically under the stub and the real PySide6.
"""
class _CaptureSignal:
def connect(self, fn):
pass
def emit(self, msg):
logs.append(msg)
return _CaptureSignal()

19
tests/test_chunk_method_retry.py
View File

@ -26,6 +26,7 @@ from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
OPENTRANSCODE_PATH = Path(__file__).resolve().parent.parent / "open-transcode.py"
@ -279,7 +280,10 @@ class TestY4mBreakRetry:
worker._ffmpeg_fallback_encode = MagicMock(return_value=True)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
# v4.2.1+: RETRY log lines are verbose-only; the test asserts on
# them, so enable verbose output.
worker.verbose = True
# Setup required attributes
worker._current_temps = []
@ -379,7 +383,7 @@ class TestY4mBreakRetry:
worker._ffmpeg_fallback_encode = mock_ffmpeg_fallback
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
worker._current_temps = []
worker._stop = False
@ -455,7 +459,7 @@ class TestY4mBreakRetry:
worker._ffmpeg_fallback_encode = MagicMock(return_value=True)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
worker._current_temps = []
worker._stop = False
@ -490,9 +494,16 @@ class TestVSPluginProbe:
"""When no VS plugin .so files exist in any search dir, the probe
should return an empty list."""
# Point HOME at an empty tmp dir so the home-dir search paths
# don't accidentally find real plugins
# don't accidentally find real plugins. v4.7.1: the probe also
# scans the python site-packages plugin dirs — patch those too,
# or a machine with a git-built BestSource (user site) would
# legitimately report it and break the hermetic expectation.
monkeypatch.setenv("HOME", str(tmp_path))
monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path / "share"))
import opentranscode.env_probe as _ep
monkeypatch.setattr(_ep.site, "getusersitepackages",
lambda: str(tmp_path / "site-packages"))
monkeypatch.setattr(_ep.site, "getsitepackages", lambda: [])
result = opentranscode_module._probe_vs_source_plugins()
assert result == [], \

41
tests/test_e2e_real_encode.py
View File

@ -32,6 +32,26 @@ from unittest.mock import MagicMock
import pytest
def _capture_signal(logs: list):
"""Return a drop-in replacement for a Qt Signal that captures emit()ed
messages into *logs*.
PySide6's SignalInstance attributes are read-only, so tests running
against a REAL PySide6 install cannot patch ``worker.log_msg.emit``
directly (they can under the conftest stubs). Replacing the whole
signal object works in both modes.
"""
class _CaptureSignal:
def connect(self, fn):
pass
def emit(self, msg):
logs.append(msg)
return _CaptureSignal()
# ─────────────────────────────────────────────────────────────────────────────
# Module loading — the open-transcode.py file has a dash in its name, can't use import
# ─────────────────────────────────────────────────────────────────────────────
@ -270,12 +290,10 @@ class TestRealEncodePipeline:
subtitle_lang=None,
)
# Collect log messages
# Collect log messages (replace the signal — SignalInstance
# attributes are read-only under a real PySide6 install)
logs: list[str] = []
worker.log_msg.connect = lambda fn: setattr(worker, "_log_fn", fn)
# Patch the log_msg signal emit to capture messages
original_emit = worker.log_msg.emit
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
# Run the worker synchronously (bypass QThread.start)
worker.run()
@ -385,7 +403,7 @@ class TestRealEncodePipeline:
)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
worker.run()
@ -458,7 +476,7 @@ class TestRealEncodePipeline:
)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
worker.run()
@ -525,7 +543,7 @@ class TestMovflagsFix:
)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
worker.run()
# Find the CMD log line — it should contain -movflags +faststart for MP4
@ -572,7 +590,7 @@ class TestMovflagsFix:
)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
worker.run()
output_files = list(out_dir.rglob("*_archived.mkv"))
@ -759,7 +777,10 @@ class TestV7Y4mBreakRecovery:
worker._ffmpeg_fallback_encode = MagicMock(return_value=True)
logs: list[str] = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = _capture_signal(logs)
# v4.2.1+: RETRY log lines are verbose-only; the test asserts on
# them, so enable verbose output.
worker.verbose = True
# Run the full pipeline (not just _encode_one)
worker.run()

7
tests/test_ffmpeg_fallback.py
View File

@ -14,6 +14,7 @@ from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
OPENTRANSCODE_PATH = Path(__file__).resolve().parent.parent / "open-transcode.py"
@ -211,7 +212,7 @@ class TestPerFileFallbackRetry:
worker._ffmpeg_fallback_encode = mock_ffmpeg_fallback
logs = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
# Set up _current_temps and _file_res_map (needed by _process_one_file)
worker._current_temps = []
@ -293,7 +294,7 @@ class TestPerFileFallbackRetry:
worker._ffmpeg_fallback_encode = MagicMock(return_value=True)
logs = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
worker._current_temps = []
worker._stop = False
@ -356,7 +357,7 @@ class TestPerFileFallbackRetry:
worker._ffmpeg_fallback_encode = MagicMock(return_value=True)
logs = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
worker._current_temps = []
worker._stop = False # will be set by the pattern matcher

9
tests/test_force_validation.py
View File

@ -16,6 +16,7 @@ from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
# ─────────────────────────────────────────────────────────────────────────────
@ -172,7 +173,9 @@ class TestSkipInvalidFiles:
worker.env.ffprobe_path = shutil.which("ffprobe") or "/usr/bin/ffprobe"
logs = []
worker.log_msg = MagicMock()
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
# v4.2.1+: file-type diagnostics on SKIP are verbose-only.
worker.verbose = True
skip, info, src_w, src_h = worker._validate_file(fake_video)
@ -202,7 +205,7 @@ class TestSkipInvalidFiles:
worker.env.ffprobe_path = shutil.which("ffprobe") or "/usr/bin/ffprobe"
logs = []
worker.log_msg = MagicMock()
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
skip, info, src_w, src_h = worker._validate_file(fake_video)
@ -330,7 +333,7 @@ class TestPreFlightValidation:
)
logs = []
worker.log_msg.emit = lambda msg: logs.append(msg)
worker.log_msg = capture_signal(logs)
# Run the worker — should abort in pre-flight validation
worker.run()

421
tests/test_gpu_engine.py Normal file
View File

@ -0,0 +1,421 @@
"""v4.6.0 GPU (NVENC) engine tests.
Covers:
- ``GpuInfo`` defaults and properties
- ``_probe_gpu``: compiled-in detection + functional smoke test gate
(the driver/ffmpeg NVENC API mismatch case), with subprocess mocked
- ``resolve_gpu_encoder``: the auto/gpu/cpu decision matrix
- codec profile gpu fields + NVENC vargs shape
- ``FFMPEG_LIB_KEY_MAP`` nvenc entries
- CLI ``--engine`` flag
- EncoderWorker engine attribute + env fallback
- launcher-script parity (resolve_gpu_encoder + engine param exist)
"""
import inspect
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
from opentranscode.codec_profiles import (
FFMPEG_LIB_KEY_MAP,
VIDEO_CODECS,
_av1_nvenc_args,
_hevc_nvenc_args,
)
from opentranscode.encoder_worker import EncoderWorker, resolve_gpu_encoder
from opentranscode.env_probe import GpuInfo, _NVENC_ENCODER_NAMES, _probe_gpu
# ─────────────────────────────────────────────────────────────────────────────
# GpuInfo
# ─────────────────────────────────────────────────────────────────────────────
class TestGpuInfo:
def test_defaults_have_no_gpu(self):
info = GpuInfo()
assert info.has_gpu is False
assert info.usable_encoders == []
assert info.first_failure_detail == ""
assert info.name == ""
def test_usable_encoders_in_preference_order(self):
info = GpuInfo()
for enc in reversed(_NVENC_ENCODER_NAMES):
info.functional[enc] = True
assert info.usable_encoders == list(_NVENC_ENCODER_NAMES)
def test_first_failure_detail_skips_empty(self):
info = GpuInfo()
info.details["av1_nvenc"] = ""
info.details["hevc_nvenc"] = "driver too old"
assert info.first_failure_detail == "driver too old"
# ─────────────────────────────────────────────────────────────────────────────
# _probe_gpu (subprocess mocked)
# ─────────────────────────────────────────────────────────────────────────────
def _fake_run_factory(encoders_output: str, smoke_ok: bool):
"""Return a subprocess.run stand-in: -encoders prints encoders_output;
the smoke test succeeds or fails per smoke_ok."""
def fake_run(cmd, **kw):
if "-encoders" in cmd:
res = MagicMock(returncode=0)
res.stdout = encoders_output
return res
# Smoke test — fail with the real-world driver mismatch text.
if smoke_ok:
res = MagicMock(returncode=0)
res.stderr = ""
return res
res = MagicMock(returncode=-22)
res.stderr = (
"[hevc_nvenc @ 0x56146c9e2240] Driver does not support the "
"required nvenc API version. Required: 13.1 Found: 13.0\n"
"[hevc_nvenc @ 0x56146c9e2240] The minimum required Nvidia "
"driver for nvenc is 610.00 or newer\n"
)
return res
return fake_run
class TestProbeGpu:
def test_no_ffmpeg_returns_empty(self):
assert _probe_gpu(None).has_gpu is False
def test_no_nvenc_in_build(self, monkeypatch):
monkeypatch.setattr(
"opentranscode.env_probe.subprocess.run",
_fake_run_factory("V..... libsvtav1 SVT-AV1 encoder\n", smoke_ok=True),
)
info = _probe_gpu("/usr/bin/ffmpeg")
assert any(info.encoders.values()) is False
assert info.has_gpu is False
assert info.functional == {}
def test_compiled_in_but_driver_too_old(self, monkeypatch):
out = (" V....D hevc_nvenc NVIDIA NVENC hevc encoder\n"
" V....D h264_nvenc NVIDIA NVENC H.264 encoder\n")
monkeypatch.setattr(
"opentranscode.env_probe.subprocess.run",
_fake_run_factory(out, smoke_ok=False),
)
info = _probe_gpu("/usr/bin/ffmpeg")
assert info.encoders["hevc_nvenc"] is True
assert info.functional["hevc_nvenc"] is False
assert "API version" in info.first_failure_detail
assert info.has_gpu is False
def test_functional_nvenc(self, monkeypatch):
out = " V....D hevc_nvenc NVIDIA NVENC hevc encoder\n"
monkeypatch.setattr(
"opentranscode.env_probe.subprocess.run",
_fake_run_factory(out, smoke_ok=True),
)
monkeypatch.setattr(
"opentranscode.env_probe.shutil.which", lambda name: None
)
info = _probe_gpu("/usr/bin/ffmpeg")
assert info.has_gpu is True
assert info.usable_encoders == ["hevc_nvenc"]
def test_smoke_test_is_live_encode(self, monkeypatch):
"""The functional gate must run an actual encode, not just grep."""
seen_cmds = []
def fake_run(cmd, **kw):
seen_cmds.append(cmd)
res = MagicMock(returncode=0)
res.stdout = " V....D hevc_nvenc NVIDIA NVENC hevc encoder\n"
res.stderr = ""
return res
monkeypatch.setattr("opentranscode.env_probe.subprocess.run", fake_run)
monkeypatch.setattr("opentranscode.env_probe.shutil.which", lambda n: None)
_probe_gpu("/usr/bin/ffmpeg")
# Stage 1 (-encoders) + Stage 2 (lavfi source + -c:v hevc_nvenc).
smoke = [c for c in seen_cmds if "lavfi" in c]
assert len(smoke) == 1
assert "hevc_nvenc" in smoke[0]
assert "-f" in smoke[0] and "null" in smoke[0]
# ─────────────────────────────────────────────────────────────────────────────
# resolve_gpu_encoder decision matrix
# ─────────────────────────────────────────────────────────────────────────────
class _FakeGpu:
def __init__(self, functional):
self.functional = functional
self.name = "Test GPU"
class _FakeEnv:
def __init__(self, functional):
self.gpu = _FakeGpu(functional)
self.av1an_flags = {}
def _codec_by_label(fragment):
return next(c for c in VIDEO_CODECS if fragment in c.label)
class TestResolveGpuEncoder:
def test_auto_uses_functional_gpu(self):
env = _FakeEnv({"hevc_nvenc": True})
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, env) == ("hevc_nvenc", "nvenc")
def test_auto_falls_back_when_smoke_failed(self):
# The real GTX 1070 + ffmpeg 9.0.2 case: encoder listed but the
# live test failed → CPU.
env = _FakeEnv({"hevc_nvenc": False})
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_engine_cpu_never_gpu(self):
env = _FakeEnv({"hevc_nvenc": True})
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("cpu", codec, env) == (None, None)
def test_engine_gpu_requires_functional(self):
env = _FakeEnv({"hevc_nvenc": False})
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("gpu", codec, env) == (None, None)
env2 = _FakeEnv({"hevc_nvenc": True})
assert resolve_gpu_encoder("gpu", codec, env2) == ("hevc_nvenc", "nvenc")
def test_vp9_needs_vaapi_profile(self):
env = _FakeEnv({"av1_nvenc": True, "hevc_nvenc": True, "h264_nvenc": True})
codec = _codec_by_label("VP9")
# No profile → legacy nvenc fallback → VP9 has none.
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_av1_gpu_only_when_functional(self):
env = _FakeEnv({"av1_nvenc": False})
codec = _codec_by_label("AV1")
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
env2 = _FakeEnv({"av1_nvenc": True})
assert resolve_gpu_encoder("auto", codec, env2) == ("av1_nvenc", "nvenc")
def test_env_without_gpu_attr_is_safe(self):
class _NoGpu:
pass
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, _NoGpu()) == (None, None)
# ── v4.8.0: profile-driven resolution ──
def test_profile_selects_vaapi_encoder(self):
from opentranscode.gpu_profiles import gpu_profile_by_key
env = _FakeEnv({"hevc_vaapi": True, "h264_vaapi": True})
env.av1an_flags = {"gpu_profile": "amd-rdna3"}
codec = _codec_by_label("x265")
enc, api = resolve_gpu_encoder("auto", codec, env)
assert (enc, api) == ("hevc_vaapi", "vaapi")
def test_profile_respects_functional_gate(self):
env = _FakeEnv({"hevc_vaapi": False})
env.av1an_flags = {"gpu_profile": "amd-rdna3"}
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_compute_profile_has_no_encoders(self):
# CMP 170HX / A100 class: claims nothing → CPU.
env = _FakeEnv({})
env.av1an_flags = {"gpu_profile": "nv-compute"}
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_pascal_profile_blocks_av1(self):
# AV1 on a 1070-class card: profile has no av1 encoder → CPU.
env = _FakeEnv({"av1_nvenc": True}) # functional (e.g. different card)
env.av1an_flags = {"gpu_profile": "nv-pascal"}
codec = _codec_by_label("AV1")
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_matched_profile_used_when_no_selection(self):
from opentranscode.gpu_profiles import gpu_profile_by_key
env = _FakeEnv({"hevc_qsv": True})
env.av1an_flags = {}
env.gpu.profile_key = "intel-arc"
codec = _codec_by_label("x265")
assert resolve_gpu_encoder("auto", codec, env) == ("hevc_qsv", "qsv")
# ─────────────────────────────────────────────────────────────────────────────
# Codec profiles: gpu fields + NVENC vargs
# ─────────────────────────────────────────────────────────────────────────────
class TestCodecProfileGpuFields:
def test_av1_has_av1_nvenc(self):
av1 = _codec_by_label("AV1")
assert av1.gpu_encoder == "av1_nvenc"
assert av1.gpu_vargs_fn is not None
def test_hevc_has_hevc_nvenc(self):
x265 = _codec_by_label("x265")
assert x265.gpu_encoder == "hevc_nvenc"
def test_vp9_has_no_gpu_encoder(self):
vp9 = _codec_by_label("VP9")
assert vp9.gpu_encoder == ""
def test_nvenc_vargs_use_cq_mode(self):
args = _hevc_nvenc_args(28, 7)
assert args[0] == "-c:v" and args[1] == "hevc_nvenc"
assert "-cq" in args and "28" in args
assert "-rc" in args and "vbr" in args
assert "-b:v" in args and "0" in args
assert "-pix_fmt" in args and "yuv420p" in args
av1_args = _av1_nvenc_args(32, 6)
assert "av1_nvenc" in av1_args
def test_skip_existing_codec_name_unchanged_by_gpu(self):
"""hevc_nvenc outputs the same ffprobe codec_name ("hevc") as
libx265, so skip-existing works across engine switches."""
x265 = _codec_by_label("x265")
assert x265.ffprobe_codec_name == "hevc"
av1 = _codec_by_label("AV1")
assert av1.ffprobe_codec_name == "av1"
def test_ffmpeg_lib_key_map_has_nvenc(self):
assert FFMPEG_LIB_KEY_MAP["hevc_nvenc"] == "hevc_nvenc"
assert FFMPEG_LIB_KEY_MAP["h264_nvenc"] == "h264_nvenc"
assert FFMPEG_LIB_KEY_MAP["av1_nvenc"] == "av1_nvenc"
# ─────────────────────────────────────────────────────────────────────────────
# CLI --engine flag
# ─────────────────────────────────────────────────────────────────────────────
class TestCliEngineFlag:
def test_engine_choices(self):
from opentranscode.cli import build_parser
p = build_parser()
assert p.parse_args(["--engine", "gpu"]).engine == "gpu"
assert p.parse_args(["--engine", "cpu"]).engine == "cpu"
assert p.parse_args([]).engine == "auto"
def test_engine_rejects_unknown(self):
from opentranscode.cli import build_parser
p = build_parser()
with pytest.raises(SystemExit):
p.parse_args(["--engine", "tape"])
# ─────────────────────────────────────────────────────────────────────────────
# EncoderWorker engine attribute + GPU encode path
# ─────────────────────────────────────────────────────────────────────────────
class TestEncoderWorkerEngine:
def _make_worker(self, mock_env, engine=None, use_ffmpeg_fallback=True):
from opentranscode.codec_profiles import (
AUDIO_PROFILES, CONTAINER_PROFILES, RESOLUTION_PRESETS,
)
return EncoderWorker(
in_dir=Path("/tmp"),
out_dir=Path("/tmp"),
video_codec=_codec_by_label("x265"),
audio_profile=AUDIO_PROFILES[0],
container=CONTAINER_PROFILES[0],
crf=28,
preset_label="Medium (7)",
delete_source=False,
env=mock_env,
extensions={".mkv"},
resolution=RESOLUTION_PRESETS[0],
use_ffmpeg_fallback=use_ffmpeg_fallback,
engine=engine,
)
def test_engine_attr_stored(self, mock_env):
w = self._make_worker(mock_env, engine="gpu")
assert w.engine == "gpu"
def test_engine_falls_back_to_env_flags(self, mock_env):
mock_env.av1an_flags["engine"] = "cpu"
try:
w = self._make_worker(mock_env) # engine not passed
assert w.engine == "cpu"
finally:
mock_env.av1an_flags.pop("engine", None)
def test_engine_rejects_unknown_value(self, mock_env):
mock_env.av1an_flags.pop("engine", None)
w = self._make_worker(mock_env, engine="quantum")
assert w.engine == "auto"
def test_gpu_encoder_unresolved_before_run(self, mock_env):
w = self._make_worker(mock_env)
assert w._gpu_encoder is None
def test_gpu_mode_builds_nvenc_command(self, mock_env):
"""engine=auto + functional hevc_nvenc → the ffmpeg command uses
hevc_nvenc with -cq instead of libx265 with -crf."""
mock_env.av1an_flags["verbose"] = True
mock_env.gpu = _FakeGpu({"hevc_nvenc": True})
mock_env.ffmpeg_libs["hevc_nvenc"] = True
w = self._make_worker(mock_env, engine="auto", use_ffmpeg_fallback=False)
# run() resolves the engine before the encode loop; mirror that
# here (this test targets the command-construction path only).
w._gpu_encoder, w._gpu_api = resolve_gpu_encoder(
w.engine, w.video_codec, mock_env)
w.use_ffmpeg_fallback = True
w.resolution = type("R", (), {"width": None, "height": None})()
w._temp_dir = Path("/tmp")
w._current_temps = []
w._file_res_map = {}
w._stop = False
w._consecutive_fail_count = 0
w._last_fail_pattern = None
w.audio_level_db = 0.0
w.encode_timeout = 60
logs: list[str] = []
w.log_msg = capture_signal(logs)
w._run_with_stop_check = MagicMock(
return_value=("ok", 0, "", "")
)
src = Path("/tmp/fake_src.mkv")
out = Path("/tmp/fake_out.mkv")
ok = w._ffmpeg_fallback_encode(src, src, out)
assert ok is False # "output exists" check fails (mock ran nothing)
cmd = w._run_with_stop_check.call_args[0][0]
assert "hevc_nvenc" in cmd
assert "-cq" in cmd
assert "libx265" not in cmd
def test_cpu_mode_keeps_libx265(self, mock_env):
mock_env.gpu = _FakeGpu({"hevc_nvenc": True})
w = self._make_worker(mock_env, engine="cpu", use_ffmpeg_fallback=True)
w.resolution = type("R", (), {"width": None, "height": None})()
w.audio_level_db = 0.0
w.encode_timeout = 60
w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", ""))
src = Path("/tmp/fake_src.mkv")
w._ffmpeg_fallback_encode(src, src, Path("/tmp/fake_out.mkv"))
cmd = w._run_with_stop_check.call_args[0][0]
assert "libx265" in cmd
assert "hevc_nvenc" not in cmd
# ─────────────────────────────────────────────────────────────────────────────
# Launcher parity
# ─────────────────────────────────────────────────────────────────────────────
class TestLauncherParity:
def test_launcher_defines_resolve_gpu_encoder(self, opentranscode_module):
assert hasattr(opentranscode_module, "resolve_gpu_encoder")
def test_launcher_worker_accepts_engine(self, opentranscode_module):
sig = inspect.signature(opentranscode_module.EncoderWorker.__init__)
assert "engine" in sig.parameters
def test_launcher_profiles_have_gpu_fields(self, opentranscode_module):
for vc in opentranscode_module.VIDEO_CODECS:
assert hasattr(vc, "gpu_encoder")
assert hasattr(vc, "gpu_vargs_fn")

202
tests/test_gpu_profiles.py Normal file
View File

@ -0,0 +1,202 @@
"""v4.8.0 GPU capability-profile tests.
Covers the combined generation dropdown entries: profile data
integrity, the auto-match matrix (including Tesla/data-center and the
crypto-era CMP oddballs), profile-driven encoder resolution, and the
rebuild dep-tree extension.
"""
import os
import pytest
from opentranscode.codec_profiles import VIDEO_CODECS
from opentranscode.encoder_worker import resolve_gpu_encoder
from opentranscode.gpu_profiles import (
GPU_PROFILES,
encoder_filter_chain,
encoder_quality_args,
encoder_pre_args,
gpu_profile_by_key,
match_gpu_profile,
)
class TestProfileData:
def test_keys_unique(self):
keys = [p.key for p in GPU_PROFILES]
assert len(keys) == len(set(keys))
def test_labels_unique(self):
labels = [p.label for p in GPU_PROFILES]
assert len(labels) == len(set(labels))
def test_cpu_only_profile_claims_nothing(self):
cpu = gpu_profile_by_key("cpu")
assert cpu.encoders == {}
assert cpu.api == "none"
def test_compute_boards_claim_nothing(self):
"""V100/A100/H100-class boards and CMP 170HX have no NVENC —
selecting them must land on the CPU path, not crash."""
compute = gpu_profile_by_key("nv-compute")
assert compute.encoders == {}
def test_ada_has_av1_older_nvidia_does_not(self):
ada = gpu_profile_by_key("nv-ada")
assert ada.encoders["av1"] == "av1_nvenc"
for key in ("nv-pascal", "nv-turing", "nv-ampere"):
assert "av1" not in gpu_profile_by_key(key).encoders
def test_intel_arc_has_av1_qsv(self):
arc = gpu_profile_by_key("intel-arc")
assert arc.api == "qsv"
assert arc.encoders["av1"] == "av1_qsv"
def test_amd_rdna3_has_av1_vaapi_rdna12_does_not(self):
assert gpu_profile_by_key("amd-rdna3").encoders["av1"] == "av1_vaapi"
assert "av1" not in gpu_profile_by_key("amd-rdna12").encoders
def test_vp9_only_via_vaapi(self):
from opentranscode.codec_profiles import VIDEO_CODECS
vp9 = next(c for c in VIDEO_CODECS if "VP9" in c.label)
assert vp9.gpu_encoders_by_api == {"vaapi": "vp9_vaapi"}
class TestMatchMatrix:
@pytest.mark.parametrize("name,expected_key", [
("NVIDIA GeForce GTX 1070", "nv-pascal"),
("NVIDIA GeForce GTX 1080 Ti", "nv-pascal"),
("NVIDIA GeForce RTX 2060", "nv-turing"),
("NVIDIA GeForce GTX 1660 SUPER", "nv-turing"),
("NVIDIA GeForce RTX 3060", "nv-ampere"),
("NVIDIA GeForce RTX 4090", "nv-ada"),
("NVIDIA GeForce RTX 5080", "nv-ada"),
("Tesla P40", "nv-pascal"),
("Tesla T4", "nv-turing"),
("NVIDIA A10", "nv-ampere"),
("NVIDIA CMP 50HX", "nv-turing"),
("NVIDIA CMP 90HX", "nv-ampere"),
("NVIDIA CMP 170HX GA100", "nv-compute"),
("NVIDIA CMP 170HX", "nv-compute"),
("NVIDIA A100-SXM4-40GB", "nv-compute"),
("NVIDIA H100", "nv-compute"),
("Intel(R) Arc(TM) B580", "intel-arc"),
("Intel(R) UHD Graphics 770", "intel-xe"),
])
def test_match(self, name, expected_key):
assert match_gpu_profile(name).key == expected_key
def test_no_match_returns_none(self):
assert match_gpu_profile("") is None
assert match_gpu_profile("Unknown Widget Graphics") is None
class TestProfileResolution:
def _env(self, functional, flags=None, profile_key=""):
class G:
pass
g = G()
g.functional = functional
g.profile_key = profile_key
class E:
pass
e = E()
e.gpu = g
e.av1an_flags = flags or {}
return e
def test_hevc_via_pascal_profile(self):
env = self._env({"hevc_nvenc": True}, {"gpu_profile": "nv-pascal"})
codec = next(c for c in VIDEO_CODECS if "x265" in c.label)
assert resolve_gpu_encoder("auto", codec, env) == ("hevc_nvenc", "nvenc")
def test_arc_qsv_hevc(self):
env = self._env({"hevc_qsv": True}, {"gpu_profile": "intel-arc"})
codec = next(c for c in VIDEO_CODECS if "x265" in c.label)
assert resolve_gpu_encoder("auto", codec, env) == ("hevc_qsv", "qsv")
def test_vp9_on_amd_stays_cpu(self):
# AMD VCN has no VP9 encode — even though ffmpeg has vp9_vaapi,
# the RDNA 3 profile truthfully claims none.
env = self._env({"vp9_vaapi": True}, {"gpu_profile": "amd-rdna3"})
codec = next(c for c in VIDEO_CODECS if "VP9" in c.label)
assert resolve_gpu_encoder("auto", codec, env) == (None, None)
def test_vaapi_encoder_args_shape(self):
profile = gpu_profile_by_key("amd-rdna3")
assert encoder_pre_args(profile)[0] == "-vaapi_device"
assert "format=nv12,hwupload" in encoder_filter_chain(profile)
args = encoder_quality_args("vaapi", "hevc_vaapi", 28, 9)
assert args[:2] == ["-rc_mode", "CQP"]
class TestRebuildGpuDeps:
def test_nvenc_needs_codec_headers(self):
from opentranscode.source_builder import gpu_dep_packages
pkgs = gpu_dep_packages("nvenc", "arch")
assert "nv-codec-headers" in pkgs
def test_vaapi_qsv_driver_packages(self):
from opentranscode.source_builder import gpu_dep_packages
assert any("libva" in p for p in gpu_dep_packages("vaapi", "arch"))
assert any("intel-media-driver" in p for p in gpu_dep_packages("qsv", "arch"))
def test_unpackaged_family_returns_empty(self):
from opentranscode.source_builder import gpu_dep_packages
assert gpu_dep_packages("nvenc", "nixos") == []
def importlib_pyside_available():
import importlib.util
return importlib.util.find_spec("PySide6") is not None
class TestGpuComboEngineInteraction:
@pytest.mark.skipif(
not importlib_pyside_available(),
reason="requires a real PySide6 (stub mode cannot instantiate Qt)",
)
def test_cpu_engine_disables_gpu_menu(self):
"""v4.8.1: ENGINE = CPU greys out the GPU dropdown; the explicit
'None (CPU-only encode)' entry remains for opting out while a
GPU-capable engine is selected."""
os.environ.setdefault("QT_QPA_PLATFORM", "offscreen")
from PySide6.QtWidgets import QApplication
from opentranscode.ui_window import OpenCodecMaster
app = QApplication.instance() or QApplication([])
window = OpenCodecMaster()
try:
gpu_idx = next(i for i in range(window.gpu_combo.count())
if "None (CPU-only" in window.gpu_combo.itemText(i))
assert gpu_idx > 0, "None (CPU-only) must be an explicit option"
window.engine_combo.setCurrentIndex(0) # Auto
assert window.gpu_combo.isEnabled()
window.engine_combo.setCurrentIndex(2) # CPU
assert not window.gpu_combo.isEnabled(), (
"ENGINE = CPU must disable the GPU dropdown"
)
window.engine_combo.setCurrentIndex(1) # GPU
assert window.gpu_combo.isEnabled()
finally:
window.close()
class TestLauncherParity:
def test_launcher_has_gpu_profiles(self, opentranscode_module):
assert hasattr(opentranscode_module, "GPU_PROFILES")
assert hasattr(opentranscode_module, "match_gpu_profile")
def test_launcher_match_same_as_package(self, opentranscode_module):
from opentranscode.gpu_profiles import match_gpu_profile as pkg_match
assert opentranscode_module.match_gpu_profile("RTX 4090").key \
== pkg_match("RTX 4090").key
def test_launcher_worker_gpu_param(self, opentranscode_module):
import inspect
sig = inspect.signature(
opentranscode_module.SourceBuildWorker.__init__)
assert "gpu_profile_key" in sig.parameters

278
tests/test_hybrid_scheduler.py Normal file
View File

@ -0,0 +1,278 @@
"""v4.7.0 hybrid (GPU + CPU lanes) scheduler tests.
Covers:
- ``plan_hybrid``: degenerate cases, LPT size-balanced split, CPU
thread budget, unknown-size fallback
- ``EncoderWorker.file_subset``: a lane only processes its partition
- ``ffmpeg_threads``: injected on the CPU software path, never GPU
- per-lane temp dirs (``_worker_temp_dir`` lane suffix)
- ``scan_input_files``: extension filter + intermediate exclusion
- CLI ``--engine hybrid``
- launcher parity (plan_hybrid + lane params exist)
"""
import inspect
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
from opentranscode.codec_profiles import (
AUDIO_PROFILES,
CONTAINER_PROFILES,
RESOLUTION_PRESETS,
VIDEO_CODECS,
)
from opentranscode.encoder_worker import EncoderWorker, scan_input_files
from opentranscode.hybrid_scheduler import (
HYBRID_CPU_RESERVE_THREADS,
HybridPlan,
plan_hybrid,
)
def _hevc():
return next(c for c in VIDEO_CODECS if "x265" in c.label)
def _files(tmp_path, names):
files = []
for i, name in enumerate(names):
f = tmp_path / name
f.write_bytes(b"\x00" * 1024)
files.append(f)
return files
# ─────────────────────────────────────────────────────────────────────────────
# plan_hybrid
# ─────────────────────────────────────────────────────────────────────────────
class TestPlanHybrid:
def test_no_gpu_encoder_returns_none(self, tmp_path):
files = _files(tmp_path, ["a.mp4", "b.mp4"])
assert plan_hybrid(files, "", True, 28) is None
assert plan_hybrid(files, "hevc_nvenc", False, 28) is None
assert plan_hybrid(files, "hevc_nvenc", True, 28) is not None
def test_empty_file_list_returns_none(self):
assert plan_hybrid([], "hevc_nvenc", True, 28) is None
def test_single_file_goes_to_gpu_lane(self, tmp_path):
files = _files(tmp_path, ["a.mp4"])
plan = plan_hybrid(files, "hevc_nvenc", True, 28)
assert plan.gpu_files == files
assert plan.cpu_files == []
def test_split_covers_all_files_exactly_once(self, tmp_path):
files = _files(tmp_path, [f"v{i}.mp4" for i in range(10)])
sizes = {f: (i + 1) * 100_000_000 for i, f in enumerate(files)}
plan = plan_hybrid(files, "hevc_nvenc", True, 28, sizes=sizes)
assert plan.total_files == len(files)
assert sorted(plan.gpu_files + plan.cpu_files) == sorted(files)
assert not set(plan.gpu_files) & set(plan.cpu_files)
def test_lpt_prefers_gpu_for_largest_files(self, tmp_path):
"""The largest file goes to the GPU lane (lowest per-byte cost),
and the greedy split keeps the smaller CPU load in check."""
files = _files(tmp_path, ["big.mp4", "small.mp4"])
sizes = {files[0]: 1_000_000_000, files[1]: 10_000_000}
plan = plan_hybrid(files, "hevc_nvenc", True, 28, sizes=sizes)
assert plan.gpu_files[0] == files[0] # biggest → GPU
def test_cpu_budget_reserves_gpu_threads(self, tmp_path):
files = _files(tmp_path, ["a.mp4", "b.mp4"])
plan = plan_hybrid(files, "hevc_nvenc", True, 28)
assert plan.cpu_budget_threads == 28 - HYBRID_CPU_RESERVE_THREADS
plan2 = plan_hybrid(files, "hevc_nvenc", True, 2)
assert plan2.cpu_budget_threads == 1 # never below 1
def test_unknown_sizes_fall_back_to_average(self, tmp_path):
a, b, c = _files(tmp_path, ["a.mp4", "b.mp4", "c.mp4"])
sizes = {a: 1000, b: 1000} # c unknown
plan = plan_hybrid([a, b, c], "hevc_nvenc", True, 28, sizes=sizes)
assert plan.total_files == 3 # no crash, all files placed
def test_plan_is_pure_dataclass(self):
plan = HybridPlan()
assert plan.total_files == 0
assert plan.gpu_encoder == ""
assert plan.cpu_budget_threads == 1
# ─────────────────────────────────────────────────────────────────────────────
# EncoderWorker hybrid support
# ─────────────────────────────────────────────────────────────────────────────
def _worker(mock_env, tmp_path, **kw):
return EncoderWorker(
in_dir=tmp_path / "in",
out_dir=tmp_path / "out",
video_codec=_hevc(),
audio_profile=AUDIO_PROFILES[2],
container=CONTAINER_PROFILES[0],
crf=28,
preset_label="Faster (10)",
delete_source=False,
env=mock_env,
extensions={".mkv", ".mp4"},
resolution=RESOLUTION_PRESETS[0],
**kw,
)
class TestWorkerHybridSupport:
def test_file_subset_restricts_queue(self, mock_env, tmp_path, tiny_test_video):
# Real videos — the worker ffprobe-validates its queue.
(tmp_path / "in").mkdir(parents=True)
a = tmp_path / "in" / "a.mkv"
b = tmp_path / "in" / "b.mkv"
a.write_bytes(tiny_test_video.read_bytes())
b.write_bytes(tiny_test_video.read_bytes())
(tmp_path / "out").mkdir(parents=True)
mock_env.ffmpeg_libs["hevc_nvenc"] = True
mock_env.gpu.functional["hevc_nvenc"] = True # live probe passes
w = _worker(mock_env, tmp_path, file_subset=[a], lane="gpu",
engine="gpu", use_ffmpeg_fallback=True)
logs: list[str] = []
w.log_msg = capture_signal(logs)
w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", ""))
# Fake a successful output for the encoded file.
def fake_run(cmd, **kw2):
# ffmpeg path: output is the last arg (after -y)
out = Path(cmd[-1])
out.parent.mkdir(parents=True, exist_ok=True)
out.write_bytes(b"\x00" * 8192)
return ("ok", 0, "", "")
w._run_with_stop_check = fake_run
w.audio_level_db = 0.0
w.verbose = False
w.run()
assert w.success_count == 1
# Only the subset file was encoded.
assert (tmp_path / "out" / "a_archived.mkv").exists()
assert not (tmp_path / "out" / "b_archived.mkv").exists()
def test_lane_temp_dirs_are_distinct(self, mock_env, tmp_path):
gpu = _worker(mock_env, tmp_path, lane="gpu")
cpu = _worker(mock_env, tmp_path, lane="cpu")
plain = _worker(mock_env, tmp_path)
assert gpu._temp_dir != cpu._temp_dir
assert "-gpu" in gpu._temp_dir.name
assert "-cpu" in cpu._temp_dir.name
assert "-" not in plain._temp_dir.name.replace("worker-", "").lstrip("0123456789") or \
plain._temp_dir.name == f"worker-{plain._temp_dir.name.split('-')[1]}"
def test_ffmpeg_threads_on_cpu_software_path(self, mock_env, tmp_path):
w = _worker(mock_env, tmp_path, ffmpeg_threads=13,
use_ffmpeg_fallback=True, engine="cpu")
w.resolution = RESOLUTION_PRESETS[0]
w.audio_level_db = 0.0
w.encode_timeout = 60
w._gpu_encoder = None # CPU lane
w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", ""))
src = tmp_path / "in" / "x.mp4"
src.parent.mkdir(parents=True, exist_ok=True)
src.write_bytes(b"\x00" * 64)
w._ffmpeg_fallback_encode(src, src, tmp_path / "out.mkv")
cmd = w._run_with_stop_check.call_args[0][0]
assert "-threads" in cmd and "13" in cmd
def test_failed_ffmpeg_encode_removes_partial_output(self, mock_env, tmp_path):
"""A failed ffmpeg encode must not leave a truncated file that
skip-existing could later mistake for a finished archive."""
w = _worker(mock_env, tmp_path, use_ffmpeg_fallback=True, engine="cpu")
w.resolution = RESOLUTION_PRESETS[0]
w.audio_level_db = 0.0
w.encode_timeout = 60
w._gpu_encoder = None
w._run_with_stop_check = MagicMock(return_value=("ok", 1, "", "boom"))
src = tmp_path / "in" / "x.mp4"
src.parent.mkdir(parents=True, exist_ok=True)
src.write_bytes(b"\x00" * 64)
out = tmp_path / "out.mkv"
out.write_bytes(b"partial") # ffmpeg left a partial file
ok = w._ffmpeg_fallback_encode(src, src, out)
assert ok is False
assert not out.exists(), "partial output must be removed on failure"
def test_no_threads_cap_on_gpu_path(self, mock_env, tmp_path):
mock_env.ffmpeg_libs["hevc_nvenc"] = True
w = _worker(mock_env, tmp_path, ffmpeg_threads=13,
use_ffmpeg_fallback=True, engine="gpu")
w.resolution = RESOLUTION_PRESETS[0]
w.audio_level_db = 0.0
w.encode_timeout = 60
w._gpu_encoder = "hevc_nvenc" # GPU lane — NVENC is silicon-bound
w._run_with_stop_check = MagicMock(return_value=("ok", 0, "", ""))
src = tmp_path / "in" / "x.mp4"
src.parent.mkdir(parents=True, exist_ok=True)
src.write_bytes(b"\x00" * 64)
w._ffmpeg_fallback_encode(src, src, tmp_path / "out.mkv")
cmd = w._run_with_stop_check.call_args[0][0]
assert "-threads" not in cmd
assert "hevc_nvenc" in cmd
# ─────────────────────────────────────────────────────────────────────────────
# scan_input_files
# ─────────────────────────────────────────────────────────────────────────────
class TestScanInputFiles:
def test_filters_extensions_and_intermediates(self, tmp_path):
keep = tmp_path / "keep.mp4"; keep.write_bytes(b"\x00")
drop = tmp_path / "drop.txt"; drop.write_bytes(b"\x00")
inter = tmp_path / "keep.scaled_tmp.mkv"; inter.write_bytes(b"\x00")
nested = tmp_path / "sub"; nested.mkdir()
n2 = nested / "nested.mkv"; n2.write_bytes(b"\x00")
result = scan_input_files(tmp_path, {".mp4", ".mkv"})
assert keep in result and n2 in result
assert drop not in result
assert inter not in result
def test_sorted_output(self, tmp_path):
for name in ("c.mp4", "a.mp4", "b.mp4"):
(tmp_path / name).write_bytes(b"\x00")
assert scan_input_files(tmp_path, {".mp4"}) == sorted(
scan_input_files(tmp_path, {".mp4"}))
# ─────────────────────────────────────────────────────────────────────────────
# CLI + launcher parity
# ─────────────────────────────────────────────────────────────────────────────
class TestCliHybrid:
def test_engine_hybrid_accepted(self):
from opentranscode.cli import build_parser
assert build_parser().parse_args(["--engine", "hybrid"]).engine == "hybrid"
def test_engine_default_still_auto(self):
from opentranscode.cli import build_parser
assert build_parser().parse_args([]).engine == "auto"
class TestLauncherHybridParity:
def test_launcher_has_plan_hybrid(self, opentranscode_module):
assert hasattr(opentranscode_module, "plan_hybrid")
assert hasattr(opentranscode_module, "HybridPlan")
def test_launcher_worker_lane_params(self, opentranscode_module):
sig = inspect.signature(opentranscode_module.EncoderWorker.__init__)
for p in ("file_subset", "lane", "ffmpeg_threads"):
assert p in sig.parameters, p
def test_launcher_plan_matches_package_behavior(self, opentranscode_module, tmp_path):
files = _files(tmp_path, ["a.mp4", "b.mp4", "c.mp4", "d.mp4"])
plan = opentranscode_module.plan_hybrid(
files, "hevc_nvenc", True, 28,
sizes={f: i * 10_000_000 for i, f in enumerate(files)},
)
pkg_plan = plan_hybrid(
files, "hevc_nvenc", True, 28,
sizes={f: i * 10_000_000 for i, f in enumerate(files)},
)
assert [Path(x) for x in plan.gpu_files] == pkg_plan.gpu_files
assert [Path(x) for x in plan.cpu_files] == pkg_plan.cpu_files

4
tests/test_intelligent_workers.py
View File

@ -302,6 +302,8 @@ def test_live_tail_emits_lines(opentranscode_module, mock_env, monkeypatch):
worker = make_minimal_worker(opentranscode_module, env=mock_env)
worker._stop = False
# v4.2.1+: the live stderr/stdout tail only emits when verbose=True.
worker.verbose = True
# Patch time.sleep so the poll loop runs instantly.
monkeypatch.setattr("time.sleep", lambda *a, **k: None)
@ -359,6 +361,8 @@ def test_live_tail_handles_carriage_return(opentranscode_module, mock_env, monke
worker = make_minimal_worker(opentranscode_module, env=mock_env)
worker._stop = False
# v4.2.1+: the live stderr/stdout tail only emits when verbose=True.
worker.verbose = True
monkeypatch.setattr("time.sleep", lambda *a, **k: None)
monkeypatch.setattr("os.killpg", lambda *a, **k: None)

344
tests/test_large_file_fixes.py Normal file
View File

@ -0,0 +1,344 @@
"""v4.6.0 large-file + av1an chunking fix tests.
Root causes these lock in:
1. The pure-ffmpeg path pre-scaled to a CRF-16 intermediate — a flat
1800s timeout and a 0.5-0.8x source size temp file killed large
files. Now the ffmpeg path scales inline and only the av1an path
pre-scales (with the per-file timeout and STOP support).
2. av1an's FRAME MISMATCH / "encoder crashed: exit status: 0" chunk
drift fell through to "Unknown av1an failure"; the ffmpeg ≥ 7
"-vsync" removal broke segment/hybrid chunking silently. Both are
now diagnosed and trigger the select retry.
3. Severe disk-space warnings were verbose-only, so quiet mode gave
zero notice before "No space left on device".
4. Loudnorm analysis decoded the whole VIDEO stream (-af only), pushing
large files past its 120s timeout; -vn makes it audio-only.
"""
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from conftest import capture_signal
from opentranscode.codec_profiles import (
AUDIO_PROFILES,
CONTAINER_PROFILES,
RESOLUTION_PRESETS,
VIDEO_CODECS,
)
from opentranscode.encoder_worker import EncoderWorker
def _worker(mock_env, use_ffmpeg_fallback, codec=None):
return EncoderWorker(
in_dir=Path("/tmp"),
out_dir=Path("/tmp"),
video_codec=codec or VIDEO_CODECS[0],
audio_profile=AUDIO_PROFILES[0],
container=CONTAINER_PROFILES[0],
crf=32,
preset_label="Medium (6)",
delete_source=False,
env=mock_env,
extensions={".mkv"},
resolution=RESOLUTION_PRESETS[0],
use_ffmpeg_fallback=use_ffmpeg_fallback,
)
# ─────────────────────────────────────────────────────────────────────────────
# _prepare_input: pre-scale only on the av1an path
# ─────────────────────────────────────────────────────────────────────────────
class TestPreScaleGating:
def test_ffmpeg_path_skips_prescale(self, mock_env, tmp_path):
"""With the ffmpeg path (incl. GPU mode), the source file is used
directly — no CRF-16 intermediate, no 30-minute pre-scale cap."""
w = _worker(mock_env, use_ffmpeg_fallback=True)
w._temp_dir = tmp_path
w._current_temps = []
w.resolution = RESOLUTION_PRESETS[2] # 1080p — scaling requested
w.verbose = False
src = tmp_path / "source.mp4"
src.write_bytes(b"\x00" * 1024)
mock_env.ffmpeg_path = "/usr/bin/ffmpeg"
result = w._prepare_input(
src, 1920, 1080, needs_scale=True,
scale_filter="scale=1920:1080",
)
assert result is not None
encode_input, output_f = result
assert encode_input == src # NO intermediate
# No pre-scale intermediate tracked (the .av1an work-dir path is
# still tracked by the symlink section — harmless, never created
# on this path).
assert not any(".scaled_tmp" in str(t) for t in w._current_temps)
def test_av1an_path_still_prescales(self, mock_env, tmp_path, monkeypatch):
"""The av1an path keeps the CRF-16 intermediate (VS plugins need
it) but now runs it via _run_with_stop_check with the per-file
timeout instead of a flat 1800s subprocess.run cap."""
w = _worker(mock_env, use_ffmpeg_fallback=False)
w._temp_dir = tmp_path
w._current_temps = []
w.resolution = RESOLUTION_PRESETS[2]
w.verbose = False
w.encode_timeout = 86400
logs: list[str] = []
w.log_msg = capture_signal(logs)
src = tmp_path / "source.mp4"
src.write_bytes(b"\x00" * 1024)
seen = {}
def fake_stop_check(cmd, env=None, timeout=None, log_prefix=" "):
seen["timeout"] = timeout
seen["cmd"] = cmd
# Fake a successful pre-scale: create the intermediate.
out_idx = cmd.index("-y")
Path(cmd[out_idx + 1]).write_bytes(b"\x00" * 4096)
return ("ok", 0, "", "")
w._run_with_stop_check = fake_stop_check
result = w._prepare_input(
src, 1920, 1080, needs_scale=True,
scale_filter="scale=1920:1080",
)
assert result is not None
encode_input, output_f = result
assert encode_input != src # intermediate produced
assert str(encode_input).endswith(".scaled_tmp.mkv")
assert seen["timeout"] == 86400 # per-file timeout, NOT 1800
assert any("libx265" in c for c in seen["cmd"]) # CRF-16 intermediate
def test_prescale_timeout_fails_cleanly(self, mock_env, tmp_path):
"""A pre-scale timeout now reports a dedicated message (the old
code surfaced a raw TimeoutExpired as 'pre-scale error')."""
w = _worker(mock_env, use_ffmpeg_fallback=False)
w._temp_dir = tmp_path
w._current_temps = []
w.resolution = RESOLUTION_PRESETS[2]
w.verbose = False
w.encode_timeout = 3600
w.fail_count = 0
logs: list[str] = []
w.log_msg = capture_signal(logs)
src = tmp_path / "source.mp4"
src.write_bytes(b"\x00" * 1024)
w._run_with_stop_check = MagicMock(return_value=("timeout", -1, "", ""))
result = w._prepare_input(
src, 1920, 1080, needs_scale=True,
scale_filter="scale=1920:1080",
)
assert result is None
assert w.fail_count == 1
assert any("pre-scale timeout" in l for l in logs)
def test_prescale_stop_aborts_without_fail(self, mock_env, tmp_path):
"""User STOP during pre-scale is not a transcode failure."""
w = _worker(mock_env, use_ffmpeg_fallback=False)
w._temp_dir = tmp_path
w._current_temps = []
w.resolution = RESOLUTION_PRESETS[2]
w.verbose = False
w.fail_count = 0
logs: list[str] = []
w.log_msg = capture_signal(logs)
src = tmp_path / "source.mp4"
src.write_bytes(b"\x00" * 1024)
w._run_with_stop_check = MagicMock(return_value=("stop", -1, "", ""))
result = w._prepare_input(
src, 1920, 1080, needs_scale=True,
scale_filter="scale=1920:1080",
)
assert result is None
assert w.fail_count == 0
# ─────────────────────────────────────────────────────────────────────────────
# Disk-space severe warnings are user-facing
# ─────────────────────────────────────────────────────────────────────────────
class TestDiskSpaceWarnings:
def _setup(self, mock_env, tmp_path, verbose):
w = _worker(mock_env, use_ffmpeg_fallback=True)
w._temp_dir = tmp_path
w.verbose = verbose
logs: list[str] = []
w.log_msg = capture_signal(logs)
src = tmp_path / "big.mkv"
src.write_bytes(b"\x00" * (2 * 1024 ** 3)) # 2 GB (sparse-ish; st_size counts)
return w, logs, src
def test_severe_output_warning_in_quiet_mode(self, mock_env, tmp_path, monkeypatch):
w, logs, src = self._setup(mock_env, tmp_path, verbose=False)
monkeypatch.setattr(
"opentranscode.encoder_worker.shutil.disk_usage",
lambda _p: MagicMock(free=512 * 1024 ** 2), # 0.5 GB free < 2 GB
)
w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=False)
assert any("low disk space on output" in l for l in logs), logs
def test_no_warning_when_plenty_free_quiet(self, mock_env, tmp_path, monkeypatch):
w, logs, src = self._setup(mock_env, tmp_path, verbose=False)
monkeypatch.setattr(
"opentranscode.encoder_worker.shutil.disk_usage",
lambda _p: MagicMock(free=100 * 1024 ** 3),
)
w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=False)
assert logs == []
def test_severe_temp_warning_in_quiet_mode_av1an_scale(
self, mock_env, tmp_path, monkeypatch,
):
w, logs, src = self._setup(mock_env, tmp_path, verbose=False)
w.use_ffmpeg_fallback = False # av1an path — intermediate on temp
calls = {"n": 0}
def fake_disk_usage(_p):
calls["n"] += 1
# First call: output partition (plenty); second: temp (low).
free = 100 * 1024 ** 3 if calls["n"] == 1 else 512 * 1024 ** 2
return MagicMock(free=free)
monkeypatch.setattr("opentranscode.encoder_worker.shutil.disk_usage",
fake_disk_usage)
w._check_disk_space(src, tmp_path / "out.mkv", needs_scale=True)
assert any("low disk space on temp" in l for l in logs), logs
# ─────────────────────────────────────────────────────────────────────────────
# Loudnorm analysis: audio-only decode
# ─────────────────────────────────────────────────────────────────────────────
class TestLoudnormAudioOnly:
def test_analysis_cmd_has_vn(self, mock_env, tmp_path):
w = _worker(mock_env, use_ffmpeg_fallback=True)
w.audio_level_db = -16.0 # enable normalization path
mock_env.ffmpeg_path = "/usr/bin/ffmpeg"
seen = {}
def fake_run(cmd, **kw):
seen["cmd"] = cmd
res = MagicMock(returncode=1)
res.stdout = ""
res.stderr = "no json" # no JSON → returns None (fine)
return res
# _analyze_audio_loudness uses subprocess.run directly
import opentranscode.encoder_worker as ew
monkeypatch = pytest.MonkeyPatch()
try:
monkeypatch.setattr(ew.subprocess, "run", fake_run)
gain = w._analyze_audio_loudness(tmp_path / "video.mp4")
finally:
monkeypatch.undo()
assert gain is None # stderr had no JSON — graceful fallback
assert "-vn" in seen["cmd"], (
"loudnorm analysis must skip video decoding (-vn) or large "
"files blow the 120s timeout"
)
# ─────────────────────────────────────────────────────────────────────────────
# av1an chunking: new failure patterns trigger the select retry
# ─────────────────────────────────────────────────────────────────────────────
_FRAME_MISMATCH_STDERR = (
"ERROR av1an_core::broker: [chunk 2] encoder failed 3 times, shutting "
"down worker: encoder crashed: exit status: 0\n"
)
_FRAME_MISMATCH_STDOUT = (
" FRAME MISMATCH: chunk 2: 53/52 (actual/expected frames)\n"
)
_VSYNC_STDERR = (
"Error: FFmpeg failed to segment:\n"
' stderr: "Unrecognized option \'vsync\'.\\nError splitting the '
'argument list: Option not found\\n"\n'
)
class TestChunkFailurePatterns:
def _encode_one_with_mocks(self, mock_env, tmp_path, stdout, stderr):
w = _worker(mock_env, use_ffmpeg_fallback=False)
w._temp_dir = tmp_path
w._current_temps = []
w._file_res_map = {}
w._stop = False
w._consecutive_fail_count = 0
w._last_fail_pattern = None
w.verbose = True
w.audio_level_db = 0.0
w.encode_timeout = 60
w._current_idx = 1
w._current_total = 1
w._current_filename = "test.mkv"
w._current_scale_filter = ""
mock_env.av1an_flags["svt_name"] = "svt-av1"
# conftest's mock_env pre-sets the env_probe "select" override;
# remove it so the first attempt runs with auto (the retry path
# under test).
mock_env.av1an_flags.pop("chunk_method_override", None)
logs: list[str] = []
w.log_msg = capture_signal(logs)
calls: list[str | None] = []
def fake_stop_check(cmd, env=None, timeout=None, log_prefix=" "):
cm = None
if "--chunk-method" in cmd:
cm = cmd[cmd.index("--chunk-method") + 1]
calls.append(cm)
if cm in (None, "hybrid"):
return ("ok", 1, stdout, stderr)
# select retry: fake a valid output
out = Path(cmd[cmd.index("-o") + 1])
out.parent.mkdir(parents=True, exist_ok=True)
out.write_bytes(b"\x00" * 8192)
return ("ok", 0, "", "encoding finished")
w._run_with_stop_check = fake_stop_check
w._ffmpeg_fallback_encode = MagicMock(return_value=True)
src = tmp_path / "input.mkv"
src.write_bytes(b"\x00" * 4096)
out = tmp_path / "out" / "input_archived.mkv"
result = w._encode_one(src, src, out, 1)
return w, logs, calls, result
def test_frame_mismatch_triggers_select_retry(self, mock_env, tmp_path):
"""FRAME MISMATCH (frames land in av1an's STDOUT) previously fell
to 'Unknown av1an failure'; now it retries with select."""
w, logs, calls, result = self._encode_one_with_mocks(
mock_env, tmp_path, _FRAME_MISMATCH_STDOUT, _FRAME_MISMATCH_STDERR,
)
assert result is True
assert calls == [None, "select"], calls
assert w.env.av1an_flags.get("chunk_method_override") == "select"
def test_vsync_removal_diagnosed_and_retried(self, mock_env, tmp_path):
"""ffmpeg ≥ 7 removed -vsync; av1an segment chunking dies with
'Unrecognized option' — must be diagnosed, not 'Unknown'."""
w, logs, calls, result = self._encode_one_with_mocks(
mock_env, tmp_path, "", _VSYNC_STDERR,
)
assert result is True
assert calls == [None, "select"]
assert any("vsync" in l for l in logs), logs
def test_frame_mismatch_not_misdiagnosed_as_concat(self, mock_env, tmp_path):
w, logs, calls, result = self._encode_one_with_mocks(
mock_env, tmp_path, _FRAME_MISMATCH_STDOUT, _FRAME_MISMATCH_STDERR,
)
assert not any("concat failure" in l for l in logs), logs

3
tests/test_package_structure.py
View File

@ -250,4 +250,5 @@ class TestEntryPoints:
capture_output=True, text=True, timeout=10,
)
assert result.returncode == 0
assert "4.5.0" in result.stdout
import opentranscode
assert opentranscode.__version__ in result.stdout

162
tests/test_source_builder.py Normal file
View File

@ -0,0 +1,162 @@
"""v4.7.1 rebuild-from-git tests.
The REBUILD FROM GIT button must be always usable: on a bare system it
generates its own dependency tree (distro-aware package install), then
builds VapourSynth + BestSource + av1an into user-owned prefixes.
Covers:
- ``build_dep_plan``: per-family package lists + install commands,
rust stripped when av1an isn't being built, manual note for
unsupported families
- ``_probe_vs_source_plugins``: discovers plugins in the python
site-packages tree (where the git VS stack installs)
- ``_av1an_env``: carries the fresh VS stack paths
- the REBUILD button is enabled without a probe (offscreen UI test)
- launcher parity
"""
import os
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from opentranscode.distro_probe import DistroProfile, detect_distro
from opentranscode.source_builder import BUILD_DEPS_BY_FAMILY, build_dep_plan
def _profile(family):
return DistroProfile(
family=family, name=f"Test {family}", version_id="?",
pkg_manager="pkg", install_cmd_template="", binary_extra_paths=[],
av1an_known_encoder_names=[], ffmpeg_pkg="ffmpeg", av1an_pkg="av1an",
notes="",
)
class TestBuildDepPlan:
def test_arch_uses_pacman_with_zimg_and_rust(self):
plan = build_dep_plan(_profile("arch"))
assert plan.install_cmd[:2] == ["pacman", "-S"]
assert "zimg" in plan.packages
assert "rust" in plan.packages
assert "meson" in plan.packages
assert plan.manual_note is None
def test_debian_uses_apt_with_zimg_dev(self):
plan = build_dep_plan(_profile("debian"))
assert plan.install_cmd[:2] == ["apt-get", "install"]
assert "libzimg-dev" in plan.packages
assert "cargo" in plan.packages
def test_redhat_and_suse_covered(self):
assert build_dep_plan(_profile("redhat")).install_cmd[0] == "dnf"
assert build_dep_plan(_profile("suse")).install_cmd[0] == "zypper"
for fam in ("redhat", "suse"):
assert any("zimg" in p for p in BUILD_DEPS_BY_FAMILY[fam])
def test_unknown_family_gets_manual_note(self):
plan = build_dep_plan(_profile("freebsd"))
assert plan.install_cmd is None
assert "manually" in plan.manual_note
def test_no_av1an_strips_rust(self):
plan = build_dep_plan(_profile("arch"), build_av1an=False)
assert "rust" not in plan.packages
assert "zimg" in plan.packages # still needed for VapourSynth
def test_real_distro_detection_has_plan(self):
plan = build_dep_plan(detect_distro())
assert plan.packages # this repo targets the covered families
def test_every_covered_family_installs_zimg(self):
"""zimg is VapourSynth's one hard library dep — a bare system
build must pull it in."""
for family, packages in BUILD_DEPS_BY_FAMILY.items():
assert any("zimg" in p for p in packages), family
class TestPluginProbeSeesGitStack:
def test_probe_discovers_user_site_plugins(self, tmp_path, monkeypatch):
import opentranscode.env_probe as ep
plugins = tmp_path / "site-packages" / "vapoursynth" / "plugins"
plugins.mkdir(parents=True)
(plugins / "libbestsource.so").write_bytes(b"\x7fELFfake")
monkeypatch.setattr(ep.site, "getusersitepackages",
lambda: str(tmp_path / "site-packages"))
monkeypatch.setattr(ep.site, "getsitepackages", lambda: [])
assert "bestsource" in ep._probe_vs_source_plugins()
def test_av1an_env_carries_fresh_stack(self, tmp_path, monkeypatch):
import opentranscode.env_probe as ep
vs_dir = tmp_path / "site-packages" / "vapoursynth"
vs_dir.mkdir(parents=True)
(vs_dir / "libvsscript.so").write_bytes(b"\x7fELFfake")
monkeypatch.setattr(ep.site, "getusersitepackages",
lambda: str(tmp_path / "site-packages"))
monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
monkeypatch.delenv("PYTHONPATH", raising=False)
env = ep._av1an_env()
assert str(vs_dir) in env["LD_LIBRARY_PATH"]
assert str(tmp_path / "site-packages") in env["PYTHONPATH"]
def test_av1an_env_without_git_stack_unchanged(self, tmp_path, monkeypatch):
import opentranscode.env_probe as ep
monkeypatch.setattr(ep.site, "getusersitepackages",
lambda: str(tmp_path / "empty"))
monkeypatch.delenv("LD_LIBRARY_PATH", raising=False)
env = ep._av1an_env()
assert ".local/lib" in env["LD_LIBRARY_PATH"]
assert "PYTHONPATH" not in env
def importlib_pyside_available():
import importlib.util
return importlib.util.find_spec("PySide6") is not None
class TestRebuildButtonAlwaysEnabled:
@pytest.mark.skipif(
not importlib_pyside_available(),
reason="requires a real PySide6 (stub mode cannot instantiate Qt)",
)
def test_enabled_before_probe(self):
os.environ.setdefault("QT_QPA_PLATFORM", "offscreen")
from PySide6.QtWidgets import QApplication
from opentranscode.ui_window import OpenCodecMaster
app = QApplication.instance() or QApplication([])
window = OpenCodecMaster()
try:
assert window.btn_rebuild.isEnabled(), (
"REBUILD FROM GIT must be usable without a successful probe"
)
assert any(
"Hybrid (GPU + CPU)" in window.engine_combo.itemText(i)
for i in range(window.engine_combo.count())
)
finally:
window.close()
class TestLauncherParity:
def test_launcher_has_dep_tree(self, opentranscode_module):
assert hasattr(opentranscode_module, "build_dep_plan")
assert hasattr(opentranscode_module, "BUILD_DEPS_BY_FAMILY")
def test_launcher_worker_builds_bestsource(self, opentranscode_module):
assert hasattr(opentranscode_module.SourceBuildWorker,
"_build_bestsource")
def test_launcher_plan_matches_package(self):
from opentranscode.source_builder import build_dep_plan as pkg_plan
plan_l = build_dep_plan(_profile("arch"))
plan_p = pkg_plan(_profile("arch"))
assert plan_l.packages == plan_p.packages
assert plan_l.install_cmd == plan_p.install_cmd