973 lines
36 KiB
Markdown
Executable File
973 lines
36 KiB
Markdown
Executable File
# OpenTranscode v4.4.3 — Production Release
|
||
|
||
**v4.4.3 fixes the crash + adds a UI toggle for av1an.**
|
||
|
||
## What changed
|
||
|
||
### 1. Fixed the AttributeError crash
|
||
|
||
v4.4.2's launcher script crashed on START with:
|
||
```
|
||
AttributeError: 'EncoderWorker' object has no attribute 'verbose'
|
||
```
|
||
|
||
The launcher script's `EncoderWorker.__init__` never set `self.verbose`,
|
||
so when `_check_disk_space` referenced it, the worker thread crashed.
|
||
Fixed — `self.verbose` is now set from `env.av1an_flags["verbose"]` in
|
||
both the package and the launcher script.
|
||
|
||
### 2. UI toggle for av1an
|
||
|
||
Instead of the `--use-av1an` CLI flag (which was weird to require), the
|
||
GUI now has an **"av1an (chunk-parallel)"** checkbox in the options row
|
||
next to "Force (skip validation)" and "Delete source after verify".
|
||
|
||
- **Default OFF** (unchecked) = ffmpeg-only, the reliable default
|
||
- **ON** = av1an chunk-parallel, for users with a working VapourSynth setup
|
||
|
||
The CLI flag `--use-av1an` still works (for scripting), but the UI
|
||
toggle takes precedence when set.
|
||
|
||
### 3. Fixed misleading "single-file script" terminology
|
||
|
||
The docs called `open-transcode.py` "the single-file script" — but it's
|
||
a 6,500-line launcher that mirrors the `opentranscode/` package (16
|
||
modules). Calling it "single-file" was misleading. Now consistently
|
||
referred to as "the launcher script" throughout docs and comments.
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.4.2 — Production Release
|
||
|
||
**v4.4.2 kills the live tail spam and the confusing "FAIL then OK" double-status.**
|
||
|
||
## What changed
|
||
|
||
Two fixes for issues found in v4.4.1 testing:
|
||
|
||
### 1. Live tail gated behind `--verbose` (launcher script)
|
||
|
||
The package had the live tail (`│ Encoding: 1373/1376 Frames @ 51.70 fps...`)
|
||
gated behind `--verbose` since v4.1.1. The launcher script (which the
|
||
tests run against) never got that gate — it was emitting every frame of
|
||
SVT-AV1's progress bar to the log in quiet mode. Now gated.
|
||
|
||
### 2. "FAIL: av1an exit code 1" moved to `_vlog`
|
||
|
||
When av1an fails but the ffmpeg fallback succeeds, the user was seeing:
|
||
```
|
||
[22/180] file.mkv — FAIL: av1an exit code 1
|
||
[22/180] file.mkv — OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
Two status lines for one file — one FAIL, one OK. Confusing. Now the
|
||
av1an failure is logged to `_vlog` (verbose only), and the user only
|
||
sees the final outcome:
|
||
```
|
||
[22/180] file.mkv — OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
If both av1an AND ffmpeg fail, the user sees:
|
||
```
|
||
[22/180] file.mkv — FAIL: av1an + ffmpeg both failed
|
||
```
|
||
|
||
One line. Clear outcome.
|
||
|
||
### Note on `--use-av1an`
|
||
|
||
The log that revealed these issues showed av1an running — which means
|
||
`--use-av1an` was passed. The default (ffmpeg-only) doesn't hit the
|
||
av1an concat bug at all. If you're seeing av1an failures, drop the
|
||
`--use-av1an` flag and use the default ffmpeg path.
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.4.1 — Production Release
|
||
|
||
**v4.4.1 shuts up. Two lines per file: start, finish. Nothing else.**
|
||
|
||
## What changed
|
||
|
||
v4.4.0 tried to be helpful with heartbeats and disk-space warnings.
|
||
The user told us to shut up. So we did.
|
||
|
||
**Default log output is now exactly:**
|
||
|
||
```
|
||
Found 180 file(s) to process.
|
||
[1/180] filename.mkv
|
||
[1/180] filename.mkv — OK: 1.6MB -> 1.3MB (81%)
|
||
[2/180] already_done.mkv — SKIP (already av1/opus)
|
||
[3/180] next.mkv
|
||
[3/180] next.mkv — OK: 2.4MB -> 1.8MB (75%)
|
||
...
|
||
QUEUE COMPLETE. Success: 178, Failed: 0, Skipped: 2.
|
||
```
|
||
|
||
That's it. Two lines per file that encodes (start + finish). One line
|
||
per file that's skipped or fails. No heartbeats. No disk-space
|
||
warnings. No per-frame chatter. No "still encoding" messages.
|
||
|
||
**What's gated behind `--verbose`:**
|
||
- Heartbeat (`... 30s elapsed`) — was always-on since v4.3.0, now opt-in
|
||
- Disk-space warnings (`WARN: low disk space on output...`) — now opt-in
|
||
- Live tail of ffmpeg/av1an stderr — already opt-in since v4.1.1
|
||
- CMD: lines, DIAGNOSIS blocks, resolution map — already opt-in since v4.2.1
|
||
- Pre-flight validation table — already opt-in since v4.2.1
|
||
|
||
**If a 10-hour encode looks hung without the heartbeat**, run with
|
||
`--verbose` to see it. Or just trust that it's working — ffmpeg
|
||
doesn't hang, it just takes a long time.
|
||
|
||
```bash
|
||
opentranscode # quiet: start + finish only
|
||
opentranscode --verbose # full detail: heartbeats, disk warnings, CMD, etc.
|
||
```
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.4.0 — Production Release
|
||
|
||
**v4.4.0 adds massive-file support (30GB+ BluRay rips) and reduces log noise.**
|
||
|
||
## The v4.4.0 changes
|
||
|
||
### 1. Massive-file support
|
||
|
||
Three changes to prevent failures on 30GB+ source files:
|
||
|
||
**Per-file timeout raised from 2h to 24h (configurable via `--timeout`)**
|
||
|
||
A 30GB 1080p BluRay rip at SVT-AV1 preset 6 (~5-10 fps) on a 2-hour
|
||
movie takes 4-10 hours. The old 7200s (2h) timeout killed massive-file
|
||
encodes partway through. New default is 86400s (24h). Configurable:
|
||
|
||
```bash
|
||
opentranscode --timeout 36000 # 10 hours per file
|
||
opentranscode --timeout 0 # no timeout (not recommended)
|
||
```
|
||
|
||
**5%-of-source integrity check replaced with absolute 1KB minimum**
|
||
|
||
The old check rejected outputs smaller than 5% of the source size. For
|
||
a 50GB high-bitrate BluRay source, 5% = 2.5GB — but valid AV1 at CRF 32
|
||
produces 1-2GB for a 2-hour movie. This caused false "output too small"
|
||
failures on valid encodes of massive files.
|
||
|
||
The new check uses an absolute 1KB minimum (a valid container header
|
||
alone is ~1KB; anything below is definitely corrupt). The real
|
||
integrity gate is the duration check in `_verify_and_finalize` (output
|
||
must be >= 95% of source duration).
|
||
|
||
**Disk space pre-check (warn, not abort)**
|
||
|
||
Before encoding each file > 1 GB, the worker checks free space on the
|
||
output and temp partitions:
|
||
|
||
```
|
||
[N/180] big_movie.mkv
|
||
WARN: low disk space on output (5.2 GB free, source is 30.0 GB) — encode may fail partway through
|
||
... 30s elapsed
|
||
...
|
||
```
|
||
|
||
When scaling, also checks the temp partition (the lossless intermediate
|
||
can be 2-3x source size):
|
||
|
||
```
|
||
WARN: low disk space on temp (20.0 GB free, lossless intermediate may need ~60.0 GB) — consider scaling to a smaller resolution or freeing space
|
||
```
|
||
|
||
This is a WARNING, not an abort — the user might be encoding to a
|
||
different partition or know the output will be smaller. If the encode
|
||
fails partway due to ENOSPC, the existing error handling catches it.
|
||
|
||
### 2. Reduced log noise
|
||
|
||
**Combined status lines** — every file now produces ONE line instead of two:
|
||
|
||
Before (v4.3.0):
|
||
```
|
||
[1/180] filename.mkv
|
||
OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
After (v4.4.0):
|
||
```
|
||
[1/180] filename.mkv — OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
Same for SKIP and FAIL:
|
||
```
|
||
[2/180] already_done.mkv — SKIP (already av1/opus)
|
||
[3/180] broken.mkv — FAIL: not a valid video (ffprobe could not read it)
|
||
```
|
||
|
||
**Disk-space warnings no longer fire for skipped files** — the check
|
||
moved after the skip-existing check, so a folder of 175 already-encoded
|
||
files produces 175 SKIP lines, not 175 SKIP lines buried in 175 disk-
|
||
space warnings.
|
||
|
||
**Heartbeat stays user-facing** (fixed in v4.3.0) — one line per 30
|
||
seconds during long encodes:
|
||
```
|
||
[1/180] big_movie.mkv
|
||
... 30s elapsed
|
||
... 60s elapsed
|
||
... 90s elapsed
|
||
[1/180] big_movie.mkv — OK: 30.2GB -> 2.1GB (7%)
|
||
```
|
||
|
||
The `[N/total] filename` banner appears once at the start of the encode
|
||
(so the user knows what's being processed), heartbeats fire every 30s
|
||
(so the user knows it's alive), and the final OK/FAIL line repeats the
|
||
prefix (so the user can match status to file at a glance).
|
||
|
||
### What v4.4.0 keeps
|
||
|
||
- **Skip-existing detection** (v4.3.0) — probes output with ffprobe,
|
||
skips files whose codec matches
|
||
- **Live progress tail + heartbeat** (v4.1.1, fixed in v4.3.0) —
|
||
av1an/ffmpeg stderr in `--verbose` mode, heartbeat always on
|
||
- **`--use-av1an` opt-in** (v4.2.0) — ffmpeg is the default encode path
|
||
- **`--verbose`** (v4.2.1) — full tech detail (CMD:, DIAGNOSIS, etc.)
|
||
- **`--force-reencode`** (v4.3.0) — bypass skip-existing
|
||
- **`--skip-existing` / `--force-reencode`** (v4.3.0) — defaults to skip
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
opentranscode # default: ffmpeg, skip-existing, 24h timeout
|
||
opentranscode --timeout 36000 # 10h per-file timeout
|
||
opentranscode --force-reencode # re-encode even if output exists
|
||
opentranscode --use-av1an # opt into av1an chunk-parallel
|
||
opentranscode --verbose # full tech detail
|
||
```
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.3.0 — Production Release
|
||
|
||
**v4.3.0 adds skip-existing detection + fixes the v4.2.1 heartbeat regression.**
|
||
|
||
## The v4.3.0 changes
|
||
|
||
### 1. Skip-existing detection (the big feature)
|
||
|
||
When you re-run OpenTranscode on a folder you've already encoded, it now
|
||
**skips files whose output already exists with a matching codec** instead
|
||
of re-encoding them from scratch.
|
||
|
||
Detection strategy (all must pass):
|
||
1. Output file exists at the computed path (`<stem>_archived.<ext>`)
|
||
2. ffprobe can read it (not corrupt)
|
||
3. Video stream `codec_name` matches the selected encoder (`av1`/`vp9`/`hevc`)
|
||
4. Audio stream `codec_name` matches the selected audio profile (`opus`/`vorbis`/`flac`)
|
||
5. If scaling was requested, output resolution matches the target
|
||
|
||
**What is NOT verified** (and why):
|
||
- CRF/preset — these are encoder settings, not reliably stored in container
|
||
metadata. To re-encode at a different CRF with the same codec, use
|
||
`--force-reencode`.
|
||
- Audio bitrate — varies per file based on loudness normalization.
|
||
|
||
**CLI flags:**
|
||
```bash
|
||
opentranscode # skip-existing ON (default)
|
||
opentranscode --force-reencode # re-encode everything, even if output exists
|
||
```
|
||
|
||
**Log output in quiet mode:**
|
||
```
|
||
[1/180] already_encoded.mkv
|
||
SKIP: already encoded (av1/opus)
|
||
[2/180] new_file.mkv
|
||
... 30s elapsed
|
||
OK: 1.6MB -> 1.3MB (81%)
|
||
...
|
||
QUEUE COMPLETE. Success: 45, Failed: 0, Skipped: 135.
|
||
```
|
||
|
||
Skipped files are counted separately — they're NOT successes (nothing
|
||
was encoded) and NOT failures (the output exists and matches).
|
||
|
||
### 2. Heartbeat regression fix
|
||
|
||
v4.2.1 gated the 30-second heartbeat behind `--verbose`, which caused
|
||
the "hangs on first transcode, forever timer" symptom in quiet mode.
|
||
The user saw:
|
||
|
||
```
|
||
[1/180] filename.mkv
|
||
```
|
||
|
||
...and nothing else for the entire encode duration. Looked exactly like
|
||
a hang.
|
||
|
||
**v4.3.0 fix**: the heartbeat is **always user-facing** (quiet or
|
||
verbose). The live tail of `frame= 67 fps= 12...` stays gated behind
|
||
`--verbose` (that IS noisy), but the heartbeat is the "is it alive?"
|
||
signal — one line per 30 seconds, not "absurd amounts of useless info."
|
||
|
||
Quiet mode now shows:
|
||
```
|
||
[1/180] filename.mkv
|
||
... 30s elapsed
|
||
... 60s elapsed
|
||
... 90s elapsed
|
||
OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
One line per 30 seconds + the success line. Not noisy. Not silent.
|
||
|
||
### What the skip detection does NOT cover
|
||
|
||
The detection is conservative — if in doubt, re-encode:
|
||
- No ffprobe available → re-encode (can't verify codec)
|
||
- Output file exists but ffprobe can't read it → re-encode (treat as corrupt)
|
||
- Video codec matches but audio doesn't → re-encode
|
||
- Codec matches but resolution doesn't (when scaling requested) → re-encode
|
||
|
||
This means the only time a file is skipped is when we're **confident**
|
||
the existing output matches your current encoder selection. If you
|
||
switch from AV1 to x265, all files re-encode. If you switch from Opus
|
||
to Vorbis audio, all files re-encode. If you change the target
|
||
resolution, all files re-encode.
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.2.1 — Production Release
|
||
|
||
**v4.2.1 makes the log quiet by default. Tech detail is gated behind
|
||
`--verbose`.**
|
||
|
||
*(Note: v4.3.0 fixes the heartbeat regression introduced here — the
|
||
heartbeat is now always user-facing, not gated behind `--verbose`.)*
|
||
|
||
## The v4.2.1 change
|
||
|
||
### Why
|
||
|
||
v4.2.0's log was filling up with thousands of lines of useless info
|
||
per file: the `CMD:` banner, ffmpeg's `frame= 67 fps= 12 q=32 ...`
|
||
progress chatter, the resolution map, the pre-flight validation table,
|
||
the DIAGNOSIS blocks, the 30s heartbeat, the RETRY messages. For a
|
||
180-file run that's thousands of lines nobody reads.
|
||
|
||
The user just needs: did it encode or not?
|
||
|
||
### What changed
|
||
|
||
**Default log output is now quiet.** Per file, you see exactly two
|
||
lines:
|
||
|
||
```
|
||
[1/180] filename.mkv
|
||
OK: 1.6MB -> 1.3MB (81%)
|
||
```
|
||
|
||
Or on failure:
|
||
|
||
```
|
||
[2/180] next.mkv
|
||
FAIL: ffmpeg exit code 1
|
||
```
|
||
|
||
Final summary stays:
|
||
|
||
```
|
||
QUEUE COMPLETE. Success: 178, Failed: 2.
|
||
```
|
||
|
||
### What's gated behind `--verbose`
|
||
|
||
- `Found N file(s) to process.` + `Temp dir:` banner
|
||
- `─── FILE RESOLUTION MAP ───` + per-file resolution lines
|
||
- `─── PRE-FLIGHT VALIDATION ───` + valid/invalid counts
|
||
- ` Mode: ffmpeg (...)`, ` Chunking: select ...`
|
||
- ` CMD: av1an -i ... --workers 4 --chunk-method select ...`
|
||
- ` Source: 1920x1080 -> Output: 1920x1080`
|
||
- Live tail of av1an/ffmpeg stderr (`│ frame= 67 fps= 12 ...`)
|
||
- `... still encoding (30s elapsed)` heartbeat (every 30s)
|
||
- `DIAGNOSIS: ...` blocks (root-cause analysis for av1an failures)
|
||
- ` ─── av1an stderr (last 25 lines) ───` dumps
|
||
- ` RETRY: ...` / ` RETRY OK: ...` / ` RETRY FAIL: ...` messages
|
||
- ` Pre-scale OK (... MB intermediate)`
|
||
- ` SUBS:` mux messages
|
||
- `CLEANUP: Swept N residual temp file(s)...`
|
||
- ` File type: ...` (yt-dlp-download detection)
|
||
|
||
### What stays user-facing (quiet mode)
|
||
|
||
- `Found N file(s) to process.` — single banner line
|
||
- `[N/total] filename` — per-file header
|
||
- ` OK: XMB -> YMB (Z%)` — success
|
||
- ` FAIL: <short reason>` — failure (one line, not a multi-line dump)
|
||
- ` SKIP: <short reason>` — skipped file (no video stream, too short, etc.)
|
||
- ` STOP: skipping remaining files (<reason>)` — when queue auto-aborts
|
||
- `CLEANED: Removed N source file(s).` — after delete-source transcodes
|
||
- `QUEUE COMPLETE. Success: N, Failed: M.` — final summary
|
||
- `ABORT: All N file(s) are invalid. Aborting queue.` — when pre-flight finds zero valid files
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
opentranscode # quiet (default)
|
||
opentranscode --verbose # full tech detail
|
||
opentranscode --use-av1an # opt into av1an chunk-parallel
|
||
opentranscode --use-av1an --verbose # full tech detail + av1an
|
||
```
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.2.0 — Production Release
|
||
|
||
**v4.2.0 makes ffmpeg the default encode path. av1an chunk-parallel is
|
||
now opt-in via `--use-av1an`.**
|
||
|
||
## The v4.2.0 change
|
||
|
||
### Why
|
||
|
||
v4.1.x tried to make av1an chunk-parallel work reliably. It doesn't,
|
||
across distros. Failure modes observed in production:
|
||
|
||
- **`Unprocessed tokens: --threads`** (v4.1.0/v4.1.1) — `SvtAv1EncApp`
|
||
CLI doesn't accept `--threads`, only `--lp`. Every chunk failed 3x.
|
||
- **No av1an output for 270+ seconds** (v4.1.2) — av1an wedged at
|
||
startup or its stderr is buffered and won't flush until exit.
|
||
- **y4m pipe breaks** (v4.0.0) — Hybrid chunk method fails on phone-
|
||
recorded MP4s with sparse keyframes.
|
||
- **VapourSynth plugin issues** — `lsmash`/`ffms2`/`bestsource` are
|
||
separate packages that most distros don't install by default.
|
||
|
||
Each fix uncovered a new failure mode. The pattern is clear: av1an is
|
||
too fragile to be the default.
|
||
|
||
### What changed
|
||
|
||
**Default encode path is now ffmpeg-only.** The av1an pre-flight smoke
|
||
test is skipped entirely. `_on_run_clicked` sets `use_ffmpeg_fallback =
|
||
True` directly, and `EncoderWorker` runs the `_ffmpeg_fallback_encode`
|
||
path for every file.
|
||
|
||
This means:
|
||
- **No VapourSynth dependency** — ffmpeg invokes `libsvtav1` as a
|
||
library, no `SvtAv1EncApp` subprocess, no `libvapoursynth-script.so`.
|
||
- **No chunk-method selection** — single-pass ffmpeg per file.
|
||
- **No `--threads` CLI quirks** — `-threads` is a valid ffmpeg/libsvtav1
|
||
library option.
|
||
- **Immediate progress output** — ffmpeg's progress bar flushes to
|
||
stdout line-by-line, picked up by the live tail (v4.1.1).
|
||
- **Slower than chunk-parallel av1an** — single-pass, no scene-split
|
||
parallelism. But it actually completes, which is the only thing that
|
||
matters.
|
||
|
||
### The `--use-av1an` flag
|
||
|
||
Users who specifically want av1an chunk-parallel (e.g. they have a
|
||
known-good VapourSynth + lsmash/ffms2 setup) can opt in:
|
||
|
||
```bash
|
||
opentranscode --use-av1an
|
||
```
|
||
|
||
When set, the full av1an pre-flight + smoke test runs as before, and
|
||
the encode uses the chunk-parallel path. When not set (default), the
|
||
smoke test is skipped and ffmpeg is used directly.
|
||
|
||
### What v4.2.0 keeps
|
||
|
||
- **Live progress tail** (v4.1.1) — ffmpeg's stdout/stderr emits to the
|
||
GUI log as it arrives, with `\r` progress bar handling.
|
||
- **30-second heartbeat** (v4.1.1) — `... still encoding (Xs elapsed)`.
|
||
- **`--max-workers` / `--threads-per-worker` CLI flags** (v4.1.0) —
|
||
still wired through `env.av1an_flags`. `--threads-per-worker` is now
|
||
actually useful in the ffmpeg path (libsvtav1 accepts `-threads`).
|
||
- **`--chunk-method`** (v4.0.0) — still honored when `--use-av1an` is
|
||
set. No-op in the default ffmpeg path.
|
||
|
||
### What the ffmpeg command looks like
|
||
|
||
```
|
||
ffmpeg -i input.mkv \
|
||
-c:v libsvtav1 -preset 6 -crf 32 -pix_fmt yuv420p10le -g 240 \
|
||
-c:a libopus -b:a 64k \
|
||
-y output.mkv
|
||
```
|
||
|
||
Simple, reliable, no surprises. The same command works on any distro
|
||
with `ffmpeg` compiled against `libsvtav1` (which is the default on
|
||
Arch, Debian, Ubuntu, Fedora).
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.1.2 — Production Release
|
||
|
||
**v4.1.2 reverts the `--threads N` injection that broke av1an in v4.1.0/v4.1.1.**
|
||
|
||
*(Note: v4.2.0 makes ffmpeg the default, so this av1an issue is no
|
||
longer reachable unless you opt in with `--use-av1an`.)*
|
||
|
||
## The v4.1.2 fix (the actual root cause)
|
||
|
||
### What was broken
|
||
|
||
v4.1.0 added "intelligent chunking" that injected `--threads N` into
|
||
av1an's `--video-params` string, intending to cap each per-chunk encoder
|
||
instance to N threads. The theory was sound — without a cap, SVT-AV1's
|
||
default `--threads 0` means "use all logical cores," so 13 chunk-parallel
|
||
workers × 28 threads = ~364 threads on a 28-thread Xeon → kernel
|
||
scheduler drowned → hard lock.
|
||
|
||
The implementation was wrong. av1an invokes `SvtAv1EncApp` (the
|
||
standalone SVT-AV1 CLI binary) per-chunk, not the libsvtav1 library.
|
||
**`SvtAv1EncApp` does not accept `--threads`** — it uses `--lp N`
|
||
(logical processors) instead. The result, visible in the user's log:
|
||
|
||
```
|
||
Svt[info]: -------------------------------------------
|
||
Svt[info]: SVT [version]: SVT-AV1 Encoder Lib v4.2.0
|
||
Svt[info]: -------------------------------------------
|
||
Unprocessed tokens: --threads
|
||
Unprocessed arguments: 6
|
||
Error in configuration, could not begin encoding!
|
||
```
|
||
|
||
Every chunk failed 3 times with this error → av1an exited with code 1
|
||
→ no output file → OpenTranscode fell back to ffmpeg → ffmpeg succeeded
|
||
(because it invokes libsvtav1 as a library, where `-threads` IS valid).
|
||
But this happened *per file*, making every encode go through the slow
|
||
single-pass ffmpeg fallback path instead of chunk-parallel av1an.
|
||
|
||
### The fix
|
||
|
||
**Don't inject `--threads` into av1an's `--video-params` at all.**
|
||
Revert `params_fn` to its v4.0.0 signature `(crf, preset) -> str` with
|
||
no threads parameter. Thread capping now happens via:
|
||
|
||
1. **av1an's `--workers` flag** (chunk-parallel count) — this is what
|
||
`_compute_intelligent_worker_count` actually controls. Fewer workers
|
||
= fewer concurrent SVT-AV1 processes = less thread pressure.
|
||
2. **`-threads` in the ffmpeg fallback path** — `_svtav1_ffmpeg_args` /
|
||
`_vp9_ffmpeg_args` / `_x265_ffmpeg_args` already accept `-threads`,
|
||
passed to libsvtav1/libvpx/libx265 as library options (where it
|
||
works).
|
||
|
||
The intelligent worker-count logic from v4.1.0 is **kept** — it still
|
||
caps `--workers` to `physical_cores - 1` and reserves 1 logical thread
|
||
for OS/UI. But the per-chunk thread cap that broke SVT-AV1 is gone.
|
||
|
||
### What v4.1.2 keeps from v4.1.0/v4.1.1
|
||
|
||
- **Intelligent `--workers` count** — `_compute_intelligent_worker_count`
|
||
still caps chunk-parallel workers based on CPU topology.
|
||
- **Live progress tail** (v4.1.1) — av1an's stdout/stderr now emits to
|
||
the GUI log as it arrives, with `\r` progress bar handling.
|
||
- **30-second heartbeat** (v4.1.1) — `... still encoding (Xs elapsed)`
|
||
so you always know the encode is alive.
|
||
- **`--max-workers` / `--threads-per-worker` CLI flags** — still wired
|
||
through `env.av1an_flags`. `--max-workers` controls av1an's
|
||
`--workers`. `--threads-per-worker` is currently a no-op in the
|
||
av1an path (kept for future use if/when SVT-AV1's `--lp` flag is
|
||
wired in correctly).
|
||
|
||
### What the av1an command looks like now
|
||
|
||
```
|
||
/usr/bin/av1an -i input.mkv --workers 4 --chunk-method select \
|
||
--encoder svt-av1 \
|
||
--video-params --preset 6 --crf 32 --keyint 240 \
|
||
--audio-params -c:a libopus -b:a 64k \
|
||
--concat mkvmerge -o output.mkv
|
||
```
|
||
|
||
No `--threads 6` in `--video-params` — SvtAv1EncApp accepts this cleanly.
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.1.1 — Production Release
|
||
|
||
**v4.1.1 adds live progress tail + heartbeat** so you can see av1an is
|
||
working during long encodes. Also bumps threads-per-worker from 4 to 6
|
||
for better SVT-AV1 per-chunk throughput.
|
||
|
||
*(Note: the threads-per-worker bump in v4.1.1 was reverted in v4.1.2 —
|
||
see above.)*
|
||
|
||
## The v4.1.1 fix (in detail)
|
||
|
||
### The bug
|
||
|
||
v4.1.0 capped threads per encoder instance at 4 to prevent the
|
||
thread-oversubscription hard-lock. That worked (no more hard locks),
|
||
but it made each chunk ~7x slower than v4.0.0's "grab all 28 threads"
|
||
behavior. The result: file 1 took 10+ minutes, and the user saw
|
||
**nothing** in the GUI log the entire time.
|
||
|
||
The "nothing" was the real killer. The drainer threads in
|
||
`_run_with_stop_check` read av1an's stdout/stderr into StringIO
|
||
buffers but **only emitted them to the GUI log on process exit**. So
|
||
during a 10-minute encode, the user stared at:
|
||
|
||
```
|
||
[1/46] Encoding: video1.mp4
|
||
Source: 1920x1080 -> Output: 1920x1080
|
||
Chunking: select (av1an default if no override)
|
||
CMD: av1an -i ... --workers 4 --chunk-method select ...
|
||
```
|
||
|
||
...and nothing else for 10 minutes. Looked identical to a wedged
|
||
process. User assumed it was "borked" and killed it.
|
||
|
||
### The fix (three parts)
|
||
|
||
1. **Live tail** — the drainer now emits each line of av1an's
|
||
stdout/stderr to the GUI log **as it arrives**, prefixed with `│ `
|
||
to distinguish from orchestrator messages. Handles both `\n` (log
|
||
lines like "scenecut: found 8 scene(s)") and `\r` (progress bar
|
||
updates like "Encoding 45%") as line boundaries, so av1an's progress
|
||
bar renders correctly in real-time.
|
||
|
||
2. **30-second heartbeat** — the poll loop emits
|
||
`... still encoding (Xs elapsed)` every 30 seconds, so even if
|
||
av1an isn't producing line-delimited output (e.g. during a long
|
||
SVT-AV1 encode that only updates a `\r` progress bar), the user
|
||
knows the process is alive.
|
||
|
||
3. **`IDEAL_THREADS_PER_WORKER` bumped from 4 to 6** — SVT-AV1 with
|
||
only 4 threads was too slow per-chunk. With 6 threads, each chunk
|
||
gets ~50% better throughput while staying under the logical-thread
|
||
budget. On a 28-thread Xeon, this changes the split from 6×4=24
|
||
to 4×6=24 (same total, better per-chunk latency — the first chunk
|
||
completes sooner, so the user sees progress faster).
|
||
|
||
### What the log looks like now
|
||
|
||
```
|
||
[1/46] Encoding: video1.mp4
|
||
Source: 1920x1080 -> Output: 1920x1080
|
||
Chunking: select (av1an default if no override)
|
||
CMD: av1an -i ... --workers 4 --chunk-method select ...
|
||
│ INFO encode_file: Input: 1920x1080 @ 29.763 fps, YUVJ420P, SDR
|
||
│ INFO encode_file: scenecut: found 8 scene(s)
|
||
│ DEBUG encode_file: Segmenting video
|
||
│ DEBUG encode_file: Segment done
|
||
│ INFO encode_chunk: Encoding chunk 1
|
||
... still encoding (30s elapsed)
|
||
│ INFO encode_chunk: Encoding chunk 2
|
||
... still encoding (60s elapsed)
|
||
│ SUMMARY ------------------------------------------
|
||
│ Average Speed: 4.231 fps
|
||
SUCCESS: video1.mp4 (245.3MB -> 28.7MB, 12%)
|
||
```
|
||
|
||
### Worker-count math, updated for v4.1.1
|
||
|
||
| Machine | budget | workers | threads | active | reserved |
|
||
|----------------------------|--------|---------|---------|--------|----------|
|
||
| 4-core / 8-thread laptop | 7 | 1 | 7 | 7 | 1 |
|
||
| 8-core / 16-thread desktop | 15 | 2 | 7 | 14 | 1 |
|
||
| **14-core / 28-thread Xeon** | 27 | **4** | **6** | **24** | **4** |
|
||
| 32-core / 64-thread EPYC | 63 | 10 | 6 | 60 | 4 |
|
||
| 1-core / 2-thread VM | 1 | 1 | 1 | 1 | 1 |
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.1.0 — Production Release
|
||
|
||
**v4.1.0 adds intelligent chunking** to prevent the thread-oversubscription
|
||
hard-lock that v4.0.0 hit on high-core-count machines.
|
||
|
||
## The v4.1.0 fix (in detail)
|
||
|
||
### The bug
|
||
|
||
v4.0.0 introduced the `--chunk-method select` auto-override for phone-
|
||
recorded MP4s with sparse keyframes (see the v4.0.0 section below for
|
||
that fix). But `select` mode keeps the encode pipeline tighter than
|
||
the previous `hybrid` default — chunks warm up faster, more encoder
|
||
instances hit full tilt at the same instant.
|
||
|
||
Meanwhile, `EncoderWorker.run()` was still computing
|
||
`worker_count = max(1, physical_cores - 1)` and passing no per-chunk
|
||
thread cap to the encoder. SVT-AV1's default `--threads 0` means "use
|
||
all logical cores," so each chunk-parallel worker spawned an
|
||
SvtAv1EncApp process that grabbed every logical thread.
|
||
|
||
On a 28-thread Xeon (14 physical cores) with 13 chunk-parallel workers,
|
||
the math was: `13 × 28 = ~364 active threads on 28 logical CPUs`. The
|
||
kernel scheduler drowned, I/O wait escalated, and the box hard-locked
|
||
even though no single process was at fault. The 1-second STOP-button
|
||
poll in `_run_with_stop_check` couldn't get scheduled, so even clicking
|
||
STOP didn't recover it.
|
||
|
||
### The fix (three parts)
|
||
|
||
1. **`_compute_intelligent_worker_count()`** in `EncoderWorker` now
|
||
computes `(worker_count, threads_per_worker)` such that
|
||
`worker_count * threads_per_worker <= logical_threads - 1` (one
|
||
logical thread reserved for OS / UI / av1an orchestrator). The
|
||
budget is split using an ideal `4 threads per worker` — the empirical
|
||
sweet spot for SVT-AV1, x265, and vpxenc. Beyond ~6 threads per
|
||
encoder instance you hit memory-bandwidth contention and diminishing
|
||
returns.
|
||
|
||
2. **The resolved `threads_per_worker` is passed to `params_fn`** so each
|
||
`_av1_params` / `_vp9_params` / `_x265_params` appends
|
||
`--threads N` to the encoder's `--video-params` string. Every per-
|
||
chunk encoder instance (SvtAv1EncApp / vpxenc / x265) now respects
|
||
its share of the thread budget instead of grabbing all cores.
|
||
|
||
3. **Two new CLI flags** let the user override the auto math when
|
||
needed:
|
||
|
||
```bash
|
||
opentranscode --max-workers 4 # cap chunk-parallel worker count
|
||
opentranscode --threads-per-worker 6 # per-encoder thread cap
|
||
opentranscode --max-workers 4 --threads-per-worker 6 # full override
|
||
```
|
||
|
||
Both are also visible in `--dry-run` and the GUI env-probe banner.
|
||
|
||
### Worker-count math, by machine
|
||
|
||
| Machine | budget | workers | threads | active | reserved |
|
||
|----------------------------|--------|---------|---------|--------|----------|
|
||
| 4-core / 8-thread laptop | 7 | 1 | 7 | 7 | 1 |
|
||
| 8-core / 16-thread desktop | 15 | 3 | 5 | 15 | 1 |
|
||
| **14-core / 28-thread Xeon** | 27 | **6** | **4** | **24** | **4** |
|
||
| 32-core / 64-thread EPYC | 63 | 15 | 4 | 60 | 4 |
|
||
| 1-core / 2-thread VM | 1 | 1 | 1 | 1 | 1 |
|
||
|
||
The Xeon row is the user's box. v4.0.0 hit 13 × 28 = 364 threads →
|
||
hard lock. v4.1.0 hits 6 × 4 = 24 threads with 4 reserved for OS/UI.
|
||
|
||
### Also new in v4.1.0
|
||
|
||
- `--dry-run` now prints the computed worker math so you can verify
|
||
the thread budget before launching a real encode.
|
||
- The GUI env-probe banner shows the same math on startup.
|
||
- `EncoderWorker.__init__` reads `env.av1an_flags["max_workers"]` and
|
||
`["threads_per_worker"]` as fallback when the explicit constructor
|
||
args aren't supplied — so the CLI flags reach the GUI-spawned
|
||
worker without `ui_window.py` code changes.
|
||
|
||
### Backwards compatibility
|
||
|
||
- `params_fn` accepts an optional third arg `threads=0`. Existing
|
||
callers passing `(crf, preset)` still work because `threads`
|
||
defaults to 0 (the v4.0.0 behavior — no `--threads` flag, encoder
|
||
auto-selects).
|
||
- The `_av1_params` / `_vp9_params` / `_x265_params` functions are
|
||
byte-identical to v4.0.0 when `threads=0`.
|
||
|
||
---
|
||
|
||
# OpenTranscode v4.0.0 — Production Release
|
||
|
||
**v4.0.0 resolves the "works up until near the end, never saves chunks into a
|
||
full file" bug** that affected phone-recorded MP4s with sparse keyframes.
|
||
|
||
## The v4.0.0 fix (in detail)
|
||
|
||
### The bug
|
||
|
||
When no VapourSynth source plugins are installed (the common case on most
|
||
distros — `lsmash`, `ffms2`, `bestsource` are all separate packages),
|
||
av1an auto-selects the **Hybrid** chunk method. Hybrid does:
|
||
|
||
1. `ffmpeg -c copy -f segment` to split the source at scene boundaries
|
||
2. Re-decode each segment to y4m via a second ffmpeg invocation
|
||
3. Pipe the y4m to the encoder (SvtAv1EncApp / vpxenc / x265)
|
||
|
||
Phone-recorded MP4s (the `20190707_112725.11b774bacde3.mp4` files in
|
||
the production log) only have I-frames every 5–10 seconds. Scene
|
||
boundaries detected by av1an's `av_scenechange` rarely align with those
|
||
sparse keyframes. The segment muxer can only split on keyframes, so the
|
||
segment for scene N actually starts somewhere inside scene N-1's GOP.
|
||
|
||
The result: the decoder has no I-frame reference → errors with
|
||
`[h264 @ 0x...] error while decoding MB 35 25` → the y4m pipe breaks →
|
||
the encoder reads EOF mid-frame → `Failed to read y4m frame delimiter.
|
||
Read broken. EOF: 1` → every chunk fails after 3 retries → no chunks
|
||
to concat → **no output file**.
|
||
|
||
The previous ffmpeg fallback rescued the file, but it was slow (single-pass,
|
||
no chunk-parallel) and the diagnostic was misleading ("concat failure"
|
||
when it was actually a chunk-extraction failure).
|
||
|
||
### The fix (three parts)
|
||
|
||
1. **`_encode_one` now accepts a `chunk_method` parameter**. When av1an
|
||
fails with the y4m break pattern, it recursively retries with
|
||
`--chunk-method select`. Select uses VapourSynth's `select()` filter
|
||
to extract frames one-by-one — slower than Hybrid but reliable for
|
||
any file VapourSynth can open. This is faster than the ffmpeg
|
||
fallback (chunk-parallel still works) and produces identical-quality
|
||
output (same encoder, same params).
|
||
|
||
2. **The working chunk_method is cached** in
|
||
`env.av1an_flags["chunk_method_override"]` so subsequent files skip
|
||
the wasted first attempt.
|
||
|
||
3. **`env_probe` now probes for VapourSynth source plugins** via
|
||
`_probe_vs_source_plugins()`. When NONE are found, it pre-sets
|
||
`chunk_method_override = "select"` to avoid the wasted first attempt
|
||
entirely. The probe checks `~/.local/lib/vapoursynth/`,
|
||
`/usr/lib/vapoursynth/`, `/usr/local/lib/vapoursynth/`, and the
|
||
Debian multiarch path.
|
||
|
||
### Also fixed
|
||
|
||
The "SUMMARY block + non-zero exit" diagnostic previously misdiagnosed
|
||
y4m break failures as "concat failure" (because SVT-AV1 prints a
|
||
SUMMARY block per-chunk before the pipe breaks). The check is now
|
||
guarded by `"Failed to read y4m frame delimiter" not in stderr_full` so
|
||
it only fires for true concat failures.
|
||
|
||
### New CLI flag
|
||
|
||
```bash
|
||
opentranscode --chunk-method {auto,select,hybrid,segment,ffms2,lsmash,bestsource,dgdecnv}
|
||
```
|
||
|
||
Lets the user force a specific chunk method. Useful for debugging or
|
||
for environments where the probe picks the wrong default. `auto` clears
|
||
any override the probe set.
|
||
|
||
## Quick start
|
||
|
||
### Option A: Install as a package (recommended)
|
||
|
||
```bash
|
||
cd /path/to/this/directory
|
||
pip install -e . # editable install (dev)
|
||
# OR
|
||
pip install -e ".[dev]" # with pytest + build tools
|
||
|
||
# Now you can run it three ways:
|
||
opentranscode # console entry point
|
||
python -m opentranscode # module entry point
|
||
python -m opentranscode --version # → opentranscode 4.4.3
|
||
```
|
||
|
||
### Option B: Run the launcher script (backwards compat)
|
||
|
||
```bash
|
||
python open-transcode.py # the launcher script still works
|
||
```
|
||
|
||
### Option C: Verify your environment without encoding
|
||
|
||
```bash
|
||
opentranscode --dry-run # probe env + smoke test, no GUI, no encode
|
||
opentranscode --dry-run --chunk-method select # preview a forced chunk method
|
||
opentranscode --dry-run --max-workers 4 # preview worker-count override
|
||
opentranscode --verify-only /path/to/existing_output.mkv # re-verify an output
|
||
```
|
||
|
||
### Option D: Override the intelligent worker math (v4.1.0)
|
||
|
||
If the auto-computed thread budget still hard-locks the box (rare), or
|
||
if you have fast storage and want more parallelism, dial it manually:
|
||
|
||
```bash
|
||
opentranscode --max-workers 4 # 4 chunks in parallel
|
||
opentranscode --threads-per-worker 6 # 6 threads per encoder
|
||
opentranscode --max-workers 8 --threads-per-worker 2 # full override
|
||
```
|
||
|
||
Both flags are stored on `env.av1an_flags` and picked up by the
|
||
GUI-spawned worker — no UI changes needed.
|
||
|
||
## Files
|
||
|
||
| Path | Description |
|
||
|------|-------------|
|
||
| `opentranscode/` | **Package** — 16 modules. Importable as `import opentranscode`. |
|
||
| `open-transcode.py` | Launcher script (~6,500 lines). Kept for backwards compat + as the test target for the mocked tests. Mirrors the package behavior. |
|
||
| `pyproject.toml` | PEP 621 build config. Entry point: `opentranscode = opentranscode.__main__:main`. Ready for `pip install -e .` and `python -m build`. |
|
||
| `tests/` | 103 tests across 12 files. All pass in ~10 seconds. |
|
||
| `pytest.ini` | pytest config (also in pyproject.toml). |
|
||
| `README.md` | This file. |
|
||
| `logs/av1an.log.2026-07-13` | The production log that revealed the y4m break bug. Kept for reference. |
|
||
|
||
## Test suite
|
||
|
||
```bash
|
||
cd /path/to/this/directory
|
||
python -m pytest tests/ -v
|
||
|
||
# 103 tests, ~10 seconds:
|
||
# 25 mocked unit tests (smoke test, encoder pipeline, audio loudnorm, etc.)
|
||
# 12 real-ffmpeg e2e tests — requires ffmpeg + ffprobe
|
||
# 36 package-structure tests
|
||
# 13 chunk-method retry tests
|
||
# 1 chunk-method e2e recovery test
|
||
```
|
||
|
||
The e2e tests generate a real 2-second test video with ffmpeg, run the
|
||
full EncoderWorker pipeline on it (AV1→MKV, x265→MKV, VP9→WebM), and
|
||
verify the output file exists, is non-empty, has the correct codec, and
|
||
has the expected duration. The chunk-method e2e test additionally
|
||
simulates the y4m break pattern and verifies the retry-with-select
|
||
produces a valid output file.
|
||
|
||
## Verification (run these to confirm v4.0.0 works)
|
||
|
||
```bash
|
||
# 1. Package imports cleanly
|
||
python -c "import opentranscode; print(opentranscode.__version__)" # → 4.0.0
|
||
|
||
# 2. CLI works
|
||
python -m opentranscode --version # → opentranscode 4.0.0
|
||
python -m opentranscode --help # → usage (includes --chunk-method)
|
||
python -m opentranscode --dry-run # → env probe report (shows VS plugins + chunk method)
|
||
|
||
# 3. All tests pass
|
||
python -m pytest tests/ -q # → 103 passed in ~10s
|
||
|
||
# 4. Install works
|
||
pip install -e . # → installs opentranscode + PySide6
|
||
opentranscode --version # → opentranscode 4.0.0
|
||
|
||
# 5. Launcher script still works (backwards compat)
|
||
python open-transcode.py # → launches GUI (if PySide6 + display)
|
||
```
|
||
|
||
## Publishing to PyPI
|
||
|
||
v4.0.0 is the production release. To publish:
|
||
|
||
```bash
|
||
python -m build # produces dist/opentranscode-4.0.0.tar.gz + .whl
|
||
twine upload dist/* # publishes to PyPI
|
||
```
|
||
|
||
## The production log that revealed the bug
|
||
|
||
The file `logs/av1an.log.2026-07-13` is the actual av1an log from the
|
||
user's production run that revealed the y4m break bug. Key markers:
|
||
|
||
```
|
||
INFO encode_file: Input: 1920x1080 @ 29.763 fps, YUVJ420P, SDR
|
||
INFO encode_file: scenecut: found 8 scene(s) [with extra_splits: 16 scene(s)]
|
||
DEBUG encode_file: Segmenting video
|
||
DEBUG encode_file: Segment done
|
||
INFO encode_file:
|
||
Encoding Failed to read y4m frame delimiter. Read broken. EOF: 1
|
||
[h264 @ 0x55da365b30c0] error while decoding MB 35 25
|
||
WARN encode_chunk: Encoder failed (on chunk 11):
|
||
Encoding Failed to read y4m frame delimiter. Read broken. EOF: 1
|
||
SUMMARY -----------------------------------------------------------------
|
||
Average Speed: 2.501 fps
|
||
ERROR av1an_core::broker: [chunk 4] encoder failed 3 times, shutting down worker
|
||
```
|
||
|
||
The smoke test (line 1 of the log) succeeds because it uses
|
||
`chunk_method: Select` — but the real encode (line 4 onward) uses
|
||
`chunk_method: Hybrid` (av1an's auto-selection when no VS plugins are
|
||
installed) and fails on every chunk.
|