191 lines
7.5 KiB
Markdown
Executable File
191 lines
7.5 KiB
Markdown
Executable File
# OpenTranscode v4.0.0 — Production Release
|
||
|
||
**v4.0.0 resolves the "works up until near the end, never saves chunks into a
|
||
full file" bug** that affected phone-recorded MP4s with sparse keyframes.
|
||
|
||
## The v4.0.0 fix (in detail)
|
||
|
||
### The bug
|
||
|
||
When no VapourSynth source plugins are installed (the common case on most
|
||
distros — `lsmash`, `ffms2`, `bestsource` are all separate packages),
|
||
av1an auto-selects the **Hybrid** chunk method. Hybrid does:
|
||
|
||
1. `ffmpeg -c copy -f segment` to split the source at scene boundaries
|
||
2. Re-decode each segment to y4m via a second ffmpeg invocation
|
||
3. Pipe the y4m to the encoder (SvtAv1EncApp / vpxenc / x265)
|
||
|
||
Phone-recorded MP4s (the `20190707_112725.11b774bacde3.mp4` files in
|
||
the production log) only have I-frames every 5–10 seconds. Scene
|
||
boundaries detected by av1an's `av_scenechange` rarely align with those
|
||
sparse keyframes. The segment muxer can only split on keyframes, so the
|
||
segment for scene N actually starts somewhere inside scene N-1's GOP.
|
||
|
||
The result: the decoder has no I-frame reference → errors with
|
||
`[h264 @ 0x...] error while decoding MB 35 25` → the y4m pipe breaks →
|
||
the encoder reads EOF mid-frame → `Failed to read y4m frame delimiter.
|
||
Read broken. EOF: 1` → every chunk fails after 3 retries → no chunks
|
||
to concat → **no output file**.
|
||
|
||
The previous ffmpeg fallback rescued the file, but it was slow (single-pass,
|
||
no chunk-parallel) and the diagnostic was misleading ("concat failure"
|
||
when it was actually a chunk-extraction failure).
|
||
|
||
### The fix (three parts)
|
||
|
||
1. **`_encode_one` now accepts a `chunk_method` parameter**. When av1an
|
||
fails with the y4m break pattern, it recursively retries with
|
||
`--chunk-method select`. Select uses VapourSynth's `select()` filter
|
||
to extract frames one-by-one — slower than Hybrid but reliable for
|
||
any file VapourSynth can open. This is faster than the ffmpeg
|
||
fallback (chunk-parallel still works) and produces identical-quality
|
||
output (same encoder, same params).
|
||
|
||
2. **The working chunk_method is cached** in
|
||
`env.av1an_flags["chunk_method_override"]` so subsequent files skip
|
||
the wasted first attempt.
|
||
|
||
3. **`env_probe` now probes for VapourSynth source plugins** via
|
||
`_probe_vs_source_plugins()`. When NONE are found, it pre-sets
|
||
`chunk_method_override = "select"` to avoid the wasted first attempt
|
||
entirely. The probe checks `~/.local/lib/vapoursynth/`,
|
||
`/usr/lib/vapoursynth/`, `/usr/local/lib/vapoursynth/`, and the
|
||
Debian multiarch path.
|
||
|
||
### Also fixed
|
||
|
||
The "SUMMARY block + non-zero exit" diagnostic previously misdiagnosed
|
||
y4m break failures as "concat failure" (because SVT-AV1 prints a
|
||
SUMMARY block per-chunk before the pipe breaks). The check is now
|
||
guarded by `"Failed to read y4m frame delimiter" not in stderr_full` so
|
||
it only fires for true concat failures.
|
||
|
||
### New CLI flag
|
||
|
||
```bash
|
||
opentranscode --chunk-method {auto,select,hybrid,segment,ffms2,lsmash,bestsource,dgdecnv}
|
||
```
|
||
|
||
Lets the user force a specific chunk method. Useful for debugging or
|
||
for environments where the probe picks the wrong default. `auto` clears
|
||
any override the probe set.
|
||
|
||
## Quick start
|
||
|
||
### Option A: Install as a package (recommended)
|
||
|
||
```bash
|
||
cd /path/to/this/directory
|
||
pip install -e . # editable install (dev)
|
||
# OR
|
||
pip install -e ".[dev]" # with pytest + build tools
|
||
|
||
# Now you can run it three ways:
|
||
opentranscode # console entry point
|
||
python -m opentranscode # module entry point
|
||
python -m opentranscode --version # → opentranscode 4.0.0
|
||
```
|
||
|
||
### Option B: Run the single-file script (backwards compat)
|
||
|
||
```bash
|
||
python open-transcode.py # the single-file version still works
|
||
```
|
||
|
||
### Option C: Verify your environment without encoding
|
||
|
||
```bash
|
||
opentranscode --dry-run # probe env + smoke test, no GUI, no encode
|
||
opentranscode --dry-run --chunk-method select # preview a forced chunk method
|
||
opentranscode --verify-only /path/to/existing_output.mkv # re-verify an output
|
||
```
|
||
|
||
## Files
|
||
|
||
| Path | Description |
|
||
|------|-------------|
|
||
| `opentranscode/` | **Package** — 16 modules. Importable as `import opentranscode`. |
|
||
| `open-transcode.py` | Single-file script (~6,100 lines). Kept for backwards compat + as the test target for the mocked tests. Mirrors the package behavior. |
|
||
| `pyproject.toml` | PEP 621 build config. Entry point: `opentranscode = opentranscode.__main__:main`. Ready for `pip install -e .` and `python -m build`. |
|
||
| `tests/` | 103 tests across 12 files. All pass in ~10 seconds. |
|
||
| `pytest.ini` | pytest config (also in pyproject.toml). |
|
||
| `README.md` | This file. |
|
||
| `logs/av1an.log.2026-07-13` | The production log that revealed the y4m break bug. Kept for reference. |
|
||
|
||
## Test suite
|
||
|
||
```bash
|
||
cd /path/to/this/directory
|
||
python -m pytest tests/ -v
|
||
|
||
# 103 tests, ~10 seconds:
|
||
# 25 mocked unit tests (smoke test, encoder pipeline, audio loudnorm, etc.)
|
||
# 12 real-ffmpeg e2e tests — requires ffmpeg + ffprobe
|
||
# 36 package-structure tests
|
||
# 13 chunk-method retry tests
|
||
# 1 chunk-method e2e recovery test
|
||
```
|
||
|
||
The e2e tests generate a real 2-second test video with ffmpeg, run the
|
||
full EncoderWorker pipeline on it (AV1→MKV, x265→MKV, VP9→WebM), and
|
||
verify the output file exists, is non-empty, has the correct codec, and
|
||
has the expected duration. The chunk-method e2e test additionally
|
||
simulates the y4m break pattern and verifies the retry-with-select
|
||
produces a valid output file.
|
||
|
||
## Verification (run these to confirm v4.0.0 works)
|
||
|
||
```bash
|
||
# 1. Package imports cleanly
|
||
python -c "import opentranscode; print(opentranscode.__version__)" # → 4.0.0
|
||
|
||
# 2. CLI works
|
||
python -m opentranscode --version # → opentranscode 4.0.0
|
||
python -m opentranscode --help # → usage (includes --chunk-method)
|
||
python -m opentranscode --dry-run # → env probe report (shows VS plugins + chunk method)
|
||
|
||
# 3. All tests pass
|
||
python -m pytest tests/ -q # → 103 passed in ~10s
|
||
|
||
# 4. Install works
|
||
pip install -e . # → installs opentranscode + PySide6
|
||
opentranscode --version # → opentranscode 4.0.0
|
||
|
||
# 5. Single-file script still works (backwards compat)
|
||
python open-transcode.py # → launches GUI (if PySide6 + display)
|
||
```
|
||
|
||
## Publishing to PyPI
|
||
|
||
v4.0.0 is the production release. To publish:
|
||
|
||
```bash
|
||
python -m build # produces dist/opentranscode-4.0.0.tar.gz + .whl
|
||
twine upload dist/* # publishes to PyPI
|
||
```
|
||
|
||
## The production log that revealed the bug
|
||
|
||
The file `logs/av1an.log.2026-07-13` is the actual av1an log from the
|
||
user's production run that revealed the y4m break bug. Key markers:
|
||
|
||
```
|
||
INFO encode_file: Input: 1920x1080 @ 29.763 fps, YUVJ420P, SDR
|
||
INFO encode_file: scenecut: found 8 scene(s) [with extra_splits: 16 scene(s)]
|
||
DEBUG encode_file: Segmenting video
|
||
DEBUG encode_file: Segment done
|
||
INFO encode_file:
|
||
Encoding Failed to read y4m frame delimiter. Read broken. EOF: 1
|
||
[h264 @ 0x55da365b30c0] error while decoding MB 35 25
|
||
WARN encode_chunk: Encoder failed (on chunk 11):
|
||
Encoding Failed to read y4m frame delimiter. Read broken. EOF: 1
|
||
SUMMARY -----------------------------------------------------------------
|
||
Average Speed: 2.501 fps
|
||
ERROR av1an_core::broker: [chunk 4] encoder failed 3 times, shutting down worker
|
||
```
|
||
|
||
The smoke test (line 1 of the log) succeeds because it uses
|
||
`chunk_method: Select` — but the real encode (line 4 onward) uses
|
||
`chunk_method: Hybrid` (av1an's auto-selection when no VS plugins are
|
||
installed) and fails on every chunk.
|