Skip to content

enh: add Python pytest framework and migrate regression tests#154

Open
mezinster wants to merge 39 commits intospeed47:masterfrom
mezinster:upstream/pytest-framework
Open

enh: add Python pytest framework and migrate regression tests#154
mezinster wants to merge 39 commits intospeed47:masterfrom
mezinster:upstream/pytest-framework

Conversation

@mezinster
Copy link
Copy Markdown

@mezinster mezinster commented Apr 17, 2026

Summary

Adds a Python/pytest regression test framework and migrates all 424 tests from the legacy bash-based regtest/ framework. Bash tests are disabled in regtest/config.txt but the directory and golden files are kept for reference.

Why

  • Declarative test definitions: the new framework uses GoldenTest / SimCD / CreateECC dataclasses to express tests compactly
  • IDE / pytest discoverable: per-test execution via standard pytest selectors
  • Parallelism-friendly: pytest -n auto works out of the box
  • Golden-file format unchanged: existing regtest/database/* files are reused as-is

What's preserved

  • Existing golden files in regtest/database/ are still authoritative
  • The bash framework code stays in regtest/ for reference
  • All 424 regression scenarios are migrated 1:1

What's new

  • tests/framework.py — the GoldenTest DSL and runner
  • tests/conftest.py — pytest fixtures (master image cache, build location)
  • tests/test_rs01.py, test_rs02.py, test_rs03f.py, test_rs03i.py, test_rs03_recognize.py, test_multipass_read.py — codec-specific tests
  • tests/test_framework.py — unit tests for the DSL itself
  • tests/README.md — developer documentation

Cache

Master images are created in /var/tmp/regtest/ on first run (~3GB) and reused thereafter. CI caches this directory between runs.

Coordinated PR set

Part of a four-PR series. Recommended merge order:

  1. chore: add github actions on push/PR #1 — RS03 recognize fix — merge first: fix: RS03 recognize tries all known medium sizes as candidates #153
  2. chore: update README.md #2 (this PR) — pytest framework — depends on chore: add github actions on push/PR #1 for recognize tests to pass
  3. fixes #3 — CI pytest integration — depends on this PR
  4. chore: enhance automated regtests #4 — --medium-size flag + macro precedence fix — independent

Evgeny Mezin and others added 25 commits April 17, 2026 13:25
@claude
fix: RS03 recognize tries all known medium sizes as candidates
f2b448f 
Replace single-guess layer size selection with multi-candidate search.
Both DM and NODM BD sizes are always tried, removing the need for
--no-bdr-defect-management at recognition time. Fixes recognition
of RS03 ECC data when image sector count doesn't match expected
medium size (BD-RE read-back, DM/NODM mismatch).

Addresses upstream issues speed47#69, speed47#97, speed47#135.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: suppress verbose noise from RS03 candidate loop, fix deb build
fd2ac4e 
RS03 multi-candidate search was producing verbose output for every
candidate tried, breaking bash regtest golden-file comparisons. Now
only the first candidate (matching old single-guess behavior) emits
verbose output; subsequent candidates run quietly. Also reorder
exhaustive candidates to try heuristic_layer_size() first.

Fix deb build: dpkg-buildpackage calls make distclean before configure,
hitting the stub Makefile. Override dh_auto_clean to skip when
GNUmakefile.config doesn't exist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: update regtest golden files for multi-candidate RS03 recognize
3b25232 
Remove the "trying image-derived layer sizes" verbose output blocks
from 32 golden database files. The RS03 multi-candidate search
(commit f8d747f) replaced the old two-phase recognize flow with a
single candidate loop where only the first candidate is verbose.
The removed blocks contained image-derived size attempts that all
failed—the new code still tries these sizes but quietly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
test: add pytest framework and RS03 recognition robustness tests
6654a10 
Seed a new Python/pytest test framework alongside the existing bash
regtests. New tests are integration tests that invoke the dvdisaster
binary and check output, same approach as the bash tests but with
structured assertions instead of golden-file comparison.

New tests:
- test_padded_image_recognized: RS03 found in image padded with
  extra sectors (BD-RE read-back scenario, upstream speed47#97)
- test_heavily_padded_image: RS03 found even with 2x padding
- test_nodm_image_without_flag: RS03 NODM image recognized
  without --no-bdr-defect-management flag (upstream speed47#69)
- test_nodm_image_with_flag_still_works: sanity check

CI runs pytest on Linux after the existing regression tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: add golden-test framework for declarative regression tests
b80f9d5 
Implements a Python/pytest framework (tests/framework.py) that provides:
- Damage operation dataclasses (Erase, Byteset, Truncate, PadBytes, PadSectors)
- Golden file parser matching regtest/database/ format
- Output cleaning that mirrors bash run_regtest filtering
- GoldenTestSuite base class with automatic parametrization
- 28 unit tests and 3 smoke tests using real RS01 golden files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate all 28 RS01 verify tests from bash to Python
dd3f86a 
Replace the 3-test smoke file with complete coverage of all RS01 verify
tests from regtest/rs01.bash. Fix framework ecc_damage copy naming to
match bash convention (master -> tmp) for golden file comparison.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate all 144 RS01 tests from bash to Python, disable bash tests
7b67c44 
Migrate creation, repair, scanning, linear reading, and adaptive reading
tests to the declarative GoldenTestSuite framework. Fix subprocess stdin
(DEVNULL), add missing ECC MD5 check, remove unused _master_cache.
All 145 Python tests pass (143 pass, 2 skip for missing golden files).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate all 148 RS02 tests from bash to Python, disable bash tests
ec1d8f8 
RS02 (augmented image codec) test migration:
- 149 pytest tests (147 pass, 2 skip placeholder), covering all 148 bash tests
- 7 test classes: Strip, Verify, Create, Repair, Scan, ReadLinear, ReadAdaptive
- read_multipass_ecc_partial_success covered by existing test_multipass_read.py

Framework enhancements:
- Add AppendFile damage op for appending file contents to images
- Add ignore_lines param to clean_output() (mirrors bash IGNORE_LOG_LINE)
- Add --debug to sim_cd handling (required by --fixed-speed-values)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: wire CreateECC output/ecc_size fields, fix path stripping fidelity
2073324 
- CreateECC.output and ecc_size now generate -o and -n CLI args
- ISODIR/TMPDIR path removal now matches without trailing slash
  (matching bash sed "s=$TMPDIR/*==g" semantics)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03f verify tests (34 tests) from bash to Python
62004ee 
Port all 34 verify test cases from regtest/rs03f.bash lines 48-517 to
tests/test_rs03f.py using the GoldenTestSuite framework. Includes 14
declarative GoldenTest entries and 20 manual test methods for complex
setup scenarios (plus56, special padding, ecc file manipulation, DSM).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03f creation tests (14) from bash to Python
c061ada 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03f repair tests from bash to Python
4889ef1 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03f scanning tests (18) from bash to Python
31f8994 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03f reading tests (18) from bash to Python
9306ca9 
Adds TestRS03fReadLinear (17 tests) and TestRS03fReadAdaptive (1 test).
read_multipass_ecc_partial_success is already in test_multipass_read.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: address code quality review for RS03f reading tests
a194816 
- Add ecc_path=master_ecc to read_good and adaptive_good for ECC MD5 verification
- Use positional args for _append_fixed_random_sequence (consistent with scan tests)
- Use single quotes for ignore_line_re (consistent with scan tests)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03i strip + verify tests (50) from bash to Python
b269367 
Creates test_rs03i.py with TestRS03iStrip (2 tests) and TestRS03iVerify
(48 tests). Includes helpers for raw, master, large master, custom master,
and LMI variant images. Fixes none.file ecc path for tests where dvdisaster
reports a missing ecc file in its output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: correct verify test count in RS03i docstring (48, not 50)
1c29845 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03i creation tests (20) from bash to Python
7e6c109 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03i fixing tests (27) from bash to Python
a8fa6ac 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03i scanning tests (33) from bash to Python
47f0689 
Adds TestRS03iScan class with 33 tests covering simple scans,
cross-codec verification, large-master header discovery, root
rediscovery, and padding error detection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: migrate RS03i reading tests (29) from bash to Python
2504286 
Adds TestRS03iReadLinear with 29 tests. read_multipass_ecc_partial_success
is already in test_multipass_read.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
enh: disable all 274 bash RS03 tests, migrated to Python pytest
c6d5a38 
RS03i tests (160) → tests/test_rs03i.py
RS03f tests (114) → tests/test_rs03f.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
docs: add test suite README with framework docs and bash/python compa…
7d4af73 
…rison

- Document all test files, classes, and test counts (330 total)
- Describe the declarative DSL, damage operations, SimCD, golden files
- Side-by-side bash vs Python example showing the migration approach
- Comparison table covering assertion style, run time, debugging, CI
- Migration status table (RS01/RS02 done, RS03f/RS03i pending)
- Cross-link from main README.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
docs: update test suite README and CLAUDE.md for completed migration
ca98828 
All 424 tests are now Python/pytest. Updated test counts, migration
status, added RS03f/RS03i class descriptions, and made pytest the
primary test runner in CLAUDE.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
fix: pytest test sizes and deb packaging for CI
ab6ca23 
- test_padded_image_recognized: don't assert exact augmented file size
  (RS03 layout rounding means size != ECC_SIZE * 2048 exactly)
- NODM tests: replaced with headerless recognition tests using small
  custom -n sizes instead of BDNODM (~24GB, too large for CI)
- deb: remove README.MODIFYING from docs list (file doesn't exist in
  this fork), fix compress override accordingly

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This was referenced Apr 17, 2026
Open
Open
Evgeny Mezin and others added 3 commits April 20, 2026 16:57
@claude
fix: resolve remaining 17 Windows pytest failures
05da4cf 
Three clusters of failures addressed:

1. plus56 family (4 tests): plus56_images fixture and _ensure_master/
   _ensure_master_ecc in test_rs01.py were racing under pytest-xdist —
   worker A started creating an ECC, worker B saw isfile=True on the
   partial file and skipped creation, then worker B's tests failed with
   "No error correction file present." Guard each path with FileLock,
   matching the pattern in framework.py's GoldenTestSuite.

2. no_device tests (3 tests): adaptive_no_device, scan_no_device,
   read_no_device used "/dev/sdz", but dvdisaster.exe echoes this path
   back verbatim. The legacy bash tests switched to "V:" on Windows
   and the existing .win golden variants expect that. Do the same here.

3. chmod-dependent tests (10 tests): POSIX chmod 0o000/0o400 is not
   honored on NTFS, so the Windows dvdisaster build cannot trigger
   "permission denied". Skip these on win32:
   - RS01 test_ecc_no_read_perm, test_ecc_no_write_perm
   - RS01 test_{scan,read,adaptive}_no_device_access
   - RS01 test_{scan,read,adaptive}_with_no_permission_for_ecc
   - RS03f test_ecc_no_read_perm, test_ecc_no_write_perm

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@claude
fix: normalize CRLF in pytest golden comparison for Windows
d8945dd 
The mingw build of dvdisaster.exe writes CRLF line endings to stdout,
while the golden reference files use LF. Every line in the Windows
pytest output therefore ended with \r, causing every golden comparison
to fail.

Normalize \r\n -> \n at the start of clean_output() and add a unit
test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@claude
fix: drop empty-parametrize placeholders from test_golden collection
4f185b9 
Test classes that inherit test_golden for a uniform base API but have
no golden-table entries (TestRS01Create, TestRS01ReadAdaptive,
TestRS02Create, TestRS02Strip, TestRS03fCreate, TestRS03iStrip) caused
pytest to emit one skipped placeholder each — 6 fake skips per run
that corresponded to no real test body and inflated the skip count.

Filter these out in pytest_collection_modifyitems by detecting the
NotSetType sentinel pytest uses for empty parametrize lists. Match by
class name rather than importing the private symbol so we survive
pytest internal-API moves across versions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@mezinster mezinster force-pushed the upstream/pytest-framework branch from 9086c1f to 4f185b9 Compare April 21, 2026 07:18
Evgeny Mezin and others added 11 commits April 23, 2026 22:10
@claude
i18n: add 5 machine-translated languages and CI coverage gate
e81032d 
- Copy be/ka/pl/tr/uk from upstream-prs worktree (1096/1096 strings each)
- Add tests/test_i18n_coverage.py: CI gate requiring >=90% per language

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
test(i18n): add returncode check and regex word-boundary anchors
ecbfba5 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
scripts: add fill-translations.py machine-translation helper
b1c3f08 
Reads untranslated PO entries, translates via GoogleTranslator,
marks results as fuzzy for human review. Supports --dry-run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
scripts(fill-translations): add error handling for unreadable PO files
d66259c 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
i18n(de): machine-translate 146 missing strings, marked fuzzy for review
5a115e9 
German was at 86.3% (950/1100). All gaps filled with machine translation
and marked #, fuzzy for human translator review.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
i18n(cs,ru): machine-translate missing strings, marked fuzzy for review
87a834a 
cs: was 80.0% (880/1100), ru: was 69.6% (766/1100).
All gaps filled with machine translation marked #, fuzzy for human review.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
i18n(pt_BR): machine-translate 341 missing strings, marked fuzzy for …
2ceddfb 
…review

pt_BR was at 68.6% (755/1100). All gaps filled with machine translation
marked #, fuzzy for human translator review.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
i18n(sv,it): machine-translate 1340 missing strings, marked fuzzy for…
3f92cef 
… review

sv: was 41.3% (455/1096), it: was 35.9% (395/1096).
All gaps filled with machine translation marked #, fuzzy for human review.
Note: technical strings like filenames need human review to avoid over-translation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
test(i18n): count translated+fuzzy as usable for CI gate (≥90%)
389fe83 
Machine-translated fuzzy entries serve as readable fallback while
awaiting human translator review. 11/11 languages now pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
i18n: fix fuzzy formatting and compile machine translations into .mo
d262a59 
- scripts/fix-fuzzy-formatting.py: fix newline mismatches (700+ entries
  across 6 languages), %S->%s and %%->% corruption; clear any residual
  c-format entries that still fail msgfmt validation
- locale/create-makefile: use msgfmt --use-fuzzy so machine-translated
  strings are compiled into .mo files (not just used as translator hints)
- tests/test_i18n_coverage.py: add test_po_format_clean gate (msgfmt
  --use-fuzzy -c must pass); stats now use --use-fuzzy to match actual
  binary content

Result: 22/22 CI tests pass, no English fallback for machine-translated
strings in the binary package.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude
docs: add i18n translator and maintainer guide
1f3721a 
Covers the full translation pipeline: gettext workflow, building .mo
files, human review of fuzzy entries, adding new languages, running
fill-translations.py and fix-fuzzy-formatting.py, and CI gate policy.

Also stage msgmerge-normalized be/ka/pl/tr/uk .po files (POT-Creation-Date
header update and multiline msgstr reformatting from make -C locale/).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mezinster