enh: integrate pytest into CI and add slow-test gating#155
Open
mezinster wants to merge 43 commits intospeed47:masterfrom
Open
enh: integrate pytest into CI and add slow-test gating#155mezinster wants to merge 43 commits intospeed47:masterfrom
mezinster wants to merge 43 commits intospeed47:masterfrom
Conversation
Replace single-guess layer size selection with multi-candidate search. Both DM and NODM BD sizes are always tried, removing the need for --no-bdr-defect-management at recognition time. Fixes recognition of RS03 ECC data when image sector count doesn't match expected medium size (BD-RE read-back, DM/NODM mismatch). Addresses upstream issues speed47#69, speed47#97, speed47#135. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RS03 multi-candidate search was producing verbose output for every candidate tried, breaking bash regtest golden-file comparisons. Now only the first candidate (matching old single-guess behavior) emits verbose output; subsequent candidates run quietly. Also reorder exhaustive candidates to try heuristic_layer_size() first. Fix deb build: dpkg-buildpackage calls make distclean before configure, hitting the stub Makefile. Override dh_auto_clean to skip when GNUmakefile.config doesn't exist. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the "trying image-derived layer sizes" verbose output blocks from 32 golden database files. The RS03 multi-candidate search (commit f8d747f) replaced the old two-phase recognize flow with a single candidate loop where only the first candidate is verbose. The removed blocks contained image-derived size attempts that all failed—the new code still tries these sizes but quietly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Seed a new Python/pytest test framework alongside the existing bash regtests. New tests are integration tests that invoke the dvdisaster binary and check output, same approach as the bash tests but with structured assertions instead of golden-file comparison. New tests: - test_padded_image_recognized: RS03 found in image padded with extra sectors (BD-RE read-back scenario, upstream speed47#97) - test_heavily_padded_image: RS03 found even with 2x padding - test_nodm_image_without_flag: RS03 NODM image recognized without --no-bdr-defect-management flag (upstream speed47#69) - test_nodm_image_with_flag_still_works: sanity check CI runs pytest on Linux after the existing regression tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements a Python/pytest framework (tests/framework.py) that provides: - Damage operation dataclasses (Erase, Byteset, Truncate, PadBytes, PadSectors) - Golden file parser matching regtest/database/ format - Output cleaning that mirrors bash run_regtest filtering - GoldenTestSuite base class with automatic parametrization - 28 unit tests and 3 smoke tests using real RS01 golden files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the 3-test smoke file with complete coverage of all RS01 verify tests from regtest/rs01.bash. Fix framework ecc_damage copy naming to match bash convention (master -> tmp) for golden file comparison. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migrate creation, repair, scanning, linear reading, and adaptive reading tests to the declarative GoldenTestSuite framework. Fix subprocess stdin (DEVNULL), add missing ECC MD5 check, remove unused _master_cache. All 145 Python tests pass (143 pass, 2 skip for missing golden files). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RS02 (augmented image codec) test migration: - 149 pytest tests (147 pass, 2 skip placeholder), covering all 148 bash tests - 7 test classes: Strip, Verify, Create, Repair, Scan, ReadLinear, ReadAdaptive - read_multipass_ecc_partial_success covered by existing test_multipass_read.py Framework enhancements: - Add AppendFile damage op for appending file contents to images - Add ignore_lines param to clean_output() (mirrors bash IGNORE_LOG_LINE) - Add --debug to sim_cd handling (required by --fixed-speed-values) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CreateECC.output and ecc_size now generate -o and -n CLI args - ISODIR/TMPDIR path removal now matches without trailing slash (matching bash sed "s=$TMPDIR/*==g" semantics) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port all 34 verify test cases from regtest/rs03f.bash lines 48-517 to tests/test_rs03f.py using the GoldenTestSuite framework. Includes 14 declarative GoldenTest entries and 20 manual test methods for complex setup scenarios (plus56, special padding, ecc file manipulation, DSM). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds TestRS03fReadLinear (17 tests) and TestRS03fReadAdaptive (1 test). read_multipass_ecc_partial_success is already in test_multipass_read.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ecc_path=master_ecc to read_good and adaptive_good for ECC MD5 verification - Use positional args for _append_fixed_random_sequence (consistent with scan tests) - Use single quotes for ignore_line_re (consistent with scan tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Creates test_rs03i.py with TestRS03iStrip (2 tests) and TestRS03iVerify (48 tests). Includes helpers for raw, master, large master, custom master, and LMI variant images. Fixes none.file ecc path for tests where dvdisaster reports a missing ecc file in its output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds TestRS03iScan class with 33 tests covering simple scans, cross-codec verification, large-master header discovery, root rediscovery, and padding error detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds TestRS03iReadLinear with 29 tests. read_multipass_ecc_partial_success is already in test_multipass_read.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RS03i tests (160) → tests/test_rs03i.py RS03f tests (114) → tests/test_rs03f.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rison - Document all test files, classes, and test counts (330 total) - Describe the declarative DSL, damage operations, SimCD, golden files - Side-by-side bash vs Python example showing the migration approach - Comparison table covering assertion style, run time, debugging, CI - Migration status table (RS01/RS02 done, RS03f/RS03i pending) - Cross-link from main README.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 424 tests are now Python/pytest. Updated test counts, migration status, added RS03f/RS03i class descriptions, and made pytest the primary test runner in CLAUDE.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_padded_image_recognized: don't assert exact augmented file size (RS03 layout rounding means size != ECC_SIZE * 2048 exactly) - NODM tests: replaced with headerless recognition tests using small custom -n sizes instead of BDNODM (~24GB, too large for CI) - deb: remove README.MODIFYING from docs list (file doesn't exist in this fork), fix compress override accordingly Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three clusters of failures addressed:
1. plus56 family (4 tests): plus56_images fixture and _ensure_master/
_ensure_master_ecc in test_rs01.py were racing under pytest-xdist —
worker A started creating an ECC, worker B saw isfile=True on the
partial file and skipped creation, then worker B's tests failed with
"No error correction file present." Guard each path with FileLock,
matching the pattern in framework.py's GoldenTestSuite.
2. no_device tests (3 tests): adaptive_no_device, scan_no_device,
read_no_device used "/dev/sdz", but dvdisaster.exe echoes this path
back verbatim. The legacy bash tests switched to "V:" on Windows
and the existing .win golden variants expect that. Do the same here.
3. chmod-dependent tests (10 tests): POSIX chmod 0o000/0o400 is not
honored on NTFS, so the Windows dvdisaster build cannot trigger
"permission denied". Skip these on win32:
- RS01 test_ecc_no_read_perm, test_ecc_no_write_perm
- RS01 test_{scan,read,adaptive}_no_device_access
- RS01 test_{scan,read,adaptive}_with_no_permission_for_ecc
- RS03f test_ecc_no_read_perm, test_ecc_no_write_perm
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mezinster
force-pushed
the
upstream/ci-pytest-integration
branch
from
531098c to
8e617fb
Compare
The mingw build of dvdisaster.exe writes CRLF line endings to stdout, while the golden reference files use LF. Every line in the Windows pytest output therefore ended with \r, causing every golden comparison to fail. Normalize \r\n -> \n at the start of clean_output() and add a unit test. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mezinster
force-pushed
the
upstream/ci-pytest-integration
branch
from
8e617fb to
293323c
Compare
Test classes that inherit test_golden for a uniform base API but have no golden-table entries (TestRS01Create, TestRS01ReadAdaptive, TestRS02Create, TestRS02Strip, TestRS03fCreate, TestRS03iStrip) caused pytest to emit one skipped placeholder each — 6 fake skips per run that corresponded to no real test body and inflated the skip count. Filter these out in pytest_collection_modifyitems by detecting the NotSetType sentinel pytest uses for empty parametrize lists. Match by class name rather than importing the private symbol so we survive pytest internal-API moves across versions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously only Linux ran the Python test suite. macOS and Windows only ran the bash runtests.sh which now skips everything. Added pytest step to all three platforms and installed Python in MSYS2 for Windows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
macOS 15 runners enforce PEP 668 externally-managed-environment, so use --break-system-packages for pip. Windows MSYS2 installs pytest via pacman (mingw-w64-x86_64-python-pytest) instead of pip. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bare `pip` installs to a different Python than `python3` on macOS 15 runners (pip→3.13, python3→3.14), causing "No module named pytest". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Master images (~3GB) were recreated from scratch on every CI run, causing 3+ hour test times on macOS/Windows. Now cached via actions/cache with keys based on OS and test file hashes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Regular pushes/PRs skip all 41 slow tests on every platform. Slow tests run on Linux only, triggered by: weekly schedule (Monday 3am UTC), release tags (v*), or manual workflow_dispatch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e build Ubuntu 18.04 reached end-of-standard-support on 2023-04-30; the main, updates, and backports pockets were moved off archive.ubuntu.com, breaking the linux64-appimage job's prerequisite install with connection timeouts and "no installation candidate" errors. Rewrite the apt sources before update so packages resolve. The build remains pinned to bionic to preserve the AppImage's glibc 2.27 compatibility floor. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Documents each workflow's purpose, the slow-test gating strategy, and the AppImage / bionic-EOL apt-rewrite rationale. Establishes a new docs/ directory in the upstream tree.
…untu.com The previous fix (5371514) was based on a wrong premise: bionic is NOT on old-releases.ubuntu.com (returns 404), so apt update failed immediately instead of timing out. The actual root cause is that the public Canonical mirror IPs (91.189.91.x / 91.189.92.x) are unreachable from inside docker on GitHub Actions runners. azure.archive.ubuntu.com is Microsoft's intra-Azure Ubuntu mirror, served from the same Azure infra GHA runners live in. It hosts all bionic pockets (main, updates, backports, security) and routes reliably from inside the docker container.
- Add `pytest-xdist` and `filelock` deps on Mac, Win, Linux. Run pytest with `-n auto` to use all available cores. - Wrap `_ensure_master` and `_ensure_master_ecc` with a per-path `FileLock`. `--random-image` is non-deterministic (regtest mode does not seed RNG), so two parallel workers racing on master creation would write divergent content and break golden-file comparisons. - Bump Mac and Win `timeout-minutes` from 120 to 240. Even with xdist the Mac runner is slow (~100s per RS02 read test); the higher cap prevents the suite from being killed mid-run.
Rebasing onto the renamed framework.filter_empty_golden_placeholders helper left two colliding pytest_collection_modifyitems definitions in conftest.py — the second (slow-gate) def silently shadowed the first (placeholder filter). Merge them into a single hook that calls the filter before applying the slow-gate logic. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
mezinster
force-pushed
the
upstream/ci-pytest-integration
branch
from
293323c to
54541d2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the new pytest framework (PR #154) into GitHub Actions, adds a
@pytest.mark.slowgate for large-image tests, and documents the workflow design in a newdocs/workflow_readme.mdfile.Changes
.github/workflows/tests.yml: pytest now runs on Linux/macOS/Windows for CLI builds. Slow tests opt-in via--run-slow. Trigger expanded to also run on push-to-master, version tags, weekly schedule, and manual dispatch..github/workflows/release.yml: AppImage build's bionic apt sources rewritten toold-releases.ubuntu.com(Bionic reached EOL 2023-04-30 and was relocated; previously the dockerapt updatestep failed with connection timeouts).tests/conftest.py: registers the--run-slowoption and theslowmarker.tests/test_rs02.py,tests/test_rs03i.py: ~40 large-image tests marked slow.docs/workflow_readme.md: new file, documents each workflow's purpose, the slow-test schedule, the test-image cache, and the bionic-EOL rationale.New convention
This PR establishes a new top-level
docs/directory. Currently upstream usesdocumentation/for PDFs but has no markdown documentation tree. The new file lives atdocs/workflow_readme.mdrather than.github/workflows/README.mdto keep contributor-facing docs in a discoverable location.Slow-test schedule
v*), manual workflow_dispatch: full suite including slow testsCoordinated PR set
Part of a four-PR series. Recommended merge order:
--medium-size+ macro precedence — independent (link to be added)