fix(ci): make nightly Valgrind job load the extension; guard bench tests under memcheck - #204
Merged
Merged
Conversation
…sts under memcheck The nightly Valgrind job (new in #201) invoked run-tests.php directly, which never loads the freshly built modules/judy.so — 227/232 tests silently SKIPped on extension_loaded('judy') and the job has failed on every run since it was introduced. Switch to make test, which passes -d extension_dir=modules -d extension=judy.so; the -m/-q/--show-diff flags ride in TESTS, which the generated Makefile appends to the run-tests.php invocation. Two classes of false positives surface once the suite actually runs under memcheck (--trace-children=yes, USE_ZEND_ALLOC=0): - The three bench-harness tests spawn child PHP processes. Children are traced too: memcheck inflates their RSS by an order of magnitude (breaking bench_memory_measurement_001's heap-vs-RSS assertions) and child memory_limit fatals leave allocations unfreed by design, which memcheck attributes to the test as a leak (bench_memory_limit_001). All three now skip when USE_ZEND_ALLOC=0 is exported, the standard run-tests marker for a Valgrind run. - PCRE2's sljit-generated code trips uninitialised-value reports inside the php binary itself (no judy.so frame), flagging every preg_*-using test as LEAK. -d pcre.jit=0 in the Valgrind run silences it. Verified in a Linux container (PHP 8.4, valgrind): the fixed command runs 229/232 under memcheck with 0 failures and 0 leaks (3 skips are the guarded bench tests); the normal suite still passes 232/232 with 0 skips. Fixes #202
Test Results
Benchmark Results
Run Status
Memory (Linux, PHP 8.6)Peak RSS of a child process building one structure, n = 500,000, minus an empty-process floor. Judy's memory advantage is type- and scale-dependent — see BENCHMARK.md for the curve across sizes on a dedicated host.
API Speedups (Linux, PHP 8.6)Speedup = PHP time / Judy time. Bold = Judy faster.
Core Types (Linux, PHP 8.6)Speedup = PHP time / Judy time. Bold = Judy faster.
All types detailCore Types (Linux, PHP 8.6) — AllSpeedup = PHP time / Judy time. Bold = Judy faster.
Cross-version detail (Linux)Core Types — LinuxEach cell: Judy ms (speedup vs PHP array).
API — Linux
Cross-version detail (Windows)Core Types — WindowsEach cell: Judy ms (speedup vs PHP array).
API — Windows
Windows results (PHP 8.6)Memory (Windows, PHP 8.6)Peak RSS of a child process building one structure, n = 500,000, minus an empty-process floor. Judy's memory advantage is type- and scale-dependent — see BENCHMARK.md for the curve across sizes on a dedicated host.
API Speedups (Windows, PHP 8.6)Speedup = PHP time / Judy time. Bold = Judy faster.
Core Types (Windows, PHP 8.6)Speedup = PHP time / Judy time. Bold = Judy faster.
All types detailCore Types (Windows, PHP 8.6) — AllSpeedup = PHP time / Judy time. Bold = Judy faster.
Release Comparison (2.6.0 → 2.7.1)
Summary:
Memory ComparisonMemory: 0 regressions, 10 unchanged
Full benchmark outputLinux — PHP 8.6Windows — PHP 8.6 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The nightly Valgrind job (introduced in #201) has failed on every run since it was added — but not because of a memory leak. It invoked
run-tests.phpdirectly, which never loads the freshly builtmodules/judy.so, so 227/232 tests silently SKIPped onextension_loaded('judy')and the only tests that executed were the three bench-harness tests, which genuinely misbehave under memcheck.Changes
make test TESTS="-j$(nproc) -m -q --show-diff -d pcre.jit=0 tests/"so the built extension is loaded (make testpasses-d extension_dir=modules -d extension=judy.so; the generated Makefile appendsTESTSto the run-tests invocation).pcre.jit=0is required because PCRE2's sljit-generated code trips uninitialised-value reports inside the php binary itself (no judy.so frame), flagging everypreg_*-using test as LEAK. The auto-filed issue's repro command is corrected to match.bench_compare_drift_001,bench_memory_limit_001,bench_memory_measurement_001) spawn child PHP processes, and run-tests' memcheck mode uses--trace-children=yes: children's RSS is inflated by an order of magnitude (breaking the heap-vs-RSS assertions) and childmemory_limitfatals leave allocations unfreed by design, which memcheck attributes to the test as a leak. They now skip whenUSE_ZEND_ALLOC=0is exported — the standard run-tests marker for a Valgrind run.Verification
pcre.jit=0, the only reports are uninitialised-value warnings in PCRE2 JIT code with no judy.so frames; with it, all 8preg_*-using tests are clean.Fixes #202