[ci] compare-results.xslt: surface runner-JAR drift warning when build SHAs differ#6332
Open
joewiz wants to merge 1 commit into
Open
[ci] compare-results.xslt: surface runner-JAR drift warning when build SHAs differ#6332joewiz wants to merge 1 commit into
joewiz wants to merge 1 commit into
Conversation
…d SHAs differ Adds a `cr:warnings` block to the comparison report that fires when the previous and current XQTS runs were produced by different runner JAR builds. Without this, develop-branch deltas, runner-source bugfixes, and shaded-`exist-core` differences all silently appear as if they were caused by the PR's source change. Why: see eXist-db#6326. A 2026-05-09 investigation found the same eXist SHA producing +831 XQTS passes and a 25x wallclock improvement when re-run against a different runner JAR build (the runner-side `applyVersionHint` cap-at-3.1 patch). The existing comparison report attributes those deltas to the PR's source change, which is wrong. This makes runner-JAR drift directly attributable rather than masquerading as engine improvements. What changed: - Reads `runner-info.xml` from each run's output dir root (two levels up from the junit-data path). The metadata is emitted by the companion `exist-xqts-runner` change; older runs predating that change will have the file absent. - Emits a `cr:warning kind="runner-drift"` element when the runner JAR's git-sha or sha256 differs between the runs, including the full `runner-jar` payloads from each side for transparency. - Emits a `cr:warning kind="embedded-exist-core-drift"` when the runner JAR appears identical but the shaded `exist-core` version differs (the second drift mode surfaced during PR eXist-db#6331's perf-6322 investigation). - Graceful degradation: when either side's `runner-info.xml` is missing or malformed (`doc-available` returns false), no warning is emitted and the rest of the comparison runs unchanged. Older baselines and contributor environments without the runner-side patch keep working with no behaviour change. Hand-tested with synthetic inputs against Saxon-HE 9.9 (the same version CI uses), covering: SHAs differ, SHAs match, missing on one side, malformed XML, and embedded-exist-core-only drift. `ci-xqts.yml` needs no edit -- the artifact upload uploads the entire `/tmp/xqts-output` dir and `runner-info.xml` round-trips automatically. Partially addresses eXist-db#6326 (mitigation A only). Mitigations B (runner version pin) and C (N-run flake filtering) remain open. Companion runner PR: https://github.com/eXist-db/exist-xqts-runner/pull/<TBD> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
cr:warningsblock to the comparison report that fires when the previous and current XQTS runs were produced by different runner JAR builds. Without this, develop-branch deltas, runner-source bugfixes, and shaded-exist-coredifferences all silently appear as if they were caused by the PR's source change.Partially addresses #6326 (mitigation A only). Mitigations B (runner version pin) and C (N-run flake filtering) remain open.
Companion runner PR (the producer of the metadata this consumes): eXist-db/exist-xqts-runner#51
Why
A 2026-05-09 investigation found the same eXist SHA producing +831 XQTS passes and a 25x wallclock improvement when re-run against a different runner JAR build (the runner-side
applyVersionHintcap-at-3.1 patch). The existing comparison report attributed those deltas to the PR's source change, which is wrong. This makes runner-JAR drift directly attributable rather than masquerading as engine improvements.A second drift mode surfaced during PR #6331's perf-6322 investigation: locally-built runner JARs shade
exist-core, so two runs of the same runner-source git SHA can still differ if assembled from differentexist-coresnapshots. The XSLT detects both modes.What changed
exist-xqts/src/main/xslt/compare-results.xslt:runner-info.xmlfrom each run's output dir root (two levels up from the junit-data path). The metadata is emitted by the companionexist-xqts-runnerchange; older runs predating that change will have the file absent.cr:warning kind=\"runner-drift\"element when the runner JAR's git-sha or sha256 differs between the runs, including the fullrunner-jarpayloads from each side for transparency.cr:warning kind=\"embedded-exist-core-drift\"when the runner JAR appears identical but the shadedexist-coreversion differs.runner-info.xmlis missing or malformed (doc-availablereturns false), no warning is emitted and the rest of the comparison runs unchanged. Older baselines and contributor environments without the runner-side patch keep working with no behaviour change.`ci-xqts.yml` needs no edit -- the artifact upload uploads the entire `/tmp/xqts-output` dir and `runner-info.xml` round-trips automatically.
Sample output (runner-drift case)
```xml
<cr:comparison xmlns:cr="http://exist-db.org/exist-xqts/compare-results\">
cr:warnings
<cr:warning kind="runner-drift">
cr:summaryRunner JAR build SHA or sha256 differs between the previous and current XQTS runs. Test deltas may include runner-side effects unrelated to this PR. See #6326.</cr:summary>
cr:previous
<runner-jar xmlns="http://exist-db.org/exist-xqts-runner/runner-info\">
aaaa1111...
...
</cr:previous>
cr:current...</cr:current>
</cr:warning>
</cr:warnings>
cr:previous...</cr:previous>
cr:current...</cr:current>
cr:change...</cr:change>
</cr:comparison>
```
Test plan
Hand-tested locally against Saxon-HE 9.9 (the same version CI invokes), with synthetic `runner-info.xml` files in five scenarios:
Risk and rollback
The change is purely additive. Reverting removes the warning element but doesn't break existing comparison output -- the per-test-comparison structure is unchanged, so any downstream consumer of `comparison-results.xml` remains compatible. The two PRs (this one and the runner-side) can land in either order; the XSLT degrades to a no-op when `runner-info.xml` is absent.