Skip to content

[ci] compare-results.xslt: surface runner-JAR drift warning when build SHAs differ#6332

Open
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:feature/ci-runner-drift-visibility
Open

[ci] compare-results.xslt: surface runner-JAR drift warning when build SHAs differ#6332
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:feature/ci-runner-drift-visibility

Conversation

@joewiz
Copy link
Copy Markdown
Member

@joewiz joewiz commented May 10, 2026

Summary

Adds a cr:warnings block to the comparison report that fires when the previous and current XQTS runs were produced by different runner JAR builds. Without this, develop-branch deltas, runner-source bugfixes, and shaded-exist-core differences all silently appear as if they were caused by the PR's source change.

Partially addresses #6326 (mitigation A only). Mitigations B (runner version pin) and C (N-run flake filtering) remain open.

Companion runner PR (the producer of the metadata this consumes): eXist-db/exist-xqts-runner#51

Why

A 2026-05-09 investigation found the same eXist SHA producing +831 XQTS passes and a 25x wallclock improvement when re-run against a different runner JAR build (the runner-side applyVersionHint cap-at-3.1 patch). The existing comparison report attributed those deltas to the PR's source change, which is wrong. This makes runner-JAR drift directly attributable rather than masquerading as engine improvements.

A second drift mode surfaced during PR #6331's perf-6322 investigation: locally-built runner JARs shade exist-core, so two runs of the same runner-source git SHA can still differ if assembled from different exist-core snapshots. The XSLT detects both modes.

What changed

exist-xqts/src/main/xslt/compare-results.xslt:

  • Reads runner-info.xml from each run's output dir root (two levels up from the junit-data path). The metadata is emitted by the companion exist-xqts-runner change; older runs predating that change will have the file absent.
  • Emits a cr:warning kind=\"runner-drift\" element when the runner JAR's git-sha or sha256 differs between the runs, including the full runner-jar payloads from each side for transparency.
  • Emits a cr:warning kind=\"embedded-exist-core-drift\" when the runner JAR appears identical but the shaded exist-core version differs.
  • Graceful degradation: when either side's runner-info.xml is missing or malformed (doc-available returns false), no warning is emitted and the rest of the comparison runs unchanged. Older baselines and contributor environments without the runner-side patch keep working with no behaviour change.

`ci-xqts.yml` needs no edit -- the artifact upload uploads the entire `/tmp/xqts-output` dir and `runner-info.xml` round-trips automatically.

Sample output (runner-drift case)

```xml
<cr:comparison xmlns:cr="http://exist-db.org/exist-xqts/compare-results\">
cr:warnings
<cr:warning kind="runner-drift">
cr:summaryRunner JAR build SHA or sha256 differs between the previous and current XQTS runs. Test deltas may include runner-side effects unrelated to this PR. See #6326.</cr:summary>
cr:previous
<runner-jar xmlns="http://exist-db.org/exist-xqts-runner/runner-info\">
aaaa1111...
...

</cr:previous>
cr:current...</cr:current>
</cr:warning>
</cr:warnings>
cr:previous...</cr:previous>
cr:current...</cr:current>
cr:change...</cr:change>
</cr:comparison>
```

Test plan

Hand-tested locally against Saxon-HE 9.9 (the same version CI invokes), with synthetic `runner-info.xml` files in five scenarios:

  • Different runner-jar git-sha -> emits `cr:warning kind="runner-drift"` with both sides' payloads.
  • Identical `runner-info.xml` files -> no warning emitted.
  • `runner-info.xml` absent on one side -> no warning, no error.
  • Malformed `runner-info.xml` -> no warning, no error (`doc-available` returns false).
  • Same runner JAR git-sha + sha256 but different embedded `exist-core` version -> emits `cr:warning kind="embedded-exist-core-drift"`.
  • End-to-end smoke test on next CI run will produce a comparison report with no warning (matching SHAs from the same workflow build).

Risk and rollback

The change is purely additive. Reverting removes the warning element but doesn't break existing comparison output -- the per-test-comparison structure is unchanged, so any downstream consumer of `comparison-results.xml` remains compatible. The two PRs (this one and the runner-side) can land in either order; the XSLT degrades to a no-op when `runner-info.xml` is absent.

…d SHAs differ

Adds a `cr:warnings` block to the comparison report that fires when the
previous and current XQTS runs were produced by different runner JAR
builds. Without this, develop-branch deltas, runner-source bugfixes,
and shaded-`exist-core` differences all silently appear as if they
were caused by the PR's source change.

Why: see eXist-db#6326. A 2026-05-09
investigation found the same eXist SHA producing +831 XQTS passes and
a 25x wallclock improvement when re-run against a different runner
JAR build (the runner-side `applyVersionHint` cap-at-3.1 patch). The
existing comparison report attributes those deltas to the PR's source
change, which is wrong. This makes runner-JAR drift directly
attributable rather than masquerading as engine improvements.

What changed:
- Reads `runner-info.xml` from each run's output dir root (two levels
  up from the junit-data path). The metadata is emitted by the
  companion `exist-xqts-runner` change; older runs predating that
  change will have the file absent.
- Emits a `cr:warning kind="runner-drift"` element when the runner
  JAR's git-sha or sha256 differs between the runs, including the
  full `runner-jar` payloads from each side for transparency.
- Emits a `cr:warning kind="embedded-exist-core-drift"` when the
  runner JAR appears identical but the shaded `exist-core` version
  differs (the second drift mode surfaced during PR eXist-db#6331's
  perf-6322 investigation).
- Graceful degradation: when either side's `runner-info.xml` is
  missing or malformed (`doc-available` returns false), no warning
  is emitted and the rest of the comparison runs unchanged. Older
  baselines and contributor environments without the runner-side
  patch keep working with no behaviour change.

Hand-tested with synthetic inputs against Saxon-HE 9.9 (the same
version CI uses), covering: SHAs differ, SHAs match, missing on one
side, malformed XML, and embedded-exist-core-only drift.

`ci-xqts.yml` needs no edit -- the artifact upload uploads the entire
`/tmp/xqts-output` dir and `runner-info.xml` round-trips automatically.

Partially addresses eXist-db#6326 (mitigation A only). Mitigations B (runner
version pin) and C (N-run flake filtering) remain open.

Companion runner PR: https://github.com/eXist-db/exist-xqts-runner/pull/<TBD>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@joewiz joewiz requested a review from a team as a code owner May 10, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant