[bugfix] following::* axis position-dependence (#2129)#6325
Open
joewiz wants to merge 2 commits into
Open
Conversation
Wildcard `following::*` (and `preceding::*`) is evaluated by walking an
XMLStreamReader through the document and applying a FollowingFilter /
PrecedingFilter that knows where the reference node lives. The reader
was being started at the document root irrespective of where the
reference node was, so a query like
$w/following::*[1]
ran in O(refPosition + K) rather than O(distance-to-Kth-match), and
the late-position case in issue eXist-db#2129 reproduced today: at xml:id=45000
in a 50,000-element flat document, the user's reproducer ran ~3x slower
than at xml:id=5000.
JFR on the late-position case (reproducer in tight loop, 60s, 10ms
sample period) showed the dominant time in EmbeddedXMLStreamReader.filter
(17.5% inclusive), RawNodeIterator.next (17.7%), DLN.compareTo (16.5%
self), and FastQSort.IntroSortLoop (13.9% inclusive) -- consistent with
"walk every event from doc start, compare each NodeId to the reference,
accumulate matches into a sorted result set".
For FOLLOWING_AXIS we now start the reader at the reference node when
it lies inside the document-child's subtree being walked. The
FollowingFilter already skips the reference node and its descendants
(via its isAfter / isDescendantOf checks) and terminates on END_ELEMENT
at the document-child's tree level, so starting later in document order
is safe. PRECEDING_AXIS keeps the historical doc-start walk because
PrecedingFilter must still see every preceding event; that residual
position-dependence is tracked separately.
Measured on craigberry's 50,000-element reproducer (preceding + following
combined, trimmed mean of 5 runs):
before fix after fix
xml:id=5000 127 112
xml:id=15000 183 144
xml:id=25000 255 186
xml:id=35000 326 220
xml:id=45000 390 259
Position-dependence ratio (45k/5k) drops from 3.07x to 2.30x. The
late-position case improves by 33%. The residual 2.30x is the
preceding::* path, addressed separately.
Isolating the following::* axis after the fix shows it is now
position-independent:
xml:id=5000: 95 ms
xml:id=15000: 88 ms
xml:id=25000: 86 ms
xml:id=35000: 87 ms
xml:id=45000: 86 ms
Adds FollowingAxisPositionRegressionTest with three correctness checks
(midpoint output, late-position output, descendant exclusion) and a
position-independence guard asserting that following:: at xml:id=45000
in a 50,000-element document completes within 3x of xml:id=5000.
Refs eXist-db#2129
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 9, 2026
reinhapa
requested changes
May 10, 2026
| @ClassRule | ||
| public static final ExistXmldbEmbeddedServer existEmbeddedServer = | ||
| new ExistXmldbEmbeddedServer(false, true, true); | ||
|
|
Member
There was a problem hiding this comment.
Address Codacy issue: Avoid unused private fields such as 'SMALL_DOC'.
Member
Author
There was a problem hiding this comment.
[This response was co-authored with Claude Code. -Joe]
Done in e628749e20 — SMALL_DOC removed (the test exercises only LARGE_DOC).
…OC field Address reinhapa Codacy comment on PR eXist-db#6325 at line 57. SMALL_DOC was declared but never referenced; the test exercises only LARGE_DOC. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
line-o
added
xquery
joewiz
added a commit
to joewiz/exist
that referenced
this pull request
May 12, 2026
Wildcard `preceding::*[K]` previously accumulated every preceding match into the result NodeSet between document start and the reference node, then let the predicate machinery pick the K-th. On a 50,000-element flat document at @xml:id=45000, that meant ~45,000 NodeProxy allocations, ~45,000 result.add() calls, and an O(N log N) sort downstream, even though only 5 elements could ever be selected by `[5]`. The fix: when LocationStep.computeLimit() yields a positive K (the existing positional-predicate detection used by FollowingFilter), PrecedingFilter switches to a K-bounded sliding window. Matches are buffered in an ArrayDeque sized to K, with the oldest evicted as new ones arrive. The window flushes into the result NodeSet on filter termination (reference node reached or root END_ELEMENT). Why a sliding window instead of "stop after K" (the FollowingFilter shape from PR eXist-db#6325): preceding-axis positional `[K]` selects the K-th match in axis order = (K-th-from-end) in document order. We have to keep walking past every match to know which K are the most recent. Performance impact (50,000-element flat doc, 5 trials median): - ratio lateMs/earlyMs: ~2.55x -> ~2.02x (position-dependence damped) - wildcard-vs-sibling gap: ~1.75x -> ~1.52x (closer to craigberry's reported ~1.33x sibling baseline, not yet at parity) The StAX walk itself remains O(refPosition) since matches must be emitted before the reference and the reader is forward-only, so absolute time still grows with position. Eliminating that would require a different approach (e.g., backward navigation through the persistent NodeId structure for wildcard tests). The K-bounded buffer is a clean, conservative win on the allocation/sort axis and a prerequisite for any later walk-avoidance work. Tests: - PrecedingAxisPositionRegressionTest mirrors PR eXist-db#6325's FollowingAxisPositionRegressionTest: correctness at 3 reference positions (early, mid, late), ancestor exclusion, K=1..4 axis-order semantics, position-independence threshold, and a wildcard-vs-preceding-sibling comparison. - positional.xqm:180 `optimize-simple-preceding` documented the prior gap (no POSITIONAL_PREDICATE optimization on preceding axis); the assertion is flipped to expect the optimization, mirroring the existing `optimize-simple-following-nested` case at line 170. - exist-core suite: 6599 tests, 0 failures, 0 errors (106 pre-existing skips). Partially addresses eXist-db#2129 (the preceding-axis half; following-axis half is closed by PR eXist-db#6325). Full closure of the sibling-vs-wildcard gap requires walk-avoidance, left as follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wildcard
following::*is no longer position-dependent. On craigberry's 50k-element reproducer at@xml:id=45000, the late-position case is 33% faster (390ms → 259ms); the position-dependence ratio (45k vs 5k) drops from 3.07× to 2.30×; isolating justfollowing::*[i][self::w]shows it now runs in flat ~86ms regardless of reference position.Closes the
following::half of #2129. Thepreceding::half remains and is being tracked as a follow-up tasking — it requires a different optimisation (last-K circular buffer inPrecedingFilterdriven by an optimizer-level rewrite).Root cause
JFR profile of the late-position case at xml:id=45000 (60s recording, 10ms sample cadence):
RawNodeIterator.nextEmbeddedXMLStreamReader.filterDLN.compareTo(self)FastQSort.IntroSortLoopLocationStep.getPrecedingOrFollowingRead together: walk every StAX event from document start, NodeId-compare each to the reference, accumulate, sort. The walker's start position was always the document-child root regardless of where the reference node actually lived.
Fix
exist-core/src/main/java/org/exist/xquery/LocationStep.java— forFOLLOWING_AXISonly, start the StAX reader at the reference node when it lies inside the document-child being walked. The existingFollowingFilter'sisAfter/isDescendantOfchecks correctly skip the reference and its descendants, so the later start position is safe and produces identical results. New helperreaderStartForWildcardAxiskeepsgetPrecedingOrFollowing's NPath at its baseline of 350.PRECEDING_AXISis intentionally unchanged — its filter must still observe every preceding event to honor positional predicates, so the fix shape is different and requires last-K recognition. Tracked separately.Numbers (craigberry's reproducer, 50,000 element flat doc, trimmed mean of 5 runs)
@xml:idPosition-dependence ratio 45k/5k: 3.07× → 2.30×.
Isolated
following::*[i][self::w]after fix: 95 / 88 / 86 / 87 / 86 ms at the 5 positions — flat (position-independent).Test plan
FollowingAxisPositionRegressionTest(4 cases): midpoint output, late-position output, descendant exclusion (<a><inner1/><inner2/></a><b/><c><inner3/></c>→a/following::*returns exactlyb, c, inner3), position-independence guard (following::at xml:id=45000 ≤ 3× xml:id=5000)XPathQueryTest148/148XQueryTest99/99mvn test -pl exist-core6596/6596 (0 failures, 0 errors, 106 skipped)getPrecedingOrFollowingNPath unchanged from baseline (350)🤖 Generated with Claude Code