[feature] Inline let-bound persistent paths for index pre-select

[joewiz]

Summary

Closes #873.

A let $v := <persistent path> return $v[Optimizable-pred] query was running ~167x slower than the equivalent direct form <persistent path>[Optimizable-pred]. This PR adds a narrow rewrite at the LetExpr.optimize(cc) stage that converts the indirect form into the direct form when six soundness gates hold, so the legacy Optimizer pass attaches the (#exist:optimize#) pragma to the resulting LocationStep and routes through the lucene/range pre-select.

Base: this PR targets develop but is built on top of optimizer/expression-optimize-method, the orphan-rescue branch carrying the four foundational Expression.optimize(CompileContext) framework commits (e6993cc020, ef07bb8cbc, 50b512b611, c922cefc2c). Those need to land first; the rewrite uses cc.replaceWith(...) and the per-expression optimize pass introduced there.

Diagnosis

For

let $a := collection('/db/system/test')//LINE
return $a[ft:query(., 'Denmark')]

Optimizer.visitFilteredExpr does wrap the FilteredExpression in the optimize pragma. At runtime Optimize.eval runs the lucene pre-select and obtains a small result NodeSet. But because the FE source is a VariableReference rather than a LocationStep, before() records contextStep == null (BasicExpressionVisitor.findFirstStep finds no LocationStep at the top level). With contextStep == null, Optimize.eval falls through to innerExpr.eval(result, null) -- but FilteredExpression.eval then calls expression.eval(contextSequence, ...) where expression is the VariableReference, and VariableReference.eval reads the bound value off the local variable stack and ignores contextSequence. So seq becomes the full //LINE NodeSet and ft:query runs once per LINE; the pre-selected result is computed and discarded.

The same issue affects range and any other Optimizable-bearing predicate; the predicate type is incidental.

What changed

File	Purpose
exist-core/src/main/java/org/exist/xquery/LetInliner.java	New helper. tryInline(LetExpr, CompileContext) checks the six gates; on success, attaches the FE's predicate to the input path's last LocationStep and returns the input path as the LetExpr's replacement (logged via cc.replaceWith).
exist-core/src/main/java/org/exist/xquery/LetExpr.java	New isScoreBinding() accessor; calls LetInliner.tryInline(this, cc) after the existing literal-drop gate.
extensions/indexes/lucene/src/test/java/org/exist/indexing/lucene/LetInliningRegressionTest.java	Six JUnit tests: one correctness pair, one optimizer-log inspection of the inline rewrite firing, three negative-gate tests (used twice / bound to count() / typed declaration), one loose perf bound (20x ceiling + 500ms slack -- catches a 167x regression without CI flakiness).

The six gates (LetInliner.tryInline)

1. Variable name present and not a score binding -- XQFT 3.0 §2.3 score bindings synthesize a double rather than the input value; inlining would change semantics.
2. Standalone let, not a chain link -- getPreviousClause() == null and the body is not a FLWORClause. Limits v1 to standalone lets so we don't have to repair previousClause pointers.
3. No declared type on the binding -- typed declarations impose a runtime cardinality+type check on $v's bound value; inlining bypasses it. v2 may relax this when the inlined static type still satisfies the declaration.
4. Input is node-typed -- strings, atomics, etc. don't gain from a downstream index pre-select.
5. Input contains a non-wildcard LocationStep -- the predicate has to attach somewhere indexable.
6. Body is a FilteredExpression (or a length-1 PathExpr / DebuggableExpression wrapping one) whose source is the bound variable, with exactly one Optimizable predicate, and the variable does not appear anywhere else in the body -- mirrors what Optimizer.visitFilteredExpr's instanceof LocationStep simplification expects, so the post-rewrite shape is exactly what the legacy pass already knows how to wrap. The "exactly one predicate" + "no other refs" guards prevent positional-predicate or substituted-out-of-scope correctness regressions.

The "inlining is not always desirable, in particular if there's no index defined" caveat noted in a 2016 comment on the issue is exactly gate 6: we only inline when the inlined form would expose an Optimizable to visitLocationStep.

Why option (a) inline at AST level, vs option (b) special-case Optimize.eval

• (a) generalises -- the same rewrite catches any Optimizable predicate (ft:query, GeneralComparison, FunMatches, range Lookup, ...) without per-predicate-type repair.
• (a) addresses the 2016 caveat (gate 6 is the if-index-defined check).
• (a) fits the framework already on this branch; cc.replaceWith handles logging and the re-analyze flag.
• (a) mirrors BaseX's Let.inlineExpr / GFLWOR.inlineForLet pattern.

Option (b) is kept as a follow-up for shapes (a) doesn't catch (e.g., let bindings to function-call results that don't expose a LocationStep in the input).

Test plan

• mvn test -pl extensions/indexes/lucene -- new LetInliningRegressionTest passes (6/6); existing OptimizerTest, LuceneIndexTest pass; 5 pre-existing ft-facets failures unchanged (verified by stash-and-rerun).
• mvn test -pl extensions/indexes/range -- 425/425 pass + 3 skipped, no new failures.
• mvn test -pl exist-core -- baseline 34 fail / 20 err vs with-fix 35 fail / 20 err (+1 = known flaky 503 HTTP test); unique failure set is byte-identical.
• Codacy PMD on changed files -- clean.
• (Deferred) JMH LetIndirectionBenchmark -- depends on the exist-indexes-jmh module which lives on a separate branch (feature/index-jmh-benchmarks); will land as a follow-up once that module reaches develop.

XQTS scores (QT4, 3.1, FTTS) should be unchanged because the rewrite preserves correctness; will flag if any score moves.

[joewiz]

Closing -- the base branch (optimizer/expression-optimize-method) does not exist on eXist-db/exist, so the diff incorrectly includes the 256 framework commits that this fix is stacked on. Will reopen once the foundation branch is upstream, or open a within-fork PR for review in the meantime.

[joewiz]

Work is now in joewiz#13