Repository navigation
Realize MarkJoin as HashJoin for IN/NOT IN (issue #272) - #323
Conversation
Replace nested-loop execution with hash-based execution for IN/NOT IN derived MarkJoin. Hash on correlation (residual) keys, then evaluate the marker predicate per-row for three-valued NULL logic. Falls back to nested loop when no hashable residual keys exist. EXISTS/NOT EXISTS MarkJoin continues to use nested loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Implements hash-based execution for IN/NOT IN-derived PhysicMarkJoin to replace nested-loop evaluation when correlation (residual) equi-join keys exist, while retaining nested-loop execution for EXISTS/NOT EXISTS and as a fallback when no hashable residual keys are available.
Changes:
- Route
PhysicMarkJoin.Exec()to a new hash-based execution path forIN/NOT INmark joins, with a nested-loop fallback when no residual equi-keys can be extracted. - Add new regression coverage for the hash path and fallback behavior (
subqueryd_hashmj.sql+ expected output). - Update existing regression expected output (
subqueryd_or.txt) to reflect the removal of per-row loop counts under hash execution.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
qpmodel/subquery.cs |
Adds hash-based execution logic for IN/NOT IN-derived mark joins and introduces a nested-loop fallback helper. |
test/regress/sql/subqueryd_hashmj.sql |
New SQL regression tests exercising hash-based mark join behavior and no-correlation fallback. |
test/regress/expect/subqueryd_hashmj.txt |
Expected output for the new regression test file. |
test/regress/expect/subqueryd_or.txt |
Updates expected plan output (removes loops=3 on right scan due to hash execution). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| rchild_().Exec(r => | ||
| { | ||
| Row combined = new Row(l, r); | ||
|
|
||
| if (residualFilter != null) | ||
| { | ||
| bool boolMarker = marker is true; | ||
| fixMarkerValue(n, semi ? boolMarker : !boolMarker); | ||
| var flag = residualFilter.Exec(context, combined); | ||
| if (!(flag is true)) | ||
| return; | ||
| } | ||
|
|
||
| callback(n); | ||
| RisEmpty = false; | ||
| if (r.ColsHasNull()) | ||
| RHasNull = true; | ||
|
|
||
| if (markerExpr.Exec(context, combined) is true) | ||
| marker = true; | ||
| }); |
There was a problem hiding this comment.
ExecNestedInClause scans the entire right side even after marker becomes true. For IN/NOT IN semantics a single true match is sufficient to determine the final marker (and for NOT IN, a true match definitively makes the predicate false), so this can be short-circuited to avoid unnecessary work on large subqueries (e.g., by checking marker is true at the top of the rchild_().Exec callback and returning early, or using context.stop_ if that’s the established cancellation mechanism).
|
|
||
| rchild_().Exec(r => | ||
| { | ||
| Row fakeLeft = new Row(lColCount); |
There was a problem hiding this comment.
In ExecHash, fakeLeft is allocated inside the rchild_().Exec callback, so a new all-null Row is created for every right-side row. This is unnecessary allocation/GC pressure since the left placeholder is constant; consider allocating fakeLeft once outside the loop (or avoid the combined row entirely if rightKeys are guaranteed right-only).
| rchild_().Exec(r => | |
| { | |
| Row fakeLeft = new Row(lColCount); | |
| Row fakeLeft = new Row(lColCount); | |
| rchild_().Exec(r => | |
| { |
Summary
subqueryd_hashmj.sqlregression tests covering IN, NOT IN, with/without correlation keysCloses #272
Test plan
subqueryd_hashmj.sqlregression test covers hash path (IN, NOT IN, with extra filters, no-correlation fallback)subqueryd_or.txtexpect file (no moreloops=3for hash-executed MarkJoin)🤖 Generated with Claude Code