Repository navigation
fix(query): IsNull selects row groups whose documents lack the path (#89) - #90
Merged
Merged
Conversation
- IsNull selects one-document row groups whose document lacks the path - Oracle and pinned tests use the uniform rule
…#89) - IsNull adds root presence minus path presence to the null and absent sets - Fail open to all row groups when root evidence is missing - No wire format change; v10/v11 indexes are fixed on read
…per row group (#89) - Property test compares against NullRGBitmap union AbsentRGs - SIMD parser root presence test behind the simdjson tag
#89) - Absent key counts as null at any row group document count - CHANGELOG Unreleased gets Changed and Fixed entries; no version cut
Decode rejects every version other than v11, so v10 indexes cannot be read and the claim that they are fixed on read was wrong.
Drop nil checks the builder and decoder cannot trigger, union in place instead of cloning twice, shorten the comments, and reuse requirePredicateResult in the new tests.
…y union wording (#89)
…ling ledger and verification
tazarov
force-pushed
the
fix/issue-89-isnull-absent
branch
from
October 6, 2026 15:52
356f5a1 to
c1bc6d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #89
Summary
IsNull(P)now selects every row group (RG) holding a document with an explicitnullfor P or without P, whatever the RG's document count. Before, an absent key counted as null only in RGs with 2+ documents, so a one-document RG without P was under-selected andNE ∪ IsNullskipped it.Every committed document marks
$present, sopresent($) − present(P)is the set of RGs whose documents all lack P.AbsentRGsstill covers mixed RGs ({"env":"prod"}+{"app":"x"}), where P is present in one document.The data was already in the index, so this is query-side only:
builder.go,serialize.goand the golden.binfiles are unchanged.Header.NumRowGroups, so a short decoded bitmap cannot drop high RGs.The new meaning also applies to plain 1:1 indexes. This follows the common convention (MongoDB
{f:null}, ClickHouse JSON, Iceberg/Delta missing columns, SQL->>). README, theIsNullgodoc and theAbsentRGscomment now state one rule. CHANGELOG Unreleased has one bullet for the 1:1 behavior change and one for the #89 fix.Evidence
df42273, issue repro):isnull_uniform_test.gocovers one-document RGs, RGs where every document lacks P, mixed RGs, empty RGs (never selected), sparse DocIDs, nested and array paths, Encode/Decode round trip, root presence fornull/5/[]/"s"/{}roots on the stdlib, materializing and SIMD parsers, and fail-open on missing or short presence evidence.IsNullexactly on every RG that received documents, and checks that the result is a superset of the pre-IsNull misses a row group whose only document lacks the path #89 rule derived from the documents.df42273on 400 seeded indexes: every other operator gave identical results.IsNulldiffered only by adding RGs.make test: 1261 passed, 1 skipped (missingtestdata/test.parquet, unrelated).make lint: 0 issues.Merge Danger
Door: two-way
Query-side only, no stored data changes. A revert restores the old answers for every index.
Blast Radius: wider
IsNullresultsA 1:1 caller who reads
IsNullas "explicit null only" now gets RGs whose document lacks the key. For pruning this means extra scans, never a missed match. A caller that treats the result as an exact answer sees more RGs. The first CHANGELOG bullet calls this out. No API, operator or format change.