Skip to content

fix(query): IsNull selects row groups whose documents lack the path (#89) - #90

Merged
oss-amikos merged 10 commits into
mainfrom
fix/issue-89-isnull-absent
Oct 6, 2026
Merged

oss-amikos merged 10 commits into
mainfrom
fix/issue-89-isnull-absent

Conversation

@oss-amikos

@oss-amikos oss-amikos commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #89

Summary

IsNull(P) now selects every row group (RG) holding a document with an explicit null for P or without P, whatever the RG's document count. Before, an absent key counted as null only in RGs with 2+ documents, so a one-document RG without P was under-selected and NE ∪ IsNull skipped it.

 evaluateIsNull(P)
   if P or root "$" has no NullIndex
-    return NoRGs        (P missing)
+    return AllRGs       (fail open)
-  return null(P) ∪ AbsentRGs(P)
+  return null(P) ∪ AbsentRGs(P) ∪ (present($) − present(P))

Every committed document marks $ present, so present($) − present(P) is the set of RGs whose documents all lack P. AbsentRGs still covers mixed RGs ({"env":"prod"} + {"app":"x"}), where P is present in one document.

The data was already in the index, so this is query-side only:

  • Wire format stays v11. builder.go, serialize.go and the golden .bin files are unchanged.
  • Existing v11 indexes give the new answer on read, with no rebuild.
  • The complement is taken against Header.NumRowGroups, so a short decoded bitmap cannot drop high RGs.

The new meaning also applies to plain 1:1 indexes. This follows the common convention (MongoDB {f:null}, ClickHouse JSON, Iceberg/Delta missing columns, SQL ->>). README, the IsNull godoc and the AbsentRGs comment now state one rule. CHANGELOG Unreleased has one bullet for the 1:1 behavior change and one for the #89 fix.

Evidence

  • Before (df42273, issue repro):
    IsNull($.env)        -> [2]      want [2 3]
    NE ∪ IsNull          -> [2]
    golden nulls-and-missing.bin  IsNull($.b) -> [3]
    
    After:
    IsNull($.env)        -> [2 3]
    NE ∪ IsNull          -> [2 3]
    golden nulls-and-missing.bin  IsNull($.b) -> [2 3]   (same v11 bytes)
    
  • isnull_uniform_test.go covers one-document RGs, RGs where every document lacks P, mixed RGs, empty RGs (never selected), sparse DocIDs, nested and array paths, Encode/Decode round trip, root presence for null/5/[]/"s"/{} roots on the stdlib, materializing and SIMD parsers, and fail-open on missing or short presence evidence.
  • The property-test oracle no longer copies the old 2+ document gate (that is why it missed the bug). It checks IsNull exactly on every RG that received documents, and checks that the result is a superset of the pre-IsNull misses a row group whose only document lacks the path #89 rule derived from the documents.
  • A separate run against df42273 on 400 seeded indexes: every other operator gave identical results. IsNull differed only by adding RGs.
  • make test: 1261 passed, 1 skipped (missing testdata/test.parquet, unrelated). make lint: 0 issues.

Merge Danger

Door: two-way

Query-side only, no stored data changes. A revert restores the old answers for every index.

Blast Radius: wider IsNull results

A 1:1 caller who reads IsNull as "explicit null only" now gets RGs whose document lacks the key. For pruning this means extra scans, never a missed match. A caller that treats the result as an exact answer sees more RGs. The first CHANGELOG bullet calls this out. No API, operator or format change.

- IsNull selects one-document row groups whose document lacks the path
- Oracle and pinned tests use the uniform rule
…#89)

- IsNull adds root presence minus path presence to the null and absent sets
- Fail open to all row groups when root evidence is missing
- No wire format change; v10/v11 indexes are fixed on read
…per row group (#89)

- Property test compares against NullRGBitmap union AbsentRGs
- SIMD parser root presence test behind the simdjson tag
#89)

- Absent key counts as null at any row group document count
- CHANGELOG Unreleased gets Changed and Fixed entries; no version cut
Decode rejects every version other than v11, so v10 indexes cannot be
read and the claim that they are fixed on read was wrong.
Drop nil checks the builder and decoder cannot trigger, union in place
instead of cloning twice, shorten the comments, and reuse
requirePredicateResult in the new tests.
@tazarov
tazarov force-pushed the fix/issue-89-isnull-absent branch from 356f5a1 to c1bc6d4 Compare October 6, 2026 15:52
@oss-amikos
oss-amikos merged commit 39deafc into main Oct 6, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IsNull misses a row group whose only document lacks the path

1 participant