perf(lexical/index): fold .lens into a 1-byte quantised norms column and drop the derivable .fstats - #1123
Merged
Conversation
Introduces a byte-quantised length table (exact below 40, log-spaced above, generated from its recurrence rather than hand-copied) as the foundation for the .norms segment part (#555). Standalone: not yet wired into any reader or writer.
Adds NormsBuilder/NormsReader for a columnar, 1-byte-quantised field-length format that will replace .lens/.fstats (#555). The writer now emits .norms alongside the existing files (additive only, ~1500 files unaffected) so this lands with zero read-path risk; the reader does not consult .norms until a later phase. Deviates from the original design sketch in one respect: fields are loaded eagerly rather than lazily per-field (unlike .dv). A .norms file is already an order of magnitude smaller than the .lens/.fstats pair it replaces, so the lazy-materialization complexity wasn't worth it here.
write_segment_files now builds one NormsBuilder and feeds both write_inverted_index's score-bound precomputation and write_norms (#555 Phase 3), instead of two independent traversals of buffered_docs that could disagree on a field's length. The Block-Max-WAND bound is now computed from the decoded (quantised, then decoded back) length, matching what the reader will substitute in once it starts consulting .norms -- a bound anchored to the exact length would understate the tf component the reader later scores against, letting real matches get pruned. The bound only loosens in the interim (search still reads exact lengths from .lens until Phase 4), never tightens, since the decoded length never exceeds the exact one. Also folds compute_field_avg_lengths into NormsBuilder's avg_length_f32, closing a latent f32/f64 rounding mismatch between it and the old .fstats path.
SegmentReader now unifies its field-length/statistics cache behind a single SegmentNorms enum: the .norms columnar format when present, falling back to the pre-#555 .lens/.fstats pair (read exactly, never quantised -- their .dict was computed against exact lengths) for older segments (#555 Phase 4). Legacy segments are naturally rewritten in the new format on their next merge. field_length/field_stats keep their exact signatures, so every caller (searcher, bmw, per_segment_view, merge_engine, query::term/phrase) is unchanged. Verified end-to-end with a RED-GREEN proof: temporarily un-quantising NormsBuilder::decoded_length reproduces a real Block-Max-WAND soundness violation (a document's actual BM25 score exceeding its precomputed upper bound) once the reader substitutes the quantised length back in, confirming the invariant the previous two commits set up. Also adds legacy-segment and format-precedence coverage, and cross-checks that a .norms-based merge reproduces what a fresh single-segment build would score for unquantised fields.
.norms has fully replaced them: write_field_lengths, write_field_stats, and calculate_field_stats are removed from InvertedIndexWriter (#555 Phase 5). Existing pre-#555 segments keep reading correctly through SegmentNorms::Legacy until their next merge rewrites them. Updates the compound-segment loose-file check and a dictionary lookup-miss test to use .norms instead of the now-unwritten .lens/.fstats, refreshes the segment-file docs (English and Japanese) to describe .norms, and clarifies the merge engine's comment on length preservation to account for the (idempotent) norms quantisation.
This was referenced Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Folds
.lens(per-doc, per-fieldstring + u32) and the derivable.fstatsinto a singlecolumnar, 1-byte-quantised
.normssegment part, matching the LuceneNormsFormat/ Tantivyfieldnormprecedent. Closes #555.What changed
laurus/src/lexical/index/structures/norms.rs): a Tantivy-styletable (exact identity below 40, log-spaced in octaves of 8 steps above that), generated from
its recurrence by a
const fnrather than a hand-copied literal array..normssegment part: magic + version + codec-id header, a doc-id slot map(contiguous fast path, sparse delta-list fallback for gaps), a field directory carrying the
exact pre-quantisation
sum_length/present_count/min_length/max_length(soavg_lengthstays bit-identical to before), and a presence bitmap distinguishing "fieldabsent" from "analyzed to zero tokens" —
BM25Scorer::score'sunwrap_or(avg)fallbackdepends on that distinction.
write_segment_filesnow builds a singleNormsBuildershared by both
write_inverted_index(the#403Block-Max-WAND score-bound precomputation)and
write_norms, instead of two independent traversals ofbuffered_docsthat coulddisagree on a field's length once quantisation entered the picture.
SegmentReader'sfield_lengths/field_statscaches are unified behindSegmentNorms(V1for.norms,Legacyfor pre-existing.lens/.fstatssegments, readexactly/unquantised since their
.dictwas computed against exact lengths). Every caller(
searcher.rs,bmw.rs,per_segment_view.rs,merge_engine.rs,query::term,query::phrase) is unchanged —field_length/field_statskeep their exact signatures.write_field_lengths/write_field_stats/calculate_field_statsare removed. Existing pre-migration segments keep reading correctly through
SegmentNorms::Legacyuntil their next merge naturally rewrites them in the new format.Landed as 5 commits, each independently test/fmt/clippy-clean:
a8d75a5d(quantisation table) →4ff1ae4c(.normspart, additive) →31f27027(score-bound anchoring) →d1f9c78e(reader switch) →b09da6ff(drop.lens/.fstats, docs).Design decisions / deviations from the original design sketch
.dv): a.normsfile is already roughly anorder of magnitude smaller than the
.lens/.fstatspair it replaces, so the whole segmentis read eagerly at
NormsReader::load— the added complexity of a seek-based per-field loaderwasn't worth it here.
SegmentNormshas two variants, not three: aMissingvariant was dropped since itbehaves identically to
Legacywith empty maps (both returnNonefrom every query).Tests
.normsformat round-trip tests (dense/sparse doc ids, absent fields, zero-length fields,exact header stats even when the per-doc column is quantised, corrupt-header rejection).
bounded relative error,
u32::MAXsaturation).NormsBuilder::decoded_lengthto return the exact (unquantised) length, and confirmed twotests fail as a result before restoring it:
score_bound_uses_decoded_length_and_never_tightens_relative_to_exact(writer-side)block_max_bound_is_never_violated_after_norms_quantisation(reader-side, end-to-end: areal flush + a real
TermInfo/BM25Scorerbuilt from it, checked against the actual scorefor a document whose field is well above the exact-quantisation window). Each fixture
document carries its own unique term so its bound is derived solely from its own length —
a shared term would let a short, unquantised document's looser bound mask a long document's
quantisation error.
.lens/.fstats-only segments stillread exactly; a
.normsfile wins over stale legacy files on the same segment).laurus/tests/lexical_norms_test.rs): a merge across segmentswith both long (quantised) and short (exact) fields preserves hit counts; a
.norms-basedmerge of exact fields reproduces exactly what a fresh single-segment build of the same
documents would score.
Scope
Three items were deliberately left out and filed as follow-ups rather than folded into this
format-migration PR:
avg_field_lengthand per-segmentmax_score_factorSegmentNorms::Legacyreader path once.normsis universal (next major)zero tokens; this PR reproduces that behavior faithfully rather than fixing it
Compatibility
New binary opening an old (
.lens/.fstats) index: reads correctly via the legacy path, noreindex needed. Old binary opening a new (
.norms-only) index: finds no.lens/.fstats,falls back to a flat
avg_field_length: 10.0and no per-doc length — scores degrade to a flatranking rather than erroring. Same one-way format contract as #1024.
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --features embeddings-all -- -D warningscargo test -p laurus --lib(1463 passed)cargo test -p laurus(integration tests + doctests, full pass)cargo test --workspace --features embeddings-all(full workspace pass)