perf(lexical/index): stored field values are written twice per segment (docs + dv) with no opt-out

[mosuka]

Background

Every stored field value is persisted twice per lexical segment: once in the docs part and once in the dv (DocValues) part. There is no way to opt a field out. Found while re-triaging the lexical/index backlog; verified on a6ad63c7.

Both sinks are fed from the same analyzed_doc.stored_fields:

• DocValues, at add time: laurus/src/lexical/index/inverted/writer.rs:494-498 clones every (field_name, value) into doc_values_writer. The rebuild path repeats it verbatim (writer.rs:1854-1866). DocValuesWriter::add_value (laurus/src/lexical/index/structures/doc_values.rs:83-88) has no gate of any kind.
• Stored documents, at flush time: write_stored_documents (writer.rs:1312-1401) serializes the same doc.stored_fields for every buffered document.

Consequences:

• Any stored: true field is duplicated on disk regardless of whether it is ever sorted, faceted, or read as a doc value — including DataValue::Bytes and DataValue::Vector payloads.
• The schema has no doc_values / fast flag to control it: FieldOption variants carry only indexed / stored / term_vectors (laurus/src/lexical/core/field.rs).
• The read side amplifies it further: DocValuesReader::load (doc_values.rs:174-249) materializes every field of the segment into memory on the first doc-value probe, even when the query sorts on a single field.

Note the two encodings are not byte-identical (the docs part is type-tagged, dv is rkyv), so this is duplicated content, not duplicated bytes; the point is the write amplification and the absence of any opt-out.

Proposal

1. Add a per-field doc-values flag to the schema, defaulting to on so existing indexes keep working, and gate writer.rs:494-498 on it.
2. Independently of the flag, make DocValuesReader::load lazy per field — the existing layout already length-prefixes each field record (doc_values.rs:126-162) and StorageInput is Read + Seek, so this needs no format change. (This half overlaps the narrowed scope of perf(lexical/index): columnar packed DocValues #547.)

Acceptance criteria

• A field can be excluded from DocValues without losing stored-field retrieval
• Sorting on one field does not deserialize the other fields' doc values
• Segment size measurably drops for a schema with large non-sortable stored fields
• Existing indexes (no flag recorded) behave exactly as today

Refs #547, #555, #548

Task List

Scope: full implementation, as decided in the investigation comment above. The prior comments' PR-1 content (type-based exclusion, sort-side fallback) is already merged via #1053/#1108. This adds: a latent mixed-segment fallback bug fix (prerequisite for the flag), lazy per-field DocValues loading with bounds checking and deterministic ordering, and the doc_values: bool field option itself with full merge/rebuild/server/binding wiring. DocumentParser's unrelated stored/indexed gap is filed as a separate issue.

Phase 1: Mixed-segment fallback bug fix (read-side correctness)

• Add stored-document fallback to TopFieldCollector::get_field_value on a DocValues miss (Ok(None)/Err), via document_fields
• Add the missing document_fields override on PerSegmentReaderView
• Make FacetCollector::collect_doc's document fetch lazy so a DV-miss document still gets a fallback contribution
• Update has_doc_values/get_doc_value doc comments (semantics + fallback contract)
• Unit tests (mock reader: has_doc_values=true, get_doc_value=Ok(None))
• Integration test with an actually column-absent segment (deleted .dv file)
• cargo test -p laurus / cargo fmt / cargo clippy clean

Phase 2: Lazy loading + robustness (no behavior change)

• Move alloc_bounds from vector::index to util, generalize error wording
• DocValuesWriter.fields -> BTreeMap, sort values_vec by doc_id
• DocValuesReader lazy per-field loading (storage + directory + cache)
• Bounds-check header-declared sizes in DocValuesReader::load
• Remove unused get_field; keep field_names (directory-backed)
• Rewrite doc_values_load_once_per_segment as a stability check
• Lazy-load tests (byte-count based) + bounds-check tests + determinism test
• cargo test -p laurus / cargo fmt / cargo clippy clean

Phase 3: doc_values flag — core wiring

• Add doc_values: bool to 7 field-option structs (not BytesOption)
• Add store_doc_values to InvertedIndexConfig/LexicalIndexConfig
• Add field_doc_values/stores_doc_values to InvertedIndexWriterConfig, gate both feed sites
• Add SegmentReader::doc_values_field_names() descoped: the merge path's detection only ever needed a per-field has_doc_values(field) check, so the enumeration API ended up with zero real callers and was removed rather than kept dead
• New integration tests (flag effect, byte-size comparison)

Phase 4: Merge / field-rebuild path

• MergeConfig::{field_doc_values, default_doc_values}, schema-first resolution
• Wire perform_merge/reconstruct_segment*/rebuild_field_across_segments
• Wire InvertedIndex::writer()/rebuild_field/merge_segment_set
• Merge tests (both field-name orderings, schema-priority pin, no-values-present pin)
• [~] Field-rebuild tests: covered indirectly through the merge-path tests above (rebuild_field_across_segments shares the same reconstruction code). No Engine::update_field end-to-end test was added: Phase 1's stored-document fallback means a doc_values-only change never changes an observable result (unlike term_vectors/analyzer changes, which do), so there is nothing for such a test to assert beyond what classify_change_table and the merge tests already pin

Phase 5: Schema-change classification

• Add classify_doc_values, fold into the 7 lexical arms
• Add cases to classify_change_table

Phase 6: Server surface

• optional bool doc_values on 7 proto messages
• convert/schema.rs both directions
• gateway/convert.rs both directions
• Server conversion tests

Phase 7: Bindings + CLI

• Add doc_values parameter to 7 methods across all 5 bindings
• Add a doc_values prompt to the CLI wizard's two option-builder functions
• Python/Ruby integration tests

Phase 8: Docs

• Update EN + JA docs (schema_and_fields.md, faceting.md, schema_format.md, grpc_api.md, 5x api_reference.md) -- lexical_indexing.md / http_gateway.md / laurus/api_reference.md left unchanged (no term_vectors precedent to mirror there either; reasons noted in the Phase 8 PR comment)
• mdbook build clean for both docs trees

Phase 9: Verification, implementation report, PR

• cargo test --workspace (full suite) -- 106 binaries all green (also caught and fixed one missed doctest)
• cargo fmt --all -- --check / cargo clippy --workspace --all-targets --features embeddings-all -- -D warnings -- both clean
• cargo check on every binding crate (python/ruby/nodejs/php/mcp + wasm --target wasm32-unknown-unknown --tests)
• Direct Python/Ruby scripts confirming the doc_values effect -- covered by Phase 7's integration tests
• Manual backward-compat check against an existing sample schema -- loaded resources/schema.toml (no doc_values key) via Python, committed, searched, and round-tripped through to_toml(); an omitted doc_values correctly resolves to true and existing behavior is unchanged
• File a separate issue for DocumentParser's stored/indexed gap -- fix(lexical): DocumentParser ignores indexed/stored field options #1114
• PR (Closes #1047, Refs #547 #555 #548) -- perf(lexical): add a doc_values opt-out for stored-field DocValues duplication #1115

[mosuka]

Investigation, and a design correction

Investigated at main 78f4c075 before implementing. Two findings change the shape of this issue.

1. A doc_values flag has to default to true, so on its own it improves nothing by default

Faceting already falls back to the stored document when a field has no DocValues column (laurus/src/lexical/search/features/facet.rs:296-305: needs_document is set when any facet field lacks DocValues, and the values are read from reader.document(doc_id) instead).

Sorting has no such fallback. TopFieldCollector::get_field_value (laurus/src/lexical/query/collector.rs:162-170) returns FieldValue::Null when get_doc_value yields nothing. So a field with doc_values: false would keep faceting correctly but sort every document as Null — silently wrong rather than merely slow.

That forces the flag to default to true, which means the write amplification stays exactly as it is for everyone who does not go and tune their schema. The flag is worth having, but it is not by itself a fix.

2. Bytes and Vector can be excluded today, with no API change and no meaningful behavior change

Sorting cannot distinguish them: sort_type_rank (collector.rs:336-352) assigns Bytes and Vector a rank each and there is no inner comparison, so every value of such a field compares equal. Sorting by a binary payload already carries no information.

Faceting is unaffected because of the fallback above — dropping the column just routes those fields through reader.document, which yields the same values.

So excluding Bytes and Vector from the DocValues feed removes the duplication of the largest payloads — embeddings and binary blobs — while leaving observable behavior intact.

Revised plan: two PRs

PR-1 (no API change)

• Stop feeding DataValue::Bytes and DataValue::Vector into DocValuesWriter (laurus/src/lexical/index/inverted/writer.rs:494-498 and the rebuild mirror at :1854-1866).
• Give sorting the same stored-document fallback faceting already has, so a field without a DocValues column sorts correctly (just more slowly) instead of collapsing to Null. This also removes the asymmetry above on its own merits.
• Deterministic gate: segment bytes for a schema with a large stored binary field, before and after.

PR-2 (the flag)

• doc_values: bool on the eight option structs, defaulting to true, with serde defaults, propagation to the five language bindings, and docs.
• Safe to opt out of because PR-1 landed the sort fallback first: doc_values: false then costs speed, not correctness.

Sequencing matters here — shipping the flag before the fallback would hand users an opt-out that silently breaks sorting on the field they opted out of.

[mosuka]

Implementation report — PR-1 (type-driven exclusion)

Branch perf/skip-doc-values-for-binary-payloads on main 78f4c075. No API change, no format change, no binding impact.

What changed

is_doc_values_candidate(&DataValue) keeps Bytes and Vector out of the DocValues feed. Both feed sites are gated — upsert_analyzed_document on the ingest path and rebuild_in_memory_index, which would otherwise reintroduce the payloads on the next rebuild after a deletion.

Tests

• binary_payloads_are_kept_out_of_doc_values inspects the serialized DocValues payload: title present, thumbnail and embedding absent, total length below the payload size.
• doc_values_payload_does_not_scale_with_binary_fields is the real gate — the serialized payload is byte-for-byte identical for a 64-byte and a 16 KiB binary field. Bytes on disk are what this issue is about, so that is what the gate measures.

RED proven before the change (the payload contained thumbnail). Inert-proven after: disabling the exclusion turns exactly these two red while the three #557 estimator tests stay green, so the components are separable.

cargo fmt --all --check clean; cargo test --workspace --no-fail-fast all green; CI-identical clippy clean.

Dropped from this PR, and why

The plan above put the sort-side stored-document fallback in PR-1. It is not here. I could not demonstrate it running: a test built on a mixed-type field (some documents Bytes, some Text) produced the same ordering with the fallback disabled as with it enabled, which means the test was not exercising the path I believed it was. Rather than ship code I cannot show is reached, it moves to PR-2, where doc_values: false makes a stored, genuinely sortable field without a column constructible — and therefore testable — through the ordinary search path.

Known unproven consequence

With the exclusion in place, a Bytes value in a sort field resolves to Null (rank 0) instead of Bytes (rank 7). In a field holding mixed types this changes where those documents land relative to documents of other types. Sorting on such a field carries no information either way — sort_type_rank gives every Bytes value the same rank with no inner comparison — but I want it recorded that I could not construct a test exercising the difference, so this is stated rather than demonstrated. PR-2's fallback restores the old ordering.

[mosuka]

Investigation + implementation plan (full scope)

Re-investigated at current main. Two findings update the shape of this issue beyond the two existing comments.

1. The two prior comments' PR-1 content is already merged, via a different issue

The type-based exclusion (Bytes/Vector never reach DocValuesWriter, via is_doc_values_candidate in writer.rs) and — despite the PR-1 report saying it was dropped — the sort-side stored-document fallback (TopFieldCollector::has_dv + fallback in collector.rs) are both already on main, landed via #1053 (PR #1108). That sequencing precondition ("land the fallback before shipping an opt-out flag") is satisfied.

2. A latent bug in the existing fallback, which the flag would make load-bearing

InvertedIndexReader::has_doc_values resolves any(...) across all segments once per collector. TopFieldCollector::get_field_value only falls back to the stored document when has_dv == false; when has_dv == true but the per-doc get_doc_value returns Ok(None) (e.g. a segment lacking the column while another segment has it), the value is silently dropped to Null instead of falling back. FacetCollector::collect_doc has the analogous gap — a facet field considered to "have DV" drops the contribution of any document whose column read misses.

This is already reachable today whenever segments disagree on whether a field has a DV column (e.g. straddling the #1052 fix). It becomes load-bearing the moment doc_values: false exists, since opting out immediately creates old/new segments with different column layouts. Confirmed this can't be reproduced with Bytes (its sort rank collapses with Null either direction, which is why the PR-1 report couldn't get a mixed-type test to fail) — reproducing it requires an actual column-absent segment, e.g. via a deleted .dv file or truly mixed doc_values settings.

Revised plan

Phase 1 (bug fix, no flag yet): fix the Ok(None)-miss fallback in both TopFieldCollector::get_field_value and FacetCollector::collect_doc, routing through document_fields (not document) to avoid a full-document clone. Add the missing document_fields override on PerSegmentReaderView. This must land before the flag; it is a correctness bug on main today, latent only because no code path yet produces mixed-column segments.

Phase 2 (no behavior change): make DocValuesReader load lazily per field. The on-disk layout already length-prefixes each field record, so this needs no format change — load reads only the directory (name/offset/len) and defers payload deserialization to first access, holding storage + segment name rather than an open handle (matching BKDReader's pattern, since holding a handle would block segment deletion on Windows). Bundled in the same phase: bound the header-declared sizes against the file (currently unchecked — a corrupted data_len header can drive an unbounded allocation) via a crate::util::alloc_bounds module (moved from crate::vector::index::alloc_bounds, generalizing its "vector segment" wording since lexical needs it too), and make the .dv field order deterministic (HashMap -> BTreeMap, matching #1109's Schema fix).

Phase 3+ (the flag): doc_values: bool on 7 of the 8 field-option structs (BytesOption excluded — it's already unconditionally excluded by type, so the flag would be permanently dead there), defaulting to true. Same 3-level resolution pattern term_vectors used (#1110): InvertedIndexWriterConfig::field_doc_values (merge/rebuild replay override) > the field's own schema setting > store_doc_values (index-wide default). Sequencing after Phase 1's fix matters for the same reason #1110 sequenced its default-reconciliation phase first — shipping the flag before the fallback fix would hand users an opt-out that silently breaks sorting/faceting on the field they opted out of.

One deliberate divergence from #1110's term_vectors merge handling: DocValues columns are fully regenerable from stored_fields (unlike term positions, which are not), so the merge/rebuild replay resolves the current schema first, falling back to per-segment detection only for schema-less fields — not detection-only. This keeps doc_values: true -> false (classified MetadataOnly) honest: the bytes are actually reclaimed on the next merge/optimize instead of being silently kept forever.

doc_values is defined as "copy the stored value into the column store too" — a subset of stored. stored: false + doc_values: true is silently ignored (not an error), consistent with the existing indexed: false + term_vectors: true precedent.

Explicitly out of scope (separate issue)

DocumentParser (the public add_analyzed_document entry point's parser) ignores stored/indexed entirely and always populates every field into stored_fields — same class of bug as #1083's term_vectors no-op, but on a different code path and requiring a constructor signature change to fix. This issue's new doc_values flag is unaffected (the write-path gate lives in upsert_analyzed_document, downstream of both entry points), so it's being filed and fixed separately rather than folded in here.

Detailed phase-by-phase file list and test plan are kept locally and will be summarized in the implementation report at PR time.

[mosuka]

Phase 1 done: mixed-segment fallback bug fix

Confirmed a latent bug while investigating this issue: InvertedIndexReader::has_doc_values is an index-wide any(...) across segments, not a per-document guarantee. TopFieldCollector::get_field_value and FacetCollector::collect_doc both treated a get_doc_value miss under has_dv == true as "the value is Null" / "no contribution" instead of falling back to the stored document. This is unreachable via the existing "no segment has the column at all" test (already fixed by #1053), but is very real whenever segments disagree on whether they have the column — which mixed old/new segments, or a future doc_values: false field, will do routinely.

This had to be fixed before the doc_values flag lands, since the flag would otherwise surface it constantly.

Changes

• laurus/src/lexical/query/collector.rs: TopFieldCollector::get_field_value now falls back to reader.document_fields(doc_id, &[field_name]) on any get_doc_value miss (Ok(None)/Err), not just when has_dv is false index-wide. Uses document_fields (not document) to avoid cloning every field of a wide-schema document.
• laurus/src/lexical/index/inverted/per_segment_view.rs: added the missing document_fields override on PerSegmentReaderView (it was falling through to the trait default, which forwards to document() and clones every field).
• laurus/src/lexical/search/features/facet.rs: FacetCollector::collect_doc's stored-document fetch is now lazy (get_or_insert_with, at most once per collect_doc per perf(lexical/facet): replace HashMap<FacetPath, _> with interned id-based counters #409's existing invariant) and triggers on any DV miss, not only "no column at all".
• laurus/src/lexical/index/inverted/reader.rs: doc comments on get_doc_value/has_doc_values now spell out the index-wide-vs-per-document distinction and the fallback contract. No behavior change here.

Tests

• Unit: new mock readers (DocValuesMissMockReader in collector.rs, DvMockReader::with_dv_miss in facet.rs) reproduce has_doc_values == true with a per-document get_doc_value miss — the exact case the old mock (has_doc_values == false index-wide) couldn't reach.
• Integration (laurus/tests/lexical_doc_values_mixed_segments_test.rs, new file): builds a real two-segment index (use_compound: false), deletes one segment's .dv file outright via Storage::delete_file, and verifies both the field-sort path (via LexicalStore::search) and FacetCollector (driven directly against the real multi-segment InvertedIndexReader) still produce the mathematically correct result instead of collapsing the affected segment's documents to Null / zero facet contribution.
• All four new/changed tests were RED-proven: temporarily reverting each fix reproduces the exact undercount/misorder described above, confirming the tests depend on the fix rather than passing vacuously.

Verification

• cargo test -p laurus --lib: 1414 passed, 0 failed (no regressions)
• cargo test -p laurus --test lexical_doc_values_mixed_segments_test --test lexical_field_sort_test --test doc_values_test: all green
• cargo fmt --all, cargo clippy -p laurus --all-targets --features embeddings-all -- -D warnings: clean

Moving on to Phase 2 (lazy per-field DocValuesReader loading + robustness).

[mosuka]

Phase 2 done: lazy per-field DocValues loading + robustness

DocValuesReader::load used to deserialize every field's doc_id -> value payload into memory on first access, even when a query only sorted on one field. It's now lazy per field, matching the on-disk length-prefixed layout (no format change):

Changes (laurus/src/lexical/index/structures/doc_values.rs)

• DocValuesReader::load now reads only the header and each field's directory entry (name, on-disk offset, payload length), skipping past each payload via seek instead of reading it. The actual doc_id -> value map is deserialized on first get_value/materialize call for that field and cached (Arc<FieldDocValues> in a RwLock<HashMap<...>>) for later calls. has_field/field_names never touch the payload at all — pure directory lookups.
• Mirrors BKDReader's storage + path pattern rather than holding an open handle: each materialization re-opens the file, so nothing here can block a concurrent merge's delete_segment_files on Windows.
• Added bounds checking on every header-declared size (num_fields, field name length, payload length) against the file's true byte length, reusing checked_capacity/checked_len — moved from vector::index::alloc_bounds to crate::util::alloc_bounds in a preparatory commit and generalized from a vector-specific error message. A flipped byte in a .dv header now surfaces a clean "corrupted" error instead of an OOM abort or an out-of-bounds read.
• DocValuesWriter.fields changed from HashMap to BTreeMap, and each field's values_vec is now sorted by doc_id before serialization, so two writers fed identical (doc_id, field, value) triples in different orders produce byte-identical .dv files.
• Removed get_field (zero callers); kept field_names (now directory-backed, no I/O).
• DocValuesReader::get_value changed from Option<&FieldValue> to Result<Option<FieldValue>> — materialization is now fallible I/O, and the one caller (InvertedIndexReader::get_doc_value) already treated it as fallible.

Tests

• New unit tests in doc_values.rs: directory-only has_field/field_names (no materialization), Ok(None) on a genuinely missing field, byte-count-based laziness (materializing_one_field_does_not_read_another_fields_payload — a small field's read stays under 1KB while a ~1MB sibling field goes untouched until it's actually requested), two corrupted-header rejection tests, and a determinism test (forward vs. reverse insertion order produce identical bytes).
• Rewrote lexical_field_sort_test.rs::doc_values_load_once_per_segment, which pinned a literal "2 opens" tied to the old eager whole-file load. Lazy per-field loading changes the exact count (directory open + one payload open per touched field, per segment), so the rewritten test instead asserts the real invariant: a second, identical search adds zero further .dv opens.
• All new/changed tests RED-proven (temporarily reverting the lazy-skip to an eager read reproduces the byte-count failure; reverting the bounds check reproduces an unhandled read-past-EOF).

Verification

• cargo test -p laurus --lib: 1420 passed, 0 failed
• cargo test -p laurus --tests (full integration suite, 92 binaries): all green, 0 failures
• cargo fmt --all -- --check, cargo clippy -p laurus --all-targets --features embeddings-all -- -D warnings: clean

No behavior change from this phase — same values, same .dv format, only when/how the payload is read changes. Moving on to Phase 3 (the doc_values: bool field option itself).

[mosuka]

Phases 3-5 done: the doc_values flag itself, merge/rebuild wiring, schema classification

Phase 3 — core wiring

• doc_values: bool (default true) added to TextOption, IntegerOption, FloatOption, BooleanOption, DateTimeOption, GeoOption, Geo3dOption. Not added to BytesOption (is_doc_values_candidate already excludes Bytes/Vector unconditionally, so the flag would always be dead there).
• InvertedIndexConfig::store_doc_values / LexicalIndexConfigBuilder::store_doc_values: index-wide default, mirrors store_term_vectors.
• InvertedIndexWriterConfig::{field_doc_values, store_doc_values} and stores_doc_values(field_name): the same 3-level resolution shape as stores_term_positions (bug(lexical): TextOption::term_vectors is a no-op — never read at index-build time #1083) — per-doc override > field's own schema setting > index-wide default.
• Both DocValues feed sites (upsert_analyzed_document, rebuild_in_memory_index) now gate on is_doc_values_candidate(value) && stores_doc_values(field_name), not just the type check.
• Descoped one planned item: SegmentReader::doc_values_field_names(). Once the merge-path detection logic was actually written (Phase 4), it only ever needed a per-field has_doc_values(field) check — the enumeration API would have shipped with zero production callers, so I removed it rather than keep dead code alive just to satisfy the original plan.

Phase 4 — merge / field-rebuild path

Deliberately does not mirror how #1083 handled term_vectors. A discarded term position is unrecoverable, so #1083 could only detect what a source segment already had on disk. A DocValues column can always be regenerated from stored_fields, so here the current schema wins:

• MergeConfig::{field_doc_values, default_doc_values}: the owning index's current per-field settings + index-wide default, populated via a new InvertedIndex::current_field_doc_values() helper (merges config.fields with runtime-added extra_fields, same union writer() already builds).
• reconstruct_segment / reconstruct_segment_with_field_override thread a doc_values_by_field map alongside the existing positions_by_field: schema-declared fields are pre-seeded (so per-segment detection's .entry().or_insert_with() can never override them); anything the schema doesn't mention falls back to detecting each source segment's actual column. rebuild_field_across_segments gained a target_doc_values parameter, seeded the same way target_term_vectors already is.
• Tests: independent per-field state across a merge (both field-name orderings), the current schema winning over a stale on-disk column (RED-proven — reverting the schema seed reproduces the bug), and a value-less field not being misdetected as "opted out" when another segment genuinely has the column.
• No separate Engine::update_field end-to-end test: a doc_values-only change never changes an observable result (Phase 1's stored-document fallback guarantees correctness either way), unlike the existing term_vectors/analyzer update_field tests, which do prove a behavior change. The classification + merge-path tests already pin the mechanics that matter.

Phase 5 — schema-change classification

• classify_doc_values(old, new, old_stored) reuses classify_indexed_only's exact logic (same shape: an off→on transition needs the original value to rebuild from) and is folded via .max(...) into all 7 lexical arms of classify_change (not Bytes, which has no flag). Engine::update_field itself needed no changes — its Reindex/Destructive dispatch already calls the now-updated InvertedIndex::rebuild_field generically.
• classify_change_table extended with Text cases (both directions + the stored: false destructive variant) and one representative non-Text case each for Integer and Boolean, to prove the fold applies uniformly.

Verification

• cargo test -p laurus --lib: 1427 passed, 0 failed
• cargo check --workspace --all-targets --features embeddings-all: clean
• cargo fmt --all -- --check, cargo clippy -p laurus --all-targets --features embeddings-all -- -D warnings: clean
• Every new merge-path assertion RED-proven (reverting the relevant seed/gate line reproduces the exact failure it's meant to catch)

Moving on to Phase 6 (gRPC/HTTP surface).

[mosuka]

Phase 6 done: gRPC / HTTP gateway surface

• optional bool doc_values added to the 7 proto field-option messages that carry the flag (not BytesOption). Tri-state for the same reason as term_vectors (bug(lexical): TextOption::term_vectors is a no-op — never read at index-build time #1083): a plain bool can't distinguish "the client omitted this" from "the client explicitly set it to false", and the engine default is true.
• convert/schema.rs: field_option_to_proto / field_option_from_proto carry doc_values through both directions; unset defaults to true.
• gateway/convert.rs: json_to_proto_field_option reads an optional doc_values key; proto_field_option_to_json's six non-Text arms (previously fixed-shape json! literals) were rewritten to conditionally-mutated objects, matching the existing Text arm's pattern, so doc_values is only emitted when explicitly set — never a stray false.

Tests: proto round-trip (explicit true/false, and unset defaulting to true) for Text and Integer; JSON round-trip with the same tri-state contract, mirroring the existing base_weight tests.

Verification: cargo test -p laurus-server (58 passed), cargo check --workspace --all-targets --features embeddings-all, cargo fmt --all -- --check, cargo clippy -p laurus-server --all-targets -- -D warnings — all clean.

Moving on to Phase 7 (the five language bindings + the CLI wizard).

[mosuka]

Phase 7 done: bindings + CLI wizard

Added a doc_values parameter (default true) to the seven add_*_field schema-builder methods across all five bindings, mirroring each binding's existing term_vectors parameter:

• Python: keyword-only pyo3 signature, inserted before analyzer on add_text_field.
• Node.js / WASM: positional Option<bool>, inserted before the trailing analyzer param (same precedent term_vectors already set there).
• PHP: #[php(defaults(...))] entry plus a plain bool parameter.
• Ruby: extended each add_*_field's get_kwargs type tuple and name list by one Option<bool> slot — add_text_field's tuple grows to 5, still well under magnus's 9-element ScanArgsOpt limit.

add_text_field's extra parameter pushed Python's signature to 8 arguments, tripping clippy::too_many_arguments; added the same #[allow(...)] the vector field builders already carry for the same reason.

laurus-cli's create wizard gets a "Doc values?" Confirm prompt (default true) in both prompt_text_option and prompt_indexed_stored_option.

Tests: new laurus-python/tests/test_doc_values.py and laurus-ruby/test/test_doc_values.rb. Neither binding exposes field-sorted search or faceting yet, so these can't observe the DocValues column itself disappearing on disk (that's covered at the Rust level, Phases 3-4) — what they do cover: doc_values: false keeps a stored field retrievable, doesn't affect searchability, and every field option that carries the flag accepts it in both directions.

Verification: cargo check --workspace --all-targets --features embeddings-all, cargo fmt --all -- --check, cargo clippy --workspace --all-targets --features embeddings-all -- -D warnings all clean. laurus-python's 107 pytest cases and laurus-ruby's 81 minitest cases all pass (built via maturin develop / rake compile).

Moving on to Phase 8 (documentation).

[mosuka]

Phase 8 done: documentation (EN + JA)

Documented doc_values: bool (default true) across both docs trees:

• concepts/schema_and_fields.md: added to the TextOption builder example and option table, plus a note that it applies to every lexical field option except BytesOption.
• laurus/faceting.md: corrected a statement that the opt-out makes inaccurate — "every stored field is written to DocValues" is now qualified as "every stored field not excluded by type or by doc_values: false".
• laurus-cli/schema_format.md: added to all 7 carrying field types' TOML snippets and option tables, plus a new "Common option: doc_values" section spelling out the effective rule (stored && doc_values) and BytesOption's exclusion.
• laurus-server/grpc_api.md: added to the FieldOption oneof summary table, with a note on the tri-state (optional bool) wire contract mirroring term_vectors.
• All 5 bindings' api_reference.md: added doc_values to all 7 add_*_field signatures. While there, fixed a pre-existing documentation bug in laurus-php's reference: addTextField's $termVectors was documented as defaulting to false; the actual (and correct) default is true.

Left unchanged, with reasons: laurus-server/http_gateway.md (no option table, only bare JSON examples), concepts/indexing/lexical_indexing.md (segment-format overview that never mentioned term_vectors either — nothing to mirror), and laurus/api_reference.md (documents SchemaBuilder methods by option-struct name only, never enumerating struct fields).

Verification: mdbook build clean for both docs/ and docs/ja/ (no warnings).

Moving on to Phase 9 (final verification, implementation report, PR).