fix(lexical): honor indexed/stored field options in DocumentParser

[mosuka]

Summary

DocumentParser::parse ignored the schema entirely, unconditionally indexing and storing every field regardless of its FieldOption. The production ingestion path (InvertedIndexWriter::analyze_document) already gates on (indexed, stored); DocumentParser did not, even though InvertedIndexWriter::add_analyzed_document's own doc comment recommends parsing through it for "explicit control".

Investigation confirmed DocumentParser is currently unreachable from any production path (only 3 doctests + 3 unit tests exercised it), so this carried no runtime risk today -- but it's a public, re-exported type, and the doc-recommended usage pattern would have silently bypassed schema enforcement for anyone following it.

What changed

Non-breaking API. DocumentParser::new(analyzer) is unchanged and still means schema-less (index everything) -- every existing caller (3 doctests, 3 unit tests) needed zero changes. A new with_fields(fields: HashMap<String, FieldOption>) consuming builder attaches the same map shape InvertedIndexWriterConfig::fields already uses, switching the parser into schema-aware mode. A new private field_flags(field_name) mirrors analyze_document's exact resolution: a declared field follows its own option; an internal (_-prefixed) field or any field while schema-less defaults to index-and-store; a field absent from an otherwise non-empty schema is skipped entirely.

No delegation to analyze_field_value. Compared the two implementations directly: parser.rs's private tokens_to_analyzed_terms groups by term text (one entry per unique term, summed frequency), while writer.rs's groups per token occurrence (one entry per occurrence, running frequency counter); DateTime indexing text/point precision also differs. Delegating would change DocumentParser's output shape, risking the backward-compat requirement. So this PR only ports the (should_index, should_store) gating decision into parser.rs, leaving its existing per-type term-generation code untouched -- restructured only to gate on the resolved flags, with every arm's stored_fields.insert collapsed into one call at the end of the loop (each arm's reconstructed value was already equivalent to field.clone()).

Centralized a growing duplicate match. The FieldOption 8-arm (indexed, stored) derivation was already duplicated in InvertedIndexWriter::analyze_document and Engine::update_field's rebuild path (indexed only). Adding a third copy here crossed the threshold to factor it out, mirroring the FieldOption::doc_values() precedent from #1047: new pub(crate) fn indexed(&self) -> bool / fn stored(&self) -> bool on FieldOption, with both existing call sites refactored to use them (pure refactor, no behavior change, first commit).

Tests

7 new cases in parser.rs: indexed: false (terms + lengths excluded), stored: false, point-value gating, Bytes.stored, schema-less passthrough (the backward-compat pin), schema-undeclared-field skipping, and _-prefixed internal fields bypassing the schema. Plus a direct unit test for the new FieldOption::indexed()/stored() accessors covering all 8 variants. Every correctness-sensitive assertion was RED-proven (temporarily short-circuited field_flags to always return schema-less and confirmed the exact expected failures).

Docs

Added a "Schema Awareness" module section and a third doctest demonstrating with_fields(). Updated add_analyzed_document's example (previously schema-less) to use with_fields(config.fields.clone()), since that doc actively recommends this exact usage pattern.

Scope note (not fixed here)

While comparing parser.rs and writer.rs::analyze_field_value, found the term-frequency/position and DateTime-representation divergence described above is a real, separate bug for anyone actually using the DocumentParser::parse() -> add_analyzed_document() pattern (repeated-term frequency ends up wrong). Left out of this PR since fixing it changes DocumentParser's output shape and risks the backward-compat criterion here; noting it in case a maintainer wants to track it separately.

Test plan

• cargo test --workspace --features embeddings-all -- 106 test binaries, all green
• cargo fmt --all -- --check -- clean
• cargo clippy --workspace --all-targets --features embeddings-all -- -D warnings -- clean
• cargo test -p laurus --doc lexical::core::parser and the add_analyzed_document doctest -- both compile/pass
• Every new correctness-sensitive assertion RED-proven by temporarily short-circuiting the gate and confirming the exact expected failure

Closes #1114