feat: bring custom analyzer definitions (add_analyzer / TOML schema I/O) to laurus-ruby, laurus-nodejs, and laurus-wasm

[mosuka]

Summary

#1058 added Schema.add_analyzer / Schema.from_toml / Schema.from_toml_file / Schema.to_toml / Schema.to_toml_file to laurus-python, letting Python callers register a custom AnalyzerDefinition (char filters + tokenizer + token filters) and load/save the same TOML format laurus-cli create index --schema accepts. No other language binding has equivalent functionality yet:

• laurus-ruby's docs/comments reference an add_analyzer that does not exist yet.
• laurus-nodejs's Schema already accepts a serde_json::Value directly for related config (see laurus-nodejs/src/schema.rs), so it may be able to reuse the exact same JSON shape laurus-python uses ({"type": "...", ...} per component) with comparatively little new code.
• laurus-wasm's existing add_analyzer is an unrelated, different concept (it registers a runtime WasmJapaneseAnalyzer object, not an AnalyzerDefinition) and would need a distinct API rather than a direct port.

Suggested direction

For each binding, evaluate and implement (as separate PRs, likely tracked as sub-issues of this one):

1. laurus-nodejs: add add_analyzer/from_toml/from_toml_file/to_toml/to_toml_file (or the binding's idiomatic equivalent), reusing the laurus-python implementation's JSON shape and error-handling approach (see feat(python): support custom analyzer definitions and TOML schema loading in laurus-python #1058's PR description for the design rationale, e.g. why tokenizer is required and filters are optional/keyword-only in Python — evaluate whether the same reasoning applies to Node's calling conventions).
2. laurus-ruby: same, adapted to Ruby idioms (e.g. keyword arguments), and fix the stale doc/comment reference to a not-yet-implemented add_analyzer.
3. laurus-wasm: design a distinct API (its existing add_analyzer name is already taken by an unrelated concept), or pick a different method name to avoid confusion.

Decision (2026-09-17)

After investigation, a blocker was found: core's Schema::from_toml/to_toml are #[cfg(not(target_arch = "wasm32"))]-gated, so laurus-wasm cannot call them as-is. Decision: lift the core wasm32 gate (verified against an actual wasm32 build) and deliver all three bindings (Ruby, Node.js, WASM) together in a single PR rather than splitting WASM into a follow-up issue. The new WASM method is named addAnalyzerDefinition to avoid colliding with the existing, unrelated addAnalyzer. Full rationale in the investigation/plan comment below.

Acceptance criteria

• laurus-ruby, laurus-nodejs, and laurus-wasm all have feature parity with laurus-python's Schema.add_analyzer/TOML I/O.
• Stale documentation referencing a not-yet-implemented add_analyzer (e.g. in laurus-ruby) is corrected or resolved.

Task list

• Investigation & plan
• Commit 1: feat(core): allow Schema::from_toml/to_toml to compile on wasm32
• Commit 2: feat(ruby): add custom analyzer definitions and TOML schema I/O
• Commit 3: feat(nodejs): add custom analyzer definitions and TOML schema I/O
• Commit 4: feat(wasm): add custom analyzer definitions and TOML schema I/O
• Final verification (workspace fmt/clippy, core regression, docs lint)
• PR opened — feat: bring custom analyzer definitions and TOML schema I/O to laurus-ruby, laurus-nodejs, and laurus-wasm #1136

Related

• feat(python): support custom analyzer definitions (and TOML schema loading) in laurus-python #1057
• feat(python): support custom analyzer definitions and TOML schema loading in laurus-python #1058

[mosuka]

Investigation

Reviewed the reference implementation (laurus-python, #1058) and all three target bindings.

laurus-python's Schema.add_analyzer takes tokenizer as a required positional argument and
char_filters/token_filters as keyword-only optional arguments (a Python-specific constraint: required args
must precede optional ones). Each component is decoded individually
(py_to_json_value → serde_json::from_value::<TokenizerConfig|CharFilterConfig|TokenFilterConfig>) so errors
can be reported per-component ("invalid tokenizer: {e}", "invalid char_filters[{i}]: {e}"). from_toml/to_toml
are thin wrappers over the core Schema::from_toml/to_toml; from_toml_file/to_toml_file do the file I/O in
the binding layer (GIL released during I/O).

laurus-ruby: no add_analyzer/from_toml family exists yet. The doc/comment references to add_analyzer
already in the codebase are forward references that will become accurate once this ships — no wording fix needed.
rb_to_highlight_options/hash_get_str_or_sym (Symbol-first, String-fallback) is the idiomatic pattern to follow
rather than add_embedder's string-only lookup. No file I/O precedent exists; without_gvl will wrap it.

laurus-nodejs: no add_analyzer/from_toml family exists yet. add_embedder's manual
serde_json::Value field access is the closest precedent, but since AnalyzerDefinition's components are
already tagged serde types, a Python-style per-component serde_json::from_value design works directly.

laurus-wasm: the existing addAnalyzer(name, &WasmJapaneseAnalyzer) is an unrelated concept (registers a
pre-built runtime analyzer) and is not touched by this work.

Blocker found: core's Schema::from_toml/to_toml (laurus/src/engine/schema.rs) are currently
#[cfg(not(target_arch = "wasm32"))]-gated, and the toml crate dependency itself is scoped the same way in
laurus/Cargo.toml. So laurus-wasm cannot call these methods today — this wasn't mentioned in the issue.

Decision

• All three bindings (Ruby, Node.js, WASM) will be delivered together in this single PR — WASM will not be split
  into a follow-up issue.
• The core wasm32 gate on Schema::from_toml/to_toml will be lifted, since the toml crate is pure Rust and is
  expected to build for wasm32 (this will be verified with an actual cargo build --target wasm32-unknown-unknown
  before proceeding; if the toml crate or a transitive dependency turns out not to support wasm32, the fallback
  is to depend on toml directly from laurus-wasm and implement from_toml/to_toml locally there — this will
  be called out explicitly in the commit/PR if it happens).
• The new WASM method is named addAnalyzerDefinition to avoid colliding with the existing, unrelated
  addAnalyzer.

Plan

1. feat(core): allow Schema::from_toml/to_toml to compile on wasm32 — lift the cfg gate in
   laurus/src/engine/schema.rs, move the toml dependency out of the wasm32-excluded target section in
   laurus/Cargo.toml. Verified by a green cargo build -p laurus-wasm --target wasm32-unknown-unknown.
2. feat(ruby): add custom analyzer definitions and TOML schema I/O — add_analyzer, analyzer_names,
   from_toml/to_toml/from_toml_file/to_toml_file on laurus-ruby's Schema, following the
   rb_to_highlight_options-style Ruby idioms and without_gvl for file I/O.
3. feat(nodejs): add custom analyzer definitions and TOML schema I/O — same surface on laurus-nodejs's
   Schema, using per-component serde_json::from_value decoding with indexed error messages.
4. feat(wasm): add custom analyzer definitions and TOML schema I/O — addAnalyzerDefinition, analyzerNames,
   static fromToml/toToml (string-based only, no file variants since there's no filesystem in the browser
   target) on laurus-wasm's Schema.

Each commit includes new tests (mutation-checked: RED before the feature exists, GREEN after), docs updates
(EN/JA api_reference.md), and the relevant per-crate verification (cargo check/make test-laurus-*, or the
wasm32 build + clippy + wasm-pack test --node for WASM). Final verification covers workspace-wide fmt,
clippy -D warnings, and a laurus core regression run.

No PHP work in this PR — the issue's suggested direction only lists Ruby/Node.js/WASM. No test coverage for
Lindera (Japanese) tokenizers since this repo doesn't ship a test dictionary — same approach as the Python test
suite (whitespace/ngram/regex only).

Task list tracked in the issue description.