[futurepaul]

The second PR of the search stack (on #771): a metered /v1/embeddings slot on the model route. It is what the search capability (next PR) embeds sections and questions with, and what an agent can call itself.

What it adds

• A slot. models.embeddings in a deployment's config is where its embeddings run, plus the space their vectors are in: {provider, model, dimensions, version?}, with key and price as any OpenAI-compatible slot names them.
  • Unnamed, it is BGE-M3 on Workers AI (@cf/baai/bge-m3, 1,024 dimensions), the benchmark's model (bench/search: a search benchmark (brain today, fbrain, DO baseline, LanceDB) #771).
  • A self-hosted deployment names its own OpenAI-compatible POST <base_url>/embeddings, such as Hugging Face's TEI running BGE-M3 on its machines.
  • It is no chat slot. No call names it as one, no route falls back to it, and it may not run a chat slot's model.
  • Deploy binds its key like any slot's, with no deploy change needed.
• A route. POST /api/models/v1/embeddings[?fragment=<name>], signed by an agent: OpenAI's request and answer, plus space: {model, dimensions, version}.
  • The computer's intercept takes it as POST /v1/embeddings.
  • base64 output is supported, since OpenAI's SDKs ask for it.
• One vector space.
  • Every vector is the slot's length and unit length, so a dot product is a cosine.
  • An answer with a vector of any other length, or another count than was asked, is never passed on: 502, and charged, because it was paid for.
  • Changing the model, dimensions or version makes a new space. Whatever keeps vectors must re-embed rather than mix them.
• Metering.
  • Charged after the answer at input tokens, at the slot's model's row of the book. There is a new row for BGE-M3 at $0.012 per million.
  • Tokens are the ones the API reports. Workers AI reports none for embeddings, so they are estimated from the input at one token per 3 bytes: a price, not a bound.
  • Refusals are passed through, uncharged.
• Bounds, refused 400 before anything is sent or charged:
  • 1 to 100 inputs, each within the slot's window (below), none empty, no token arrays.
  • model must be embeddings or the slot's own; dimensions, if named, the slot's (vectors are never cut).
  • encoding_format must be float or base64.
  • A body over 1 MiB is refused 413.
• models::embedded is the call the search capability will make: gated, metered and checked exactly like the route.

Tests

• Core unit tests:
  • slot parsing and its refusals;
  • bounds;
  • requests for both upstreams;
  • answers: order, count, dimensions, zero vectors and duplicates;
  • usage reported or counted;
  • the OpenAI-shaped reply, float and base64;
  • Models::parse for the slot (default, a self-hosted price laid over the book, a chat model refused, an unpriced model refused).
• The fake answers BGE-M3's {shape, data} on its Workers AI face and /v1/embeddings on its OpenAI-compatible face, both with hashed-word vectors.
• The e2e ledger section covers:
  • vectors in order, unit length, its space named, charged at the counted tokens;
  • base64;
  • five refusals, with nothing sent or charged;
  • a 429 passed through uncharged;
  • a person refused.
• Local runs: cargo xtask e2e --only ledger gives 111 passed, 0 failed; cargo xtask check passes.

Decided since (the coordinator, for Paul)

1. Private embeddings. A deployment whose tiers are private embeds only on an embeddings slot marked "private": true, which must be a confidential API, never Workers AI or OpenRouter. Without one, every embeddings call is refused 403, saying why, and the deployment's search will answer by words alone. Its deploy is not refused. This follows Paul's standing rule that Finite Private never falls back to a model that is not private. The e2e's private-tier node checks it.
2. A slot's window. A slot names max_input_tokens (128 to 32,768; 512 unnamed). An input that might be past it is refused 400 and never cut. Since Alex's review, "might be" is embed::tokens_max, a bound for any tokenizer: each character at the UTF-8 bytes of the longest of itself, its NFD and its NFKD, plus 3 for the model's own tokens. It is checked against BGE-M3's tokenizer for every code point, and it is exact at the window on real input. ASCII is a token a byte, so 512 holds 509 bytes. The search capability chunks with the same bound. The app-facing API doesn't change, and total tokens (cost) stay about the same; only the vectors stored grow (pennies on R2).

🤖 Generated with Claude Code

[alexlwn123]

Pass for this stack layer at a1f4efd9bd3a095f0ff787136f79023e4baafd83, against #771 at c842c4cc174ad85aafcc116df1ba6145b5e38194.

The bounded embeddings route preserves the existing agent authentication, spending gates and credential boundary, meters successful responses before validating vectors, and keeps embeddings out of chat routing. The implementation appropriately reuses the existing transport and deployment key binding; I found no blocking regression in this layer.

Validation: cargo test -p fragment-core passed all 525 tests (including the seven embeddings tests). CI check and images check are green; e2e shards and real-Hermes Docker checks were still pending when inspected. I did not run local e2e or paid vendor calls. The external engineering-style document was unavailable at the documented path.

Non-blocking: crates/core/src/embed.rs describes ceil(bytes / 3) as never undercounting tokens, but an average English characters-per-token assumption does not establish that bound. The test verifies the formula rather than comparing it with BGE-M3 tokenization. Describe this as a billing estimate, or substantiate the guarantee with the actual tokenizer and representative short, punctuation-heavy and multilingual inputs before relying on it as an upper bound.

[alexlwn123]

Fail for now at 71951b4568ecd8e7a076c61d113190337550f5ee, against #771 at c842c4cc174ad85aafcc116df1ba6145b5e38194.

The previous head had no blocking findings. The new private-slot guard is appropriately before dispatch and charging, and reuses the existing configuration checks. However, this head now uses the earlier review's unproven billing estimate to enforce a model window, making that concern a correctness issue in the newly promised behavior.

Blocking:

1. crates/core/src/embed.rs:103–104 (max_input_bytes, used by bound at line 206) admits inputs beyond max_input_tokens. Reproduction with BAAI/bge-m3's published tokenizer.json, loaded with Python tokenizers.Tokenizer.from_file and truncation disabled: ("1,2,3,4,5,6,7,8,9,0\n" * 70) is 1,400 UTF-8 bytes, so the default bound accepts it and estimates 467 tokens. The tokenizer produces 771 tokens excluding special tokens (773 including them), exceeding the configured 512-token window. This is ordinary numeric/tabular content. request forwards the entire input without further window validation: a provider enforcing that window must truncate or reject it, rather than this route refusing it before dispatch as promised. Shape/dimension validation cannot detect a vector made from a truncated prefix.

Please enforce the configured window with the relevant tokenizer or a substantiated conservative bound, accounting for special tokens. Apply the same safe boundary to callers' chunk sizing; do not use the average three-bytes-per-token billing estimate as a maximum. Cover prose, numeric/punctuation-heavy and multilingual inputs, including exact-window boundaries. This is a demonstrated admission failure; I did not call a live provider to observe its truncation policy.

Validation: cargo test -p fragment-core passed all 526 tests. CI check, images check, e2e shards and real-Hermes Docker checks were still pending when inspected. Local e2e and paid vendor calls were not run. The external engineering-style document was unavailable at its documented path.

[alexlwn123]

Pass for this stack layer at 6a0ff549ba7745a001492459d717b7bbbc72cd28, against #771 at 6d1c3b679db53118bf694576d76c061d7a5bcd78.

The previous token-window blocker is fixed: admission now uses a conservative decomposition-aware bound with special-token allowance, independently of the billing estimate. The numeric reproduction is refused before dispatch, and the tests cover punctuation, multilingual text, normalization expansion and exact-window boundaries. The earlier misleading billing guarantee is also removed.

The metered embeddings route retains the existing authentication, spending and private-slot gates. Reusing ICU normalization and the existing transport keeps the correction focused; I found no blocking regression in this layer.

Validation: cargo test -p fragment-core --quiet passed all 528 tests. CI images check passed; check, four e2e shards and real-Hermes Docker were pending when inspected. Local e2e and paid vendor calls were not run. The external engineering-style document was unavailable at its documented path.

Non-blocking: the “any tokenizer” guarantee in crates/core/src/embed.rs and docs/api.md is broader than the demonstrated BGE-M3 evidence: it assumes the listed normalization forms and at most three overhead tokens. Document those assumptions for custom slots, and retain a reproducible tokenizer-comparison script before extending the guarantee to other model families. No additional concrete admission failure was established in this review.

[alexlwn123]

Fail for now at 98c1081af2b65f444f14ce2aa79d676e056c6272.

The earlier token-window blocker remains fixed. The three embeddings commits are patch-equivalent to the previously approved head after rebasing onto master. However, that review missed a deterministic failure in the ledger e2e test.

Blocking:

1. crates/e2e/src/lanes/ledger.rs:1132–1143 still requires past the embeddings slot's 512 tokens. For its exact input, ["ok", "word ".repeat(400)], the current embed::bound(...).unwrap_err().message() returns input 1 may be 2003 tokens, past the embeddings slot's 512: split it, as nothing is cut. I ran that input against this head's core library: the e2e's message.contains(why) is false. Consequently the required ledger lane fails even when the route correctly refuses 400 and sends/charges nothing. This is a test regression from the changed refusal wording, not an admission regression.

Update the expected message or retain the intended wording in Refusal::TooLong, then run cargo xtask e2e --only ledger to verify the route and the unchanged no-dispatch/no-charge assertions together.

Validation: all 530 core tests passed; the exact ledger message predicate reproduced false in a temporary executable (removed afterward). CI check and images check passed; e2e shards and real-Hermes Docker were pending when inspected. Full local e2e and paid vendor calls were not run. The external engineering-style document is unavailable at its documented path. The implementation remains focused and reuses the existing transport, gates and ICU normalization appropriately.

[alexlwn123]

Fail for now at 63058fdad1976412425f2c05f22990b6fc5eb580.

The previous ledger-e2e blocker remains. This head only changes the minimum embedding window, its documentation and tests; it leaves the failing assertion and refusal message unchanged. The earlier token-admission blocker remains fixed.

Blocking:

1. crates/e2e/src/lanes/ledger.rs:1132 expects past the embeddings slot's 512 tokens, but crates/core/src/embed.rs:218 produces input 1 may be 2003 tokens, past the embeddings slot's 512: split it, as nothing is cut for its input ["ok", "word ".repeat(400)]. The route forwards that message at cell/src/models.rs:716, so the contains(why) assertion at ledger.rs:1140 remains false and the required ledger lane fails despite correct refusal behavior.

Align the expected substring with the intended refusal wording, preserving the status and no-dispatch/no-charge assertions, then run cargo xtask e2e --only ledger.

Validation: all 531 core tests passed (cargo test -p fragment-core --quiet); the unchanged failure is established by the source trace above. CI check, images check, e2e shards and real-Hermes Docker were pending when inspected. Full local e2e and paid vendor calls were not run. The external engineering-style document remains unavailable at its documented path. The new minimum-window change is small and direct; I found no additional blocker in it.

[futurepaul]

Thanks, Alex. Fixed at the root in 98c1081, with a follow-up in 63058fd.

1. The window holds on real tokens (blocking).

bound no longer uses three bytes a token. embed::tokens_max is an upper bound for any tokenizer an embeddings model runs:

• each character counts the UTF-8 bytes of the longest of itself, its NFD and its NFKD;
• plus 3 for the model's own tokens: <s> and </s> (or [CLS]/[SEP]), and SentencePiece's leading ▁.

Why it holds:

• A tokenizer normalizes first (not at all, NFC/NFD or NFKC/NFKD).
• It then reads tokens of at least one character (SentencePiece unigram, WordPiece) or one byte (byte-level BPE, byte fallback).
• Composing never lengthens UTF-8. I checked every canonical pair, so no normalization makes a character longer than its longest decomposition.

Bytes alone are not a bound either: 30 × U+FDFA is 90 bytes but 122 BGE-M3 tokens, because NFKC turns it into 18 characters. The bound says 993.

Substantiated with BAAI/bge-m3's own tokenizer.json (sha256 21106b6d…dd08, HF tokenizers 0.21, truncation off), counted offline:

• Every Unicode scalar value, alone, in runs of 16, spaced, and after a letter: 4,448,256 strings, none over the bound, equal at worst.
• Your input: 1,400 bytes is bounded at 1,403 tokens, past 512, so it is refused (BGE-M3 reads 773).
• the_window_holds_on_real_tokens carries BGE-M3's real counts for prose, numbers, JSON, pipes, Chinese, Japanese, Korean, Arabic, Hindi, Russian, polytonic Greek, emoji ZWJ sequences, U+FDFA, ligatures and combining marks. All are at or under the bound.
• Exact window: 509 bytes of |a|b|c|\n is exactly 512 BGE-M3 tokens and is accepted; 510 bytes is 513 and is refused. Russian prose at exactly its bound (488) is accepted with a 488 window and refused with 487.

Chunk sizing. The capability's chunker (#783, searchable::chunked) now measures with the same per-character bound, the header and the model's 3 tokens included. Every chunk it makes is one bound takes: every_chunk_is_one_the_embeddings_route_takes asks bound for each chunk of numbers, U+FDFA, Korean and prose.

A slot's max_input_tokens is now 128 to 32,768 (63058fd), so a window always holds any character plus a header. The most one character costs is 33 (U+FDFA); no_character_is_read_as_more_than_char_tokens_max checks that over every code point.

Cost. ASCII counts a token a byte, so BGE-M3's 512 holds 509 bytes, about 115 English tokens. Prose chunks are about 3× smaller than at 1,536 bytes. Billing is unchanged.

The normalization data comes from icu_normalizer, which the cell already linked through url's idna. The release Wasm is 6,190,713 bytes, within the 6.5 MB budget.

Non-blocking. estimated_tokens, its constant and the docs now call it a billing estimate, never a bound.

[alexlwn123]

Pass at 5b643e9d714b768f30ed6ef1fd5b89f46e2b53a6.

The previous ledger-e2e blocker is fixed. This head changes only that refusal fixture: the numeric reproduction is bounded at 1,403 tokens, and the expected substring now matches Refusal::TooLong::message. The route still forwards that message as a 400, and the test preserves its no-dispatch/no-charge assertions. The earlier token-admission blocker remains fixed.

The correction is small and direct, with no new implementation machinery. I found no additional blocking regression in the delta from the previously reviewed head.

Validation: cargo test -p fragment-core --quiet passed all 531 tests, including the numeric bound regression. CI check and images check passed; four e2e shards and real-Hermes Docker were pending when inspected. I attempted cargo xtask e2e --only ledger: the Worker and native builds passed, but I stopped it while the prerequisite Docker image was still building, before the lane ran. Paid vendor calls were not run. The external engineering-style document was unavailable at its documented path.