[futurepaul]

PR 3b of the search stack, on #777. It adds the capability: a fragment's own index, served by the search service (#777) beside the deployment. It is opt-in, and a deployment that names no service is unchanged.

What it adds

• Routes (docs/api.md, "A fragment's search"; wire types in fragment_proto::search):

  • POST /api/f/{name}/search/upsert, /delete: an editor writes or removes a source whole.
  • GET …/search/sources?after=&limit=: an editor pages its sources, ascending (for reconciliation).
  • GET …/search/status: a reader sees what the index holds (semantic, forgetting).
  • POST /api/f/{name}/search: a reader searches; hits carry source, section, location, snippet and signals.

  The shapes are the ones the Session Archive's writer (sessions: an import writes its archive's search #774) already sends.

• Scope and roles.

  • The index is the service's table for f<object id>-<incarnation>-members. The cell derives it; no caller names it.
  • Roles are checked before anything is read or sent: viewers search, editors write. A stranger or a removed member is refused.

• Embedding.

  • Sections are cut into chunks no larger than the embeddings slot's window (models: a metered embeddings slot, one vector space #772's max_input_tokens), so nothing is truncated.
  • Each chunk's input (title, context, heading, then text) is embedded once, through models: a metered embeddings slot, one vector space #772's models::embedded. A chunk the index already holds (known, by the input's sha256) isn't re-embedded, so a grown conversation embeds only its new turns.
  • Writes are paid by the fragment's owner and questions by the asker (an agent's owner).
  • Anonymous readers search by words alone. So does a deployment whose tiers are private with no private embeddings slot (semantic: "off"), per the coordinator's decision for models: a metered embeddings slot, one vector space #772.

• Stale writes. replaces is a compare-and-set. A mismatch is 409 with the new, additive ErrorCode::Conflict (conflict), naming current; the CLI maps it to its existing conflict code.

• Deletes follow the files. When a move of main deletes a file whose path is a source, the source is forgotten:

  • recorded in search_forget, the Fragment's schema step 4, in the move's turn;
  • left out of every answer at once;
  • sent to the service by the alarm (retried every 5 minutes if the service is down);
  • gone from the service's store within its purge window (search: the portable search service (LanceDB on R2, S3 or a disk) #777).

• A life's index ends with it. A deleted or wiped fragment's index is ended in clear_ended (the service's end: tombstoned, then dropped), with its database and blobs, under that machinery's retries. A write re-checks its life after each wait, so one the end overtook writes nothing and marks nothing of a life made again under the name.

• One source, one send at a time. A write of a source sends that source's pending forget first, whatever its retry time, and a drain never sends a source a write holds. So no forget lands after a later write.

• Deploy and dev.

  • A config's optional search: {url, token} sets FRAGMENT_SEARCH_URL (https only) and binds SEARCH_TOKEN; the token is never a variable.
  • devstack seeds fragment-search-token.
  • A search upsert's body may be up to 4 MiB; other /api/f bodies stay at 2 MiB.
  • On Cloudflare this needs the service at a URL until 3c runs it in Containers on R2. A self-hosted deployment runs it beside celld today (search/README.md).

Tests

• Core: checks and limits, chunking that keeps every word (paragraph, then sentence, word, character), chunk naming by input hash, and the service body. Migrate: step 4 migrates from step 3, a restart runs nothing, and a remake empties it. Deploy: the URL variable and token binding, and a bad URL refused. CLI: the new code's mapping.
• e2e search section (shard 1; the real service image in Docker on a disk store), 9 checks, all passing locally:
  1. Valid write, found by words and by meaning, with location and status.
  2. A replay writes and embeds nothing; a grown source embeds only its new turn.
  3. A stale write gets 409 conflict naming current.
  4. Limits are refused (400 or 413) with nothing embedded.
  5. A viewer reads but doesn't write, a stranger is refused, and a removed member is refused.
  6. A pushed delete of a source's file forgets it: gone from search, from sources, and from the service.
  7. A restarted service still searches what was written.
  8. A deleted fragment's table is dropped from the service's store.
  9. A private deployment works by words alone, with nothing sent to embed.
• CI: shard 1 builds the service's image with a BuildKit cache (scope e2e-search) and has a 50-minute limit (others stay at 20). A cold LanceDB compile on a runner takes about 25 minutes, and only when search/ changes.
• cargo xtask check passes.

The R2 one-off (waiting on Paul's OK)

This proves #777 on R2 itself (conditional-put commits, cold reads), with synthetic data only, from this machine. Steps on the claude-* preview account:

1. wrangler r2 bucket create fragment-search-test
2. Make an R2 API token, Object Read & Write, scoped to that bucket alone (dashboard: R2 → Manage API tokens). Its S3 access key id is the token's id; its secret is the token's secret.
3. In search/:

   SEARCH_TEST_STORAGE=s3://fragment-search-test/run1 \
   SEARCH_TEST_STORAGE_AWS_ENDPOINT=https://<account>.r2.cloudflarestorage.com \
   SEARCH_TEST_STORAGE_AWS_REGION=auto \
   SEARCH_TEST_STORAGE_AWS_ACCESS_KEY_ID=<id> SEARCH_TEST_STORAGE_AWS_SECRET_ACCESS_KEY=<secret> \
   cargo test --release --test service on_an_object_store -- --ignored --nocapture

   The test writes, reopens, replaces, deletes, purges and drops its own table. A few hundred requests and kilobytes.
4. Cost: within R2's free tier (1M Class A, 10M Class B, 10 GB-month), so $0.00.
5. Teardown: wrangler r2 bucket delete fragment-search-test (the test leaves it empty), then revoke the token.

No secret goes into the Secrets Store for this. That step (secret set, the bucket's keys) comes with 3c, which runs the service in Containers on R2 and is Paul's to approve.

🤖 Generated with Claude Code

[alexlwn123]

Fail for now at a87c220b8384ef6ae8cbda6f35cac4bfc88c5b87, reviewing this layer against #777 at 956d111df1afe379400e9bf20cc9ae7bc07d5097.

The opt-in capability fits the service layer, and the diff's API, deployment, migration and test surfaces are appropriate. Two blocking failures remain in ordering writes against deletion. These are source-traced interleavings, not failures reproduced by the existing e2e.

Blocking:

1. A deferred forget deletes a subsequently successful write. cell/src/searchable.rs:160 treats search_drain as a barrier, but it returns no result and selects only due entries, at most 100 (:373–382). Delete a file while the service is unavailable: its forget is deferred five minutes (:365). Restore/reimport that path after the service recovers but before the retry is due. The upsert skips the pending delete, succeeds, and leaves the outbox row untouched. Queries and sources hide the new revision; the alarm subsequently deletes it unconditionally. The same failure applies to a source beyond the 100-entry drain batch. Resolve the target source's prior forget before accepting its new write, or refuse/retry the write until that ordering is established. Coordinate concurrent drains too; merely removing the local row after an upsert does not cancel a delete already in flight. Cover recovery during backoff and a backlog larger than one pass.

2. An in-flight upsert can recreate an index after its life has been cleaned. search_upsert captures scope/owner, awaits embedding (cell/src/searchable.rs:183), then writes that scope (:186) without a lifecycle fence. While embedding is pending, the owner can delete the fragment (or an operator wipe it); end_life remakes the live tables, and clear_ended drops the index and marks storage cleaned (cell/src/ended.rs:358–369). When embedding returns, the old request creates the old scope again, and no remaining cleanup row is obliged to remove it. Deleted/wiped content can therefore remain in the service indefinitely. Order the final external write against lifecycle cleanup, including writes already dispatched, and prevent stale requests from marking a new life's metadata. Cover a paused embedding and a paused service write across delete/wipe, including recreation of the fragment name.

Validation: all five cargo test -p fragment-core searchable --lib tests and the cargo test -p fragment-core search_forget --lib migration test passed. CI check and images check passed; e2e and real-Hermes Docker checks were still running. I did not run local e2e or the paid R2 proof. The engineering-style file named by AGENTS.md is absent on this machine.

[alexlwn123]

Fail for now at dc73e4e179f0712c7aca0615285be2c7241ad608, against #777 at 108d4eea652dd8cad52f1bdf70f0a7d86ee918d0.

Both blockers from my previous review remain. This head only merges the base's environment-file support and object-store test changes; the capability's write/deletion paths are unchanged. The scope and implementation size remain appropriate for this capability.

Blocking:

1. A pending forget can erase a successful replacement. cell/src/searchable.rs:160 calls search_drain, which selects only due entries, at most 100 (:373–382), and does not signal incomplete deletion. After a file deletion fails to reach the service, its forget waits five minutes. Restore/reimport that source during the backoff: the upsert succeeds, but the retained outbox row hides the new revision and later deletes it. A source beyond the drain batch has the same problem. Establish ordering for the target source before accepting a new write, including already-running drains; cover backoff recovery and more than one batch.

2. A late upsert can recreate a cleaned, ended life's index. cell/src/searchable.rs:183–189 awaits embedding and then writes the previously captured scope and live metadata without a lifecycle fence. Delete/wipe can meanwhile remake the live tables and finish clear_ended (cell/src/ended.rs:350–369). A subsequent upsert with no replaces recreates the dropped table (search/src/store.rs, upsert/create), leaving deleted content without further cleanup. Order external writes against lifecycle cleanup, including dispatched writes, and prevent stale requests from updating a new life's metadata. Cover paused embedding and paused service writes across delete/wipe and name recreation.

Validation: re-traced both failures on this head and inspected the complete delta from my prior reviewed head. These are source-traced failures, not fresh runtime reproductions. Local tests were not rerun for the unchanged capability paths; CI checks and e2e are currently pending. No local e2e or paid R2 proof was run. The engineering-style file named by AGENTS.md is absent on this machine.

[alexlwn123]

Fail for now at e169538feea81ffb48902a6586ffd3e923942819, against #777 at 2fdfc1ae9377a4456c63c417d4924dc54a8c5955.

Both blockers from my previous review remain. The complete delta since that review only adds automatic temporary-directory cleanup in search/tests/service.rs; the capability and service write/deletion paths are unchanged. That small test cleanup is appropriate and adds no material complexity.

Blocking:

1. A deferred forget can erase a successful replacement. cell/src/searchable.rs:161 calls search_drain, which selects only due entries, at most 100 (:373–382), without reporting unfinished deletion. After a file deletion fails to reach the service, its forget waits five minutes. Reimport that source after recovery but before the retry: the upsert succeeds, yet the retained outbox row hides the replacement and later deletes it. A source beyond the drain batch has the same problem. Establish ordering for the target source before accepting its write, including deletes already in flight. Cover recovery during backoff and a backlog larger than one batch.

2. A late upsert can recreate an ended life's cleaned index. cell/src/searchable.rs:183–189 awaits embedding, then writes the captured scope and live metadata without a lifecycle fence. Delete/wipe can meanwhile remake the live tables and finish clear_ended (cell/src/ended.rs:350–369). The resumed upsert with no replaces recreates the dropped table (search/src/store.rs:257–262), leaving deleted content without further cleanup. Order external writes against lifecycle cleanup, including writes already dispatched, and prevent stale requests from updating a new life's metadata. Cover paused embedding and paused service writes across delete/wipe and name recreation.

Validation: inspected the full delta since my previous verdict and re-traced both failures at this head. These are source-traced failures, not fresh runtime reproductions. Local tests were not rerun for the unchanged capability; CI check, e2e, images check and real-Hermes Docker checks remain pending. No local e2e or paid R2 proof was run. The engineering-style file named by AGENTS.md is absent on this machine.

[futurepaul]

Thanks, Alex. Both blockers are fixed in 096aadc, plus #777's b408388. The local e2e search section passes 13/13. I also reran it with the fixes mutated out: no forget-first, no life re-checks, and a plain drop instead of end. With the fixes gone, both forget checks fail and the lifecycle check stops, because the ended life's tombstone never appears.

1. A forget never lands after a later write.

The DO sends one source one thing at a time, using a per-source async lock (search_sending, the same pattern as settling in channels.rs).

• An upsert holds its source from before its first send until its answer. First it sends that source's pending forget, if one exists, regardless of next_at: that covers the 5-minute retry and rows past a pass's 100. If that send fails, the write is refused 502. Only then does the upsert embed and write.
• A drain try-holds each source. If a source is already held, the holder is a write (which sends that forget first) or another pass, so the drain skips it. Otherwise it re-reads the row under the hold, then sends. A forget already in flight therefore finishes before a write of its source starts, and none is sent after a later write.
• search_delete holds its source too.
• The non-barrier drain at the start of an upsert is gone.

e2e (search section):

• Outage backoff. The service is stopped and the file deleted; a read drains the forget, which fails and is deferred five minutes. The service starts again, and the source is written during that wait. It is found at once and forgetting is 0. Before the fix, the row stayed and hid the new revision until the retry deleted it.
• Backlog past one pass. 101 files are deleted in one push, and the 101st source is written again right after. It stays; the other 100 go.

One window the cell can't close: a delete sent by a DO instance that is reset mid-call, which then reaches the service after the new instance's later write. The service takes a scope's writes in arrival order, so this needs the old request delayed in transit for longer than the new instance's start plus its embedding. A per-source generation fenced at the service would close it. I left that out for its complexity; say if you want it.

2. The lifecycle fence.

A write captures its life (CreatedAt) and re-checks it at three points:

• after holding its source;
• after embedding, before anything is sent;
• after the service answers, before setting MetaKey::Search.

A changed life gets 404, and the write marks nothing of a life made again under the name.

For writes already dispatched: a life's end now calls the service's POST /v1/{scope}/end (#777, b408388). It tombstones the scope before dropping its table, so a write landing after the end is 410, whatever the order and across restarts. A scope is one life of one fragment, so it is never needed again.

e2e:

• Held at its embedding. The Workers AI fake gained a deterministic hold (hold_embeddings): the call is recorded and its answer waits. The write is held at its embedding; the fragment is deleted, its index ended, and the name made again; then the hold is released. The write gets 404, the old table is never made again, and the new life has no index.
• Held at the service's door. A forwarding door between the node and the service holds an upsert the cell has already sent. While it is held: delete, the index ended, the name made again; then release. The service answers 410, so the write gets 404. The table is never made again, and the new life's search is empty.

A wipe ends a life through the same end() as a delete (ended.rs: end_life_wiped calls end(false), which records the same ended row that clear_ended ends). So I covered delete plus name recreation, and didn't add an operator wipe to this lane.

Also:

• Chunks are now sized by models: a metered embeddings slot, one vector space #772's bound (chunked measures with tokens_max). Test: every_chunk_is_one_the_embeddings_route_takes.
• docs/api.md describes the forget ordering and the life fence.

[alexlwn123]

Pass for this stack layer at 096aadc8a9b2d3697e82e4667fc02016a6f09c29, against #777 at b40838865cb9bfc550a79a8b3fb52f5ba66f745e.

Both earlier blockers are addressed: per-source serialization and a mandatory target-source forget order replacements after pending/in-flight deletion, including backoff and batches over 100; incarnation checks and the service's persistent end tombstone prevent a late upsert from recreating a cleaned life or marking the next life. I traced these paths and inspected the new regression scenarios for backoff, 101 deletes, paused embedding and a dispatched write across deletion/name recreation.

The opt-in capability remains appropriately scoped. Its API, deployment, migration and test surfaces are justified, and the added synchronization stays local to search. The revised chunker also uses the embeddings route's token bound. I found no remaining blocking regression in this layer.

Validation: all six cargo test -p fragment-core searchable --lib tests and the cargo test -p fragment-core search_forget --lib migration test passed. CI check and images check passed; e2e and real-Hermes Docker checks were still pending. I did not run local e2e or the paid R2 proof. The engineering-style file named by AGENTS.md is unavailable on this machine.

[alexlwn123]

Pass for this stack layer at 1ee3d9adb3b2dff3c3126c1ddc2472ba3af6b376, against #777 at 3214c72a2c25bd3200f4b8c6834eff72007dc798.

Both earlier blockers remain fixed as accepted in my previous approval: target-source serialization orders pending forgets before replacements, and incarnation checks plus the service's persistent end tombstone fence writes across deletion. Those capability paths are unchanged.

I inspected the complete delta since that approval: the inherited embeddings refusal fixture and service pruning changes. The pruning cutoff retains a superseded version for ongoing reads; I checked its strict timestamp boundary against the pinned LanceDB/Lance implementation. The small helper and test adjustments are appropriate in scope and complexity. I found no new blocking regression in this layer.

Validation: all six cargo test -p fragment-core searchable --lib tests and the cargo test -p fragment-core search_forget --lib migration test passed. CI check, images check, e2e and real-Hermes Docker checks were still pending. I did not run the service suite, local e2e or paid R2 proof. The engineering-style file named by AGENTS.md is unavailable on this machine.

[alexlwn123]

Pass for this stack layer at 2cb3a0fe2ac17ba711c4d8a5ca1093fcb1ccab08, against #777 at d5b9a4dd61ab903ef574311c245afce4394a465b.

Both earlier blockers remain fixed as accepted in my previous approval: per-source serialization orders forgets before replacements, and incarnation checks plus persistent end tombstones fence writes across deletion. The capability paths are unchanged since that approval. I inspected the inherited service changes that schedule cleanup before optional index folding, retry failed startup sweeps, and finish interrupted ends; they preserve the capability's deletion boundary. The added retry machinery and regression scenarios are appropriate to that purpose. I found no new blocking regression in this layer.

Validation: all six cargo test -p fragment-core searchable --lib tests and the cargo test -p fragment-core search_forget --lib migration test passed. CI images check passed. CI check failed in front::tests::a_read_is_seen_before_it_is_passed_on; its file is identical to the base branch, so this is inherited rather than introduced by this layer. E2e shards and real-Hermes Docker checks were pending. I did not run the service suite, local e2e or paid R2 proof. The engineering-style file named by AGENTS.md is unavailable on this machine.