[yatesdr]

What

This PR shares prefix KV cache between Flare instances and hosts. It builds on #80 (the host tiers in selene/) and #101 (the flattened layout).

• selene (library) gains the pool client and a shared module of definitions used by both sides.
• selene-manager (new crate, binary selene) is an optional sidecar that owns one pool's DRAM and NVMe store.
• Flare gains a small integration: checkpoint export chains, eviction headroom, and pool attach at startup.

Without --prefix-cache-pool, Flare runs selene's host tiers in process as today: no sidecar, nothing new running.

selene --listen unix:/run/selene/default.sock --memory 256G --nvme /nvme/selene --nvme-limit 2T
flare run PACKAGE --prefix-cache-memory 32G --prefix-cache-pool unix:/run/selene/default.sock

Pool addresses take three forms:

• unix:PATH: the same host, using shared memory;
• HOST:PORT: plain TCP;
• tls:HOST:PORT: TLS.

How it works

• One manager per pool. It keeps a live index of what every attached instance holds (GPU, DRAM, NVMe) plus its own store. Its store is an in-process library on selene's disk engine and record format.
• Lookup, claim, copy, release.
  • A request's prefix is looked up while it waits in the queue. A late answer is still taken if the request hasn't started its first prefill.
  • A restore comes from one source: the longest match, then the closest tier.
  • Copies move through shared-memory windows on one host, a TCP or TLS data channel between hosts, or a relay through the manager.
  • Every page is checksummed end to end. A lost race costs a recompute, never wrong bytes.
• Fencing. Memory is reused only after every party has drained or been fenced. One party list per claim records who that is. A failed teardown quarantines the memory instead of reusing it.
• Records never drop mid-chain.
  • Records move GPU → DRAM → local NVMe → the pool manager → dropped. Each tier demotes its coldest records ahead of need, so a new record always has a place.
  • A refusal or a busy manager keeps the record local and retries it.
  • Only the end of the chain drops records: oldest first, or for oversize or I/O errors.
• Shutdown. By default, a stop loses DRAM-only pooled records, as a crash would. --prefix-cache-drain-on-shutdown[=DURATION] opts in to persisting them, most recent first.
• Startup-only registration. Pool memory is registered with CUDA once, before admission, and only if the whole plan fits Flare's host and device budgets. Otherwise the instance runs on its local tiers, with one warning.
• Degrades to local. A protocol mismatch, an untrusted TLS manager, or a manager that's down all leave the instance serving from its local tiers.
• Multi-rank. A single-process instance is a world of one rank. Multi-host tensor-parallel instances use per-rank slices and one rank link. Their Flare startup wiring comes in a follow-up PR.
• Experimental RDMA, behind the rdma cargo feature, as a write-only carrier: one signalled write per window.

Layout

Path	Contents
selene/src/shared/	Wire protocol, framing, rank slices, identity, limits, record format and disk engine, prefix index, regions, TLS, and RDMA verbs. The manager imports these from here; nothing is defined twice.
selene/src/pool/	The client: engine (lookups, claims, one outbound path for exports and puts), connection, transport, cascade.
selene-manager/src/	store/ (records), pool/ (sessions, index, claims, relay), data/ (data channel and RDMA), service and config.
Flare and Helios	Scheduler export chains and headroom, worker hooks, pool attach and budget, flags and docs.

The dependency direction is one-way: the manager depends on selene, and selene never depends on the manager. CI runs both crates' tests.

Validation

CPU, on this commit:

Suite	Passed	Failed
selene, with test-support	all	0
selene-manager	all	0
Helios lib, with runtime-package,server	1,150	0
flare	207	1

• The flare failure is an Orbit repository-download test that hits a cache permission error inside the test container. It's unrelated to this change.
• The rdma --tests checks pass for both crates.
• The root and manager container images build.

GPU (SM120; Qwen TP1 and TP2, and a TP4 MoE model), on the same code before the final rebase:

• Correctness. Every restore checked matched a GPU-resident continuation by its top-20 logprob distribution: local and pooled, from DRAM and NVMe, across replicas, after restarts, and across hosts.

• Restore speed, 32k-token prefix:

  Source	Time
  GPU hit	34 ms
  Own DRAM	64 ms
  Neighbour GPU	130 ms
  Own NVMe after a restart	155 ms
  Recompute	about 2.9 s

  Across hosts over plain TCP, a 30k-token restore takes about 0.6 s.

• Concurrency. Up to 4 concurrent cross-replica restores, with no recomputes.

• Performance against main. Offload off, local tiers and pool all fell within main's own run-to-run spread for TTFT, decode and throughput, on TP1, TP2 replicas and the TP4 model.

• Faults. Manager kill and restart, source killed mid-export, replica restart and TLS refusals were all safe and fell back to local tiers. Manager disk recovery came back intact.

• Soaks. 30–100 minutes with mixed lengths and random restarts: no errors outside injected faults, and flat resource use.

• Bursts on a TP4 MoE model, on this commit:

  • back-to-back restores right after a publication burst: 18 of 18;
  • publishing into a full pool, then restoring: 7 of 7;
  • a 30-minute soak: 0 errors and 0 manager put errors, with a 73.6% continuation hit rate.

• Local tiers keep the same number of records as main under eviction.

Status and limitations

• Pooled mode costs host resources: at 8–16 concurrent requests on TP1, about 25% more instance CPU and about 23% more instance RSS than local tiers. Latency is unchanged.
• RDMA was tested only against software RoCE; no hardware NIC pair was available.
• Multi-host TP startup wiring is a follow-up PR.
• This PR removes --prefix-cache-disk-transport and the prefix_cache.disk_transport setting. The direct (GPUDirect Storage) value was declared but rejected as unavailable, so staged is the only transport and needs no option.
• Windows is not built.