[yatesdr]

What

This PR makes the host cache tiers (DRAM and disk) hold far less data for the same conversations, and stops the GPU from evicting snapshots it could keep.

Changes

1. Drop host rows a conversation has moved past.
   • Every host record kept its row (the resume state at that exact point). A growing conversation only resumes at its tip, at branch points and at roots, so the rows of interior records were dead weight: on a long agent conversation rows were 93% of the bytes copied.
   • A non-root record with exactly one committed child in its tier now drops its row and keeps its pages once the child is ready (in memory, in the composite tier and on disk, where it is a truncate of the row segment). Leaves, branch points and roots keep their rows.
   • Removal waits while anything still reads the row. Lookup uses row-less records only as page carriers, so a restore never uses a missing row; a later fork or rewind resumes at the nearest record with a row and prefills the rest.
   • Recovery drops page-only records with no children, which can only appear after a crash or a drain timeout.
   • Metric: flare_host_cache_rows_thinned_total{tier}.
2. Store TP-rank-identical record data once.
   • For layouts that replicate their cache across tensor-parallel ranks (DeepSeek V4.1 TP4), the ranks' shares of a record are byte-identical, so 75% of host bytes were duplicates.
   • At copy-out Selene checks, separately for a record's pages and its row, whether every local rank's share is identical (a memcmp of the unpadded bytes, then one hash). If so, one share is stored, the record header (version 2) marks it replicated, and restore copies it to every local rank after verifying its checksum. Otherwise the record is stored per rank as before.
   • The decision made in DRAM is carried to disk. The disk-only path decides from the first window; a later window that differs abandons that one store (counted, never a partial file).
   • Version-1 records still recover. TP1 and head-sharded layouts are unaffected. Pool publications and memory-only split pools stay per rank.
   • Metrics: flare_host_cache_packed_records_total{tier}, flare_host_cache_per_rank_records_total{tier}, flare_host_cache_abandoned_stores_total.
   • Also: a row whose truncate submission fails is no longer counted as thinned or dropped and stays usable; pool announcement refuses packed records explicitly.
3. Offer the host store only checkpoints a request may resume.
   • On a growing conversation about four checkpoints are published per turn and the next request resumes at one. A message checkpoint is shadowed a few tokens later by its own prompt checkpoint, and a reply-end checkpoint is rarely resumed by clients that re-render the reply.
   • The scheduler now defers the host offer (never the GPU capture) of a message checkpoint whose prompt checkpoint follows less than one page later, and of terminal checkpoints. A deferred checkpoint is offered when a request resumes exactly at it, when its request is cancelled or fails, or (for a message checkpoint whose prompt capture got no snapshot) when it is resumed while pinned.
   • The parent reported for a publication is the nearest offered ancestor, with the tail pages since it, so announced chains stay connected. Captures, prefill cuts, GPU retention and eviction order are unchanged.
4. Bound GPU snapshot rows by memory, not a fixed count.
   • Snapshot capacity was eight rows per concurrent request (256 at concurrency 32) regardless of memory. On a long agent conversation the GPU evicted 94% of the checkpoints it created while 55-73% of its KV pool sat free.
   • Capacity is now whatever the borrowable KV backing can alias, which the planner already caps by memory and virtual extent. Rows and request pages still yield to each other through the existing pressure eviction. The per-request capture budget stays at most eight, so prefill cuts are unchanged.
   • No package re-export is needed. The design docs are updated.

Behaviour changes

• Host tiers hold much less. Interior records keep pages only, and replicated data is stored once. Only a leaf, a branch point or a root has a row, so a fork or rewind to an interior point prefills from the nearest record with a row.
• Fewer checkpoints are offered to the host store. Deferred ones are offered if a request resumes at them, so resume points are unchanged.
• Version-2 record files. Older binaries treat them as invalid and remove them; this build still reads version 1.
• GPU snapshots can use most spare KV memory. At high concurrency they took 98% of KV pages, with free pages near zero and no preemptions. Snapshots are evicted for active requests, so it behaves as a cache, but free KV pages no longer indicate headroom.
• Four new host cache metrics (above).

Testing

DeepSeek V4.1 Flash, TP4, on an SM120 host (4x RTX PRO 6000), production-style flags (MTP 5, 32 slots, 16 GiB host memory, 120 GB disk limit, 60 s shutdown drain). Base is the parent commit (upstream main plus #141/#142). Single conversation: 170 turns to about 130k tokens, once with a client that echoes the reply back unchanged and once with a client that re-renders it. Concurrent: 32 conversations for 30 minutes.

Single long conversation

Measure	Base	This change	Base, re-rendering client	This change, re-rendering client
Host memory used at end	9,026 MB (681 records)	263 MB (510)	9,067 MB	250 MB (341)
Disk after the shutdown drain	8.86 GB	0.15 GB	8.90 GB	0.15 GB
GPU snapshots resident / capacity	256 / 256	681 / 1991	256 / 256	681 / 1991
GPU snapshot evictions	425	0	425	0
KV pages holding cached state	13%	34%	13%	34%
Records packed / per-rank	0 / 0	509 / 1	0 / 0	341 / 0
GPU-to-host copy per turn	51.5 MB	39.3 MB (-24%)	51.7 MB	27.1 MB (-48%)
TTFT p50	0.302 s	0.301 s	0.295 s	0.296 s

32 concurrent conversations, 30 minutes

Measure	Base	This change
Requests completed / errors	3,388 / 0	3,504 / 0
Throughput	142.6 gen tok/s, 113 req/min	137.6 gen tok/s, 117 req/min
TTFT p50 / p99 (queue-dominated)	6.08 s / 68.2 s	5.56 s / 70.6 s
Host memory	full (15.0 GB, 1,151 records)	7,582 records in the same 15.0 GB
Disk written	125 GB (hit the 120 GB limit after about 25 min)	2.7 GB
GPU snapshots evicted	13,315 of 13,571	12,309 of 14,037
KV pages holding cached state, max	13%	98% (no preemptions)
Records packed / per-rank	0 / 0	12,531 / 1,346
Abandoned stores / failed stores / failed fetches	0 / 0 / 0	0 / 0 / 0
GPU-to-host copy	162.7 GB	141.9 GB (-13%)
Shutdown drain (60 s budget)	9 of 1,061 records written, the rest skipped: disk full	6,991 of 6,991 in 20 s

Idle TTFT of a cached turn at 130k tokens is 0.30 s on both sides, and engine time, GPU wait on the host and probe latency are the same, so no scan or packing cost shows at this scale. The packing time itself is not exposed as a metric and was not measured directly.

Correctness (this change only)

• Burst of 128 restores of 3k-token prompts at concurrency 32: 128/128 restored, 0 recomputed, 0 errors, three runs (memory twice, disk-only once).
• Restore oracle at 12k tokens, 10 trials each: memory path 10/10 and disk path 10/10 pass the logprob gate and resume at the reference point.
• Restart: 64 conversations of 20k tokens, SIGTERM with drain, restart on the same disk directory. The drain took 11.3 s (258 records, 1.55 GB); after the restart 64/64 conversations restored from disk with their whole prompt cached, TTFT p50 0.06 s, 0 abandoned stores.

Unit tests (Linux, upstream CI commands): selene 388 pass (with test-support); selene-manager passes; helios 1191 pass and 7 fail, the same 7 as on main (a DeepSeek tokenizer-fixture defect). New tests cover chain thinning in memory, composite and disk, forks and rewinds restored through disk, identical, mixed and differing shares, multi-window disk-only spills in any order, abandoned stores, recovery including a version-1 file written by the previous code, deferral at page boundaries, offer on reuse, capture-failure and preemption paths, compaction with a deferred heir, and capacity growing with memory.

Notes

• GPU-to-host copy bytes fall only 13-48%, not by an order of magnitude, because every rank is still copied to the host before deduplication; the saving is in what the host tiers keep, not in what crosses the bus. Deduplicating before the copy is a possible follow-up.
• The test host reads its embedding tables from disk (it lacks the memory to hold them). Occasional 30 s stalls in those reads, which coincide with NVMe command timeouts, affect both builds; they dominate the 30 s TTFT p99 in the single-conversation runs and are not a tiering effect. With the stalled requests removed, p99 is 0.385 s against 0.387 s for base.