[yatesdr]

What

This adds eviction and backpressure for Selene's host tiers. It stops checkpoints being lost between the GPU and host memory under pressure, replaces the fixed row and page slabs with one pool of equal slots plus an automatic intake, adds an optional maximum record age, and makes local mode write to disk only when memory needs the room.

It also contains two fixes found while validating on a disk-heavy workload: record files are preallocated so direct writes overlap, and a parked record's ancestors are written to disk before it.

Changes

1. One pool of equal host slots.
   • Host memory was two fixed slab classes (a quarter for rows, the rest for pages), so a row-heavy model filled its rows while most page memory sat empty.
   • Now there is one mapping of page-sized slots. A row takes as many slots as it needs, laid out per rank exactly as pages are, so one resolver serves split, registered and pool addresses.
   • The on-disk record format and checksum values are unchanged, so existing records recover. A record written by the previous two-rank format is covered by a test.
   • The memory tier and the disk tier's staging now share this one pool. The separate fixed staging pool is gone.
2. One host-memory setting, plus an automatic intake.
   • The setting is the warm-tier size, and 0 is allowed.
   • On top of it, an intake buffer receives GPU publications. It is sized at startup from the layout and the selected concurrency (room for two rows per active request and two transfer windows). It is shrunk to fit the host budget, never refused.
   • Allocation keeps the intake free by evicting the coldest clean records on demand, so staging never waits on a warm tier full of durable copies.
   • The coldest dirty records are written to disk (or retired to the pool) once free plus clean slots fall below two intakes. At most about two intakes of the coldest records are written ahead of need, so a large-memory host does not write warm records early.
   • This replaces the earlier headroom rule, the memory-only pool retirement rule, and the rows-against-concurrency warning. Startup now warns once, only when the intake had to shrink, and says how much memory it needs.
   • Every store gets at least two windows of intake.
   • Behaviour change: a disk-only instance now reserves a larger intake (about 0.5-8 GB of host memory depending on model and concurrency, where it used 0.2-0.7 GB).
3. Every tier combination works. Warm or none, disk or none, pool or none. Pool-only allocates no local pool. Multi-host with a warm tier is still refused. With nothing configured the cache stays disabled.
4. Local mode spills on pressure.
   • It no longer writes every record through to disk at publish. It uses the parked/spill path that pool mode already uses.
   • The opt-in --prefix-cache-drain-on-shutdown now covers local mode.
   • Behaviour change: by default, a stop or crash loses records that were only in memory, the same rule as pool mode.
5. Optional maximum record age.
   • Set with --prefix-cache-max-age DURATION, [prefix_cache] max_age, or the manager's --max-age. It is off by default, and off means no extra work.
   • Records unused for that long are dropped from memory, disk and the manager store. The sweep runs at most once a minute, oldest first, for at most 64 records, and stops at the first fresh leaf.
   • After a restart, disk records that were reused but never rewritten can expire up to one max age early. That costs a recompute, never a wrong answer.
6. Metrics: host_cache_expired_total{tier}, a reason label on retry_abandoned, and slot metrics (slots in use, slots total, intake slots) in place of the row gauge.
7. Restore cost estimate. A fetch's parallel disk reads are now timed as one transfer. Before, each read was timed separately, so a restore looked about twice as expensive as it was and could be refused as slower than prefill.
8. Preallocate each record file.
   • Record files were written with O_DIRECT into a fresh, never-sized temporary file, so every write extended it. On ext4 an extending direct write holds the inode lock exclusively, which serialized the disk ring's chunks at queue depth 1.
   • On a host whose record NVMe occasionally loses a completion interrupt (the kernel recovers after its 30 s I/O timeout), one lost interrupt froze the whole disk pipeline and publications were lost.
   • Create now allocates the record's final length with fallocate (Linux only; a filesystem without it behaves as before). Every direct write is then an overwrite, which runs under the shared lock, so writes overlap. The temporary is removed if preallocation fails.
9. Write a parked record's ancestors to disk before it.
   • Parked records spill coldest first, so a hot shared ancestor (the root every conversation shares) was never written while its children were. The disk held ready children under an unwritten parent, and a republish at that key was retried until its GPU snapshot was evicted and counted as abandoned.
   • Spill now writes the root-most ancestor that has not committed instead of the record, and the record waits for it. Ancestors started this way count toward the pressure target and the in-flight cap.
   • The shutdown drain starts the ancestor and waits for it, or gives the record up if the ancestor cannot start. A parent that fails twice, or whose file fails to commit, drops its parked children.
10. Test-only: directory-lock acquisition in tests retries a lock briefly inherited by another test's child process. One shared helper replaces a duplicate retry.

Behaviour changes

• Local mode no longer writes at publish; by default a stop or crash loses records held only in memory. Use --prefix-cache-drain-on-shutdown to keep them.
• Disk-only instances reserve a larger host intake.
• Max age is new and off by default.
• Startup warns once, only if the intake had to shrink.
• A restart without the shutdown drain restores the same as before; no restart benefit is claimed for the ancestor fix.

Testing

DeepSeek V4.1 Flash, TP4, on an SM120 host with 4x RTX PRO 6000. Bursts are 3k-token conversations, 32 concurrent, 128 per run. "Abandoned" means publications lost between the GPU and the host.

Test	Result
Earlier version of this PR (before the two fixes), 8G burst	lost publications in 5/8 runs (up to 115); main 0/6
Same version, disk-only	lost in 6/6 (up to 590); main 1/6 (45)
With preallocation, disk-only	8/8 runs: 0 abandoned, 128/128 restored; 7 of them hit an NVMe completion timeout
Same window, build without preallocation, disk-only	3/4 runs lost (376, 69, 350)
With preallocation, 8G burst	6/6: 0 lost during the burst
With ancestor-first spill, 8G burst and disk-only	4/4 and 3/3: 0 abandoned in burst and probe phases, 128/128 restored (before: 16-19 harmless duplicate inflight abandons per burst run)
Restart without the shutdown drain	62 restored / 65 partial / 1 recomputed, same as without the fix
Combined build (all six open Selene PRs on main): smoke	tool calls with thinking 5/5 single and 8/8 parallel, arithmetic 20/20, 25.5k-token repeat 1.68 s to 0.06 s, host-cache hit after restart
Combined build: 8G burst x2, disk-only x1	0 abandoned, 128/128 restored
Unified host memory, 20-minute Qwen soak	0 errors, flat RSS
Restore oracle (memory, disk, pool), DeepSeek and Qwen	all pass
Qwen 4G and 16G bursts	0 abandoned (main lost 4 in one 4G run)
Qwen throughput, 1-16 clients	equal or higher than main

The slot pool, intake and soak results were measured on the revision before the two fixes; the fixes change only record-file creation, spill order, the shutdown drain and the drop of a failed parent's parked children. The disk-only and 8G burst rows above were rerun on the final revision.

• Unit tests (Linux): selene 366 pass. They cover multi-piece rows across the tier matrix, a record written by the previous format, the intake and floors, preallocated length, a cold child waiting for its parked parent, the drain when an ancestor cannot start, and a parent that fails twice. helios passes. selene-manager passes except two kill-process multi-host tests that need a non-container host environment.
• The upstream page_only_nodes_supply_pages_but_never_match test passes unmodified.

Notes

• Drop counts and restore behaviour depend on the restore cost estimate. The separate PR Selene: let a slow transfer-rate estimate recover without new samples #155 (rate-estimate recovery) fixes a sticky slow-disk estimate that can refuse restores after one slow read.
• The NVMe completion timeouts come from the test host's NVMe path (they continued after a drive firmware update, so the drive, platform or driver is not yet known). Preallocation makes the pipeline tolerate them rather than removing them.
• Rows become several pieces (for example 23 for DeepSeek, 135 for Qwen TP1), and a row smaller than a slot uses one whole slot. This is documented in the prefix-cache docs.