Skip to content

Commit f7c468c

Browse files
voipmonitorclaude
andauthored
Fix TP16 host prefix records and retain deep DRAM prefixes (#264)
* Fix TP16 host prefix records and retain deep DRAM prefixes Version 1 and 2 record headers cannot describe the Kimi TP16 geometry: sixteen ranks with 768 planes require 6513 bytes, exceeding the fixed 4096-byte header block and rejecting every host-cache publication. Add record format version 3 with u32 plane sizes, reducing that header to 3441 bytes without changing segment offsets. Preserve version 1 and 2 decoding and byte-identical rewrites, keep version metadata out of semantic header equality, and reject plane sizes that do not fit the new encoding. Keep redundant interior rows while the DRAM pool has more than two row slots of headroom. Resume eager row thinning automatically under real memory pressure so deep-prefix restore does not trade unbounded DRAM growth for retention. Update retention tests to cover direct deep-endpoint restores and pressure behavior. Log host-store submission and completion errors instead of silently discarding their causes. Validation: all 393 Selene unit tests pass. Native TP16 store/drain and DRAM restore tests show zero I/O errors and a complete 153295-token prompt reused as 132593 GPU tokens plus 20702 host-restored tokens with zero new prefill. Those service tests used the db29 C1 diagnostic package. A separate C4 long-request CUDA_ERROR_ILLEGAL_ADDRESS remains under investigation and is not covered by this commit. Previous-core serving beyond signed package inspection was not separately revalidated. Design-Signed-Off-By: Kimi (Kimi); design-review=cfcf Co-Authored-By: Kimi <noreply@anthropic.com> * Preserve host record compatibility and reclaim rows under pressure Reserved transfer intake made the row-thinning threshold unreachable during ordinary retention pressure. Reclaim eligible redundant rows using free warm slots, and do so before incoming allocations evict reusable endpoints. Remove queued thinning keys with removed or compacted records so bookkeeping remains bounded by the resident index. Pinned rows remain protected. Record validation used the v3 size limit even for decoded legacy headers, and cross-rank agreement hashed wire versions despite semantic equality. Validate the stored encoding, retain lossless legacy writes for wide planes, select valid legacy replication versions, and hash a canonical encoding for rank agreement. Existing record bytes remain stable when rewritten without semantic changes. Update the cache format and retention documentation. These changes preserve usable endpoints under pressure and keep equivalent records from being rejected across format versions. Metadata-only tests cover 4 GiB planes, byte-exact rewrites, replication flags and mixed-format agreement. Pressure tests cover intake reserves, multi-slot rows, larger incoming records, pins and repeated chain removal. All 401 Selene tests and the CLI cargo check pass. No GPU or end-to-end serving test was run during this review. Design-Signed-Off-By: Codex GPT-6; design-review=cfcf Signed-off-by: Martin Vit <martin@voipmonitor.org> --------- Signed-off-by: Martin Vit <martin@voipmonitor.org> Co-authored-by: Kimi <noreply@anthropic.com>
1 parent 4c0e4bc commit f7c468c

6 files changed

Lines changed: 488 additions & 56 deletions

File tree

‎docs/prefix-cache-offload.md‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -247,11 +247,16 @@ policy allows it: the memory store standalone evicts any unpinned node, the
247247
disk store evicts any unpinned node, and the composite restricts the memory
248248
store to durable nodes.
249249

250-
Each tier drops a non-root record's row once its only child commits, keeping
251-
the pages; readers and spills delay removal until they release the row. DRAM
252-
releases row slots, disk truncates at the row segment, and later spills omit it.
253-
A rewind or later fork restores the nearest row-bearing ancestor and re-prefills
254-
the remaining tokens.
250+
Disk drops a non-root record's row once its only child commits, keeping the
251+
pages. DRAM retains these redundant rows while its free retention space exceeds
252+
two continuation rows. Reserved transfer intake does not count as retention
253+
headroom. When that threshold is reached, or an incoming record needs more
254+
space than is free, DRAM reclaims eligible redundant rows before evicting other
255+
records. Readers and spills delay row removal until their pins are released.
256+
DRAM releases row slots, disk truncates at the row segment, and later spills
257+
omit removed rows. A rewind or later fork restores its retained endpoint when
258+
available; otherwise it restores the nearest row-bearing ancestor and
259+
re-prefills the remaining tokens.
255260
With lazy write-behind, a crash or an incomplete shutdown drain can lose the
256261
DRAM-only descendants of a page-only disk prefix, leaving recovery to an older
257262
row-bearing ancestor. Startup prunes childless page-only records and any
@@ -393,11 +398,18 @@ the storage namespace identity, the layout fingerprint, the key (consumed count
393398
digest), the parent key, the bucket digest, the page token count, the tail
394399
page count, the rank count, the per-rank plane sizes and row bytes, the page
395400
segment offset, the row segment offset, the row checksum, and the creation
396-
time. Version 2 uses flag bit 31 for a replicated page segment and bit 30 for a
397-
replicated row. Page offsets use one fixed stride for the whole record. Version-1
398-
records still recover, and the layout fingerprint is unchanged. An older binary
399-
treats replicated version-2 files as invalid and **deletes them during recovery**;
400-
downgrading therefore starts those records cold. A checksum segment follows,
401+
time. Versions 1 and 2 encode each plane size as `u64`; version 3 uses `u32`
402+
plane sizes so wider rank/plane catalogs fit the same header block. Row sizes
403+
remain `u64`. New records use version 3 when every plane fits `u32`, otherwise
404+
the `u64` format is retained. All encodings still require the header to fit 4 KiB.
405+
Versions 2 and 3 use flag bit 31 for a replicated page segment and bit 30 for a
406+
replicated row; version 3 also permits non-replicated records. Page offsets use
407+
one fixed stride for the whole record. Version-1 and version-2 records still
408+
recover and preserve their encoding on rewrite; cross-rank agreement compares
409+
canonical header fields independently of the file's encoding version. The
410+
layout fingerprint is unchanged. A binary that does not support a record's
411+
version treats it as invalid and **deletes it during recovery**; downgrading
412+
therefore starts those records cold. A checksum segment follows,
401413
one 64-bit xxh3 over each stored page record, padded to
402414
4 KiB, so a restore that reads only a suffix of the pages verifies each page
403415
it read. The page segment follows, page-major: page `i`, then rank `r`, then

‎helios/src/server/runtime.rs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2224,6 +2224,12 @@ impl<'a> Worker<'a> {
22242224
StoreEvent::Stored {
22252225
checkpoint, result, ..
22262226
} => {
2227+
if let Err(error) = &result {
2228+
eprintln!(
2229+
"HOST_CACHE_STORE_ERROR checkpoint={} error={error}",
2230+
checkpoint.value()
2231+
);
2232+
}
22272233
self.metrics.host_retry_abandoned[5]
22282234
.fetch_add(u64::from(result.is_err()), Ordering::Relaxed);
22292235
self.scheduler.unpin_checkpoint(checkpoint)?;
@@ -2304,6 +2310,12 @@ impl<'a> Worker<'a> {
23042310
for publication in publications {
23052311
progress = true;
23062312
let outcome = store.store(&publication);
2313+
if let Err(error) = &outcome {
2314+
eprintln!(
2315+
"HOST_CACHE_STORE_SUBMIT_ERROR checkpoint={} error={error}",
2316+
publication.id.value()
2317+
);
2318+
}
23072319
if let Ok(outcome) = outcome {
23082320
self.metrics.host_store_outcome(outcome);
23092321
match outcome {

‎selene/src/memory.rs‎

Lines changed: 159 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -396,6 +396,14 @@ impl Core {
396396
if !self.fits(needed) {
397397
return Ok(None);
398398
}
399+
if self
400+
.pool
401+
.free_slots()
402+
.saturating_sub(self.pool.intake_slots())
403+
< needed
404+
{
405+
self.thin_rows_for(needed);
406+
}
399407
if self.spill && self.retire_for(self.shortfall(needed)) {
400408
return Ok(None);
401409
}
@@ -408,6 +416,7 @@ impl Core {
408416
Some(Eviction::Leaves(nodes)) => {
409417
self.stats.dropped[0] += if self.flat { 0 } else { nodes.len() as u64 };
410418
for node in &nodes {
419+
self.thinning.remove(&node.key);
411420
self.clean_slots -= self.clean_slots(node);
412421
self.release(&node.location);
413422
}
@@ -674,6 +683,7 @@ impl Core {
674683
if !node.ready || node.pinned() {
675684
return false;
676685
}
686+
self.thinning.remove(key);
677687
let clean = self.clean_slots(node);
678688
self.index.drop_row(key);
679689
let record = &mut self.index.node_mut(key).unwrap().location;
@@ -696,6 +706,20 @@ impl Core {
696706
}
697707

698708
fn thin_rows(&mut self) {
709+
self.thin_rows_for(self.layout.row_slots().saturating_mul(2));
710+
}
711+
712+
fn thin_rows_for(&mut self, headroom: usize) {
713+
// Intake slots belong to transfers, not checkpoint retention. Reclaim
714+
// redundant rows before an incoming record evicts reusable endpoints.
715+
if self
716+
.pool
717+
.free_slots()
718+
.saturating_sub(self.pool.intake_slots())
719+
> headroom
720+
{
721+
return;
722+
}
699723
for key in std::mem::take(&mut self.thinning) {
700724
if !self.index.row_redundant(&key, false) {
701725
continue;
@@ -728,6 +752,7 @@ impl Core {
728752

729753
fn remove_record(&mut self, key: &PrefixKey) -> Result<()> {
730754
for node in self.index.remove(key)? {
755+
self.thinning.remove(&node.key);
731756
self.clean_slots -= self.clean_slots(&node);
732757
self.release(&node.location);
733758
}
@@ -1816,7 +1841,7 @@ mod tests {
18161841
}
18171842

18181843
#[test]
1819-
fn conversation_rows_thin_to_root_and_tip_and_restore_through_page_carriers() {
1844+
fn conversation_rows_survive_with_headroom_and_restore_at_each_endpoint() {
18201845
let geometry = [geometry(PT, &[("kv", PAGE)], ROW)];
18211846
let (copier, gpu) = crate::testing::SimulatedGpu::shared_mapped(&geometry).unwrap();
18221847
let mut store = MemoryStore::new(IDENTITY, host_pool(32), copier).unwrap();
@@ -1842,20 +1867,17 @@ mod tests {
18421867
store.poll(&mut events).unwrap();
18431868
assert_eq!(
18441869
store.core.index.nodes().filter(|n| n.row).count(),
1845-
(turn as usize).min(2)
1870+
turn as usize
18461871
);
18471872
assert_clean_slots(&store.core);
18481873
}
1849-
assert_eq!(store.pool().slot_count() - store.pool().free_slots(), 10);
1850-
assert_eq!(store.stats().rows_thinned, [6, 0]);
1851-
for (tokens, expected) in [(8 * PT, 8 * PT), (7 * PT, PT)] {
1874+
assert_eq!(store.pool().slot_count() - store.pool().free_slots(), 16);
1875+
assert_eq!(store.stats().rows_thinned, [0, 0]);
1876+
for (tokens, expected) in [(8 * PT, 8 * PT), (7 * PT, 7 * PT)] {
18521877
let hit = store.lookup(&history(tokens, 1), 0, u64::MAX).unwrap();
18531878
assert_eq!(hit.consumed_tokens, expected);
1854-
// A rewind resumes at the root and re-prefills the remaining tokens.
1855-
assert_eq!(
1856-
tokens - hit.consumed_tokens,
1857-
if tokens == 7 * PT { 6 * PT } else { 0 }
1858-
);
1879+
// Every endpoint resumes directly at its retained row.
1880+
assert_eq!(tokens - hit.consumed_tokens, 0);
18591881
store
18601882
.fetch(
18611883
hit.clone(),
@@ -1887,7 +1909,7 @@ mod tests {
18871909
}
18881910

18891911
#[test]
1890-
fn thinning_waits_for_commit_and_pins_and_preserves_existing_branches() {
1912+
fn rows_survive_commit_and_pins_and_existing_branches() {
18911913
let layout = Layout::new(&[geometry(PT, &[("kv", PAGE)], 3 * PAGE)]).unwrap();
18921914
let mut core = Core::new(IDENTITY, host_pool(64), layout, false);
18931915
let full = history(320, 1);
@@ -1910,19 +1932,19 @@ mod tests {
19101932
let before = core.pool.free_slots();
19111933
core.unpin_record(&parent.key).unwrap();
19121934
core.settle(&mut Vec::new());
1913-
assert_eq!(core.pool.free_slots(), before + 3);
1914-
assert!(core.record(&parent.key).unwrap().row.is_none());
1935+
assert_eq!(core.pool.free_slots(), before);
1936+
assert!(core.record(&parent.key).unwrap().row.is_some());
19151937
assert_eq!(core.record(&parent.key).unwrap().pages.len(), 1);
19161938
assert_clean_slots(&core);
19171939

1918-
// A late fork preserves the page carrier without reconstructing its lost row.
1940+
// A late fork retains both the page carrier and its parent's row.
19191941
let mut branch = history(128, 1);
19201942
branch.append(&[99; 128], &[]).unwrap();
19211943
let sibling = published(&branch, 192, Some(128), 10, 4);
19221944
core.store(&sibling).unwrap();
19231945
finish(&mut core);
19241946
assert_eq!(core.index[&parent.key].children(), 2);
1925-
assert!(!core.index[&parent.key].row);
1947+
assert!(core.index[&parent.key].row);
19261948
assert!(core.index[&child.key].row && core.index[&sibling.key].row);
19271949

19281950
// A branch formed while its row is pinned retains that row and both tips.
@@ -1944,11 +1966,130 @@ mod tests {
19441966
finish(&mut core);
19451967
assert_eq!(
19461968
core.lookup(&history(256, 1), 0, u64::MAX).unwrap().key,
1947-
child.key
1969+
leaf.key
19481970
);
19491971
assert_clean_slots(&core);
19501972
}
19511973

1974+
#[test]
1975+
fn row_thinning_uses_warm_headroom_for_each_row_geometry() {
1976+
for row_slots in [1, 3] {
1977+
for headroom in [2 * row_slots, 2 * row_slots + 1] {
1978+
let layout =
1979+
Layout::new(&[geometry(PT, &[("kv", PAGE)], row_slots as u64 * PAGE)]).unwrap();
1980+
assert_eq!(layout.row_slots(), row_slots);
1981+
let intake = crate::offload::intake_slots(&layout, 1);
1982+
let warm = 3 * (1 + row_slots) + headroom;
1983+
let pool = SlabPool::new(PAGE, warm + intake, intake).unwrap();
1984+
let mut core = Core::new(IDENTITY, pool, layout, false);
1985+
let full = history(192, 1);
1986+
let checkpoints = [
1987+
published(&full, 64, None, 0, 1),
1988+
published(&full, 128, Some(64), 0, 2),
1989+
published(&full, 192, Some(128), 0, 3),
1990+
];
1991+
for checkpoint in &checkpoints {
1992+
assert!(core.store(checkpoint).unwrap());
1993+
finish(&mut core);
1994+
}
1995+
let thinned = headroom == 2 * row_slots;
1996+
assert_eq!(
1997+
core.index[&checkpoints[1].key].row, !thinned,
1998+
"row_slots={row_slots}, warm headroom={headroom}, intake={intake}"
1999+
);
2000+
assert_eq!(
2001+
core.pool.free_slots(),
2002+
intake + headroom + usize::from(thinned) * row_slots
2003+
);
2004+
assert!(core.index[&checkpoints[0].key].row);
2005+
assert!(core.index[&checkpoints[2].key].row);
2006+
assert_clean_slots(&core);
2007+
}
2008+
}
2009+
}
2010+
2011+
#[test]
2012+
fn pressure_thins_redundant_rows_before_allocating_another_endpoint() {
2013+
for (warm_slots, next_endpoint) in [(7, 256), (9, 384)] {
2014+
let mut core = store_core(warm_slots - 1, 1, false);
2015+
let full = history(next_endpoint, 1);
2016+
let root = published(&full, 64, None, 0, 1);
2017+
let middle = published(&full, 128, Some(64), 0, 2);
2018+
let child = published(&full, 192, Some(128), 0, 3);
2019+
let tip = published(&full, next_endpoint, Some(192), 0, 4);
2020+
for checkpoint in [&root, &middle, &child, &tip] {
2021+
assert!(core.store(checkpoint).unwrap());
2022+
finish(&mut core);
2023+
}
2024+
assert!(core.index[&root.key].row, "warm slots={warm_slots}");
2025+
assert!(!core.index[&middle.key].row);
2026+
assert_eq!(
2027+
core.lookup(&history(64, 1), 0, u64::MAX).map(|hit| hit.key),
2028+
Some(root.key)
2029+
);
2030+
assert!(core.pool.free_slots() >= core.pool.intake_slots());
2031+
assert_clean_slots(&core);
2032+
}
2033+
}
2034+
2035+
#[test]
2036+
fn pressure_thinning_waits_for_the_last_row_pin() {
2037+
let mut core = store_core(6, 1, false);
2038+
let full = history(192, 1);
2039+
let root = published(&full, 64, None, 0, 1);
2040+
let parent = published(&full, 128, Some(64), 0, 2);
2041+
let child = published(&full, 192, Some(128), 0, 3);
2042+
for checkpoint in [&root, &parent] {
2043+
assert!(core.store(checkpoint).unwrap());
2044+
finish(&mut core);
2045+
}
2046+
core.pin_record(&parent.key).unwrap();
2047+
assert!(core.store(&child).unwrap());
2048+
finish(&mut core);
2049+
assert!(core.index[&parent.key].row);
2050+
assert!(core.thinning.contains(&parent.key));
2051+
let before = core.pool.free_slots();
2052+
core.unpin_record(&parent.key).unwrap();
2053+
core.settle(&mut Vec::new());
2054+
assert!(!core.index[&parent.key].row);
2055+
assert_eq!(core.pool.free_slots(), before + core.layout.row_slots());
2056+
assert!(!core.thinning.contains(&parent.key));
2057+
assert_clean_slots(&core);
2058+
}
2059+
2060+
#[test]
2061+
fn thinning_queue_keeps_only_resident_records_after_removal_and_eviction() {
2062+
for remove_explicitly in [false, true] {
2063+
let mut core = store_core(7, 1, false);
2064+
for salt in 1..=64 {
2065+
let full = history(192, salt);
2066+
let checkpoints = [
2067+
published(&full, 64, None, 0, 1),
2068+
published(&full, 128, Some(64), 0, 2),
2069+
published(&full, 192, Some(128), 0, 3),
2070+
];
2071+
for checkpoint in &checkpoints {
2072+
assert!(core.store(checkpoint).unwrap());
2073+
finish(&mut core);
2074+
}
2075+
if remove_explicitly {
2076+
// Expiration and explicit removal share this record-removal path.
2077+
for checkpoint in checkpoints.iter().rev() {
2078+
if core.index.contains(&checkpoint.key) {
2079+
core.remove_record(&checkpoint.key).unwrap();
2080+
}
2081+
}
2082+
}
2083+
assert!(
2084+
core.thinning.iter().all(|key| core.index.contains(key)),
2085+
"stale thinning keys after chain {salt}, explicit removal={remove_explicitly}"
2086+
);
2087+
assert!(core.thinning.len() <= core.index.len());
2088+
assert_clean_slots(&core);
2089+
}
2090+
}
2091+
}
2092+
19522093
#[test]
19532094
fn a_record_one_slot_larger_than_warm_is_oversize() {
19542095
let geometry = [geometry(PT, &[("kv", PAGE)], ROW)];
@@ -2773,8 +2914,8 @@ mod tests {
27732914
stored(&finish(&mut core)),
27742915
[(grandchild.key, Some(3 * PAGE + ROW))]
27752916
);
2776-
// Commit thins the child: nine page slots and the tip's one row slot
2777-
// occupy ten of twelve warm slots. The root's row was already freed.
2917+
// With only one warm slot free, commit thins the redundant child row.
2918+
// Its pages remain available and the grandchild retains its endpoint.
27782919
assert!(core.record(&child.key).unwrap().row.is_none());
27792920
assert!(core.record(&grandchild.key).unwrap().row.is_some());
27802921
assert_eq!(core.pool.free_slots(), core.pool.intake_slots() + 2);

‎selene/src/shard.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1201,7 +1201,7 @@ fn shape(header: &Header) -> Result<[u8; 16]> {
12011201
header.fingerprint = [0; 16];
12021202
header.row_checksum = 0;
12031203
let mut hash = Blake2b::<U16>::new();
1204-
hash.update(header.encode()?);
1204+
hash.update(header.canonical_encoding()?);
12051205
Ok(hash.finalize().into())
12061206
}
12071207

‎selene/src/shard/tests.rs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,42 @@ use std::net::TcpListener;
512512
assert!(agree(&zero, &peers).unwrap().is_empty());
513513
}
514514

515+
#[test]
516+
fn agreement_compares_legacy_and_v3_headers_by_geometry() {
517+
for (version, replication) in [
518+
(1u32, record::Replication::default()),
519+
(2u32, record::Replication { pages: true, row: true }),
520+
] {
521+
let mut compact = scanned(key(128, 1), None, true, 0);
522+
compact.header.set_replication(replication).unwrap();
523+
let mut bytes = compact.header.encode().unwrap();
524+
assert_eq!(u32::from_le_bytes(bytes[8..12].try_into().unwrap()), 3);
525+
bytes[8..12].copy_from_slice(&version.to_le_bytes());
526+
// This fixture has one rank and one plane. Legacy entries retain
527+
// an eight-byte plane size followed by the eight-byte row size.
528+
bytes[181..189].copy_from_slice(&compact.header.ranks[0].planes[0].to_le_bytes());
529+
bytes[189..197].copy_from_slice(&compact.header.ranks[0].row_bytes.to_le_bytes());
530+
bytes[197..].fill(0);
531+
let mut legacy = compact.clone();
532+
legacy.header = Header::decode(&bytes).unwrap();
533+
assert_eq!(legacy.header, compact.header);
534+
assert_eq!(legacy.header.encode().unwrap(), bytes);
535+
536+
for (local, peer) in [(&compact, &legacy), (&legacy, &compact)] {
537+
let peers = [decode_summary(&summary(std::slice::from_ref(peer)).unwrap()[5..]).unwrap()];
538+
assert_eq!(
539+
agree(std::slice::from_ref(local), &peers).unwrap(),
540+
HashMap::from([(local.header.key, true)]),
541+
);
542+
}
543+
544+
let mut different = legacy;
545+
different.header.ranks[0].row_bytes += 4096;
546+
let peers = [decode_summary(&summary(&[different]).unwrap()[5..]).unwrap()];
547+
assert!(agree(&[compact], &peers).unwrap().is_empty());
548+
}
549+
}
550+
515551
#[test]
516552
fn events_and_peer_file_names_round_trip_and_reject_paths() {
517553
let event = TransportEvent::StoreInvalidated {

0 commit comments

Comments
 (0)