Repository navigation
Fix native peer-access OOM without reducing cache capacity - #286
Conversation
Cache planning can consume nearly all free GPU memory before post-allocation cuMemSetAccess calls create driver access state. The loader also repeats static grants and uses every signed reader declaration, including ranks that never import the allocation. This can turn an otherwise valid automatic KV capacity into an allocation OOM and force an unnecessary manual reserve. Use validated imports as the access authority and make single-host cache allocation transactional so a failed attempt can be reduced safely. Replace the two all-descriptor grant passes with one owner-plus-actual-importer grant per physical mapping. Apply the same path to static and runtime-sized storage, preserve owner-only continuation aliases, and log per-reader free-memory deltas around cache grants. Keep core sizing declarations until every rank maps and grants its cache; only then publish execution state and capacity bindings. For cache cuMemCreate, cuMemMap, or detailed cuMemSetAccess OOM on one host, synchronize all contexts, unmap every cache mapping and alias, account for retained free-memory loss, and retry with doubling KV-budget reductions. Cross-host allocation remains one-shot; other errors and cleanup failures abort startup. Existing signed cores remain loadable without re-export. Successful first attempts keep the capacity selected by the original memory plan and pay no retry allowance. A single-host cache OOM can now release the attempt and recover at a smaller common KV capacity instead of requiring a guessed manual reserve. Steady-state allocation, graph contracts, and model computation are unchanged. Focused runtime-core tests on current main pass: 133 passed, 7 ignored. Existing exported TP16 cores with 8-rank and 16-rank import groups completed native serving with no manual memory reserve, including C1/C4 generation, 32K prefill, and non-empty response checks. The 8-rank case succeeded on its first attempt with 3,145,728 KV tokens and 153 snapshot slots; all 384 grants had exactly eight readers. The 16-rank case hit a final-grant OOM, unmapped the attempt, and succeeded after a 96 MiB/GPU budget reduction with 4,485,120 KV tokens and 109 slots; successful grants had 16 readers. Separate 8-rank runs also passed with -1G and -0G reserves. The broader Helios suite had 1,366 passes; four missing-template failures and one cache-selection failure reproduced on the parent source. That full run predates the rebase onto current main. Native retry exhaustion at minimum KV capacity and cross-host retry were not tested; cross-host retry is intentionally not enabled. Design-Signed-Off-By: Claude Code (Claude Fable 5); design-review=ed22 Co-Authored-By: Claude Code <noreply@anthropic.com>
original-el8
added
area:caching
|
Categorized as P2 cache-allocation correctness work requiring review. The proposed transaction delays publishing capacity bindings until every local rank succeeds, then cleans up a failed cache attempt before retrying at a smaller common budget. The reported TP16 run exercised a real final-grant OOM and recovered after a 96 MiB/GPU reduction. Review cleanup and owner/importer grants, especially continuation aliases, error paths and retry exhaustion at minimum capacity. The PR explicitly leaves cross-host allocation one-shot and has not exercised native retry exhaustion. No model label is inferred from the TP16 rank count. |
Native startup with sixteen-rank cache import groups (DCP16) can exhaust UVM/RM mapping handles while framebuffer memory remains free. Small KV backing allocations used by snapshot aliases create separate allocation and per-device mapping objects. CUDA exposes the resulting NV_ERR_INSUFFICIENT_RESOURCES as an out-of-memory error. The backing can be merged without changing virtual cache layout, so handle pressure need not be relieved by shrinking KV. Replace cache allocation retry and budget backoff with one allocation attempt. Permute backing-plane assignments only between chunks with identical bytes per page, byte offset, and mapped length, then reorder whole chunks within one snapshot row and merge adjacent spans of the same KV plane. Keep spans crossing row boundaries and partial row spans in place. Preserve physical byte coverage, recorded addresses, capacities, and each snapshot's exact set of conflicting KV pages. Signed core metadata and owner/importer permission sets remain compatible; no re-export is needed. For a Kimi K3.1 Pro TP16 layout with 2,234 local KV pages and 112 snapshots, KV backing chunks per rank fall from 1,920 to 1,625. With default memory sizing or --memory-limit=-0G, DCP16 retains 4,575,232 KV tokens and 112 snapshot slots. Eight-rank import groups (DCP8) retain 3,145,728 tokens and 153 snapshot slots. Capacity selection is unchanged; no additional reserve or allocation retry remains. Native startup and replay are qualified on sixteen RTX PRO 6000 Blackwell GPUs with driver 615.71.10: both import group sizes passed default, -0G, and -1G, including generation at concurrency one and four, three cold 32,768-token prefills, reader checks, and clean shutdown. The runtime-core CPU suite passed 133 tests; the GPU alias test passed with distinct byte patterns, including merged backing. Bounded TLA+ models and Rust conformance checks passed without mutation testing, and the release CLI built successfully. Six other GPU fixture tests were not run. Qualification does not establish model quality or support on other hardware/driver versions. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
voipmonitor
changed the title
|
Regarding the retry-related review items in this comment: the DCP16 failure was UVM/RM mapping-resource exhaustion ( Commit d63cd1fc removes allocation retry, cleanup-for-retry, and budget backoff. It merges actual backing allocations and alias mappings within individual snapshot rows, preserving recorded addresses, physical bytes, KV capacity, snapshot count, and each snapshot's exact set of conflicting KV pages. Access remains scoped to the owner and actual importers, including for cores with overdeclared reader lists. Capacity bindings still wait until every local rank has mapped and granted its cache. Allocation is attempted once on both single-host and cross-host paths, so retry exhaustion and the 96 MiB/GPU reduction are no longer part of the implementation. Review of alias lifetime, owner/importer permissions, and error paths remains relevant. Default and explicit zero-reserve startup retain 4,575,232 KV tokens and 112 snapshots for DCP16, and 3,145,728 tokens and 153 snapshots for DCP8, without an added reserve. Native startup and replay are qualified on Frank1's sixteen RTX PRO 6000 Blackwell GPUs with driver 615.71.10: all six DCP8/DCP16 × default/ |
Why
Native startup with sixteen tensor-parallel ranks can fail in
cuMemSetAccessafter KV backing is mapped even though framebuffer memory remains available. Some cores declare all ranks as readers where only eight-rank context-parallel groups (DCP8) import the cache. When all sixteen ranks import it (DCP16), fine-grained backing for snapshot aliases can exhaust UVM's shared RM handle namespace. CUDA reports the underlyingNV_ERR_INSUFFICIENT_RESOURCES(0x1a) as OOM.Static model and peer-exchange storage is allocated before KV sizing. The failing operation grants access to the KV backing itself. The correction therefore scopes access to real importers and reduces actual backing objects while preserving the full cache layout contract.
What changed
Memory::Peerimports, including the owner. Replace the two all-descriptor grant passes with one grant per physical mapping. Existing cores whosePeerMemory.readersincludes every TP rank remain compatible, and TP16 static imports retain their required access.The change is implemented in shared native storage code. Existing signed cores load without re-export or graph changes.
Result
For the Kimi K3.1 Pro DCP16 layout with 2,234 local KV pages, 112 snapshot rows, and five execution rows, KV backing chunks per rank fall from 1,920 to 1,625, with identical physical bytes and snapshot conflicts. Native serving succeeds on the first allocation attempt with the complete physical-memory plan. KV capacity and snapshot capacity are both retained:
--memory-limit=-0G--memory-limit=-1GThe lower capacities in the
-1Gcolumn come solely from the explicitly requested reserve. There is no automatic reserve or reduction after an allocation failure.Evidence and qualification
Native qualification used sixteen RTX PRO 6000 Blackwell Workstation Edition GPUs (SM120), NVIDIA driver 615.71.10, on Frank1. The unchanged Kimi K3.1 Pro TP16 payload cores were
gb202-tp16-202611121210for DCP16 andgb202-tp16-202611131301for DCP8, under/data/kimi31-support/k31-synthetic-comm-lab/core-cache/models/kimi-k31-mxfp4/. Serving selected concurrency 4, seven DFlash2 draft tokens, adaptive verification, no history drafts, and--kv-cache-limit memory; the configuration reserve was zero.-0G/-1Gpassed generation at concurrency one and four (C1/C4, with a 1,024-token completion limit per measured request), three cold 32,768-token prefills, nonempty response and usage checks, and clean shutdown. Every case had one cache allocation attempt and 384 successful shared-allocation access records with exactly eight or sixteen readers, respectively. Capacities were asserted against the values in the table.0x1aat mapping ioctl 521,881. A separate reproducer using only 2 MiB of physical backing reached the same status with over 101 GB free per GPU. The packed DCP16 load and serving checks completed with 444,837 cumulative mapping ioctls and no failed mapping calls. These are resource counts, not measurements of a driver memory reserve.The ordinary CPU run ignored seven GPU cases; the targeted alias case was then run explicitly, leaving six other GPU fixture tests unrun. Native qualification is limited to the hardware, driver, cores, and workloads above. It is serving/alias correctness qualification, not a full model-quality evaluation. The broader Helios suite was not rerun for this revision.
Focused reproduction commands, from the repository root:
The GPU command requires an assigned idle CUDA device. The lifecycle command and pinned TLA+ tools setup are documented in verification guidance. Exact native launch arguments, response results, and raw driver/startup logs are retained on Frank1 under
/data/kimi31-dcp8-20261112/flare-dcp8-peer-access-fix/.local-dev/peer-access/; the final serving cases have thedcp16-packed-*anddcp8-packed-*prefixes and are summarized bypacked-matrix.log.