Fix native peer-access OOM without reducing cache capacity (#286)

* Fix native cache peer access and retry allocation OOM

Cache planning can consume nearly all free GPU memory before post-allocation
cuMemSetAccess calls create driver access state. The loader also repeats static
grants and uses every signed reader declaration, including ranks that never
import the allocation. This can turn an otherwise valid automatic KV capacity
into an allocation OOM and force an unnecessary manual reserve. Use validated
imports as the access authority and make single-host cache allocation
transactional so a failed attempt can be reduced safely.

Replace the two all-descriptor grant passes with one owner-plus-actual-importer
grant per physical mapping. Apply the same path to static and runtime-sized
storage, preserve owner-only continuation aliases, and log per-reader free-memory
deltas around cache grants. Keep core sizing declarations until every rank maps
and grants its cache; only then publish execution state and capacity bindings.
For cache cuMemCreate, cuMemMap, or detailed cuMemSetAccess OOM on one host,
synchronize all contexts, unmap every cache mapping and alias, account for
retained free-memory loss, and retry with doubling KV-budget reductions.
Cross-host allocation remains one-shot; other errors and cleanup failures abort
startup. Existing signed cores remain loadable without re-export.

Successful first attempts keep the capacity selected by the original memory plan
and pay no retry allowance. A single-host cache OOM can now release the attempt
and recover at a smaller common KV capacity instead of requiring a guessed
manual reserve. Steady-state allocation, graph contracts, and model computation
are unchanged.

Focused runtime-core tests on current main pass: 133 passed, 7 ignored. Existing
exported TP16 cores with 8-rank and 16-rank import groups completed native
serving with no manual memory reserve, including C1/C4 generation, 32K prefill,
and non-empty response checks. The 8-rank case succeeded on its first attempt
with 3,145,728 KV tokens and 153 snapshot slots; all 384 grants had exactly eight
readers. The 16-rank case hit a final-grant OOM, unmapped the attempt, and
succeeded after a 96 MiB/GPU budget reduction with 4,485,120 KV tokens and 109
slots; successful grants had 16 readers. Separate 8-rank runs also passed with
-1G and -0G reserves. The broader Helios suite had 1,366 passes; four
missing-template failures and one cache-selection failure reproduced on the
parent source. That full run predates the rebase onto current main. Native retry
exhaustion at minimum KV capacity and cross-host retry were not tested;
cross-host retry is intentionally not enabled.

Design-Signed-Off-By: Claude Code (Claude Fable 5); design-review=ed22
Co-Authored-By: Claude Code <noreply@anthropic.com>

* Compact cache backing without reducing KV or snapshot capacity

Native startup with sixteen-rank cache import groups (DCP16) can exhaust
UVM/RM mapping handles while framebuffer memory remains free. Small KV
backing allocations used by snapshot aliases create separate allocation
and per-device mapping objects. CUDA exposes the resulting
NV_ERR_INSUFFICIENT_RESOURCES as an out-of-memory error. The backing can
be merged without changing virtual cache layout, so handle pressure need
not be relieved by shrinking KV.

Replace cache allocation retry and budget backoff with one allocation
attempt. Permute backing-plane assignments only between chunks with
identical bytes per page, byte offset, and mapped length, then reorder
whole chunks within one snapshot row and merge adjacent spans of the
same KV plane. Keep spans crossing row boundaries and partial row spans
in place. Preserve physical byte coverage, recorded addresses,
capacities, and each snapshot's exact set of conflicting KV pages.
Signed core metadata and owner/importer permission sets remain
compatible; no re-export is needed.

For a Kimi K3.1 Pro TP16 layout with 2,234 local KV pages and 112
snapshots, KV backing chunks per rank fall from 1,920 to 1,625. With
default memory sizing or --memory-limit=-0G, DCP16 retains 4,575,232 KV
tokens and 112 snapshot slots. Eight-rank import groups (DCP8) retain
3,145,728 tokens and 153 snapshot slots. Capacity selection is
unchanged; no additional reserve or allocation retry remains.

Native startup and replay are qualified on sixteen RTX PRO 6000
Blackwell GPUs with driver 615.71.10: both import group sizes passed
default, -0G, and -1G, including generation at concurrency one and four,
three cold 32,768-token prefills, reader checks, and clean shutdown. The
runtime-core CPU suite passed 133 tests; the GPU alias test passed with
distinct byte patterns, including merged backing. Bounded TLA+ models
and Rust conformance checks passed without mutation testing, and the
release CLI built successfully. Six other GPU fixture tests were not
run. Qualification does not establish model quality or support on other
hardware/driver versions.

Design-Signed-Off-By: Codex (GPT-6); design-review=ed22

---------

Co-authored-by: Claude Code <noreply@anthropic.com>

Author: voipmonitor

docs/runtime-core-design.md

@@ -662,6 +662,40 @@ the recipe's execution rows per request. KV pages receive the remaining budget,
 within their virtual maximum. The planner accounts for mapping alignment and
 requires at least 8,192 cached tokens per active request.
 
+The number of physical allocation handles and device mappings is also a driver
+resource constraint. Linux UVM duplicates external mappings into a shared RM
+client whose automatic handle range is 524,288 entries (`RS_UNIQUE_HANDLE_RANGE`
+in NVIDIA drivers
+[580.105.08](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/580.105.08/src/nvidia/inc/libraries/resserv/rs_client.h)
+and [615.71.10](https://github.com/NVIDIA/open-gpu-kernel-modules/blob/615.71.10/src/nvidia/generated/g_rs_client_nvoc.h)).
+Exhaustion returns `NV_ERR_INSUFFICIENT_RESOURCES`, which CUDA can expose as
+`CUDA_ERROR_OUT_OF_MEMORY` despite free framebuffer memory. Combining access
+calls alone does not remove the underlying allocation and mapping objects.
+
+The native cache layout reduces those objects without reducing KV pages or
+snapshot rows. It first permutes the backing-plane assignment of chunks with
+identical bytes per page, byte offset within the plane, and mapped length. Their
+row positions and logical page coverage stay the same. It then reorders whole chunks inside an
+uninterrupted interval of one snapshot row, merging adjacent spans of the same
+KV plane into one backing allocation and one alias mapping. Only complete chunks
+wholly contained in one row are reordered or merged; spans crossing row boundaries
+and partial mappings keep their row positions and lengths.
+
+These transformations preserve every plane's byte coverage, physical byte
+count, snapshot address and stride, and the exact set of KV pages conflicting
+with each snapshot. The scheduler consumes the same exclusion ranges. Reader
+permissions remain allocation-specific. The layout uses ordinary CUDA VMM
+allocations at the recorded core addresses and requires no core re-export,
+additional framebuffer reserve, allocation retry, or reduced cache capacity.
+
+Native startup and C1/C4 generation with 32K prefill are qualified for Kimi K3.1
+Pro TP16 cores using DCP8 and DCP16 on sixteen RTX PRO 6000 Blackwell GPUs with
+NVIDIA driver 615.71.10, with default memory sizing and explicit reserves of
+zero and 1 GiB. KV and snapshot capacities match the physical-memory plan in
+all six configurations. CPU layout tests compare every row's exact conflict
+set and complete byte coverage; GPU tests preserve snapshot bytes while released
+KV pages are overwritten. These checks do not constitute model-quality evaluation.
+
 Immutable snapshot capacity is bounded by borrowable KV backing and the prepared
 virtual extent. At load, CUDA virtual mappings expose the same allocation chunks
 through the page planes and the snapshot part of the continuation range. Small
@@ -787,9 +821,34 @@ Collective signals and generation-tagged payloads are reset after all ranks
 drain. Imported mappings are recorded as peer dependencies and stored only in
 the owning rank's bundle.
 
-Helios checks that rank allocations do not overlap, creates every device
-context, loads rank storage, and grants the declared peer access through
-`cuMemSetAccess` before loading modules and instantiating execution state.
+Helios checks that rank allocations do not overlap and creates every device
+context before mapping storage. Each mapping receives one `cuMemSetAccess`
+grant containing its owner and the ranks whose bundles import that allocation
+as peer memory. The signed reader declaration must include these ranks but may
+contain unused ranks; cores with all-rank reader declarations therefore retain
+compatibility without granting access outside the actual import group. Static
+mappings receive access during allocation. Runtime-sized cache mappings receive
+access after backing is mapped, before zeroing or graph use. Continuation aliases
+have their own access permissions, independent of the KV plane they borrow.
+
+For shared cache allocations, startup logs each reader GPU's `cuMemGetInfo_v2`
+free bytes before and after its mapping grants, the signed used-byte delta,
+owner, address, backed size, backing mapping count, and reader rank/device pairs.
+Physical backing is already allocated at the first sample; no cache backing is
+allocated between these samples. These observations include driver access state
+and may include other processes' memory activity. They do not measure transient
+peaks inside a driver call or establish a safe planning reserve. Remote KV bytes are
+not charged as duplicated physical storage. A failed grant reports the exact
+mapping range and readers while retaining the CUDA status.
+
+Cache backing and access grants are attempted once. Allocation and access errors
+abort startup with their original CUDA status; the loader does not reduce cache
+capacity and retry. Capacities are published only after all ranks have mapped
+and granted their cache. The fixed model and communication buffers are allocated
+before cache sizing. Access to the runtime-sized cache itself follows its
+physical mapping, so its mapping resources must fit alongside the chosen backing.
+CUDA free bytes do not describe every driver resource limit.
+
 Address reservations are established before allocation. Allocation, payload
 loading, kernel loading, and execution preparation each run concurrently across
 ranks, joining all rank threads between stages. Loading retains a process-wide