Repository navigation
Conversation
Snapshot rows virtually alias KV pages. When the last snapshot owning a page is released, the page returns to the free list still holding row bytes. A request that later receives it as a partially filled tail page reads those bytes past its last token; in FP8 KV some decode as NaN, masked attention turns 0 x NaN into non-finite logits, and the commit reports INVALID_SPECULATION (device error 6) under KV pressure. The scheduler now marks a page when its last alias is released. When commit first moves a marked page from a request's reservation into its KV pages, the submission carries that page id and the mark is cleared. Every executor zeroes the listed pages in each local cache plane with cuMemsetD8Async on the model stream, after the input copies and before the step's graph. Tensor-parallel PREPARE frames carry the list to every rank (handshake layout 7), and peers validate it before READY. Pages are not cleared at release, where inbound host restores could still be writing, or at reservation, which may be returned unused. Inbound restore pages are overwritten whole and never enter a clear list; reservations, cancellations, and snapshot reuse keep the mark. The change is independent of model and cache scheme. Validation: - Scheduler unit test: released alias pages are listed exactly once, when a request first commits them, and survive cancellation and snapshot reuse. - TP wire tests: cleared pages round-trip through PREPARE, and ids outside the cache are rejected. - CUDA test on an RTX PRO 6000 (SM120) using real VMM aliases across three planes: FP8 NaN bytes written through released rows land in freed KV pages and stay inside their alias ranges. clear_pages zeroes exactly the freed pages and leaves other pages and the live row byte-identical. - Helios lib tests after rebasing onto 585159c: 1259 passed. The 9 failures are the frontend, engram, and package-cache tests that fail on unmodified cc696a6 in the test environment; 585159c itself was not rerun. - helios check with all targets and flare release build pass on 585159c. - Stock Qwen3.8 Flash-Next TP2 under KV pressure (--memory-limit 49.5G, 143 alternating 16k/32k prompts): stock passes 143/143. This model's QSA attention does not read partially filled FP8 tail pages, so the run does not reproduce NVIDIA#116. Limitations: the NVIDIA#116 end-to-end reproduction on dense Qwen3.8-27B has not been run. Exporting qwen3.8-27b at 6fa01a4 failed in "Warm compiled forwards" (compiled forward has unbound static Inductor kernels). No throughput comparison has been measured. Gemma 4 (NVIDIA#61) is untested. The CUDA alias test and the TP2 run predate the rebase onto 585159c and were not repeated. Addresses NVIDIA#116. Design-Signed-Off-By: Cursor Agent (Auto); design-review=a38f
16da4e2 to
a8d434c
Compare
|
The need for scheduler-level page clearing should be re-established after the attention masking fixes. Flare's pin 1f8ed8a3 includes the tail-NaN fixes that resolved the dense-Qwen reproduction in #116. FlashInfer PRs #73 and #74 additionally fix leading-window and MSA tail masking, but are newer than that Flare pin. Marked |
|
Dependency status refresh: main This PR remains |
|
@mark-yong GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
Snapshot rows virtually alias KV pages. When the last snapshot owning a page is released, the page returns to the free list still holding row bytes. A request that later receives it as a partially filled tail page reads those bytes past its last token; in FP8 KV some decode as NaN, masked attention turns 0 x NaN into non-finite logits, and the commit reports INVALID_SPECULATION (device error 6) under KV pressure.
The scheduler now marks a page when its last alias is released. When commit first moves a marked page from a request's reservation into its KV pages, the submission carries that page id and the mark is cleared. Every executor zeroes the listed pages in each local cache plane with cuMemsetD8Async on the model stream, after the input copies and before the step's graph. Tensor-parallel PREPARE frames carry the list to every rank (handshake layout 7), and peers validate it before READY. Pages are not cleared at release, where inbound host restores could still be writing, or at reservation, which may be returned unused. Inbound restore pages are overwritten whole and never enter a clear list; reservations, cancellations, and snapshot reuse keep the mark. The change is independent of model and cache scheme.
Validation:
Limitations: the #116 end-to-end reproduction on dense Qwen3.8-27B has not been run. Exporting qwen3.8-27b at 6fa01a4 failed in "Warm compiled forwards" (compiled forward has unbound static Inductor kernels). No throughput comparison has been measured.
Fixes #116.
Design-Signed-Off-By: Cursor Agent (Auto); design-review=1c01