Skip to content

Clear released snapshot-alias pages before their first KV use - #181

Draft
mark-yong wants to merge 1 commit into
NVIDIA:mainfrom
mark-yong:fix/released-alias-pages-stale-row-bytes
Draft

mark-yong wants to merge 1 commit into
NVIDIA:mainfrom
mark-yong:fix/released-alias-pages-stale-row-bytes

Conversation

@mark-yong

@mark-yong mark-yong commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Snapshot rows virtually alias KV pages. When the last snapshot owning a page is released, the page returns to the free list still holding row bytes. A request that later receives it as a partially filled tail page reads those bytes past its last token; in FP8 KV some decode as NaN, masked attention turns 0 x NaN into non-finite logits, and the commit reports INVALID_SPECULATION (device error 6) under KV pressure.

The scheduler now marks a page when its last alias is released. When commit first moves a marked page from a request's reservation into its KV pages, the submission carries that page id and the mark is cleared. Every executor zeroes the listed pages in each local cache plane with cuMemsetD8Async on the model stream, after the input copies and before the step's graph. Tensor-parallel PREPARE frames carry the list to every rank (handshake layout 7), and peers validate it before READY. Pages are not cleared at release, where inbound host restores could still be writing, or at reservation, which may be returned unused. Inbound restore pages are overwritten whole and never enter a clear list; reservations, cancellations, and snapshot reuse keep the mark. The change is independent of model and cache scheme.

Validation:

  • Scheduler unit test: released alias pages are listed exactly once, when a request first commits them, and survive cancellation and snapshot reuse.
  • TP wire tests: cleared pages round-trip through PREPARE, and ids outside the cache are rejected.
  • CUDA test on an RTX PRO 6000 (SM120) using real VMM aliases across three planes: FP8 NaN bytes written through released rows land in freed KV pages and stay inside their alias ranges. clear_pages zeroes exactly the freed pages and leaves other pages and the live row byte-identical.
  • Helios lib tests: the same 9 frontend, engram, and package-cache failures occur on unmodified cc696a6 in the test environment.
  • flare release build and heliopause check pass.
  • Stock Qwen3.8 Flash-Next TP2 under KV pressure (--memory-limit 49.5G, 143 alternating 16k/32k prompts): stock passes 143/143. This model's QSA attention does not read partially filled FP8 tail pages, so the run does not reproduce Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure) #116.

Limitations: the #116 end-to-end reproduction on dense Qwen3.8-27B has not been run. Exporting qwen3.8-27b at 6fa01a4 failed in "Warm compiled forwards" (compiled forward has unbound static Inductor kernels). No throughput comparison has been measured.

Fixes #116.

Design-Signed-Off-By: Cursor Agent (Auto); design-review=1c01

Snapshot rows virtually alias KV pages. When the last snapshot owning a
page is released, the page returns to the free list still holding row
bytes. A request that later receives it as a partially filled tail page
reads those bytes past its last token; in FP8 KV some decode as NaN,
masked attention turns 0 x NaN into non-finite logits, and the commit
reports INVALID_SPECULATION (device error 6) under KV pressure.

The scheduler now marks a page when its last alias is released. When
commit first moves a marked page from a request's reservation into its
KV pages, the submission carries that page id and the mark is cleared.
Every executor zeroes the listed pages in each local cache plane with
cuMemsetD8Async on the model stream, after the input copies and before
the step's graph. Tensor-parallel PREPARE frames carry the list to every
rank (handshake layout 7), and peers validate it before READY. Pages are
not cleared at release, where inbound host restores could still be
writing, or at reservation, which may be returned unused. Inbound
restore pages are overwritten whole and never enter a clear list;
reservations, cancellations, and snapshot reuse keep the mark. The
change is independent of model and cache scheme.

Validation:
- Scheduler unit test: released alias pages are listed exactly once,
  when a request first commits them, and survive cancellation and
  snapshot reuse.
- TP wire tests: cleared pages round-trip through PREPARE, and ids
  outside the cache are rejected.
- CUDA test on an RTX PRO 6000 (SM120) using real VMM aliases across
  three planes: FP8 NaN bytes written through released rows land in
  freed KV pages and stay inside their alias ranges. clear_pages zeroes
  exactly the freed pages and leaves other pages and the live row
  byte-identical.
- Helios lib tests after rebasing onto 585159c: 1259 passed. The 9
  failures are the frontend, engram, and package-cache tests that fail on
  unmodified cc696a6 in the test environment; 585159c itself was not
  rerun.
- helios check with all targets and flare release build pass on
  585159c.
- Stock Qwen3.8 Flash-Next TP2 under KV pressure
  (--memory-limit 49.5G, 143 alternating 16k/32k prompts): stock
  passes 143/143. This model's QSA attention does not read partially
  filled FP8 tail pages, so the run does not reproduce NVIDIA#116.

Limitations: the NVIDIA#116 end-to-end reproduction on dense Qwen3.8-27B has
not been run. Exporting qwen3.8-27b at 6fa01a4 failed in "Warm compiled
forwards" (compiled forward has unbound static Inductor kernels). No
throughput comparison has been measured. Gemma 4 (NVIDIA#61) is untested.
The CUDA alias test and the TP2 run predate the rebase onto 585159c
and were not repeated.

Addresses NVIDIA#116.

Design-Signed-Off-By: Cursor Agent (Auto); design-review=a38f
@mark-yong
mark-yong force-pushed the fix/released-alias-pages-stale-row-bytes branch from 16da4e2 to a8d434c Compare November 4, 2026 14:35
@mark-yong
mark-yong marked this pull request as draft November 4, 2026 14:53
@original-el8 original-el8 added area:caching Prefix/KV reuse, cache retention, and model or payload cache behavior. area:memory Memory budgets, allocation lifetimes, scratch sharing, and capacity. triage:verify-for-closure May already be fixed. Submitter: retest cited fix; close if resolved or report remaining symptoms. labels Nov 5, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

The need for scheduler-level page clearing should be re-established after the attention masking fixes. Flare's pin 1f8ed8a3 includes the tail-NaN fixes that resolved the dense-Qwen reproduction in #116. FlashInfer PRs #73 and #74 additionally fix leading-window and MSA tail masking, but are newer than that Flare pin.

Marked triage:verify-for-closure: re-export with the applicable fixes and repeat the remaining DeepSeek/SM121 reproduction from #116. If it passes, this broad page-clearing workaround may be unnecessary. If it still fails, record the package/dependency revisions and failing workload before reviewing the added scheduler, executor, and TP wire changes. The residual failure has not been ruled out by this triage.

@original-el8

Copy link
Copy Markdown
Collaborator

Dependency status refresh: main 86c217a0 pins FlashInfer 07c41f329a0e. Its source contains the window-lower-bound masking in attention/paged_decode/_kernel.py and the MSA visible-extent load checks in attention/paged/forward_extend_generic.py. The older triage note that placed those fixes beyond Flare's pin is no longer applicable.

This PR remains triage:verify-for-closure. Re-export the affected model with the applicable pinned kernels and repeat the remaining DeepSeek/SM121 failure in #116 before deciding whether scheduler-level page clearing is needed. The original dense-Qwen case has separate passing evidence; it does not resolve that DeepSeek failure. This refresh checked dependency source, not a GPU reproduction.

@original-el8

Copy link
Copy Markdown
Collaborator

@mark-yong GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head a8d434c113d3 against main 97cfeb0001f1. Conflicting paths:

  • helios/src/scheduler.rs

Please reconcile the branch with main and rerun the checks affected by the resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:caching Prefix/KV reuse, cache retention, and model or payload cache behavior. area:memory Memory budgets, allocation lifetimes, scratch sharing, and capacity. model:qwen3.8-27b Affected recipe or documented reproducer: qwen3.8-27b. Impact may be broader. triage:verify-for-closure May already be fixed. Submitter: retest cited fix; close if resolved or report remaining symptoms.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure)

2 participants