Repository navigation
Append DeepSeek V4.1 window snapshots during cache writes - #108
Merged
Merged
Conversation
Prepare the verification and draft sliding-window history copy with the cache append, using caller-owned history and continuation row IDs. Preserve the existing prefill copy and ring-commit behavior. Pin the published FlashInfer snapshot-writer revision so this main-based branch builds independently. Validation: 5 CPU tests and 4 SM120 compiled native GPU cases passed with the pinned FI revision. The GPU cases cover verification width eight and draft width seven at one and three rows, exact staging and persistent cache bytes, poisoned reusable storage, allocation stability, and one fewer graph node. The combined feature set native throughput is not attributed to this isolated writer. Design-Signed-Off-By: Codex GPT-6.1; design-review=2d93
Keep the existing CUDA toolkit platform markers byte-for-byte while updating only the three immutable FlashInfer revision references. This avoids unrelated Windows and non-Linux dependency-resolution changes in the window snapshot consumer PR. Validation: uv lock --check --offline and uv lock --locked --offline pass; the source and tested FI revision are unchanged, so no GPU replay test was repeated. Design-Signed-Off-By: Codex GPT-6.1; design-review=2d93
This was referenced Oct 28, 2026
lukealonso
added a commit
that referenced
this pull request
Oct 29, 2026
Combine immutable window-prefix copies with appended KV encoding during verify and draft preparation. Preserve lockfile platform markers. Cherry-picked from cb5b34e and 02d7328 (PR #108), with dependency conflicts resolved against local main. Validation: 5 Ignition snapshot preparation tests and 27 FlashInfer snapshot/word-exchange contract tests passed. CUDA and full-model export checks were not run; target GPUs are occupied. Design-Signed-Off-By: Codex GPT-6; design-review=1c01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DeepSeek V4.1 draft and verification passes need the saved 128-token window history alongside the KV rows they have just computed. They currently prepare these in two operations: copy the history into a working buffer, then append the KV rows.
This PR makes the b12x cache writer do both in one operation. It removes one graph node while producing exactly the same cache bytes. The recipe supplies the history buffer and request row IDs explicitly, so the copy remains safe when request slots are reused and graphs are replayed natively. Prefill and the commit of accepted tokens keep their existing behavior.
The change is implemented and qualified for the tested cache layouts. Using it requires re-exporting the model; the package ABI and serving API stay the same. Unsupported history lengths or destination page layouts are rejected during preparation.
The required writer API is already in FlashInfer main through FI #57. This branch pins the published revision used for its tests. When resolving the dependency conflict with main, retain a revision containing that API rather than downgrading FlashInfer.
Validation passed:
That combined build averaged 130.47 steps/s across three Tetris requests. Several changes are included in that number; the end-to-end speedup of this cache-writer change has not been measured separately.