Repository navigation
Gather DeepSeek V4.1 compressor state in one kernel - #87
Merged
lukealonso merged 1 commit intoOct 26, 2026
Merged
Conversation
Read pending FP32 values, FP32 gates and Int64 positions through one typed row gather into separate buffers. Preserve compression arithmetic, continuation layout and accepted-prefix commits while removing two captured CUDA nodes per ratio-two compressor. Re-export packages to include the fused gather. Implemented independently on NVIDIA main 3267c05 with unchanged dependency pins. The pinned FlashInfer 43eaf93fa compression API matches the qualified 98a53bf5 source; no additional FlashInfer API is required. Qualified here: 12 CPU checks with CUDA hidden and initialization prohibited. Existing Flare 60197fcd / FlashInfer 98a53bf5 evidence covers 12 compiled/native GPU cases, byte identity, mutable rows, inactive requests, accepted counts 0–8, source poisoning and allocation stability. Saved component ABBA measurements at one request/eight token rows/layer 2/five accepted tokens are 17.208 to 15.144 us cache-flushed and 13.004 to 11.112 us with repeated inputs; projection inputs are precomputed. Six integrated natural-EOS service pairs completed without request failures, with no statistically isolated HTTP gain established. GPU replay, package export and service qualification of this main-based branch remain pending. Design-Signed-Off-By: Codex (GPT-6); design-review=babb
lukealonso
marked this pull request as ready for review
October 26, 2026 13:50
lukealonso
pushed a commit
that referenced
this pull request
Oct 26, 2026
* Pin FlashInfer to the supported tuning-tier API Flare preparation declares off/default/audit tuning, but FlashInfer 43eaf93fa accepts off/low/medium/high/max and rejects default preparation before any operator is declared. Pin public FlashInfer 344e48375 in pyproject.toml and uv.lock so dependency resolution supplies the matching tuning API. Implemented: update only the FlashInfer Git revision; retain every other locked dependency. Existing environments require synchronization to use the corrected revision. Kernel execution changes remain those of the pinned public dependency. Qualified: the CPU reproducer rejects default and audit with 43eaf93fa and accepts both with 344e48375. uv lock --check --offline passes. Raw reproducer and output are under /data/flare-pr-preparation-20261026/validation-flare. GPU and native component qualification with the corrected pin is pending. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Preserve FP32 DeepSeek V4.1 target vocabulary logits (#86) Use BF16 checkpoint head weights with FP32 accumulation and output for target prefill, decode, verification, prompt scoring and cached-prefix resume. Keep BF16 proposal logits on a separate fixed-width PCIe channel. Shared gathers transport FP32 words through bit-preserving BF16 views; resume and prompt scoring retain BF16 defaults for other recipes. Implemented: remove target-logit BF16 rounding without expanding or quantizing weights. Existing packages require re-export to use the corrected graphs. The reference expands BF16 weights and inputs to FP32; BF16-input FP32-output GEMM has a different reduction order and is not bit-identical to that SGEMM. Qualified on this upstream-based branch: 40 focused CPU contracts with CUDA hidden and CUDA initialization prohibited. Existing integration evidence at Flare 33ef5407 / FlashInfer 98a53bf5 covers actual-head rows 1/8/32/64, scored prefill, TP2 native gathering and resume capture. The M64 expanded-FP32 comparison measured relative RMS about 5.6e-6 and maximum error about 5e-5. Those GPU results do not qualify this branch with its pinned dependency. GPU/native replay, package export and cross-host serving remain unrun here. This is a correctness correction with no speedup claim. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Gather DeepSeek V4.1 compressor continuation fields in one launch (#87) Read pending FP32 values, FP32 gates and Int64 positions through one typed row gather into separate buffers. Preserve compression arithmetic, continuation layout and accepted-prefix commits while removing two captured CUDA nodes per ratio-two compressor. Re-export packages to include the fused gather. Implemented independently on NVIDIA main 3267c05 with unchanged dependency pins. The pinned FlashInfer 43eaf93fa compression API matches the qualified 98a53bf5 source; no additional FlashInfer API is required. Qualified here: 12 CPU checks with CUDA hidden and initialization prohibited. Existing Flare 60197fcd / FlashInfer 98a53bf5 evidence covers 12 compiled/native GPU cases, byte identity, mutable rows, inactive requests, accepted counts 0–8, source poisoning and allocation stability. Saved component ABBA measurements at one request/eight token rows/layer 2/five accepted tokens are 17.208 to 15.144 us cache-flushed and 13.004 to 11.112 us with repeated inputs; projection inputs are precomputed. Six integrated natural-EOS service pairs completed without request failures, with no statistically isolated HTTP gain established. GPU replay, package export and service qualification of this main-based branch remain pending. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Resolve compiler math libraries before native preparation (#88) Select the CUDA math library required by compiled-forward eager precision before constructing recorded preparation sessions or entering standalone operator preparation. Native program discovery and materialization then use the same compiler options across serial and overlapping graph catalogs. Inductor initializes toolkit libdevice when Flare's emulate_precision_casts option first compiles a forward. Previously, b12x could memoize discovery with Triton's bundled library first, then request another library's program key while runtime compilation was forbidden. The two library files differ in content; retaining their distinct identities is required. Implemented in shared preparation without changing numerical options, kernel selection, cache equivalence or compilation guards. Qualified with 80 CPU checks, including two library-selection regressions that fail on prerequisite a2c9278 and pass with this correction. The regressions prohibit CUDA initialization and perform no compilation. GPU serial-to-overlap preparation and native replay remain unqualified pending an assigned validation window. Design-Signed-Off-By: Codex (GPT-6); design-review=babb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DeepSeek V4.1 ratio-two compression copies pending values, gates and positions with three CUDA launches. Gather those FP32/FP32/Int64 fields in one launch, using the same row map and separate output buffers. Compressor arithmetic, continuation layout and accepted-prefix commits remain unchanged; each compressor graph has two fewer kernel nodes.
Implemented as a shared typed-gather component and its recipe call site. No dependency or preparation-policy changes are included. Existing packages require re-export to contain the fused gather. This PR depends on #85 and is based on its branch to keep the feature diff independent.
Qualified on this branch with public FlashInfer
344e48375: 12 CPU checks and 12 compiled/native GPU cases pass on RTX PRO 6000 Blackwell. Coverage includes strided and mutable row selection, exact mixed-type bytes, tails, inactive requests, accepted counts 0–8, source/scratch reuse, allocation stability and the two-node reduction.This requires the qualification-enabled Heliopause extension described in the preparation guide.
Component ABBA timing for the same gather/compressor implementation, one request/eight rows/layer 2/five accepted tokens:
The interval includes gathering, compression and accepted-state commit with precomputed projection inputs, not the whole attention layer or model. Hardware: RTX PRO 6000 Blackwell, CUDA 13.2, Torch 2.14.1+cu132, automatic core clocks and +6000 memory offset. All four measured component cases improved. Those timings were collected on integration Flare
60197fcd/ FI98a53bf5; the separated branch’s fresh checks above establish correctness, not a repeated latency measurement.Six integrated TP4/C1/K7 service pairs also completed without errors. They do not establish an isolated HTTP speedup for this change. Standalone package export/load and wider service configurations remain unqualified on this separated branch.