Repository navigation
Resolve compiler math libraries before native preparation - #88
Merged
lukealonso merged 1 commit intoOct 26, 2026
Merged
Conversation
Select the CUDA math library required by compiled-forward eager precision before constructing recorded preparation sessions or entering standalone operator preparation. Native program discovery and materialization then use the same compiler options across serial and overlapping graph catalogs. Inductor initializes toolkit libdevice when Flare's emulate_precision_casts option first compiles a forward. Previously, b12x could memoize discovery with Triton's bundled library first, then request another library's program key while runtime compilation was forbidden. The two library files differ in content; retaining their distinct identities is required. Implemented in shared preparation without changing numerical options, kernel selection, cache equivalence or compilation guards. Qualified with 80 CPU checks, including two library-selection regressions that fail on prerequisite a2c9278 and pass with this correction. The regressions prohibit CUDA initialization and perform no compilation. GPU serial-to-overlap preparation and native replay remain unqualified pending an assigned validation window. Design-Signed-Off-By: Codex (GPT-6); design-review=babb
lukealonso
marked this pull request as ready for review
October 26, 2026 13:50
lukealonso
pushed a commit
that referenced
this pull request
Oct 26, 2026
* Pin FlashInfer to the supported tuning-tier API Flare preparation declares off/default/audit tuning, but FlashInfer 43eaf93fa accepts off/low/medium/high/max and rejects default preparation before any operator is declared. Pin public FlashInfer 344e48375 in pyproject.toml and uv.lock so dependency resolution supplies the matching tuning API. Implemented: update only the FlashInfer Git revision; retain every other locked dependency. Existing environments require synchronization to use the corrected revision. Kernel execution changes remain those of the pinned public dependency. Qualified: the CPU reproducer rejects default and audit with 43eaf93fa and accepts both with 344e48375. uv lock --check --offline passes. Raw reproducer and output are under /data/flare-pr-preparation-20261026/validation-flare. GPU and native component qualification with the corrected pin is pending. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Preserve FP32 DeepSeek V4.1 target vocabulary logits (#86) Use BF16 checkpoint head weights with FP32 accumulation and output for target prefill, decode, verification, prompt scoring and cached-prefix resume. Keep BF16 proposal logits on a separate fixed-width PCIe channel. Shared gathers transport FP32 words through bit-preserving BF16 views; resume and prompt scoring retain BF16 defaults for other recipes. Implemented: remove target-logit BF16 rounding without expanding or quantizing weights. Existing packages require re-export to use the corrected graphs. The reference expands BF16 weights and inputs to FP32; BF16-input FP32-output GEMM has a different reduction order and is not bit-identical to that SGEMM. Qualified on this upstream-based branch: 40 focused CPU contracts with CUDA hidden and CUDA initialization prohibited. Existing integration evidence at Flare 33ef5407 / FlashInfer 98a53bf5 covers actual-head rows 1/8/32/64, scored prefill, TP2 native gathering and resume capture. The M64 expanded-FP32 comparison measured relative RMS about 5.6e-6 and maximum error about 5e-5. Those GPU results do not qualify this branch with its pinned dependency. GPU/native replay, package export and cross-host serving remain unrun here. This is a correctness correction with no speedup claim. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Gather DeepSeek V4.1 compressor continuation fields in one launch (#87) Read pending FP32 values, FP32 gates and Int64 positions through one typed row gather into separate buffers. Preserve compression arithmetic, continuation layout and accepted-prefix commits while removing two captured CUDA nodes per ratio-two compressor. Re-export packages to include the fused gather. Implemented independently on NVIDIA main 3267c05 with unchanged dependency pins. The pinned FlashInfer 43eaf93fa compression API matches the qualified 98a53bf5 source; no additional FlashInfer API is required. Qualified here: 12 CPU checks with CUDA hidden and initialization prohibited. Existing Flare 60197fcd / FlashInfer 98a53bf5 evidence covers 12 compiled/native GPU cases, byte identity, mutable rows, inactive requests, accepted counts 0–8, source poisoning and allocation stability. Saved component ABBA measurements at one request/eight token rows/layer 2/five accepted tokens are 17.208 to 15.144 us cache-flushed and 13.004 to 11.112 us with repeated inputs; projection inputs are precomputed. Six integrated natural-EOS service pairs completed without request failures, with no statistically isolated HTTP gain established. GPU replay, package export and service qualification of this main-based branch remain pending. Design-Signed-Off-By: Codex (GPT-6); design-review=babb * Resolve compiler math libraries before native preparation (#88) Select the CUDA math library required by compiled-forward eager precision before constructing recorded preparation sessions or entering standalone operator preparation. Native program discovery and materialization then use the same compiler options across serial and overlapping graph catalogs. Inductor initializes toolkit libdevice when Flare's emulate_precision_casts option first compiles a forward. Previously, b12x could memoize discovery with Triton's bundled library first, then request another library's program key while runtime compilation was forbidden. The two library files differ in content; retaining their distinct identities is required. Implemented in shared preparation without changing numerical options, kernel selection, cache equivalence or compilation guards. Qualified with 80 CPU checks, including two library-selection regressions that fail on prerequisite a2c9278 and pass with this correction. The regressions prohibit CUDA initialization and perform no compilation. GPU serial-to-overlap preparation and native replay remain unqualified pending an assigned validation window. Design-Signed-Off-By: Codex (GPT-6); design-review=babb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serial and overlapping attention preparation in one process can fail with an “uncached Triton program” error even after the expected program was compiled.
Flare records native program identities while Triton is using its bundled
libdevicemath library. The first Inductor forward compilation then selects the CUDA toolkit’s library for Flare’s existingemulate_precision_castspolicy. Discovery has retained the earlier compiler options, while materialization uses the other library and requests a different program identity.Resolve that existing Inductor choice before recorded-session initialization and standalone operator preparation. Both stages then see the same library. The fix is shared across recipes and introduces no setting, kernel selection change, weakened cache check or runtime compilation.
Implemented and qualified with public FlashInfer
344e48375, Torch 2.14.1+cu132 and Triton 3.8.0:The GPU reproducer combines this commit with the separately reviewed query-overlap change. The two
libdevicefiles differ in content, so their identities remain distinct even though this quantizer happened to emit identical PTX. No numerical oracle or no-compilation guard was changed, and no inference speedup is claimed.Depends on #85; the diff contains only this preparation-order correction and its tests.