Repository navigation
Fuse DSpark draft scoring and sparse support restoration - #95
Merged
Merged
Conversation
Read strided BF16/FP32 draft-head rows and optional bias directly with FP32 addition, then combine policy scoring and local candidate generation. Emit canonical support bytes during selection and restore only locally written support after each broadcast. Preserve score inspection and selector-dependent materialization, proposal probabilities, RNG reservations and rejection inputs. Implemented on the public dependency and stable-toolchain prerequisites. Markov projections remain normally prepared; target arithmetic, graph families and native metadata are unchanged. Re-export packages to capture the fusion. Qualified here with 66 CPU contracts. Historical TP2 native proposal-pipeline checks retain byte-exact dense q and sampled drafts with stable allocations; their timings exclude model forwards, Markov GEMM and target verification. GPU/native qualification of this separated branch remains pending; no isolated HTTP throughput gain is claimed. Design-Signed-Off-By: Codex (GPT-6); design-review=babb
This was referenced Oct 26, 2026
lukealonso
marked this pull request as ready for review
October 26, 2026 16:54
tunamitom
added a commit
to tunamitom/flare-local-inference
that referenced
this pull request
Oct 27, 2026
Fold every upstream commit past 3267c05, including fused DSpark draft scoring (NVIDIA#95), the selene crate, the FlashInfer 344e4837 pin, GPU clock locking and programmatic-dependency preservation in native graph loading (NVIDIA#104). Conflicts: - Dockerfile/.dockerignore: keep ignition/recipes in the image next to upstream's eos/ and selene/. - collectives: upstream transport views plus our gather block limit. - draft_sampling: upstream's support payloads, support restore and bound head-row scoring ported onto our split selection/sampling helpers; the vocabulary-shard proposals keep their own pooled path. - parallel_draft: bound logits apply only to dense proposals; the vocabulary-shard and draft-position block paths are unchanged. - tests: upstream fakes gain our block_limit and round attributes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tunamitom
added a commit
to tunamitom/flare-local-inference
that referenced
this pull request
Nov 11, 2026
Flag plumbing for the transplanted upstream NVIDIA#94/NVIDIA#95 direct-table geometry: --trellis-prefill-table 0|1|2 (default off) stamps B12X_W4A16_PREFILL_TABLE before any kernel compiles and records the selection in the plan settings, the same contract as --trellis-prefill-shapes (W3 R3). 1 = 48-row blocks, 2 stages, tile 128,128,64,256 for 1536..2560-row launches; 2 = additionally 64-row blocks, 3 stages, tile 64,128,64,128 for >=2560-row launches. Decode / verify / draft launches never see the switch (kernel-side gate). Design-Signed-Off-By: Claude (hermes-agent); design-review=a38f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resulting behavior
DSpark proposal preparation reads strided BF16 or FP32 draft-head rows directly, adds Markov bias in FP32, and combines policy scoring with local candidate selection. Selection emits canonical sparse support; after the TP result broadcast, restoration clears only support locations written by the local proposal. The materialized scoring and dense postbroadcast restore path no longer runs for this composition.
Request filters, score inspection, selector-dependent materialization, normalization order, token-zero padding, proposal probabilities, RNG reservations and target rejection inputs retain their contracts. Markov projections keep their selected preparation plans. Captured kernels change, so existing packages require re-export; the graph family, package ABI and native transport contract do not change.
Source and validation
Implemented:
astra/dspark-draft-scoringatce4729f5602341363f12aebe5b0ede2734eaa52fis one commit on Flare main82cb15a069d0756457604d8c25b45e6770f172e9. The source and tests touched by this commit are byte-identical to the qualified component revisionee285316982e262c92673c62f7a72ae109c6d853.Qualified component behavior: With FlashInfer
344e48375564a0f55af21039efd34cab3c5fd8cc, 66 CPU contracts and 64 GPU cases passed on the qualified revision. GPU checks covered independent policy/tie arithmetic, compiled and native strided-input/bias paths, poisoned producer/support storage, token-zero byte preservation, allocation stability, and TP2 divergent-peer result delivery. After rebasing onto main, 43 focused CPU tests passed with CUDA hidden. GPU tests were not repeated against Flare main or FlashInfer main9b9ed3bb7; complete model preparation and serving on that combination remain unqualified.Component measurement
Conditions: two RTX PRO 6000 Blackwell GPUs; TP2, vocabulary 129280, seven draft positions, rows 1/8/32; default autotuning, compiled CUDA graphs, real result broadcasts, 25 replays per sample and balanced ABBA order. Each sample is the maximum of the two rank-local CUDA-event intervals. The materialized control and fused candidate produce byte-exact retained dense q and sampled tokens with stable replay allocation counts. GPU clocks were not sampled.
Reproduction from
/data/flare-pr-preparation-20261026on the validation host:validation-flare/.venv/bin/python validation-flare/proposal-timing/run_timing.py scoring --output /tmp/scoring-tp2.json. The driver selects the validation configuration and GPU0/1 UUIDs before Torch imports. Raw per-rank samples and scope:validation-flare/proposal-timing/scoring-gpu01.json.Research-only performance scope: The measured chain excludes the model forwards, Markov transition GEMM, vocabulary head and target verification. It measures the combined scoring/support changes, not separate gains for each kernel. Unsupported claim: no full-model or HTTP throughput improvement has been established from this component result.