Repository navigation
Add independent DFlash2 body quantization - #291
Open
voipmonitor wants to merge 3 commits into
Open
voipmonitor wants to merge 3 commits into
voipmonitor wants to merge 3 commits into
Conversation
Kimi K3.1 planning and weight loading already declare an all conversion group, but the preparation CLI fixed online conversion to KDA inputs. Exposing the existing group is necessary to prepare an MXFP8 DFlash2 body with an independently quantized proposal head without changing the recipe default. The preparation dependency must also name the merged FlashInfer sources that implement the selected dynamic KDA and DCP8 collective contracts. Add --online-mxfp8-scope to export, debug-serve, and graph benchmarking, pass it through the existing plan, and document that all also converts eligible target attention, shared-expert, and MoE latent projections. Update the FlashInfer pin and lock entry to aa6c4221586f7607c2bca64abbcdc0699845cf3b. The default remains kda-input, target vocabulary verification remains independent, and existing exported cores are unchanged. This enables an explicit combined MXFP8 DFlash2 and NVFP4 proposal-head export with the merged DCP8 collective implementation for acceptance and native serving measurements. No performance improvement is claimed by the CLI or pin change. Integrated planning tests pass with 183 passed and two deselected; both deselected collective-ladder failures reproduce on the parent without this patch. The merged FlashInfer dynamic fused-KDA host tests pass 62/62, and the release native build succeeds. Full vision export and native serving remain separate qualification steps. Design-Signed-Off-By: Claude Code (Claude Fable 5); design-review=ed22 Co-Authored-By: Claude Code <noreply@anthropic.com>
DFlash2 body conversion was reachable only through the target-wide online MXFP8 scope. That coupling prevented an MXFP8 draft-body experiment from preserving the baseline target's kda-input precision and independent proposal-head choice. Expose the existing body converter through a separate preparation setting rather than adding more meanings to the target scope. Add --draft-body-quantization mxfp8 to export, debug-serve, and bench. Thread the optional value through Speculation, plan display, Kimi preparation, and rebuild metadata. Enable DFlash2 body conversion when either the new option or the existing all scope requests it, preserving prior behavior. Reject DSpark, non-DFlash2 methods, disabled speculation, zero draft depth, and undeclared recipes. Keep target conversion and target tuning identity unchanged, and retain independent draft-head NVFP4/MXFP8 conversion. Record the body choice as optional RebuildSpeculation field 14 and include it in semantic combine and cached-weight compatibility checks. Missing metadata keeps its prior meaning; serving contracts and existing captured cores are unchanged. Document the eligible body projections and the precision retained by selector, normalization, convolution, absorbed key/value, and target-head storage. This permits a DFlash2 body in MXFP8 with an NVFP4 proposal head while the target stays kda-input and its verification head stays BF16. No throughput, memory, or quality improvement is claimed by the configuration change itself. Focused Python tests pass with 246 passed, 9 skipped, and two deselected failures that reproduce on the parent. Runtime-core tests pass 133/133, the semantic rebuild compatibility regression passes, and the cached-weight body mismatch test passes. Release Flare and Heliopause builds succeed. Full vision export, numerical replay, and native serving are separate qualification steps. Design-Signed-Off-By: Claude Code (Claude Fable 5); design-review=ed22 Co-Authored-By: Claude Code <noreply@anthropic.com>
DFlash2 bodies already use MXFP8 when online conversion selects the all scope. Comparing the optional body override directly treats omission and an explicit mxfp8 value as different even though their conversion is identical. This rejects otherwise compatible core combination and cached-weight reuse. Add shared normalization on RebuildManifest and use it in both compatibility checks. Resolve the documented DFlash2 inheritance for all and absent scopes; retain explicit and unknown values without rewriting metadata. For exports without drafts, leave body precision out of the reuse guard because only target views are consumed. Target-format and weight-view validation remain in place. Redundant explicit MXFP8 overrides can now reuse or combine with inherited MXFP8. Actual checkpoint-precision versus MXFP8 body mismatches still reject, and disabling drafts does not block reuse of compatible target weights. Stored weights, wire fields, and serving behavior are unchanged. CPU validation passed 48 core-conversion tests, 136 runtime-core tests, and five cached-weight tests. Signed-core combine cases accept explicit and inherited MXFP8, preserve each recorded override, and reject genuine format differences. Tests also cover empty scope, disabled conversion, other methods, unknown overrides, and target-only reuse. Release Flare and Heliopause builds passed. Seven GPU tests remained ignored; GPU export and native serving were not repeated for this metadata-only correction. Design-Signed-Off-By: Codex (GPT-6); design-review=ed22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DFlash2 body conversion was reachable only through the target-wide
online_mxfp8_scope=allsetting. That couples an isolated draft-body precision experiment to unrelated target attention, shared-expert, and MoE latent conversions. We need to preserve the baseline target'skda-inputprecision while independently selecting DFlash2 body and proposal-head formats.The DFlash2 loader already has the required MXFP8 body converter. This PR exposes it through an independent speculation setting rather than adding combined meanings to the target scope.
What changed
--draft-body-quantization {mxfp8}to export, debug-serve, and bench. Thread the optional value throughSpeculation, plan display, Kimi preparation, and rebuild metadata.allscope requests it, preserving existing behavior. Reject DSpark, non-DFlash2 methods, disabled speculation, zero draft depth, and undeclared recipes.--draft-head-quantization {nvfp4,mxfp8}remains independent, as does target verification-head precision.--online-mxfp8-scopeand update the FlashInfer pin toaa6c4221586f7607c2bca64abbcdc0699845cf3bfor the merged dynamic-KDA and DCP collective implementation.draft_body_quantizationasRebuildSpeculationfield 14. Combine and cached-weight checks compare effective DFlash2 body precision: omission under online MXFP8all(including an absent stored scope) matches an explicitmxfp8override. Actual BF16/MXFP8 body mismatches still reject. Target-only reuse ignores unused draft-body settings while retaining target-format and tensor-view checks.Missing metadata retains its previous meaning. Existing captured cores and serving contracts are unchanged.
Result
The requested configuration is now expressible directly: target
kda-input, DFlash2 body MXFP8, NVFP4 proposal head, and BF16 target verification head. No performance improvement is claimed from the CLI or metadata changes themselves.A real DCP8 vision export with this configuration completed with 71 graphs. Native serving started without a manual memory reserve or OOM and allocated 1,776,640 KV tokens with 86 snapshot slots. An actual image request produced the expected answer with 100 multimodal image tokens.
Post-signing native measurements were:
The DCP16 variant with the same precision configuration also completed with 71 graphs. Native serving started without a manual reserve or OOM and allocated 3,557,376 KV tokens with 88 snapshot slots. Its post-signing measurements were:
Both variants answered an actual image request correctly with 100 multimodal image tokens. C4 acceptance was 4.3165 tokens/step for DCP8 and 3.9536 for DCP16 versus 6.4543 for the earlier combined target-and-draft
allconfiguration. These runs are not a controlled same-vision A/B, so a draft-body performance or quality benefit is not established.Validation
kda-input, DFlash2 bodymxfp8, proposal headnvfp4, and MoonViT metadata. C1/C4 responses are nonempty without replacement characters.The effective-precision compatibility correction was validated with CPU packaging/reuse tests and release builds; the native model measurements above were not rerun for that metadata-only correction.
No full model-quality benchmark or complete Helios suite was run for this PR.
🤖 Generated with Claude Code