Repository navigation
Calibrate the EXL3 MXFP8 conversion credit from measured rank storage - #18
Open
tunamitom wants to merge 1 commit into
Conversation
_exl3_mxfp8_savings credited 37.5% of counted BF16 projection bytes while the measured conversion saves more, leaving the planner's per-rank weights estimate at 89.16 GiB against 88.03 GiB of measured converted storage (1.11 GiB per rank of bookkeeping pessimism). The credit is now calibrated against measurement and bounded below the worst measured rank, so the estimate can never land under the measured storage and a shortfall surfaces as a planner refusal instead of a preparation OOM. Arithmetic (Kimi-K3.1-Pro EXL3-2bit, dflash2 draft, TP8): - counted BF16 projections: 2 x elements = 95,626,985,472 B; latent pair credited at 1/16 = 590,872,576 B; - per-rank counted weights before credit = device 802,341,903,064 / 8 = 100,292,737,883 B (device = stored 796,968,398,552 + draft 5,373,504,512); - measured converted storage per rank (weights/tp8/rank-0..7 across 12 TP8 exports of this checkpoint): largest rank 94,526,085,728 B (88.034 GiB, rank 0 in every export), smallest 94,515,137,120 B (88.024 GiB). - worst-rank bound: credit <= 8 x (100,292,737,883 - 94,526,085,728) = 46,133,217,240 B, i.e. 47.62% of counted BF16 bytes after the latent term. The credit is rounded below that bound to 47.5% (475/1000): savings 46,013,690,675 B land the per-rank estimate at 94,541,026,549 B (88.05 GiB), 14 MiB above the largest measured rank, and 0.9 point below the 48.4% ideal reduction of the format. Validation: tests/cli/test_models.py updated to the calibrated formula and bounds (CPU). Material limitations: no TP16 export of this checkpoint exists to measure, so the bound rests on TP8 measurement only; no CUDA preparation was re-run here; one calibration serves every EXL3 kimi_k3/kimi_k31 checkpoint at every TP size, and other checkpoints' conversion rates are unmeasured. Design-Signed-Off-By: Hermes Agent (hermes-agent); design-review=a38f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
_exl3_mxfp8_savingscredited 37.5% of counted BF16 projection bytes while the measured conversion saves more, leaving the planner's per-rank weights estimate at 89.16 GiB against 88.03 GiB of measured converted storage (1.11 GiB per rank of bookkeeping pessimism). The credit is now calibrated against measurement and bounded below the worst measured rank, so the estimate can never land under the measured storage — a shortfall surfaces as a planner refusal, not a preparation OOM.Calibration evidence (all ranks; Kimi-K3.1-Pro EXL3-2bit, dflash2 draft, TP8):
Provenance: the calibration target is the served signed TP8 export of this checkpoint. At the C4 P4096 profile (recorded dry-run, supply 94.010 GiB, 262,144 context) the 37.5% credit refused the C4 shape (fixed 94.055 > supply); the calibrated credit removes 1.11 GiB/rank and the shape fits with about 1.0 GiB spare. Context and credit effects do not interact in this branch — the 1M-context change is proposed separately as its own PR and this calibration lands either way.
Numerics class
NUMERICS-NEUTRAL — planner bookkeeping and a declared estimate only; no arithmetic change.
Tests
tests/cli/test_models.py— 40 passed (CPU), formula and bounds updated to the calibrated credit.tests/recipes/test_kimi31.py25 passed;tests/cli641 passed, same 2 pre-existing failures as the base.Material limitations: no TP16 export of this checkpoint exists to measure, so the bound rests on TP8 measurement only; no CUDA preparation was re-run here; one calibration serves every EXL3 kimi_k3/kimi_k31 checkpoint at every TP size, and other checkpoints' conversion rates are unmeasured.
Dependencies: none on the companion FlashInfer PRs.