Repository navigation
Kimi K3.1: raise the recipe context to 1,048,576 (maintainer decision requested) - #19
Open
tunamitom wants to merge 1 commit into
Conversation
Question for the maintainer, not a fait accompli: this commit proposes undoing the 262,144 choice of 669fdfc ("Prepare a 262144-token K3.1 profile ..."). If you want to keep 262,144, drop this commit; the MXFP8 credit calibration is a separate branch and stands either way. Why 1,048,576: the engine (kimi3/bootstrap.py validates context in [page_tokens, 1048576]) and both checkpoint configs cap positions at 1,048,576 (max_position_embeddings in the target and dflash2 configs), and the value is baked into every exported package as max_sequence_tokens and page-table width, so 262,144 caps requests at a quarter of the model's range in any re-export. Memory accounting at the C4 P4096 profile (the profile 669fdfc qualified): the bound feeds the planner's page demand (concurrency * context / page_tokens: 64 -> 256 pages per slot at P4096), max_body_bytes, the DSA record slabs (+0.52 GiB per rank resident at 1M vs 262k) and prepared_sequence_tokens. With the calibrated MXFP8 credit the c4 shape keeps about 1.0 GiB of supply spare at the 262,144 context and about 0.5 GiB at 1M (recorded supply 94.010 GiB at the qualification profile). Boot does not need one full 1M request to fit: fit_context (helios/src/model.rs) clamps max_sequence_tokens to min(prepared_sequence_tokens, pages * page_tokens), so a boot whose KV pool is smaller advertises less than 1M instead of failing; the earlier "one full 1M request must fit the pool the first boot reports" claim was wrong. Docs: docs/kimi-k3.1-exl3-handoff.md now records the signed export's 262,144 against the recipe's declared 1,048,576. The review also cited docs/core-design.md and docs/model-preparation.md: both state the Qwen recipe's 262,144 context (qwen38 caps context at 262144) and are not contradicted here. Validation: tests/recipes/test_kimi31.py and tests/cli/test_models.py on this tree at parity with the release-lane base (CPU only). Material limitations: no CUDA preparation or boot was re-run on this tree; the +0.52 GiB/rank slab figure and the 94.010 GiB supply are recorded qualification measurements, not re-measured here. Design-Signed-Off-By: Hermes Agent (hermes-agent); design-review=a38f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Question for the maintainer, not a fait accompli
This PR proposes undoing the 262,144 choice of 669fdfc ("Prepare a 262144-token K3.1 profile ..."). It is submitted as a standalone commit so it can be accepted or dropped without touching anything else. If 262,144 stands, drop this PR — the MXFP8 credit calibration (separate PR) is independent and lands either way.
What / why (the question, with the memory accounting)
Why 1,048,576: the engine (kimi3/bootstrap.py validates context in [page_tokens, 1048576]) and both checkpoint configs cap positions at 1,048,576 (
max_position_embeddingsin the target and dflash2 configs), and the value is baked into every exported package asmax_sequence_tokensand page-table width, so 262,144 caps requests at a quarter of the model's range in any re-export.Memory accounting at the C4 P4096 profile (the profile 669fdfc qualified, documented at the base tip): the bound feeds the planner's page demand (
concurrency * context / page_tokens: 64 -> 256 pages per slot at P4096),max_body_bytes, the DSA record slabs (+0.52 GiB per rank resident at 1M vs 262k) andprepared_sequence_tokens. With the calibrated MXFP8 credit (separate PR) the C4 shape keeps about 1.0 GiB of supply spare at 262,144 and about 0.5 GiB at 1M (recorded supply 94.010 GiB at the qualification profile) — the spares are stated at each context so the two effects are not conflated.Boot behaviour: boot does not require one full 1M request to fit the pool;
fit_context(helios/src/model.rs) clampsmax_sequence_tokenstomin(prepared_sequence_tokens, pages * page_tokens), so a boot whose KV pool is smaller advertises less than 1M instead of failing. The real effect of the change is that the advertised context becomes pool-limited, not that boot fails.Docs:
docs/kimi-k3.1-exl3-handoff.mdnow records the signed export's 262,144 against the recipe's declared 1,048,576 (the signed package stays accurate as a qualification record; re-exports bake the larger maximum).docs/core-design.mdanddocs/model-preparation.mdstate the Qwen recipe's 262,144 context (qwen38caps context at 262144) and are not contradicted by this change — no edit made there.Numerics class
NUMERICS-NEUTRAL — a declared context bound and documentation only; no arithmetic change.
Tests
tests/recipes/test_kimi31.py25 passed;tests/cli/test_models.py40 passed;tests/cli641 passed, same 2 pre-existing failures as the base.Dependencies: none on the companion FlashInfer PRs; independent of the calibration PR.