Repository navigation
Conversation
FLARE_KIMI3_ROUTE_ROWS=16 (default 8) raises routing.MAX_ROWS, i.e. the route channel's capacity and the decoder's tokens <= ROUTE_ROWS gate, so verify batches of two concurrent speculative streams (9..16 rows) keep the b12x paired route gather + Kimi top-16 instead of falling to the general path (latent all-reduce + router all-gather + route top-k). Needs a b12x whose kimi_topk16 accepts 16 rows; with an older b12x, preparing a 9..16-row decoder fails loudly. Meant for B12X_PCIE_DCP_PAIR_PUSH=1. Any other value than 8 or 16 raises. Default 8 keeps today's graphs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 Description
FLARE_KIMI3_ROUTE_ROWS=16(default 8). It is read at import ofkimi3/routing.py, so set it at export and serve. It raisesrouting.MAX_ROWS, which sets the route channel's capacity and the decoder'stokens <= ROUTE_ROWSgate. With it, verify batches of two concurrent speculative streams (9..16 rows) keep the b12x paired route gather + Kimi top-16 instead of the general path (latent all-reduce + router all-gather + route top-k). Values other than 8 or 16 raise.It needs FlashInfer with 16-row
kimi_topk16(local-inference-lab/flashinfer NVIDIA#101), and it is meant forB12X_PCIE_DCP_PAIR_PUSH=1(NVIDIA#91). The numbers, tests and topology notes are the same as in the FlashInfer PR: above 8 rows the MoE route step goes from ~51-54 us to ~15 us per layer at 9-10 rows on TP8.