Repository navigation
Decode GLM width-one rounds in place without KDA records - #217
original-el8 wants to merge 1 commit into
Conversation
A GLM width-one speculative round verifies only the pending target token, so it has no drafts to roll back. Its target KDA layers recorded per-token corrections, and a record replay then committed one record into each live row; the round also saved and restored the draft C4 tail ring, which no draft pass touches at width one. Width-one rounds now prepare their KDA layers with the ordinary one-column decode plan, which advances each live recurrent row in place. Inactive rows keep the null state index and convolution slot -1, so their recurrent state and convolution history do not change. Width-one rounds construct no record replay and skip the convolution shift, which leaves history unchanged when a round emits at most one token, and they skip the draft tail backup and restore. They still absorb the emitted token into the draft cache and save the target output, so the following round can draft. Wider rounds are unchanged. docs/mtp-design.md describes the GLM record and in-place KDA paths, replacing a description of per-input recurrent checkpoints that the recipe does not use. Compatibility: no package metadata or prepared-contract change. Exported packages keep their captured graphs; re-export prepares the width-one graphs. A request whose width-one target distribution fails validation is marked as an error and emits no token. Its KDA row now advances by the verified token, as its convolution history already did; Helios publishes no prefix for requests with an error code. Validation on main (972e28e) with this change, on one RTX PRO 6000 (SM120), with layer 0 KDA weights of GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD at the TP4 head partition: - tests/recipes/test_glm53_mtp_state_cuda.py: 2 passed. A sequence of a width-one round, a recorded six-candidate round accepting three tokens, and a width-one round matches the same sequence with the recorded width-one plan: convolution history is bitwise equal, the recurrent row is within rtol 0.025 and atol 0.002, and the round outputs are within 1e-2 relative norm. The inactive request's row and the null row stay bitwise unchanged. test_learned_kda_verification_commits_each_prefix_before_plain_decode now resolves its plain-decode kernels before the preparation session freezes; without that step it fails on main with KernelResolutionFrozenError. - tests/recipes/test_glm53_speculative.py with the other GLM recipe, speculative-capacity and prepared-contract CPU tests: 222 passed, 47 skipped. Six tests/recipes/test_declarations.py cases fail because the Kimi and Qwen checkpoints they read are absent on the test host; they fail identically without this change. Width-one target metadata packs one state column. - DGX Spark (GB10, SM121): GLM-5.3-Flash packages exported from a tree that contains this change and other GLM recipe changes, without context parallelism, served at TP2 on two hosts and TP4 on four hosts with five drafts and adaptive verification. Each package passed a 32-prompt record-retrieval and counting check serially and at eight concurrent requests, with identical answers in both modes. Not run: a complete width-one round graph compared against ordinary decode, SM103 hardware, and a decode-time measurement that isolates this change. Design-Signed-Off-By: flare-02; design-review=4d3a
|
Categorized as The supplied KDA tests cover recurrent-state transitions and inactive rows. The remaining acceptance gap is a complete captured width-one round compared with ordinary decode, including transitions to and from wider rounds and the failed-distribution path where recurrent state has already advanced. The combined TP2/TP4 package results are useful integration evidence but do not isolate this change's numerical or timing effect. Review alongside #218, which also changes the draft-round state path. No additional GPU tests were run during triage. |
Decode GLM width-one rounds in place without KDA records Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Design-Signed-Off-By: Claude Fable 5.1 (Claude Code); design-review=4d3a Signed-off-by: Kyonjova <kyonjova@pm.me>
|
@original-el8 GitHub reports merge conflicts with
Please reconcile the branch with main and rerun the checks affected by the resolution. |
A GLM width-one speculative round verifies only the pending target token, so
it has no drafts to roll back. Its target KDA layers recorded per-token
corrections, and a record replay then committed one record into each live
row; the round also saved and restored the draft C4 tail ring, which no draft
pass touches at width one.
Width-one rounds now prepare their KDA layers with the ordinary one-column
decode plan, which advances each live recurrent row in place. Inactive rows
keep the null state index and convolution slot -1, so their recurrent state
and convolution history do not change. Width-one rounds construct no record
replay and skip the convolution shift, which leaves history unchanged when a
round emits at most one token, and they skip the draft tail backup and
restore. They still absorb the emitted token into the draft cache and save
the target output, so the following round can draft. Wider rounds are
unchanged. docs/mtp-design.md describes the GLM record and in-place KDA
paths, replacing a description of per-input recurrent checkpoints that the
recipe does not use.
Compatibility: no package metadata or prepared-contract change. Exported
packages keep their captured graphs; re-export prepares the width-one graphs.
A request whose width-one target distribution fails validation is marked as
an error and emits no token. Its KDA row now advances by the verified token,
as its convolution history already did; Helios publishes no prefix for
requests with an error code.
Validation on main (972e28e) with this change, on one RTX PRO 6000 (SM120),
with layer 0 KDA weights of GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD at the TP4 head
partition:
width-one round, a recorded six-candidate round accepting three tokens, and
a width-one round matches the same sequence with the recorded width-one
plan: convolution history is bitwise equal, the recurrent row is within
rtol 0.025 and atol 0.002, and the round outputs are within 1e-2 relative
norm. The inactive request's row and the null row stay bitwise unchanged.
test_learned_kda_verification_commits_each_prefix_before_plain_decode now
resolves its plain-decode kernels before the preparation session freezes;
without that step it fails on main with KernelResolutionFrozenError.
speculative-capacity and prepared-contract CPU tests: 222 passed,
47 skipped. Six tests/recipes/test_declarations.py cases fail because the
Kimi and Qwen checkpoints they read are absent on the test host; they fail
identically without this change. Width-one target metadata packs one state
column.
contains this change and other GLM recipe changes, without context
parallelism, served at TP2 on two hosts and TP4 on four hosts with five
drafts and adaptive verification. Each package passed a 32-prompt
record-retrieval and counting check serially and at eight concurrent
requests, with identical answers in both modes.
Not run: a complete width-one round graph compared against ordinary decode,
SM103 hardware, and a decode-time measurement that isolates this change.