Skip to content

Decode GLM width-one rounds in place without KDA records - #217

Open
original-el8 wants to merge 1 commit into
mainfrom
perf/glm53-width-one-in-place
Open

original-el8 wants to merge 1 commit into
mainfrom
perf/glm53-width-one-in-place

Conversation

@original-el8

Copy link
Copy Markdown
Collaborator

A GLM width-one speculative round verifies only the pending target token, so
it has no drafts to roll back. Its target KDA layers recorded per-token
corrections, and a record replay then committed one record into each live
row; the round also saved and restored the draft C4 tail ring, which no draft
pass touches at width one.

Width-one rounds now prepare their KDA layers with the ordinary one-column
decode plan, which advances each live recurrent row in place. Inactive rows
keep the null state index and convolution slot -1, so their recurrent state
and convolution history do not change. Width-one rounds construct no record
replay and skip the convolution shift, which leaves history unchanged when a
round emits at most one token, and they skip the draft tail backup and
restore. They still absorb the emitted token into the draft cache and save
the target output, so the following round can draft. Wider rounds are
unchanged. docs/mtp-design.md describes the GLM record and in-place KDA
paths, replacing a description of per-input recurrent checkpoints that the
recipe does not use.

Compatibility: no package metadata or prepared-contract change. Exported
packages keep their captured graphs; re-export prepares the width-one graphs.
A request whose width-one target distribution fails validation is marked as
an error and emits no token. Its KDA row now advances by the verified token,
as its convolution history already did; Helios publishes no prefix for
requests with an error code.

Validation on main (972e28e) with this change, on one RTX PRO 6000 (SM120),
with layer 0 KDA weights of GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD at the TP4 head
partition:

  • tests/recipes/test_glm53_mtp_state_cuda.py: 2 passed. A sequence of a
    width-one round, a recorded six-candidate round accepting three tokens, and
    a width-one round matches the same sequence with the recorded width-one
    plan: convolution history is bitwise equal, the recurrent row is within
    rtol 0.025 and atol 0.002, and the round outputs are within 1e-2 relative
    norm. The inactive request's row and the null row stay bitwise unchanged.
    test_learned_kda_verification_commits_each_prefix_before_plain_decode now
    resolves its plain-decode kernels before the preparation session freezes;
    without that step it fails on main with KernelResolutionFrozenError.
  • tests/recipes/test_glm53_speculative.py with the other GLM recipe,
    speculative-capacity and prepared-contract CPU tests: 222 passed,
    47 skipped. Six tests/recipes/test_declarations.py cases fail because the
    Kimi and Qwen checkpoints they read are absent on the test host; they fail
    identically without this change. Width-one target metadata packs one state
    column.
  • DGX Spark (GB10, SM121): GLM-5.3-Flash packages exported from a tree that
    contains this change and other GLM recipe changes, without context
    parallelism, served at TP2 on two hosts and TP4 on four hosts with five
    drafts and adaptive verification. Each package passed a 32-prompt
    record-retrieval and counting check serially and at eight concurrent
    requests, with identical answers in both modes.

Not run: a complete width-one round graph compared against ordinary decode,
SM103 hardware, and a decode-time measurement that isolates this change.

A GLM width-one speculative round verifies only the pending target token, so
it has no drafts to roll back. Its target KDA layers recorded per-token
corrections, and a record replay then committed one record into each live
row; the round also saved and restored the draft C4 tail ring, which no draft
pass touches at width one.

Width-one rounds now prepare their KDA layers with the ordinary one-column
decode plan, which advances each live recurrent row in place. Inactive rows
keep the null state index and convolution slot -1, so their recurrent state
and convolution history do not change. Width-one rounds construct no record
replay and skip the convolution shift, which leaves history unchanged when a
round emits at most one token, and they skip the draft tail backup and
restore. They still absorb the emitted token into the draft cache and save
the target output, so the following round can draft. Wider rounds are
unchanged. docs/mtp-design.md describes the GLM record and in-place KDA
paths, replacing a description of per-input recurrent checkpoints that the
recipe does not use.

Compatibility: no package metadata or prepared-contract change. Exported
packages keep their captured graphs; re-export prepares the width-one graphs.
A request whose width-one target distribution fails validation is marked as
an error and emits no token. Its KDA row now advances by the verified token,
as its convolution history already did; Helios publishes no prefix for
requests with an error code.

Validation on main (972e28e) with this change, on one RTX PRO 6000 (SM120),
with layer 0 KDA weights of GLM-5.3-Flash-NVFP4-MXFP8-CSF-QAD at the TP4 head
partition:
- tests/recipes/test_glm53_mtp_state_cuda.py: 2 passed. A sequence of a
  width-one round, a recorded six-candidate round accepting three tokens, and
  a width-one round matches the same sequence with the recorded width-one
  plan: convolution history is bitwise equal, the recurrent row is within
  rtol 0.025 and atol 0.002, and the round outputs are within 1e-2 relative
  norm. The inactive request's row and the null row stay bitwise unchanged.
  test_learned_kda_verification_commits_each_prefix_before_plain_decode now
  resolves its plain-decode kernels before the preparation session freezes;
  without that step it fails on main with KernelResolutionFrozenError.
- tests/recipes/test_glm53_speculative.py with the other GLM recipe,
  speculative-capacity and prepared-contract CPU tests: 222 passed,
  47 skipped. Six tests/recipes/test_declarations.py cases fail because the
  Kimi and Qwen checkpoints they read are absent on the test host; they fail
  identically without this change. Width-one target metadata packs one state
  column.
- DGX Spark (GB10, SM121): GLM-5.3-Flash packages exported from a tree that
  contains this change and other GLM recipe changes, without context
  parallelism, served at TP2 on two hosts and TP4 on four hosts with five
  drafts and adaptive verification. Each package passed a 32-prompt
  record-retrieval and counting check serially and at eight concurrent
  requests, with identical answers in both modes.

Not run: a complete width-one round graph compared against ordinary decode,
SM103 hardware, and a decode-time measurement that isolates this change.

Design-Signed-Off-By: flare-02; design-review=4d3a
@original-el8 original-el8 added area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 6, 2026
@original-el8

Copy link
Copy Markdown
Collaborator Author

Categorized as triage:review, performance, area:kernels, and area:preparation. The change removes redundant KDA recording/replay and draft-tail backup from width-one GLM rounds; it is not evidence that an existing correctness report is fixed.

The supplied KDA tests cover recurrent-state transitions and inactive rows. The remaining acceptance gap is a complete captured width-one round compared with ordinary decode, including transitions to and from wider rounds and the failed-distribution path where recurrent state has already advanced. The combined TP2/TP4 package results are useful integration evidence but do not isolate this change's numerical or timing effect. Review alongside #218, which also changes the draft-round state path. No additional GPU tests were run during triage.

kyonjova added a commit to kyonjova/flare-local-inference that referenced this pull request Nov 8, 2026
Decode GLM width-one rounds in place without KDA records

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Design-Signed-Off-By: Claude Fable 5.1 (Claude Code); design-review=4d3a
Signed-off-by: Kyonjova <kyonjova@pm.me>
@original-el8 original-el8 added the model:glm-5.3 Affected recipe or documented reproducer: glm-5.3. Impact may be broader. label Nov 8, 2026
@original-el8

Copy link
Copy Markdown
Collaborator Author

@original-el8 GitHub reports merge conflicts with main, confirmed locally with git merge-tree for PR head 8aab9c9e71ad against main 97cfeb0001f1. Conflicting paths:

  • ignition/recipes/glm53/speculative.py
  • tests/recipes/test_glm53_speculative.py

Please reconcile the branch with main and rerun the checks affected by the resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kernels GPU kernels and numerical execution. area:preparation Ignition checkpoint loading, tuning, compilation, capture, and export. model:glm-5.3 Affected recipe or documented reproducer: glm-5.3. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant