Skip to content

GLM-5.3-Flash on main: export fails and prefix-cache restores fail with device error 72 #272

Closed

Description

@yatesdr

GLM-5.3-Flash can't be served from main (0572fd1). There are two independent problems.

1. Export fails while compiling, on every rank:

torch._dynamo.exc.UserError: Could not extract specialized integer from data-dependent expression u51
  ... block_fp8.py _project -> plan_from_handle(handle)

Since 4a24d78, _project is traced and looks up its plan from a handle that preparation marks as unbacked.

2. Every prefix-cache restore fails with prepared model reported device error 72 once export works. The first request succeeds and every later request fails.

Merge 2b38bf9 kept #165's two-page trailing regions but restored the older trailing_groups[region.nbytes] key (state_transfer.py:273). That key is two pages, so each shifted MTP draft copy writes two pages at the wrong offset. The copy overwrites last_hidden and consumed_tokens, and the next restore fails validation. Only GLM-5.3 passes trailing pages, so no other model is affected.

Repro (2x RTX PRO 6000, SM120):

ignition export glm-5.3-flash --gpus 0,1 --max-concurrency 8 --prefill-chunk-size 4096 \
  --mtp-draft-tokens 5 --speculative-token-capacity 48 --language-model-only

Serve the core and send two chat requests. A core exported at f7c468c, before the merge, serves normally.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions