Repository navigation
GLM-5.3 TP2 live serving: [gMASK] reasoning leak, budget starvation, and tool-call rejections trace to one sampled degenerate draw #133
Description
Report only. This issue reproduces and classifies three symptoms observed on a live GLM-5.3 TP2 deployment on Oct 28. It attempts no fix and scopes none here. Every count below is a measured frequency at the stated N, and inferences are labeled as such.
Environment and a framing fact
The live server is the Spark TP2 pair at 56f5702, package glm53-spark-tp2-packed-202610281245.safetensors, rank0 serving :8000 on spark-37b4, measured model id GLM-5.3-Flash. The symptoms predate the Oct-28 sync, and the sync left the binary contents unchanged.
Framing fact. The server accepts temperature: 0, but the package declares the sampling range [0.95, 1] for temperature, and the server clamps sampling parameters. Two byte-identical requests returned different reasoning content (measured). Every replay below is a sampling draw, so the k/N counts are frequencies rather than determinism.
Symptom 1: tool-call parse rejections
The earlier summary miscounted. /tmp/glm53-run2.log lines 137 to 148 hold exactly 12 rejections: GLM argument names must be nonempty and unique 8, tool parameter value does not match its declared JSON type 2, GLM argument requires a value tag 2. The earlier text said about 11 with one value-tag error. All baseline request ids carry the msg- prefix, and only /v1/messages issues that prefix (measured). The Oct-28 tool traffic ran on the Messages route.
Repro. The original payloads did not survive. The shapes were reconstructed from the chat template as two tools, get_weather with city, days (integer), unit, and a zero-argument get_time, plus four prompts. Each shape ran 12 replays on chat completions, and three shapes also ran on /v1/messages, at temperature 0 with streaming, 86 requests total.
| route and shape | nonempty-and-unique | declared-JSON-type | requires-value-tag | sibling arg-key error |
|---|---|---|---|---|
| chat, four shapes x12 | 0/48 | 0/48 | 0/48 | 0/48 |
| messages, single-city forecast x12 | 0/12 | 0/12 | 0/12 | 0/12 |
| messages, zero-arg get_time x12 | 0/12 | 0/12 | 0/12 | 0/12 |
| messages, two-city compare x12 | 1/12 | 0/12 | 0/12 | 1/12 |
The two rejections in the last row are the historical dominant string once and GLM tool call requires an argument key once, which is a sibling branch of the same parser in helios/src/output/argument_tags.rs. The other two baseline strings measured 0/84 replays. If those tails sit near two percent, 84 replays cannot see them (inferred).
Client-visible behavior, measured, and relevant to any scoping decision: a rejection arrives mid-stream as an SSE error event after normal deltas, and the stream then carries no finish_reason chunk and no [DONE]. Deltas already sent stay on the wire, including one complete valid tool_use block in one run that then died on the second call's parse.
Correlation to symptom 3: both rejected runs open their thinking with The[gMASK] (measured, 2 of 2).
Symptom 2: reasoning budget starvation
The observed case was one temperature-0 chat at max_tokens=256 finishing with content null and finish_reason=length while the same prompt succeeded at 512.
Battery: 54 non-streaming requests, three prompts, settings {256, 512} at default effort and 256 at reasoning_effort=low, six runs each. Starved means content null with finish=length and completion tokens equal to max_tokens.
| prompt | 256 default | 512 default | 256 low |
|---|---|---|---|
| arithmetic 23+6 | 0/6 | 3/6 | 0/6 |
| capital of France | 2/6 | 1/6 | 0/6 |
| two-step word problem | 1/6 | 0/6 | 0/6 |
- Starvation reproduces at both budgets, 3/18 at 256 and 4/18 at 512. The 256-fails-512-succeeds contrast is sampling luck, not a budget threshold (measured at 18 runs per setting).
- All 7 starved runs share one signature: reasoning opens
Thethen a literal[gMASK]within the first 12 characters, drifts off-topic, and never closes before the budget dies. - A starved run is not a truncated copy of a good run. Streams for the same prompt diverge at byte 3 to 17.
reasoning_effort=lowproduced 0/18 starved and collapsed reasoning to tens of characters. The recipe template maps low and high, and everything else falls to its max branch, and the default is that max branch (repo read).
Symptom 3: [gMASK] in reasoning_content
Captures: 20 streaming runs, prompts 23+6 and capital of France, at max_tokens 64 and 512, five each. The literal string [gMASK] appears in reasoning_content deltas in 4/20 runs and in content deltas in 0/20. It sits right after the opening The or mid-text. Neither earlier field claim survives. The leak is not every run, and it is not truncation-only. Complete runs carry it 3/10, truncated runs 1/10.
Token facts, measured from the tokenizer snapshot (76348ddb) on the serving host: [gMASK] is added token id 154822, special, and it lives outside the 154820-entry BPE vocabulary. flare inspect on the deployed package lists eos tokens 154820, 154827, 154829. Token 154822 is not one of them. The recipe's rejected_tokens (154830 to 154835, 154854, 154855) also excludes it, and rejected_tokens validates prompt input only, not sampling (helios/src/model/prompt.rs:41).
Mechanism, all measured from code: helios/src/server/output.rs:157 builds the streamed decoder with skip_special_tokens=false, and Decoder::push suppresses decoding only for the declared eos ids (output.rs:206). A sampled 154822 therefore decodes to the literal text [gMASK] and routes into the currently open channel, which is reasoning until </think> closes the think envelope. Nothing strips it afterward. The runtime decoded faithfully, and this report claims no runtime decode leak.
Unified classification
The three symptoms read as one defect class: a per-request degenerate draw in the default effort-max sampling regime. The opening signature was measured in 7/7 starved text runs, 4/20 leak runs, and 2/2 tool rejections. On a text request the draw starves the budget or leaks token text. On a tool request the same draw eventually emits malformed GLM argument tags, and argument_tags.rs rejects them per its strict no-coercion contract. The origin of the draw itself, checkpoint behavior versus sampling configuration, is inferred and not localized by this investigation.
Scoping questions are left open for maintainers, with no recommendation implied here: whether 154822 belongs in a sampling-time rejection, whether it belongs among the eos ids, and what a stream should do with already-delivered partial tool blocks when a later call fails to parse. One adjacent code behavior surfaced while tracing and was not reproduced as a defect: an open <tool_call> at natural EOS errors with incomplete GLM tool call, while the same open call at max_tokens is silently dropped with finish_reason=length (helios/src/output/tagged.rs:251-263).
Repro
One bash driver runs all four steps: log extraction, the 7-shape by 12-replay two-route tool matrix, the 54-run budget battery, the 20-run gMASK captures, and the frequency tallies. The request bodies and prompts are inline in the script, the driver is parameterized by ssh host, and all probes are sequential read-only inference requests against the live server. Raw captures of about 160 responses are held by the reporter and can be uploaded on request.
Checks not run
- The two unreproduced error strings at sample sizes that could see a two percent tail.
- The chat-route versus Messages-route rejection difference. Protocol rendering and sampling luck remain competing explanations.
- The vision route, other recipes,
reasoning_effort=high, and other export builds. The draw rate at the checkpoint at rest was not separated from this packed export. - The speculative accept and rollback path for delimiter token identity, which the code trace flagged as open.
Activity
eduffd commentedon Oct 28, 2026
Follow-up: root cause found (preparation-side graph corruption, fixed by #129), plus corrections to this report's own repro
I re-ran this report end to end, found a defect in its repro driver, reproduced all three symptoms with valid captures, and then traced the cause. The cause is not in sampling configuration or output presentation. It is upstream, in graph preparation, and PR #129 already fixes it.
1. The report's sections C and D captured errors, not model output
repro.shsections C and D write the request body to a local temp file and then run:The
cat >inside the ssh command never receives stdin, because the local redirection< /tmp/repro-bat.jsonis missing. Every one of the 74 captures from those sections is:{"error":{"message":"invalid JSON request: EOF while parsing a value at line 1 column 0"}}So the budget-battery and gMASK-capture tables in the report body were produced from invalid requests. The tool-call section B was unaffected (its bodies were shipped by a separate
catredirection) and remains valid. A corrected driver reran the same prompts, budgets, and counts. Measured, this session, against the same live TP2 deployment:"unterminated GLM argument key","GLM argument names must be nonempty and unique")content:null,finish_reason:length)[gMASK]in streamreasoning_content, 0 incontent)Two corrections to the report's framing follow directly. Rejections are not messages-route only: the chat route logged both strings above, including a fifth parser message (
unterminated GLM argument key) not in the report's table. And starvation is fully correlated with a degenerate stream open: of the 54 battery runs, 26 began with[gMASK]within the first 8 characters and 10 of those starved; 28 did not lead with it and 0 of those starved. Measured.2. This is not a sampling draw
The report classified all three symptoms as one sampled degenerate draw. Determinism refutes that. Repeating one warm-cache request 15 times produced 15 byte-identical completions, every one leading with
[gMASK]. A seeded temperature-0 request (which clamps to 0.95, see section 4) produced identical text on runs 1 and 3 but diverged on run 2, so per-request sampling noise is present, but the identical-repeat class is not noise. Cold-cache versus warm-cache leading-[gMASK]rates also differ (12/15 versus 15/15), which points at recomputed prefill state, not at the sampler. Measured. Inference: the first sampled tokens are the deterministic output of a graph that reads the wrong buffer.3. Root cause: the exported package was built with the storage-lowering defect that #129 fixes
PR #129 (
ad994b6e Keep complete copies ordered before storage calls that read their region) fixes the Inductor storage lowering introduced in317c1ecc(2026-10-26, "Preserve compiled storage mutations without cloning backing buffers"). #129's own description states the consequence: "the final mhc_post of prefill [reads] the router logits instead of the MoE output, and batched decode without MTP [reads] num_tokens from cumsum[0]", and that "packages exported from revisions that include 317c1ec but not this change can contain the misread views and should be exported again."This deployment's package (
glm53-spark-tp2-packed-202610281245, exported 2026-10-28 12:45) came from a checkout in exactly that window:317c1eccis an ancestor of every revision in the export range. Strong inference, not a direct inspection: the captured graphs in the served package contain misread views.I ran #129's own test file against the pre-fix preparation code on the host that exported this package (GB300 Spark, CUDA tests enabled):
ignition/preparation/compilation.py+ Fix complete-copy ordering, CuTe tensor pickling and GB300 package naming #129 tests:7 failed, 32 passed, 1 skipped. Failures includetest_opaque_calls_read_the_written_destination_of_a_complete_copy[offset_view|sub_view_read|token_count|triton_reader|write_then_read]. Thetoken_countassertion is the misread itself:Mismatched elements: 1/1, Greatest absolute difference: 15 at index (0,).compilation.pyhunk:1 failed, 38 passed, 1 skipped. The single remaining failure wastest_bf16_add_out_preserves_pooled_aliases_under_native_replay, which requires thequalificationCargo feature in the installed extension and fails identically with or without the fix. After rebuilding the extension with that feature on the same host, that test also passed:1 passed in 2.43s.So the defect class is confirmed on this host and this code state, and #129 closes it at the compile level. Recommended path for this issue: land #129, re-export the GLM-5.3 TP2 package during a maintenance window (both Sparks' GPUs are occupied by the live deployment, so this needs operator scheduling), then rerun the corrected battery and compare frequencies. Frequency batteries, not paired seeds, because per-request sampling stays stochastic at 0.95.
4. Answers to the scoping questions
Does 154822 belong in a sampling-time rejection? No, on current evidence.
frontend.rejected_tokenshas always been a prompt-input modality gate, not an output policy. It originates inc4673947as"prepared GLM catalog supports text inputs only"and moved into the contract inbcd6b249. No sampling-time consumer of it has ever existed, and the prepared contract models valid sampled IDs as a vocabulary range, not a per-token exclusion list. The per-request mechanism that does exclude tokens from sampling is theGRAMMAR_MASKgraph input, built for structured-output grammars. Masking 154822 would not fix this issue anyway: with corrupted logits the draw is garbage either way, and a mask just changes which garbage is sampled. Keep this issue scoped to the preparation fix.Does 154822 belong among the eos ids? No. EOS ids are checkpoint-derived (
generation_config.jsonthroughprepare_frontend), and[gMASK]must stay legal in prompts because the packaged chat template prefix is[gMASK]<sop><|system|>. Making it an eos would truncate the degenerate draws to empty answers while leaving the corrupted first tokens, and would change stop semantics for a token that is a structural part of every GLM prompt.What should a stream do with already-delivered partial tool blocks when a later call fails to parse? What it does today, which is documented:
docs/api.md("Truncated tool envelopes are omitted and the response reports its output limit; recognized but incomplete calls at natural completion are errors") and "streams receive a terminal error and may already have delivered the first call". The drop-at-limit versus error-at-EOS asymmetry was introduced deliberately in99cfcd5be, pinned by the testoutput_limits_allow_incomplete_tools_but_eos_rejects_themin03074c11, and restated in #67. Changing stream semantics after a parse failure is a maintainer policy decision that would editdocs/api.md, not a defect fix.Why is
temperature: 0accepted? Documented current behavior: request distribution controls clamp to package ranges (docs/api.md, "temperature: 0 becomes the declared minimum, or the default if no range exists"), and GLM declares[0.95, 1]from the model-card evaluation presets. An explicit zero-temperature rejection existed (99cfcd5be, "The request is never silently clamped or replaced") and was replaced by clamping in905735a5. One doc defect to flag:docs/api.mdstill says "Temperature zero remains rejected" in the/v1/messagessection, which contradicts the clamp section and the protocol testtest_zero_temperature_clamps_to_model_sampling_policy. Worth a docs follow-up.5. Unrelated item noticed in the same area
The finish-reason classification at the output limit does not consult
ignore_eos. That is already tracked indocs/TODO.txtand open PR #102. Not part of this issue.What was measured versus inferred here
Measured: the driver defect and the 74 invalid captures; rerun frequencies; the leading-
[gMASK]correlation; 15/15 byte-identical warm repeats; the 7-failed pre-fix versus 1-failed post-fix test split on the export host. Inferred (strongly): that the served package's captured graphs carry the misread views, from #129's own export warning plus the export-window ancestry. The battery rerun against a re-exported package is the closing check, and it needs a maintenance window.eduffd commentedon Oct 28, 2026
Close-out: #129 merged and deployed on this cluster; all three symptoms collapsed on the same battery
We merged PR #129 locally (merge commit
696cf668; the only conflict was inignition/preparation/compilation.py, resolved by keeping the PR's deferred-copy pre-scan together with main's contiguous offset-0 copy-source guard). Both preparation hosts were fast-forwarded,flareand theflare_heliopauseextension were rebuilt from the merged tree, and the GLM-5.3 TP2 package was re-exported from both Sparks: cache entrygb10-tp2-202610282200, packed asglm53-spark-tp2-packed-202610282200.safetensors. Serving relaunched on Spark1:8000 against the new package.Then the same battery as the follow-up comment (same prompts, budgets, and request counts). Measured:
content:null,finish_reason:length)[gMASK]in streamed text[gMASK]Caveats. Each arm is one battery draw. Starvation going from 10/54 to 0/54 is decisive at that pre-fix rate. For parse rejections, the pre-fix 1/84 is rare enough that 0/84 in this draw only bounds the residual rate near 3.5 percent at 95 percent confidence. The GLM tool parser itself is strict and unchanged: a malformed tool call from the model would still be rejected by design, and that documented behavior did not need to change.
Two supporting observations from the deploy. The re-export produced a cache entry named
gb10-tp2-202610282200, which exercises #129's PCI-device revision naming on the real export path. And the re-export did not replay compiled artifacts from the corrupt 12:45 export: within the export's run window, the same persistent kernel cache that the corrupt export used gained 1,382 new Inductor files and 17,873 new Triton files (measured by mtime), so the corrected storage lowering genuinely compiled into the new package rather than being served from a graph cache keyed only on the unchanged input graph. The preparation suite on the export host on the merged tree was41 passed, 1 skipped; the only failure against a plain-feature extension build remains the pre-existingqualificationfeature gate noted in our review on #129.The corrupt package stays on disk for comparison and is no longer served.
original-el8 commentedon Nov 5, 2026
The preparation fix in #129 is merged and is an ancestor of main
87cc8c1e. The close-out comment reports that the re-exported GLM TP2 package reduced tool parse rejections from 1/84 to 0/84, budget starvation from 10/54 to 0/54, and literal gMASK output from 7/20 to 0/20 in the corrected battery.This is a closure candidate for the prepared-graph corruption described here. Record that affected packages require re-export from preparation code containing the fix; upgrading the serving executable alone does not replace captured kernels. Malformed tool calls from otherwise valid model output remain a separate parser/API concern. This triage checked the merged history and reported validation; it did not rerun the model.
feat(#16): Fix(helios): strip leaked [gMASK] control tokens in GLM Ar…
eduffd commentedon Nov 8, 2026
Closing this, the fix is already merged