DeepSeek V4.1 Flash: positions before token 128 return context-free logits when the reasoning-effort prefix is present

Summary

With the default DeepSeek V4.1 chat render, some positions in the first 128 tokens produce a
next-token distribution that ignores the context. The default render is thinking mode, which
injects Reasoning Effort: 75 (range 1-100, …) after <｜System｜>, as the official
encoding.py does.

Every affected position returns the same flat distribution: '\n' −2.25, '#' −2.37, '<'
−2.74, '##' −3.09. The positions before and after it are sharp. At the packaged temperature 1
and top-p 1, the sampler then picks an arbitrary token, and the reply continues around it:

"Hi! 👋 What's on your五 mind today?"
"Hi! 👋 How can\n can I help you today?"
"Hi! 👋 What's on yourIt looks like my response got cut off — let me try that again. …"
"Hi! 👋 How can<ds_safety>[用户未成年]否 … </ds_sHome>can I"

The last example is the same defect: '<' is the third alternative, and after it the model
writes its safety-annotation format. [{"role":"user","content":"Say hi."}] with default
settings was corrupted in 6 of 6 Chat Completions requests. The packaged seed is fixed, so an
affected prompt fails the same way on every identical request.

Environment

flare-inference 0.1.0~git20261027170349.7c22369f-1 (7c22369f) on every rank
local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE@gb10 (commit 607c1abf), deployment
0a377833bb1c431ab417837a50bb18a8, TP4
2× DGX Spark and 2× ASUS GX10 (GB10), RoCE through a switch, launched with flare orbit run
The README's TP4 command plus --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB --memory-limit=-2G
No other requests in flight; flare_free_kv_pages ≈ 116k and flare_cached_kv_pages ≈ 13k throughout

Reproduction

Prefill only: /v1/completions with max_tokens: 1 and logprobs: 5 on the rendered chat
prompt plus the model's own first nine output tokens:

<｜begin▁of▁sentence｜><｜System｜>Reasoning Effort: 75 (range 1-100, the higher the value, the more thorough the reasoning)\n\n<｜User｜>Say hi.<｜Assistant｜><think></think>Hi! 👋 What's on your

BLANK prompt_tokens= 42  default chat render                '\n' -2.25 | '#' -2.37 | '<' -2.74 | '##' -3.09
BLANK prompt_tokens= 42  same, Reasoning Effort 50          '\n' -2.47 | '<' -2.57 | '#' -2.91 | '##' -3.23
ok    prompt_tokens= 16  same, no effort prefix             ' mind' 0.00 | 'mind' -17.56 | ' agenda' -20.26 | ' Mind' -21.36
ok    prompt_tokens=142  same, ~100 tokens after the prefix ' mind' 0.00 | 'mind' -15.10 | ' river' -15.65 | …

Five repeats of the first request return bit-identical top-5 logprobs.

Repro script

"""Prefill-only probe: next-token distribution after each prompt via /v1/completions.
Usage: repro.py http://HOST:PORT/v1 MODEL"""
import json, sys, urllib.request
url, model = sys.argv[1].rstrip("/"), sys.argv[2]
B, S, U, A = "<｜begin▁of▁sentence｜>", "<｜System｜>", "<｜User｜>", "<｜Assistant｜>"
H = lambda n: f"Reasoning Effort: {n} (range 1-100, the higher the value, the more thorough the reasoning)\n\n"
filler = " ".join(["the river runs past old stone walls"] * 14)
tail = f"{U}Say hi.{A}<think></think>Hi! 👋 What's on your"
cases = [("default chat render", f"{B}{S}{H(75)}{tail}"),
         ("same, Reasoning Effort 50", f"{B}{S}{H(50)}{tail}"),
         ("same, no effort prefix", f"{B}{tail}"),
         ("same, ~100 tokens after the prefix", f"{B}{S}{H(75)}Notes: {filler}{tail}")]
for name, prompt in cases:
    body = {"model": model, "prompt": prompt, "max_tokens": 1, "logprobs": 5}
    d = json.loads(urllib.request.urlopen(urllib.request.Request(url + "/completions",
        json.dumps(body).encode(), {"Content-Type": "application/json"}), timeout=300).read())
    top = sorted(d["choices"][0]["logprobs"]["top_logprobs"][0].items(), key=lambda kv: -kv[1])
    blank = top[0][0] in ("\n", "#") and top[0][1] < -1.5
    print(f"{'BLANK' if blank else 'ok   '} prompt_tokens={d['usage']['prompt_tokens']:3}  {name:34} "
          + " | ".join(f"{t!r} {l:.2f}" for t, l in top[:4]))

Evidence

Each probe asks for one token after a prefix extended one word at a time, so every
measurement is the last position of a prefill. Positions are prompt lengths.

Context	Probes	Blank positions
Effort prefix, "Say hi." reply, 0–200 filler words after the prefix	10 sweeps	42 46 · 48 · 56 · 68 · 82 84 · 104 · 122 124; none once the reply starts at ≥ 136
Effort prefix, sleep-tip reply, 0–400 filler words	17 sweeps, 561	72 · 84; none at 127–478
Effort prefix, `<think></think>` vs `</think>`	2 sweeps, 124	42 46 vs 41 45 (same tokens, shifted by `<think>`)
No system block, non-thinking	3 sweeps, 176	none
20-token plain system prompt instead of the effort prefix, both modes	2 sweeps, 117	none

In total there were 16 blank positions out of about 520 below 128, and none out of about 440
at 128 or above.

Which positions fail depends on content, not offset. Changing the effort score from 75 to 50
moves the failures but doesn't remove them.
Decode shows the same signature. Streamed Chat Completions with logprobs show the blank
distribution at the spliced token, and the next position is sharp again (' your' −0.25,
' mind' 0.00).
After <｜begin▁of▁sentence｜> alone, the distribution is '#' −1.58, '\n' −2.34, '##'
−2.94, '<' −3.44: the same family, but not identical.

This rules out:

Speculative verification: the failure reproduces at the prefill's last position with
max_tokens: 1.
Prefix-cache reuse: a random tag after BOS (cached_tokens: 1) fails the same way.
The Helios frontend and output parsers: raw /v1/completions reproduces it.
Stale page bytes (Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure) #116): the output is deterministic and the pool is mostly free.

Not tested

A reference forward on the same prompts: DeepSeek's inference/ or
ignition debug-serve --direct. Without one we can't exclude the checkpoint itself.
TP1/TP2, other GPUs, or Engram tables in memory (they do not fit in host memory here).
Packages exported after 7c22369f.

Possible direction (needs maintainer input)

The boundary is 128 tokens, which is both sliding_window and PAGE_TOKENS
(ignition/recipes/deepseek41/pool.py). Below 128, the queried position sits in a partial
first window and a partial first page, which suggests these paths:

PreparedWindowIndices.__call__ (ignition/components/window_indices.py) with
left = (position - 127).clamp_min(0) and window lengths below 128.
The compressed geometry in AttentionIndices.__call__
(ignition/recipes/deepseek41/attention.py), where lengths = (position + 1) // ratio is small.
How PreparedCompressedMLA combines the partial window, the compressed rows and sink.

One ignition debug-serve --direct request with the 42-token prompt above would show whether
the prepared graphs or the checkpoint produce the blank distribution. We can run more probes
on this deployment if that helps.

Follow-up: the trigger is absolute position below 128, not how close the effort prefix is to the query.

I ran the same prefill-only sweep (one token per prefix, extended one word at a time) on the
"Say hi." reply from the report, with about 100 filler tokens moved to either side of the
effort prefix:

Render	Reply positions	Blank
`<｜System｜>` + effort prefix + `Notes: <filler>` + user turn	134–169	none
`<｜System｜>Notes: <filler>\n\n` + effort prefix + user turn (prefix right before the reply)	135–170	none
No filler (the report's case)	34–69	42, 46

Moving the prefix up to the reply doesn't bring the failure back once the reply is past
position 128. The defect needs both the effort prefix somewhere in the context and a query
position inside the first 128 tokens, which is the first sliding_window / first KV page.
This makes a checkpoint explanation less likely and points at the first-window or first-page
handling: PreparedWindowIndices with left = 0 and window lengths below 128, compressed rows
in page 0, and how those are combined with sink in PreparedCompressedMLA.

The failing position reproduces identically through every path we can reach without
re-exporting:

a cold prefill (cached_tokens: 1);
a prefill that resumes from a 33-token cached prefix;
MTP verification during streamed decode.

Since all three give the same result, the fault is probably in a component they share rather
than in one graph shape.

Switch the default backend to FLARE and record its benchmarks

Still reproduces after the gb10 package update: flare-inference 0.1.0~git20261030013211.a161b17f-1 on every rank and agent, with package ab4f07af (deployment 7cf40bec5c5b4026a4abc801e615226c, 8192-token prefill chunks), launched with --mtp-adaptive-verification-alpha 0.5. This is the same TP4 cluster as in the report.

The repro script from the issue gives the same result as on 7c22369f / 607c1abf:

BLANK prompt_tokens= 42  default chat render, 42 tokens     '\n' -2.25 | '#' -2.37 | '<' -2.75 | '##' -3.08
BLANK prompt_tokens= 42  same, Reasoning Effort 50          '\n' -2.29 | '#' -2.48 | '<' -2.81 | '##' -3.10
ok    prompt_tokens= 16  same, no effort header             ' mind' 0.00 | 'mind' -17.09 | ' agenda' -19.54 | ' Mind' -21.04
ok    prompt_tokens=142  same, ~100 tokens of notes first   ' mind' 0.00 | 'mind' -15.43 | ' river' -15.90 | '<｜end▁of▁sentence｜>' -18.46

Neither the larger prefill chunk nor the changes in a794c99..a161b17f (#108 window snapshots, #135 distributed sampling, deepseek recipe 2.0) move the failing positions.

This remains a high-priority correctness investigation because the report shows abnormal next-token distributions without an explicit execution failure. #193 describes a related short-reply symptom on SM120, but a shared cause has not been demonstrated.

The next comparison should use a package prepared from a recorded revision containing the compilation fix in #129 and the pinned attention changes, then run the supplied effort-prefix sweep below and above token 128 through cold prefill, restored prefill and verification. Record rendered token IDs, the actual prompt positions, package/deployment identity and runtime revision. This establishes whether the original failure persists after integrated fixes; neither #129 nor #193 is currently evidence that this specific defect is resolved.

Confirming on SM120, with a different build, package and set of flags.

Setup: 4x RTX PRO 6000 Blackwell (SM120), TP4. Flare is built locally from main of 2026-10-29 plus local patches. The DeepSeek-V4.1-Flash-1017 package was prepared locally (C32, 7 MTP drafts). Launched with --embedding-table-storage ram, --scheduler-policy throughput and the prefix cache on. Stock NVIDIA open kernel module 610.57.04 (no P2P patch).

Request: [{"role":"user","content":"Say hi."}] with the default chat render (thinking mode, so the Reasoning Effort header is present), temperature: 0 (clamped to the 0.01 floor), max_tokens: 64, logprobs with top_logprobs: 5. Requests were sequential.

Result:

Corrupted in 25 of 25 runs on a first pass and 10 of 10 on a second pass.
Replies: Hi! 👋 How can\n can I help you today?, Hi! 👋 How can\n, Hi! 👋 How can\n's it going?, Hi! 👋 What can I\nI help you with today?.
The reply skips reasoning: </think> is the first completion token.
The corrupted token is the 8th completion token, at absolute position 40 (prompt_tokens 33).
Its distribution is the flat one from the report: '\n' -2.27 | '#' -2.52 | '<' -2.83 | '##' -3.08 | '\n\n' -3.77.
The positions around it are sharp. Before it: ' How' -0.16, ' can' -0.08. After it: ' can' -0.65, ' I' -0.00.

Controls on the same load:

The same request with a 110-token system message in front (prompt_tokens 124): 0 of 10 corrupted, consistent with the position-below-128 finding above.
Say hello to me. (25 runs), Say good morning. (10) and Greet me in one short sentence. (10): no flat position in any reply.

So this also reproduces on SM120 with the embedding table in RAM rather than on disk, and with the stock driver. The short-reply pattern in #193 (How can- I, How can// can, the <ds_safety> block after How can) appears here at the same place in the reply, with the same flat distribution. That may help connect the two.

An independent reference forward reproduces the flat distribution, so on the evidence this is the checkpoint, not Flare.

Method:

The official inference/ code at the checkpoint revision 1bae0e10, with model.py's block structure.
kernel.py's arithmetic reimplemented in plain PyTorch and run on the CPU, with each replacement checked against a
tiny case. That includes:
- the in-place FP8 quantize/dequantize of the window KV;
- the FP4 compressed KV;
- sparse_attn with the sink;
- the FP8/FP4 GEMMs;
- hc_split_sinkhorn.
The checkpoint's weights, run one layer at a time, with no Flare code in the path.
The inputs were the exact token IDs from the production deployment for three prompts:
- the 42-token prompt from this issue;
- the 16-token no-header control;
- the 142-token filler control.

The issue's position (the last row of the 42-token prompt):

reference           '\n' -2.245 | '#' -2.368 | '<' -2.751 | '##' -3.076 | '\n\n' -3.694
Flare (SM120, TP4)  '\n' -2.264 | '#' -2.389 | '<' -2.701 | '##' -3.076 | '\n\n' -3.701

At the other positions the reference tracks Flare:

Where Flare's distribution is sharp, the reference's is too: ' your' 0.00 one position earlier, and ' mind' 0.00
at the last position of both the no-header control and the filler control (absolute position 141).
Across the 42-token prompt it matches Flare's top-1 at 39 of 41 positions. The two misses are positions where both
are uncertain (top token below -2.0, with a near tie).

The sliding-window key set is the same in both implementations: keys max(0, p-127) .. p, from get_window_topk_idxs
in the official code and PreparedWindowIndices in Flare.

On my SM120 deployment the flat distribution is:

bit-identical across repeats;
unchanged by three different long requests sent in between;
unchanged whether 33 or 42 prompt tokens came from the prefix cache;
within about 0.03 of the GB10 numbers in this issue.

Taken together, the official math with these weights produces this distribution at this position, and Flare reproduces
it. The full position sweep across the 128-token window boundary was not repeated in the reference; at every position
compared, including the flat one, the two agree.

The independent CPU-reference comparison changes the diagnosis: it reportedly reproduces the flat distribution with the checkpoint weights and no Flare execution, while tracking Flare at the tested control positions. This points toward checkpoint behavior for the compared prompts rather than establishing a Flare cache or kernel defect.

Reduced priority from P1 to P2 while retaining triage:investigate. Please attach the reference implementation, exact token IDs, and comparison outputs so the result can be independently checked, including the untested position sweep. The CPU arithmetic is a reimplementation of the reference kernels, so the reported agreement is evidence to verify, not a blanket correctness guarantee. No Flare fix or closure is claimed. Link the matching short-prompt cases in #193 to this comparison.

Reference code, token IDs and outputs: https://gist.github.com/brandonmmusic-max/8c1a89c738fd4ca9fa7a130bfbf645ee

model.py follows the official inference/model.py at 1bae0e10, with the tilelang kernels replaced by the plain tensor math in quant.py. ref_selftest.py checks each replacement against an independent tiny case.
The weights come from a lossless container of deepseek-ai/DeepSeek-V4.1-Flash-1017 at 1bae0e10; rebuilding the 48 original shards from it matches every shard's SHA256.
prompts.json has the exact token IDs and Flare's per-position top-5 from my SM120 TP4 deployment. ref-*.json has the reference at every position, and compare.txt is the side-by-side output of compare.py.

The reference covers every position of the 142-token filler control, across the 128-token boundary. Flare's side of that prompt was sampled only at positions 129 to 141; its three top-1 differences (129 to 131) are positions where neither side is confident (top-1 logprob between -1.1 and -2.3). I'll add Flare's remaining positions so the whole sweep is compared.

#193's failing prefix, position 44 of "Say hello to me." plus "Hello! 👋 It's great to", has the same flat spread over \n, #, < and ##, each within 0.1 of the numbers here. I'll run that exact prefix through the reference and post the result in #193.

Both follow-ups from my last comment are done; the scripts, token IDs and outputs are added to the same gist: https://gist.github.com/brandonmmusic-max/8c1a89c738fd4ca9fa7a130bfbf645ee

Full sweep of the 142-token filler control. Flare's top-5 now covers every position, from one-token completions on my SM120 TP4 deployment.

The top token agrees at 126 of 142 positions.
The flat markup distribution (top token below -1.8 with at least three of \n, #, < and ## in the top five, the criterion from DeepSeek V4.1 Flash gb202 TP4: next-token logits intermittently corrupt mid-reply (stray "##", "-", "<ds_safety>" classifier text in answers) #193) appears only below position 128, on both sides: at 45, 47 and 61 in both, at 75 in the reference only, and at 29 and 103 in Flare only.
The largest disagreements (54, 75 and 103) flip between a confident continuation and a diffuse distribution, in both directions. They look like the borderline positions voipmonitor describes in DeepSeek V4.1 Flash gb202 TP4: next-token logits intermittently corrupt mid-reply (stray "##", "-", "<ds_safety>" classifier text in answers) #193, where small numerical differences decide the outcome.

#193's failing prefix, "Say hello to me." plus "Hello! 👋 It's great to", built from this issue's exact tokens (44 tokens). After "great to":

reference           '\n' -2.292 | '#' -2.417 | '<' -2.652 | '##' -3.084 | '\n\n' -3.721
Flare (SM120, TP4)  '\n' -2.254 | '#' -2.442 | '<' -2.879 | '##' -3.067 | '\n\n' -3.754

That is the only flat position in the prompt on either side, and the top token agrees at 38 of 44 positions, with every difference at a position where neither side's top token is above -0.5.

Where the attention goes at the flat positions. attn_probe_cpu.py in the reference gist (https://gist.github.com/brandonmmusic-max/8c1a89c738fd4ca9fa7a130bfbf645ee) runs the CPU reference with each head's exact softmax mass recorded in every layer, including the per-head sink logit, split by prompt segment. Positions are 0-based indexes of the last prompt token; "start" is the BOS token plus the effort header, and "own text" is the reply, or the system text for the filler control. Values are averaged over 40 layers and 64 heads:

Prompt	Flat / clean position	Start	Own text	Output norm
#193 prefix (44 tokens)	43 / 42	0.54 / 0.46	0.31 / 0.42	4.66 / 5.48
This issue's prompt (42 tokens)	41 / 40	0.55 / 0.51	0.32 / 0.41	4.68 / 5.29
Filler control	45 / 44	0.59 / 0.41	0.39 / 0.57	4.49 / 5.49
Filler control	47 / 46	0.53 / 0.37	0.45 / 0.61	4.46 / 5.31
Filler control	61 / 60	0.42 / 0.29	0.56 / 0.70	4.39 / 5.42

The sink logit takes 0.010 to 0.019 at every position, flat or clean, so it is not involved.
At each flat position, attention moves toward the start and away from the text being written, and the attention output weakens by 11 to 19 percent. Counting the window keys alone, the start's share is 0.15 to 0.25 at the flat positions and 0.06 to 0.10 at the clean ones.
At the DeepSeek V4.1 Flash gb202 TP4: next-token logits intermittently corrupt mid-reply (stray "##", "-", "<ds_safety>" classifier text in answers) #193 position, a two-sentence system message makes the next token meet (-0.20). The start's share drops to 0.29, and the system text takes 0.26. Without the effort header, the next token is meet (-0.04), and the start takes 0.14.

Odd and even positions. Every flat position in the reference outputs in the gist is at an odd index. In the ratio-2 layers, a query at an odd index also sees the compressed pair that its own token completes. Two measurements rule that out as the cause:

On Flare (SM120, TP4), lengthening the user message with 0 to 90 copies of " please", in both "Say hi." and "Say hello to me.", gives flat positions at 19 of 92 odd and 24 of 90 even indexes.
In the reference, hiding that compressed pair from every query leaves positions 41, 43, 45 and 47 flat. Only borderline positions change: 61 and 75 turn sharp, and 29 and 68 turn flat.

The odd indexes are a sampling artifact. This issue's prompt and the #193 prefix have 42 and 44 tokens, the reference runs only even-length prompts, and the filler control's flat positions follow its repeating seven-token phrase.

System message. The same sweep, 182 prompts per row, counting the position after the fixed reply prefix:

System message after the effort header	Flat
none	43 of 182
"You are a helpful assistant."	8 of 182
"You are a helpful assistant. Answer the user's questions clearly and accurately, keep replies friendly and concise, and ask a short clarifying question when a request is ambiguous."	0 of 182

Lengthening the user message does not prevent it, and a substantive system message does, with no change to the weights. The mitigation is qualified on Flare with this checkpoint for these prompts; it is not implemented in Flare's chat rendering.

The scripts and outputs behind these numbers are in the same gist.

Navigation Menu

Repository navigation

DeepSeek V4.1 Flash: positions before token 128 return context-free logits when the reasoning-effort prefix is present #118

Summary

Environment

Reproduction

Evidence

Not tested

Possible direction (needs maintainer input)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DeepSeek V4.1 Flash: positions before token 128 return context-free logits when the reasoning-effort prefix is present #118

Description

Summary

Environment

Reproduction

Evidence

Not tested

Possible direction (needs maintainer input)

Activity

nero- commented on Oct 28, 2026

nero- commented on Oct 31, 2026

original-el8 commented on Nov 5, 2026

brandonmmusic-max commented on Nov 5, 2026

brandonmmusic-max commented on Nov 5, 2026

original-el8 commented on Nov 6, 2026

brandonmmusic-max commented on Nov 6, 2026

brandonmmusic-max commented on Nov 6, 2026

brandonmmusic-max commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions