Repository navigation
DeepSeek 4.1 TP3 on GB10: 0.46 GiB/GPU less prefill scratch gave +7.6 % KV tokens (measurement only) #186
Description
On three GB10 hosts at TP3, DeepSeek V4.1 Flash starts with about 7.7M KV tokens (flare_total_kv_tokens). A local prototype that reduced the fixed prefill scratch of the DeepSeek 4.1 recipe by 0.46 GiB per GPU raised that to about 8.3M (+7.6 % on the mean of three starts), with bit-identical first-token log-probabilities. I'm reporting the measurement so you can judge whether the headroom is worth pursuing. I'm not proposing an implementation: the prototype was written with Claude Opus 5.5 and falls under the model-optimization restriction in AGENTS.md and principle 14 of the design review, so I won't submit it or describe it.
Setup. main at f571a550. Both packages were exported with --max-concurrency 4 --mtp-draft-tokens 5 and the default 4096-token prefill chunk, with the preparation host-memory floor lowered from 8 GiB to 6 GiB by a local override (main has no option for it). Two export attempts closer to the recipe defaults (max concurrency 32, 7 draft tokens) failed on these hosts: one at the host-memory check before compilation (12,269 MiB available, 12,288 required), one at a RoCE collective timeout while warming compiled forwards. Both packages were served with flare run from the same main build. Each package was started three times: the first start of each was a cold start without a fixed idle period; the second and third alternated between the packages, each after 3 minutes idle. The prototype's first start attempt failed at "Distribute rank packages over CX7" (failed to fill whole buffer) and started normally on retry.
Decode and prefill in tok/s, aggregate over the streams, mean and range of 3 starts:
main |
Prototype | Difference | |
|---|---|---|---|
Fixed device memory per GPU (flare inspect --memory-map) |
96.71 GiB | 96.25 GiB | -0.46 GiB |
flare_total_kv_tokens (3 starts) |
7,915,264 / 7,456,512 / 7,762,304 | 8,505,088 / 8,213,760 / 8,172,672 | +7.6 % on the mean |
| Decode prose x1 | 45.5 (44.5–46.3) | 44.2 (43.7–44.8) | -2.9 % |
| Decode prose x4 | 89.7 (88.3–91.7) | 92.3 (90.3–94.6) | +2.9 % |
| Decode code x1 | 82.6 (79.4–88.2) | 83.1 (80.3–87.0) | +0.7 % |
| Decode code x4 | 150.4 (142.4–154.9) | 149.2 (144.0–157.5) | -0.8 % |
| Prefill, 30k-token prompt | 2,764 (2,540–2,891) | 2,776 (2,557–2,900) | +0.4 % |
| Prefill, 120k-token prompt | 3,390 (3,281–3,448) | 3,298 (3,205–3,421) | -2.7 % |
Every decode and prefill difference is within the range of single starts, but prose x1 and 120k prefill were lower with the prototype in two of the three alternating pairs; three starts cannot separate that from noise. The KV total is sized at start from the memory then free and varied by up to 0.46M tokens between starts of the same package; the lowest prototype start (8.17M) is still above the highest main start (7.92M). At 890 bytes per token per GPU, 0.46 GiB corresponds to about 0.55M tokens; the measured mean difference is 0.59M.
The top-20 log-probabilities of the first generated token were bit-identical between the first start of each package for ten prompts from 836 to 94,341 tokens, and a 118k-token retrieval test passed 16/16 in all six starts.
The saving scales with the prefill chunk: a layout calculation for the prototype gives about 110 MiB per 1024 tokens of chunk, about 440 MiB at 4096, close to the 468 MiB measured. I have not measured other chunk sizes on this base or other recipes.
To reproduce the measurement on main.
- Export at TP3, with
ignition join <rank0>:<port> --rank {1,2} --ranks 3 --gpu 0on the peers:For both packages we loweredignition export deepseek-v4.1-flash --ranks 3 --rendezvous <rank0>:<port> --gpus 0 \ --max-concurrency 4 --mtp-draft-tokens 5ignition.preparation.host_memory._HEADROOM_BYTESto 6 GiB; we did not try this recipe with the default 8 GiB. - Fixed device memory per rank:
flare inspect --unverified --memory-map --json <package>/model.safetensors, summingranks[*].allocations[*].nbytes. - Serve with
flare run <package> --rank 0 --rendezvous <rank0>:<port> --gpus 0 --scheduler-policy throughput --prefix-cache-memory 0 --prefix-cache-disk <dir> --prefix-cache-disk-limit 200G(peers:flare run --rank {1,2} ...), aftersync; echo 3 > /proc/sys/vm/drop_cacheson every host, and readflare_total_kv_tokensfrom/metricsonce/v1/modelsanswers. On GB10 the KV pool is sized fromMemAvailableat start, so compare several alternating starts rather than one. - Decode: 512 generated tokens per stream, prose and code prompts, 1 and 4 concurrent streams. Prefill: time to first token for 30k- and 120k-token prompts.
If you look into it, I can export and measure any branch on the same three hosts with the same protocol.
Activity
original-el8 commentedon Nov 5, 2026
Categorized as
performance,area:memory,area:preparation, andtriage:design-review. The demonstrated result is increased KV capacity from reduced fixed scratch, not a demonstrated throughput improvement. The next decision is whether that capacity saving warrants an implementation under the repository's design requirements; any implementation needs its own lifetime analysis, numerical checks, and fresh matched measurements. This is not a confirmed correctness bug or a closure-verification item.feat(#23): Feat(ignition): allow overriding the 8 GiB host-memory flo…
javieratomic commentedon Nov 12, 2026
Update for
mainat0572fd12: the scratch this issue measured is gone. 4a24d78 (merged through 2b38bf9 and e5e7881) replaced the DeepSeek V4.1 recipe's manual scratch pools with compiler-planned temporaries. A core exported with recipe defaults from0572fd12, which includes that change among other storage changes sincebbad44fe, holds 4.7 GiB less fixed device memory per GPU than one exported frombbad44feon the same three GB10 hosts (TP3, one rank per host). The difference is between the two commits, 39 first-parent commits apart, not the effect of 4a24d78 alone; the message of e5e7881 claims no fixed-memory reduction for the merged revision.Memory inventory printed at export, rank 0, in MiB (ranks 1 and 2 are within 4 MiB on both commits):
bbad44fe+ #1220572fd12+ #122At load, with the same
flare runoptions (--scheduler-policy throughput --kv-cache-limit memory --prefix-cache-memory 0 --prefix-cache-disk <dir> --prefix-cache-disk-limit 50G, default draft tokens), rank 0 reports the following in "Plan cache" and "Allocate cache". Thebbad44fecores were started by thebbad44feruntime with #251 (e774a46) applied, without which they did not start, and the0572fd12core by the0572fd12runtime; the two sides were started on different days.bbad44fe+ #122, 6 starts of 2 cores0572fd12+ #122, 3 startsWith
--mtp-draft-tokens 0the0572fd12core loaded 98.09 to 98.14 GiB and allocated 9,992,192 to 10,107,008 KV tokens in two starts, against 102.90 to 102.93 GiB and 4,564,352 tokens in two starts of thebbad44fecores.Limits of these numbers:
0572fd12core was exported fromf25a0386(Make the compiler memory headroom a preparation setting #122 on0572fd12) with--compile-headroom 4G, because the default export stops in kernel tuning on these hosts (GB10 TP3: DeepSeek V4.1 Flash export at 0572fd12 fails kernel tuning with a 0-byte scratch limit; 7.1-7.9 GiB available, below the 8 GiB floor #270).flare_total_kv_tokens, and nothing was served: no decode, prefill or output comparison.bbad44fecores were exported frombf5ca911(Make the compiler memory headroom a preparation setting #122 onbbad44fe) with--compile-headroom 6G. Both sides are recipe defaults (max concurrency 32), not the--max-concurrency 4 --mtp-draft-tokens 5exports off571a550in the issue's table, so neither these counts nor the 4.7 GiB compare directly with the issue's 7.7M tokens (flare_total_kv_tokens) and 0.46 GiB (flare inspect --memory-map).bbad44feand0572fd12also change the runtime (617ef92 changeshelios/src/cache.rsandhelios/src/scheduler/cache_memory.rs), so the token counts need not be the effect of the fixed-memory difference alone.The prototype overlapped prefill buffers inside the pools that
0572fd12no longer has, so I have nothing left to propose here; from my side this can be closed.original-el8 commentedon Nov 13, 2026
Marked
triage:verify-for-closure. The author's update confirms that compiler-planned temporaries replaced the manual scratch pools this report targeted. The measured fixed-memory reduction spans other changes too, so it should not be attributed solely to the scratch refactor.The original scratch proposal can be closed if no distinct remaining allocation problem is identified. The later default-export memory failure (#270) and transport-startup failure (#271) have their own tickets; their existence does not preserve the need for the removed scratch pools.