Repository navigation
128 shared snapshot slots cap multi-user prefix reuse on a 4-Spark DeepSeek TP4 ring #47
Description
Summary
On a four-Spark TP4 ring, DeepSeek V4.1 Flash gets a 25M-token KV pool but only 128 shared snapshot slots, and
the slot count didn't change when we shrank the pool. In a multi-user chat workload, reuse is bounded by the slots,
not by memory. Once other users' prompts publish enough snapshots, a returning user's prefix is fully re-prefilled,
even with most of the pages free. We'd like to know whether the slot count can be raised, or tied to the pool,
on large-pool multi-host packages.
Setup
- 4× DGX Spark (GB10, SM121),
flare-inference 0.1.0~git20261021004534.bb204798,flare orbit run --roce-ring - Package
local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revisiongb10 --max-concurrency 16 --mtp-draft-tokens 5 --mtp-adaptive-verification --embedding-table-storage disk --embedding-table-resident-scales --embedding-table-cache-size 1GiB- GPU clocks capped at 2100 MHz
Startup, default KV limit:
[rank 0 spark01] [ OK ] Allocate cache. 21.2 GiB · 25258240 KV tokens · 128 shared snapshot slots · limited by memory 0.9s
With --kv-cache-limit concurrency, the pool shrinks but the slots stay the same:
[rank 0 spark01] [ OK ] Allocate cache. 14.2 GiB · 16777216 KV tokens · 128 shared snapshot slots · limited by concurrency 0.6s
(For comparison, a single-GPU Qwen3.8-Flash-Next package reported 1925120 KV tokens · 64 shared snapshot slots.)
What we measured
Reuse happens at speaker changes (system→user→assistant), which matches the docs. Each speaker change appears to
use one snapshot.
Retention probe. On a quiet server we primed a ~9K-token conversation of real prose, ran some other traffic,
then re-asked with the conversation extended by one turn:
| Traffic between prime and re-ask | Prefix reused |
|---|---|
| nothing / 60 s idle / 300 s idle | 100% |
| 1 or 3 other small conversations | 100% |
| one 100K cold read, or four 50K cold reads at once | 100% |
| 1 or 2 other prompts with 60 speaker changes each | 100% |
| 4 other prompts × 60 speaker changes, or 1 × 240, or 2 × 240 | 0% |
Human-paced author simulation. Each simulated author writes, asks book questions, fetches chapters and edits,
with human think-time between turns; 4 or 8 more authors join every 15 minutes. We used a compact layout to keep
speaker changes few: the book in one system message, and older history folded into one pair. Same simulator, seed
and schedule on both engines. "Reply start" is the median TTFT of ordinary turns.
| Authors | Reply start, Flare | Prefix reused, Flare | Decode p50, Flare |
|---|---|---|---|
| 4 | 2.0 s | 74% | 45 tok/s |
| 8 | 1.7 s | 73% | 43 tok/s |
| 12 | 6.2 s | 58% | 27 tok/s |
| 16 | 164 s | 18% | 6 tok/s |
In an earlier run with a chattier layout (resumed sessions of 100+ short messages), every turn at just 4 authors
was a full cold read (140–170 s to first token), while about 94% of KV pages were free.
On the same four Sparks and the same simulator, an SGLang TP4 build of the same model held 88% reuse and a 0.56 s
reply start at 16 authors. It overloaded at 24, against 16 for Flare.
Related observation: fully cached long prompts with many messages
A ~208K-token prompt with ~200 messages, reported fully cached (208,030 of 208,047 tokens), still took about 30 s to
first token per question. A ~192K prompt with ~80 messages took 7.2 s. With few messages, warm TTFT is 0.29 s at
10K and 5.8 s at 169K. Cold prefill also slows with message count, at ~14K tokens:
| Messages | Cold TTFT | Warm TTFT |
|---|---|---|
| 4 | 6.4 s | 0.4 s |
| 30 | 10.4 s | 0.6 s |
| 120 | 25.7 s | 1.0 s |
| 240 | 44.0 s | 1.7 s |
That suggests a per-snapshot cost on publish and restore.
Questions
- Is 128 a fixed cap, or derived from something we can tune on a multi-host package? It didn't change with the KV
limit. - Could the slot count scale with the KV pool on large-pool packages? With 25M tokens, memory isn't the constraint
for this workload. - Is per-snapshot publish/restore cost expected to grow with message count as above?
Multi-host disk prefix caching just landed in #29 (closing #23) and should soften this for returning users; we haven't tested it yet. Raising the GPU slot count would still help. We're happy to rerun the probes against
a branch.
Activity
calvarado2004 commentedon Oct 25, 2026
We hit the same limit on a four-Spark TP4 ring (DeepSeek V4.1 Flash 1017,
--max-concurrency 4, main atbce032c0), from a single coding-agent session rather than many users, and have a fix we would like to propose.What we see
The startup line reports
22952064 KV tokens · 32 shared snapshot slots. A Qwen Code session at ~380K tokens shares those 32 slots with its own subagents and side queries (9–35K-token prompts, several turns each). A few minutes of subagent work retires the main conversation's endpoint, and the next turn prefills the whole 381K context again (cached_tokens: 0, about 4–6 minutes to first token). At that moment/metricsshowedflare_free_snapshots 0,flare_cached_checkpoints 32, andflare_free_kv_tokens 22226688. Idle time alone does not cause it: one 75-minute pause still reused 305K tokens.Cause
On question 1: the count is derived in
capacities()inhelios/src/runtime_package/memory.rsascached = active * 8, then bounded by the row virtual extent and the aliased bytes. So it scales with--max-concurrency(16 × 8 = 128 in your setup) and not with the KV pool.The second half is the eviction order.
PrefixCheckpointCache::evicttakes the least recently used leaf, so a 380K-token endpoint and a 9K-token endpoint are equal candidates, and the long one is usually the oldest.Proposed change
--prefix-cache-snapshots N. A runtime flag for the slot total, forwarded to every rank like the other prefix-cache options. Rows still alias KV backing and stay bounded by the prepared extent, so no re-export is needed.Tests:
cache::tests: one checks that a 1,024-token endpoint survives exactly(1024/64 − 1) × 8evictions of 64-token endpoints, and that a lookup renews its credit. The other checks that compaction keeps the credit.scheduler::testscase that drives admit, prefill, snapshot, and finish with 3 slots: a 512-token conversation, then 6 short requests, then a follow-up that must reuse 512 tokens.capacitiestest for the flag, and an extended CLI forwarding test.qualify/prefix_retention.py, which runs the same scenario against a live server: one long conversation, then twice as many multi-turn short conversations as there are slots, then a follow-up. It checkscached_tokensand asks for a code from the first message.The three new behavior tests fail on
bce032c0(the conversation goes first) and pass with the change. The helios library suite passes (1224 passed). Twoflare/tests/unverified_payload.rstests fail both with and without the change.We have not yet measured it on the live ring; we will post the
prefix_retention.pynumbers at ~300K and ~900K tokens once we can restart the card.On question 3: our cold prefill also slowed with message count, and part of it was frontend tokenization of every message prefix. That loop is gone on main since 67ba265 and 03940fa.
Would you accept this as a PR? We are glad to change the policy, the flag name, or how the default is derived, for example from the page pool.
yatesdr commentedon Nov 3, 2026
I believe this was mostly addressed with PR#168 (now merged). The count is now limited by the GPU memory available, and test machine went from 256 resume points to 1,991 (ds41f 1017).
Conversation length priority is not planned to prioritize eviction, it’s strictly last referenced. DRAM and NVME offloading are now implemented, so rehydration of evicted long context is a lesser concern.
Please test latest main to see if this is resolved
feat(#22): Feat(ignition): accept --prefix-cache-states on ignition e…