Follow me on X for more updates: https://x.com/jayleaton
Support me here: https://buymeacoffee.com/jayleaton
A recipe for serving DeepSeek-V4.1-Flash-1017 on two NVIDIA DGX Sparks behind an OpenAI-compatible API. It ships our EXL3 pack (flash17) and a pinned upstream TensorFold native build. The measurements below describe the previously deployed engine and profile.
| Model | deepseek-ai/DeepSeek-V4.1-Flash-1017 |
| Pack | EXL3 2.8 bpw, 203.8 GB: jayleaton/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpw |
| Hardware | 2x DGX Spark (GB10, 128 GB each), tensor parallel 2 over the 200 Gb/s CX7 link |
| Engine | upstream TensorFold, pinned in config/tensorfold.rev |
| Serving | 1M context, four slots, vision (native image input), tool calling, structured output, built-in Python tool (compute lane) |
This branch migrates the recipe to upstream native serving. The pin is TensorFold 12257eb51 on
dsv41/undeclared-tool-passthrough: 1ea71240a plus two server commits, so a whole tool call to a tool the request did
not declare comes back as a tool call, as on vLLM, and the client can refuse it
(ENGINE). 1ea71240a (dsv41/session-lcp-resume) is
7b79ef92d plus opt-in prompt checkpoints for rewritten histories
(TF_DSV41_SESSION_LCP_RESUME, off by default) and a log line on each costly session miss
(#80). 7b79ef92d (dsv41/compute-lane-release-s86) is
01a62132c plus the NVMe session-restore fix of upstream #2182 (sessions parked on
the NVMe tier come back instead of failing with BadBlob; #86).
01a62132c is the engine source the compute-lane studies ran (07106059c,
byte-for-byte) plus TF_DSV41_COMPUTE_LANE_DEFAULT and its follow-ups (a client's own python function keeps the name;
compute_executed counts only runs; the built-in tool only for thinking chats with room for a call; no runner cell
for half an expression). That is release
candidate G (58bece1d3, fallback; F 56200688d before it) plus deadline_s, the engine-held compute lane and the
experimental in-model runner. The flash17 profile's default is the in-thinking "tool-full" profile, whose runner and
trust guide run for every thinking chat whatever the client sends (the built-in tool only when its reply has room for a
call); it needs the runner image (scripts/build-runner-image.sh). The measured lane-only arm is an opt-in block. The
pin's history holds the stable DeepSeek head H (20ad77878, re-recorded there as
b073317ab) and includes flash17’s dsq-1 admission support. The pin is not on upstream main yet, so the build fetches
it from JJ's TensorFold fork (config/tensorfold.repo) until the upstream PRs merge; a local
checkout or bundle also works. See ENGINE.
On the 2026-10-29/30 spot checks against the previous engine, H passed every correctness gate, with 0 token differences. It was at parity on 8K prefill (1.06×), 1-stream decode (1.00×) and auto tools (1.02×); 4-stream decode was 0.97×. The details are in ENGINE. Full engine/pack/device qualification is still pending, and the historical production results below do not qualify this pin.
Work in progress. Measured on one pair of Sparks. Read what is not solved before relying on it.
Tasks. 356 matched items (pass@1, greedy, thinking on, effort high), against Mia-AiLab's 2.9 bpw pack and our previous q28-v2 pack:
| Suite | flash17 | Mia 2.9 bpw | q28-v2 |
|---|---|---|---|
| HumanEval+ | 34 / 40 | 34 / 40 | 33 / 40 |
| MBPP+ | 50 / 60 | 49 / 60 | 50 / 60 |
| MMLU-Pro STEM | 93 / 100 | 91 / 100 | 90 / 100 |
| MATH-500 | 36 / 40 | 36 / 40 | 36 / 40 |
| SciCode (first step) | 6 / 10 | 6 / 10 | 5 / 10 |
| LiveCodeBench | 16 / 20 | 13 / 20 | 13 / 20 |
| BFCL | 39 / 40 | 39 / 40 | 39 / 40 |
| Tool chains | 6 / 6 | 6 / 6 | 4 / 6 |
| MGSM | 40 / 40 | 40 / 40 | 40 / 40 |
Small samples: every paired 95 % confidence interval includes zero, so these show no statistically significant difference. Intervals, method and limits are in FLASH-1017.
Retention of the 1017 release, as served (top-1 agreement with the release, with the serving expert pruning): code 95.17 %, science 94.93 %, tools 87.64 %. The full table, with KL and the other packs: FLASH-1017.
Speed at the recommended profile (2.1M pool, 2100 MHz cap), single stream: short decode 87.9 tok/s; decode after a 32K / 131K prompt 64.1 / 64.9 tok/s; prompt tokens over time to first token 2,349 / 2,198 tok/s (TTFT includes HTTP and admission). q28-v2's earlier measurements, under a different clock and its 3.0M pool, were 85.75, 66.7 / 64.5 and 2,630 / 2,460; not a matched A/B. Details: FLASH-1017.
Images and agents at the recommended profile: VQA 20 / 20 plus 3 / 3 image_url probes; a clean OpenCode
session passed (7 valid tool calls, normal stop).
The full guide is ZIG-SERVE. Build the image and the per-rank assets as it describes, then:
git clone https://github.com/jayleaton/deepseek-v41-tensorfold-spark.git
cd deepseek-v41-tensorfold-spark
scripts/build-native.sh -t tensorfold-dsv41:local # the engine at the pin
scripts/build-runner-image.sh tensorfold-dsv41:local -t tensorfold-dsv41:local-runner # + runner assets (downloaded, sha256-checked)
hf download jayleaton/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpw --local-dir /srv/models/flash17 # both nodes
cp config/prod-zig-flash17.env.example config/prod-zig.env && $EDITOR config/prod-zig.env # both nodes
scripts/run-zig.sh 1 --check-aot # after generating assets; compare both rank identities before starting- Fill in the node paths and link settings from
config/prod-zig-flash17.env.exampleand keep its engine settings. It needs the 2.1M-token pool (TF_DSV41_POOL_TOKENS=2100000). With q28-v2's 3.0M pool this pack goes under the 4 GiB memory floor during 4K prefills (why). - Pack the 1017 Engram shards with
scripts/pack_engram.py(Engram setup, withdeepseek-ai/DeepSeek-V4.1-Flash-1017as the source); the September shards do not match this pack. - We suggest a 2100-2200 MHz GPU clock cap (
nvidia-smi -lgc 300,2100) to help with temperature control. - Check the answer to
What is 17 * 23?is 391 before serving traffic.
The previous pack, q28-v2
(jayleaton/DeepSeek-V4.1-Flash-EXL3-2.8bpw),
keeps its own unchanged profile, config/prod-zig.env.example (September Engram shards,
3.0M pool). Stop rank 0, then start both ranks with that config:
FLASH-1017, rollback. Its measurements are in
PREVIOUS-RESULTS.
| FLASH-1017 | the flash17 pack: profile, memory, task results, retention, rollback |
| ZIG-SERVE / ZIG-RESULTS | Zig build, TP=2 launch, exact settings, q28-v2 comparisons |
| PREVIOUS-RESULTS | q28-v2 kit results and the historical 2.9 bpw Python vs vLLM tables |
| INSTALL | native requirements, weights and Engram shards |
| API / IMAGES | endpoints, thinking and effort, image input |
| OPERATIONS | settings, memory gates, the optional thermal and memory guard, turning each lever off |
| SECURITY | where keys live, which processes hold them, what the output filter promises |
| METHOD / RESULTS / BENCHMARKS | how it was measured, every table, how to rerun |
| CHANGELOG | bug fixes and new levers since G13 |
| KNOWN-ISSUES | what is not solved |
| PROJECT / ENGINE / ARCHITECTURE / DECODE | patches, layout, model split, decode roofline |
| campaign/ | the development log, windows G1-G19 |
Built on TensorFold by Ash Hart (ashhart), and on DeepSeek's model, papers and code. Measured against Mia / MiaAI-Lab's 2x DGX Spark kit and 2.9 bpw pack. EXL3 is by turboderp / ExLlamaV3. Every contributor, project and benchmark we used is credited in CREDITS.
This project's code, patches, scripts, benchmarks and docs: Apache License 2.0 (LICENSE,
NOTICE). TensorFold, the added files, the adapted kernels, the base image and the model weights each keep
their own terms: docs/LICENSING.md.
HTTP tooling: bench/speed.py and scripts/preflight.py; see BENCHMARKS.