DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works)

Hi, and thank you for Flare. We run DeepSeek-V4.1-Flash-1017 at TP=4 on four DGX Spark (GB10) nodes on the RoCE ring, and it serves well from a bce032c package. When we moved to main at eb71702b, the export fails in "Warm compiled forwards" with a GPU fault on one rank. We are not sure where to look next and would appreciate advice.

What happens

Two runs with the same options. Each one fails on a different node and in a different graph, but the Xid ESR values are identical:

Run	Rank (node)	Forward being warmed	Step
1	2	`packed-speculative4x6-drafts5-tokens21`	Warm compiled forwards 29/72
2 (cache warm)	3	`prefill3072`	Warm compiled forwards 2/72

NVRM: Xid (PCI:000f:01:00): 13, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Illegal Instruction Encoding
NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x505730=0x20009 0x505734=0x20 0x505728=0x1c81fb60 0x50572c=0x1174
NVRM: Xid (PCI:000f:01:00): 43, pid=..., name=ignition, channel 0x00000002

The faulting rank reports CUDA_ERROR_ILLEGAL_INSTRUCTION (715). The other ranks then report RoCE collective on rank N timed out waiting for rank 2 (run 1) or RoCE proxy failed: two-shot op 475 abandoned: kernel timed out (run 2). NIC counters show no discards, receive errors or link-down events, so we read the RoCE errors as a consequence of the fault.

Setup

4x DGX Spark GB10 (compute capability 12.1), DGX OS OTA 26.09.2, driver 580.178.04, CUDA 13.3
Flare eb71702b with four small local patches (frontend tool choice, image placeholder text, prefix snapshot eviction, payload hashing). None of them touch kernels or Ignition preparation.
torch 2.14.1+cu132, triton 3.8.0, flashinfer-python 0.7.1 (local-inference-lab/flashinfer@f91f257e), nvidia-cutlass-dsl 4.7.1
ignition export <official checkpoint> --gpus 0 --ranks 4 --roce-ring --transfer-streams 8 --max-concurrency 4 --mtp-draft-tokens 5 --prefill-chunk-size 4096 --expert-scale-compression csf
The same options at bce032c export and serve with no fault, on the same OS and hardware, today.

Questions

Is a kernel on this path known to need a newer FlashInfer or b12x build for SM121? We noticed 085ae4a7 (FlashInfer MXFP4 MoE), Prepare DeepSeek V4.1 MXFP8 attention with public b12x plans #109 (MXFP8 attention) and Fuse DSpark draft scoring and sparse support restoration #95 (fused draft scoring) between the two pins, but we have not bisected.
Is there a flag or an environment setting that makes the faulting kernel easier to identify, for example by disabling per-rank tuning selection, so that we can send a more useful report?

We can run a bisect or a debug build on the four nodes if that helps. Thank you.

Hello, I found this same problem yesterday morning 3 dgx spark also deepseek v4.1 flash 1017, I have been since chasing it but still unsure what does it cause it, hope to update with news soon after some more testing, now thx to you im sure its not my setup.

Follow-up with what I have narrowed down so far, on 3x GB10, driver 580.178.04, DeepSeek V4.1 Flash TP3:

Regressing commit. Bisecting gives cb6187b ("Agree and freeze Inductor kernels before graph capture"), which falls inside your bce032c..eb71702 range. Its parents (e.g. ca1f960) export and serve fine for me, so building from a commit before cb6187b is a working stopgap.

Where it fails. In InductorSelections.warm(), during the second (frozen) warm pass, after _reconcile -> _install has rebuilt the kernel launchers. The dmesg signature is always the same as yours:

Xid 13, Graphics SM Warp Exception ... Illegal Instruction Encoding
Xid 13, Graphics Exception: ESR ...=0x10009 ...=0x20 ...=0x1c81fb60 ...=0x1174
Xid 43

What I have ruled out (each one is a full export, with the result):

_install doing nothing: passes. Installing the selection without rebuilding the launchers: passes.
Rebuilding the launchers from the live compile result (no pickling, no restored config; just a second module load of the same cubin): fails. So it is the reload itself, not the agreed selection.
Keeping the old launchers alive so no module is ever unloaded: fails, so it is not a dangling handle after unload.
One install per kernel instead of repeated installs: fails.
Capping the GPU clock at 2300 MHz: fails.
TORCHINDUCTOR_USE_FAST_TRITON_LAUNCHER=0 (no cached raw CUfunction): fails.

The faulting kernel is not at fault. With a torch.cuda.synchronize() after every static launch in the frozen warm, the fault lands on a specific kernel, but a different one per run (a reduction with R0_BLOCK 512 / 4 warps once, a small pointwise kernel another time). The SASS of the cubin that was loaded disassembles cleanly with nvdisasm and is identical to a fresh compile. Loading and launching that same cubin in isolation passes thousands of times, including with heavy memory churn and thousands of other modules loaded.

Not Flare-specific. vLLM on the same nodes has died with the very same Xid 13 and the same ESR values, which points at the driver/platform rather than at the kernels.

So far: reloading already-loaded Inductor cubins as new modules mid-process and launching from them triggers it, intermittently and at a random kernel. I'm still investigating and working on a standalone reproducer outside Flare; will post here when I have one.

Update: on the same 3x GB10 setup, upgrading the driver from 580.178.04 to 615.71.09 makes the problem go away.

With 580.178.04, every export of DeepSeek V4.1 Flash TP3 at a commit including cb6187b failed in "Warm compiled forwards" (between warm 13 and 32 of 72) with the Xid 13 / ESR signature above.
With 615.71.09 on all three nodes, the same export at the same commit, with no code changes, has completed fully twice ("Warm compiled forwards 72/72", package exported), with no Xid in dmesg on any node. One other run stopped in the first warm on a RoCE collective timeout, again without any Xid, and has not happened since.

Driver details, in case it helps anyone reproduce: Ubuntu 24.04 noble-updates packages nvidia-driver-615-open plus the prebuilt, signed linux-modules-nvidia-615-open-$(uname -r) (no DKMS needed, works with Secure Boot), on the DGX OS 7.0.0-1019-nvidia kernel (4K pages). nvidia-smi reports CUDA 13.4. The only new boot message is a harmless NV_ERR_NOT_SUPPORTED from koobInitAperture.

Note that NVIDIA currently validates only the DGX OS R580 driver on DGX Spark, so this is ahead of the supported stack; building from a commit before cb6187b remains the option on R580.

@javieratomic

I'll test it with NVIDIA Flare on my 4 DGX Sparks @javieratomic!

Data point from a different model and TP size: on main at 6fa01a49 (includes cb6187bf), a Qwen3.8 Flash Next TP2 export finished on two DGX Spark (GB10) nodes without this fault, on the R580 driver.

Driver 580.178.04, kernel 7.0.0-1019-nvidia, GPU clocks capped at about 2350 MHz
nvidia/Qwen3.8-Flash-Next-FP4-Local @ b9ce18a6, FlashInfer f5f5341d (from uv.lock)
ignition export <checkpoint> --ranks 2 --gpus 0 --max-concurrency 8 (default MTP 5)
"Warm compiled forwards" 60/60 in 134.9 s, no Xid in the kernel log on either node during the export and the serving that followed

This is one export, and the fault is reported as intermittent, so it doesn't show Qwen is immune. It may help narrow the problem to the model, the graph count or the rank count.

@calvarado2004

@calvarado2004 were you able to test with the driver version >= 615.71.09?
Please note that there was a fix for this, a794c99, but wanna see if driver fixes it, so please use a version without it

A note connecting this to #187. Two independent changes each avoid the fault, which suggests cb6187bf is the trigger and the R580 driver is the cause:

Driver. 615.71.09 avoids it with no code changes (above).
Memory retention. a794c99d keeps transient physical memory alive across compiled-forward warmup and was added for the same illegal instruction on DGX Spark. It does not stop the launcher rebuild. On a 96 GB SM120 GPU running driver 615.71.09, disabling that retention produced no fault in a full qwen3.8-27b export (qwen3.8-27b export runs out of memory during compiled-forward warmup on 96 GB SM120: transient physical retention accumulates across forwards #187), though the driver alone may explain that.

cb6187bf makes _reconcile call _install for every newly registered kernel. That reloads its cubin through _make_launchers(), and the frozen warm pass immediately runs the reloaded module, which matches the reload-only failure bisected above.

A question for maintainers: on a single GPU, or when every rank already holds the winning selection, _install reloads the selection the tuner already has. Could _install be skipped when the current launcher matches the winner's config and cubin, if its config normalization and dynamic_scale_rblock setting are preserved? That would cut most reloads without weakening agreement, and it might reduce the need for retention on R580. It has not been tested, and I have no Spark to test it on.

@mark-yong

@mark-yong indeed, Driver 615.71.09 works pretty well.

@lukealonso

@lukealonso suspect that driver version was the cause of the issue that necessitated a794c99, triggered by cb6187b's kernel reloads

@mark-yong

@mark-yong We upgraded today, but this isn't the test you asked for: our build still includes a794c99d, and we haven't run an export on 615 yet. Here's what we have so far.

Two DGX Spark (GB10) nodes, TP2, upgraded from 580.178.04 to 615.71.09 on the DGX OS 7.0.0-1019-nvidia kernel.
Flare main at 6fa01a49, which contains both cb6187bf and a794c99d, serving a prepared Qwen3.8 Flash Next TP2 package.
No Xid in the kernel log on either node after three starts and about 80 minutes of load: decode runs up to 16 concurrent requests, Hotel Lights, and image renders on one node alongside.
With the same flags, decode at 8K context was about 10% faster than on 580, at both one and four concurrent requests. That's one comparison.

Two install notes for anyone following the steps above on DGX OS:

On our nodes, apt-get install nvidia-driver-615-open pulls in nvidia-dkms-615-open and a new kernel (7.0.0-1024) through the linux-*-nvidia-hwe-24.04 meta-packages, even with linux-modules-nvidia-615-open-$(uname -r) named. To change only the driver, we installed nvidia-headless-no-dkms-615-open, nvidia-utils-615, the libnvidia-*-615 libraries you need and the prebuilt module, with those meta-packages pinned at their installed versions. Afterwards we ran apt-mark hold on them, so a later upgrade can't boot a kernel that has no 615 module. Run apt-get -s first to check the plan.
If you pointed LD_LIBRARY_PATH at a CUDA forward-compatibility libcuda on R580 (ours was 595.58.03), remove it after upgrading. Otherwise the older compat library loads instead of the 615 driver's own.

Navigation Menu

Repository navigation

DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works) #137

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DeepSeek V4.1 TP4 export on GB10: Xid 13 "Illegal Instruction Encoding" while warming compiled forwards at eb71702b (bce032c works) #137

Description

Activity

javieratomic commented on Oct 29, 2026

javieratomic commented on Oct 29, 2026

javieratomic commented on Oct 29, 2026

calvarado2004 commented on Oct 29, 2026

CaptKitCarson commented on Nov 2, 2026

mark-yong commented on Nov 3, 2026

mark-yong commented on Nov 3, 2026

calvarado2004 commented on Nov 3, 2026

mark-yong commented on Nov 3, 2026

CaptKitCarson commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions