Multi-host RoCE cores exported at 0572fd12 do not start: pinned FlashInfer 7b35510b records proxy ABI 14, the runtime provides 11 #271

What happens

A DeepSeek V4.1 Flash core exported with tensor parallelism 3 on three GB10 hosts (one rank per host, RoCE) does not start under the runtime built from the same tree. Every rank loads its weights and allocates its cache, then stops at "Connect RoCE transport":

[  FAILED  ] Connect RoCE transport.  0.0s
Error: Execution(Poisoned(Failure { message: "core RoCE kernels use proxy ABI 14, this runtime provides 11", source: None }))

The exporter and the runtime of 0572fd12 disagree on the RoCE proxy ABI:

• c74b613 pins FlashInfer 7b35510b, whose flashinfer/experimental/b12x/comm/roce/_roce_proxy.c defines ROCE_ABI_VERSION 14. local-inference-lab/flashinfer c9b42612 ("Route RoCEnante over direct cables with queue-pair lanes, a partner relay and overrun credits", [b12x] Route RoCEnante over direct cables with queue-pair lanes, a partner relay and overrun credits local-inference-lab/flashinfer#81) raised it from 11. The exporter stores the value that library reports as the proxy_abi of the core's transport (flashinfer/experimental/b12x/comm/roce/roce_oneshot.py:1206).
• The runtime's vendored proxy still defines ROCE_ABI_VERSION 11 (helios/native/roce_proxy.c:103), PROXY_ABI is 11 (helios/src/execution/roce.rs:23), and open accepts only cores that record ABI 9, 10 or 11 (roce.rs:125).

#251 (e774a46) matched the runtime to ABI 11, which the pin of that time, 85ea6e76, and the later 4e977a53 (d095e9e) both define. c74b613 replaced ead6cfc1 (b3bf45f), a revision GitHub no longer resolves, so I could not check its ABI. The message of c74b613 says the pinned revision sits "on top of FlashInfer main's RoCE transport changes" and that "cross-host transport has not been retested here".

Why it matters

A multi-host RoCE core exported at 0572fd12 cannot be served by the runtime of 0572fd12. The mismatch is reported only when a rank connects its transport, after the export has completed (9527 s here) and each start has loaded the weights.

Evidence

• Five starts of one core, three ranks each: 13 of the 15 logs contain the message above. The other two, one of rank 0 and one of rank 1, end at the failed "Connect RoCE transport" step (rank 0 also prints a failed "Load tokenizer" line) without the error text. The block above is from a peer rank.
• The core was exported from f25a0386 (Make the compiler memory headroom a preparation setting #122 on 0572fd12) with --compile-headroom 4G, because the default export of 0572fd12 stops in kernel tuning on these hosts (GB10 TP3: DeepSeek V4.1 Flash export at 0572fd12 fails kernel tuning with a 0-byte scratch limit; 7.1-7.9 GiB available, below the 8 GiB floor #270). f25a0386 changes no Rust or C source and no dependency pin relative to 0572fd12.
• The runtime was built from the same tree with cargo build --release --locked --manifest-path flare-cli/Cargo.toml --features roce.
• direct_url.json of the installed flashinfer-python 0.7.3 records commit 7b35510b, as uv.lock pins.

Not measured

• Whether ABI 12 to 14 change what the captured kernels see, or only the proxy's own side. The vendored proxy and the pinned FlashInfer's _roce_proxy.c differ in more than the version constant.
• Single-host cores and other models. The check runs only where a core records a RoCE transport.

Possible direction

• The repair Match native RoCE transport to the pinned FlashInfer proxy ABI 11 #251 made for ABI 11: bring helios/native/roce_proxy.c, PROXY_ABI and the accepted set in open up to the pinned FlashInfer's ABI 14. Route multi-host RoCE rails over direct cables #232 (open, "Route multi-host RoCE rails over direct cables") changes roce.rs and does not change these constants.
• A test that compares the vendored proxy's ROCE_ABI_VERSION with the one in the pinned FlashInfer, so that a pin that changes the ABI fails in CI. native_proxy_abi_matches_core_contract (roce.rs:387-390) compares the vendored proxy only with PROXY_ABI.

Environment

• Three GB10 hosts (DGX Spark class), one rank per host, RoCE between them.
• Checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017.
• 0572fd12 runtime; core exported from f25a0386; FlashInfer 7b35510b.