Host-bridge TP4: cross-quadrant peer-read stalls and fan-out write collapse make decode ~40% slower; forced phased-write PoC reaches switch level #146

Summary

On frank2, DeepSeek V4.1 TP4 on GPUs behind different CPU IO-die quadrants (0,4,8,12) runs ~40% slower per decode round than on switch-local GPUs (12-15). The same package and settings are used on both. Two fabric effects cause it:

1. Peer reads across quadrants stall in periodic windows (1-1.6 ms per 80 KiB read). This hits pull, bulk_pull and every other pull-style transport.
2. Writes that fan out to several quadrants at once collapse by 8-13x. This hits full, payload_rsag and bulk_push above a few KiB.

Every joint mode in the GB202 ladder is one of these two kinds, so the runtime race can only pick between stalling reads and collapsing writes.

A proof of concept forces write-only, one-destination-at-a-time ("phased") collectives through a ladder with one profile per bucket. The rest of the package is unchanged. It brings host-bridge GPUs to the level of the switches:

The forced phased modes also run on switches (0-3) and cost 4-6% there. On host bridges the PoC is within 1-2% of the same PoC on switches: once reads and fan-out are gone, the host-bridge fabric adds almost nothing.

PoC branches. The phased kernels are Claude-written, so they are not mergeable under design-review principle 14:

• Flare: claude/poc-hostbridge-phased (10cceb2): ladder mode names, the forced ladder, tests, export/bench scripts.
• FlashInfer: claude/poc-hostbridge-phased: FI 443d8fad + phased joint submodes (4703658b8, from FI Parse Gemma 4 reasoning channels and tool calls #62), a PoC ladder-tool filter (39cfd638b), and the diagnostic harness (c0cb3d5a5).

1. Hardware and topology

frank2:

• AMD EPYC 9575F, 1 socket, NPS1;
• kernel pcie_aspm=off pcie_port_pm=off amd_iommu=off iommu=off;
• driver 615.71.10;
• 17x RTX PRO 6000 behind c-payne Switchtec Gen5 boards.

Each switch chip is split into two partitions with separate x16 uplinks, and both uplinks land in the same IO-die quadrant (one NBIO):

GPUs 0-3 are all in quadrant 0, which is why 0-3 performs like frank1. GPUs 0,4,8,12 span four quadrants. Side note: GPU 16 links at Gen5 x4.

2. What we measured

The tool is a standalone harness (benchmarks/experimental/b12x/pcie_fabric_stall/ on the FI branch):

• one process, one persistent kernel per GPU;
• every operation is timestamped with %globaltimer (start, after a cross-GPU start barrier, end), with clock alignment;
• patterns: read/write ring/all/incast/pair, one-shot pull/push, phased push, ring, RSAG push/phased/pull, host read/write;
• a second harness runs NCCL all-reduce inside CUDA graphs with per-operation stamps.

2.1 Cross-quadrant peer reads stall in periodic windows

• Windows: every quadrant has a ~150-200 ms window every ~2.30 s. The quadrants take turns ~300 ms apart (00 -> 40 -> 80 -> c0). The phase is fixed in wall-clock time and does not depend on the measuring process. A 4-quadrant group therefore has a degraded target ~30% of the time.
• Size of the slowdown: inside a window, reads whose target is in that quadrant are 10-50x slower. Single 80 KiB reads take 1.1-1.6 ms, and up to 2.7 ms at 400 KiB.
• Not the GPU: a reader behind the same switch reads the same target at the same moment without any stall.
• Within one quadrant reads are clean: reads between host bridges 00 and 20 (GPU 1 <-> 2,3) had 0 stalls in 700k reads.
• Posted writes never stall: write-only patterns peaked at <= 55 us over 150k+ operations spanning many windows. Host-memory reads and writes are clean too.

2.2 Fan-out writes across quadrants collapse

• A GPU that interleaves writes to destinations in different quadrants is 8x (3 quadrants) to 13x (4 quadrants) slower than one writing one destination at a time.
• Interleaving to one remote quadrant, or within a switch, is fine.
• This looks like the source side ordering posted writes across fabric destinations. Rotating destinations per GPU-wide phase, or a ring, avoids it.

2.3 Same patterns on 0-3 versus 0,4,8,12

80 KiB, 16 CTAs; per-GPU mean / max per operation, including the ~5.5 us start barrier.

2.4 Transport comparison on 1,5,9,13 (same switches and quadrants as 0,4,8,12)

= 3 s per cell, so every cell spans several windows.

Pull has the best median and the worst mean and tail. A median-based startup race prefers it, but serving pays the mean.

The phased cost on a fabric where fan-out works is small. On one quadrant (13-16) at 16 KiB: pull 8.2 us, phased push 8.3 us, fan-out push 9.2 us.

2.5 NCCL and vLLM

• NCCL 2.30.7 is not affected by default. On this topology it picks SHM/direct (GPU writes to host memory), which gives 22 us at 80 KiB with no stalls.

  • With NCCL_P2P_LEVEL=SYS it uses an LL ring of posted writes, one destination per GPU, and is the fastest small transport (14.3 us).
  • Only NCCL_P2P_READ_ENABLE=1 NCCL_PROTO=Simple makes it read across quadrants, and then it shows the same stalls (max 1.63 ms).

• vLLM (fraserprice DeepSeek-4.1-Flash-1017-BRRR image):

  • <= 96 KB: b12x one-shot pull, including the fused add+RMSNorm path;
  • 96 KB-1 MB: its own phased_all_reduce.cu (posted writes, one destination per GPU-wide phase, rotation);

  • 1 MB: NCCL with NCCL_P2P_LEVEL=SYS NCCL_P2P_READ_ENABLE=1.

  Its small path would hit the same read windows on 0,4,8,12; we did not serve it there.

3. Ruled out

All HSMP changes were read back and restored. One side observation: CPU config-space reads of a GPU (our 0.5 ms LnkSta sampler) add extra read stalls targeting that GPU.

Conclusion: the windows come from per-NBIO periodic firmware activity that no OS-visible control reaches. Only BIOS options or AMD can change it. Software can avoid it completely by not reading across quadrants.

4. Why the current runtime selection cannot fix it

• No good candidate on host bridges. Every joint mode is either a peer read (pull, bulk_pull) or a fan-out write (full, payload_rsag, bulk_push).
• The race reads medians. It decides on medians over a few ms, which hides 2.3 s windows. Pull wins on median.
• Choices change per start. Two starts of the same package on 0,4,8,12 picked Pull 8 / Bulk Pull 8 / Full 4 / Payload RSAG 2, then Pull 9 / Bulk Pull 7 / Payload RSAG 4 / Full 2. In the second start, c32 fell to 1007-1026 tok/s because the adaptive scheduler mostly ran b16w8 rounds.

5. Everything we tried

6. The PoC

What changed against the measured reference package. The reference is Flare 4f20ac5 (= main ladder code) with FI 443d8fad, DS4.1 c32 CSF, --collective-tuning runtime with gb202_pcie_tp4.json. The PoC changes:

1. FlashInfer: FI 443d8fad plus full_phased, payload_rsag_phased and bulk_push_phased. They move the same data as their base modes, but every CTA of a rank writes one peer at a time in rank-staggered order. The joint compile key goes to 211 so stale cache entries are never reused.
2. Flare: the three names are added to ignition/preparation/collective_ladder.py.
3. Ladder: ignition/preparation/ladders/frank2_hostbridge_tp4_forced.json, 255 buckets from 16 B to 8 MiB, one profile per bucket, so the startup race has one candidate.
   • It is built from a b12x.tools.collective_ladder sweep on 0,4,8,12 with pull and bulk_pull excluded: 84,200 configurations, 1200 s.
   • Write-only modes have no stall tail, so each bucket keeps the lowest mean among profiles within 3% of the best median.

The resulting ladder:

For comparison, the existing modes on 0,4,8,12 at 32 rows (320 KiB) take 32.5 us (pull, with stalls) and 28.9 us (bulk pull); the phased choice takes ~20 us.

Export and tests.

• Same command and environment as the reference package (frank1 GPUs 8-11), only adding the ladder. The export took ~18 min and resource release 14.7 s.
• Weights are byte-identical to the reference; only 14 deployment/index files differ.
• FI test_pcie_joint_gpu.py passes on 0,4,8,12 (13 passed, 12 skipped).
• Flare tests/preparation/test_collective_ladder.py passes for the new ladder, including b12x accepting every pinned geometry at both ends of every bucket. The one remaining failure is a native-extension mismatch in the test environment, and it fails the same way on main.

Startup:

• the selection is identical on every start: Full 2, full_phased 6, payload_rsag_phased 11, bulk_push_phased 2, Payload RSAG 1;
• no pull of any kind;
• startup tuning takes 1.33 s instead of 2.45 s.

Serving measurement.

• frank2, --max-concurrency 32 --mtp-draft-tokens 7 --mtp-verification-token-budget 128 --embedding-table-storage ram --prefix-cache-memory 64G.
• llm_decode_bench.py with 2 x 30 s cells per concurrency at temperature 1, after a warm-up.
• Runs alternated: current, PoC, current, PoC, on 0,4,8,12 and again on the switched GPUs 0,1,2,3 (two PIX pairs in one quadrant). 12-15 has one run of the current package. Ratios use the mean of the cells.

c1 differences are within draft-acceptance noise at temperature 1.

GPU time per speculative round, from Flare's flare_speculative_gpu_nanoseconds_total / flare_speculative_rounds_total (slowest rank). This does not depend on draft acceptance.

On 0-3 the current package's startup race picks Bulk Push 7, Full 4, Payload RSAG 11 (no reads), and that is the best result on this host.

Result.

• On host bridges the PoC makes a decode round ~40% faster: 8.2-8.3 ms against 13.9-14.1 ms.
• That is within 4-8% of the best switch rounds (7.6 ms at b1w8), and within 0.5-2% of the same forced phased ladder on switches.
• Throughput lands within 5-6% of the switches at c8-c32.
• It is stable across starts, which the current selection is not.
• The remaining gap is the cost of the phased modes themselves, which are 4-6% slower than the switch-optimal fan-out modes on 0-3. It is not the host-bridge fabric. A per-fabric choice (switch modes on switches, phased modes on host bridges) therefore gives switch-level results on both.

7. Recommended direction

1. Across quadrants, never read peers and never fan writes out. Use write-only transports with one destination per GPU at a time:
   • <= ~12 KiB: fan-out is fine (latency-bound);
   • ~12-110 KiB: phased one-shot push (or an LL-style write ring, which is what NCCL does);
   • ~115 KiB-1.6 MiB: phased RSAG (a write ring is within a few %);

   • = 1.7 MiB: phased bulk push.

2. Select by fabric class, not by a short median race. Peer reads look best on median and worst on mean. Options:
   • ship a ladder per fabric class (switch-local vs host-bridge/quadrant-crossing) and choose it from PCI topology at load;
   • or keep the startup race but leave read submodes out of the candidates when ranks span IO-die quadrants.
3. Kernels need a permitted author. The phased submodes in the PoC are Claude-written (FI Parse Gemma 4 reasoning channels and tool calls #62), so principle 14 rules them out. The measurements above are the spec; the branches show the shape that works.
4. Do not force phased modes on switches. On 0-3 they cost 4-6% against the switch-optimal fan-out modes. On host bridges they are within 0.5-2% of their own switch result. So the remaining gap to switches is the phased modes' own cost (phase barriers), not the fabric. A per-fabric choice gets both fabrics to their best.

8. Caveats

• One host (frank2) and one PoC export. The forced ladder is frank2-specific; on switches (0-3) it costs 4-6%, so it should only be used where ranks span IO-die quadrants.
• Throughput varies with draft acceptance at temperature 1; use the per-round GPU times for the comparison.
• Branches: the Flare branch is based on main 921afd5 (its ladder code is identical to 4f20ac5). The FI branch is based on 443d8fad. Neither carries a design sign-off. Do not merge.

Reproduce

• Harness: FI branch benchmarks/experimental/b12x/pcie_fabric_stall/README.md.
• PoC export, serving and A/B: Flare branch scripts/poc_hostbridge_collectives/README.md.
• frank2 data: /data/ssd2/claude-ds41/work/p2pstall/ (harness runs: dumps-*) and /data/ssd2/claude-ds41/work/poc-phased/ (ladder report, forced ladder, bench/{base-1,base-2,poc-1,poc-2,sw-base-1,sw03-base-1,sw03-base-2,sw03-poc-1,sw03-poc-2}).
• PoC package: /data/ssd2/claude-ds41/poc-package/.