[brandonmmusic-max]

A tensor-parallel group can mix RTX PRO 6000 Blackwell Max-Q Workstation Edition and RTX PRO 6000 Blackwell
Workstation Edition cards: the same GB202 silicon under device names that differ only by "Max-Q". A DeepSeek V4.1
TP4 export on such a group (one PCIe host, two cards of each, 96 GB per card) failed during compiled preparation
with BindingMismatch: rank-local forward does not match compiled guards. The guard named layer 20's experts
binding and execution_plan.decode_config.csf_inline_scales, false on two ranks and true on the other two: the
ranks had chosen different MoE plans.

RecordedPreparationSession.tune decides whether to shard tuning and share selections by comparing
(device name, local tuning catalog) across ranks. The names differ only by "Max-Q", so the ranks neither
sharded nor shared. Each rank raced its own candidates and kept its own winner, and the compiled tensor-parallel
forwards, which bind one rank's selections, failed their guards on the others.

The ranks compare the device name with the Max-Q token removed. Ranks that agree on that name adopt rank
zero's cached selections before tuning, shard candidate races so every rank takes one winner, and
agree_selections verifies agreement before compilation. Ranks whose names differ after removing the token
shard only on an exact name match and share selections only across hosts.

Validation

• test_max_q_rank_shares_and_shards_tuning_only_with_same_silicon in tests/preparation/test_tuning.py has rank
  zero report the Max-Q name and a peer report the same silicon without it: the peer adopts rank zero's selections,
  shards the race, and verifies selections. A peer reporting a GB10 does none of these. The same-silicon case fails
  on a161b17f and passes with this change; all 100 tests in the file pass.
• Full CPU suite on this host, CUDA hidden: 25 failed, 4,189 passed, 2,462 skipped with this change, against
  25 failed, 4,187 passed, 2,462 skipped on unmodified a161b17f. The failing ids are identical; no test fails
  only with this change.
• DeepSeek V4.1 TP4 exports on the group above, from 2026-10-29 builds of main (921afd5c) that also carried
  three local changes this PR does not include: acceptance of the host's CSF checkpoint layout (2-D scale
  planes), an OpenAI-compatible serving surface, and an override of torch's compile-package GPU-name check
  for the two card names (see Limitations):
  • Without this change the export stopped at the guard mismatch after 54 minutes.
  • With this change, the export completed kernel tuning and loaded every compiled forward (113/113) with no guard
    mismatch. It then stopped in "Warm compiled forwards" with CUDA out of memory. That failure comes from transient
    physical backing retained across the warmup catalog (a794c99d), not from tuning; pull request Reuse retained warmup mappings across compiled warmup #154 addresses it.
  • A diagnostic export that also released warmup transient backing immediately completed all 113 forwards,
    and its package booted and passed its quality checks, so no later preparation stage depends on per-rank
    selections.

Limitations

• The unit test covers two device names on one host; a cross-host group with mixed Max-Q and Workstation Edition
  names was not run.
• Both card types run rank zero's winning plan, which need not be the fastest for the lower-power card; compiled
  forwards require one selection, so agreement is the requirement.
• Other mixed groups are unchanged, because their names differ after removing the token.
• Agreement on tuning is necessary but not sufficient for these groups: tensor-parallel forwards compile once
  and every rank loads the same artifacts, and torch's compile-package check compares GPU names exactly, so a
  rank refuses forwards compiled on the other card name ("Compile package was created with different GPU").
  The exports above passed that check only through the local override; a GLM-5.3 Flash TP4 export on the same
  group without it stopped there. That check lives in torch/_dynamo/package.py, outside the files Flare's
  bundled patches may modify, so it is not addressed here.

Export evidence (DeepSeek V4.1 TP4, two Max-Q and two Workstation Edition GB202 cards)

Without this change (export rc=1 after 54 minutes):

RuntimeError: compiled forward binding failed: ['BindingMismatch: rank-local forward does not match compiled guards: GuardDebugInfo(
result=0,
verbose_code_parts=["L['forward'].decoder.layers[20].experts.binding.execution_plan.decode_config.csf_inline_scales == False  # if decoder is None or binding.execution_plan.decode_config.csf_inline_scales:  # flashinfer/experimental/b12x/moe/fused_moe/_binding_ops.py:16 in scale_buffers"],
num_guards_executed=34, ...]', '... csf_inline_scales == True ...', '... csf_inline_scales == False ...', '... csf_inline_scales == True ...']

With this change: no BindingMismatch; Load compiled forwards. 113/113, then Warm compiled forwards. 47/113
with CUDA out of memory (49 MiB free on GPU 0).

Diagnostic export with this change and immediate release of warmup transient backing: export rc=0 elapsed 77 min,
Load compiled forwards. 113/113, Warm compiled forwards. 113/113, package booted healthy after 200 s, all
quality checks passed.