[adenzhou1350]

Dependency

This is a stacked follow-up to #8511, which is currently a draft pending exact-head multi-GPU requalification. Until that PR merges, GitHub also shows its prerequisite commits; the change specific to this PR is the final ZeRO-1/2: configure quantized weight group size commit.

Summary

• add zero_quantized_weights_group_size for ZeRO-1/2;
• keep the default at 64;
• require positive multiples of 8;
• reject non-default use with ZeRO-3;
• forward the setting through the engine and optimizer.

This exposes the accuracy/scale-traffic tradeoff without changing defaults. A prior qualification workload used group size 8192, but this PR does not propose 8192 as a shared default.

Current status (80e452a)

• rebased on the prerequisite correctness and bounded-memory fixes in ZeRO-1/2: add bounded quantized weight all-gather #8511
• DCO and GitHub test collection checks pass
• exact-head multi-GPU correctness and performance requalification is pending

This PR remains a draft until both its prerequisite and the configurable group path have device-backed distributed evidence on their current heads.

Historical pre-fix evidence

The following results came from the earlier stack, before the current prerequisite fix/rebase. They screen the idea but are not current-head qualification:

• Qwen3-4B, sequence 2048, BF16, ZeRO-2 DP4, fused AdamW, four PCIe-connected RTX 5090 GPUs
• 20-step trajectory: all losses and global norms finite; zero parameter-sample tolerance violations
• four order-balanced performance blocks: 1.03633x, 1.03936x, 1.03794x, 1.03614x
• median speedup: 1.03713x; bootstrap 95% interval [1.03614, 1.03936]
• peak allocated/reserved memory: 28.85/30.21 GB
• the earlier config-path probe observed group sizes 64 and 8192 on every rank and reproduced its own qualified final parameter hashes

The comparison also used a 100M reduce bucket. That was a workload setting, not a new default or source constant.

Scope and risks

Larger groups trade scale traffic for precision and must be validated per workload. Existing evidence does not establish behavior for ZeRO-1, FP16, offload, MoE, H100/B300, NVLink/RDMA, other models, or long-horizon convergence.

[adenzhou1350]

Closing this stacked follow-up to keep the upstream queue focused. Its prerequisite is still undergoing exact-head requalification, and this configurable-group layer does not yet have independent current-head distributed evidence. The branch and local evidence remain available if a maintainer later requests this surface after the prerequisite is qualified.