[DataFlow runtime] Domino disagg: 1-server + DP=7 launcher + working --num-epochs

[maocheng23]

Summary

Adds a 1-inference-server + DP=7-trainer launcher for Qwen3-8B domino disagg, makes --num-epochs actually work in the online/disagg path, and adds config-only perf tuning that ~doubles single-node throughput (28.8 → 50.1 samples/s) plus the profiling instrumentation and analysis behind it. Stacks on domino-disagg (#660).

• examples/disagg/run_qwen3_8b_domino_disagg_1srv_dp7.sh — single capture server (1 GPU) + DP=7 domino trainer (7 GPUs).
• examples/disagg/run_disagg_domino.py — the producer now replicates the prompt pool by --num-epochs before streaming (the online channel is consume-once).
• New tuning knobs (all off by default) + env-gated profiling across the data path.
• examples/disagg/PERF_FINDINGS.md — full write-up.

Perf: 28.8 → 50.1 samples/s, config only (measured on 1×8 H200, ≥390 s integrated)

⚠️ Supersedes the earlier "trainer-bound / DP7 = 26.9" framing in this PR's history. Those single-~20 s-window numbers were noisy; everything below is total-samples ÷ total-time over ≥390 s, which is the only reliable method here.

Best config (see PERF_FINDINGS.md for the full recipe):

DISAGG_SERVER_URLS="$U,$U,$U,$U,$U,$U,$U,$U"   # 8 concurrent producer workers, one server
CLONE_ON_FETCH=0  LOADER_PREFETCH=2  SERVER_MEM_FRACTION=0.5  ACCUM=8  BATCH_SIZE=2

Result: 50.1 samples/s, all 8 GPUs healthy (server util 98% / SM 76%; trainers util 77–100% / SM 76–85%; trainer mem ~45 GB).

The pipeline is a supply/demand seesaw on 8 GPUs, and 1:7 is the best split:

Split / config	samples/s	regime
1 srv + 7 trn (this PR, tuned)	50.1	supply-bound (best)
2 srv + 6 trn (b2)	39.0	demand-bound — only 6 trainers
2 srv + 6 trn (b8)	39.5	demand-bound — bigger batch changes nothing

Bigger batch does not help (compute-latency-bound, MFU ~14%): b2/a8 = 50.1, b4/a4 = 47.5, b8/a2 = 47.2. Batch 8 drives trainer memory to 113 GB / 100% util but throughput falls — full util/memory ≠ fast.

Why per-sample trainer demand is "low" (num_anchors): the DFlash draft forward expands every sample to num_anchors × block_size = 256 × 16 = 4096 draft positions (+ full 151,936-vocab head). Per-microstep compute fits ≈ 53 ms + 0.73 ms × anchors, so ~78% of compute at 256 anchors is the anchor expansion — one "sample" is ~45 TFLOP (fwd+bwd), ≈9× a normal 1.1 B forward. This is by design (denser training signal), not inefficiency, and it runs at a healthy ~44 % MFU (measured, bench_domino_mfu.py) — the trainer is compute-bound, not stalled. num_anchors is not a throughput lever: reducing it raises sequences/s but lowers training-signal/s (anchor-updates/s), so it's a quality/data-efficiency setting only (the launcher keeps it fixed at 256).

MFU / FLOPs (both sides ≈ 43–44 %, compute-bound)

• Trainer ≈ 44 % MFU — measured isolated (single GPU, no data-path/comm, FLOP-counted): ~430 TFLOP/s, ~45 TFLOP/sample, per-sample time flat across batch (compute-throughput-bound). Supersedes an earlier "~14 %" note that came from a cuda.synchronize-distorted timing.
• Inference (target prefill) ≈ 43 % MFU — estimated from the server's ~27 k tok/s prefill at ~550-tok prompts (2·params·tokens, 8 B target).

Remaining ceilings (code, not config)

1. Supply ≈ 50/s is PREFILL-COMPUTE-BOUND, not sink-bound. At ≥8 producer workers the server GPU prefills ~100 % of the time at ~27 k tok/s (~43 % MFU); the mooncake sink thread keeps up (~55/s) and is only the next wall. So an async/parallel sink is a second-order lever, not the primary fix. Real supply levers: higher prefill MFU (CUDA graphs / bigger prefill batches / lighter aux-hidden capture copy, ~+30 %); fewer aux layers (a training change); or a 2nd inference GPU (the seesaw — worse on 8 GPUs).
2. Trainer demand ≈ 50/s at ~44 % MFU. Since the system is supply-bound, raising trainer MFU (CUDA-graphing the anchor-block kernels, fusion, reduced-vocab head) does not raise system throughput today — only matters once supply is lifted or to cut trainer GPU cost.

Notes for reviewers

• The perf commit is split from the instrumentation commit; both are default-preserving (every knob is off unless its env var is set).
• DISAGG_TOTAL_STEPS must be ceil(NUM_EPOCHS * num_filtered_prompts / (TRAIN_DP * BATCH_SIZE * ACCUM)) — too small decays the LR to ~0 early (documented inline).
• With --num-epochs > 1 the producer re-captures the pool each epoch.
• A GPU-cat optimization for the capture sink lives in the pod's installed SGLang (patches/sglang/.../spec-capture.patch territory); note the sink is not the current supply ceiling (prefill compute is), so sink work is secondary.
• No credentials committed; W&B/HF auth via env at launch.

🤖 Generated with Claude Code

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!