[DataFlow runtime] Domino disaggregated training (example + launcher + strategy params + capture/tracker fixes)#660
Open
maocheng23 wants to merge 14 commits into
Open
[DataFlow runtime] Domino disaggregated training (example + launcher + strategy params + capture/tracker fixes)#660maocheng23 wants to merge 14 commits into
maocheng23 wants to merge 14 commits into
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
2. del some test code to increase speed 3. refactor code without influence training
jiapingW
requested review from
FlamingoPg,
FrankLeeeee,
shuaills and
sleepcoo
as code owners
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
…--num-epochs
- examples/disagg/run_qwen3_8b_domino_disagg_1srv_dp7.sh: single inference
server (1 GPU) + DP=7 trainer (7 GPUs) variant of the multiserver launcher.
The domino disagg workload is trainer-bound and the capture server is
heavily overprovisioned in the 2-server layout (~6% util); this rebalances
to 1 server + 7 trainers, which raises both throughput and server util.
- examples/disagg/run_disagg_domino.py: the producer now replicates the prompt
pool by --num-epochs before streaming. The online streaming channel is
consume-once, so previously the pool streamed once regardless of
--num-epochs; multi-epoch training now works as expected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
perf(domino-disagg): env-tunable data-plane knobs + findings (28.8->50.1 samples/s)
Config-only tuning that ~doubles single-node online domino-disagg throughput,
with util/SM/mem healthy on all 8 GPUs. All knobs are off by default.
Launchers (1srv+dp7 and multiserver):
- DISAGG_SERVER_URLS now overridable; repeating one URL N times spawns N
concurrent producer workers with disjoint leases against one server, which
breaks the single-producer supply ceiling (the blocking HTTP prefill releases
the GIL so workers genuinely overlap).
- SERVER_MEM_FRACTION knob (default 0.85): 0.5 right-sizes the capture-only
server from ~126 GB to ~78 GB with no perf cost.
- NUM_ANCHORS / BLOCK_SIZE knobs on the 1srv launcher (quality<->throughput).
- multiserver launcher now honors BATCH_SIZE / ACCUM (were hardcoded to 2 / none).
FeatureDataLoader:
- LOADER_PREFETCH=N: background-thread batch prefetch so the training step never
pays mooncake get/collate latency inline (ack still after the trainer consumes).
- CLONE_ON_FETCH=0: skip the redundant defensive clone on the mooncake zero-copy
path (get() already allocates a fresh tensor).
See examples/disagg/PERF_FINDINGS.md for the full analysis: the pipeline is a
supply/demand seesaw (1:7 beats 2:6), bigger batch does not help (compute-latency
bound, MFU ~14%), and ~78% of per-sample trainer compute is the num_anchors=256
draft expansion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
perf(domino-disagg): env-gated profiling instrumentation for the data path
Default-off timers that produced the PERF_FINDINGS analysis. Each is gated on an
env var and adds nothing when unset.
- PROFILE_PRODUCER=N -> [prod] (backpressure-park / run_once / publish + in_flight)
in launch.py, and [prod-http] (build / HTTP RTT / parse)
in the server-capture adapter.
- PROFILE_LOADER=N -> [loader rK] queue-wait / get / clone / release / gc.
- PROFILE_STORE=N -> [store rK] get/put GB/s, rm_ok/rm_fail(-706), pending depth.
- PROFILE_DISTRIB=sec -> [dist] commit / count / inbox-publish + per-rank lag.
- PROFILE_STEPS=N -> [profile] data-wait vs compute, [profile2] fwd/bwd/opt + padded len.
- PROFILE_TORCH=N -> rank-0 torch.profiler top-ops + chrome trace.
- FSDP_SHARDING -> override the FSDP sharding strategy (e.g. NO_SHARD).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
revert(domino-disagg): drop NUM_ANCHORS/BLOCK_SIZE knob; num_anchors is not a throughput lever
Reducing num_anchors raises sequences/s but LOWERS training-signal/s
(anchor-updates/s): each anchor is a supervised target, and a fixed ~53ms/
microstep per-sequence overhead means fewer anchors amortize that cost over less
signal (256 -> ~2130 vs 64 -> ~1280 anchor-updates/s). It also forces ~4x more
captured sequences for equal signal, loading the supply bottleneck. So it is a
quality/data-efficiency setting, not a speed knob — revert the launcher override
back to the fixed --num-anchors 256 / --block-size 16 and correct PERF_FINDINGS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs(domino-disagg): measured MFU/FLOPs + correct the inference-ceiling analysis
Adds bench_domino_mfu.py (isolated single-GPU, FLOP-counted, CUDA-event-timed
trainer benchmark) and folds its results into PERF_FINDINGS.md:
- Trainer MFU ≈ 44% (measured, b2/b4/b8; ~45 TFLOP/sample fwd+bwd; per-sample
time flat across batch → compute-bound, not stalled). Supersedes the earlier
"MFU ~14% / occupied-but-stalled" note, which came from a cuda.synchronize-
distorted timing and was wrong.
- Inference (target prefill) ≈ 43% MFU, estimated from ~27k tok/s at ~550-tok
prompts (2·params·tokens for the 8B target).
- Corrects the "supply ceiling = capture sink" framing: at >=8 producer workers
the server is PREFILL-COMPUTE-BOUND (GPU prefills ~100% duty at ~27k tok/s);
the sink thread keeps up (~55/s) and is only the *next* wall. So an async/
parallel sink is a second-order lever, not the primary fix — the real supply
levers are higher prefill MFU (CUDA graphs / bigger prefill batches / lighter
aux-hidden capture copy), fewer aux layers (training change), or more inference
GPUs (the seesaw). Trainer-MFU work does not raise system throughput while
supply-bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix lint
fix lint
[DataFlow runtime] Domino disagg: 1-server + DP=7 launcher + working --num-epochs
Match stock train_dflash.py's logging so the disagg (online/DP) curves overlay the in-process trainer: - controller._result: all_reduce(loss)/world and all_reduce(acc)/world before logging (was: single rank's local-batch value -> ~sqrt(world) noisier, spiky). - _dflash_family_disagg.py logger: log accuracy under train/accuracy (was train/acc) and add train/lr from the optimizer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eps) Adds the DFlash counterpart of the Domino 1srv+DP7 launcher so DFlash disagg training is reproducible from the repo: - run_disagg_dflash.py entry, qwen3-8b-dflash.json, drops domino-only --lambda-base-* args, NUM_ANCHORS/WANDB_NAME overridable. - auto-derives DISAGG_TOTAL_STEPS = ceil(NUM_EPOCHS * prompts / (TRAIN_DP * BATCH_SIZE)) from the dataset when unset, so only --num-epochs is needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
Curve here, also checked the accept length, 5.9 main and 5.75 refactored (acceptable diff?)
|
Add a reusable two-node Qwen3-8B DFlash launcher with rank-specific inference and training roles, cross-node Mooncake coordination, allocated-GPU UUID handling, and symmetric failure cleanup. Document the RCLI launch flow and apply the formatter changes required by PR 660's lint job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end Domino disaggregated training on top of the DataFlow runtime integration branch (
dataflow-up-16-zerocopy), plus a few robustness fixes surfaced along the way. Cleanly stacked: 2 commits, +718 / −71 across 12 files, no rebase needed.Commits
6cd68297579d8eWhat's in it
New Domino disagg entrypoints
examples/disagg/run_disagg_domino.py(+298) — end-to-end domino disaggregated training exampleexamples/.../run_qwen3_8b_domino_disagg_multiserver.sh(+216) — multiserver launcher; plus edits to the qwen3.6-27b dflash disagg launcherconfigs/qwen3-8b-domino.json— addsemb_dim: 256Parameterized domino strategy
training/strategies/registry.py— domino strategy now acceptslambda_start/decay_ratiovia a new_make_domino_strategybuildertraining/trainer.py+launch.py— thread a newstrategy_kwargsdict throughTrainer,_assemble_trainer, andbuild_disagg_online_consumerso strategy construction can be parameterizedregistry.py— online collate fix: bothdflashanddominonow use_dflash_offline_collatefor the online path instead ofconcat_collate(behavioral change — worth a close look)Robustness fixes
inference/adapters/server_capture.py(+51) — handle batch/list-wrapped capture results from the spec-capture server; select this task's row bysample_id, with explicit errors on ambiguous or non-object results (_flatten_list_wrappers,_capture_result_for_task)tracker.py— treat a localwandb/output dir mis-imported as a namespace package as "wandb unavailable"; raise a clear error if--report-to wandbis used without the real clientruntime/data_plane/mooncake_store.py— include the underlying exception + CUDA-runtime hint in the mooncake load-failure messageTests
tests/test_runtime/test_server_capture.py(+19) — batch-result selectiontests/test_runtime/test_strategy_registry.py(+22) — domino strategy params / online collateReviewer notes
fix bugs,[bug]) — worth squashing/rewording before merge.make_online_collatechange fromconcat_collate→_dflash_offline_collateaffects the online path for both dflash and domino, not just domino — confirm this is intended for dflash too.tests/test_runtimeshould validate.Base
Targets
dataflow-up-16-zerocopy(the DataFlow runtime integration branch), consistent with how the rest of the stack lands. The merge-base is exactly that branch's current tip (#657 o3/o4 rollup).🤖 Generated with Claude Code