Re-enable ft-short FT e2e tests in CI

[fzyzcjy]

Re-enable the 11 ft-short FT e2e tests (deterministic / no_failure / with_failure × {pp2, tp2_ep2, dp4_cp2} + the three real_rollout variants). They were opted out on the 2026-07-08 :dev image because the FT dependency PRs had not yet merged; those are now merged, and the GPU CI stages reinstall sglang/megatron fresh at test time (skip_dependency_install=false), so the opt-out is obsolete. Add run-ci-ft-short to run them; ft-long (soak) stays disabled.

What this fixes

Reproduced on the faithful CI env (:dev image + sglang-miles + miles-main reinstalled fresh, via rcli / #1461 py-spy dumper on the H100 CI itself):

1. Weight-checksum consumer (two latent bugs the disabled tests masked, both on the real_rollout per-rollout checksum path):

   • Nesting: RolloutManager.check_weights returns the single updatable server's result (server_group -> engine_body; pinned by test_check_weights_targets_only_updatable_model), but flatten_inference_engine_checksums iterated one level too deep → AttributeError: 'str' object has no attribute 'get'. Flatten the actual shape.
   • Per-role parallelism_info: sglang's /weights_checker now returns each rank's parallelism_info as a per-role list (target + draft) → TypeError: list indices must be integers. Collapse to the single shared GPU rank. Unit tests + test_group mock updated to the real schema, with regression coverage.

2. est_time for the 8 non-real-rollout tests → 2400s. The 06-13 phase_a/phase_b unification roughly doubled their cost; the old est_time (from 06-12 measurements) under-budgeted them so run_suite's max(1800, est*1.25) per-file timeout killed them mid dump-comparison. Raising est_time only lifts the timeout ceiling.

3. Start the FT-test ray cluster with actor + rollout GPUs (the real_rollout "hang"). When MILES_SCRIPT_EXTERNAL_RAY is unset (CI), execute_train boots the ray head with --num-gpus <num_gpus_per_node>, and run_training passed only mode.train_gpus_per_node (the actor GPUs). The disaggregated real_rollout modes put the sglang engines on separate GPUs (actor 4 + rollout 4 = 8), so ray came up with just 4 GPUs and the 8-GPU placement group could never schedule (scheduling_state: NO_RESOURCES) → train.py hung ~30 min at ray.get(pg.ready()) until the timeout. Diagnosed on the H100 CI via ray's cluster_resources() / placement_group_table() (the machine has 8 physical GPUs; ray was just started with 4). Fix: pass mode.total_node_gpus (train + rollout), matching what scenario_realistic_gsm8k already does. rcli passed because MILES_SCRIPT_EXTERNAL_RAY=1 there uses an external 8-GPU cluster.

4. Clean up FT e2e dumps after each test. The comparison scenarios write dense per-param value/grad dumps under /node_public/dumps/<test>, which on CI is the runner's container-writable layer; run_pipeline never removed a test's dump root, so across a shard's tests they accumulated and filled the runner disk (No space left on device). Now the per-test dump root is removed in a finally once its comparison is done.

5. expandable_segments for the real_rollout train actor (H100 OOM). The disaggregated real_rollout modes put the actor on only train_gpus_per_node GPUs (4) with no tensor/pipeline/expert sharding, so each actor GPU holds the full unsharded 30B model plus the deterministic dumper's value/grad tensors, and dp2 doubles the per-dp-group batch vs the 8-GPU dp4_cp2. On an 80 GB H100 the true peak fits (~71 GiB) but the caching allocator fragments (~11 GiB reserved-but-unallocated against a 10.5 GiB request on the MegatronTrainRayActor) and spuriously OOMs mid-run — surfacing downstream as a detokenizer health-check timeout and "all cells failed". Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True for has_real_rollout modes, scoped to the train actor via --train-env-vars (matching the precedent in tests/e2e/fsdp/test_qwen3_0.6B_megatron_fsdp_align.py), leaving the sglang engine's allocator untouched. Safe: the OOM is purely on the actor (sglang runs on separate GPUs), no --offload-train, so it cannot collide with torch_memory_saver; determinism is unaffected. (H200 never OOMed thanks to its larger memory, which is why rcli always passed.)

6. Do not disable sglang cuda graph in real_rollout. Dropped --sglang-disable-cuda-graph so the real_rollout tests exercise the sglang engine with cuda graphs enabled, matching real usage.

7. est_time for the 30B real_rollout tests. deterministic_dp2_cp2_real_rollout runs the same 2-phase deterministic compare as deterministic_dp2_cp2_pp2 (~2008 s on H100) plus real sglang generation, but est_time=1500 capped the per-file timeout at 1875 s. Raise it to 3600 (timeout 4500 s) and no_failure_dp2_cp2_real_rollout 1200→2400 so real rollout generation has headroom; the dense 0.6B real_rollout (passed at 1176 s) is unchanged.

Validation

On rcli (H200, faithful deps) all 11 ft-short tests pass end-to-end. On H100 CI the last full run was 10/11: one shard 5/5 green (incl. with_failure_..._real_rollout_dense), the other 4/6 with the only genuine failure being the deterministic_dp2_cp2_real_rollout actor OOM described in fix 5 (the 6th test never ran because run_suite stops at the first failure on regular PRs). Fixes 5 and 6 target exactly that OOM and the real_rollout timeout budget. The OOM only reproduces on 80 GB H100 (H200 has the headroom), so those two fixes are validated by the memory arithmetic + existing miles-wide expandable_segments precedent rather than a local repro.

ci-sglang-pr: sglang-miles
ci-megatron-pr: miles-main

[gemini-code-assist]

Code Review

This pull request re-enables several fault-tolerance end-to-end tests by removing the "disabled" parameter. It also updates the weight checking logic in rollout_manager.py to return a list, and refactors checksum utility functions to handle multi-role parallelism information (e.g., target and draft roles) sharing a single GPU rank. Feedback was provided on checksum_utils.py to ensure backward compatibility by checking if "parallelism_info" is a dictionary before treating it as a list of dictionaries.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To ensure backward compatibility and robust handling of different parallelism_info formats (e.g., if older versions or other components return a single dictionary instead of a list of dictionaries), we should check if parallelism_info is a dictionary before attempting to iterate over it as a list. This prevents a TypeError when iterating over a dictionary's keys.

Suggested change

	def _gpu_rank(rank_info: dict[str, Any]) -> int:
	parallelism_info = rank_info["parallelism_info"]
	gpu_ranks = {role_info["rank"] for role_info in parallelism_info}
	assert len(gpu_ranks) == 1, f"expected one GPU rank across roles, got {gpu_ranks}: {rank_info!r}"
	return next(iter(gpu_ranks))
	def _gpu_rank(rank_info: dict[str, Any]) -> int:
	parallelism_info = rank_info["parallelism_info"]
	if isinstance(parallelism_info, dict):
	return parallelism_info["rank"]
	gpu_ranks = {role_info["rank"] for role_info in parallelism_info}
	assert len(gpu_ranks) == 1, f"expected one GPU rank across roles, got {gpu_ranks}: {rank_info!r}"
	return next(iter(gpu_ranks))

[fzyzcjy]

(rm label to avoid the new push triggering expensive ci)