Skip to content

fix(ci): separate image and nightly FT tag scopes #1632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guapisolo merged 16 commits into from
Jul 12, 2026
Merged

fix(ci): separate image and nightly FT tag scopes #1632

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4cb0877
Re-enable ft-short FT e2e tests in CI
fzyzcjy Jul 10, 2026
6d5a122
Fix inference-engine weight-checksum consumer
fzyzcjy Jul 10, 2026
4349c47
Raise ft-short non-real-rollout est_time to 2400s
fzyzcjy Jul 10, 2026
068bc69
Start the FT-test ray cluster with actor + rollout GPUs
fzyzcjy Jul 10, 2026
297efde
Clean up FT e2e dumps after each test to avoid filling CI disk
fzyzcjy Jul 10, 2026
10cfce3
Use expandable_segments for FT real_rollout actor to avoid H100 OOM
fzyzcjy Jul 10, 2026
f4cd575
Raise est_time for the 30B real_rollout FT tests
fzyzcjy Jul 10, 2026
731dc5a
Do not disable sglang cuda graph in FT real_rollout tests
fzyzcjy Jul 10, 2026
d901760
Scope real_rollout expandable_segments to the train actor via --train…
fzyzcjy Jul 10, 2026
0930f59
Run fault-tolerance suites on full-node H200 CI
guapisolo Jul 10, 2026
27d9811
Merge branch 'main' into ft/reenable-ft-short
guapisolo Jul 10, 2026
ae5aaec
Merge branch 'main' into ft/reenable-ft-short
guapisolo Jul 11, 2026
62b211b
tune FT CI estimates from passed runtime data
guapisolo Jul 11, 2026
5a350bc
fix(ci): separate image and nightly FT tag scopes
guapisolo Jul 11, 2026
9121497
Merge branch 'main' into fix/ci-tag-scopes
guapisolo Jul 12, 2026
500cffb
test(ft): drop the expandable_segments investigation comment
guapisolo Jul 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions .github/workflows/pr-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled]
schedule:
# Nightly full CI on main: a scheduled run is treated like workflow_dispatch
# (every stage, --match-all-labels). 15:00 UTC = 08:00 PDT / 07:00 PST (cron is UTC, no DST).
# Nightly CI on main runs every enabled tag except ft-long, with fast-fail disabled.
# 15:00 UTC = 08:00 PDT / 07:00 PST (cron is UTC, no DST).
- cron: '0 15 * * *'
workflow_dispatch:
inputs:
Expand Down Expand Up @@ -43,9 +43,12 @@ concurrency:
# `--labels`. run_suite.py strips the `run-ci-` prefix internally and ignores
# any label that does not start with `run-ci-` (see tests/ci/run_suite.py
# `strip_run_ci_prefix`). For `workflow_dispatch`, no PR labels exist, so the
# `--labels` list collapses to empty; `--match-all-labels` is then added
# unconditionally to bypass the labels predicate inside run_suite.py and run
# every enabled test in the suite.
# `--labels` list collapses to empty; `--match-all-labels` is then added to
# bypass the labels predicate inside run_suite.py. Broad scopes can add a final
# `--exclude-labels`: `run-ci-image` excludes ft-short and ft-long, while
# nightly excludes only ft-long. `run-ci-all` and workflow_dispatch pass no
# exclusions and therefore include every enabled tag. When scope labels
# overlap, all > nightly > image.
#
# The `bypass-fastfail` PR label opts a PR out of fast-fail on two levels: GPU
# stages run even when stage-a-cpu fails (cross-stage), and every stage passes
Expand Down Expand Up @@ -97,6 +100,7 @@ jobs:
--auto-partition-id ${{ matrix.partition_id }} --auto-partition-size 4
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}

@yushengsu-thu yushengsu-thu Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ~470-char expression is now copy-pasted 7 times — this PR is itself evidence of the cost (7 near-identical line edits), and a one-stage drift would be invisible until a nightly quietly runs the wrong test set. The precedence logic here is also the riskiest artifact of the PR, and it's only covered by actionlint (syntax, not behavior) — the new unit tests verify filter_tests given already-correct flags, not that the workflow emits those flags per trigger.

Suggested follow-up: compute a scope-flags output once in a preliminary job (resolve-ci-image is a natural home) and reference it per stage — or better, pass the raw event name + labels into run_suite.py and compute the scope in Python where it's unit-testable.

(nit: clause 2's github.event_name != 'schedule' && !contains(..., 'nightly') guards are unreachable — any case where they'd matter is already consumed by clause 1 or killed by !contains(run-ci-all) — so they could be dropped to lighten the line.)

${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -111,6 +115,7 @@ jobs:
python tests/ci/run_suite.py --hw cpu --suite stage-b-cpu
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -130,6 +135,7 @@ jobs:
python tests/ci/run_suite.py --hw cuda --suite stage-b-2-gpu-h200
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -154,6 +160,7 @@ jobs:
--auto-partition-id ${{ matrix.partition_id }} --auto-partition-size 2
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -178,6 +185,7 @@ jobs:
--auto-partition-id ${{ matrix.partition_id }} --auto-partition-size 2
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -202,6 +210,7 @@ jobs:
--auto-partition-id ${{ matrix.partition_id }} --auto-partition-size 3
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit

Expand All @@ -226,5 +235,6 @@ jobs:
--auto-partition-id ${{ matrix.partition_id }} --auto-partition-size 2
--labels ${{ join(github.event.pull_request.labels.*.name, ' ') }}
${{ (github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-ci-image') || contains(github.event.pull_request.labels.*.name, 'run-ci-all') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--match-all-labels' || '' }}
${{ ((github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly')) && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && '--exclude-labels ft-long') || (contains(github.event.pull_request.labels.*.name, 'run-ci-image') && !contains(github.event.pull_request.labels.*.name, 'run-ci-all') && github.event_name != 'schedule' && !contains(github.event.pull_request.labels.*.name, 'nightly') && '--exclude-labels ft-short ft-long') || '' }}
${{ (contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') || (github.event_name == 'schedule' || contains(github.event.pull_request.labels.*.name, 'nightly'))) && '--continue-on-error' || '' }}
secrets: inherit
6 changes: 4 additions & 2 deletions docs/ci/00-stage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,11 @@ Stage names follow `stage-<tier>-<gpus>-<hw>` (or `stage-<tier>-<hw>` for CPU, e

## What each stage does

**Image resolution (`resolve-ci-image`).** Before the GPU stages, a small `ubuntu-latest` job resolves the container image: it reads `ci-image-tag:` from the PR description (or the `ci_image_tag` dispatch input), defaults to `dev`, validates it is a bare tag, and outputs `radixark/miles:<tag>`. Every GPU stage uses this as its `container_image`. Distinct from this, the **`run-ci-image` label** (alongside `run-ci-all`, any `workflow_dispatch`, and the nightly `schedule` run on `main`) makes each stage add `--match-all-labels`, running the full suite regardless of per-test labels — this is how you validate a PR that bumps the image.
**Image resolution (`resolve-ci-image`).** Before the GPU stages, a small `ubuntu-latest` job resolves the container image: it reads `ci-image-tag:` from the PR description (or the `ci_image_tag` dispatch input), defaults to `dev`, validates it is a bare tag, and outputs `radixark/miles:<tag>`. Every GPU stage uses this as its `container_image`. Distinct from this, the **`run-ci-image` label** makes each stage add `--match-all-labels --exclude-labels ft-short ft-long`, running every enabled non-FT tag — this is how you validate a PR that bumps the image without scheduling FT tests.

A **nightly** run is the full suite with fast-fail off, triggered either by the nightly `schedule` cron on `main` or by a PR carrying the `nightly` label. Concretely it adds `--match-all-labels` and turns off fast-fail like the `bypass-fastfail` label.
A **nightly** run includes `ft-short` but excludes `ft-long`, with fast-fail off. It is triggered either by the nightly `schedule` cron on `main` or by a PR carrying the `nightly` label; concretely it adds `--match-all-labels --exclude-labels ft-long` and turns off fast-fail like the `bypass-fastfail` label.

`run-ci-all` and a manual `workflow_dispatch` add `--match-all-labels` without exclusions, so every enabled tag is included. If scope labels overlap, the precedence is `run-ci-all` > nightly > `run-ci-image`.

**Dependencies / gating.** The job graph is `resolve-ci-image` → `stage-a-cpu` → all GPU stages (in parallel). GPU stages require `resolve-ci-image` to succeed; by default they also require `stage-a-cpu` to succeed, so a CPU-test failure short-circuits the expensive GPU fleet. The `bypass-fastfail` PR label relaxes only the `stage-a-cpu` failure gate and passes `--continue-on-error` to each stage; it does not bypass `resolve-ci-image`. `stage-b-cpu` has no dependency and runs alongside `stage-a-cpu`, outside the GPU gating path.

Expand Down
20 changes: 11 additions & 9 deletions docs/ci/01-label.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Labels
description: The three kinds of CI label — domain labels that gate tests, meta-labels that run everything, and bypass-fastfail.
description: The three kinds of CI label — domain labels that gate tests, scope labels that broaden selection, and bypass-fastfail.
---

# Labels
Expand All @@ -10,10 +10,12 @@ A label is a GitHub PR label that changes what CI runs or how it fails. Three ki
| Kind | Example | Effect |
|---|---|---|
| Domain label | `run-ci-megatron` | selects which tests run |
| Meta-label | `run-ci-image`, `run-ci-all` | run the full suite, ignoring domain labels |
| Scope label | `run-ci-image` | run every enabled non-FT tag |
| Scope label | `nightly` | run every enabled tag except `ft-long`, with fast-fail disabled |
| Scope label | `run-ci-all` | run every enabled tag |
| Behavior label | `bypass-fastfail` | opt out of fast-fail; one run surfaces every failure |

Only domain labels are declared by tests; the other two are workflow switches in `pr-test.yml`.
Only domain labels are declared by tests; scope and behavior labels are workflow switches in `pr-test.yml`.

## Domain labels: `register_*_ci(labels=...)` ↔ `run-ci-<x>`

Expand All @@ -34,13 +36,13 @@ Domain labels live in `tests/ci/labels.py` (`KNOWN_LABELS`); a `labels=[...]` va

To add one: add the entry to `KNOWN_LABELS`, then create the matching `run-ci-<key>` label on the PR. No workflow edit needed.

## Meta-labels: run everything
## Broad CI scopes

`run-ci-image` and `run-ci-all` are the same switch: both add `--match-all-labels` to run every enabled test in the suite, ignoring domain labels. Neither is in `KNOWN_LABELS`.
`run-ci-image` adds `--match-all-labels --exclude-labels ft-short ft-long`, so image validation ignores ordinary domain gating but does not schedule any FT tests.

A manual `workflow_dispatch` run — and the nightly `schedule` run on `main` — gets `--match-all-labels` too, because neither has PR labels to filter on.
A nightly run — the `schedule` on `main` or a PR carrying `nightly` — adds `--match-all-labels --exclude-labels ft-long`, so it includes `ft-short` but omits the FT soak tests. It also disables fast-fail on both levels.

The `nightly` label runs a PR as a nightly: `--match-all-labels` plus fast-fail bypass on both levels.
`run-ci-all` and a manual `workflow_dispatch` add `--match-all-labels` without exclusions, so every enabled tag is included. If scope labels overlap, the precedence is `run-ci-all` > nightly > `run-ci-image`.

## Registration and scan scope

Expand All @@ -65,6 +67,6 @@ The `bypass-fastfail` PR label turns both off so one run surfaces every failure:
- Cross-stage: each GPU stage's check becomes `(needs.stage-a-cpu.result == 'success' || (needs.stage-a-cpu.result == 'failure' && contains(..., 'bypass-fastfail')))`, so GPU stages run even after `stage-a-cpu` fails.
- Within-stage: each stage adds `--continue-on-error` (drops `pytest -x`; sets `continue_on_error=True` for CUDA). The stage still ends red — it changes coverage, not the verdict.

A nightly run — the `schedule` cron, or a PR carrying the `nightly` label — bypasses fast-fail on both levels: the same cross-stage `if` and per-stage `--continue-on-error` match `github.event_name == 'schedule' || contains(..., 'nightly')`, because a nightly is meant to exercise every test and surface every failure (one datapoint per test), not stop at the first.
A nightly run — the `schedule` cron, or a PR carrying the `nightly` label — bypasses fast-fail on both levels: the same cross-stage `if` and per-stage `--continue-on-error` match `github.event_name == 'schedule' || contains(..., 'nightly')`, because a nightly is meant to exercise every enabled test except `ft-long` and surface every failure (one datapoint per test), not stop at the first.

Like the meta-labels, `bypass-fastfail` is matched directly in `pr-test.yml` and is not in `KNOWN_LABELS`.
Like the scope labels, `bypass-fastfail` is matched directly in `pr-test.yml` and is not in `KNOWN_LABELS`.
13 changes: 10 additions & 3 deletions miles/utils/audit_utils/checksum_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def flatten_inference_engine_checksums(check_weights_result: Any) -> list[Infere


def _flatten_to_inference_engine_bodies(check_weights_result: Any) -> list[Any]:
return [engine_body for server in check_weights_result for server_group in server for engine_body in server_group]
return [engine_body for server_group in check_weights_result for engine_body in server_group]


def _merge_inference_engine_ranks(engine_body: dict[str, Any]) -> InferenceEngineChecksums:
Expand All @@ -24,11 +24,18 @@ def _merge_inference_engine_ranks(engine_body: dict[str, Any]) -> InferenceEngin
ranks: list[dict[str, Any]] = engine_body.get("ranks", []) or []
assert ranks, f"check_weights engine body has no ranks: {engine_body!r}"

ranks_sorted = sorted(ranks, key=lambda r: r["parallelism_info"]["rank"])
ranks_sorted = sorted(ranks, key=_gpu_rank)

merged: InferenceEngineChecksums = {}
for rank_info in ranks_sorted:
rank = rank_info["parallelism_info"]["rank"]
rank = _gpu_rank(rank_info)
for name, value in rank_info["checksums"].items():
merged[f"rank{rank}/{name}"] = value
return merged


def _gpu_rank(rank_info: dict[str, Any]) -> int:
parallelism_info = rank_info["parallelism_info"]
gpu_ranks = {role_info["rank"] for role_info in parallelism_info}
assert len(gpu_ranks) == 1, f"expected one GPU rank across roles, got {gpu_ranks}: {rank_info!r}"
return next(iter(gpu_ranks))
6 changes: 3 additions & 3 deletions tests/ci/labels.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@
The workflow does not need editing -- the generic stage job filters tests
by labels at runtime.

The meta-labels `run-ci-image` / `run-ci-all` are intentionally NOT listed
here: they bypass the per-test labels filter and run the full suite via the
`--match-all-labels` flag (handled in run_suite.py).
The workflow-only scope labels are intentionally NOT listed here. `run-ci-all`
uses `--match-all-labels` without exclusions; `run-ci-image` and nightly pair
it with scope-specific `--exclude-labels` values (handled in run_suite.py).
"""

KNOWN_LABELS: dict[str, str] = {
Expand Down
Loading
Loading