This document explains the diff-driven test-selection system that lets some GPU CI workflows run only the tests a PR could affect, instead of the whole suite, while still satisfying Required status checks.
It currently drives modal-torch-latest (which runs tests/unit/v1/ on a
Modal GPU or its bounded AWS capacity fallback), but is built to drive more
workflows from one config — see Adding a workflow.
- TL;DR
- Why
- Moving parts
- How a decision is made
- Driving it (as a contributor)
- Changing it (as a maintainer)
- Security model
- Failure modes & guarantees
- FAQ / troubleshooting
- On a PR,
ci/tests_fetcher.pydiffs your branch against the base branch, traces the import graph from your changed files to the impacted tests, and writes the list toci/.test_selection/test_list.txt. A nightly scheduled run always executes the full suite. - The trusted controller validates that list, then fetches, installs, and tests
the exact candidate SHA inside a no-secret Modal Sandbox. If the fixed
L40S:2Sandbox cannot start within its acquisition window, the controller tries a fixed, ordered pool of AWSg7.12xlargeSpot locations. The backend runs only for opted-in merge queue entries, the nightly scheduled full run, and manual runs — a plain PR event never spends GPU quota. - A manual
infrastructure_smokedispatch skips selection and pytest entirely. It starts one short OCIL40S:2Sandbox check and three independent regional AWS jobs in parallel. Each AWS job directly selects one region, launches a G7.12 Spot instance, reaches it through SSM, verifies two host and container GPUs, stores a checksum-verified proof in the private regional log bucket, and terminates its own run-tagged instance. The matrix usesfail-fast: false, so one regional result cannot cancel the other regions. - It is fail-safe: anything it can't reason about safely → run the full suite. It never silently runs fewer tests than reality.
- Preview locally:
python ci/tests_fetcher.py --base origin/master # what would CI run? python ci/tests_fetcher.py --base origin/master --explain # ...and why?
- Force full test selection with
[test all](or[no filter]). Queue GPU execution also requires the exactneeds-gpu-testlabel.
modal-torch-latest is a Required check, so it must report a status on every
PR — which means we can't use GitHub's path filters (on.<event>.paths) to skip
it, because a skipped Required job blocks merges. Instead, the job always runs but
selects what to test: a docs-only PR resolves to "no impacted tests" and exits
fast (still green), while a PR touching a leaf module runs only the tests that
import it.
The design is a small, self-contained take on HuggingFace transformers'
utils/tests_fetcher.py.
PRs and merge queue entries whose entire diff is under docs/**, under
blogs/**, or matches **/*.md skip CPU/GPU test suites, Python install smoke,
and native-op precompilation. This includes all documentation and blog assets
and configuration files, plus Markdown at any directory depth. Formatting, DCO,
and documentation checks are not disabled.
The active python, cpu-torch-latest, and nv-pre-compile-ops workflows use
the reusable check-paths.yml gate. Disabled legacy workflows are left unchanged.
The shared gate also skips root version.txt changes on both PRs and merge queue
entries, including diffs mixing version and documentation changes. Any other
changed path still requires CI unless explicitly excluded for that workflow.
The Modal selector already selects no tests for a version-only diff.
The optional mps-torch-latest and no-torch workflows rely on their PR trigger
paths instead, which exclude Markdown. A README under a watched source directory
cannot start these workflows merely because the PR also changes unrelated code.
Their gate job is commented out because it could never skip: every PR run they
start includes a non-documentation change. Re-enable it if either workflow must
drop its path filters (to become a required check or run in the merge queue) or
if is_docs_path() starts covering more than Markdown.
ci/check_paths.py is the single definition of documentation (is_docs_path()).
The gate diffs the checked-out commit against the exact commit it was built on: a
PR's merge commit against its first parent, the base-branch commit GitHub merged
the PR into, and a merge queue entry against merge_group.base_sha. It does not
use the event's pull_request.base.sha, which can be days older and would add the
base branch's own changes, so docs-only PRs behind master would not skip. The gate
runs the base branch's copy of the script, read from that same commit.
A PR that edits the script is judged by the base copy, which runs its tests because
the script is not documentation; the edits apply after merge. This guards against
mistakes, not a malicious PR, which controls the workflow files on pull_request
events. Renames count both the old and new paths: moving module.py to
docs/module.py is a code deletion, not a documentation-only change. An empty
diff runs tests, and detection failures, including a base branch without the
script, emit a warning and fall back to running tests. If the gate job itself
fails or exceeds its 10-minute limit, the required jobs fail instead of skipping.
Callers may pass pr-ignore (files, or directories ending in /, such as
deepspeed/inference/v2/) to skip more paths on pull requests only; merge queue
entries use only the shared documentation and root version.txt exclusions.
Preview the decision with
python ci/check_paths.py --base "$(git merge-base origin/master HEAD)".
Scheduled, manual, and push runs retain their existing behavior. Required
workflows still start and report their statuses; only their heavy jobs or steps
skip. The Modal selector imports the same is_docs_path() and drops documentation
paths before applying its run-all path globs and import graph, so documentation
never widens a selection; explicit [test all] / [no filter] overrides still win.
Root README.md is package metadata, so the formatting workflow always runs
python scripts/check-readme.py to check that it exists and is valid UTF-8
without installing DeepSpeed or PyTorch.
| File | Role |
|---|---|
.github/workflows/modal-torch-latest.yml |
The workflow: a no-secret collect-tests job gating a trusted deploy controller job. |
ci/tests_fetcher.py |
The selector. AST-parses the repo, builds an import graph, decides all / subset / none, writes the test-list file, emits a job summary. |
ci/check_paths.py |
Defines documentation paths (is_docs_path()) for the selector and the check-paths.yml CPU/GPU workflow gate. |
ci/test_tests_fetcher.py |
Self-tests for the selector and ci/check_paths.py (pure stdlib; run in collect-tests and, on the PR's own code, in the formatting workflow). |
ci/torch_latest.py |
Pure-stdlib metadata/selection validation plus the trusted Modal-first, AWS-capacity-fallback controller. |
ci/.test_selection/test_list.txt |
The hand-off artifact (one pytest target per line). Git-ignored. |
pull_request_target / merge_group / schedule / ordinary dispatch
│
┌─────────────┴─────────────┐
│ collect-tests │ (no secrets)
│ checkout exact base SHA │
│ public-fetch exact head SHA│
│ self-test the fetcher │
│ parse candidate as data │
│ → mode (all|subset|none) │
│ → validate/upload list │
└─────────────┬──────────────┘
│ needs:
▼
┌────────────────────────────┐
│ deploy │ (trusted controller only)
│ if mode != none │
│ checkout exact base SHA │
│ validate mode/list + SHA │
│ try one L40S:2 Sandbox │
│ no secrets or mounts │
│ startup capacity only: │
│ try G7.12 Spot by region │
│ via GitHub OIDC + SSM │
│ fetch exact head SHA │
│ install + run pytest │
│ terminate chosen backend │
└────────────────────────────┘
A manual dispatch with infrastructure_smoke=true takes a separate path:
workflow_dispatch (infrastructure_smoke=true)
│
┌─────────────────┴─────────────────┐
│ │
short Modal OCI smoke AWS regional matrix
startup + two GPUs fail-fast: false
always terminates east-1 / east-2 / west-2
launch + SSM + two GPUs
container + private proof
per-region termination
That path never creates a test-selection artifact, installs DeepSpeed, or runs pytest. It is an explicit infrastructure diagnostic and is not available to scheduled, pull-request, or merge-queue executions, so it cannot weaken the Required check.
mode controls the selected test scope:
none→deployis skipped. The Required status is still satisfied (a skipped dependent job counts as success here).subset→ an eligibledeployruns pytest on exactly the impacted files.all→ an eligibledeployruns the whole scope (tests/unit/v1).
Independent of mode, deploy is skipped on pull_request_target runs and on
merge-group runs whose entry PR lacks the exact needs-gpu-test label. The
collect-tests summary still previews the selected scope. The backend executes
for opted-in merge_group entries (the merged tree, gating the merge),
schedule, and workflow_dispatch. On merge_group the candidate is the
merge-group commit in the base repository, diffed against the queue's base SHA,
so the selection covers exactly what the entry would introduce. There is
deliberately no push trigger: master receives no direct pushes and the merge
queue evaluates every entry before landing. Cross-entry interaction regressions
are what the nightly full suite exists to catch.
For a manual infrastructure-only check, select infrastructure_smoke when
dispatching the workflow. That skips both collect-tests and deploy; the
Modal and per-region AWS smoke jobs run directly and concurrently instead.
Leaving the input at its default false preserves the ordinary manual full
scope. The AWS smoke path does not depend on Modal's result, so a successful
Modal allocation cannot bypass the regional checks.
A scheduled nightly (cron, see modal-torch-latest.yml) always runs the full
suite. Its outcome feeds nightly-triage.yml:
- Green → the
nightly-last-greentag moves to the tested SHA. The tag anchors the last verified-good master revision (and is exactly the good endpoint a later manual bisect would need), so days that fail for operational reasons (no GPU instance, timeout) simply leave the tag in place. - Infra (neither Modal nor any configured AWS location acquired a GPU) → nothing is concluded; no report, no tag move.
- Timeout (the backend/job time budget ran out) → an issue is opened; a timeout points at an operational problem, not a candidate regression.
- Unknown (no sentinel in the logs) → an issue is opened. A failed run can lack a sentinel for reasons other than a timeout — checkout, setup, or a job kill before the controller could classify itself — so triage reports it as unclassified rather than guessing a class.
- Real test failures → an issue is opened listing the failing test files
and the run. Triage deliberately stops there: an opted-out queue entry may
have introduced a regression that nightly is the first GPU run to expose, or
the failure may be a cross-entry interaction or operational flake. If nightly
regressions turn out to be frequent enough to justify automated bisection,
the
nightly-last-greentag is the good endpoint it would start from.
The controller (ci/torch_latest.py) classifies its own failures for this
routing. It waits up to 10 minutes for Modal Sandbox startup; only that
startup-capacity timeout returns 75 to the workflow without a final sentinel
so that AWS can run. The selected backend prints exactly one final
DS_CI_FAILURE_CLASS=infra|timeout|test sentinel on failure. If no backend runs
the outcome is infra (75); 300 seconds without pytest output is a timeout
(124); setup and test errors are test failures (1). A failed run with no
sentinel routes to the unknown class instead of guessing.
Triage reports (timeout, unknown, test failures) go through ci/nightly_report.sh:
every report carries the nightly-triage label, and a recurring outcome
comments on its still-open issue instead of filing a new one. Dedup keys on
the title, so titles are stable across recurrences (per-night SHAs live in
the body).
Triage checks out the SHA the nightly actually ran rather than current master, so the classifier and the sentinel protocol it parses come from the same revision.
cpu-torch-multi-latest.yml covers the axis the selector does not model:
world_size. cpu-torch-latest advertises a single device, so the per-device gate
in tests/unit/common.py skips every world_size > 1 test there. The nightly sets
LOCAL_SIZE=4 to admit world_size <= 4 and selects only those tests with
DS_MULTIRANK_ONLY=1 (a pytest_collection_modifyitems hook in
tests/conftest.py): one phase for the non-v1 half (120m budget) and one for the
sequential-marked tests (25m), which the parallel phase deselects.
It is deliberately not a pre-merge or merge-queue check: the non-v1 half needs 70-90 minutes on a 4-vCPU runner, against 39 minutes for the longest current pre-merge job, and a wedged rank would hold up the queue.
cpu-multi-triage.yml routes its outcome through the same sentinel protocol and
reporter as the modal nightly, but with its own issue titles and its own
nightly-cpu-multi-last-green tag, so nightly-last-green keeps meaning "the last
revision the full GPU suite verified". Issue #8805 (bf16 check_grad_overflow
semantics) is an example of a regression only this selection can see.
TestSelector.select() in ci/tests_fetcher.py runs these checks in order; the
first that matches wins:
- No base ref (nightly / manual) →
all. - Base ref unresolvable →
all. - No merge-base with the base (e.g. shallow clone, unrelated history) →
all. A diff here would be wrong, so we never narrow on it. - Commit message tag
[test all]/[no filter]anywhere on the branch →all. - Only documentation paths changed (
docs/**,blogs/**, or**/*.md, as defined byis_docs_path()inci/check_paths.py), including deleted paths and both sides of renames →none. Otherwise documentation paths are dropped, so they cannot trigger the steps below. - A changed or deleted file matches a run-all glob (
COMMON_RUN_ALL_GLOBS+ the workflow'sextra_run_all_globs) →all. These are files too central or too dynamic to narrow safely: CI scripts, build system,csrc/,op_builder/,accelerator/, shared fixtures (tests/unit/common.py,tests/conftest.py,pytest.ini), and core runtime hubs (deepspeed/__init__.py,deepspeed/runtime/engine.py,deepspeed/comm/**,deepspeed/accelerator/**, …). - A deleted module is still imported by a surviving file (a dangling import
the graph can't follow) →
all. A clean deletion (importers removed/updated in the same PR) does not trigger this. - Otherwise, narrow via the import graph (below). If nothing is impacted →
none; if everything is →all; else →subset.
- Nodes are Python files under the package roots:
deepspeed/**and theunittest-helper package attests/unit/**. - An edge
A → Bmeans "B imports A". The selector walks backwards from each changed file to every test that (transitively) imports it. - Opaque hub modules (
OPAQUE_MODULES:deepspeed,deepspeed.comm,deepspeed.accelerator): almost every test importsunit.common, which imports these. Their__init__.pyfiles eagerly pull in huge subtrees, so if treated as normal nodes anydeepspeed/**change would fan out to the whole suite. We therefore don't expand their__init__imports; instead, changes to the hubs themselves are caught by the run-all globs in step 6. conftest.pychanges select every test under that conftest's directory.- New test files are selected directly (they have no importers yet).
Some dependencies are wired at runtime (monkey-patching, plugin/registry lookup,
JIT-loaded ops, deepspeed.initialize()-time replace_module injection), so a
test can depend on code it never imports. DYNAMIC_EDGES is a curated map of
changed-file glob → extra test-path globs that patches these blind spots. It is
additive on top of the static graph (and is only consulted if step 6 didn't
already short-circuit to all).
The fetcher is pure stdlib — no DeepSpeed/torch install needed, just git:
# Selection for your branch vs. the base branch
python ci/tests_fetcher.py --base origin/master
cat ci/.test_selection/test_list.txt
# Explain *why* each test was selected (prints import chains)
python ci/tests_fetcher.py --base origin/master --explain--explain output looks like:
deepspeed/shared.py impacts:
tests/unit/v1/test_shared.py <- deepspeed/shared.py
tests/unit/v1/moe/test_moe.py <- tests/unit/v1/moe/test_moe.py <- deepspeed/shared.py
- Force full test selection: include
[test all](or[no filter]) anywhere in a commit message on the branch. Queue GPU execution still needs the exactneeds-gpu-testlabel. - Touch an infra file: changing a non-documentation file that matches a run-all glob runs everything.
- Found a missed test? It's likely a runtime/dynamic dependency the static
graph can't see — add a
DYNAMIC_EDGESentry (see below) and/or report it.
All knobs live in ci/tests_fetcher.py. After any change, run the self-tests:
python ci/test_tests_fetcher.py # standalone
# or: pytest ci/test_tests_fetcher.pyAdd a glob to TestSelector.COMMON_RUN_ALL_GLOBS (shared across workflows) or to
a specific workflow's extra_run_all_globs. Globs match repo-root-relative POSIX
paths; dir/** matches everything under dir/.
Add to TestSelector.DYNAMIC_EDGES:
DYNAMIC_EDGES = {
# changed-file glob : test-path globs to pull in when it changes
"deepspeed/module_inject/**": ("tests/unit/v1/moe/**",),
}Keep entries conservative: too broad just wastes GPU time; too narrow misses coverage. Prefer fixing it here over widening a run-all glob when only a slice of tests is truly affected.
If a new universal hub starts fanning every change out to the whole suite, add it
to OPAQUE_MODULES and (usually) add the hub file itself to the run-all globs so
real changes to it still run everything.
The engine is config-driven (WORKFLOWS registry of WorkflowConfig). To drive
another workflow:
- Add an entry:
WORKFLOWS["my-workflow"] = WorkflowConfig( name="my-workflow", test_scopes=("tests/unit/v2",), # dirs this workflow runs extra_run_all_globs=(".github/workflows/my-workflow.yml",), )
- In that workflow's YAML, mirror
modal-torch-latest.yml'scollect-testsjob and call the fetcher with--workflow my-workflow. - Point the runner at
ci/.test_selection/test_list.txt. - Add coverage to
ci/test_tests_fetcher.pyif the scope/behavior differs.
ci/test_tests_fetcher.py builds throwaway git repos (TmpRepo) from a synthetic
BASELINE tree and asserts the resulting mode/tests. Add a test_* function
for any new behavior; it runs both standalone and under pytest, and executes in
the collect-tests CI job, so a broken selector is caught before it mis-picks
tests.
The workflow triggers on pull_request_target, so it runs in the base repo's
context and the deploy controller can authenticate to Modal or, only for the
capacity fallback, assume its AWS role. The trust boundary is:
- Both jobs check out only the exact trusted base SHA (or the exact
push/manual SHA), with persisted credentials, LFS, and submodules disabled.
They never use the fork head with
actions/checkout. collect-testsholds no repository secrets. Trusted base code fetches the public head and base repositories at their exact event SHAs, disables credentials, hooks, submodules, and LFS smudging, and rejects checkout symlinks that escape the candidate root. It treats the candidate tree only as git/AST data; it never imports, installs, builds, or executes candidate code.- The selector logic and its self-tests always come from the trusted checkout.
A PR's
ci/changes still appear in the diff, so the base selector'sci/**run-all rule widens them to the full suite unless they are documentation. - The handoff is a fixed
all/subset/nonemode plus one bounded, validated regular file. Test paths must stay undertests/unit/v1, cannot be options or traversal paths, and are passed after pytest's--separator. deployruns only the trusted controller. Candidate metadata is restricted to a publicowner/repositoryand full 40-hex SHA; event values enter Python through environment variables, not workflow shell interpolation. GitHub's OIDC permission is scoped to this job and AWS credentials are requested only after the Modal controller returns its dedicated startup-capacity result.- The controller role ARN and opaque regional fallback JSON are stored as masked repository secrets. The controller logs region and ordinal location transitions, but never the launch-template, subnet, bucket, instance, or SSM command identifiers, and AWS CLI failures expose only a bounded error code. The trusted job exchanges its GitHub OIDC token directly for only the three credential fields, masks each value before exporting it, and never requests or prints the STS assumed-role identity.
- The Modal service token remains in the controller process. The Sandbox gets no Modal, GitHub, Hugging Face, OIDC, or other secret; no local checkout, mount, volume, network filesystem, port, proxy, or workload identity is attached. The controller constructs a positive allowlist of non-sensitive git/pip environment flags instead of forwarding its environment.
- Candidate acquisition, dependency installation, DeepSpeed installation, and
pytest all happen inside one selected backend. Modal uses an isolated
l40s:2Sandbox with a bounded server-side lifetime. Its capacity fallback uses onlyg7.12xlargeSpot, tries configured subnets in fixedus-east-1,us-east-2,us-west-2order, exposes exactly GPUs 0 and 1 to the container, and controls it through SSM. The instance has no inbound rules and a 60-minute termination guard. SSM sends prepare and test output to a scoped, expiring CloudWatch Logs group; the controller forwards new stdout/stderr events through the same public-output masking path while each command is running. Every subprocess uses structural arguments, and every nonzero clone/install/test status fails the check. - The selected backend retains outbound network access because it must reach public GitHub, package indexes, PyTorch wheels, and optionally Transformers. Those services do not have stable CIDRs suitable for the SDK's CIDR allowlist, so the containment control is the absence of secrets, identities, and mounts.
- The
pull_request_targettrigger types arereview_requested,ready_for_review, andsynchronize. Becausesynchronizere-runs on every push to an open PR (not just on a maintainer action), the maintainer review is a mitigation, not the trust boundary. Exact trusted base code plus backend isolation is the primary protection.
merge_group runs are a separate trust context. GitHub runs a queued
entry's workflows from the merge-group commit — the PR's merged tree — so the
workflow YAML itself is candidate-controlled. The jobs restore the repo's
invariant by resolving every GitHub-side checkout to the trusted base revision
(merge_group.base_sha): the selector and controller come from master there,
and the merged candidate enters only as
validated git data (fetched by exact SHA into a separate root) or inside the
no-secret execution backend. The residual exposure — a queued PR rewriting the
workflow YAML itself to echo secrets — is inherent to GitHub's merge queue;
review PRs that touch .github/workflows/* with that in mind.
Consequence: changes to
ci/*(includingtests_fetcher.pyitself) take effect underpull_request_targetonly after they're merged. A PR that changes this launcher cannot prove its new PR-triggered end-to-end path by running itself. Before merge, use pure-stdlib/unit/static validation and, only when separately authorized credentials are available, a direct backend smoke. Do not describe mock coverage as live provider evidence.
The selector is built to fail safe — to all, never to none:
- Missing/unresolvable base, no merge-base, a parse error on a file, or any
unexpected exception in the selector → it falls back to the full suite (the
top-level handler in
main()logs the traceback and setsmode=all). - Checkout, selector self-test, list validation, write, or upload failures make
collect-testsfail.deploythen enters its explicit failure guard, so a broken collector cannot pass the Required check. - The only way to run fewer tests is a clean, well-understood narrow decision; every uncertain case widens to everything.
- Missing or invalid repository/SHA metadata, an inconsistent mode/list pair, and any failed clone, install, version probe, or pytest command fail the controller. There is no moving-branch fallback.
- Only the bounded 10-minute Modal Sandbox startup timeout enables AWS. Modal API, authentication, image-build, setup, test, cleanup, and lifetime failures are terminal. After an AWS instance is allocated, boot, SSM, setup, test, and cleanup failures are also terminal and never move to another location.
- Before allocation, AWS advances through the fixed regional subnet list only for explicit EC2 insufficient-capacity responses. If every location reports capacity unavailable, no backend ran and the Required check fails as infra.
- Once created, the task-owned Sandbox or EC2 instance is terminated on success and failure. Cleanup failure is reported without hiding the primary command error; a tag-scoped final reconciliation covers interrupted AWS controllers.
nonecreates no backend.subsetandallrun at most one test backend; the GPU shape, region order, and time bounds are constants rather than PR or manual inputs. A 300-second pytest-output watchdog is a terminal timeout and is never made green by running the test again on another provider.
Every run writes a summary to the GitHub job summary (mode, reason, and the
selected files) so the decision is auditable from the Actions UI. Candidate
text is control-escaped before logs/summaries and is never written to
GITHUB_OUTPUT.
My PR shows mode=none but I changed code.
Either the change is non-Python / out of the workflow's scope, or your edits
aren't committed (the fetcher diffs committed history, merge-base..HEAD).
A relevant test wasn't selected.
Likely a runtime/dynamic dependency. Confirm with --explain, then add a
DYNAMIC_EDGES entry. As a stop-gap, use [test all]; the entry still needs
the exact needs-gpu-test label to execute that selection on the merge queue.
Everything runs even for a tiny change.
You touched a run-all glob (CI/build/shared fixture/core runtime), or a hub module
fanned out. Check the job summary's reason. If a hub over-fans, consider
OPAQUE_MODULES.
It ran the full suite and the summary says "shallow clone?" / "no merge-base". The exact event commits did not provide a usable merge-base. The collector fetches the exact head and base SHAs into fixed refs; verify both event commits are still publicly reachable. Do not replace this with a moving branch fallback.
Why didn't this PR exercise its new pull_request_target controller?
GitHub intentionally runs that event's workflow from the trusted base. The new
controller becomes the trusted code only after merge; before then, rely on the
focused static/unit evidence described in the security model. (The merge-queue
run does use the PR's merged controller, so ci/* changes are first
exercised live there.)