Skip to content

Latest commit

 

History

History
559 lines (466 loc) · 30 KB

TEST_SELECTION.md

File metadata and controls

559 lines (466 loc) · 30 KB

Diff-based CI test selection

This document explains the diff-driven test-selection system that lets some GPU CI workflows run only the tests a PR could affect, instead of the whole suite, while still satisfying Required status checks.

It currently drives modal-torch-latest (which runs tests/unit/v1/ on a Modal GPU or its bounded AWS capacity fallback), but is built to drive more workflows from one config — see Adding a workflow.

TL;DR

  • On a PR, ci/tests_fetcher.py diffs your branch against the base branch, traces the import graph from your changed files to the impacted tests, and writes the list to ci/.test_selection/test_list.txt. A nightly scheduled run always executes the full suite.
  • The trusted controller validates that list, then fetches, installs, and tests the exact candidate SHA inside a no-secret Modal Sandbox. If the fixed L40S:2 Sandbox cannot start within its acquisition window, the controller tries a fixed, ordered pool of AWS g7.12xlarge Spot locations. The backend runs only for opted-in merge queue entries, the nightly scheduled full run, and manual runs — a plain PR event never spends GPU quota.
  • A manual infrastructure_smoke dispatch skips selection and pytest entirely. It starts one short OCI L40S:2 Sandbox check and three independent regional AWS jobs in parallel. Each AWS job directly selects one region, launches a G7.12 Spot instance, reaches it through SSM, verifies two host and container GPUs, stores a checksum-verified proof in the private regional log bucket, and terminates its own run-tagged instance. The matrix uses fail-fast: false, so one regional result cannot cancel the other regions.
  • It is fail-safe: anything it can't reason about safely → run the full suite. It never silently runs fewer tests than reality.
  • Preview locally:
    python ci/tests_fetcher.py --base origin/master            # what would CI run?
    python ci/tests_fetcher.py --base origin/master --explain  # ...and why?
  • Force full test selection with [test all] (or [no filter]). Queue GPU execution also requires the exact needs-gpu-test label.

Why

modal-torch-latest is a Required check, so it must report a status on every PR — which means we can't use GitHub's path filters (on.<event>.paths) to skip it, because a skipped Required job blocks merges. Instead, the job always runs but selects what to test: a docs-only PR resolves to "no impacted tests" and exits fast (still green), while a PR touching a leaf module runs only the tests that import it.

The design is a small, self-contained take on HuggingFace transformers' utils/tests_fetcher.py.

Documentation-only changes

PRs and merge queue entries whose entire diff is under docs/**, under blogs/**, or matches **/*.md skip CPU/GPU test suites, Python install smoke, and native-op precompilation. This includes all documentation and blog assets and configuration files, plus Markdown at any directory depth. Formatting, DCO, and documentation checks are not disabled.

The active python, cpu-torch-latest, and nv-pre-compile-ops workflows use the reusable check-paths.yml gate. Disabled legacy workflows are left unchanged.

The shared gate also skips root version.txt changes on both PRs and merge queue entries, including diffs mixing version and documentation changes. Any other changed path still requires CI unless explicitly excluded for that workflow. The Modal selector already selects no tests for a version-only diff.

The optional mps-torch-latest and no-torch workflows rely on their PR trigger paths instead, which exclude Markdown. A README under a watched source directory cannot start these workflows merely because the PR also changes unrelated code. Their gate job is commented out because it could never skip: every PR run they start includes a non-documentation change. Re-enable it if either workflow must drop its path filters (to become a required check or run in the merge queue) or if is_docs_path() starts covering more than Markdown.

ci/check_paths.py is the single definition of documentation (is_docs_path()). The gate diffs the checked-out commit against the exact commit it was built on: a PR's merge commit against its first parent, the base-branch commit GitHub merged the PR into, and a merge queue entry against merge_group.base_sha. It does not use the event's pull_request.base.sha, which can be days older and would add the base branch's own changes, so docs-only PRs behind master would not skip. The gate runs the base branch's copy of the script, read from that same commit. A PR that edits the script is judged by the base copy, which runs its tests because the script is not documentation; the edits apply after merge. This guards against mistakes, not a malicious PR, which controls the workflow files on pull_request events. Renames count both the old and new paths: moving module.py to docs/module.py is a code deletion, not a documentation-only change. An empty diff runs tests, and detection failures, including a base branch without the script, emit a warning and fall back to running tests. If the gate job itself fails or exceeds its 10-minute limit, the required jobs fail instead of skipping. Callers may pass pr-ignore (files, or directories ending in /, such as deepspeed/inference/v2/) to skip more paths on pull requests only; merge queue entries use only the shared documentation and root version.txt exclusions. Preview the decision with python ci/check_paths.py --base "$(git merge-base origin/master HEAD)".

Scheduled, manual, and push runs retain their existing behavior. Required workflows still start and report their statuses; only their heavy jobs or steps skip. The Modal selector imports the same is_docs_path() and drops documentation paths before applying its run-all path globs and import graph, so documentation never widens a selection; explicit [test all] / [no filter] overrides still win.

Root README.md is package metadata, so the formatting workflow always runs python scripts/check-readme.py to check that it exists and is valid UTF-8 without installing DeepSpeed or PyTorch.

Moving parts

File Role
.github/workflows/modal-torch-latest.yml The workflow: a no-secret collect-tests job gating a trusted deploy controller job.
ci/tests_fetcher.py The selector. AST-parses the repo, builds an import graph, decides all / subset / none, writes the test-list file, emits a job summary.
ci/check_paths.py Defines documentation paths (is_docs_path()) for the selector and the check-paths.yml CPU/GPU workflow gate.
ci/test_tests_fetcher.py Self-tests for the selector and ci/check_paths.py (pure stdlib; run in collect-tests and, on the PR's own code, in the formatting workflow).
ci/torch_latest.py Pure-stdlib metadata/selection validation plus the trusted Modal-first, AWS-capacity-fallback controller.
ci/.test_selection/test_list.txt The hand-off artifact (one pytest target per line). Git-ignored.

Job flow

           pull_request_target / merge_group / schedule / ordinary dispatch
                                  │
                    ┌─────────────┴─────────────┐
                    │        collect-tests       │   (no secrets)
                    │  checkout exact base SHA   │
                    │ public-fetch exact head SHA│
                    │  self-test the fetcher     │
                    │  parse candidate as data   │
                    │   → mode (all|subset|none) │
                    │   → validate/upload list   │
                    └─────────────┬──────────────┘
                                  │ needs:
                                  ▼
                    ┌────────────────────────────┐
                    │           deploy            │   (trusted controller only)
                    │  if mode != none            │
                    │  checkout exact base SHA    │
                    │  validate mode/list + SHA   │
                    │  try one L40S:2 Sandbox     │
                    │    no secrets or mounts     │
                    │  startup capacity only:     │
                    │    try G7.12 Spot by region │
                    │    via GitHub OIDC + SSM    │
                    │    fetch exact head SHA      │
                    │    install + run pytest     │
                    │  terminate chosen backend  │
                    └────────────────────────────┘

A manual dispatch with infrastructure_smoke=true takes a separate path:

                  workflow_dispatch (infrastructure_smoke=true)
                                   │
                    ┌─────────────────┴─────────────────┐
                    │                                     │
          short Modal OCI smoke                  AWS regional matrix
          startup + two GPUs                     fail-fast: false
          always terminates                      east-1 / east-2 / west-2
                                                 launch + SSM + two GPUs
                                                 container + private proof
                                                 per-region termination

That path never creates a test-selection artifact, installs DeepSpeed, or runs pytest. It is an explicit infrastructure diagnostic and is not available to scheduled, pull-request, or merge-queue executions, so it cannot weaken the Required check.

mode controls the selected test scope:

  • none → deploy is skipped. The Required status is still satisfied (a skipped dependent job counts as success here).
  • subset → an eligible deploy runs pytest on exactly the impacted files.
  • all → an eligible deploy runs the whole scope (tests/unit/v1).

Independent of mode, deploy is skipped on pull_request_target runs and on merge-group runs whose entry PR lacks the exact needs-gpu-test label. The collect-tests summary still previews the selected scope. The backend executes for opted-in merge_group entries (the merged tree, gating the merge), schedule, and workflow_dispatch. On merge_group the candidate is the merge-group commit in the base repository, diffed against the queue's base SHA, so the selection covers exactly what the entry would introduce. There is deliberately no push trigger: master receives no direct pushes and the merge queue evaluates every entry before landing. Cross-entry interaction regressions are what the nightly full suite exists to catch.

For a manual infrastructure-only check, select infrastructure_smoke when dispatching the workflow. That skips both collect-tests and deploy; the Modal and per-region AWS smoke jobs run directly and concurrently instead. Leaving the input at its default false preserves the ordinary manual full scope. The AWS smoke path does not depend on Modal's result, so a successful Modal allocation cannot bypass the regional checks.

Nightly full-suite runs and regression triage

A scheduled nightly (cron, see modal-torch-latest.yml) always runs the full suite. Its outcome feeds nightly-triage.yml:

  • Green → the nightly-last-green tag moves to the tested SHA. The tag anchors the last verified-good master revision (and is exactly the good endpoint a later manual bisect would need), so days that fail for operational reasons (no GPU instance, timeout) simply leave the tag in place.
  • Infra (neither Modal nor any configured AWS location acquired a GPU) → nothing is concluded; no report, no tag move.
  • Timeout (the backend/job time budget ran out) → an issue is opened; a timeout points at an operational problem, not a candidate regression.
  • Unknown (no sentinel in the logs) → an issue is opened. A failed run can lack a sentinel for reasons other than a timeout — checkout, setup, or a job kill before the controller could classify itself — so triage reports it as unclassified rather than guessing a class.
  • Real test failures → an issue is opened listing the failing test files and the run. Triage deliberately stops there: an opted-out queue entry may have introduced a regression that nightly is the first GPU run to expose, or the failure may be a cross-entry interaction or operational flake. If nightly regressions turn out to be frequent enough to justify automated bisection, the nightly-last-green tag is the good endpoint it would start from.

The controller (ci/torch_latest.py) classifies its own failures for this routing. It waits up to 10 minutes for Modal Sandbox startup; only that startup-capacity timeout returns 75 to the workflow without a final sentinel so that AWS can run. The selected backend prints exactly one final DS_CI_FAILURE_CLASS=infra|timeout|test sentinel on failure. If no backend runs the outcome is infra (75); 300 seconds without pytest output is a timeout (124); setup and test errors are test failures (1). A failed run with no sentinel routes to the unknown class instead of guessing.

Triage reports (timeout, unknown, test failures) go through ci/nightly_report.sh: every report carries the nightly-triage label, and a recurring outcome comments on its still-open issue instead of filing a new one. Dedup keys on the title, so titles are stable across recurrences (per-night SHAs live in the body).

Triage checks out the SHA the nightly actually ran rather than current master, so the classifier and the sentinel protocol it parses come from the same revision.

Multi-rank CPU nightly

cpu-torch-multi-latest.yml covers the axis the selector does not model: world_size. cpu-torch-latest advertises a single device, so the per-device gate in tests/unit/common.py skips every world_size > 1 test there. The nightly sets LOCAL_SIZE=4 to admit world_size <= 4 and selects only those tests with DS_MULTIRANK_ONLY=1 (a pytest_collection_modifyitems hook in tests/conftest.py): one phase for the non-v1 half (120m budget) and one for the sequential-marked tests (25m), which the parallel phase deselects.

It is deliberately not a pre-merge or merge-queue check: the non-v1 half needs 70-90 minutes on a 4-vCPU runner, against 39 minutes for the longest current pre-merge job, and a wedged rank would hold up the queue.

cpu-multi-triage.yml routes its outcome through the same sentinel protocol and reporter as the modal nightly, but with its own issue titles and its own nightly-cpu-multi-last-green tag, so nightly-last-green keeps meaning "the last revision the full GPU suite verified". Issue #8805 (bf16 check_grad_overflow semantics) is an example of a regression only this selection can see.

How a decision is made

TestSelector.select() in ci/tests_fetcher.py runs these checks in order; the first that matches wins:

  1. No base ref (nightly / manual) → all.
  2. Base ref unresolvable → all.
  3. No merge-base with the base (e.g. shallow clone, unrelated history) → all. A diff here would be wrong, so we never narrow on it.
  4. Commit message tag [test all] / [no filter] anywhere on the branch → all.
  5. Only documentation paths changed (docs/**, blogs/**, or **/*.md, as defined by is_docs_path() in ci/check_paths.py), including deleted paths and both sides of renames → none. Otherwise documentation paths are dropped, so they cannot trigger the steps below.
  6. A changed or deleted file matches a run-all glob (COMMON_RUN_ALL_GLOBS + the workflow's extra_run_all_globs) → all. These are files too central or too dynamic to narrow safely: CI scripts, build system, csrc/, op_builder/, accelerator/, shared fixtures (tests/unit/common.py, tests/conftest.py, pytest.ini), and core runtime hubs (deepspeed/__init__.py, deepspeed/runtime/engine.py, deepspeed/comm/**, deepspeed/accelerator/**, …).
  7. A deleted module is still imported by a surviving file (a dangling import the graph can't follow) → all. A clean deletion (importers removed/updated in the same PR) does not trigger this.
  8. Otherwise, narrow via the import graph (below). If nothing is impacted → none; if everything is → all; else → subset.

The import graph

  • Nodes are Python files under the package roots: deepspeed/** and the unit test-helper package at tests/unit/**.
  • An edge A → B means "B imports A". The selector walks backwards from each changed file to every test that (transitively) imports it.
  • Opaque hub modules (OPAQUE_MODULES: deepspeed, deepspeed.comm, deepspeed.accelerator): almost every test imports unit.common, which imports these. Their __init__.py files eagerly pull in huge subtrees, so if treated as normal nodes any deepspeed/** change would fan out to the whole suite. We therefore don't expand their __init__ imports; instead, changes to the hubs themselves are caught by the run-all globs in step 6.
  • conftest.py changes select every test under that conftest's directory.
  • New test files are selected directly (they have no importers yet).

Dynamic edges (DYNAMIC_EDGES)

Some dependencies are wired at runtime (monkey-patching, plugin/registry lookup, JIT-loaded ops, deepspeed.initialize()-time replace_module injection), so a test can depend on code it never imports. DYNAMIC_EDGES is a curated map of changed-file glob → extra test-path globs that patches these blind spots. It is additive on top of the static graph (and is only consulted if step 6 didn't already short-circuit to all).

Driving it (as a contributor)

Preview what CI will run

The fetcher is pure stdlib — no DeepSpeed/torch install needed, just git:

# Selection for your branch vs. the base branch
python ci/tests_fetcher.py --base origin/master
cat ci/.test_selection/test_list.txt

# Explain *why* each test was selected (prints import chains)
python ci/tests_fetcher.py --base origin/master --explain

--explain output looks like:

deepspeed/shared.py impacts:
    tests/unit/v1/test_shared.py <- deepspeed/shared.py
    tests/unit/v1/moe/test_moe.py <- tests/unit/v1/moe/test_moe.py <- deepspeed/shared.py

Escape hatches

  • Force full test selection: include [test all] (or [no filter]) anywhere in a commit message on the branch. Queue GPU execution still needs the exact needs-gpu-test label.
  • Touch an infra file: changing a non-documentation file that matches a run-all glob runs everything.
  • Found a missed test? It's likely a runtime/dynamic dependency the static graph can't see — add a DYNAMIC_EDGES entry (see below) and/or report it.

Changing it (as a maintainer)

All knobs live in ci/tests_fetcher.py. After any change, run the self-tests:

python ci/test_tests_fetcher.py          # standalone
# or: pytest ci/test_tests_fetcher.py

Add a run-all trigger

Add a glob to TestSelector.COMMON_RUN_ALL_GLOBS (shared across workflows) or to a specific workflow's extra_run_all_globs. Globs match repo-root-relative POSIX paths; dir/** matches everything under dir/.

Add a dynamic edge

Add to TestSelector.DYNAMIC_EDGES:

DYNAMIC_EDGES = {
    # changed-file glob : test-path globs to pull in when it changes
    "deepspeed/module_inject/**": ("tests/unit/v1/moe/**",),
}

Keep entries conservative: too broad just wastes GPU time; too narrow misses coverage. Prefer fixing it here over widening a run-all glob when only a slice of tests is truly affected.

Mark a module opaque

If a new universal hub starts fanning every change out to the whole suite, add it to OPAQUE_MODULES and (usually) add the hub file itself to the run-all globs so real changes to it still run everything.

Add a new workflow

The engine is config-driven (WORKFLOWS registry of WorkflowConfig). To drive another workflow:

  1. Add an entry:
    WORKFLOWS["my-workflow"] = WorkflowConfig(
        name="my-workflow",
        test_scopes=("tests/unit/v2",),               # dirs this workflow runs
        extra_run_all_globs=(".github/workflows/my-workflow.yml",),
    )
  2. In that workflow's YAML, mirror modal-torch-latest.yml's collect-tests job and call the fetcher with --workflow my-workflow.
  3. Point the runner at ci/.test_selection/test_list.txt.
  4. Add coverage to ci/test_tests_fetcher.py if the scope/behavior differs.

Add a self-test

ci/test_tests_fetcher.py builds throwaway git repos (TmpRepo) from a synthetic BASELINE tree and asserts the resulting mode/tests. Add a test_* function for any new behavior; it runs both standalone and under pytest, and executes in the collect-tests CI job, so a broken selector is caught before it mis-picks tests.

Security model

The workflow triggers on pull_request_target, so it runs in the base repo's context and the deploy controller can authenticate to Modal or, only for the capacity fallback, assume its AWS role. The trust boundary is:

  • Both jobs check out only the exact trusted base SHA (or the exact push/manual SHA), with persisted credentials, LFS, and submodules disabled. They never use the fork head with actions/checkout.
  • collect-tests holds no repository secrets. Trusted base code fetches the public head and base repositories at their exact event SHAs, disables credentials, hooks, submodules, and LFS smudging, and rejects checkout symlinks that escape the candidate root. It treats the candidate tree only as git/AST data; it never imports, installs, builds, or executes candidate code.
  • The selector logic and its self-tests always come from the trusted checkout. A PR's ci/ changes still appear in the diff, so the base selector's ci/** run-all rule widens them to the full suite unless they are documentation.
  • The handoff is a fixed all / subset / none mode plus one bounded, validated regular file. Test paths must stay under tests/unit/v1, cannot be options or traversal paths, and are passed after pytest's -- separator.
  • deploy runs only the trusted controller. Candidate metadata is restricted to a public owner/repository and full 40-hex SHA; event values enter Python through environment variables, not workflow shell interpolation. GitHub's OIDC permission is scoped to this job and AWS credentials are requested only after the Modal controller returns its dedicated startup-capacity result.
  • The controller role ARN and opaque regional fallback JSON are stored as masked repository secrets. The controller logs region and ordinal location transitions, but never the launch-template, subnet, bucket, instance, or SSM command identifiers, and AWS CLI failures expose only a bounded error code. The trusted job exchanges its GitHub OIDC token directly for only the three credential fields, masks each value before exporting it, and never requests or prints the STS assumed-role identity.
  • The Modal service token remains in the controller process. The Sandbox gets no Modal, GitHub, Hugging Face, OIDC, or other secret; no local checkout, mount, volume, network filesystem, port, proxy, or workload identity is attached. The controller constructs a positive allowlist of non-sensitive git/pip environment flags instead of forwarding its environment.
  • Candidate acquisition, dependency installation, DeepSpeed installation, and pytest all happen inside one selected backend. Modal uses an isolated l40s:2 Sandbox with a bounded server-side lifetime. Its capacity fallback uses only g7.12xlarge Spot, tries configured subnets in fixed us-east-1, us-east-2, us-west-2 order, exposes exactly GPUs 0 and 1 to the container, and controls it through SSM. The instance has no inbound rules and a 60-minute termination guard. SSM sends prepare and test output to a scoped, expiring CloudWatch Logs group; the controller forwards new stdout/stderr events through the same public-output masking path while each command is running. Every subprocess uses structural arguments, and every nonzero clone/install/test status fails the check.
  • The selected backend retains outbound network access because it must reach public GitHub, package indexes, PyTorch wheels, and optionally Transformers. Those services do not have stable CIDRs suitable for the SDK's CIDR allowlist, so the containment control is the absence of secrets, identities, and mounts.
  • The pull_request_target trigger types are review_requested, ready_for_review, and synchronize. Because synchronize re-runs on every push to an open PR (not just on a maintainer action), the maintainer review is a mitigation, not the trust boundary. Exact trusted base code plus backend isolation is the primary protection.

merge_group runs are a separate trust context. GitHub runs a queued entry's workflows from the merge-group commit — the PR's merged tree — so the workflow YAML itself is candidate-controlled. The jobs restore the repo's invariant by resolving every GitHub-side checkout to the trusted base revision (merge_group.base_sha): the selector and controller come from master there, and the merged candidate enters only as validated git data (fetched by exact SHA into a separate root) or inside the no-secret execution backend. The residual exposure — a queued PR rewriting the workflow YAML itself to echo secrets — is inherent to GitHub's merge queue; review PRs that touch .github/workflows/* with that in mind.

Consequence: changes to ci/* (including tests_fetcher.py itself) take effect under pull_request_target only after they're merged. A PR that changes this launcher cannot prove its new PR-triggered end-to-end path by running itself. Before merge, use pure-stdlib/unit/static validation and, only when separately authorized credentials are available, a direct backend smoke. Do not describe mock coverage as live provider evidence.

Failure modes & guarantees

The selector is built to fail safe — to all, never to none:

  • Missing/unresolvable base, no merge-base, a parse error on a file, or any unexpected exception in the selector → it falls back to the full suite (the top-level handler in main() logs the traceback and sets mode=all).
  • Checkout, selector self-test, list validation, write, or upload failures make collect-tests fail. deploy then enters its explicit failure guard, so a broken collector cannot pass the Required check.
  • The only way to run fewer tests is a clean, well-understood narrow decision; every uncertain case widens to everything.
  • Missing or invalid repository/SHA metadata, an inconsistent mode/list pair, and any failed clone, install, version probe, or pytest command fail the controller. There is no moving-branch fallback.
  • Only the bounded 10-minute Modal Sandbox startup timeout enables AWS. Modal API, authentication, image-build, setup, test, cleanup, and lifetime failures are terminal. After an AWS instance is allocated, boot, SSM, setup, test, and cleanup failures are also terminal and never move to another location.
  • Before allocation, AWS advances through the fixed regional subnet list only for explicit EC2 insufficient-capacity responses. If every location reports capacity unavailable, no backend ran and the Required check fails as infra.
  • Once created, the task-owned Sandbox or EC2 instance is terminated on success and failure. Cleanup failure is reported without hiding the primary command error; a tag-scoped final reconciliation covers interrupted AWS controllers.
  • none creates no backend. subset and all run at most one test backend; the GPU shape, region order, and time bounds are constants rather than PR or manual inputs. A 300-second pytest-output watchdog is a terminal timeout and is never made green by running the test again on another provider.

Every run writes a summary to the GitHub job summary (mode, reason, and the selected files) so the decision is auditable from the Actions UI. Candidate text is control-escaped before logs/summaries and is never written to GITHUB_OUTPUT.

FAQ / troubleshooting

My PR shows mode=none but I changed code. Either the change is non-Python / out of the workflow's scope, or your edits aren't committed (the fetcher diffs committed history, merge-base..HEAD).

A relevant test wasn't selected. Likely a runtime/dynamic dependency. Confirm with --explain, then add a DYNAMIC_EDGES entry. As a stop-gap, use [test all]; the entry still needs the exact needs-gpu-test label to execute that selection on the merge queue.

Everything runs even for a tiny change. You touched a run-all glob (CI/build/shared fixture/core runtime), or a hub module fanned out. Check the job summary's reason. If a hub over-fans, consider OPAQUE_MODULES.

It ran the full suite and the summary says "shallow clone?" / "no merge-base". The exact event commits did not provide a usable merge-base. The collector fetches the exact head and base SHAs into fixed refs; verify both event commits are still publicly reachable. Do not replace this with a moving branch fallback.

Why didn't this PR exercise its new pull_request_target controller? GitHub intentionally runs that event's workflow from the trusted base. The new controller becomes the trusted code only after merge; before then, rely on the focused static/unit evidence described in the security model. (The merge-queue run does use the PR's merged controller, so ci/* changes are first exercised live there.)