From c480974ee3401bd1431820aaa1771cd28f721548 Mon Sep 17 00:00:00 2001 From: Affan Amir Mir Date: Mon, 6 Apr 2026 12:27:52 +0500 Subject: [PATCH 1/3] docs: add "Coverage with test sharding" section to test.mdx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When using `pants test --shard=N/T` to split tests across parallel CI runners, each shard only exercises a fraction of test targets so the per-shard coverage output is artificially low. The existing docs say nothing about this pattern, leaving users to discover the solution on their own. Add a new "Coverage with test sharding" subsection under the Coverage section that covers the complete four-step workflow: 1. Enable `report = ["raw", ...]` to generate the binary `.coverage` file (not just XML) so shards can be merged later. 2. Set `relative_files = true` in `.coveragerc` — required because Pants runs each shard in its own sandbox; relative paths let `coverage combine` match files across sandboxes. 3. Upload the `.coverage` dotfile from GitHub Actions with `include-hidden-files: true` (the default silently drops it). 4. In a post-shard job, copy binaries to the repo root, run `coverage combine`, generate the final XML, and enforce `fail_under` only at this point (not per shard). Co-Authored-By: Claude Sonnet 4.6 --- docs/docs/python/goals/test.mdx | 92 +++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/docs/docs/python/goals/test.mdx b/docs/docs/python/goals/test.mdx index 53dffaacc8e..1286308e8eb 100644 --- a/docs/docs/python/goals/test.mdx +++ b/docs/docs/python/goals/test.mdx @@ -578,6 +578,98 @@ branch = true When generating HTML, XML, and JSON reports, you can automatically open the reports through the option `--test-open-coverage`. +### Coverage with test sharding + +Pants supports running a subset of tests on each CI runner using the `--shard` flag: + +```bash +# Run the first half of tests on one runner, the second half on another +❯ pants test --shard=0/2 :: +❯ pants test --shard=1/2 :: +``` + +When sharding, each runner only exercises ~1/N of your test targets, so the per-shard coverage report is artificially low. You must combine the binary `.coverage` files from all shards to get an accurate result. + +#### Step 1: generate binary coverage data + +Include `"raw"` in your reports so Pants writes the binary `.coverage` file (required for combining): + +```toml title="pants.ci.toml" +[coverage-py] +report = ["raw", "xml", "console"] +output_dir = "coverage-report" +# Do NOT set fail_under here — per-shard coverage is artificially low. +# Enforce fail_under after combining all shards instead. +``` + +#### Step 2: configure `relative_files = true` + +This is **required** for cross-shard merging. Pants runs each shard in its own sandbox directory, so absolute paths in `.coverage` files differ between shards. `relative_files = true` lets `coverage combine` match files across sandboxes by resolving paths relative to the repository root. + +```ini title=".coveragerc" +[run] +relative_files = true +branch = true +``` + +#### Step 3: upload per-shard artifacts (GitHub Actions) + +`actions/upload-artifact` skips dotfiles by default. Set `include-hidden-files: true` so the `.coverage` binary is included: + +```yaml title=".github/workflows/ci.yml" +- name: Upload coverage reports + uses: actions/upload-artifact@v4 + with: + name: coverage-shard-${{ matrix.shard }} + include-hidden-files: true # Required: .coverage is a dotfile + path: coverage-report/ +``` + +#### Step 4: combine binaries and generate the final report + +After all shards complete, download their artifacts, copy the `.coverage` binaries to your repository root (so `relative_files` path resolution works), combine them, and then enforce any coverage threshold: + +```yaml title=".github/workflows/ci.yml" +coverage-report: + needs: [test] + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Download all shard coverage + uses: actions/download-artifact@v4 + with: + pattern: coverage-shard-* + path: all-coverage/ + merge-multiple: false + + - name: Combine coverage across shards + run: | + pip install coverage + + # Rename each shard's .coverage to a unique name at the repo root + # (required so relative_files resolves against the repo root) + idx=0 + for f in $(find all-coverage/ -name ".coverage" -not -name ".coverage.*"); do + cp "$f" ".coverage.shard${idx}" + idx=$((idx + 1)) + done + + coverage combine --rcfile=.coveragerc .coverage.shard* + coverage xml --rcfile=.coveragerc -o coverage-report/coverage.xml + + # Enforce threshold only after combining — not per shard + coverage report --rcfile=.coveragerc --fail-under=80 +``` + +:::caution Don't set `fail_under` in `[coverage-py]` when sharding +Each shard only runs a fraction of your targets, so per-shard coverage is intentionally incomplete. Setting `fail_under` in `pants.toml` or `pants.ci.toml` will cause every shard to fail. Remove it from your Pants config and enforce it via `coverage report --fail-under` after merging all shard binaries. +::: + +:::note `global_report` and sharding +If `[coverage-py] global_report = true` is set, per-shard reports will show 0% for all files that shard didn't touch. The setting remains useful for the final merged report — consider applying it only in the post-merge step rather than in `pants.ci.toml`. +::: + ## JUnit XML results Pytest can generate [JUnit XML result files](https://docs.pytest.org/en/6.2.x/usage.html#creating-junitxml-format-files). This allows you to hook up your results, for example, to dashboards. From a6d2b7637289d97d41db8540fdc6e59c421a035d Mon Sep 17 00:00:00 2001 From: Affan Amir Mir Date: Wed, 8 Apr 2026 12:34:03 +0500 Subject: [PATCH 2/3] docs: restructure sharding section per review feedback Reorganize "Coverage with test sharding" from four numbered steps into two logical subsections (Configuration + Merging coverage reports) and frame the GitHub Actions workflow as an example rather than the only approach. Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/docs/python/goals/test.mdx | 52 ++++++++++----------------------- 1 file changed, 15 insertions(+), 37 deletions(-) diff --git a/docs/docs/python/goals/test.mdx b/docs/docs/python/goals/test.mdx index 1286308e8eb..54f70f395f4 100644 --- a/docs/docs/python/goals/test.mdx +++ b/docs/docs/python/goals/test.mdx @@ -580,56 +580,42 @@ When generating HTML, XML, and JSON reports, you can automatically open the repo ### Coverage with test sharding -Pants supports running a subset of tests on each CI runner using the `--shard` flag: +When using the `--shard` flag to split tests across CI runners, each shard only exercises a fraction of your test targets. The per-shard coverage report will be artificially low. To get accurate coverage you need to combine the binary `.coverage` files from all shards. -```bash -# Run the first half of tests on one runner, the second half on another -❯ pants test --shard=0/2 :: -❯ pants test --shard=1/2 :: -``` - -When sharding, each runner only exercises ~1/N of your test targets, so the per-shard coverage report is artificially low. You must combine the binary `.coverage` files from all shards to get an accurate result. +#### Configuration -#### Step 1: generate binary coverage data - -Include `"raw"` in your reports so Pants writes the binary `.coverage` file (required for combining): +Add `"raw"` to your coverage reports so Pants writes the `.coverage` binary, and set `relative_files = true` so that `coverage combine` can match paths across different sandbox directories: ```toml title="pants.ci.toml" [coverage-py] report = ["raw", "xml", "console"] output_dir = "coverage-report" -# Do NOT set fail_under here — per-shard coverage is artificially low. -# Enforce fail_under after combining all shards instead. ``` -#### Step 2: configure `relative_files = true` - -This is **required** for cross-shard merging. Pants runs each shard in its own sandbox directory, so absolute paths in `.coverage` files differ between shards. `relative_files = true` lets `coverage combine` match files across sandboxes by resolving paths relative to the repository root. - ```ini title=".coveragerc" [run] relative_files = true branch = true ``` -#### Step 3: upload per-shard artifacts (GitHub Actions) +:::caution Don't set `fail_under` in `[coverage-py]` when sharding +Each shard only runs a fraction of your targets, so per-shard coverage is intentionally incomplete. Setting `fail_under` in `pants.toml` or `pants.ci.toml` will cause every shard to fail. Enforce the threshold after combining all shards instead. +::: + +#### Merging coverage reports -`actions/upload-artifact` skips dotfiles by default. Set `include-hidden-files: true` so the `.coverage` binary is included: +After all shards complete, collect their `.coverage` binaries, combine them with `coverage combine`, and generate the final report. The following example uses GitHub Actions, but the same approach applies to any CI system: ```yaml title=".github/workflows/ci.yml" -- name: Upload coverage reports +# In each shard job: upload the coverage output +- name: Upload coverage uses: actions/upload-artifact@v4 with: name: coverage-shard-${{ matrix.shard }} - include-hidden-files: true # Required: .coverage is a dotfile + include-hidden-files: true # .coverage is a dotfile path: coverage-report/ -``` - -#### Step 4: combine binaries and generate the final report -After all shards complete, download their artifacts, copy the `.coverage` binaries to your repository root (so `relative_files` path resolution works), combine them, and then enforce any coverage threshold: - -```yaml title=".github/workflows/ci.yml" +# Post-shard job: combine and enforce threshold coverage-report: needs: [test] runs-on: ubuntu-latest @@ -643,12 +629,10 @@ coverage-report: path: all-coverage/ merge-multiple: false - - name: Combine coverage across shards + - name: Combine and report run: | pip install coverage - # Rename each shard's .coverage to a unique name at the repo root - # (required so relative_files resolves against the repo root) idx=0 for f in $(find all-coverage/ -name ".coverage" -not -name ".coverage.*"); do cp "$f" ".coverage.shard${idx}" @@ -657,17 +641,11 @@ coverage-report: coverage combine --rcfile=.coveragerc .coverage.shard* coverage xml --rcfile=.coveragerc -o coverage-report/coverage.xml - - # Enforce threshold only after combining — not per shard coverage report --rcfile=.coveragerc --fail-under=80 ``` -:::caution Don't set `fail_under` in `[coverage-py]` when sharding -Each shard only runs a fraction of your targets, so per-shard coverage is intentionally incomplete. Setting `fail_under` in `pants.toml` or `pants.ci.toml` will cause every shard to fail. Remove it from your Pants config and enforce it via `coverage report --fail-under` after merging all shard binaries. -::: - :::note `global_report` and sharding -If `[coverage-py] global_report = true` is set, per-shard reports will show 0% for all files that shard didn't touch. The setting remains useful for the final merged report — consider applying it only in the post-merge step rather than in `pants.ci.toml`. +With `[coverage-py] global_report = true`, per-shard reports show 0% for untouched files. Consider applying this setting only in the post-merge step rather than in `pants.ci.toml`. ::: ## JUnit XML results From c5c64dc971303307a2b8e1af71f8a092f73f4c0f Mon Sep 17 00:00:00 2001 From: Affan Amir Mir Date: Thu, 9 Apr 2026 00:31:31 +0500 Subject: [PATCH 3/3] docs: Cross-reference coverage-with-sharding from advanced target selection Add a tip admonition in the "Sharding the input targets" section of advanced-target-selection.mdx linking to the new coverage-with-sharding guide in test.mdx, so users discover the workflow before hitting artificially low per-shard coverage numbers. Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/docs/using-pants/advanced-target-selection.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/docs/using-pants/advanced-target-selection.mdx b/docs/docs/using-pants/advanced-target-selection.mdx index a35fdb8c4d7..ff7b349ecf8 100644 --- a/docs/docs/using-pants/advanced-target-selection.mdx +++ b/docs/docs/using-pants/advanced-target-selection.mdx @@ -206,6 +206,10 @@ For other goals, you can leverage shell piping to partition the input targets in pants list :: | awk 'NR % 5 == 0' | xargs pants package ``` +:::tip Coverage and sharding +When using `--shard` with test coverage enabled, each shard only exercises a fraction of your targets, producing artificially low coverage numbers. You need to combine the binary `.coverage` files from all shards in a post-shard CI step to get accurate results. See [Coverage with test sharding](../python/goals/test.mdx#coverage-with-test-sharding) for the full configuration and CI workflow. +::: + ## Using CLI aliases If setting tags on individual targets is not feasible, there are a few other options available to refer to multiple targets.