Skip to content

docs: Add "Coverage with test sharding" section to test.mdx#23219

Open
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir:docs/coverage-with-sharding
Open

docs: Add "Coverage with test sharding" section to test.mdx#23219
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir:docs/coverage-with-sharding

Conversation

@Affanmir
Copy link
Copy Markdown

@Affanmir Affanmir commented Apr 6, 2026

What this PR does

Adds a new "Coverage with test sharding" subsection to the Python test goal docs, covering the complete workflow for getting accurate combined coverage when using pants test --shard=N/T.

Why this is needed

The --shard flag is a common pattern for speeding up CI by splitting tests across parallel runners. However, the existing docs cover coverage and parallelism/batching in isolation — there is no guidance on what happens to coverage when sharding is in use.

The problem: each shard only exercises ~1/N of test targets, so the per-shard coverage report is artificially low (e.g. ~50% of actual coverage with 2 shards). Without knowing to combine the binary .coverage files, users will either:

  • See misleadingly low coverage numbers and disable coverage, or
  • Have fail_under fire on every shard and give up on sharding.

What's documented

A four-step workflow:

  1. Generate binary data — add "raw" to [coverage-py].report so Pants writes the .coverage binary alongside XML.
  2. relative_files = true — required in .coveragerc because each shard runs in its own sandbox; relative paths let coverage combine match files across sandbox directories.
  3. Upload dotfilesactions/upload-artifact silently drops dotfiles by default; set include-hidden-files: true to capture .coverage.
  4. Post-shard combine job — download all shard artifacts, copy .coverage binaries to the repo root, run coverage combine, generate final XML, and enforce fail_under only at this point.

Two admonitions call out the two most common gotchas:

  • Don't set fail_under in pants.ci.toml when sharding (fires prematurely on every shard)
  • global_report = true produces misleading per-shard zeroes; apply it post-merge instead

Checklist

  • Added new ### Coverage with test sharding subsection in the Coverage section of docs/docs/python/goals/test.mdx
  • Placement: after the fail_under / --test-open-coverage paragraphs, before ## JUnit XML results
  • All code blocks closed, admonitions use :::caution / :::note syntax
  • No changes to any other files

When using `pants test --shard=N/T` to split tests across parallel CI
runners, each shard only exercises a fraction of test targets so the
per-shard coverage output is artificially low.  The existing docs say
nothing about this pattern, leaving users to discover the solution on
their own.

Add a new "Coverage with test sharding" subsection under the Coverage
section that covers the complete four-step workflow:

1. Enable `report = ["raw", ...]` to generate the binary `.coverage`
   file (not just XML) so shards can be merged later.
2. Set `relative_files = true` in `.coveragerc` — required because
   Pants runs each shard in its own sandbox; relative paths let
   `coverage combine` match files across sandboxes.
3. Upload the `.coverage` dotfile from GitHub Actions with
   `include-hidden-files: true` (the default silently drops it).
4. In a post-shard job, copy binaries to the repo root, run
   `coverage combine`, generate the final XML, and enforce
   `fail_under` only at this point (not per shard).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for this, it is a great addition to the docs, but I think it could use some rephrasing.

The first two "steps" aren't really steps, they're one-time config. And then the other two steps are specific to github actions.

So I recommend showing the config needed, and then just saying that you need to merge the coverage reports, and give the github actions workflow as an example of how to do this.

Reorganize "Coverage with test sharding" from four numbered steps into
two logical subsections (Configuration + Merging coverage reports) and
frame the GitHub Actions workflow as an example rather than the only
approach.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Affanmir Affanmir requested a review from benjyw April 8, 2026 07:36
…ection

Add a tip admonition in the "Sharding the input targets" section of
advanced-target-selection.mdx linking to the new coverage-with-sharding
guide in test.mdx, so users discover the workflow before hitting
artificially low per-shard coverage numbers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and will be useful for folks. Thanks!

@benjyw benjyw added the release-notes:not-required [CI] PR doesn't require mention in release notes label Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes:not-required [CI] PR doesn't require mention in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants