docs: Add "Coverage with test sharding" section to test.mdx#23219
Open
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Open
docs: Add "Coverage with test sharding" section to test.mdx#23219Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Affanmir wants to merge 3 commits intopantsbuild:mainfrom
Conversation
When using `pants test --shard=N/T` to split tests across parallel CI runners, each shard only exercises a fraction of test targets so the per-shard coverage output is artificially low. The existing docs say nothing about this pattern, leaving users to discover the solution on their own. Add a new "Coverage with test sharding" subsection under the Coverage section that covers the complete four-step workflow: 1. Enable `report = ["raw", ...]` to generate the binary `.coverage` file (not just XML) so shards can be merged later. 2. Set `relative_files = true` in `.coveragerc` — required because Pants runs each shard in its own sandbox; relative paths let `coverage combine` match files across sandboxes. 3. Upload the `.coverage` dotfile from GitHub Actions with `include-hidden-files: true` (the default silently drops it). 4. In a post-shard job, copy binaries to the repo root, run `coverage combine`, generate the final XML, and enforce `fail_under` only at this point (not per shard). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
benjyw
reviewed
Apr 8, 2026
Contributor
benjyw
left a comment
There was a problem hiding this comment.
Hi, thanks for this, it is a great addition to the docs, but I think it could use some rephrasing.
The first two "steps" aren't really steps, they're one-time config. And then the other two steps are specific to github actions.
So I recommend showing the config needed, and then just saying that you need to merge the coverage reports, and give the github actions workflow as an example of how to do this.
Reorganize "Coverage with test sharding" from four numbered steps into two logical subsections (Configuration + Merging coverage reports) and frame the GitHub Actions workflow as an example rather than the only approach. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ection Add a tip admonition in the "Sharding the input targets" section of advanced-target-selection.mdx linking to the new coverage-with-sharding guide in test.mdx, so users discover the workflow before hitting artificially low per-shard coverage numbers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
benjyw
approved these changes
Apr 9, 2026
Contributor
benjyw
left a comment
There was a problem hiding this comment.
Looks good, and will be useful for folks. Thanks!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds a new "Coverage with test sharding" subsection to the Python
testgoal docs, covering the complete workflow for getting accurate combined coverage when usingpants test --shard=N/T.Why this is needed
The
--shardflag is a common pattern for speeding up CI by splitting tests across parallel runners. However, the existing docs cover coverage and parallelism/batching in isolation — there is no guidance on what happens to coverage when sharding is in use.The problem: each shard only exercises ~1/N of test targets, so the per-shard coverage report is artificially low (e.g. ~50% of actual coverage with 2 shards). Without knowing to combine the binary
.coveragefiles, users will either:fail_underfire on every shard and give up on sharding.What's documented
A four-step workflow:
"raw"to[coverage-py].reportso Pants writes the.coveragebinary alongside XML.relative_files = true— required in.coveragercbecause each shard runs in its own sandbox; relative paths letcoverage combinematch files across sandbox directories.actions/upload-artifactsilently drops dotfiles by default; setinclude-hidden-files: trueto capture.coverage..coveragebinaries to the repo root, runcoverage combine, generate final XML, and enforcefail_underonly at this point.Two admonitions call out the two most common gotchas:
fail_underinpants.ci.tomlwhen sharding (fires prematurely on every shard)global_report = trueproduces misleading per-shard zeroes; apply it post-merge insteadChecklist
### Coverage with test shardingsubsection in the Coverage section ofdocs/docs/python/goals/test.mdxfail_under/--test-open-coverageparagraphs, before## JUnit XML results:::caution/:::notesyntax