Skip to content
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions docs/docs/python/goals/test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -578,6 +578,98 @@ branch = true

When generating HTML, XML, and JSON reports, you can automatically open the reports through the option `--test-open-coverage`.

### Coverage with test sharding

Pants supports running a subset of tests on each CI runner using the `--shard` flag:

```bash
# Run the first half of tests on one runner, the second half on another
❯ pants test --shard=0/2 ::
❯ pants test --shard=1/2 ::
```

When sharding, each runner only exercises ~1/N of your test targets, so the per-shard coverage report is artificially low. You must combine the binary `.coverage` files from all shards to get an accurate result.

#### Step 1: generate binary coverage data

Include `"raw"` in your reports so Pants writes the binary `.coverage` file (required for combining):

```toml title="pants.ci.toml"
[coverage-py]
report = ["raw", "xml", "console"]
output_dir = "coverage-report"
# Do NOT set fail_under here — per-shard coverage is artificially low.
# Enforce fail_under after combining all shards instead.
```

#### Step 2: configure `relative_files = true`

This is **required** for cross-shard merging. Pants runs each shard in its own sandbox directory, so absolute paths in `.coverage` files differ between shards. `relative_files = true` lets `coverage combine` match files across sandboxes by resolving paths relative to the repository root.

```ini title=".coveragerc"
[run]
relative_files = true
branch = true
```

#### Step 3: upload per-shard artifacts (GitHub Actions)

`actions/upload-artifact` skips dotfiles by default. Set `include-hidden-files: true` so the `.coverage` binary is included:

```yaml title=".github/workflows/ci.yml"
- name: Upload coverage reports
uses: actions/upload-artifact@v4
with:
name: coverage-shard-${{ matrix.shard }}
include-hidden-files: true # Required: .coverage is a dotfile
path: coverage-report/
```

#### Step 4: combine binaries and generate the final report

After all shards complete, download their artifacts, copy the `.coverage` binaries to your repository root (so `relative_files` path resolution works), combine them, and then enforce any coverage threshold:

```yaml title=".github/workflows/ci.yml"
coverage-report:
needs: [test]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Download all shard coverage
uses: actions/download-artifact@v4
with:
pattern: coverage-shard-*
path: all-coverage/
merge-multiple: false

- name: Combine coverage across shards
run: |
pip install coverage

# Rename each shard's .coverage to a unique name at the repo root
# (required so relative_files resolves against the repo root)
idx=0
for f in $(find all-coverage/ -name ".coverage" -not -name ".coverage.*"); do
cp "$f" ".coverage.shard${idx}"
idx=$((idx + 1))
done

coverage combine --rcfile=.coveragerc .coverage.shard*
coverage xml --rcfile=.coveragerc -o coverage-report/coverage.xml

# Enforce threshold only after combining — not per shard
coverage report --rcfile=.coveragerc --fail-under=80
```

:::caution Don't set `fail_under` in `[coverage-py]` when sharding
Each shard only runs a fraction of your targets, so per-shard coverage is intentionally incomplete. Setting `fail_under` in `pants.toml` or `pants.ci.toml` will cause every shard to fail. Remove it from your Pants config and enforce it via `coverage report --fail-under` after merging all shard binaries.
:::

:::note `global_report` and sharding
If `[coverage-py] global_report = true` is set, per-shard reports will show 0% for all files that shard didn't touch. The setting remains useful for the final merged report — consider applying it only in the post-merge step rather than in `pants.ci.toml`.
:::

## JUnit XML results

Pytest can generate [JUnit XML result files](https://docs.pytest.org/en/6.2.x/usage.html#creating-junitxml-format-files). This allows you to hook up your results, for example, to dashboards.
Expand Down