Skip to content

Switch to PCOV for the coverage report runner#11618

Open
johnbillion wants to merge 7 commits into
WordPress:trunkfrom
johnbillion:pcov
Open

Switch to PCOV for the coverage report runner#11618
johnbillion wants to merge 7 commits into
WordPress:trunkfrom
johnbillion:pcov

Conversation

@johnbillion
Copy link
Copy Markdown
Member

@johnbillion johnbillion commented Apr 21, 2026

Instead of using Xdebug in coverage mode to generate a coverage report, we can use PCOV.

This has reduced the test coverage step time from around 50 minutes to around 25. A comparable unit test step without coverage takes around 11 minutes. I was expecting the time reduction to be even greater, but it's still a 50% saving.

It might be worth installing PCOV directly into the container images. This PR installs it prior to running the coverage tests but it only takes a few seconds. A task for another time.

Trac ticket: https://core.trac.wordpress.org/ticket/64893

Why has the HTML report generation been removed?

It creates a 400MB artifact that expands to 7GB of HTML files. Many of them are unusable. We've got Codecov for this reason.

Why has the coverage changed?

The Codecov coverage report is showing a -0.5% coverage reduction with this change. The source code itself has not changed, the change comes from the differences in reporting between Xdebug and PCOV. The PCOV docs notes a difference in switch handling, for example.

What about phpdbg?

PCOV is faster and more accurate than phpdbg, although admittedly there aren't a huge number of comparisons available. Regardless, PHPUnit as of version 10 no longer supports phpdbg for coverage reporting so PCOV is a better choice for future-proofing.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Opus 4.7
Used for: Initial planning for the switch to PCOV. Tweaked and verified by me.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

@johnbillion johnbillion marked this pull request as ready for review April 21, 2026 16:16
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 21, 2026

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props johnbillion, desrosj.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the GitHub Actions coverage workflow to generate PHPUnit code coverage using PCOV instead of enabling Xdebug coverage mode, with the goal of significantly reducing CI runtime and artifact size.

Changes:

  • Removes Xdebug coverage configuration from the coverage workflow path.
  • Installs/enables PCOV inside the PHP container when coverage-report is enabled and switches the PHPUnit invocation to run against the running container.
  • Drops HTML coverage report generation/artifact upload, keeping only the Clover XML for Codecov.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/test-coverage.yml Removes Xdebug coverage env vars and (currently) comments out the repo guard for running coverage.
.github/workflows/reusable-phpunit-tests-v3.yml Adds PCOV installation/config and updates PHPUnit coverage invocation to output Clover XML only.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

permissions:
contents: read
if: ${{ github.repository == 'WordPress/wordpress-develop' }}
# if: ${{ github.repository == 'WordPress/wordpress-develop' }}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be reverted prior to merge. Can stay in place for now.

Comment thread .github/workflows/reusable-phpunit-tests-v3.yml
Copy link
Copy Markdown
Member

@desrosj desrosj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Nice speed improvement! It would be great to reach a point where we can generate and submit coverage reports for every commit. But I'm not sure we're there just yet.

Added some small feedback here, but I have also opened WordPress/wpdev-docker-images#210 to introduce PCOV into the Docker images. I think that's preferable because it not only eliminates the need for some of this installation code, but it also allows for a coverage report to be generated locally using PCOV more easily. Speaking of, it looks like the --coverage-html report is still configured in the test:coverage command within package.json.

If WordPress/wpdev-docker-images#210 is merged, then the .env.example files should be updated to include:

  • LOCAL_PHP_PCOV set to false with a Docblock explaining what it would be used for.
  • LOCAL_PCOV_FILES that's commented out by default, but would allow someone to specify a number that's passed to pcov.initial.files.

The docker.compose.yml file would need to be updated to set pcov.initial.files to the value of LOCAL_PCOV_FILES when set.

Comment on lines +203 to +226
# Installs PCOV as the code coverage driver for the PHPUnit run below.
#
# The INI directives tune PCOV for WordPress's codebase:
# - `pcov.enabled` keeps the Zend hooks active (this is the default, but
# stated explicitly for clarity).
# - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not
# record hits for `vendor/`, `tests/`, or WordPress test fixtures that
# PHPUnit would discard at report time anyway.
# - `pcov.initial.files` pre-sizes the internal file tracking array for
# the thousands of files under `src/`, avoiding reallocation churn
# during test warmup. The default of 64 is far too low here.
- name: Install PCOV coverage driver
if: ${{ inputs.coverage-report }}
run: |
docker compose exec -T -u 0 php sh -c '
pecl install pcov &&
docker-php-ext-enable pcov &&
{
echo "pcov.enabled=1"
echo "pcov.directory=/var/www/src"
echo "pcov.initial.files=2000"
} >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini &&
php -m | grep -i pcov
'
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setup-php action supports PCOV as a coverage tool. When we are finally able to explore running the unit tests outside of the Docker environment, this becomes simply supplying values to the ini-values input.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Comment on lines +203 to +227
# Installs PCOV as the code coverage driver for the PHPUnit run below.
#
# The INI directives tune PCOV for WordPress's codebase:
# - `pcov.enabled` keeps the Zend hooks active (this is the default, but
# stated explicitly for clarity).
# - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not
# record hits for `vendor/`, `tests/`, or WordPress test fixtures that
# PHPUnit would discard at report time anyway.
# - `pcov.initial.files` pre-sizes the internal file tracking array for
# the thousands of files under `src/`, avoiding reallocation churn
# during test warmup. The default of 64 is far too low here.
- name: Install PCOV coverage driver
if: ${{ inputs.coverage-report }}
run: |
docker compose exec -T -u 0 php sh -c '
pecl install pcov &&
docker-php-ext-enable pcov &&
{
echo "pcov.enabled=1"
echo "pcov.directory=/var/www/src"
echo "pcov.initial.files=2000"
} >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini &&
php -m | grep -i pcov
'

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked and there are 1,852 PHP files in src/, so this number should work. However, I'm concerned that this number will just end up being set and never updated over time.

What if we add a simple file counting step before this one and round the number of files up to the nearest hundred before using that value for pcov.initial.files?

Suggested change
# Installs PCOV as the code coverage driver for the PHPUnit run below.
#
# The INI directives tune PCOV for WordPress's codebase:
# - `pcov.enabled` keeps the Zend hooks active (this is the default, but
# stated explicitly for clarity).
# - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not
# record hits for `vendor/`, `tests/`, or WordPress test fixtures that
# PHPUnit would discard at report time anyway.
# - `pcov.initial.files` pre-sizes the internal file tracking array for
# the thousands of files under `src/`, avoiding reallocation churn
# during test warmup. The default of 64 is far too low here.
- name: Install PCOV coverage driver
if: ${{ inputs.coverage-report }}
run: |
docker compose exec -T -u 0 php sh -c '
pecl install pcov &&
docker-php-ext-enable pcov &&
{
echo "pcov.enabled=1"
echo "pcov.directory=/var/www/src"
echo "pcov.initial.files=2000"
} >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini &&
php -m | grep -i pcov
'
- name: Count PHP source files
id: count_php_files
if: ${{ inputs.coverage-report }}
run: |
total=$(find src -type f -name '*.php' -print | wc -l | tr -d '[:space:]')
rounded=$(( ( total + 99 ) / 100 * 100 ))
echo "count=$rounded" >> "$GITHUB_OUTPUT"
echo "::info::PHP files under src/: ${total}; pcov.initial.files=${rounded}"
# Installs PCOV as the code coverage driver for the PHPUnit run below.
#
# The INI directives tune PCOV for WordPress's codebase:
# - `pcov.enabled` keeps the Zend hooks active (this is the default, but
# stated explicitly for clarity).
# - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not
# record hits for `vendor/`, `tests/`, or WordPress test fixtures that
# PHPUnit would discard at report time anyway.
# - `pcov.initial.files` pre-sizes the internal file tracking array for
# the rough number of files under `src/`, avoiding reallocation churn
# during test warmup. The default of 64 is far too low here.
- name: Install PCOV coverage driver
if: ${{ inputs.coverage-report }}
env:
PCOV_INITIAL_FILES: ${{ steps.count_php_files.outputs.count }}
run: |
docker compose exec -T -u 0 php sh -c '
pecl install pcov &&
docker-php-ext-enable pcov &&
{
echo pcov.enabled=1
echo pcov.directory=/var/www/src
echo pcov.initial.files=$PCOV_INITIAL_FILES
} >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini &&
php -m | grep -i pcov
'

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I ran the count before deciding on 2,000. This setting reserves the given size for a hash table used during the coverage run, if it's exceeded it just means some resizing occurs. I wonder if the find might take longer than the hash table resizing. No clue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude tells me just to use 10,000 which will allocate ~512KB of memory and is plenty big enough. Sound good?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, I think works here. But if we choose to go the "PCOV installed in Docker for us" route, we'd have to apply that number there. A number that high also negates the need for a related environment variable in the local environment, which is a positive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In review

Development

Successfully merging this pull request may close these issues.

3 participants