Switch to PCOV for the coverage report runner#11618
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
There was a problem hiding this comment.
Pull request overview
This PR updates the GitHub Actions coverage workflow to generate PHPUnit code coverage using PCOV instead of enabling Xdebug coverage mode, with the goal of significantly reducing CI runtime and artifact size.
Changes:
- Removes Xdebug coverage configuration from the coverage workflow path.
- Installs/enables PCOV inside the PHP container when
coverage-reportis enabled and switches the PHPUnit invocation to run against the running container. - Drops HTML coverage report generation/artifact upload, keeping only the Clover XML for Codecov.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| .github/workflows/test-coverage.yml | Removes Xdebug coverage env vars and (currently) comments out the repo guard for running coverage. |
| .github/workflows/reusable-phpunit-tests-v3.yml | Adds PCOV installation/config and updates PHPUnit coverage invocation to output Clover XML only. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| permissions: | ||
| contents: read | ||
| if: ${{ github.repository == 'WordPress/wordpress-develop' }} | ||
| # if: ${{ github.repository == 'WordPress/wordpress-develop' }} |
There was a problem hiding this comment.
This needs to be reverted prior to merge. Can stay in place for now.
desrosj
left a comment
There was a problem hiding this comment.
Thanks! Nice speed improvement! It would be great to reach a point where we can generate and submit coverage reports for every commit. But I'm not sure we're there just yet.
Added some small feedback here, but I have also opened WordPress/wpdev-docker-images#210 to introduce PCOV into the Docker images. I think that's preferable because it not only eliminates the need for some of this installation code, but it also allows for a coverage report to be generated locally using PCOV more easily. Speaking of, it looks like the --coverage-html report is still configured in the test:coverage command within package.json.
If WordPress/wpdev-docker-images#210 is merged, then the .env.example files should be updated to include:
LOCAL_PHP_PCOVset to false with a Docblock explaining what it would be used for.LOCAL_PCOV_FILESthat's commented out by default, but would allow someone to specify a number that's passed topcov.initial.files.
The docker.compose.yml file would need to be updated to set pcov.initial.files to the value of LOCAL_PCOV_FILES when set.
| # Installs PCOV as the code coverage driver for the PHPUnit run below. | ||
| # | ||
| # The INI directives tune PCOV for WordPress's codebase: | ||
| # - `pcov.enabled` keeps the Zend hooks active (this is the default, but | ||
| # stated explicitly for clarity). | ||
| # - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not | ||
| # record hits for `vendor/`, `tests/`, or WordPress test fixtures that | ||
| # PHPUnit would discard at report time anyway. | ||
| # - `pcov.initial.files` pre-sizes the internal file tracking array for | ||
| # the thousands of files under `src/`, avoiding reallocation churn | ||
| # during test warmup. The default of 64 is far too low here. | ||
| - name: Install PCOV coverage driver | ||
| if: ${{ inputs.coverage-report }} | ||
| run: | | ||
| docker compose exec -T -u 0 php sh -c ' | ||
| pecl install pcov && | ||
| docker-php-ext-enable pcov && | ||
| { | ||
| echo "pcov.enabled=1" | ||
| echo "pcov.directory=/var/www/src" | ||
| echo "pcov.initial.files=2000" | ||
| } >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini && | ||
| php -m | grep -i pcov | ||
| ' |
There was a problem hiding this comment.
The setup-php action supports PCOV as a coverage tool. When we are finally able to explore running the unit tests outside of the Docker environment, this becomes simply supplying values to the ini-values input.
| # Installs PCOV as the code coverage driver for the PHPUnit run below. | ||
| # | ||
| # The INI directives tune PCOV for WordPress's codebase: | ||
| # - `pcov.enabled` keeps the Zend hooks active (this is the default, but | ||
| # stated explicitly for clarity). | ||
| # - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not | ||
| # record hits for `vendor/`, `tests/`, or WordPress test fixtures that | ||
| # PHPUnit would discard at report time anyway. | ||
| # - `pcov.initial.files` pre-sizes the internal file tracking array for | ||
| # the thousands of files under `src/`, avoiding reallocation churn | ||
| # during test warmup. The default of 64 is far too low here. | ||
| - name: Install PCOV coverage driver | ||
| if: ${{ inputs.coverage-report }} | ||
| run: | | ||
| docker compose exec -T -u 0 php sh -c ' | ||
| pecl install pcov && | ||
| docker-php-ext-enable pcov && | ||
| { | ||
| echo "pcov.enabled=1" | ||
| echo "pcov.directory=/var/www/src" | ||
| echo "pcov.initial.files=2000" | ||
| } >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini && | ||
| php -m | grep -i pcov | ||
| ' | ||
|
|
There was a problem hiding this comment.
I checked and there are 1,852 PHP files in src/, so this number should work. However, I'm concerned that this number will just end up being set and never updated over time.
What if we add a simple file counting step before this one and round the number of files up to the nearest hundred before using that value for pcov.initial.files?
| # Installs PCOV as the code coverage driver for the PHPUnit run below. | |
| # | |
| # The INI directives tune PCOV for WordPress's codebase: | |
| # - `pcov.enabled` keeps the Zend hooks active (this is the default, but | |
| # stated explicitly for clarity). | |
| # - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not | |
| # record hits for `vendor/`, `tests/`, or WordPress test fixtures that | |
| # PHPUnit would discard at report time anyway. | |
| # - `pcov.initial.files` pre-sizes the internal file tracking array for | |
| # the thousands of files under `src/`, avoiding reallocation churn | |
| # during test warmup. The default of 64 is far too low here. | |
| - name: Install PCOV coverage driver | |
| if: ${{ inputs.coverage-report }} | |
| run: | | |
| docker compose exec -T -u 0 php sh -c ' | |
| pecl install pcov && | |
| docker-php-ext-enable pcov && | |
| { | |
| echo "pcov.enabled=1" | |
| echo "pcov.directory=/var/www/src" | |
| echo "pcov.initial.files=2000" | |
| } >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini && | |
| php -m | grep -i pcov | |
| ' | |
| - name: Count PHP source files | |
| id: count_php_files | |
| if: ${{ inputs.coverage-report }} | |
| run: | | |
| total=$(find src -type f -name '*.php' -print | wc -l | tr -d '[:space:]') | |
| rounded=$(( ( total + 99 ) / 100 * 100 )) | |
| echo "count=$rounded" >> "$GITHUB_OUTPUT" | |
| echo "::info::PHP files under src/: ${total}; pcov.initial.files=${rounded}" | |
| # Installs PCOV as the code coverage driver for the PHPUnit run below. | |
| # | |
| # The INI directives tune PCOV for WordPress's codebase: | |
| # - `pcov.enabled` keeps the Zend hooks active (this is the default, but | |
| # stated explicitly for clarity). | |
| # - `pcov.directory` restricts instrumentation to `src/`, so PCOV does not | |
| # record hits for `vendor/`, `tests/`, or WordPress test fixtures that | |
| # PHPUnit would discard at report time anyway. | |
| # - `pcov.initial.files` pre-sizes the internal file tracking array for | |
| # the rough number of files under `src/`, avoiding reallocation churn | |
| # during test warmup. The default of 64 is far too low here. | |
| - name: Install PCOV coverage driver | |
| if: ${{ inputs.coverage-report }} | |
| env: | |
| PCOV_INITIAL_FILES: ${{ steps.count_php_files.outputs.count }} | |
| run: | | |
| docker compose exec -T -u 0 php sh -c ' | |
| pecl install pcov && | |
| docker-php-ext-enable pcov && | |
| { | |
| echo pcov.enabled=1 | |
| echo pcov.directory=/var/www/src | |
| echo pcov.initial.files=$PCOV_INITIAL_FILES | |
| } >> /usr/local/etc/php/conf.d/docker-php-ext-pcov.ini && | |
| php -m | grep -i pcov | |
| ' | |
There was a problem hiding this comment.
Yeah I ran the count before deciding on 2,000. This setting reserves the given size for a hash table used during the coverage run, if it's exceeded it just means some resizing occurs. I wonder if the find might take longer than the hash table resizing. No clue.
There was a problem hiding this comment.
Claude tells me just to use 10,000 which will allocate ~512KB of memory and is plenty big enough. Sound good?
There was a problem hiding this comment.
Ya, I think works here. But if we choose to go the "PCOV installed in Docker for us" route, we'd have to apply that number there. A number that high also negates the need for a related environment variable in the local environment, which is a positive.
Instead of using Xdebug in coverage mode to generate a coverage report, we can use PCOV.
This has reduced the test coverage step time from around 50 minutes to around 25. A comparable unit test step without coverage takes around 11 minutes. I was expecting the time reduction to be even greater, but it's still a 50% saving.
It might be worth installing PCOV directly into the container images. This PR installs it prior to running the coverage tests but it only takes a few seconds. A task for another time.
Trac ticket: https://core.trac.wordpress.org/ticket/64893
Why has the HTML report generation been removed?
It creates a 400MB artifact that expands to 7GB of HTML files. Many of them are unusable. We've got Codecov for this reason.
Why has the coverage changed?
The Codecov coverage report is showing a -0.5% coverage reduction with this change. The source code itself has not changed, the change comes from the differences in reporting between Xdebug and PCOV. The PCOV docs notes a difference in
switchhandling, for example.What about phpdbg?
PCOV is faster and more accurate than phpdbg, although admittedly there aren't a huge number of comparisons available. Regardless, PHPUnit as of version 10 no longer supports phpdbg for coverage reporting so PCOV is a better choice for future-proofing.
Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Model(s): Opus 4.7
Used for: Initial planning for the switch to PCOV. Tweaked and verified by me.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.