Repository navigation
Conversation
Playwright's --shard deals spec files out by test count, so shard 2 of 8 ran 15 minutes of tests while shard 8 ran 3, and the slowest shard set the length of every full CI run. Each E2E job now asks Playwright for the spec list and picks its share with .github/scripts/e2e-shard.mjs, which packs files by the durations recorded in e2e-shard-times.json. Every shard uploads its JSON report so the times can be refreshed with the script's --record mode.
A directory holding the reports of several runs made --record add a file's times together. The shard script's tests also ran in every shard job after the install and build; the Test Plan job runs them once now.
A spec file that fails to load (a missing import, a duplicate test title, test.only under forbidOnly) leaves the list report with no suites, so the shard script failed with "has no spec files" and the real error never reached the log. The input-order test also gets two files of equal weight, so it covers the name tie-break.
Three of the shard script's tests run it in a temporary folder and left that folder behind on every run. Each now removes its folder when it finishes, as scripts/archive-changelogs.test.mjs does.
|
github-actions
Bot
added
review/needs-review
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
Duration-weighted, file-level sharding is the right fix for the shard imbalance under fullyParallel: false and workers: 1. The allocator is deterministic and dependency-free, and it fits the existing .github/scripts/*.mjs tooling and node --test checks. Both reviewers approved and raised no findings. They checked the shard allocator, the timing recorder, the workflow selection and failure handling, and the Playwright reporter configuration. The timing data covers every current non-table spec. The workflow keeps Node running before Cloudflare, and the Table E2E job stays separate. The review was static, so neither reviewer ran tests or CI.
Reviewed by anthropic/claude-opus-5-5 and openai/gpt-6.1-sol.
🚀 Deploying Preview to Cloudflare 🚀Preview URL: https://daniel-ci-e2e-shard-balance.try.emdashcms.com, https://daniel-ci-e2e-shard-balance-emdash-playground.emdash-cms.workers.dev (commit 6555550)This URL reflects your latest Preview deploymentPreview Deployments by commit
|
What does this PR do?
Gives the eight E2E shards about the same amount of test time. Playwright's
--shardsplits the tests by count, not by time, and withfullyParallel: falseeach shard gets a block of neighbouring spec files. Onmainthat left shard 2 with 15 minutes of tests and shard 8 with 3, and the slowest shard set the length of a full CI run.An E2E job now asks Playwright for the spec list and takes its share from
.github/scripts/e2e-shard.mjs, which packs the files by the durations ine2e-shard-times.json, longest first. A new or renamed spec is on that list too, so it still runs, counted as an average one until its time is recorded. The shards upload Playwright's JSON reports as artifacts, ande2e-shard.mjs --record <dir>refreshes the times from a downloaded set. The tests, the Table E2E job and the order of the Node and Cloudflare steps stay as they are.#4329 changes the same E2E steps and gets updated once this merges.
Type of change
Your connection to this change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges except in translation PRs — a workflow extracts catalogs on merge tomain. (n/a: no admin UI change)AI assistance disclosure
Screenshots / test output
node --test .github/scripts/e2e-shard.test.mjs: 9 tests pass, and the Test Plan job now runs them next toapproval.test.mjs.playwright test --list --reporter=jsontakes about 4 seconds and runs no global setup. With a spec file that can't load (atest.onlyunder CI, a duplicate test title), the pick step fails with Playwright's own error.e2e-timings-<n>artifact. The shards ran 6.3 to 9.8 minutes of tests each, against 3.0 to 15.1 onmain. The slowest E2E job took 14.0 minutes, against 18.3 to 20.1 in the six runs, and the whole run finished 14.6 minutes after it started, against about 20.