Skip to content

[codex] clean up test suite commands#210

Merged
joesobo merged 21 commits into
mainfrom
codex/test-suite-cleanup
May 21, 2026
Merged

[codex] clean up test suite commands#210
joesobo merged 21 commits into
mainfrom
codex/test-suite-cleanup

Conversation

@joesobo
Copy link
Copy Markdown
Owner

@joesobo joesobo commented May 20, 2026

Summary

This PR cleans up and speeds up the monorepo test/CI flow.

  • Simplifies root test scripts around the real test lanes: unit, Playwright, and local-only VS Code validation.
  • Removes release-check and release-package script clutter from the root workflow surface.
  • Drops the separate Playwright lint script; the normal lint lane covers tests.
  • Removes obsolete release test coverage.
  • Splits CI into parallel lanes for build, lint, typecheck, Playwright, and unit tests.
  • Splits monorepo unit tests by package area, then splits extension webview Vitest by behavior group.
  • Names the unit-test matrix checks after the behavior they cover instead of arbitrary shard ids.
  • Keeps VS Code Electron E2E available as a local sanity check without putting it on the PR critical path.
  • Speeds up scoped mutation loops by ignoring package-local VS Code test bundles, narrowing top-level source-file Vitest includes, tuning Stryker runner reuse/concurrency, skipping file-scoped preflight typecheck, printing mutation progress heartbeats, and making scoped targets share Stryker's package-level incremental report.
  • Updates test and quality docs to match the current CI lanes, local VS Code E2E role, and mutation workflow.

CI Speed Evidence

Latest green PR run: https://github.com/joesobo/CodeGraphyV4/actions/runs/26199267945

  • Total wall-clock: 1m03s, from 2026-05-21 01:06:47Z to 01:07:50Z.
  • Baseline before the final CI-speed iterations: 4m55s.
  • Net improvement from that baseline: 3m52s faster, about 79% shorter.
  • Goal was under 3m; current measured result is 1m57s under target.

Latest lane timings:

  • build: 28s
  • lint: 32s
  • typecheck: 26s
  • playwright: 54s
  • Unit tests / Packages: 1m00s
  • Unit tests / Extension node: 25s
  • Unit tests / Extension webview app shell and plugins: 27s
  • Unit tests / Extension webview panels, search, and exports: 28s
  • Unit tests / Extension webview graph interaction and rendering: 23s

Previous iteration checkpoints:

Mutation Loop Evidence

  • Preferred file command works without extra flags: pnpm run mutate -- packages/extension/src/webview/vscodeApi.ts.
  • The direct Vitest files for vscodeApi.ts run 9 tests in about 2.016s wall time, with only about 8ms spent in the test bodies. The mutation loop is dominated by Stryker startup, dry-run, sandbox, and runner orchestration overhead rather than assertion time.
  • The repo uses Stryker's native incremental mode instead of a wrapper-level cache shortcut. Scoped file and directory targets now share the package-level incremental file, for example reports/mutation/extension/stryker-incremental-extension.json.
  • Cold package-level incremental file for vscodeApi.ts: 6 mutants, 100% score, 11.12s wall time.
  • Warm package-level incremental file for vscodeApi.ts: Stryker reused 6 of 6 mutant results, 100% score, 7.51s wall time.
  • Forced rerun: pnpm run mutate -- --force packages/extension/src/webview/vscodeApi.ts retested all 6 mutants at 100% score in 10.67s wall time.
  • File-scoped mutation targets skip the preflight typecheck by default; package and directory targets still run it.
  • The mutation wrapper prints a progress heartbeat every 60 seconds while Stryker is running.
  • pnpm run mutate -- extension/ now clears Stryker's instrumented dry run but did not finish mutant execution within the requested 30-minute cutoff. It timed out at 1800.156s after finding 892 files to mutate and instrumenting 22,525 mutants.
  • Direct mutation-compatible Vitest for the full extension suite passed separately: 868 files, 5,437 tests, 92.32s.
  • pnpm run mutate -- --skip-typecheck --mutate packages/extension/src/webview/components/graph/model/node/display.ts: 49 mutants, 85.42% score, Stryker exited successfully with the existing survivors reported, 29.923s wall time.
  • Direct Stryker timing check on display.ts: one worker took about 32.3s; two workers took about 27.9s on the same 49-mutant target.

Local Validation

  • ruby -ryaml -e 'YAML.load_file(".github/workflows/ci.yml"); puts "workflow yaml ok"'
  • pnpm --filter @codegraphy/extension exec vitest run --config vitest.config.ts --project node packages/extension/tests/vitest.includes.test.ts
  • CODEGRAPHY_VITEST_WEBVIEW_GROUP=graph pnpm exec turbo run test:webview --filter=@codegraphy/extension
  • CODEGRAPHY_VITEST_WEBVIEW_GROUP=appPlugins pnpm exec turbo run test:webview --filter=@codegraphy/extension
  • CODEGRAPHY_VITEST_WEBVIEW_GROUP=panelsExport pnpm exec turbo run test:webview --filter=@codegraphy/extension
  • pnpm --filter @codegraphy/quality-tools exec vitest run --config vitest.config.ts tests/mutation/runner/command.test.ts tests/mutation/runner/run.effects.test.ts tests/mutation/runner/directIncludes.test.ts tests/mutation/runner/vitestIncludes.test.ts
  • pnpm --filter @codegraphy/quality-tools exec vitest run --config vitest.config.ts tests/mutation/runner/run.test.ts tests/mutation/runner/run.effects.test.ts tests/mutation/reporting/reportArtifacts.test.ts tests/mutation/runner/command.test.ts
  • pnpm --filter @codegraphy/quality-tools test
  • pnpm run mutate -- packages/extension/src/webview/vscodeApi.ts
  • pnpm run mutate -- --force packages/extension/src/webview/vscodeApi.ts
  • pnpm run mutate -- extension/ with a 30-minute watchdog: timed out at 1800.156s.
  • CODEGRAPHY_MUTATION_RUN=1 CODEGRAPHY_VITEST_SCOPE=extension pnpm --filter @codegraphy/extension exec vitest run --config vitest.config.ts
  • pnpm run lint
  • pnpm run typecheck
  • pnpm run test
  • Commit hook reran pnpm run typecheck and staged-file ESLint.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 20, 2026

⚠️ No Changeset found

Latest commit: 8a1a8d2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@joesobo joesobo marked this pull request as ready for review May 21, 2026 21:55
@joesobo joesobo merged commit 943f2ef into main May 21, 2026
9 checks passed
@joesobo joesobo deleted the codex/test-suite-cleanup branch May 21, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant