Automated testing

[raineorshine]

Placeholder conversation for the discussion of the relationship between automated testing and coding agents.

[fbmcipher]

Some bullet points on this from our conversation a couple of weeks ago:

• A lot of test writing is rote/boilerplate – AI could help increase test coverage significantly
• Designing tests and choosing which tests properly cover the behavior is not trivial – that shouldn't be taken for granted
  • Raine cautioned about overconfidence: "just because the AI has written a lot of tests, doesn't mean everything is working great. It has a knack, like people, for writing tests to cover just the parts that work"
• Potentially assigning the agent to some of the test engineering backlog
  • Potential tasks include splitting the GitHub Actions workflow into two: snapshot Puppeteer tests and non-snapshot Puppeteer tests - parallelism benefit & allows agents to be more specialized in handling failures
• Raine first wants to make sure tests can actually be run by AI and that agents can see the results — "we're not quite there yet"
  • The Vitest upgrade PR was the concrete example of this gap — all Puppeteer tests timing out and the agent unable to diagnose or fix them
• Increased test coverage will increase our confidence in AI submissions

[fbmcipher]

The Vitest upgrade PR was the concrete example of this gap

We might be close to solving that. The test results in #4180 were really positive – the agent was able to solve 12 of the 13 failing tests in one shot. The last 1 failure required a Puppeteer snapshot update, and for some reason the agent wasn't fully able to do this. Investigating!

[fbmcipher]

A couple of weeks ago, we spoke about how writing test code is fundamentally different to writing application code. In application code we're really interested in abstraction, encapsulation, maintainability, DRY – whereas test code actually benefits from being much more explicit. Repetition is preferred because it makes test code more readable.

Though test writing can be sort of boilerplate, designing tests is still non-trivial. One solution might be a "Test Engineer" agent – that understands best practices about how to write tests, and has access to a headless Chrome instance so it can actually run tests.

An agent feels appropriate here because of how different writing tests is to writing application code – it would be great to be able to separate those two roles – but the inability to @mention custom agents is really painful! It would be great to @mention the test engineer to say "fix this failing test", for example.

So maybe the answer is a set of skills instead – I'm not sure yet!

[fbmcipher]

As an update to this post, we now have a skill called tdd-write-failing-test that writes regression tests for all issues the agent solves.

To make sure that tests conform to the project's standards, docs/testing.md was recently expanded with specific instructions for agents.