Out-of-process tests for the MCP server, with CI and an example agent

[elopez]

The MCP server is the one part of Echidna that another program talks to, and
the Haskell suite cannot say whether that conversation works: it runs in
process, while what matters here is what goes over the wire and what the tools
answer about a campaign that is actually fuzzing. So this suite starts one.

tests/mcp/conftest.py launches a real echidna against a fixture contract,
waits for the server to accept an MCP initialize rather than merely for the
socket to open, and hands the same campaign to every module. Four layers ride
on it:

• test_mcp.py — the nine tools: what each reports about a live campaign
  and what it says when it cannot answer. status while the campaign fuzzes,
  coverage marked up line by line, an LCOV file that is really on disk, an
  injected sequence the workers keep fuzzing through, a replayed sequence
  reported call by call including the revert reason, a sampled function whose
  counts appear in status a few seconds later. Anything that steers the
  campaign is undone afterwards, so no test depends on the order it ran in.
• test_mcp_conformance.py — the regression guard for "real clients can
  still connect": a notification gets 202 with no body, a GET asking for a
  server stream gets 405 rather than the JSON blob a strict client tries to
  read as SSE, initialize negotiates a version. These are exactly the two
  behaviours mcp-server 0.2.0.1 fixed, and the reason that version is the
  floor.
• test_mcp_codex.py — replays Codex's rmcp handshake step by step.
• test_mcp_claude.py — drives the server with the reference mcp SDK
  through a real session.

Both client checks are transport-level and need no API key, so both run in CI.

examples/mcp_agent.py is the live-model counterpart, and is not in CI: it
reads status, and when the campaign has gone a minute without finding
coverage it hands Claude the ABI and the coverage report and injects the
sequences that come back. A demonstration that those tools are enough to close
the loop, not a tuned strategy.

Running it

pip install -r tests/mcp/requirements-test.txt
pytest tests/mcp -v

ECHIDNA_BIN picks the binary, ECHIDNA_MCP_PORT the port, and
ECHIDNA_MCP_URL points the transport suites at a campaign you started
yourself. Also documented in the README, which previously said nothing about
this suite — the workflow was the only entry point.

Adapted, not cherry-picked

The tool tests are written against the server as it is rather than the
prototype they come from: status answers JSON, show_coverage takes a
contract, and a tool that cannot do what it was asked answers with an MCP
error result. They assert on all three, where the originals asserted that some
substring appeared somewhere in the text. They also go through the shared wire
helpers after a handshake, rather than a second raw client that posted
tools/call with no handshake and no headers — asserting leniency next to a
suite that asserts strictness said two different things about what the server
promises.

The fixture contracts lost their spec-kit headers, two properties that could
not fail (one returned a literal true), and the test prefix on their entry
points, which means something else in Foundry mode. What is left is a contract
that gives the tools something to report on: calls that succeed, calls that
revert with a reason, and properties that hold so the campaign keeps running
for as long as the session needs it.

tests/mcp/requirements.txt is gone: nothing referenced it, and it
contradicted both its sibling and the example's README on what to install. So
is pytest.ini, which was only read when pytest was given the directory as an
argument — the one async test carries an explicit marker instead, which works
either way.

Worth a look

• The mcp SDK's 2.0 is a breaking change for this test. It renamed the
  transport helper, dropped the third stream it used to yield, and moved its
  result models to snake_case (serverInfo → server_info). Rather than
  straddle both majors, requirements-test.txt follows the current one and the
  test skips itself on an older SDK. The other pins are bounded by major
  version only — a compatibility suite should be running against what clients
  actually ship, not against whatever was current when it was written.
• CI cost. The workflow builds the whole closure and then fuzzes, so it
  triggers on master and on pull requests against it, like every other
  workflow here, and only for paths that can change what the server answers.
  Its Cachix token is unavailable to pull requests from forks, so those runs
  read the public cache and push nothing; a miss makes them slow rather than
  red. That is noted in the workflow where it will be read.

Verification

pytest tests/mcp -v — 24 passed, against a campaign the fixture starts
itself. The example was driven against a live campaign as well: handshake,
status, target, show_coverage, the error path, and the injection path
with the model stubbed out.