Out-of-process tests for the MCP server, with CI and an example agent

.github/workflows/mcp-tests.yml

@@ -0,0 +1,99 @@
+name: MCP integration tests
+
+# Out-of-process MCP testing: spins up a real Echidna campaign with the MCP
+# server and drives it the way real agent clients do. Besides the tool tests it
+# runs the client-compatibility suite:
+#   * test_mcp_conformance.py -- wire-protocol regression guard (202/405/handshake)
+#   * test_mcp_codex.py       -- replays Codex's strict rmcp handshake
+#   * test_mcp_claude.py      -- drives the server with the official `mcp` SDK
+# Both client checks are transport-level and need no API keys, so they run on
+# every push/PR that touches the code they cover. (A live-model smoke using
+# examples/mcp_agent.py needs ANTHROPIC_API_KEY / Codex auth and is
+# intentionally not part of CI.)
+
+# Building the closure and running a campaign is expensive, so this is limited
+# to changes that can affect what the server answers. (GitHub Actions does not
+# support YAML anchors, hence the repetition.)
+on:
+  push:
+    branches:
+    - master
+    paths:
+    - '.github/workflows/mcp-tests.yml'
+    - 'tests/mcp/**'
+    - 'lib/**'
+    - 'src/**'
+    - 'package.yaml'
+    - 'flake.nix'
+    - 'flake.lock'
+  pull_request:
+    branches:
+    - master
+    paths:
+    - '.github/workflows/mcp-tests.yml'
+    - 'tests/mcp/**'
+    - 'lib/**'
+    - 'src/**'
+    - 'package.yaml'
+    - 'flake.nix'
+    - 'flake.lock'
+
+# Least-privilege GITHUB_TOKEN: this workflow only checks out the repo and runs
+# tests, so it needs no write scopes.
+permissions:
+  contents: read
+
+# A new push to the same ref cancels an in-flight run: this one builds the whole
+# closure and then fuzzes, so it is worth not piling up.
+concurrency:
+  group: mcp-tests-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  mcp-client-compat:
+    name: MCP client compatibility (Claude + Codex)
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Install Nix
+        uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
+
+      # Pull prebuilt echidna/hevm/deps from the project's Cachix so the build
+      # takes minutes instead of compiling the whole closure from source.
+      # CACHIX_AUTH_TOKEN is not exposed to pull requests from forks, so those
+      # runs read the public cache and push nothing back: a fork PR that misses
+      # is slow rather than red.
+      - name: Configure Cachix
+        uses: cachix/cachix-action@5f2d7c5294214f71b873db4b969586b980625e71 # v17
+        with:
+          name: trailofbits
+          authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
+
+      - name: Build echidna
+        run: nix build .#echidna --out-link result
+
+      - name: Add echidna to PATH
+        run: echo "$GITHUB_WORKSPACE/result/bin" >> "$GITHUB_PATH"
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.11"
+
+      - name: Install test dependencies
+        run: pip install -r tests/mcp/requirements-test.txt
+
+      # Echidna shells out to crytic-compile + solc to compile the test contract.
+      - name: Install crytic-compile and solc
+        run: |
+          pip install crytic-compile solc-select
+          solc-select install 0.8.25
+          solc-select use 0.8.25
+
+      - name: Run MCP test suite (tools + conformance + Claude + Codex)
+        run: pytest tests/mcp -v

README.md

@@ -318,6 +318,25 @@ $ nix develop # alternatively nix-shell
 [nix-shell]$ cabal new-repl
 ```
 
+### Running the test suites
+
+`cabal run tests` is the Haskell suite, and covers everything Echidna does in
+process.
+
+The MCP server is tested out of process instead, by starting a real campaign and
+driving it the way an agent client would. That suite is Python, and needs an
+`echidna` on `$PATH`:
+
+```sh
+$ pip install -r tests/mcp/requirements-test.txt
+$ pytest tests/mcp -v
+```
+
+Point it at a campaign you started yourself with `ECHIDNA_MCP_URL`, at a
+different port with `ECHIDNA_MCP_PORT`, or at a specific binary with
+`ECHIDNA_BIN`. An example of an agent driving a campaign through the same
+interface is in [examples/](examples/README.md).
+
 ## Public use of Echidna
 
 ### Property testing suites

examples/README.md

@@ -0,0 +1,73 @@
+# Echidna MCP agent example
+
+An agent that connects to a live Echidna campaign through its [MCP](https://modelcontextprotocol.io/)
+server, watches what the fuzzer is reaching, and aims it at what it is not.
+
+## Requirements
+
+```
+pip install langchain-anthropic langgraph httpx
+```
+
+## Usage
+
+**1. Start Echidna with the MCP server:**
+
+```bash
+echidna MyContract.sol --server 8080 --format text
+```
+
+`--format text` is required — the interactive TUI otherwise owns the terminal.
+
+**2. Run the agent:**
+
+```bash
+export ANTHROPIC_API_KEY=your_key_here
+python examples/mcp_agent.py
+```
+
+It polls until you stop it with Ctrl-C. Point it at another campaign with
+`ECHIDNA_MCP_URL=http://127.0.0.1:9000/mcp`.
+
+## What it does
+
+Every 30 seconds it calls `status`. When the campaign has gone a minute without
+finding new coverage, it reads `target` and `show_coverage`, asks Claude for
+call sequences that would reach lines the campaign has not, and injects them
+with `inject_fuzz_transactions` — clearing the previous round first, so the
+fuzzer is not left splitting its budget across every ordering ever suggested.
+
+## The tools the server exposes
+
+Four report on the campaign:
+
+| Tool | Description |
+|------|-------------|
+| `status` | Corpus size, iterations, coverage, failing tests, optimization values, how long since the last coverage and which functions found it, and whatever `sample` is recording |
+| `target` | The contract under test and the functions it exposes |
+| `show_coverage` | One contract's source, line by line, marked with what the campaign reached |
+| `dump_lcov` | Write the coverage so far to an LCOV file |
+
+Five steer it:
+
+| Tool | Description |
+|------|-------------|
+| `inject_fuzz_transactions` | Spend part of the fuzzer's budget on a specific ordering of calls |
+| `clear_fuzz_priorities` | Forget every injected ordering and return to the corpus |
+| `execute_sequence` | Run a concrete sequence and report what each call did, without disturbing the campaign |
+| `sample` | Record what one function does as the campaign calls it; results appear in `status` |
+| `reload_corpus` | Pick up whatever was written to the corpus directory since the campaign started |
+
+`inject_fuzz_transactions` and `execute_sequence` take a sequence written the way
+the calls would be in Solidity, separated by `;`. `inject_fuzz_transactions`
+additionally accepts `?` for an argument the fuzzer should choose:
+
+```
+approve(0x10, ?); transferFrom(?, ?, 100)
+```
+
+## Testing the server
+
+`tests/mcp/` drives a real campaign the way these clients do — tool semantics,
+wire-protocol conformance, and the official MCP SDK. See the
+[README](../README.md#running-the-test-suites) for how to run it.

examples/mcp_agent.py

@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+"""
+LangGraph agent for Echidna's MCP server.
+
+Connects to a running campaign, watches what the fuzzer is reaching, and when
+coverage stalls asks Claude for call sequences to aim it at. A demonstration of
+what the server is for rather than a tuned strategy: the interesting part is
+that `status` and `show_coverage` are enough for a model to decide what to try
+next, and `inject_fuzz_transactions` is enough to act on it.
+
+Requirements:
+    pip install langchain-anthropic langgraph httpx
+
+Usage:
+    # Start Echidna with the MCP server. --format text is required: the
+    # interactive TUI otherwise owns the terminal.
+    echidna MyContract.sol --server 8080 --format text
+
+    # Run the agent:
+    ANTHROPIC_API_KEY=... python examples/mcp_agent.py
+"""
+
+import json
+import os
+import re
+import time
+from typing import TypedDict
+
+import httpx
+from langchain_anthropic import ChatAnthropic
+from langgraph.graph import END, StateGraph
+
+MCP_URL = os.environ.get("ECHIDNA_MCP_URL", "http://127.0.0.1:8080/mcp")
+PROTOCOL_VERSION = "2025-06-18"
+
+# How long the campaign may go without finding coverage before the agent steps
+# in, and how often it looks.
+STALL_SECONDS = 60
+INTERVAL_SECONDS = 30
+
+
+# ---------------------------------------------------------------------------
+# MCP client
+# ---------------------------------------------------------------------------
+
+def _rpc(method: str, params: dict | None = None, request_id: int | None = 1):
+    body = {"jsonrpc": "2.0", "method": method}
+    if request_id is not None:
+        body["id"] = request_id
+    if params is not None:
+        body["params"] = params
+    headers = {
+        "Content-Type": "application/json",
+        "Accept": "application/json, text/event-stream",
+    }
+    if method != "initialize":
+        headers["MCP-Protocol-Version"] = PROTOCOL_VERSION
+    response = httpx.post(MCP_URL, json=body, headers=headers, timeout=60)
+    response.raise_for_status()
+    return response.json() if request_id is not None else None
+
+
+def connect() -> str:
+    """Run the MCP handshake and return the server's name."""
+    result = _rpc("initialize", {
+        "protocolVersion": PROTOCOL_VERSION,
+        "capabilities": {},
+        "clientInfo": {"name": "echidna-mcp-agent", "version": "0"},
+    }, request_id=0)["result"]
+    _rpc("notifications/initialized", request_id=None)
+    return result["serverInfo"]["name"]
+
+
+def call_tool(name: str, arguments: dict | None = None) -> str:
+    """Call a tool and return its report, raising if the tool could not answer."""
+    result = _rpc("tools/call", {"name": name, "arguments": arguments or {}})["result"]
+    text = "".join(part.get("text", "") for part in result.get("content", []))
+    if result.get("isError"):
+        raise RuntimeError(f"{name}: {text}")
+    return text
+
+
+# ---------------------------------------------------------------------------
+# Graph
+# ---------------------------------------------------------------------------
+
+class State(TypedDict):
+    contract: str
+    iterations: int
+    coverage: int
+    stalled_for: int
+
+
+def observe(state: State) -> State:
+    """Read how the campaign is going."""
+    status = json.loads(call_tool("status"))
+    stalled_for = status["time_since_last_coverage_sec"] or 0
+    print(f"  coverage={status['coverage_points']}  "
+          f"iterations={status['iterations']}  "
+          f"last coverage {stalled_for}s ago "
+          f"({', '.join(status['recent_covered_functions'][:3]) or 'nothing yet'})")
+    return {
+        **state,
+        "iterations": status["iterations"],
+        "coverage": status["coverage_points"],
+        "stalled_for": stalled_for,
+    }
+
+
+def inject(state: State) -> State:
+    """Ask Claude which orderings to try, and point the fuzzer at them."""
+    # temperature is not accepted on current models; steer with the prompt.
+    llm = ChatAnthropic(model="claude-opus-4-8", max_tokens=1024)
+
+    abi = call_tool("target")
+    coverage = call_tool("show_coverage", {"contract": state["contract"]})
+
+    answer = llm.invoke(
+        "You are helping an Echidna fuzzing campaign reach code it has not "
+        "reached. It has found no new coverage for a while.\n\n"
+        f"The contract under test exposes:\n{abi}\n\n"
+        f"Its coverage so far — lines marked * were executed, r reverted, "
+        f"o ran out of gas, e errored, blank was never reached:\n"
+        f"{coverage[:8000]}\n\n"
+        "Reply with up to 3 call sequences that would reach unmarked lines, one "
+        "per line and nothing else. Separate the calls in a sequence with ';', "
+        "and write '?' for any argument the fuzzer should choose. For example:\n"
+        "approve(0x10, ?); transferFrom(?, ?, 100)\n"
+    ).content
+
+    # Anything with a call in it; the model's prose, if any, has no parentheses.
+    sequences = [line.strip() for line in answer.splitlines()
+                 if re.search(r"\w+\s*\(", line)]
+
+    # Drop what was injected last time first, so the fuzzer is not left
+    # splitting its budget across every ordering the agent has ever suggested.
+    call_tool("clear_fuzz_priorities")
+    for sequence in sequences[:3]:
+        try:
+            print(f"  injecting: {sequence}")
+            print(f"    {call_tool('inject_fuzz_transactions', {'transactions': sequence})}")
+        except RuntimeError as complaint:
+            print(f"    rejected: {complaint}")
+
+    return state
+
+
+def route(state: State) -> str:
+    return "inject" if state["stalled_for"] >= STALL_SECONDS else END
+
+
+def build_graph():
+    graph = StateGraph(State)
+    graph.add_node("observe", observe)
+    graph.add_node("inject", inject)
+    graph.set_entry_point("observe")
+    graph.add_conditional_edges("observe", route, {"inject": "inject", END: END})
+    graph.add_edge("inject", END)
+    return graph.compile()
+
+
+def main() -> None:
+    if not os.getenv("ANTHROPIC_API_KEY"):
+        print("Set ANTHROPIC_API_KEY before running.")
+        return
+
+    try:
+        print(f"Connected to {connect()} at {MCP_URL}.")
+    except Exception as unreachable:
+        print(f"Cannot reach the MCP server: {unreachable}")
+        print("Start Echidna with:  echidna MyContract.sol --server 8080 --format text")
+        return
+
+    # "Contract: path/to/File.sol:Name" — the name is what show_coverage wants.
+    contract = call_tool("target").splitlines()[0].rsplit(":", 1)[1].strip()
+    print(f"Watching {contract}. Stepping in after {STALL_SECONDS}s without coverage.")
+
+    graph = build_graph()
+    state: State = {"contract": contract, "iterations": 0, "coverage": 0, "stalled_for": 0}
+
+    step = 0
+    while True:
+        step += 1
+        print(f"\n--- step {step} ---")
+        state = graph.invoke(state)
+        time.sleep(INTERVAL_SECONDS)
+
+
+if __name__ == "__main__":
+    main()