Concrete sequence replay with a JSON report - #1606
Open
elopez wants to merge 2 commits into
Open
Conversation
Add ExecuteSequence, the command that answers "what does this sequence of calls actually do?". A caller hands a worker a concrete sequence and gets back a JSON report: what each call was, whether it completed, reverted or failed an assertion, the gas it burned and the events it emitted, then the block number and timestamp the sequence ended on and, if asked for, the EVM trace. The replay goes through execTx rather than callseq, so the campaign is left exactly as it was: no coverage recorded, nothing added to the corpus, no test falsified. Answering a question about the contract must not change what the campaign does next, and a test pins that down. The report summarises the sequence by its worst transaction rather than its first failure. An assertion failure anywhere outranks a revert, even one that happened earlier -- reverts are common enough in a random sequence that reporting one would bury the thing the caller was looking for. Assertion failures are recognised with checkAssertionEvent and checkPanicEvent, the same pair an assertion test uses, so both the emit-AssertionFailed convention and solc's Panic(1) are covered; the tests exercise one path each. The command is addressed to a single worker, which answers through a one-shot Reply channel. The replay runs on that worker's own thread, so it stops fuzzing for as long as the caller is waiting on it. Nothing sends this command yet; the MCP server that does arrives next. Co-authored-by: gustavo-grieco <gustavo.grieco+github@gmail.com>
GHC 9.8 turned head into a -Wall warning, and CI builds the test suite with -Werror, so the single-transaction assertion in the Panic(1) test failed the Windows and Linux builds. Match on the list instead. The test replays exactly one transaction, so pinning that down says what the assertion already assumed and reports a useful failure if it ever stops holding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A caller hands a fuzzing worker a concrete sequence of transactions and gets back a JSON report of what it did. This is what lets something outside the campaign ask "what does this sequence actually do?" without guessing from coverage.
ExecuteSequencejoins the commands a worker accepts over the bus. The replay goes throughexecTxrather thancallseq, so the campaign is left exactly as it was: no coverage recorded, nothing added to the corpus, no test falsified. Answering a question about the contract must not change what the campaign does next, and a test pins that down.executeSeqlands in its ownEchidna.Worker.Replay— nothing about it is fuzz-specific, andWorker/Fuzz.hsis not the place to accumulate it.{"failed_tx_index":3,"final_block_number":"0x42ae50","final_timestamp":"0x5ae26348", "status":"assertion_failed","transaction_count":3,"transactions":[ {"call":"Test.assert_revert(1)","gas_used":21609,"index":1,"logs":[], "result":"Stop","status":"completed"}, {"call":"Test.assert_unreachable()","gas_used":21202,"index":2,"logs":[], "result":"ErrorRevert","status":"reverted"}, {"call":"Test.assert_revert(200)","gas_used":23398,"index":3, "logs":["AssertionFailed(«error») from: 0xa329c0648769a73afac7f9381e08fb43dbea72"], "result":"ErrorRevert","status":"assertion_failed"}]}Asking for the trace adds a
tracefield. Traces are cleared before each transaction unlessallEventsis set, so in the usual case that is the last transaction's trace alone, which is what keeps the cost bounded —showTraceTreeis not cheap.Nothing sends this command yet; the MCP server that does is the next rung.
Notes for review
Four places where this deliberately departs from the
dev-agents-3original:"AssertFail"and"Panic(AbiUInt 256 1)"; neither string is ever produced. Echidna emitsAssertionFailed(...), which does not containAssertFail, and hevm'sShow AbiValuerendersAbiUInt _ nas justn, so the revert text isPanic(1).assertion_failedcould never be reported. This usescheckAssertionEventandcheckPanicEvent "1"fromEchidna.Testinstead — the same paircheckAssertionFailureuses — and a test covers each path."reverted"for a sequence where tx 2 reverted and tx 3 failed an assertion. Reverts are routine in a random sequence, so first-failure buries the finding.TxStatus/TxOutcomereplace the four-tuple accumulator, whosembFailedandmbFailedStatuswere always set together for the same index, and the stringly-typedcase mbFailedStatus of Just "assertion_failed" -> …that read them back.resultusesTxResult's existingToJSONrather thanshow.stripAnsiEscapeCodes, asUI/Widgets.hsalready does for the same reason: whoever reads this report is not a terminal, so the escape codes are only tokens wasted.Also dropped the original's
when (workerId == 0)guard inside the handler —checkMessagesalready filters ontid == workerId, so exactly one worker ever replies.Replyis anewtypeoverTMVarwith an opaqueShow, soFuzzerCmdkeeps its derivedShowinstead of needing a hand-written instance that can drift as constructors are added.Follow-up for the MCP rung
The worker replies on its own thread, so a replay stops that worker fuzzing while the caller waits. More importantly, if the addressed worker has already reached its test limit, nobody replies at all — the MCP tool needs a timeout rather than a bare
takeTMVar.Testing
cabal buildclean, no warnings, no new dependencies.cabal run tests— 195/195, including five newSequence replaycases. One asserts the campaign's coverage stats and corpus size are unchanged across a replay; another checks solc'sPanic(1)path, gated on solc ≥ 0.8.hlint lib srcclean for the files touched.basic/flags.solstill runs and terminates, exercising thecheckMessagessignature change in the hot loop.