AI review: collect the inputs from the bundle and Sourcify

[marcocastignoli]

Step 2 of #3057, onto ci/ai-review.

After the checks of step 1, the job runs .github/scripts/ai-review-collect.js from the base branch: for every deployment of every affected descriptor it fetches the verified sources, the ABI, the NatSpec and the proxy resolution from Sourcify (fields=all, one call per address, two at a time, a shared pause on 429), follows a proxy to its implementations, and decodes the constructor arguments with viem. Deployments that share an implementation source become one review unit; deployments with different code get their own, because two chains can run different code behind one descriptor.

Each unit is one JSON file: the descriptor at head and base, its format keys, its test cases and results from the bundle (rendered screens dropped where they equal the expected ones), the embedded calldata formats it declares, and the contracts with the subset of the Sourcify response the model needs. Bytecode, compiler input and output, metadata and storage layout are not kept. An input above the size cap (1.5 MB by default) loses its largest source files first, never the main file of a contract, and lists what it dropped.

The inputs are the artifact ai-review-inputs, kept 30 days. The bundle is no longer uploaded on its own. The same script runs locally from a bundle file (--bundle <file> --out <dir>), which is how step 3, the price simulation, gets its inputs.

Not in this pull request: the sources of the callees of embedded calldata formats, which need the raw transactions of the test cases; and the trimmed spec, which belongs to the system prompt of step 4.

Tested locally

• The fork's test pull request (one EIP-712 descriptor, one deployment): one input, 41 KB, constructor argument decoded.
• feat(symbiotic): add vault, RFQ, rewards and V2 curator descriptors #2990, Symbiotic, the worst case: 73 inputs for 66 descriptors, 26 MB in total, the largest 726 KB, nothing dropped, every contract verified. Seven descriptors split into several units because their implementations differ between chains. Without a Sourcify token, two 429 pauses.

A run on the fork with this version is linked in the comments.

🤖 Generated with Claude Code

[marcocastignoli]

Run on the fork with this version, through the real trigger and the approval: run 36441059487 on fork PR #2. Green 26 seconds after the click: checks, scripts from the base branch, npm ci --ignore-scripts, the Sourcify fetch, and the artifact ai-review-inputs (one input, the QuickSwap permit descriptor, 54 KB).

Posted with Claude Code

[marcocastignoli]

• how do we differentiate between different contracts
• we should include also comments and pr description

[marcocastignoli]

how do we differentiate between different contracts? this is just a question

Every address of the descriptor is fetched from Sourcify; a proxy also brings its implementation, and each contract carries a role, deployment or implementation. Deployments are then grouped by a key: the SHA-256 of the ABI and of every verified source file (path and content, sorted by path) of the code a call runs, so the implementation for a proxy and the contract itself otherwise. Same key, one review unit; different keys, one unit each, reviewed on its own, and the comment names the implementation and the deployments of each unit. Two side effects worth knowing: the same code verified under different file paths gives two units, and deployments Sourcify does not know share one empty key. The docs page in #3071 spells this out.

we should include also comments and pr description

Done in #3081, stacked on this one: the workflow reads the title, description, comments, reviews and review comments through the API into a JSON file, and the collector adds them to every unit as pullRequest, bot comments removed.

This branch also gained the commit that focuses the sources on the reviewed functions (b155f40), which the benchmark of #3069 was run with.

Posted with Claude Code