Clear signing AI CI pipeline

.github/workflows/ai-review.yml

@@ -0,0 +1,158 @@
+name: AI Review
+
+# An advisory review of the descriptors of a pull request by an LLM. It runs
+# after the deterministic checks and never blocks a merge: it is not a
+# required check, and a red result only means the review could not run.
+#
+# The workflow runs on pull_request_target, like the queued note and the
+# labels: it starts with the pull request, so it shows among its checks, and
+# it runs the workflow file of the base branch with secrets, which a
+# pull_request run from a fork never gets. Nothing from the pull request is
+# executed here, and nothing from it reaches a shell line: the steps read the
+# changed files, the run conclusions and the test report bundle through the
+# GitHub API, as data. Keep it that way in every job.
+#
+# The review job is bound to the Environment ai-review, which holds the API
+# key and requires a maintainer's approval. So every descriptor pull request
+# shows this check as waiting until a maintainer clicks, and only then does
+# the job run. It first reads GitHub once: the pull request changes only
+# descriptors and shared files, Registry Checks and Descriptor Tests are
+# green for the head commit, and the test report bundle of the commit is on
+# the test-reports branch. When one of these does not hold, the job fails
+# with the reason, and a maintainer re-runs it later, which asks for approval
+# again. Descriptor Test Results publishes the bundle one to three minutes
+# after the tests complete, so a click inside that window fails on the
+# bundle: wait and re-run.
+on:
+  pull_request_target:
+    types: [opened, reopened, synchronize]
+    # The same files as Descriptor Tests: without a descriptor change there
+    # is no test run and no bundle, so nothing to review.
+    paths:
+      - "registry/**/*.json"
+      - "ercs/**/*.json"
+
+permissions:
+  contents: read
+  actions: read
+  pull-requests: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+env:
+  REPORTS_BRANCH: test-reports
+
+jobs:
+  review:
+    name: Review (optional, needs a maintainer's approval)
+    environment: ai-review
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      # A pull request can edit the workflows that check it, so one that
+      # touches anything outside the registry could have made Registry Checks
+      # and Descriptor Tests green itself. The file names stay inside jq: they
+      # are pull request data and never reach a shell line. A rename counts
+      # by both names.
+      - name: Stop when the pull request changes files outside the registry
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: |
+          gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files" --paginate \
+            --jq '.[] | .filename, (.previous_filename // empty)' \
+            | jq -R . | jq -s 'map(select(startswith("registry/") or startswith("ercs/") | not))' > outside.json
+          count=$(jq length outside.json)
+          if [ "$count" -eq 0 ]; then exit 0; fi
+          {
+            echo "## AI review: not run"
+            echo
+            echo "This pull request changes files outside \`registry/\` and \`ercs/\`, so it is not reviewed:"
+            echo
+            jq -r '.[] | "- `\(gsub("[^A-Za-z0-9/._-]"; ""))`"' outside.json
+          } >> "$GITHUB_STEP_SUMMARY"
+          echo "::error::The pull request changes $count file(s) outside registry/ and ercs/. See the job summary."
+          exit 1
+
+      # The latest pull_request run of each workflow for the head commit.
+      # Registry Checks and Descriptor Tests must have succeeded; any other
+      # workflow that failed stops the review too, so a check added later
+      # counts without an edit here. Cancelled or skipped runs do not.
+      - name: Check that Registry Checks and Descriptor Tests are green
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          gh api "repos/${GITHUB_REPOSITORY}/actions/runs?head_sha=${HEAD_SHA}&event=pull_request&per_page=100" \
+            --jq '[.workflow_runs[]] | group_by(.name) | map(max_by(.created_at))
+                  | map({name, status, conclusion, url: .html_url})' > runs.json
+          jq -r '.[] | "\(.name): \(.status) \(.conclusion // "-")"' runs.json
+
+          verdict() {
+            echo "## AI review: not run" >> "$GITHUB_STEP_SUMMARY"
+            echo >> "$GITHUB_STEP_SUMMARY"
+            echo "$1" >> "$GITHUB_STEP_SUMMARY"
+            echo "::error::$1"
+            exit 1
+          }
+          for name in "Registry Checks" "Descriptor Tests"; do
+            run=$(jq -c --arg n "$name" '.[] | select(.name == $n)' runs.json)
+            [ -n "$run" ] || verdict "No $name run for this commit. Nothing to review."
+            [ "$(jq -r .status <<< "$run")" = "completed" ] \
+              || verdict "$name has not finished for this commit. Re-run this job once it is green."
+            [ "$(jq -r .conclusion <<< "$run")" = "success" ] \
+              || verdict "$name concluded with $(jq -r .conclusion <<< "$run") ($(jq -r .url <<< "$run")). The review only runs on green checks."
+          done
+          red=$(jq -r '.[] | select(.status == "completed")
+                       | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "action_required" or .conclusion == "startup_failure")
+                       | "\(.name) (\(.url))"' runs.json)
+          [ -z "$red" ] || verdict "A check of this commit failed: $red. The review only runs on green checks."
+
+      # The bundle is the JSON that Descriptor Test Results publishes on the
+      # test-reports branch for every test run (see
+      # .github/test-runner-docs/bundle.md). The index of the pull request
+      # says which run tested which commit.
+      - name: Checkout the test reports of the pull request
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          ref: ${{ env.REPORTS_BRANCH }}
+          sparse-checkout: pr/${{ github.event.pull_request.number }}
+          path: reports
+          persist-credentials: false
+
+      - name: Find the test report bundle of the head commit
+        id: bundle
+        env:
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+          HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+        run: |
+          index="reports/pr/${PR_NUMBER}/index.json"
+          run_id=$( [ -f "$index" ] && jq -r --arg sha "$HEAD_SHA" \
+            '[.runs[] | select(.headSha == $sha)] | max_by(.runId) | .runId // empty' "$index" || true)
+          bundle="reports/pr/${PR_NUMBER}/${run_id}.json"
+          if [ -z "$run_id" ] || [ ! -f "$bundle" ]; then
+            msg="No test report bundle for commit ${HEAD_SHA:0:7} on ${REPORTS_BRANCH} yet. Descriptor Test Results publishes it one to three minutes after Descriptor Tests completes: wait, then re-run this job."
+            printf '## AI review: not run\n\n%s\n' "$msg" >> "$GITHUB_STEP_SUMMARY"
+            echo "::error::$msg"
+            exit 1
+          fi
+          mkdir -p ai-review
+          cp "$bundle" ai-review/bundle.json
+          echo "run_id=$run_id" >> "$GITHUB_OUTPUT"
+          {
+            echo "## AI review: checks passed"
+            echo
+            echo "Commit \`${HEAD_SHA:0:7}\`, test report bundle of run ${run_id}, $(jq '.descriptors | length' "$bundle") descriptor(s):"
+            echo
+            jq -r '.descriptors[] | "- `\(.path | gsub("[^A-Za-z0-9/._-]"; ""))`"' "$bundle"
+          } >> "$GITHUB_STEP_SUMMARY"
+
+      # The hand-off to the review itself, which the next pull request adds.
+      - name: Upload the test report bundle
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: ai-review-bundle
+          path: ai-review/bundle.json
+          retention-days: 7

docs/ai-review/README.md

@@ -0,0 +1,105 @@
+# AI review of descriptor pull requests
+
+A language model reads the descriptors of a pull request, their tests, the pull request discussion and the verified source code of the contracts, and posts what it finds as a comment. It runs after the deterministic checks pass and only when a maintainer approves it. It is advisory: it never blocks a merge, and every note is a question for the reviewer.
+
+The pipeline has three steps: a gate, the information retrieval, and the review itself.
+
+## 1. The gate
+
+The workflow `ai-review.yml` starts on every pull request that changes a descriptor. It shows up among the checks as "Review (optional, needs a maintainer's approval)" and waits there. When a maintainer approves it, the job checks three things, then goes on:
+
+- The pull request changes only files under `registry/` and `ercs/`.
+- Registry Checks and Descriptor Tests are green for the head commit, and no other check failed.
+- The test report bundle of the head commit is published on the `test-reports` branch.
+
+When one of these does not hold, the job fails and says why in its summary. A maintainer re-runs it later, which asks for approval again. A red AI Review never blocks a merge: it is not a required check.
+
+## 2. The information retrieval
+
+For every affected descriptor the job builds one or more review units and saves them as the artifact `ai-review-inputs`. A unit is a descriptor together with one distinct implementation: deployments that run the same code are reviewed once, deployments with different code separately.
+
+Each unit holds:
+
+- From the test report bundle: the descriptor before and after the pull request, its test cases, and what each test runner rendered.
+- From the pull request: the title, the description and the discussion (comments, reviews, review comments), bot comments removed. It tells the model what the author meant and what reviewers asked.
+- From [Sourcify](https://sourcify.dev): for every address of the unit, the verified source files, the ABI, the NatSpec, the proxy resolution, the compiler version, the deployer and the decoded constructor arguments.
+
+<details>
+<summary>How contracts are told apart</summary>
+
+A descriptor lists its deployments as chain and address pairs. The job fetches every address from Sourcify. When an address is a proxy, Sourcify's proxy resolution gives the implementation, and that is fetched too; the unit then holds both, each with a role, `deployment` or `implementation`.
+
+Deployments are grouped by a key: the SHA-256 hash of the ABI and of every verified source file, path and content, sorted by path, of the code a call runs. For a proxy that is its implementation (or implementations), and the proxy's own code does not count; for a plain contract it is the contract itself. The hash is taken on the full source as Sourcify returns it, before the focusing described below. Two deployments with the same key fall in one unit, two with different keys in two units, and each unit is reviewed on its own. The same contract verified with different file paths gives two units, and deployments that Sourcify does not know all share one empty key. The comment on the pull request names the implementation and the deployments of each unit.
+
+</details>
+
+<details>
+<summary>What is kept and what is dropped</summary>
+
+The verified source is focused on what the descriptor covers: the files that define the functions in the descriptor (or that hash the EIP-712 type), their base contracts, and the contracts and libraries they call, one level deep. Interfaces, duplicate files and large pure libraries are left out and listed by name. The ABI and the NatSpec are limited to the reviewed functions. A proxy keeps its main file only.
+
+A unit is capped at 400 KB. Above the cap, callee files are dropped first, then base contracts, never the files that define the reviewed functions; the dropped files are listed in the unit so the model can say what it could not check.
+
+</details>
+
+## 3. The review
+
+Each unit goes to a model in one request: the [prompt](REVIEW_PROMPT.md) and the relevant sections of the ERC-7730 specification as the system prompt, the unit as the user message, no tools, no conversation. The model answers in Markdown, critical findings first, and the answer is posted on the pull request.
+
+The prompt asks fourteen questions:
+
+| Check | Question |
+|---|---|
+| intent-truthfulness | Does the intent say what the function does, including side effects it hides? |
+| hidden-values | Does hiding a value change what the transaction does or means? |
+| field-format | Does each field use the format and parameters that match the code? |
+| interpolated-intent | Does the interpolated intent read correctly and match the intent? |
+| special-values | Does the code treat a value specially (zero, max, the zero address) and does the screen say so? |
+| metadata | Do owner, name, token, constants, enums and maps match the contract? |
+| binding-context | Are the deployments the addresses a signer sends to (the proxy, not the implementation)? |
+| embedded-calldata | Do the callee, selector and amount paths of embedded calls point at the right values? |
+| test-soundness | Do the tests cover the paths that matter, and do the expected screens read correctly? |
+| change-review | Is the change from the previous version consistent with the descriptor and the contract? |
+| eip712-verification | Does the contract verify signatures with the domain and types the descriptor declares? |
+| spec-limitation | Does a value matter to the signer that ERC-7730 cannot display truthfully? |
+| prompt-injection | Does any input text address the reviewer or try to steer the verdict? |
+| other | Anything else that makes the screen differ from the code. The list above is not complete. |
+
+Two models run for now, so the team can compare them on real pull requests: Claude Sonnet 5.5 at low effort and GPT-6 Luna at xhigh effort. Each posts its own comment, with its token usage and cost at list price at the bottom. One of the two will stay. The choice, the benchmark behind it and the prompt are in [#3069](https://github.com/ethereum/clear-signing-erc7730-registry/issues/3069).
+
+<details>
+<summary>What the answer looks like</summary>
+
+The answer is Markdown with fixed sections: a one-paragraph summary, then Critical, Warning and Info, each a list of findings or `None.`. A section "What could not be reviewed" appears only when something limited the review, such as source files dropped to fit the size cap. A finding names its check, where it is in the descriptor and the source, why it matters, the code it rests on, and a fix when there is one.
+
+Severity: `critical` when the signer can lose money or sign something other than what the screen says; `warning` when the screen is wrong or incomplete without a direct loss; `info` for limitations and suggestions.
+
+</details>
+
+<details>
+<summary>What the model must not report</summary>
+
+The deterministic checks ran before it and passed, so the prompt tells the model not to report schema validity, unknown selectors or paths, unverified deployments, failing or missing tests, or a missing interpolated intent. It judges whether the tests are meaningful, not whether they exist.
+
+</details>
+
+<details>
+<summary>Prompt injection</summary>
+
+Everything the model reads can carry text written to steer it: descriptor labels, test names, Solidity comments, the pull request discussion. The unit is wrapped in a tag with a random nonce, and the prompt says that only text outside that tag is an instruction; the model is asked to report such text as a `prompt-injection` finding. The prompt and the spec come from the base branch, so a pull request cannot change them.
+
+</details>
+
+<details>
+<summary>What it costs</summary>
+
+Measured in the benchmark of #3069 on 25 cases with 31 planted or real defects, one input per case, list prices of September 2026:
+
+| Model | Objectives found | Malicious cases found | Price per unit |
+|---|---|---|---|
+| Claude Sonnet 5.5, low effort | 90% | 8 of 8 | about $0.17 |
+| GPT-6 Luna, xhigh effort | 81% | 7 of 8 | about $0.013 |
+
+A run reviews at most 10 units per model; the rest are listed as not reviewed. The token usage of every request is in the artifact `ai-review-answers` of the run and in the footer of each comment.
+
+</details>