Skip to content

feat: self-test the review workflow on this repository's pull requests - #33

Merged
tobyhede merged 8 commits into
mainfrom
feat/self-test-ready
Nov 19, 2026
Merged

tobyhede merged 8 commits into
mainfrom
feat/self-test-ready

Conversation

@cipherstash-agent

Copy link
Copy Markdown

Summary

This repository now tests its own review workflow on its own pull requests, from start to end. When a pull request here is opened, reopened or marked ready for review, a new workflow calls that pull request's own version of the review workflow with a stub Claude routine, which only posts a comment, and then waits for the stub's reply. So a change that breaks deciding, sending or reaching the routine shows as a red check here, before any other repository uses it.

To let the test find its own reply, every review request now ends with a link to the workflow run that sent it. Real reviews get the link too, so a review can be traced back to its run.

Changes

  • Request decision (request-decision/decision.jq): takes an optional run_url and, when it is a workflow run's link, adds Sent by workflow run: <link> as the text's third paragraph. Four new saved cases; the 73 existing ones are unchanged.
  • Review workflow (.github/workflows/claude-review.yml): the decide job passes the run's link (from GITHUB_SERVER_URL, GITHUB_REPOSITORY, GITHUB_RUN_ID) into the decision; new workflow_call outputs decision and sent.
  • Self-test (.github/workflows/self-test.yml): review calls ./.github/workflows/claude-review.yml with vars.CLAUDE_ROUTINE_INTEGRATION_TEST_TRIGGER_ID and secrets.CLAUDE_ROUTINE_INTEGRATION_TEST_TOKEN; verify (contents, issues and pull-requests read only) runs the check; skipped explains drafts, forks and bots' pull requests in a notice.
  • Check (steps/verify.py): fails naming the skip reason, or saying the request was not sent; otherwise reads the pull request's comments every 15 seconds for up to 5 minutes for one with integration-test: received and this run's exact link.
  • Tests: tests/test_verify.py (against the stand-in server), tests/test_decide.py (the decide step as written), tests/test_self_test.py (what the self-test calls, with which routine, and when). tests/harness.py can now set runner variables and read another workflow.
  • README: a Self-test section (what it does, its settings, the stub's prompt, how to read a failure), the new outputs, and the run_url input.

Verification

Run locally in the worktree:

  • request-decision/test-runner: all checks ok.
  • request-decision/test: 77 cases, 0 failed. git diff shows only additions under request-decision/cases/.
  • python3 -m unittest discover -s tests: 147 tests, OK.
  • ruff check .: all checks passed. mypy: no issues in 18 files.
  • Each new structural test was checked to fail against a broken workflow (real token name, wrong condition) before being kept.
  • Not run locally: actionlint, shellcheck, zizmor and the actions-lock check (not installed here); CI runs them.
  • The live self-test run is this pull request's own Self-test check; its result is reported in a comment below.

Related

Closes #29. Part of #1; see also #19.
Stacked on #13 (feat/requested-reviewer).

Review notes

  • I put the run link in the request decision, not in the fire step as first suggested. The issue asks for the saved decision cases to cover it, and it keeps a comment command's instructions block as the last thing in the text, which is the documented protection against a >>> inside them adding text after the block. Existing cases have no run_url, so their text is unchanged.
  • Open point: I cannot see whether CLAUDE_REVIEW_APP_ID and CLAUDE_REVIEW_APP_PRIVATE_KEY reach this repository. They are passed through under the same names. Pull requests opened by cipherstash-agent need no membership check, so the test passes without them; a pull request opened by anyone else would fail with membership-lookup-failed if they are missing.
  • A full re-run keeps the same run link, so it can find an earlier attempt's reply. The README says so.
  • Only Ready requests are tested; Re-run the self-test by requesting the review agent #30 and Self-test the comment command #31 add the requested reviewer and the comment command.

Generated by Claude Code

Copy link
Copy Markdown
Author

received <token>


Generated by Claude Code

Copy link
Copy Markdown
Author

Live self-test result for ae7ba0f (run https://github.com/cipherstash/workflows/actions/runs/44453696196):

  • review / decide: fire (Ready, by @cipherstash-agent, no membership check). The request text ended with Sent by workflow run: https://github.com/cipherstash/workflows/actions/runs/44453696196.
  • review / fire: sent, HTTP 200 from the stub's trigger.
  • The stub replied as cipherstash-agent at 03:47:05Z with the text received &lt;token&gt; (then the Claude Code footer). It does not follow the expected prompt, so it holds neither the marker nor the run link.
  • verify: failed after 300 seconds with "No reply from the stub routine …", as designed.

To fix this, the stub routine's prompt needs to be changed to the one in the README's Self-test section. That is outside this repository. The test runs again when the pull request is reopened, or marked as a draft and then ready again.

zizmor also fails on this pull request, with one finding: self-repository on uses: ./.github/workflows/claude-review.yml in self-test.yml. It suggests the $/... form instead. This is still open; see the review notes.


Generated by Claude Code

…kflow

The review request is about to name the workflow run that sent it, which
the decide job builds from the runner's GITHUB_SERVER_URL,
GITHUB_REPOSITORY and GITHUB_RUN_ID, and the self-test adds a step file
that runs from its own workflow rather than the reusable one. The harness
only knew the reusable workflow and a fixed set of runner variables, so
neither could be tested through it.

run_step and run_step_as_written now take runner, which changes, adds or
(with None) removes a runner variable, and every helper that reads a
workflow takes workflow, defaulting to the reusable one. By default a step
now also sees GITHUB_SERVER_URL and GITHUB_RUN_ID for a fixed run, whose
link is RUN_URL. Jobs that call a reusable workflow have no steps, so the
step listings tolerate that.

Refs #29

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
A review request said which pull request to review and why, but not which
workflow run asked for it. Without that, a review cannot be traced back
to its run, and the self-test (#29) cannot tell its own stub reply from a
reply to another run.

The decide job now adds run_url, the run's link built from the runner's
GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_RUN_ID, to the decision's
input, and the request decision adds "Sent by workflow run: <link>" as a
third paragraph when that value is a run's link, and leaves it out
otherwise. It lives in the decision rather than the fire step so that the
saved cases cover it and a comment command's instructions stay the end of
the text, which is what stops a ">>>" in them from adding text after the
block. Inputs without run_url, such as every existing saved case, give the
same text as before.

Refs #29

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The review workflow was only tested in parts: the saved decision cases and
the step files' unit tests. Nothing checked that the workflow as a whole,
as a pull request changes it, still decides, sends a request and reaches a
routine. This runs it end to end on every pull request here that becomes
ready, against a stub routine that only posts a comment, so a test run is
cheap and never starts a real review.

The self-test workflow calls the pull request's own claude-review.yml by
its local path, with the stub's trigger id variable and token secret, never
the real routine's. The reusable workflow now gives its decision and
whether the request was sent as outputs, so the verify job can fail with
the skip reason, or say the request was not sent, before it waits. It then
reads the pull request's comments for up to 5 minutes for one holding
"integration-test: received" and this run's link, which the request now
ends with, matching the link exactly so another run's reply never counts.
That logic is steps/verify.py, unit tested against the stand-in server.

Drafts, forks and bots' pull requests get none of the secrets the test
needs, so a separate job skips them with a notice instead of failing.

Closes #29

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
… author

A re-run kept the run id, so the self-test could pass on the reply to an
earlier attempt. The run link now names the attempt
(<server>/<repo>/actions/runs/<id>/attempts/<n>), which the request decision
accepts and verify.py matches exactly. verify.py also ignores replies from
anyone but an owner, member or collaborator, checks its deadline between
pages, fails at once on a 401, 403 or 404, explains an unsent request that a
newer request may have replaced, and warns instead of failing when the job
summary cannot be written. The verify job drops the permissions it does not
use, and the README says what the self-test does and does not re-run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor 1.32.0 reports self-repository for a `./` reusable workflow call.
`$/` resolves to this repository at the exact commit that is running.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
actionlint rejects `uses: $/.github/workflows/claude-review.yml` ("not
following the format owner/repo/path/to/workflow.yml@ref nor
./path/to/workflow.yml"), so the actionlint check failed. Back to `./`,
which zizmor 1.32.0 reports as self-repository.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…s association

The verify step timed out on PR #34 although the stub had replied correctly: it
accepted only comments whose author_association was OWNER, MEMBER or
COLLABORATOR, and the job's token (issues: read) can see another association for
an account with private organisation membership. The association is not a
reliable signal there.

A reply now counts only when its author's login equals REPLY_AUTHOR (compared
without regard to case), which the workflow sets to cipherstash-agent, the
account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so
the check can never accept everyone. A comment with the marker and this
attempt's link from another author is noticed (login and association, escaped),
and the timeout error says how many comments were read and how many such
near-matches were ignored.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor fails on a reusable workflow called by a local path and asks for the
`$/` form, which names this repository at the exact commit that is running.
actionlint v1.7.12 does not know that form yet, so its one message about it
is ignored, in self-test.yml only. The user chose this over a zizmor ignore.

This restores 637778a, which was reverted while the choice was open.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede force-pushed the feat/self-test-ready branch from 98a82be to bc3c536 Compare November 19, 2026 22:16
@tobyhede
tobyhede merged commit 2f7f9f8 into main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-test the review workflow on this repository's pull requests

3 participants