Repository navigation
feat: self-test the review workflow on this repository's pull requests - #33
Merged
Merged
Conversation
Author
|
received <token> Generated by Claude Code |
Author
|
Live self-test result for ae7ba0f (run https://github.com/cipherstash/workflows/actions/runs/44453696196):
To fix this, the stub routine's prompt needs to be changed to the one in the README's Self-test section. That is outside this repository. The test runs again when the pull request is reopened, or marked as a draft and then ready again.
Generated by Claude Code |
This was referenced Nov 19, 2026
cipherstash-agent
changed the base branch from
feat/requested-reviewer
to
main
November 19, 2026 08:16
…kflow The review request is about to name the workflow run that sent it, which the decide job builds from the runner's GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_RUN_ID, and the self-test adds a step file that runs from its own workflow rather than the reusable one. The harness only knew the reusable workflow and a fixed set of runner variables, so neither could be tested through it. run_step and run_step_as_written now take runner, which changes, adds or (with None) removes a runner variable, and every helper that reads a workflow takes workflow, defaulting to the reusable one. By default a step now also sees GITHUB_SERVER_URL and GITHUB_RUN_ID for a fixed run, whose link is RUN_URL. Jobs that call a reusable workflow have no steps, so the step listings tolerate that. Refs #29 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
A review request said which pull request to review and why, but not which workflow run asked for it. Without that, a review cannot be traced back to its run, and the self-test (#29) cannot tell its own stub reply from a reply to another run. The decide job now adds run_url, the run's link built from the runner's GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_RUN_ID, to the decision's input, and the request decision adds "Sent by workflow run: <link>" as a third paragraph when that value is a run's link, and leaves it out otherwise. It lives in the decision rather than the fire step so that the saved cases cover it and a comment command's instructions stay the end of the text, which is what stops a ">>>" in them from adding text after the block. Inputs without run_url, such as every existing saved case, give the same text as before. Refs #29 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The review workflow was only tested in parts: the saved decision cases and the step files' unit tests. Nothing checked that the workflow as a whole, as a pull request changes it, still decides, sends a request and reaches a routine. This runs it end to end on every pull request here that becomes ready, against a stub routine that only posts a comment, so a test run is cheap and never starts a real review. The self-test workflow calls the pull request's own claude-review.yml by its local path, with the stub's trigger id variable and token secret, never the real routine's. The reusable workflow now gives its decision and whether the request was sent as outputs, so the verify job can fail with the skip reason, or say the request was not sent, before it waits. It then reads the pull request's comments for up to 5 minutes for one holding "integration-test: received" and this run's link, which the request now ends with, matching the link exactly so another run's reply never counts. That logic is steps/verify.py, unit tested against the stand-in server. Drafts, forks and bots' pull requests get none of the secrets the test needs, so a separate job skips them with a notice instead of failing. Closes #29 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
… author A re-run kept the run id, so the self-test could pass on the reply to an earlier attempt. The run link now names the attempt (<server>/<repo>/actions/runs/<id>/attempts/<n>), which the request decision accepts and verify.py matches exactly. verify.py also ignores replies from anyone but an owner, member or collaborator, checks its deadline between pages, fails at once on a 401, 403 or 404, explains an unsent request that a newer request may have replaced, and warns instead of failing when the job summary cannot be written. The verify job drops the permissions it does not use, and the README says what the self-test does and does not re-run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor 1.32.0 reports self-repository for a `./` reusable workflow call. `$/` resolves to this repository at the exact commit that is running. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
actionlint rejects `uses: $/.github/workflows/claude-review.yml` ("not
following the format owner/repo/path/to/workflow.yml@ref nor
./path/to/workflow.yml"), so the actionlint check failed. Back to `./`,
which zizmor 1.32.0 reports as self-repository.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…s association The verify step timed out on PR #34 although the stub had replied correctly: it accepted only comments whose author_association was OWNER, MEMBER or COLLABORATOR, and the job's token (issues: read) can see another association for an account with private organisation membership. The association is not a reliable signal there. A reply now counts only when its author's login equals REPLY_AUTHOR (compared without regard to case), which the workflow sets to cipherstash-agent, the account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so the check can never accept everyone. A comment with the marker and this attempt's link from another author is noticed (login and association, escaped), and the timeout error says how many comments were read and how many such near-matches were ignored. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor fails on a reusable workflow called by a local path and asks for the `$/` form, which names this repository at the exact commit that is running. actionlint v1.7.12 does not know that form yet, so its one message about it is ignored, in self-test.yml only. The user chose this over a zizmor ignore. This restores 637778a, which was reverted while the choice was open. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
tobyhede
force-pushed
the
feat/self-test-ready
branch
from
November 19, 2026 22:16
98a82be to
bc3c536
Compare
tobyhede
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This repository now tests its own review workflow on its own pull requests, from start to end. When a pull request here is opened, reopened or marked ready for review, a new workflow calls that pull request's own version of the review workflow with a stub Claude routine, which only posts a comment, and then waits for the stub's reply. So a change that breaks deciding, sending or reaching the routine shows as a red check here, before any other repository uses it.
To let the test find its own reply, every review request now ends with a link to the workflow run that sent it. Real reviews get the link too, so a review can be traced back to its run.
Changes
request-decision/decision.jq): takes an optionalrun_urland, when it is a workflow run's link, addsSent by workflow run: <link>as the text's third paragraph. Four new saved cases; the 73 existing ones are unchanged..github/workflows/claude-review.yml): the decide job passes the run's link (fromGITHUB_SERVER_URL,GITHUB_REPOSITORY,GITHUB_RUN_ID) into the decision; newworkflow_calloutputsdecisionandsent..github/workflows/self-test.yml):reviewcalls./.github/workflows/claude-review.ymlwithvars.CLAUDE_ROUTINE_INTEGRATION_TEST_TRIGGER_IDandsecrets.CLAUDE_ROUTINE_INTEGRATION_TEST_TOKEN;verify(contents, issues and pull-requests read only) runs the check;skippedexplains drafts, forks and bots' pull requests in a notice.steps/verify.py): fails naming the skip reason, or saying the request was not sent; otherwise reads the pull request's comments every 15 seconds for up to 5 minutes for one withintegration-test: receivedand this run's exact link.tests/test_verify.py(against the stand-in server),tests/test_decide.py(the decide step as written),tests/test_self_test.py(what the self-test calls, with which routine, and when).tests/harness.pycan now set runner variables and read another workflow.run_urlinput.Verification
Run locally in the worktree:
request-decision/test-runner: all checks ok.request-decision/test: 77 cases, 0 failed.git diffshows only additions underrequest-decision/cases/.python3 -m unittest discover -s tests: 147 tests, OK.ruff check .: all checks passed.mypy: no issues in 18 files.Self-testcheck; its result is reported in a comment below.Related
Closes #29. Part of #1; see also #19.
Stacked on #13 (
feat/requested-reviewer).Review notes
>>>inside them adding text after the block. Existing cases have norun_url, so their text is unchanged.CLAUDE_REVIEW_APP_IDandCLAUDE_REVIEW_APP_PRIVATE_KEYreach this repository. They are passed through under the same names. Pull requests opened bycipherstash-agentneed no membership check, so the test passes without them; a pull request opened by anyone else would fail withmembership-lookup-failedif they are missing.Generated by Claude Code