Repository navigation
feat: re-run the self-test by requesting the review agent - #34
Merged
Merged
Conversation
Author
|
integration-test: received ready request from https://github.com/cipherstash/workflows/actions/runs/44456449461/attempts/1 Generated by Claude Code |
cipherstash-agent
pushed a commit
that referenced
this pull request
Nov 19, 2026
…s association The verify step timed out on PR #34 although the stub had replied correctly: it accepted only comments whose author_association was OWNER, MEMBER or COLLABORATOR, and the job's token (issues: read) can see another association for an account with private organisation membership. The association is not a reliable signal there. A reply now counts only when its author's login equals REPLY_AUTHOR (compared without regard to case), which the workflow sets to cipherstash-agent, the account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so the check can never accept everyone. A comment with the marker and this attempt's link from another author is noticed (login and association, escaped), and the timeout error says how many comments were read and how many such near-matches were ignored. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
This was referenced Nov 19, 2026
tobyhede
pushed a commit
that referenced
this pull request
Nov 19, 2026
…s association The verify step timed out on PR #34 although the stub had replied correctly: it accepted only comments whose author_association was OWNER, MEMBER or COLLABORATOR, and the job's token (issues: read) can see another association for an account with private organisation membership. The association is not a reliable signal there. A reply now counts only when its author's login equals REPLY_AUTHOR (compared without regard to case), which the workflow sets to cipherstash-agent, the account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so the check can never accept everyone. A comment with the marker and this attempt's link from another author is noticed (login and association, escaped), and the timeout error says how many comments were read and how many such near-matches were ignored. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
cipherstash-agent
force-pushed
the
feat/self-test-rerun
branch
from
November 19, 2026 22:20
239c601 to
5766b5a
Compare
Stacked pull requests often get new commits after they are opened, and pushing does not run the self-test again. Requesting, or re-requesting, a review from cipherstash-agent now runs it on the pull request's latest commit, and tests the requested-reviewer kind of review request, which the self-test did not cover. A request for any other reviewer, or for a team, starts no job at all, not even the skipped note, so ordinary review requests add nothing to the checks. A draft is tested on a reviewer request, as the review workflow takes those on drafts; forks and bot-authored pull requests are still skipped with a note. The verify step now also checks that the decision's kind matches the event (Ready for opened, reopened and ready_for_review; requested-reviewer for review_requested), so a wrong decision cannot pass just because the stub replied. The job conditions are tested by evaluating them over sample events rather than by comparing their text. The README says how to re-run the test, and that GitHub does not let anyone request a review from a pull request's own author, so pull requests opened by cipherstash-agent still need draft then ready. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…request The review workflow skips a request on a closed pull request with the reason `closed`, so the verify job went red for a harmless event. The skipped and review jobs now also require the pull request to be open for a review_requested event, so none of them starts. The conditions' tests cover the closed case. Also documents, in the README, that a request by a bot, the review agent or a non-member fails the test by design (reasons bot, review-agent, not-a-team-member), and that a pull request opened with the agent already in Reviewers sends two events, where a third request replaces a waiting one and its verify reports the request as not sent. A new test runs the skipped job's DRAFT expression and script, which nothing exercised, and checks the reason text for a draft, a fork and a bot. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
tobyhede
force-pushed
the
feat/self-test-rerun
branch
from
November 19, 2026 22:30
5766b5a to
ff6871c
Compare
tobyhede
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
You can now run the self-test again on a pull request by requesting, or re-requesting, a review from
cipherstash-agent. This tests the pull request's latest commit, which matters for stacked pull requests that get new commits after they are opened. It also tests the requested-reviewer kind of review request, which the self-test did not cover before.Changes
.github/workflows/self-test.ymlalso runs onreview_requested.reviewjob runs when the requested reviewer iscipherstash-agent(GitHub compares the login ignoring case, as the review workflow does). It runs on drafts too, because the review workflow takes a requested reviewer on a draft.skippednote job. I chose this so that ordinary review requests add nothing to the pull request's checks.verifystep gets the event's action asEVENT_ACTION.steps/verify.pychecks that the decision's kind matches the event:readyfor opened, reopened and ready_for_review, andrequested-reviewerfor review_requested. A wrong kind fails before any comments are read, naming the kind. An action the test does not run on also fails.tests/test_self_test.pynow checks theskippedandreviewjobs' conditions by evaluating them over sample events (ready events, drafts, forks, bot authors, the review agent in different case, other users, teams). Before, it compared the condition text. The evaluator is small and covers only the parts of GitHub's expression language that these conditions use. A test also checks that the login in the conditions matches the review workflow'sreview-agentdefault.tests/test_verify.pyadds cases for each kind of request, for a wrong kind, and for an unknown action.cipherstash-agent; for those, mark the pull request as a draft and then as ready.No saved decision case changed.
request-decision/decision.jqand.github/workflows/claude-review.ymlare unchanged.Verification
Run locally on 8e99c84:
request-decision/test-runner: passedrequest-decision/test: 79 cases, 0 failedpython3 -m unittest discover -s tests: 166 tests, OKruff check .: passedmypy: no issues in 18 source filesactionlint and zizmor are not installed locally, so they run in CI only. Opening this pull request runs the self-test's Ready path live. That run does not test the reviewer-request path: this pull request is opened by
cipherstash-agent, and GitHub does not let you request a review from a pull request's author.Related
feat/self-test-ready)Review notes
self-repositoryfinding onuses: ./.github/workflows/claude-review.ymlcomes from feat: self-test the review workflow on this repository's pull requests #33 and is expected here. That decision is still open and is not changed in this pull request.cipherstash-agent, the review workflow skips it with the reasonbot, so the test fails. I did not add a sender check, because nothing here requests reviews as a bot.Generated by Claude Code