Skip to content

feat: re-run the self-test by requesting the review agent - #34

Merged
tobyhede merged 2 commits into
mainfrom
feat/self-test-rerun
Nov 19, 2026
Merged

tobyhede merged 2 commits into
mainfrom
feat/self-test-rerun

Conversation

@cipherstash-agent

Copy link
Copy Markdown

Summary

You can now run the self-test again on a pull request by requesting, or re-requesting, a review from cipherstash-agent. This tests the pull request's latest commit, which matters for stacked pull requests that get new commits after they are opened. It also tests the requested-reviewer kind of review request, which the self-test did not cover before.

Changes

  • .github/workflows/self-test.yml also runs on review_requested.
    • The review job runs when the requested reviewer is cipherstash-agent (GitHub compares the login ignoring case, as the review workflow does). It runs on drafts too, because the review workflow takes a requested reviewer on a draft.
    • A request for any other reviewer, or for a team, starts no job at all, not even the skipped note job. I chose this so that ordinary review requests add nothing to the pull request's checks.
    • Forks and pull requests opened by bots are still skipped with a note, for every event. For a reviewer request, the draft reason never applies.
    • The verify step gets the event's action as EVENT_ACTION.
  • steps/verify.py checks that the decision's kind matches the event: ready for opened, reopened and ready_for_review, and requested-reviewer for review_requested. A wrong kind fails before any comments are read, naming the kind. An action the test does not run on also fails.
  • tests/test_self_test.py now checks the skipped and review jobs' conditions by evaluating them over sample events (ready events, drafts, forks, bot authors, the review agent in different case, other users, teams). Before, it compared the condition text. The evaluator is small and covers only the parts of GitHub's expression language that these conditions use. A test also checks that the login in the conditions matches the review workflow's review-agent default.
  • tests/test_verify.py adds cases for each kind of request, for a wrong kind, and for an unknown action.
  • README, Self-test section: the new trigger, the skip rules, the new failure messages, and a "Running the test again" subsection. It explains that GitHub does not let you request a review from a pull request's own author. So this way works only on pull requests not opened by cipherstash-agent; for those, mark the pull request as a draft and then as ready.

No saved decision case changed. request-decision/decision.jq and .github/workflows/claude-review.yml are unchanged.

Verification

Run locally on 8e99c84:

  • request-decision/test-runner: passed
  • request-decision/test: 79 cases, 0 failed
  • python3 -m unittest discover -s tests: 166 tests, OK
  • ruff check .: passed
  • mypy: no issues in 18 source files

actionlint and zizmor are not installed locally, so they run in CI only. Opening this pull request runs the self-test's Ready path live. That run does not test the reviewer-request path: this pull request is opened by cipherstash-agent, and GitHub does not let you request a review from a pull request's author.

Related

Review notes

  • The zizmor self-repository finding on uses: ./.github/workflows/claude-review.yml comes from feat: self-test the review workflow on this repository's pull requests #33 and is expected here. That decision is still open and is not changed in this pull request.
  • If a bot requests a review from cipherstash-agent, the review workflow skips it with the reason bot, so the test fails. I did not add a sender check, because nothing here requests reviews as a bot.
  • A reviewer request always needs the membership check for the person who requested it, so it depends on the App secrets reaching this repository (see the README's Settings).

Generated by Claude Code

Copy link
Copy Markdown
Author

integration-test: received ready request from https://github.com/cipherstash/workflows/actions/runs/44456449461/attempts/1


Generated by Claude Code

cipherstash-agent pushed a commit that referenced this pull request Nov 19, 2026
…s association

The verify step timed out on PR #34 although the stub had replied correctly: it
accepted only comments whose author_association was OWNER, MEMBER or
COLLABORATOR, and the job's token (issues: read) can see another association for
an account with private organisation membership. The association is not a
reliable signal there.

A reply now counts only when its author's login equals REPLY_AUTHOR (compared
without regard to case), which the workflow sets to cipherstash-agent, the
account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so
the check can never accept everyone. A comment with the marker and this
attempt's link from another author is noticed (login and association, escaped),
and the timeout error says how many comments were read and how many such
near-matches were ignored.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede added this pull request to stack #36 November 19, 2026 22:16
tobyhede pushed a commit that referenced this pull request Nov 19, 2026
…s association

The verify step timed out on PR #34 although the stub had replied correctly: it
accepted only comments whose author_association was OWNER, MEMBER or
COLLABORATOR, and the job's token (issues: read) can see another association for
an account with private organisation membership. The association is not a
reliable signal there.

A reply now counts only when its author's login equals REPLY_AUTHOR (compared
without regard to case), which the workflow sets to cipherstash-agent, the
account the stub posts as. An empty or missing REPLY_AUTHOR fails at once, so
the check can never accept everyone. A comment with the marker and this
attempt's link from another author is noticed (login and association, escaped),
and the timeout error says how many comments were read and how many such
near-matches were ignored.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Base automatically changed from feat/self-test-ready to main November 19, 2026 22:30
Stacked pull requests often get new commits after they are opened, and
pushing does not run the self-test again. Requesting, or re-requesting,
a review from cipherstash-agent now runs it on the pull request's
latest commit, and tests the requested-reviewer kind of review request,
which the self-test did not cover.

A request for any other reviewer, or for a team, starts no job at all,
not even the skipped note, so ordinary review requests add nothing to
the checks. A draft is tested on a reviewer request, as the review
workflow takes those on drafts; forks and bot-authored pull requests
are still skipped with a note.

The verify step now also checks that the decision's kind matches the
event (Ready for opened, reopened and ready_for_review;
requested-reviewer for review_requested), so a wrong decision cannot
pass just because the stub replied. The job conditions are tested by
evaluating them over sample events rather than by comparing their text.

The README says how to re-run the test, and that GitHub does not let
anyone request a review from a pull request's own author, so pull
requests opened by cipherstash-agent still need draft then ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…request

The review workflow skips a request on a closed pull request with the reason
`closed`, so the verify job went red for a harmless event. The skipped and
review jobs now also require the pull request to be open for a review_requested
event, so none of them starts. The conditions' tests cover the closed case.

Also documents, in the README, that a request by a bot, the review agent or a
non-member fails the test by design (reasons bot, review-agent,
not-a-team-member), and that a pull request opened with the agent already in
Reviewers sends two events, where a third request replaces a waiting one and
its verify reports the request as not sent. A new test runs the skipped job's
DRAFT expression and script, which nothing exercised, and checks the reason
text for a draft, a fork and a bot.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede force-pushed the feat/self-test-rerun branch from 5766b5a to ff6871c Compare November 19, 2026 22:30
@tobyhede
tobyhede merged commit e2ff83b into main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run the self-test by requesting the review agent

3 participants