Skip to content

Commit 40a0989

Browse files
authored
fix: start the self-test for a comment command whatever the commenter's association
GitHub reports a team member whose organisation membership is private as CONTRIBUTOR or NONE, so the OWNER/MEMBER/COLLABORATOR test silently started no run for their comment command. The review workflow's team check is the gate: a non-member's command now starts a run that ends red with not-a-team-member, by design. Bots, the review agent, closed pull requests and non-commands still start no job. Tests first failed for CONTRIBUTOR and NONE commenters. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
1 parent 5c6a9cf commit 40a0989

3 files changed

Lines changed: 29 additions & 15 deletions

File tree

‎.github/workflows/self-test.yml‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,14 @@
1010
# was of the wrong kind for the event, or if it was not sent, and otherwise waits a few minutes for the stub's reply
1111
# naming this run attempt.
1212
#
13-
# A team member's comment command, a new pull request comment whose first line is `@cipherstash-agent review`, runs the
14-
# test too, and the verify job also checks that the comment got the review workflow's 👀 reaction. A comment always
13+
# A comment command, a new pull request comment whose first line is `@cipherstash-agent review`, runs the test too,
14+
# and the verify job also checks that the comment got the review workflow's 👀 reaction. A comment always
1515
# runs the workflow file, the review workflow and the step files from the default branch, main, so this part tests
1616
# main's version, not the pull request's, and runs only once this file is on main. Comments that cannot be a comment
1717
# command start no job: other comments, comments on plain issues or closed pull requests, comments by bots or by the
18-
# review agent (the stub's own reply among them), and comments by anyone who is not an owner, member or collaborator.
18+
# review agent (the stub's own reply among them). A comment command from someone who is not a team member does start
19+
# the job, and the run ends red with `not-a-team-member`, by design: GitHub reports a team member whose organisation
20+
# membership is private as CONTRIBUTOR or NONE, so the commenter's author_association cannot tell who may ask.
1921
#
2022
# Pull requests from forks (which get none of this repository's secrets) and pull requests opened by bots are not
2123
# tested, and neither are drafts when they are opened or reopened: the skipped job says why. Marking a draft ready, or
@@ -90,10 +92,11 @@ jobs:
9092
# review workflow does). A command with other spacing, such as two spaces, is a command to the review workflow but
9193
# is not tested. fromJSON gives the tab and line breaks, which an expression's string cannot hold. Comments by bots
9294
# and by the review agent (the stub's reply among them) and comments on closed pull requests start no job, because
93-
# the review workflow skips them; the author_association test, as in examples/claude-review.yml, keeps comments from
94-
# outside the repository's members from starting a runner. The review workflow still decides, and checks team
95-
# membership. A comment's payload does not say whether the pull request comes from a fork, and the review workflow
96-
# takes comment commands on forks, so they are tested: nothing from the fork is checked out or run.
95+
# the review workflow skips them. There is no author_association test, as in examples/claude-review.yml: a team
96+
# member whose organisation membership is private shows as CONTRIBUTOR or NONE and would be skipped silently. The
97+
# review workflow decides, and checks team membership, so a non-member's command starts a run that ends red with
98+
# not-a-team-member, by design. A comment's payload does not say whether the pull request comes from a fork, and
99+
# the review workflow takes comment commands on forks, so they are tested: nothing from the fork is checked out or run.
97100
review:
98101
if: >-
99102
(github.event_name == 'pull_request'
@@ -108,7 +111,6 @@ jobs:
108111
&& github.event.issue.state == 'open'
109112
&& github.event.comment.user.type != 'Bot'
110113
&& github.event.comment.user.login != 'cipherstash-agent'
111-
&& contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association)
112114
&& (github.event.comment.body == '@cipherstash-agent review'
113115
|| startsWith(github.event.comment.body, '@cipherstash-agent review ')
114116
|| startsWith(github.event.comment.body, format('@cipherstash-agent review{0}', fromJSON('"\t"')))

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@ The stub only posts a comment saying it received the request, so a test run is f
242242

243243
### What it does
244244

245-
[`.github/workflows/self-test.yml`](.github/workflows/self-test.yml) runs when a pull request in this repository is opened, reopened or marked ready for review, when someone requests a review from the review agent, `cipherstash-agent`, on a pull request here, and when a team member posts a comment command on a pull request here (see [Comment commands](#comment-commands)).
245+
[`.github/workflows/self-test.yml`](.github/workflows/self-test.yml) runs when a pull request in this repository is opened, reopened or marked ready for review, when someone requests a review from the review agent, `cipherstash-agent`, on a pull request here, and when someone posts a comment command on a pull request here (see [Comment commands](#comment-commands)).
246246

247247
1. The `review` job calls `.github/workflows/claude-review.yml` by GitHub's self-repository form, `uses: $/.github/workflows/claude-review.yml`, which names this repository at the exact commit that is running. (`$/` needs github.com and runner 2.336.0 or later.) For a Ready or requested-reviewer request, that is the pull request's own version, so the version under test is the one in the pull request. For a comment command, it is `main`'s version (see [Comment commands](#comment-commands)). The job passes the stub routine's trigger id and token, never the real routine's.
248248
2. The review workflow decides, as it would for any repository. For a review requested from `cipherstash-agent`, the person who requested it must be a member of the `@cipherstash/developers` team, as for any requested reviewer. If it decides to request a review, it sends the request to the stub. The request text includes a link to this run attempt, `<server>/<repo>/actions/runs/<id>/attempts/<n>`.
@@ -262,17 +262,18 @@ The test covers all three kinds of review request: Ready, requested-reviewer and
262262
### Comment commands
263263

264264
A comment command is a new comment on a pull request whose first line is `@cipherstash-agent review` (see [Claude review](#claude-review)).
265-
When a team member posts one on a pull request in this repository, the self-test runs as above, with these differences:
265+
When someone posts one on a pull request in this repository, the self-test runs as above, with these differences:
266266

267267
- **It tests `main`, not the pull request.** GitHub always runs a comment's workflows from the default branch. So the self-test file, the review workflow it calls and the step files all come from `main`, whatever the pull request changes. To test a pull request's own version of the review workflow, use a Ready or requested-reviewer request (see [Running the test again](#running-the-test-again)). This part of the test cannot run at all until the self-test, with its `issue_comment` trigger, is merged to `main`.
268268
- The `verify` job checks that the decision is a comment-command request for the comment that started the run. It then checks that the comment has the review workflow's 👀 reaction. The reaction must be by `github-actions[bot]` (the `REACTION_AUTHOR` setting in the workflow), the account that the review workflow's `react` job reacts as, so a 👀 added by hand does not count. This account is not yet confirmed by a live run. If the first live run fails on the reaction check, check `REACTION_AUTHOR` first. The job reads the comment's reactions every 5 seconds, for up to 60 seconds; the `react` job has finished before `verify` starts, so the reaction is normally there at once. When it finds the reaction, it shows it in a notice, `Comment command reacted to`, and in the run's summary. Last, it waits for the stub's reply for this run attempt, as for the other kinds.
269269
- Only comments that the review workflow would take as a comment command start the test. All other comments start no self-test job at all: the workflow run shows only skipped jobs, and no `skipped` job explains it. That includes:
270270
- ordinary comments, and comments that mention `cipherstash-agent` without being a command
271271
- comments on plain issues, and on closed or merged pull requests
272272
- comments by bots, and by `cipherstash-agent` itself. The stub posts its reply as `cipherstash-agent`, so its reply can never start another run.
273-
- comments by anyone who is not an owner, member or collaborator of the repository (GitHub's `author_association`), as in [`examples/claude-review.yml`](examples/claude-review.yml). This is only a filter that saves starting a runner. The review workflow's team membership check is what decides.
274273
- edited or deleted comments
275274

275+
Comments are not filtered by the commenter's standing in the repository (GitHub's `author_association`), as in [`examples/claude-review.yml`](examples/claude-review.yml), because GitHub reports a team member whose organisation membership is private as `CONTRIBUTOR` or `NONE`, and a filter on it would skip their command without any run. The review workflow's team membership check decides. So a command from someone who is not a team member does start a run, which ends red with `not-a-team-member`, by design.
276+
276277
The filter is a job condition, which cannot match patterns, so it takes only a first line of `@cipherstash-agent review` (in any case) followed by the end of the comment, a space, a tab or a line break. A command with other spacing, such as two spaces before `review`, is still a command to the review workflow, but it does not start the test. A unit test runs the request decision on sample comments to check that the filter never lets through a comment that the review workflow would skip as not a command.
277278
- Comment commands are tested on drafts, because the review workflow takes them on drafts.
278279
- Comment commands are tested on pull requests from forks too. A comment's event does not say whether the pull request comes from a fork, and the review workflow takes comment commands on forks. A comment's run has this repository's secrets even on a fork's pull request, but nothing from the fork is checked out or run: every job uses `main`'s files only, the review workflow reviews by the pull request's link, and only a team member's command is sent. So the secrets reach no code from the fork.

‎tests/test_self_test.py‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -407,10 +407,22 @@ def test_a_comment_command_by_a_bot_or_by_the_review_agent_runs_no_job(self):
407407
with self.subTest(**case):
408408
self.assertEqual(jobs_run(comment_event(**case)), set())
409409

410-
def test_a_comment_command_from_outside_the_repositorys_members_runs_no_job(self):
410+
def test_a_comment_command_runs_the_review_job_whatever_the_commenters_association(self):
411+
# GitHub reports a team member whose organisation membership is private as CONTRIBUTOR or NONE, so the
412+
# association is no filter: the review workflow's team membership check decides, and ends the run red for a
413+
# non-member.
411414
for association in ("CONTRIBUTOR", "FIRST_TIME_CONTRIBUTOR", "FIRST_TIMER", "NONE", "MANNEQUIN", None):
412415
with self.subTest(association=association):
413-
self.assertEqual(jobs_run(comment_event(association=association)), set())
416+
self.assertEqual(jobs_run(comment_event(association=association)), {"review"})
417+
418+
def test_a_bot_or_review_agent_command_runs_no_job_whatever_its_association(self):
419+
for association in ("OWNER", "CONTRIBUTOR", "NONE", None):
420+
for case in ({"author_type": "Bot", "login": "dependabot[bot]"}, {"login": "cipherstash-agent"}):
421+
with self.subTest(association=association, **case):
422+
self.assertEqual(jobs_run(comment_event(association=association, **case)), set())
423+
424+
def test_the_review_job_does_not_filter_comments_by_author_association(self):
425+
self.assertNotIn("author_association", value(".jobs.review.if"))
414426

415427
def test_a_comment_command_on_a_closed_pull_request_or_a_plain_issue_runs_no_job(self):
416428
self.assertEqual(jobs_run(comment_event(state="closed")), set())
@@ -500,8 +512,7 @@ def test_the_verify_step_accepts_a_reply_only_from_the_stub_routines_account(sel
500512
)
501513

502514
def test_the_verify_job_does_not_filter_the_reply_by_author_association(self):
503-
# The review job's comment filter reads the comment command's author_association, as the example caller does;
504-
# the stub's reply is accepted by its author's login alone.
515+
# The stub's reply is accepted by its author's login alone.
505516
self.assertNotIn("author_association", value(".jobs.verify | tojson"))
506517

507518
def test_the_review_workflow_gives_its_decision_and_whether_it_sent_the_request(self):

0 commit comments

Comments
 (0)