Skip to content

Commit b493c5e

Browse files
authored
Merge pull request #35 from cipherstash/feat/self-test-comment
feat(self-test): self-test the comment command
2 parents e2ff83b + 0257b14 commit b493c5e

5 files changed

Lines changed: 782 additions & 113 deletions

File tree

‎.github/workflows/self-test.yml‎

Lines changed: 70 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@
1010
# was of the wrong kind for the event, or if it was not sent, and otherwise waits a few minutes for the stub's reply
1111
# naming this run attempt.
1212
#
13+
# A comment command, a new pull request comment whose first line is `@cipherstash-agent review`, runs the test too,
14+
# and the verify job also checks that the comment got the review workflow's 👀 reaction. A comment always
15+
# runs the workflow file, the review workflow and the step files from the default branch, main, so this part tests
16+
# main's version, not the pull request's, and runs only once this file is on main. Comments that cannot be a comment
17+
# command start no job: other comments, comments on plain issues or closed pull requests, comments by bots or by the
18+
# review agent (the stub's own reply among them). A comment command from someone who is not a team member does start
19+
# the job, and the run ends red with `not-a-team-member`, by design: GitHub reports a team member whose organisation
20+
# membership is private as CONTRIBUTOR or NONE, so the commenter's author_association cannot tell who may ask.
21+
#
1322
# Pull requests from forks (which get none of this repository's secrets) and pull requests opened by bots are not
1423
# tested, and neither are drafts when they are opened or reopened: the skipped job says why. Marking a draft ready, or
1524
# requesting a review from the review agent, runs the test; the review workflow takes a requested reviewer on a draft
@@ -20,24 +29,34 @@
2029
# secret CLAUDE_ROUTINE_INTEGRATION_TEST_TOKEN the stub routine's bearer token
2130
# variable CLAUDE_ROUTINE_INTEGRATION_TEST_TRIGGER_ID the stub routine's trigger id (`trig_…`); a variable, because a
2231
# secret cannot be passed as a reusable workflow's input
23-
# secrets CLAUDE_REVIEW_APP_ID, CLAUDE_REVIEW_APP_PRIVATE_KEY the membership App, used only when the pull request's
24-
# author is not the review agent
32+
# secrets CLAUDE_REVIEW_APP_ID, CLAUDE_REVIEW_APP_PRIVATE_KEY the membership App, used for every comment command,
33+
# every requested-reviewer request, and a Ready request when
34+
# the pull request's author is not the review agent
2535
name: Self-test
2636

2737
on:
2838
pull_request:
2939
types: [opened, reopened, ready_for_review, review_requested]
40+
# Runs with this repository's secrets, from main, for any comment on an issue or pull request here. Only a team
41+
# member's comment command is acted on: the review job's condition filters out comments that cannot be one, and the
42+
# review workflow checks every rule itself, team membership included. No job checks out or runs the pull request's
43+
# code; every file comes from main. zizmor flags issue_comment because anyone who can comment can start this
44+
# workflow. Accepted for those reasons, as in examples/claude-review.yml.
45+
issue_comment: # zizmor: ignore[dangerous-triggers] only a team member's comment command is acted on
46+
types: [created]
3047

3148
permissions: {}
3249

3350
jobs:
3451
# Says why a pull request is not tested. Runs exactly when the review job does not, except for a review requested
35-
# from anyone but the review agent or on a closed pull request, which runs neither. The review agent's login must match the review workflow's
36-
# review-agent input, which this file leaves at its default; a job's condition cannot read that input. GitHub
37-
# compares strings ignoring case, as the review workflow does, and a team's request has no requested_reviewer.
52+
# from anyone but the review agent or on a closed pull request, and any comment, which run neither. The review
53+
# agent's login must match the review workflow's review-agent input, which this file leaves at its default; a job's
54+
# condition cannot read that input. GitHub compares strings ignoring case, as the review workflow does, and a team's
55+
# request has no requested_reviewer.
3856
skipped:
3957
if: >-
40-
(github.event.action != 'review_requested'
58+
github.event_name == 'pull_request'
59+
&& (github.event.action != 'review_requested'
4160
|| (github.event.requested_reviewer.login == 'cipherstash-agent' && github.event.pull_request.state == 'open'))
4261
&& ((github.event.action != 'review_requested' && github.event.pull_request.draft)
4362
|| github.event.pull_request.head.repo.full_name != github.repository
@@ -64,17 +83,44 @@ jobs:
6483
echo "::notice title=Self-test skipped::Not tested: $reason"
6584
echo "Self-test: skipped ($reason)" >>"$GITHUB_STEP_SUMMARY"
6685
67-
# Calls this pull request's own version of the review workflow, with the stub routine's trigger and token. A draft is
68-
# tested only for a requested reviewer, which the review workflow takes on drafts too.
86+
# Calls the review workflow with the stub routine's trigger and token: this pull request's own version for a
87+
# pull_request event, and main's for a comment. A draft is tested only for a requested reviewer or a comment
88+
# command, which the review workflow takes on drafts too.
89+
#
90+
# For a comment, the condition is a filter only, and lets through nothing the review workflow would skip as not a
91+
# comment command, so a red test always means something: the first line must be `@cipherstash-agent review` followed
92+
# by the end of the comment, a space, a tab or a line break (startsWith and == ignore case, as the review workflow
93+
# does). This holds for ASCII text. GitHub does not document how it compares other letters, so such a comment could
94+
# pass here and then end the run red with not-a-command. A command with other spacing, such as two spaces, is a
95+
# command to the review workflow but is not tested. fromJSON gives the tab and line breaks, which an expression's
96+
# string cannot hold. Comments by bots and by the review agent (the stub's reply among them) and comments on closed
97+
# pull requests start no job, because the review workflow skips them. There is no author_association test, as in
98+
# examples/claude-review.yml: a team member whose organisation membership is private shows as CONTRIBUTOR or NONE and
99+
# would be skipped silently. The review workflow decides, and checks team membership, so a non-member's command starts
100+
# a run that ends red with not-a-team-member, by design. A comment's payload does not say whether the pull request
101+
# comes from a fork, and the review workflow takes comment commands on forks, so they are tested: nothing from the
102+
# fork is checked out or run.
69103
review:
70104
if: >-
71-
(github.event.action != 'review_requested'
105+
(github.event_name == 'pull_request'
106+
&& (github.event.action != 'review_requested'
72107
|| (github.event.requested_reviewer.login == 'cipherstash-agent' && github.event.pull_request.state == 'open'))
73108
&& (github.event.action == 'review_requested' || github.event.pull_request.draft == false)
74109
&& github.event.pull_request.head.repo.full_name == github.repository
75-
&& github.event.pull_request.user.type != 'Bot'
110+
&& github.event.pull_request.user.type != 'Bot')
111+
|| (github.event_name == 'issue_comment'
112+
&& github.event.action == 'created'
113+
&& github.event.issue.pull_request
114+
&& github.event.issue.state == 'open'
115+
&& github.event.comment.user.type != 'Bot'
116+
&& github.event.comment.user.login != 'cipherstash-agent'
117+
&& (github.event.comment.body == '@cipherstash-agent review'
118+
|| startsWith(github.event.comment.body, '@cipherstash-agent review ')
119+
|| startsWith(github.event.comment.body, format('@cipherstash-agent review{0}', fromJSON('"\t"')))
120+
|| startsWith(github.event.comment.body, format('@cipherstash-agent review{0}', fromJSON('"\n"')))
121+
|| startsWith(github.event.comment.body, format('@cipherstash-agent review{0}', fromJSON('"\r\n"')))))
76122
# The review workflow's jobs can hold no more than this, and GitHub refuses to start it if a job asks for more,
77-
# even one that will not run. Only its react job uses it, for a comment command, which this test does not make.
123+
# even one that will not run. Only its react job uses it, to add the 👀 reaction to a comment command.
78124
permissions:
79125
issues: write
80126
uses: $/.github/workflows/claude-review.yml
@@ -85,22 +131,23 @@ jobs:
85131
CLAUDE_REVIEW_APP_ID: ${{ secrets.CLAUDE_REVIEW_APP_ID }}
86132
CLAUDE_REVIEW_APP_PRIVATE_KEY: ${{ secrets.CLAUDE_REVIEW_APP_PRIVATE_KEY }}
87133

88-
# Fails unless the review workflow sent a review request and the stub replied naming this run. Runs whenever the
89-
# review job ran, even if it failed, so a failure is always explained here.
134+
# Fails unless the review workflow sent a review request, the comment command (if that is what started the run) got
135+
# the 👀 reaction, and the stub replied naming this run. Runs whenever the review job ran, even if it failed, so a
136+
# failure is always explained here.
90137
verify:
91138
needs: review
92139
if: ${{ !cancelled() && needs.review.result != 'skipped' }}
93140
runs-on: ubuntu-latest
94141
timeout-minutes: 10
95-
# Reads the pull request's comments. This repository is public, so the checkout below needs no contents permission,
96-
# as the review workflow's own jobs show.
142+
# Reads the pull request's comments, and a comment's reactions. This repository is public, so the checkout below
143+
# needs no contents permission, as the review workflow's own jobs show.
97144
permissions:
98145
issues: read
99146
env:
100147
# Where the checkout step below puts this repository's step files, as in the review workflow's jobs.
101148
WORKFLOW_FILES: ${{ github.workspace }}/.cipherstash-workflows
102149
steps:
103-
# This pull request's own steps/ directory, and nothing else. No credentials are kept.
150+
# This pull request's own steps/ directory (main's, for a comment), and nothing else. No credentials are kept.
104151
- name: Check out the step files
105152
id: checkout
106153
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -118,7 +165,14 @@ jobs:
118165
DECISION: ${{ needs.review.outputs.decision }}
119166
SENT: ${{ needs.review.outputs.sent }}
120167
# Which event started this run, so the step can check the request was of the kind that event makes.
168+
EVENT_NAME: ${{ github.event_name }}
121169
EVENT_ACTION: ${{ github.event.action }}
170+
# The comment that started a comment command's run, whose 👀 reaction is checked; a number, or empty for a
171+
# pull_request event.
172+
COMMENT_ID: ${{ github.event.comment.id }}
173+
# The account the review workflow's react job adds the reaction as: the job's token acts as this bot. Only
174+
# its reaction counts, so a 👀 added by hand does not pass the test.
175+
REACTION_AUTHOR: github-actions[bot]
122176
# The account the stub routine posts its reply as. Only a comment by this login counts as the reply: the
123177
# job's token may not see the stub's organisation membership, so the author's standing in the repository is not used.
124178
REPLY_AUTHOR: cipherstash-agent

0 commit comments

Comments
 (0)