Skip to content

Commit ff6871c

Browse files
authored
fix(self-test): start no job for a review requested on a closed pull request
The review workflow skips a request on a closed pull request with the reason `closed`, so the verify job went red for a harmless event. The skipped and review jobs now also require the pull request to be open for a review_requested event, so none of them starts. The conditions' tests cover the closed case. Also documents, in the README, that a request by a bot, the review agent or a non-member fails the test by design (reasons bot, review-agent, not-a-team-member), and that a pull request opened with the agent already in Reviewers sends two events, where a third request replaces a waiting one and its verify reports the request as not sent. A new test runs the skipped job's DRAFT expression and script, which nothing exercised, and checks the reason text for a draft, a fork and a bot. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
1 parent 370a54e commit ff6871c

3 files changed

Lines changed: 89 additions & 7 deletions

File tree

‎.github/workflows/self-test.yml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@
1313
# Pull requests from forks (which get none of this repository's secrets) and pull requests opened by bots are not
1414
# tested, and neither are drafts when they are opened or reopened: the skipped job says why. Marking a draft ready, or
1515
# requesting a review from the review agent, runs the test; the review workflow takes a requested reviewer on a draft
16-
# too. A review requested from anyone else, or from a team, starts no job at all, not even the skipped one.
16+
# too. A review requested from anyone else, or from a team, or on a closed pull request, starts no job at all, not even
17+
# the skipped one: the review workflow would skip a closed pull request, and the verify job would fail for it.
1718
#
1819
# Needs, as repository settings (never the real routine's trigger or token):
1920
# secret CLAUDE_ROUTINE_INTEGRATION_TEST_TOKEN the stub routine's bearer token
@@ -31,12 +32,13 @@ permissions: {}
3132

3233
jobs:
3334
# Says why a pull request is not tested. Runs exactly when the review job does not, except for a review requested
34-
# from anyone but the review agent, which runs neither. The review agent's login must match the review workflow's
35+
# from anyone but the review agent or on a closed pull request, which runs neither. The review agent's login must match the review workflow's
3536
# review-agent input, which this file leaves at its default; a job's condition cannot read that input. GitHub
3637
# compares strings ignoring case, as the review workflow does, and a team's request has no requested_reviewer.
3738
skipped:
3839
if: >-
39-
(github.event.action != 'review_requested' || github.event.requested_reviewer.login == 'cipherstash-agent')
40+
(github.event.action != 'review_requested'
41+
|| (github.event.requested_reviewer.login == 'cipherstash-agent' && github.event.pull_request.state == 'open'))
4042
&& ((github.event.action != 'review_requested' && github.event.pull_request.draft)
4143
|| github.event.pull_request.head.repo.full_name != github.repository
4244
|| github.event.pull_request.user.type == 'Bot')
@@ -66,7 +68,8 @@ jobs:
6668
# tested only for a requested reviewer, which the review workflow takes on drafts too.
6769
review:
6870
if: >-
69-
(github.event.action != 'review_requested' || github.event.requested_reviewer.login == 'cipherstash-agent')
71+
(github.event.action != 'review_requested'
72+
|| (github.event.requested_reviewer.login == 'cipherstash-agent' && github.event.pull_request.state == 'open'))
7073
&& (github.event.action == 'review_requested' || github.event.pull_request.draft == false)
7174
&& github.event.pull_request.head.repo.full_name == github.repository
7275
&& github.event.pull_request.user.type != 'Bot'

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,8 @@ The `verify` job fails with one error that says what went wrong:
291291
- `No reply from the stub routine within 300 seconds`: the request was sent, but no comment by `@cipherstash-agent` on the pull request names this run attempt. The error says how many comments the last read took and how many comments with the marker and this attempt's link were ignored for being from another author. Each such comment also gets a notice, `Reply ignored`, that names its author and association. If there are some, the stub posted as a different account than `REPLY_AUTHOR`. Otherwise check that the stub's prompt matches the one above, that the stub can comment on this repository, and the stub's session in Claude. If reading the comments failed as well, the error ends with the last failure.
292292
- `REPLY_AUTHOR is empty or missing`: the workflow does not name the account whose reply to accept, so the job fails at once rather than accept anyone's comment.
293293

294+
A requested-reviewer request made by a bot, by the review agent itself, or by someone who is not on the team also makes the test fail, by design: the review workflow skips it with the reason `bot`, `review-agent` or `not-a-team-member`, and the `verify` job reports that reason. Such a request tests nothing about the stub. A review requested on a closed pull request starts no self-test job at all.
295+
294296
Each attempt looks only for a reply that names its own attempt's link, so a reply to an earlier run or an earlier attempt never counts.
295297

296298
### Running the test again
@@ -307,6 +309,10 @@ For those, mark the pull request as a draft and then as ready for review.
307309
A requested-reviewer request needs the membership check, because the person who requested the review must be a team member.
308310
If the App's secrets do not reach this repository, it fails with the reason `membership-lookup-failed` (see [Settings](#settings)).
309311

312+
A pull request that is opened with `cipherstash-agent` already in Reviewers sends two events, `opened` and `review_requested`, so it starts two self-test runs.
313+
Only one request is sent at a time for a pull request, so the second waits. A third request that arrives while the second is waiting replaces the waiting one (see `concurrency` above).
314+
The replaced run's `verify` job then reports that the request was not sent, because its `fire` job was cancelled. That failure is expected, and the newer run tests the commit.
315+
310316
"Re-run all jobs" tests the same commit again, and gets a new attempt, with a new attempt link.
311317

312318
## Checks

‎tests/test_self_test.py‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
"""The self-test workflow, .github/workflows/self-test.yml: what it calls, with which routine, and when."""
22

3+
import os
34
import re
5+
import subprocess
6+
import tempfile
47
import unittest
58

69
from harness import SELF_TEST, _yq, file_steps, step_field, step_file_path, step_shells
@@ -58,10 +61,20 @@ def take(self, operator):
5861
return False
5962

6063
def evaluate(self):
64+
return bool(self.result())
65+
66+
def result(self):
6167
result = self.either()
6268
if self.position != len(self.tokens):
6369
raise ValueError("the expression has text left over")
64-
return bool(result)
70+
return result
71+
72+
def render(self):
73+
"""The value as GitHub writes it into an `env:` entry: true and false in lower case, null as empty."""
74+
result = self.result()
75+
if isinstance(result, bool):
76+
return "true" if result else "false"
77+
return "" if result is None else str(result)
6578

6679
def either(self):
6780
left = self.both()
@@ -107,11 +120,12 @@ def unary(self):
107120
return token
108121

109122

110-
def event(action="opened", draft=False, fork=False, author_type="User", reviewer=None, team=None):
123+
def event(action="opened", draft=False, fork=False, author_type="User", reviewer=None, team=None, state="open"):
111124
"""A pull_request event's payload, trimmed to what the jobs' conditions read."""
112125
payload = {
113126
"action": action,
114127
"pull_request": {
128+
"state": state,
115129
"draft": draft,
116130
"head": {"repo": {"full_name": "someone/workflows" if fork else REPOSITORY}},
117131
"user": {"login": "monalisa", "type": author_type},
@@ -124,9 +138,32 @@ def event(action="opened", draft=False, fork=False, author_type="User", reviewer
124138
return payload
125139

126140

141+
def context_of(payload):
142+
return {"github": {"event": payload, "event_name": "pull_request", "repository": REPOSITORY}}
143+
144+
145+
def skipped_job_says(payload):
146+
"""What the `skipped` job's step reports for the payload: its env filled in as GitHub would, and its script run."""
147+
context = context_of(payload)
148+
names = value(".jobs.skipped.steps[0].env | keys | .[]").split()
149+
env = {name: Expression(value(f".jobs.skipped.steps[0].env.{name}"), context).render() for name in names}
150+
script = _yq(".jobs.skipped.steps[0].run", SELF_TEST)
151+
with tempfile.TemporaryDirectory() as temp:
152+
summary = os.path.join(temp, "summary")
153+
result = subprocess.run(
154+
["bash", "-c", script],
155+
env={**os.environ, **env, "GITHUB_STEP_SUMMARY": summary},
156+
capture_output=True,
157+
text=True,
158+
check=True,
159+
)
160+
with open(summary, encoding="utf-8") as file:
161+
return result.stdout, file.read()
162+
163+
127164
def jobs_run(payload):
128165
"""The names of the self-test's jobs, out of skipped and review, that run for the payload."""
129-
context = {"github": {"event": payload, "event_name": "pull_request", "repository": REPOSITORY}}
166+
context = context_of(payload)
130167
return {job for job in ("skipped", "review") if Expression(value(f".jobs.{job}.if"), context).evaluate()}
131168

132169

@@ -191,6 +228,42 @@ def test_requesting_any_other_reviewer_or_a_team_runs_no_job(self):
191228
with self.subTest(**case):
192229
self.assertEqual(jobs_run(event("review_requested", **case)), set())
193230

231+
def test_a_review_requested_on_a_closed_pull_request_runs_no_job(self):
232+
for reviewer in ("cipherstash-agent", "CipherStash-Agent"):
233+
for case in ({}, {"draft": True}, {"fork": True}, {"author_type": "Bot"}):
234+
with self.subTest(reviewer=reviewer, **case):
235+
closed = event("review_requested", reviewer=reviewer, state="closed", **case)
236+
self.assertEqual(jobs_run(closed), set())
237+
238+
def test_the_skipped_job_says_a_draft_is_why(self):
239+
for action in ("opened", "reopened"):
240+
with self.subTest(action=action):
241+
notice, summary = skipped_job_says(event(action, draft=True))
242+
self.assertIn("::notice title=Self-test skipped::Not tested: the pull request is a draft", notice)
243+
self.assertIn("requesting a review from cipherstash-agent, runs the test", notice)
244+
self.assertIn("Self-test: skipped (the pull request is a draft", summary)
245+
246+
def test_the_skipped_job_does_not_call_a_requested_draft_a_draft(self):
247+
# A review requested on a draft is tested, so a draft never explains a skip for it: its fork or bot does.
248+
notice, _ = skipped_job_says(event("review_requested", draft=True, fork=True, reviewer="cipherstash-agent"))
249+
self.assertIn("comes from a fork", notice)
250+
notice, _ = skipped_job_says(
251+
event("review_requested", draft=True, author_type="Bot", reviewer="cipherstash-agent")
252+
)
253+
self.assertIn("opened by a bot", notice)
254+
255+
def test_the_skipped_job_says_a_fork_or_a_bot_is_why(self):
256+
notice, summary = skipped_job_says(event("opened", fork=True))
257+
self.assertIn("Not tested: the pull request comes from a fork", notice)
258+
self.assertIn("Self-test: skipped (the pull request comes from a fork", summary)
259+
notice, summary = skipped_job_says(event("ready_for_review", author_type="Bot"))
260+
self.assertIn("Not tested: the pull request was opened by a bot", notice)
261+
self.assertIn("Self-test: skipped (the pull request was opened by a bot", summary)
262+
263+
def test_the_skipped_job_names_a_draft_before_a_fork(self):
264+
notice, _ = skipped_job_says(event("opened", draft=True, fork=True))
265+
self.assertIn("is a draft", notice)
266+
194267
def test_the_review_agent_requested_is_the_one_the_review_workflow_acts_for(self):
195268
# The conditions name the review agent, as a job's `if:` cannot read the review workflow's input default.
196269
agent = _yq('.on.workflow_call.inputs."review-agent".default').strip()

0 commit comments

Comments
 (0)