Skip to content

Commit 6a9bc32

Browse files
committed
Merge feat/self-test-ready into feat/self-test-rerun
Brings in the reply-author check for the verify step. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr # Conflicts: # .github/workflows/self-test.yml # README.md # steps/verify.py # tests/test_verify.py
2 parents 8e99c84 + 8548515 commit 6a9bc32

5 files changed

Lines changed: 192 additions & 37 deletions

File tree

‎.github/workflows/self-test.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,9 @@ jobs:
115115
SENT: ${{ needs.review.outputs.sent }}
116116
# Which event started this run, so the step can check the request was of the kind that event makes.
117117
EVENT_ACTION: ${{ github.event.action }}
118+
# The account the stub routine posts its reply as. Only a comment by this login counts as the reply: the
119+
# job's token may not see the stub's organisation membership, so the author's standing in the repository is not used.
120+
REPLY_AUTHOR: cipherstash-agent
118121
# The check's file, from this pull request's checkout. Python's standard library only (3.10 features), so
119122
# nothing is installed; tests/test_verify.py runs this file.
120123
STEP_FILE: ${{ env.WORKFLOW_FILES }}/steps/verify.py

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,7 @@ The stub only posts a comment saying it received the request, so a test run is f
248248
1. The `review` job calls the pull request's own version of `.github/workflows/claude-review.yml`, by its local path, so the version under test is the one in the pull request. It passes the stub routine's trigger id and token, never the real routine's.
249249
2. The review workflow decides, as it would for any repository. For a review requested from `cipherstash-agent`, the person who requested it must be a member of the `@cipherstash/developers` team, as for any requested reviewer. If it decides to request a review, it sends the request to the stub. The request text includes a link to this run attempt, `<server>/<repo>/actions/runs/<id>/attempts/<n>`.
250250
3. The stub posts a comment on the pull request, echoing the attempt's link.
251-
4. The `verify` job checks the review workflow's decision. The request must be a Ready request when the pull request was opened, reopened or marked ready, and a requested-reviewer request when a review was requested. If it decided to request a review of the right kind and sent the request, the job reads the pull request's comments every 15 seconds, for up to 5 minutes, until it finds a comment that contains `integration-test: received` and this attempt's link. Only a comment whose `author_association` is `OWNER`, `MEMBER` or `COLLABORATOR` counts, so someone else cannot pass the test by posting the marker on a public pull request. It then shows the reply, and its author, in a notice and in the run's summary.
251+
4. The `verify` job checks the review workflow's decision. The request must be a Ready request when the pull request was opened, reopened or marked ready, and a requested-reviewer request when a review was requested. If it decided to request a review of the right kind and sent the request, the job reads the pull request's comments every 15 seconds, for up to 5 minutes, until it finds a comment that contains `integration-test: received` and this attempt's link. Only a comment posted by the stub routine's account, `cipherstash-agent` (the `REPLY_AUTHOR` setting in the workflow; the login is compared without regard to case), counts, so someone else cannot pass the test by posting the marker on a public pull request. The comment author's standing in the repository (`author_association`) is not used, because the job's token may see a different standing for an account with private organisation membership. It then shows the reply, and its author, in a notice and in the run's summary.
252252

253253
The test is skipped, and the `skipped` job says why in a notice, for:
254254

@@ -289,7 +289,8 @@ The `verify` job fails with one error that says what went wrong:
289289
- `The review workflow gave no decision to request a review`: the review workflow did not finish its decision. Look at its `decide` job.
290290
- `... but the request was not sent to the stub routine`: the stub's trigger refused the request or could not be reached. The `review / fire` job's error says why. Check the token secret and the trigger id variable. If the `fire` job was cancelled, a newer request (a newer push or event for the same pull request) may have replaced it.
291291
- `GitHub answered HTTP 401, 403 or 404 when asked for the pull request's comments`: retrying cannot fix it, so the job fails at once. Check the job's token and permissions.
292-
- `No reply from the stub routine within 300 seconds`: the request was sent, but no comment on the pull request names this run attempt. Check that the stub's prompt matches the one above, that the stub can comment on this repository, and the stub's session in Claude. If reading the comments failed as well, the error ends with the last failure.
292+
- `No reply from the stub routine within 300 seconds`: the request was sent, but no comment by `@cipherstash-agent` on the pull request names this run attempt. The error says how many comments the last read took and how many comments with the marker and this attempt's link were ignored for being from another author. Each such comment also gets a notice, `Reply ignored`, that names its author and association. If there are some, the stub posted as a different account than `REPLY_AUTHOR`. Otherwise check that the stub's prompt matches the one above, that the stub can comment on this repository, and the stub's session in Claude. If reading the comments failed as well, the error ends with the last failure.
293+
- `REPLY_AUTHOR is empty or missing`: the workflow does not name the account whose reply to accept, so the job fails at once rather than accept anyone's comment.
293294

294295
Each attempt looks only for a reply that names its own attempt's link, so a reply to an earlier run or an earlier attempt never counts.
295296

‎steps/verify.py‎

Lines changed: 68 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
"""The self-test's verify step: checks that the review workflow requested a review from the stub routine, then waits for
22
the stub's reply on the pull request.
33
4-
Passes only when a comment on the pull request, by an owner, member or collaborator of the repository, contains the
5-
marker `integration-test: received` and this run attempt's link (`<server>/<repo>/actions/runs/<id>/attempts/<n>`),
6-
which the review request includes and the stub echoes back. Fails with one error annotation that says what went wrong:
7-
the review workflow decided not to request a review (naming the skip reason), the request was not of the kind the
8-
event makes, the request was not sent, or no reply came within WAIT_SECONDS.
4+
Passes only when a comment on the pull request, by the stub routine's account (REPLY_AUTHOR), contains the marker
5+
`integration-test: received` and this run attempt's link (`<server>/<repo>/actions/runs/<id>/attempts/<n>`), which the
6+
review request includes and the stub echoes back. Fails with one error annotation that says what went wrong: no
7+
account to expect, the review workflow decided not to request a review (naming the skip reason), the request was not
8+
of the kind the event makes, the request was not sent, or no reply came within WAIT_SECONDS (saying how many comments
9+
were read and how many carried the marker and this attempt's link but came from another author).
910
1011
Inputs, all environment variables: DECISION and SENT (the review workflow's outputs), EVENT_ACTION (the action of the
11-
pull_request event that started the run), GITHUB_TOKEN (the job's token, which only reads comments) and the runner's
12-
GITHUB_API_URL, GITHUB_REPOSITORY, GITHUB_SERVER_URL, GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, GITHUB_OUTPUT and
13-
GITHUB_STEP_SUMMARY.
12+
pull_request event that started the run), REPLY_AUTHOR (the login the stub routine posts as), GITHUB_TOKEN (the job's
13+
token, which only reads comments) and the runner's GITHUB_API_URL, GITHUB_REPOSITORY, GITHUB_SERVER_URL, GITHUB_RUN_ID,
14+
GITHUB_RUN_ATTEMPT, GITHUB_OUTPUT and GITHUB_STEP_SUMMARY.
1415
Python's standard library only (3.10 features); tests/test_verify.py runs this file.
1516
"""
1617

@@ -37,9 +38,6 @@
3738

3839
# A run attempt's link, as the review workflow's decide job writes it and the request decision accepts it.
3940
RUN_URL = re.compile(r"https://[A-Za-z0-9.-]+/[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+/actions/runs/[0-9]+/attempts/[0-9]+")
40-
# Who may post the reply: people with a standing in the repository. Anyone can comment on a public pull request, so a
41-
# copy of the marker from anyone else is not the stub's reply.
42-
TRUSTED_AUTHORS = ("OWNER", "MEMBER", "COLLABORATOR")
4341
# The kind of review request each pull_request event the self-test runs on makes.
4442
KINDS = {
4543
"opened": "ready",
@@ -59,6 +57,26 @@ class Unread(Exception):
5957
"""The comments could not be read this time; the message says why."""
6058

6159

60+
class Tally:
61+
"""What the reads of the comments have seen, for the timeout's message and for noticing each near-match once."""
62+
63+
def __init__(self) -> None:
64+
self.read = 0 # comments read in the last pass over the pages
65+
self.ignored: set[Any] = set() # near-matches from another author, by comment id, across all passes
66+
67+
68+
def reply_author() -> str:
69+
"""The login the stub routine posts as. Anyone can comment on a public pull request, so a copy of the marker from
70+
another account is not the stub's reply; an empty login would accept everyone, so it is an error."""
71+
login = os.environ.get("REPLY_AUTHOR", "").strip()
72+
if not login:
73+
raise Failed(
74+
"REPLY_AUTHOR is empty or missing, so there is no account whose reply to accept: "
75+
"set it to the login the stub routine posts as in the self-test workflow"
76+
)
77+
return login
78+
79+
6280
def run_url() -> str:
6381
"""This run attempt's link. A re-run has the same run id and a new attempt, so only the attempt's link tells a reply
6482
to this attempt from a reply to an earlier one."""
@@ -130,31 +148,54 @@ def comments_page(token: str, number: int, page: int) -> list[Any]:
130148
return comments
131149

132150

133-
def find_reply(token: str, number: int, url: str, deadline: float) -> dict[str, Any] | None:
134-
"""The first comment on the pull request, by someone with a standing in the repository, that holds the marker and
135-
this run attempt's link, not a longer number. Stops looking at the deadline, even part way through the pages."""
151+
def find_reply(
152+
token: str, number: int, url: str, author: str, deadline: float, tally: Tally
153+
) -> dict[str, Any] | None:
154+
"""The first comment on the pull request by `author` (compared without regard to case) that holds the marker and
155+
this run attempt's link, not a longer number. A comment that holds both but is by someone else is counted in
156+
tally and noticed once. Stops looking at the deadline, even part way through the pages."""
136157
names_this_run = re.compile(re.escape(url) + r"(?![0-9])")
158+
tally.read = 0
137159
for page in range(1, MAX_PAGES + 1):
138160
comments = comments_page(token, number, page)
139161
for comment in comments:
140162
if not isinstance(comment, dict):
141163
continue
164+
tally.read += 1
142165
found: dict[str, Any] = comment
143166
body = found.get("body")
144-
trusted = found.get("author_association") in TRUSTED_AUTHORS
145-
if trusted and isinstance(body, str) and MARKER in body and names_this_run.search(body):
167+
if not (isinstance(body, str) and MARKER in body and names_this_run.search(body)):
168+
continue
169+
user = found.get("user")
170+
login = user.get("login") if isinstance(user, dict) else None
171+
if isinstance(login, str) and login.casefold() == author.casefold():
146172
return found
173+
key = found.get("id", len(tally.ignored))
174+
if key not in tally.ignored:
175+
tally.ignored.add(key)
176+
# The login and the association come from the comment, so they are untrusted: escaped, and quoted.
177+
who = f"@{login}" if isinstance(login, str) else "an unknown author"
178+
association = found.get("author_association")
179+
print(
180+
"::notice title=Reply ignored::"
181+
+ escaped(
182+
f"A comment with the marker and this attempt's link is by {who} (association {association}), "
183+
f"not by the expected author @{author}, so it is ignored"
184+
),
185+
flush=True,
186+
)
147187
if len(comments) < PAGE_SIZE or time.monotonic() >= deadline:
148188
break
149189
return None
150190

151191

152-
def wait_for_reply(token: str, number: int, url: str) -> dict[str, Any]:
192+
def wait_for_reply(token: str, number: int, url: str, author: str) -> dict[str, Any]:
153193
deadline = time.monotonic() + WAIT_SECONDS
194+
tally = Tally()
154195
problem = ""
155196
while True:
156197
try:
157-
reply = find_reply(token, number, url, deadline)
198+
reply = find_reply(token, number, url, author, deadline, tally)
158199
problem = ""
159200
except Unread as failure:
160201
reply = None
@@ -164,10 +205,14 @@ def wait_for_reply(token: str, number: int, url: str) -> dict[str, Any]:
164205
if time.monotonic() + POLL_SECONDS > deadline:
165206
break
166207
time.sleep(POLL_SECONDS)
208+
count = len(tally.ignored)
167209
message = (
168-
f"No reply from the stub routine within {WAIT_SECONDS:g} seconds: no comment on pull request #{number} "
169-
f"contains `{MARKER}` and this run attempt's link, {url}. Check that the stub routine's prompt asks it to post "
170-
"that comment, and that it can comment on this repository (see Self-test in the README)"
210+
f"No reply from the stub routine within {WAIT_SECONDS:g} seconds: no comment on pull request #{number} by "
211+
f"@{author} contains `{MARKER}` and this run attempt's link, {url}. The last read took {tally.read} "
212+
f"{'comment' if tally.read == 1 else 'comments'} from the pull request, and in all {count} with the marker "
213+
f"and this attempt's link were ignored for being from another author. Check that the stub routine's prompt "
214+
f"asks it to post that comment, that it posts as @{author}, and "
215+
"that it can comment on this repository (see Self-test in the README)"
171216
)
172217
if problem:
173218
message += f". The last read of the comments failed: {problem}"
@@ -193,11 +238,12 @@ def report(reply: dict[str, Any]) -> None:
193238

194239

195240
try:
241+
author = reply_author()
196242
url = run_url()
197243
number = requested_pull_request(
198244
os.environ.get("DECISION", ""), os.environ.get("SENT", ""), os.environ.get("EVENT_ACTION", "")
199245
)
200-
report(wait_for_reply(os.environ.get("GITHUB_TOKEN", ""), number, url))
246+
report(wait_for_reply(os.environ.get("GITHUB_TOKEN", ""), number, url, author))
201247
except Failed as failure:
202248
error(str(failure))
203249
sys.exit(1)

‎tests/test_self_test.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,14 @@ def test_the_verify_job_reads_the_review_workflows_outputs(self):
221221
value(".jobs.verify.steps[] | select(.id == \"verify\") | .env.SENT"), "${{ needs.review.outputs.sent }}"
222222
)
223223

224+
def test_the_verify_step_accepts_a_reply_only_from_the_stub_routines_account(self):
225+
self.assertEqual(
226+
value(".jobs.verify.steps[] | select(.id == \"verify\") | .env.REPLY_AUTHOR"), "cipherstash-agent"
227+
)
228+
229+
def test_the_workflow_does_not_filter_the_reply_by_author_association(self):
230+
self.assertNotIn("author_association", SELF_TEST.read_text(encoding="utf-8"))
231+
224232
def test_the_review_workflow_gives_its_decision_and_whether_it_sent_the_request(self):
225233
outputs = '.on.workflow_call.outputs'
226234
self.assertEqual(_yq(f"{outputs}.decision.value").strip(), "${{ jobs.decide.outputs.decision }}")

0 commit comments

Comments
 (0)