Repository navigation
feat(self-test): self-test the comment command - #35
Merged
Merged
Conversation
Author
|
integration-test: received ready request from https://github.com/cipherstash/workflows/actions/runs/44460121752/attempts/1 Generated by Claude Code |
3 tasks
cipherstash-agent
force-pushed
the
feat/self-test-comment
branch
from
November 19, 2026 22:20
09bc31a to
9a97d6c
Compare
tobyhede
force-pushed
the
feat/self-test-comment
branch
from
November 19, 2026 22:30
9a97d6c to
7e6c92a
Compare
The self-test covered Ready and requested-reviewer requests, but not the third kind, the comment command, so a change that broke it could merge unnoticed. The workflow now also runs on new comments, and the review job calls the review workflow (main's, as comments always run from the default branch) with the stub routine for a comment command. The review job's condition is a filter that lets through only comments the review workflow takes as a comment command: the first line must be `@cipherstash-agent review` followed by the end, a space, a tab or a line break. fromJSON supplies the tab and line breaks, which an expression's string cannot hold. Comments by bots or by the review agent (so the stub's own reply can never start a run), comments on closed pull requests or plain issues, and comments from outside the repository's members start no job, as the review workflow would skip them and the verify job would then fail. A test runs the request decision itself on each sample comment, so the filter can never let a non-command through. The skipped job now runs only for pull_request events: a comment's payload has no pull_request, and its fork test would otherwise pass. The verify step is given the event name, the comment's id and the account the reaction must come from, for the checks that follow. Refs #31 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
A comment command has one effect the other kinds of request do not: the review workflow's react job adds a 👀 reaction to the comment, so the person who asked knows it was received. That job keeps going on a failure (a missing reaction is only a warning there), so nothing else would notice if it broke. For a run started by a comment, the verify step now requires a comment-command decision for that very comment (the decision's comment_id must equal the event's), then reads the comment's `eyes` reactions for up to 60 seconds and accepts only one by the account the react job's token acts as, github-actions[bot], so a 👀 added by hand does not pass. It then waits for the stub's reply as before. The reactions are read with the job's existing `issues: read`. The event's name now joins its action in choosing the expected kind, because an issue_comment event's action, created, says nothing alone. The comment-listing code is shared with the reactions read, so both keep the same refusal, retry and size rules. Refs #31 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The comment-command part of the self-test differs from the rest in ways a reader cannot guess from the workflow: a comment always runs main's files, so it tests main, not the pull request, and cannot run until this stack is merged; it checks a 👀 reaction from a particular account; and most comments start no job at all. The README now says so, how to run it, why fork pull requests are tested even though a comment's run has secrets, which commands the filter misses, and what each new failure message means. Refs #31 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Two comments in the self-test workflow ran past the 120-column width the rest of the file keeps, after the comment-command change added to them. Refs #31 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The review job calls the pull request's own review workflow only for Ready and requested-reviewer requests. A comment command uses main's version. The README now says so, and marks the reaction author as not yet confirmed by a live run. It also notes that Re-run all jobs keeps the first attempt's reaction on a comment run. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor flags any workflow that runs on issue_comment, because anyone who can comment can start it. The self-test acts only on a team member's comment command and runs no pull request code, the same reasons the example caller's ignore gives. The user approved this ignore. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…'s association GitHub reports a team member whose organisation membership is private as CONTRIBUTOR or NONE, so the OWNER/MEMBER/COLLABORATOR test silently started no run for their comment command. The review workflow's team check is the gate: a non-member's command now starts a run that ends red with not-a-team-member, by design. Bots, the review agent, closed pull requests and non-commands still start no job. Tests first failed for CONTRIBUTOR and NONE commenters. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
tobyhede
force-pushed
the
feat/self-test-comment
branch
from
November 19, 2026 22:30
7e6c92a to
40a0989
Compare
tobyhede
previously approved these changes
Nov 19, 2026
… guarantees The settings header said the membership App is used only when the pull request's author is not the review agent. A comment command and a requested-reviewer request always need it. The README said only an owner, member or collaborator who is not on the team gets a red run. There is no such filter: anyone's command does. The filter's guarantee is now stated for ASCII text only, because GitHub does not document how it compares other letters. The comment no longer names contains, which the condition does not use, and the test's expression evaluator drops its unused contains branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…ng test Reads a login in one place (login_of) and writes the job summary in one place (write_summary). Renames the comment id pattern, the function that returns the requested pull request, and the set of ignored reaction ids, so each name says what it holds. No message changes. The test for a reaction that arrives while waiting now adds the reaction only after the step's first read, instead of after a fixed 0.2 seconds, so a slow start cannot fail it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
This was referenced Nov 20, 2026
tobyhede
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The self-test now covers the third kind of review request, the comment command. When a team member comments
@cipherstash-agent reviewon a pull request in this repository, the self-test calls the review workflow with the stub routine, then checks the decision, the 👀 reaction on the comment, and the stub's reply for this run. GitHub runs a comment's workflows from the default branch, so this part testsmain's review workflow, not the pull request's, and it cannot run at all until this pull request is merged tomain.Changes
.github/workflows/self-test.yml): adds theissue_comment(created) trigger. Thereviewjob's condition lets through only a comment the review workflow takes as a comment command: an open pull request, not by a bot or bycipherstash-agent(so the stub's reply can never start a run), and a first line of@cipherstash-agent reviewfollowed by the end, a space, a tab or a line break. Comments are not filtered byauthor_association, because GitHub reports a team member whose organisation membership is private asCONTRIBUTORorNONE. The review workflow's team membership check decides. Theskippedjob now runs forpull_requestevents only. Theverifystep getsEVENT_NAME,COMMENT_IDandREACTION_AUTHOR: github-actions[bot].steps/verify.py): picks the expected kind by event name and action. For a comment command, it checks that the decision is for the comment that started the run, then waits up to 60 seconds for aneyesreaction bygithub-actions[bot], then waits for the stub's reply as before. Reading comments and reactions shares one function, with the same refusal, retry and size rules.tests/test_self_test.pyevaluates the job conditions for comment events (its expression evaluator knowsstartsWith,fromJSONandformat) and runs the real request decision on sample comments to check that, for ASCII text, the filter never lets a non-command through.tests/test_verify.pycovers the reaction check.main), how to run it, which comments start nothing, why fork pull requests are tested, and the new failure messages.Verification
request-decision/test-runner: passed.request-decision/test: 79 cases, 0 failed. The saved decision cases are unchanged.python3 -m unittest discover -s tests: 217 tests, all passed.ruff check .: passed.mypy: no issues in 18 source files.main.Related
Closes #31. #33 and #34 are merged; this pull request is based on
main.Review notes
issue_commenttrigger carries a zizmor ignore comment with its reason, approved by the repository owner. The workflow is written defensively all the same: the comment's body and logins appear only in the job condition, never in a script;verifygets only the numeric comment id, by env;permissions: {}at the top andissues: readforverify.not-a-team-member, by design. No request is sent for it.main's files only and nothing from the fork is checked out or run.github-actions[bot], the account a job'sGITHUB_TOKENacts as, added withissues: write. GitHub's documentation says that permission is enough. This could not be confirmed live before merge.review) is a command to the review workflow but does not start the test: a job condition cannot match patterns.🤖 Generated with Claude Code
https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr