Skip to content

feat(self-test): self-test the comment command - #35

Merged
tobyhede merged 10 commits into
mainfrom
feat/self-test-comment
Nov 20, 2026
Merged

tobyhede merged 10 commits into
mainfrom
feat/self-test-comment

Conversation

@cipherstash-agent

@cipherstash-agent cipherstash-agent commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

The self-test now covers the third kind of review request, the comment command. When a team member comments @cipherstash-agent review on a pull request in this repository, the self-test calls the review workflow with the stub routine, then checks the decision, the 👀 reaction on the comment, and the stub's reply for this run. GitHub runs a comment's workflows from the default branch, so this part tests main's review workflow, not the pull request's, and it cannot run at all until this pull request is merged to main.

Changes

  • Workflow (.github/workflows/self-test.yml): adds the issue_comment (created) trigger. The review job's condition lets through only a comment the review workflow takes as a comment command: an open pull request, not by a bot or by cipherstash-agent (so the stub's reply can never start a run), and a first line of @cipherstash-agent review followed by the end, a space, a tab or a line break. Comments are not filtered by author_association, because GitHub reports a team member whose organisation membership is private as CONTRIBUTOR or NONE. The review workflow's team membership check decides. The skipped job now runs for pull_request events only. The verify step gets EVENT_NAME, COMMENT_ID and REACTION_AUTHOR: github-actions[bot].
  • Verify step (steps/verify.py): picks the expected kind by event name and action. For a comment command, it checks that the decision is for the comment that started the run, then waits up to 60 seconds for an eyes reaction by github-actions[bot], then waits for the stub's reply as before. Reading comments and reactions shares one function, with the same refusal, retry and size rules.
  • Tests: tests/test_self_test.py evaluates the job conditions for comment events (its expression evaluator knows startsWith, fromJSON and format) and runs the real request decision on sample comments to check that, for ASCII text, the filter never lets a non-command through. tests/test_verify.py covers the reaction check.
  • README: a "Comment commands" section: what it tests (main), how to run it, which comments start nothing, why fork pull requests are tested, and the new failure messages.

Verification

  • request-decision/test-runner: passed. request-decision/test: 79 cases, 0 failed. The saved decision cases are unchanged.
  • python3 -m unittest discover -s tests: 217 tests, all passed.
  • ruff check .: passed. mypy: no issues in 18 source files.
  • CI on the head commit: all 7 checks pass, including actionlint and zizmor.
  • The comment-command path cannot run live from this pull request: it runs only once the self-test is on main.

Related

Closes #31. #33 and #34 are merged; this pull request is based on main.

Review notes

  • The issue_comment trigger carries a zizmor ignore comment with its reason, approved by the repository owner. The workflow is written defensively all the same: the comment's body and logins appear only in the job condition, never in a script; verify gets only the numeric comment id, by env; permissions: {} at the top and issues: read for verify.
  • Anyone who can comment can post a comment command. A command from someone who is not a team member starts a run that ends red with not-a-team-member, by design. No request is sent for it.
  • Forks: a comment's event does not say whether the pull request is from a fork, and the review workflow takes comment commands on forks, so they are tested. A comment's run has secrets, but every job uses main's files only and nothing from the fork is checked out or run.
  • The reaction must be by github-actions[bot], the account a job's GITHUB_TOKEN acts as, added with issues: write. GitHub's documentation says that permission is enough. This could not be confirmed live before merge.
  • On "Re-run all jobs", a 👀 left by an earlier attempt still counts. The README states this limit.
  • A command with unusual spacing (two spaces before review) is a command to the review workflow but does not start the test: a job condition cannot match patterns.

🤖 Generated with Claude Code

https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr

Copy link
Copy Markdown
Author

integration-test: received ready request from https://github.com/cipherstash/workflows/actions/runs/44460121752/attempts/1


Generated by Claude Code

@tobyhede
tobyhede added this pull request to stack #36 November 19, 2026 22:16
@tobyhede
tobyhede force-pushed the feat/self-test-comment branch from 9a97d6c to 7e6c92a Compare November 19, 2026 22:30
Base automatically changed from feat/self-test-rerun to main November 19, 2026 22:30
The self-test covered Ready and requested-reviewer requests, but not the
third kind, the comment command, so a change that broke it could merge
unnoticed. The workflow now also runs on new comments, and the review job
calls the review workflow (main's, as comments always run from the
default branch) with the stub routine for a comment command.

The review job's condition is a filter that lets through only comments
the review workflow takes as a comment command: the first line must be
`@cipherstash-agent review` followed by the end, a space, a tab or a line
break. fromJSON supplies the tab and line breaks, which an expression's
string cannot hold. Comments by bots or by the review agent (so the
stub's own reply can never start a run), comments on closed pull
requests or plain issues, and comments from outside the repository's
members start no job, as the review workflow would skip them and the
verify job would then fail. A test runs the request decision itself on
each sample comment, so the filter can never let a non-command through.

The skipped job now runs only for pull_request events: a comment's
payload has no pull_request, and its fork test would otherwise pass.
The verify step is given the event name, the comment's id and the
account the reaction must come from, for the checks that follow.

Refs #31

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
A comment command has one effect the other kinds of request do not: the
review workflow's react job adds a 👀 reaction to the comment, so the
person who asked knows it was received. That job keeps going on a
failure (a missing reaction is only a warning there), so nothing else
would notice if it broke.

For a run started by a comment, the verify step now requires a
comment-command decision for that very comment (the decision's
comment_id must equal the event's), then reads the comment's `eyes`
reactions for up to 60 seconds and accepts only one by the account the
react job's token acts as, github-actions[bot], so a 👀 added by hand
does not pass. It then waits for the stub's reply as before. The
reactions are read with the job's existing `issues: read`.

The event's name now joins its action in choosing the expected kind,
because an issue_comment event's action, created, says nothing alone.
The comment-listing code is shared with the reactions read, so both
keep the same refusal, retry and size rules.

Refs #31

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The comment-command part of the self-test differs from the rest in ways
a reader cannot guess from the workflow: a comment always runs main's
files, so it tests main, not the pull request, and cannot run until this
stack is merged; it checks a 👀 reaction from a particular account; and
most comments start no job at all. The README now says so, how to run
it, why fork pull requests are tested even though a comment's run has
secrets, which commands the filter misses, and what each new failure
message means.

Refs #31

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Two comments in the self-test workflow ran past the 120-column width the
rest of the file keeps, after the comment-command change added to them.

Refs #31

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The review job calls the pull request's own review workflow only for Ready and requested-reviewer requests. A comment command uses main's version. The README now says so, and marks the reaction author as not yet confirmed by a live run. It also notes that Re-run all jobs keeps the first attempt's reaction on a comment run.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
zizmor flags any workflow that runs on issue_comment, because anyone who can
comment can start it. The self-test acts only on a team member's comment
command and runs no pull request code, the same reasons the example caller's
ignore gives. The user approved this ignore.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…'s association

GitHub reports a team member whose organisation membership is private as
CONTRIBUTOR or NONE, so the OWNER/MEMBER/COLLABORATOR test silently started no
run for their comment command. The review workflow's team check is the gate: a
non-member's command now starts a run that ends red with not-a-team-member, by
design. Bots, the review agent, closed pull requests and non-commands still
start no job. Tests first failed for CONTRIBUTOR and NONE commenters.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede force-pushed the feat/self-test-comment branch from 7e6c92a to 40a0989 Compare November 19, 2026 22:30
tobyhede
tobyhede previously approved these changes Nov 19, 2026
… guarantees

The settings header said the membership App is used only when the pull
request's author is not the review agent. A comment command and a
requested-reviewer request always need it.

The README said only an owner, member or collaborator who is not on the
team gets a red run. There is no such filter: anyone's command does.

The filter's guarantee is now stated for ASCII text only, because GitHub
does not document how it compares other letters. The comment no longer
names contains, which the condition does not use, and the test's
expression evaluator drops its unused contains branch.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…ng test

Reads a login in one place (login_of) and writes the job summary in one
place (write_summary). Renames the comment id pattern, the function that
returns the requested pull request, and the set of ignored reaction ids,
so each name says what it holds. No message changes.

The test for a reaction that arrives while waiting now adds the reaction
only after the step's first read, instead of after a fixed 0.2 seconds,
so a slow start cannot fail it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede merged commit b493c5e into main Nov 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-test the comment command

3 participants