Skip to content

feat: request a Claude review when a team member's pull request is ready - #11

Merged
tobyhede merged 62 commits into
mainfrom
feat/ready-review-requests
Nov 19, 2026
Merged

tobyhede merged 62 commits into
mainfrom
feat/ready-review-requests

Conversation

@cipherstash-agent

@cipherstash-agent cipherstash-agent commented Nov 18, 2026 •

Copy link
Copy Markdown

Summary

A team member's pull request now gets a Claude review when it is opened, reopened, or marked ready for review, through one shared reusable workflow instead of a copy in every repository.
Pull requests from people outside the developers team, drafts, bots and forks get no review.
A pull request opened by cipherstash-agent is reviewed without a membership check, because the review agent opens most rollout pull requests and a pull request opening cannot start a review loop.

This is the first of three stacked pull requests for the review requests in #1: this one (Ready), then the comment command (#6), then the requested reviewer (#7).

Changes

  • Reusable workflow: new claude-review.yml, called with the routine token, the membership App's id and private key, the routine trigger id, and optionally the team (default developers) and review agent (default cipherstash-agent).
    • decide job: decides in two phases. If a membership lookup is needed it creates a short-lived App token with only "Members: read", looks the requester up in the team, and decides again. It never fails the check; problems show as ::error:: annotations.
    • fire job: runs only when the decision is to fire. It keeps all the hardening of the copies it replaces: token on curl's standard input, trigger id format check, jq-built body, no retry, short timeouts, continue-on-error, permissions: {}, one fire request per pull request at a time.
  • Request decision: the jq program lives in request-decision/decision.jq, which the reusable workflow's decide job runs and the saved-case runner (request-decision/test) reads, so the tests run exactly what ships.
  • Request text: the first line is unchanged (Review pull request #<n> in <owner/repo>: <url>); a second paragraph names the kind of review request, such as Kind of review request: ready.
  • Tests: 20 saved cases covering every Ready rule, including an org member outside the team, a failed lookup, an unexpected membership value, a deleted fork, a fork opened by the review agent, a payload with no author, and a title and branch name full of quotes, $(...) and newlines.
  • Example caller: examples/claude-review.yml, linted by the Checks workflow alongside the repository's own workflows.
  • README: inputs, secrets, the caller, the decision contract and its reason codes.

Verification

  • request-decision/test: 20 cases, 0 failed. Each rule was added red, then green.
  • Checks on the head commit: actionlint (with shellcheck), zizmor, actions-lock, python-checks (Ruff and mypy) and tests on Python 3.10, 3.12 and 3.14 (with the saved cases on 3.14).
  • Decide job scripts run locally with a fake runner environment: phase 1 → lookup, phase 2 with member → fire; a failed lookup and an empty decision each produce an error annotation and a skip.
  • Membership lookup run against a local stub API: active member, pending invitation, 404, unknown team, failed token, invalid names and a connection failure each map to the expected result.
  • Pilot: ci: request a Claude code review when a pull request is ready ore.rs#143 now calls this workflow pinned to this branch's head. Marking it ready produced one run in which review / decide and review / fire both succeeded, and the routine accepted the request.
  • Not yet verified: the live membership lookup. The pilot pull request was opened by the review agent, so it fires without one.
  • Code review: one should-fix (the request did not name the kind of review request) and seven nits, all fixed: the decide job's final step can no longer fail, outputs are written with a random heredoc delimiter, lookup names reject leading dots, and a payload with no author is skipped.

Python steps and the decision live in files (#14, #15, #17, #20, #22–#24, #26)

The steps that call APIs are standard-library Python, tested offline against a local HTTP stub. Since #20 they live in ordinary files in this repository instead of inside the workflow:

  • Where the code lives: request-decision/decision.jq (the request decision, unchanged), steps/lookup.py (membership lookup, Move the membership lookup to Python, tested against a local HTTP stub #14), steps/report.py (decision fallback, Move the decision fallback step to Python, with tests #17), steps/fire.py (routine call, Move the routine call to Python, tested against the stub #15), and steps/helpers.py (the helpers every step shares: redirect blocker, time limits, annotation escaping, output writing, token redaction).
  • How the workflow reaches them (Self-checkout at the pinned commit, proven with the jq request decision #22): the decide and fire jobs start by checking out ${{ job.workflow_repository }} at ${{ job.workflow_sha }}, which is this repository at the commit the caller pinned. The checkout is sparse (only request-decision/decision.jq and steps/), keeps no credentials, goes into .cipherstash-workflows, and is skipped if either value is empty, so it can never fall back to the caller's code.
  • If the checkout fails: decide leaves the decision-failed skip with an error annotation, and fire sends nothing and shows an error. No check turns red. The same holds if a step file or helpers.py is missing.
  • Behaviour is unchanged. This is a move: the same outputs, annotations, time limits, size caps and redirect handling.
  • Tests run the files that ship. The saved-case runner reads decision.jq directly, the harness runs each step file as a process, and tests tie every path in the workflow to a real file and check that each job's checkout covers the files it runs. The test that kept copied helpers identical now has nothing to compare on this branch.
  • Checks (Ruff, mypy and a Python 3.10, 3.12 and 3.14 test matrix #26): Ruff and mypy (hash-pinned) lint and type-check all Python, and the saved cases and unit tests run on Python 3.10, 3.12 and 3.14. The check names changed: request-decision, unit-tests and unit-tests-python-3-10 are now python-checks and tests (Python 3.10/3.12/3.14). The main ruleset requires no status checks today, so nothing breaks, but update any required checks to the new names.
  • actionlint: .github/actionlint.yaml ignores exactly two messages in claude-review.yml, because actionlint v1.7.12 predates GitHub's job.workflow_repository and job.workflow_sha. Remove it when actionlint knows them.

Proven live (19 November 2026): the ore.rs pilot, pinned to ecf98782…, ran once (run). The decide and fire jobs both checked out this repository at the caller's pin with a token that has no contents permission, decided fire, and the routine accepted the request (HTTP 200). Details are on #22.

Supply-chain checks (#27)

Related

Review notes


Generated by Claude Code


Generated by Claude Code

Adds the reusable workflow every repository will call instead of carrying
its own copy, starting with the Ready kind of review request: a non-draft
pull request opened or reopened, or a draft marked ready for review.
Drafts, bots' pull requests, forks and deleted forks are skipped before any
lookup, and an author outside the team gets no review. Membership is looked
up with a short-lived token from the membership App (Members: read only);
when it cannot be checked the request is skipped with an error, never
assumed. Pull requests the review agent opens are reviewed without a
lookup, because it opens most of them.

The request decision's jq program now lives inline in the workflow: a
reusable workflow runs in the caller's repository and cannot read files
from this one. request-decision/decide reads the program out of the
workflow with yq, so the saved cases test exactly what ships. The decision
runs in two phases (decide, then decide again with the lookup's result),
which keeps it pure and offline.

The routine trigger call keeps all the hardening of the copied workflow:
token on curl's standard input, trig_ format check, body built by jq, no
retry, short timeouts, continue-on-error, no permissions, and one request
per pull request at a time without cancelling one in flight.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
Each repository will carry only this short caller, so it is kept here as
the copy to start from, pinned to the reusable workflow by full commit SHA.
It lives under examples/ rather than .github/workflows/ so it does not run
in this repository; actionlint is given the file paths explicitly so the
example is still linted on every pull request.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
Repositories adopting the shared workflow need to know which settings to
pass and what a run tells them, and contributors changing the rules need
the decision's input and output contract, its skip reasons, and how saved
cases now carry the review agent and the membership lookup's result.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The spec requires the body to say which kind of review request it is, so a
session knows why it was asked. The text now has a second paragraph, `Kind of
review request: <kind>.`, built from the decision's own kind field so the
other kinds reuse it. The first line is unchanged. The four fire cases'
expected decisions and the README are updated; the cases failed before the
change.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
Every value written to GITHUB_OUTPUT now goes through one helper that uses a
random heredoc delimiter, so no value can inject another output. The first
step writes a single needs-lookup output, which replaces the condition that
was repeated on three steps. The team and organisation names must start with
a letter or digit (a team may also start with an underscore), which keeps `.`
and `..` out of the membership URL; a requester login is letters and digits
joined by single hyphens or underscores (managed-user logins use underscores).
The Report step now has continue-on-error, writes the decision-failed skip
before it does anything else, and passes on only a skip or a fire, so the fire
job's `if` always sees valid JSON.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The reasons table left out decision-failed, and the README said the decide job
never fails, which is now true only because the report step cannot fail and
falls back to that skip. The table also covers an unexpected membership answer
and a payload with no author.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
A fork pull request by the review agent is still skipped as a fork, because the
fork check comes before the agent shortcut. An unexpected membership value
skips as membership-lookup-failed instead of being trusted. A draft marked
ready for review fires for a member.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
A null user made the decision ask for a membership lookup with a null requester,
and fire with a null requester for a member. The saved case failed before the
change. It now skips as not-a-review-request, an existing reason, so the
README gains no new code.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
actionlint (shellcheck SC2155) flagged the output helper, because a failing
command substitution inside `local x=$(...)` is masked. Splitting the two keeps
the failure visible.

Refs #5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-18T02:08:26.744779Z 27aea00 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The lookup step was shell and curl, which could only be checked by running
the workflow against GitHub. It is now standard-library Python (3.10
syntax), still inline in the reusable workflow, and a unittest suite reads
it out of the workflow with yq and runs it against an HTTP stub on
127.0.0.1. The tests cover each answer GitHub can give, a missing or
failed token, invalid names, a timeout, a refused or dropped connection,
and check that the token appears in no output.

Behaviour is kept, with three deliberate differences the tests pin down:

- A 200 whose body is not a JSON object now fails the lookup. The shell
  step treated it as not-member, which the lookup's own rule (anything but
  a clear answer is lookup-failed) did not intend.
- A redirect is still not followed, now explicitly, so the token cannot
  be sent to another host.
- Annotation text is escaped as workflow commands require, so a value
  cannot end the annotation or start another one.

The Checks workflow runs the unit tests on every pull request, and the
README says how to run and add them. The harness is shared by the next
steps to move to Python.

Refs #14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The step that always leaves the fire job a finished decision was shell
and jq. A fire decision of an unexpected shape made its summary line's jq
fail, which failed the step. It is now standard-library Python, inline in
the workflow, and catches anything unexpected, so it always exits 0 and
the decision-failed skip it writes first still stands.

Behaviour is otherwise unchanged and pinned by tests/test_report.py: a
finished skip or fire passes through as written, the decision after the
lookup wins over the first, a leftover lookup, an empty value or anything
that is not a JSON object becomes decision-failed with an error, a failed
membership lookup adds its error, and the summary line reads as before.
The summary line is kept to one line, so no value can start a workflow
command.

Refs #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The routine trigger call is a POST, and its tests need to see the exact
request body as well as the method, path and headers. StubServer now
answers POST with the same routes as GET, and records each request's body
in a list alongside the existing (method, path, headers) records, so the
lookup tests that unpack those records are unchanged.

Refs #15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The fire step was shell and curl, which could only be checked by firing
the real routine. It is now standard-library Python (3.10 syntax), inline
in the reusable workflow with id "send", and tests/test_fire.py runs it
against the stub on 127.0.0.1. The routine API's address is a constant in
the script; the tests swap it through the harness's substitutions, so no
input or event field can change where the token goes.

Kept as before: the token is checked first, then the trigger id against
trig_[A-Za-z0-9]+; one POST to the same endpoint with the same four
headers; the body is {"text": <decision text>} written by a JSON encoder;
short timeouts (10 s to connect and per read, 30 s in all); no retry;
continue-on-error keeps a failure from turning the check red.

Pinned by the tests, with these deliberate differences from curl:

- Only a 2xx answer counts as sent. curl's --fail-with-body passed a 3xx;
  a redirect is now an error and is not followed.
- A failure is one ::error:: annotation with the HTTP status and at most
  200 characters of the answer, on one line, with the token taken out.
  curl printed the whole answer as plain text, so an answer that repeated
  the token printed it.
- A decision with no string text is an error and nothing is sent. jq
  would have sent {"text": null}.
- The step writes sent=true only when the request was accepted, which
  the react job (#16) will read.

Refs #15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
cipherstash-agent pushed a commit that referenced this pull request Nov 18, 2026
Brings the Python rewrites from #11's branch up to the comment command, so
the comment command ships the tested steps rather than the shell ones they
replace:

- #14: the membership lookup step in Python, with the stub-server harness
  and its unit tests.
- #17: the decision report step in Python, so it cannot fail.
- #15: the routine trigger call in Python, and a stub server that answers
  POST requests and records their bodies.

Conflicts:

- claude-review.yml, the fire job's send step: took #11's Python step
  whole. Kept this branch's job-level `outputs: sent` and its comment (the
  Python step writes `sent=true` under the same id `send`) and the `react`
  job. The fire job's comment now says the send step's json module, not
  jq, places the decision's text in the request body.
- README.md, "What a run shows": kept #11's fuller description of `fire`
  and this branch's `react` bullet.

Both suites pass: request-decision/test (53 cases) and the unit tests (53).

Refs #14 #15 #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
cipherstash-agent pushed a commit that referenced this pull request Nov 18, 2026
Brings the comment command's branch, with the Python steps it took from
#11's branch and the Python reaction, up to the requested reviewer, so the
requested reviewer ships the tested steps:

- #14: the membership lookup step in Python, with the stub-server harness.
- #17: the decision report step in Python.
- #15: the routine trigger call in Python.
- #16: the 👀 reaction in Python, tested against the stub, and the harness's
  warnings().

The workflow merged without conflicts. README.md, "What a run shows",
conflicted: kept the comment command branch's fuller `fire` and `react`
descriptions, with this branch's note that `react` never runs for Ready or
a requested reviewer.

Both suites pass: request-decision/test (68 cases) and the unit tests (67).

Refs #14 #15 #16 #17

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The steps promise Python 3.10 features only, but the tests ran on whatever
python3 the runner has, so a newer-only call could pass CI and break on an
older image. The grammar check cannot see library calls.

Refs #14

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
…ep tests

The lookup had no total timeout (the shell step had curl --max-time 30) and
read unbounded bodies, so a server that drips bytes could hold it until the job
was killed. It now uses the send step's total timer and reads at most 64 KiB,
reporting a larger answer as malformed. The send step disarms its timer as soon
as the answer is complete, so a late alarm cannot turn an accepted request into
a failure, and both steps restore the previous SIGALRM handler and close
HTTPError objects. The proxy comment no longer claims curl's ALL_PROXY.

The total-timeout tests could not fail because the per-read timeout gave the
same message; a DRIP stub route (one byte at a time, never ending) now makes
them prove the total timer. Redirect tests cover 301/302/303/307/308, the
injection test rejects any other workflow command, the Python-step test
requires an id, dead harness code is removed, and the README lists the new
lookup-failed cases.

Refs #14
Refs #15
Refs #17

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The send step read the answer's body with the 30 s time limit still
running, even on a 2xx. A body that arrived slowly after the routine
trigger had accepted the request raised TotalTimeout, so the step
reported "not sent", wrote no `sent` output and skipped the reaction,
though a review session had started. The step now ends the time limit
as soon as opener.open returns (it returns only a 2xx answer) and does
not read the body. An error answer's body is still read under the
limit for the excerpt; if that read fails or runs out of time, the
step reports the status it already has. The membership lookup keeps
reading a 200's body under the limit, because the body is the
membership, and a body cut off there is a lookup-failed.

The redirect tests could not see where a followed redirect would go,
so they now point Location at a second recording server and check that
no request, and no Authorization header, reaches it. StubServer gains
per-route headers and drip(status) for an answer of any status whose
body never ends.

The Python steps are inline and cannot share a module, so their
scaffolding is copied. The copies had already drifted (TotalTimeout's
docstring), and a hardening fix could land in one copy only. The
alarm handling moves into a shared total_time_limit() context manager,
which also puts the previous SIGALRM handler back. Escaping moves into
escaped(). tests/test_python_steps.py now checks that every step that
makes a request defines all of the scaffolding and that every copy is
identical.

Comments describe current behaviour only: the proxy note no longer
compares urllib with curl.

Refs #14
Refs #15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
cipherstash-agent pushed a commit that referenced this pull request Nov 18, 2026
Brings in from #11: the membership lookup bounded in time and size
(a1a4225), the Python 3.10 unit-test job, and the send step that
counts an accepted request as sent without reading its body, with the
shared total_time_limit() and escaped() scaffolding, the
recording-redirect tests, StubServer.drip(status) and per-route
headers, and the scaffolding drift test (0309405).

Conflicts resolved:
- README.md, "Adding a unit test": kept this branch's run_step line
  (repository=, .warnings()) and #11's StubServer line (headers,
  drip(status)).
- tests/harness.py: dropped RUNNER_VARIABLES, as #11 did; nothing uses
  it. This branch's GITHUB_REPOSITORY and warnings() merged cleanly.
- The workflow merged without conflict. The lookup, final and send
  steps are byte-identical to #11's, so #11's lookup hardening holds;
  the react job is this branch's.

Semantic conflict: #11's drift test covers every step that makes a
request, which now includes react. So that the merge is green, react's
copies of the shared definitions are brought in line (TotalTimeout's
docstring, escaped(), with warning() built on it, total_time_limit()).
react's post() is unchanged here; the next commit moves it onto
total_time_limit() and hardens it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
cipherstash-agent pushed a commit that referenced this pull request Nov 18, 2026
Brings in from #12, by way of #11: the send step that counts an
accepted request as sent without reading its body, the shared
total_time_limit() and escaped() scaffolding and the test that keeps
every copy identical, the recording-redirect tests,
StubServer.drip(status) and per-route headers (0309405, efe1a2c), and
the react step's hardening: the previous SIGALRM handler restored, the
time limit ended as soon as the status is known, and redirect tests
over 301, 302, 303, 307 and 308 (85b0d06).

No conflicts. The workflow and README merged cleanly. The lookup,
final, send and reaction steps are byte-identical to #12's, and this
branch's requested-reviewer decision and cases are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
The fire step's new checkout tests, merged in, had lines over Ruff's limit of
120 characters (E501), which would fail the python-checks job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The override list in pyproject.toml is not enforced by mypy, so a new untyped
steps/*.py passed CI. A unit test now compares the list with steps/*.py.

Also: install Python tools with --only-binary locally as CI does, stop
claiming ubuntu-latest's Python version, and have Dependabot update the
hashed pip requirements files.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Ruff's S101 rejects assert; fail the test explicitly when the override is missing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
lookup.api and fire.post each repeated the same total time limit, opener
call, HTTPError handling and "status already known" rule, and drifted
apart in small ways. steps/helpers.py now has one send() that both use,
raising Unreachable with the failure's kind, which each step turns into
its own message, unchanged. github_headers() holds the GitHub API
headers in one place.

Each step keeps its exact behaviour: the lookup's answer still counts
only once its body is read (or, for an error status, once the answer is
closed), and the routine call's status still counts as soon as it
arrives, with the error body read for the excerpt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
report.py printed its three ::error:: lines by hand. Each message is
fixed text or an exception's class name, with nothing to escape, so
helpers.error() prints the same bytes and the step now uses it like the
others.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Each STEP_FILE and the request decision's path spelled out the whole
${{ github.workspace }}/.cipherstash-workflows prefix. Each job that
checks out this repository now names that directory once, as
WORKFLOW_FILES in its env, and each step builds its path from
${{ env.WORKFLOW_FILES }}. It is job-level rather than workflow-level
env because the decide job already relies on github.workspace there;
job env cannot read the env context, so REQUEST_DECISION_FILE moves to
the two steps that run the decision.

The harness reads STEP_FILE and REQUEST_DECISION_FILE through the job's
WORKFLOW_FILES, and a new test checks that every such job sets it to
the checkout's path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The first decide step ran jq for the decision's kind and then turned
"lookup" into true or false with an if/else. jq's own comparison prints
the same true or false, so one call does both. Checked against every
saved case: the step writes byte-identical outputs and request file.

The second step keeps its output() helper: the decision is compact JSON,
but a jq program could print more than one value, one line each, so a
plain name=value line is not provably safe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…ster

The tests ran yq over the unchanging workflow about 120 times per run,
most of them repeats, so _yq is now cached. The stand-in server polled
for shutdown every 50 ms, which every test that starts one waited out;
it now polls every 5 ms. The unit tests take about 17 s instead of 25 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Every Python step runs from a file in steps/, so the harness's inline
path (PYTHON_SHELL, python_steps(), running a script read out of the
workflow) and the checks that inline copies of the shared code match
steps/helpers.py could no longer run. One test now fails if any step
names a Python shell. run_step no longer checks file_steps() first,
because step_file_path already raises for a step with no step file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
test_report.py and test_fire.py each copied a step file into a
temporary directory, with or without helpers.py, by hand; the harness
now does it once, in step_file_copy(). Tests find step files with
step_file_path() instead of spelling ROOT/steps/<name>.py, and read one
setting of a step with step_field() instead of repeating the
select(.id == ...) query.

test_fire.py names the not-checked-out error once, as NOT_CHECKED_OUT,
as test_react.py does, and its missing-file test uses the same
assert_one_error_and_nothing_sent as the others, which asserts the same
things. test_report.py no longer compares the step with the old inline
step, which is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
…very job

CheckoutTest checked, for the decide job only, that the checkout path is
the harness's and that every steps/*.py file is checked out.
StepFilesTest.test_each_step_file_is_checked_out_by_its_own_job checks
both for every job that runs a step file, decide included, so the
decide-only copies are gone. Its checkout queries use step_field().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
request-decision/test piped the decision's compact output into a second
jq to sort and pretty-print it. jq --sort-keys does both in the same
call and prints the same text; every saved case gives byte-identical
output either way, and request-decision/test-runner still passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
request-decision/test and request-decision/test-runner use only bash and
jq, so running them in each Python version's job checked the same thing
three times. The matrix now marks the 3.14 leg with saved-cases, and
only that leg runs them; the unit tests still run in every leg, and the
job names are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@cipherstash-agent
cipherstash-agent changed the base branch from ci/checks to main November 18, 2026 23:41
@tobyhede
tobyhede added this pull request to stack #28 November 18, 2026 23:41
A timeout or a connection dropped after the request went out can still
mean the routine accepted it, so "no review was requested" was wrong and
invited a re-run that reviews twice. Those two now say to check for a
review first; a refused connection or DNS failure keeps the old message.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
With cancel-in-progress: false only one run can wait, so a third request
for the same pull request replaces it and that fire job shows cancelled.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The annotation carried the lockfile as text and base64 plus the diff, which
can pass GitHub's annotation size limit. It now points to the job summary,
which holds the full text. Also says that the deletion guard needs a base
branch, so it covers pull requests only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
The list of fully typed step modules had to be kept by hand, with a test to
catch a forgotten entry. Strict is now the default and only the test files
and their harness are relaxed, so a new step file is checked without any
list. mypy rejects a partial wildcard like test_*, so the test files are
named; a new one that is missing fails mypy loudly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Set the status result once and close the answer in one finally, instead of
branching on the same condition in two places. Behaviour is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
A new push made the older run's results useless but they still ran to the
end. Runs on main are kept, so each commit there keeps its own result.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr

@coderdan coderdan left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Recommendation: merge, after re-pinning the example caller (finding 1). Or merge the whole stack together, so that #13's pin is the one that lands on main.

Findings

  1. Should fix: the example caller pins an old version. examples/claude-review.yml pins fbf70f79, the first commit on this branch. That is the version with the Python written inline in the workflow, from before the move to steps/ (#20, #22) and the hardening after it. Between that pin and this branch's head, the shipped files differ by 6 files, +606/−129 lines. If this pull request merges on its own, the example on main points at code that main does not ship. Re-pin it to this branch's head, or to the merge commit after merging. (#12 and #13 pin commits whose shipped files match their heads, so only this one is stale.)
  2. Merge note: keep the pinned commits. The example and the ore.rs pilot (ecf98782…) pin commits on the feature branches. Merge the stack with Create a merge commit so those commits stay in main's history. A squash merge leaves them reachable only through refs/pull/N/head. Also, delete_branch_on_merge is off for this repository, so #12 will not be retargeted to main automatically when this one merges.
  3. Nit: parts of the description are out of date. "The jq program lives inline in the reusable workflow … request-decision/decide extracts it with yq" and "actionlint … and request-decision both pass" describe the layout before #20. decide is deleted, the decision is now request-decision/decision.jq, and the check names changed. Worth fixing so the history is not misleading.

What I checked

  • claude-review.yml. decide and fire hold no permissions. Untrusted text reaches scripts only through the payload file and environment variables, never through ${{ }} in a run:. The checkout is skipped when job.workflow_* is empty, so it cannot fall back to the caller's code. Every step that can fail has continue-on-error, and report writes the decision-failed skip first. fromJSON on the job output always gets valid JSON.
  • decision.jq (Ready): the order of the draft, bot, fork and deleted-fork checks, and the review agent's bypass ignoring case.
  • steps/. The token is only ever sent in a header and is redacted from excerpts. Redirects are not followed. The routine's base URL is fixed. Membership is never assumed: a pending invitation, any status but 200/404, and a malformed or oversized body all count as "not a member". A 404 is checked against the team, so a mistyped team slug does not skip everyone without a warning. Names are validated before they go into the URL.
  • Saved cases: 20 of 20 pass locally. I couldn't run the unit tests locally (no yq on this machine), but CI is green on 3.10, 3.12 and 3.14, and zizmor and actionlint pass too.

For information, not a defect: on pull_request events, anyone who can push a branch can edit the caller workflow in that branch. They can then skip the team check and read the secrets. That is GitHub's model, so the team check decides who gets a review by default. It does not limit who can reach the routine token. The comment command in #12 does not have this problem, because it runs from the default branch.


Review by Claude Code

Comment thread examples/claude-review.yml Resolved
coderdan
coderdan previously approved these changes Nov 19, 2026
…pped file

The example pinned the stack's first commit, which predates steps/ and the
hardening, so a copy of it would run the old inline workflow rather than the
one this branch ships. 8d6d526 is the last commit on the branch that changed
a shipped file (steps/helpers.py).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
Brings the stack up to date with main.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Jmn7nyyNodWTyouEMfLWr
@tobyhede
tobyhede merged commit a3accd7 into main Nov 19, 2026
7 checks passed
@tobyhede tobyhede self-assigned this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment