Skip to content

Commit 76b8084

Browse files
committed
Merge feat/ready-review-requests into feat/comment-command
Brings the Python rewrites from #11's branch up to the comment command, so the comment command ships the tested steps rather than the shell ones they replace: - #14: the membership lookup step in Python, with the stub-server harness and its unit tests. - #17: the decision report step in Python, so it cannot fail. - #15: the routine trigger call in Python, and a stub server that answers POST requests and records their bodies. Conflicts: - claude-review.yml, the fire job's send step: took #11's Python step whole. Kept this branch's job-level `outputs: sent` and its comment (the Python step writes `sent=true` under the same id `send`) and the `react` job. The fire job's comment now says the send step's json module, not jq, places the decision's text in the request body. - README.md, "What a run shows": kept #11's fuller description of `fire` and this branch's `react` bullet. Both suites pass: request-decision/test (53 cases) and the unit tests (53). Refs #14 #15 #17 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Jmn7ot2xooXpcuW44QqtU
2 parents 8d6e5d2 + d283f2e commit 76b8084

9 files changed

Lines changed: 1079 additions & 95 deletions

File tree

‎.github/workflows/checks.yml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
1-
# Checks every pull request to this repository: lints the workflows and the example callers, and runs the request
2-
# decision's saved cases.
1+
# Checks every pull request to this repository: lints the workflows and the example callers, runs the request
2+
# decision's saved cases, and runs the unit tests of the reusable workflow's Python steps.
33
name: Checks
44

55
on:
@@ -33,3 +33,14 @@ jobs:
3333
persist-credentials: false
3434
- name: Run the saved cases
3535
run: request-decision/test
36+
37+
unit-tests:
38+
runs-on: ubuntu-latest
39+
timeout-minutes: 5
40+
steps:
41+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
42+
with:
43+
persist-credentials: false
44+
# The runner's own python3 and yq; the tests need nothing installed and no network beyond 127.0.0.1.
45+
- name: Run the unit tests
46+
run: python3 -m unittest discover --start-directory tests --verbose

‎.github/workflows/claude-review.yml‎

Lines changed: 324 additions & 91 deletions
Large diffs are not rendered by default.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
__pycache__/

‎README.md‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ All three secrets, `CLAUDE_ROUTINE_TOKEN`, `CLAUDE_REVIEW_APP_ID` and `CLAUDE_RE
9393
A run has up to three jobs:
9494

9595
- `decide` reads the event and decides. It writes a one-line summary, such as `Claude review: skipped (draft)`, to the run's summary page. It does not fail: every step that can fail is allowed to, and the decision then becomes a skip with the reason `decision-failed`.
96-
- `fire` sends the request to the routine trigger. It runs only when `decide` decided to fire. A failed request shows as a failed step, but the job and the check stay green.
96+
- `fire` sends the request to the routine trigger. It runs only when `decide` decided to fire. It sends the request once, with no retry, and does not follow a redirect. Only a `2xx` answer counts as sent. A failed request shows as a failed step with an error giving the HTTP status, but the job and the check stay green.
9797
- `react` adds the 👀 reaction to a comment command's comment. It runs only for a comment command, and only once `fire` has sent the request. If the reaction cannot be added, the run shows a warning, but the job and the check stay green.
9898

9999
`decide` and `fire` hold no permissions.
@@ -157,12 +157,20 @@ Exactly one of:
157157

158158
An event that has no pull request or comment author, such as a payload whose `user` is `null`, is skipped as `not-a-review-request`.
159159

160+
## Python steps
161+
162+
Some steps, such as the membership lookup and the request to the routine trigger, are written in Python.
163+
They use only Python's standard library and keep to Python 3.10, so nothing is installed when they run.
164+
Like the request decision, each one is kept inline in the reusable workflow, with `shell: python3 {0}` and an `id`, because a reusable workflow cannot read files from this repository.
165+
The unit tests in `tests/` read each step's script out of the workflow, so they test exactly what ships.
166+
160167
## Checks
161168

162-
Every pull request runs two checks:
169+
Every pull request runs three checks:
163170

164171
- `actionlint` over the workflows in this repository and the example callers in `examples/`.
165172
- The request decision's saved cases, run by `request-decision/test`.
173+
- The unit tests of the Python steps, in `tests/`.
166174

167175
The request decision's rules are tested against saved event payloads, so the tests need no network and run anywhere with `bash`, `jq` and `yq`:
168176

@@ -176,3 +184,22 @@ request-decision/test
176184
2. Add `input.json`: the decision's input, with `event_name`, `event` (the event payload, trimmed to the fields that matter), `review_agent` (usually `"cipherstash-agent"`) and `membership` (`null` to test the decision before a lookup).
177185
3. Add `expected.json`: the decision expected for that input, such as `{"decision": "skip", "reason": "draft"}`.
178186
4. Run `request-decision/test` and check the new case passes. A case that fails prints the difference between the expected and actual decision.
187+
188+
### Running the unit tests
189+
190+
The unit tests need `python3` (3.10 or later), `bash` and `yq`, and no network: a test that needs GitHub's API or the routine trigger starts a stand-in server on 127.0.0.1.
191+
192+
```sh
193+
python3 -m unittest discover --start-directory tests
194+
```
195+
196+
### Adding a unit test
197+
198+
`tests/harness.py` runs one step of the reusable workflow on its own:
199+
200+
- `run_step(job, step_id, env, api_url=...)` reads the step's script out of the workflow with `yq`, runs it with the step's shell, and returns what it left: its exit code, standard output and error, its outputs (`.outputs`), its summary (`.summary`) and its `::error::` annotations (`.errors()`). The step sees only the `env` the test gives it, which must be names the step declares in its `env`, plus the runner's own variables, such as `GITHUB_OUTPUT`.
201+
- `StubServer(routes)` stands in for GitHub's API or the routine trigger. `routes` maps a path to a status and body, to `StubServer.HANG` for an answer that never comes, or to `StubServer.DROP` for a connection closed without one, for `GET` and `POST` alike. It records every request's method, path and headers in `.requests`, and its body in `.bodies`.
202+
- `substitutions` replaces text in the script before it runs, such as a timeout constant lowered so a test does not wait, or the routine API's fixed address swapped for the stand-in server's. Each piece of text must appear exactly once in the script.
203+
204+
1. Add a test method to the step's test file, such as `tests/test_lookup.py`, or a new `tests/test_<step>.py` for a new step. Name it for what the step does, such as `test_a_pending_membership_is_not_a_member`.
205+
2. Run the unit tests and check the new test passes.

‎tests/harness.py‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
"""Runs one step of the reusable workflow on its own, with a controlled environment, for the unit tests.
2+
3+
A reusable workflow runs in the calling repository, so its steps' scripts are kept inline in
4+
.github/workflows/claude-review.yml. The harness reads a step's script out of the workflow with yq, so the tests run
5+
exactly what ships, and runs it the way the runner would: with the step's shell, and with GITHUB_OUTPUT,
6+
GITHUB_STEP_SUMMARY and RUNNER_TEMP pointing at temporary files. Nothing else is inherited from the environment
7+
running the tests, so no proxy settings or real tokens leak in.
8+
9+
Needs python3 (3.10 or later), bash and yq. Either yq works: mikefarah's (as on GitHub's ubuntu-latest runners) or
10+
the Python jq wrapper.
11+
12+
StubServer is a GitHub API stand-in on 127.0.0.1, answering each path with the status and body a test gives it.
13+
"""
14+
15+
import functools
16+
import http.server
17+
import os
18+
import pathlib
19+
import socket
20+
import subprocess
21+
import sys
22+
import tempfile
23+
import threading
24+
25+
WORKFLOW = pathlib.Path(__file__).resolve().parent.parent / ".github" / "workflows" / "claude-review.yml"
26+
27+
# Set by the runner for every step; a test passes the step's own inputs, which must be declared in the step's env.
28+
# The shell a Python step names, so the runner runs its script with python3.
29+
PYTHON_SHELL = "python3 {0}"
30+
31+
RUNNER_VARIABLES = {"GITHUB_OUTPUT", "GITHUB_STEP_SUMMARY", "GITHUB_API_URL", "RUNNER_TEMP", "PATH", "LC_ALL"}
32+
33+
34+
def _yq(query):
35+
result = subprocess.run(["yq", "-r", query, str(WORKFLOW)], capture_output=True, text=True, check=True)
36+
return result.stdout
37+
38+
39+
def _step_query(job, step_id):
40+
return f'.jobs.{job}.steps[] | select(.id == "{step_id}")'
41+
42+
43+
class StepResult:
44+
"""What a step run left behind: its exit code, its standard output and error, its outputs and its summary."""
45+
46+
def __init__(self, returncode, stdout, stderr, output_file, summary):
47+
self.returncode = returncode
48+
self.stdout = stdout
49+
self.stderr = stderr
50+
self.output_file = output_file
51+
self.summary = summary
52+
self.outputs = parse_outputs(output_file)
53+
54+
def errors(self):
55+
"""The messages of the ::error:: annotations the step printed."""
56+
return [line[len("::error::"):] for line in self.stdout.splitlines() if line.startswith("::error::")]
57+
58+
def everything(self):
59+
"""All the text the step produced, for checking that a secret appears nowhere."""
60+
return "\n".join([self.stdout, self.stderr, self.output_file, self.summary])
61+
62+
63+
def parse_outputs(text):
64+
"""Reads a GITHUB_OUTPUT file as the runner does. A name written twice keeps its last value."""
65+
outputs = {}
66+
lines = text.split("\n")
67+
i = 0
68+
while i < len(lines):
69+
line = lines[i]
70+
i += 1
71+
if line == "":
72+
continue
73+
if "<<" in line and ("=" not in line or line.index("<<") < line.index("=")):
74+
name, delimiter = line.split("<<", 1)
75+
value = []
76+
while i < len(lines) and lines[i] != delimiter:
77+
value.append(lines[i])
78+
i += 1
79+
if i == len(lines):
80+
raise ValueError(f"output {name!r} has no closing delimiter")
81+
i += 1
82+
outputs[name] = "\n".join(value)
83+
else:
84+
name, value = line.split("=", 1)
85+
outputs[name] = value
86+
return outputs
87+
88+
89+
@functools.lru_cache(maxsize=None)
90+
def step_script(job, step_id):
91+
"""The step's script and shell, read out of the workflow."""
92+
script = _yq(_step_query(job, step_id) + " | .run")
93+
if script.strip() in ("", "null"):
94+
raise LookupError(f"no step with id {step_id!r} and a run script in job {job!r} of {WORKFLOW}")
95+
shell = _yq(_step_query(job, step_id) + ' | .shell // ""').strip()
96+
return script, shell
97+
98+
99+
def python_steps():
100+
"""(job, step id) for every step of the workflow that runs with python3."""
101+
steps = []
102+
for job in _yq(".jobs | keys | .[]").split():
103+
for step_id in _yq(f'.jobs.{job}.steps[] | select(.shell == "{PYTHON_SHELL}") | .id').split("\n"):
104+
if step_id:
105+
steps.append((job, step_id))
106+
return steps
107+
108+
109+
@functools.lru_cache(maxsize=None)
110+
def step_env_names(job, step_id):
111+
"""The names in the step's env block."""
112+
return frozenset(_yq(_step_query(job, step_id) + " | (.env // {}) | keys | .[]").split())
113+
114+
115+
def run_step(job, step_id, env, api_url="http://127.0.0.1:9", substitutions=None, timeout=20):
116+
"""Runs the step with only the given env (plus the runner's own variables) and returns a StepResult.
117+
118+
substitutions maps text in the script to its replacement, such as a timeout constant lowered for a test. Each
119+
piece of text must appear exactly once, so a test fails loudly if the script changes under it.
120+
"""
121+
script, shell = step_script(job, step_id)
122+
# Untrusted text must reach a script through env, never through an expression placed in the script.
123+
if "${{" in script:
124+
raise AssertionError(f"step {step_id!r} has a workflow expression inside its script")
125+
undeclared = set(env) - step_env_names(job, step_id)
126+
if undeclared:
127+
raise AssertionError(f"step {step_id!r} does not declare env {sorted(undeclared)}")
128+
for old, new in (substitutions or {}).items():
129+
count = script.count(old)
130+
if count != 1:
131+
raise AssertionError(f"expected {old!r} once in step {step_id!r}, found it {count} times")
132+
script = script.replace(old, new)
133+
134+
with tempfile.TemporaryDirectory() as temp:
135+
temp = pathlib.Path(temp)
136+
output_path = temp / "output"
137+
summary_path = temp / "summary"
138+
output_path.touch()
139+
summary_path.touch()
140+
script_path = temp / "step"
141+
script_path.write_text(script, encoding="utf-8")
142+
if shell == PYTHON_SHELL:
143+
command = [sys.executable, str(script_path)]
144+
elif shell in ("", "bash"):
145+
# GitHub's default for a run step on Linux.
146+
command = ["bash", "--noprofile", "--norc", "-eo", "pipefail", str(script_path)]
147+
else:
148+
raise AssertionError(f"step {step_id!r} uses shell {shell!r}, which the harness cannot run")
149+
runner_temp = temp / "runner"
150+
runner_temp.mkdir()
151+
full_env = {
152+
"PATH": os.environ.get("PATH", "/usr/bin:/bin"),
153+
"LC_ALL": "C.UTF-8",
154+
"GITHUB_OUTPUT": str(output_path),
155+
"GITHUB_STEP_SUMMARY": str(summary_path),
156+
"GITHUB_API_URL": api_url,
157+
"RUNNER_TEMP": str(runner_temp),
158+
**env,
159+
}
160+
completed = subprocess.run(command, env=full_env, capture_output=True, text=True, timeout=timeout)
161+
return StepResult(
162+
completed.returncode,
163+
completed.stdout,
164+
completed.stderr,
165+
output_path.read_text(encoding="utf-8"),
166+
summary_path.read_text(encoding="utf-8"),
167+
)
168+
169+
170+
def closed_port_url():
171+
"""A URL on 127.0.0.1 that nothing listens on, for a connection that is refused."""
172+
with socket.socket() as sock:
173+
sock.bind(("127.0.0.1", 0))
174+
port = sock.getsockname()[1]
175+
return f"http://127.0.0.1:{port}"
176+
177+
178+
class StubServer:
179+
"""A stand-in for the GitHub API on 127.0.0.1.
180+
181+
routes maps a path to (status, body), to HANG for an answer that never comes, or to DROP for a connection closed
182+
without an answer, for GET and POST alike. A path with no route answers 404. Every request is recorded in requests
183+
as (method, path, headers), and its body, as bytes (empty for a GET), at the same place in bodies. Headers are
184+
looked up by name ignoring case, as HTTP does. Use as a context manager.
185+
"""
186+
187+
HANG = object()
188+
DROP = object()
189+
190+
def __init__(self, routes=None):
191+
self.routes = dict(routes or {})
192+
self.requests = []
193+
self.bodies = []
194+
self._release = threading.Event()
195+
stub = self
196+
197+
class Handler(http.server.BaseHTTPRequestHandler):
198+
def do_GET(self):
199+
self.answer("GET")
200+
201+
def do_POST(self):
202+
self.answer("POST")
203+
204+
def answer(self, method):
205+
length = int(self.headers.get("Content-Length") or 0)
206+
body = self.rfile.read(length) if length else b""
207+
# Headers stay a message, so a lookup by name ignores case as HTTP does.
208+
stub.requests.append((method, self.path, self.headers))
209+
stub.bodies.append(body)
210+
route = stub.routes.get(self.path, (404, '{"message": "Not Found"}'))
211+
if route is StubServer.HANG:
212+
stub._release.wait(10)
213+
return
214+
if route is StubServer.DROP:
215+
# Closes the connection without an answer.
216+
self.close_connection = True
217+
return
218+
status, body = route
219+
data = body.encode("utf-8")
220+
self.send_response(status)
221+
self.send_header("Content-Type", "application/json")
222+
self.send_header("Content-Length", str(len(data)))
223+
if 300 <= status < 400:
224+
self.send_header("Location", "http://127.0.0.1:9/elsewhere")
225+
self.end_headers()
226+
self.wfile.write(data)
227+
228+
def log_message(self, *args):
229+
pass
230+
231+
self._server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
232+
self._server.daemon_threads = True
233+
self.url = f"http://127.0.0.1:{self._server.server_address[1]}"
234+
self._thread = threading.Thread(target=self._server.serve_forever, kwargs={"poll_interval": 0.05}, daemon=True)
235+
236+
def __enter__(self):
237+
self._thread.start()
238+
return self
239+
240+
def __exit__(self, *exc):
241+
self._release.set()
242+
self._server.shutdown()
243+
self._server.server_close()
244+
return False
245+
246+
def paths(self):
247+
return [path for _, path, _ in self.requests]

0 commit comments

Comments
 (0)