Skip to content

Commit c756fd0

Browse files
authored
Merge pull request #10 from cipherstash/ci/checks
ci: lint workflows and test the request decision on every pull request
2 parents b31466f + a1d6c69 commit c756fd0

8 files changed

Lines changed: 208 additions & 1 deletion

File tree

‎.github/workflows/checks.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Checks every pull request to this repository, and every push to main: lints the workflows, checks the request
2+
# decision's test script, and runs the request decision's saved cases.
3+
name: Checks
4+
5+
on:
6+
pull_request:
7+
push:
8+
branches: [main]
9+
10+
permissions:
11+
contents: read
12+
13+
jobs:
14+
actionlint:
15+
runs-on: ubuntu-latest
16+
timeout-minutes: 10
17+
steps:
18+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
19+
with:
20+
persist-credentials: false
21+
# Go's module proxy and checksum database verify the pinned version. shellcheck, which actionlint uses for
22+
# `run:` scripts, is preinstalled on ubuntu-latest.
23+
- name: actionlint
24+
run: go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 -color
25+
26+
request-decision:
27+
runs-on: ubuntu-latest
28+
timeout-minutes: 5
29+
steps:
30+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
31+
with:
32+
persist-credentials: false
33+
- name: Check the test script
34+
run: request-decision/test-runner
35+
- name: Run the saved cases
36+
run: request-decision/test

‎README.md‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,30 @@
33
Reusable GitHub Actions workflows shared by CipherStash repositories.
44

55
The first one asks Claude to review a pull request.
6-
It is being built; the plan is in [issue #1](https://github.com/cipherstash/workflows/issues/1).
6+
It is being built.
7+
The plan is in [issue #1](https://github.com/cipherstash/workflows/issues/1).
78

89
The words this repository uses, such as *review request* and *team member*, are defined in [CONTEXT.md](CONTEXT.md).
10+
11+
## Checks
12+
13+
Every pull request, and every push to `main`, runs these checks:
14+
15+
- `actionlint` over the workflows in this repository.
16+
- The request decision's saved cases, run by `request-decision/test`.
17+
It fails if a case is missing a file, if a file is not valid JSON, or if there are no cases at all.
18+
- A check of `request-decision/test` itself, run by `request-decision/test-runner`.
19+
20+
The request decision is the part that reads a GitHub event and decides whether it is a review request to send on.
21+
Its rules are tested against saved event payloads, so the tests need no network and run anywhere with `bash` and `jq`:
22+
23+
```sh
24+
request-decision/test
25+
```
26+
27+
### Adding a test case
28+
29+
1. Make a new directory under `request-decision/cases/`, named for what it checks, such as `draft-pull-request-skips`.
30+
2. Add `input.json`: an object with `event_name` (the GitHub event name, such as `pull_request`) and `event` (the event payload, trimmed to the fields that matter).
31+
3. Add `expected.json`: the decision expected for that input, such as `{"decision": "skip", "reason": "draft"}`.
32+
4. Run `request-decision/test` and check the new case passes. A case that fails prints the difference between the expected and actual decision.
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"decision": "skip", "reason": "not-a-review-request"}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"event_name": "issues",
3+
"event": {
4+
"action": "opened",
5+
"issue": {"number": 1, "title": "Not a pull request"},
6+
"sender": {"login": "octocat", "type": "User"}
7+
}
8+
}

‎request-decision/decide‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/usr/bin/env bash
2+
# Runs the request decision on the JSON object read from standard input and prints the decision as one JSON object.
3+
set -euo pipefail
4+
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
exec jq --compact-output --from-file "$here/decide.jq"

‎request-decision/decide.jq‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# The request decision: reads one review request candidate and decides whether to fire the routine trigger.
2+
#
3+
# Input (one JSON object on standard input):
4+
# event_name the GitHub event name, for example "pull_request" or "issue_comment"
5+
# event the event payload GitHub delivers
6+
#
7+
# Output (one JSON object):
8+
# {"decision": "skip", "reason": "<why>"}
9+
#
10+
# Only the skip for events that are not a kind of review request exists so far; the kinds of review request are
11+
# added by later tickets (cipherstash/workflows#5, #6 and #7).
12+
13+
{decision: "skip", reason: "not-a-review-request"}

‎request-decision/test‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
#!/usr/bin/env bash
2+
# Runs every saved case through the request decision and compares the result with the expected decision.
3+
#
4+
# Each case is a directory under request-decision/cases/ holding:
5+
# input.json the decision's input: {"event_name": ..., "event": <payload>}
6+
# expected.json the decision expected for that input
7+
#
8+
# Exits non-zero if any case fails or there are no cases. Needs only bash and jq.
9+
set -euo pipefail
10+
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11+
cases_dir="$here/cases"
12+
13+
total=0
14+
failed=0
15+
shopt -s nullglob
16+
for case_dir in "$cases_dir"/*/; do
17+
name="$(basename "$case_dir")"
18+
total=$((total + 1))
19+
if ! actual="$("$here/decide" <"$case_dir/input.json" | jq --sort-keys .)"; then
20+
echo "FAIL $name: the decision exited with an error"
21+
failed=$((failed + 1))
22+
continue
23+
fi
24+
if ! expected="$(jq --sort-keys . "$case_dir/expected.json")"; then
25+
echo "FAIL $name: expected.json is missing or is not valid JSON"
26+
failed=$((failed + 1))
27+
continue
28+
fi
29+
if [[ "$actual" == "$expected" ]]; then
30+
echo "ok $name"
31+
else
32+
echo "FAIL $name"
33+
diff <(echo "$expected") <(echo "$actual") | sed 's/^/ /' || true
34+
failed=$((failed + 1))
35+
fi
36+
done
37+
38+
if [[ "$total" -eq 0 ]]; then
39+
echo "FAIL no cases found in $cases_dir"
40+
exit 1
41+
fi
42+
43+
echo "$total cases, $failed failed"
44+
[[ "$failed" -eq 0 ]]

‎request-decision/test-runner‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
#!/usr/bin/env bash
2+
# Checks that request-decision/test reports broken or missing cases as failures instead of passing or stopping early.
3+
#
4+
# Each check copies the test script into a temporary directory beside a stand-in decision that echoes its input, sets
5+
# up the cases for that check, runs the test script, and compares its exit status and output with what is expected.
6+
# The stand-in keeps these checks about the test script alone, whatever the real decision does.
7+
#
8+
# Exits non-zero if any check fails. Needs only bash and jq.
9+
set -euo pipefail
10+
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
11+
work="$(mktemp -d)"
12+
trap 'rm -rf -- "$work"' EXIT
13+
14+
failed=0
15+
16+
# Makes a fresh copy of the test script, with the stand-in decision and an empty cases directory, and prints its path.
17+
new_copy() {
18+
local dir
19+
dir="$(mktemp -d "$work/copy.XXXXXX")"
20+
cp "$here/test" "$dir/"
21+
printf '#!/usr/bin/env bash\nexec jq --compact-output .\n' >"$dir/decide"
22+
chmod +x "$dir/decide"
23+
mkdir "$dir/cases"
24+
echo "$dir"
25+
}
26+
27+
# Adds a case that passes to a copy.
28+
add_good_case() {
29+
mkdir "$1/cases/good"
30+
echo '{"event_name": "push", "event": {}}' >"$1/cases/good/input.json"
31+
"$1/decide" <"$1/cases/good/input.json" >"$1/cases/good/expected.json"
32+
}
33+
34+
# check NAME COPY WANT_STATUS WANT_TEXT: runs the copy's test script and checks it exits with WANT_STATUS
35+
# ("zero" or "non-zero") and prints a line containing WANT_TEXT.
36+
check() {
37+
local name="$1" copy="$2" want_status="$3" want_text="$4" output status=0
38+
output="$("$copy/test" 2>&1)" || status=$?
39+
local got_status=zero
40+
[[ "$status" -eq 0 ]] || got_status=non-zero
41+
if [[ "$got_status" == "$want_status" ]] && grep --quiet --fixed-strings -- "$want_text" <<<"$output"; then
42+
echo "ok $name"
43+
else
44+
echo "FAIL $name: wanted a $want_status exit and the text \"$want_text\", got a $got_status exit and:"
45+
sed 's/^/ /' <<<"$output"
46+
failed=$((failed + 1))
47+
fi
48+
}
49+
50+
copy="$(new_copy)"
51+
add_good_case "$copy"
52+
check "a passing case passes" "$copy" zero "1 cases, 0 failed"
53+
54+
copy="$(new_copy)"
55+
check "no cases fails" "$copy" non-zero "no cases found"
56+
57+
copy="$(new_copy)"
58+
add_good_case "$copy"
59+
mkdir "$copy/cases/missing-expected"
60+
cp "$copy/cases/good/input.json" "$copy/cases/missing-expected/"
61+
check "a missing expected.json is a failed case" "$copy" non-zero "2 cases, 1 failed"
62+
63+
copy="$(new_copy)"
64+
add_good_case "$copy"
65+
mkdir "$copy/cases/invalid-expected"
66+
cp "$copy/cases/good/input.json" "$copy/cases/invalid-expected/"
67+
echo '{not json' >"$copy/cases/invalid-expected/expected.json"
68+
check "an invalid expected.json is a failed case" "$copy" non-zero "2 cases, 1 failed"
69+
70+
copy="$(new_copy)"
71+
add_good_case "$copy"
72+
mkdir "$copy/cases/missing-input"
73+
cp "$copy/cases/good/expected.json" "$copy/cases/missing-input/"
74+
check "a missing input.json is a failed case" "$copy" non-zero "2 cases, 1 failed"
75+
76+
[[ "$failed" -eq 0 ]]

0 commit comments

Comments
 (0)