Repository navigation
A save leaves out the caches a tool fetches again #3249
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # fragment CI on Linux: `cargo xtask check` (every first-party JavaScript | |
| # file through `node --check` on the pinned Node, host tests and clippy on | |
| # host and wasm, warnings denied), and `cargo xtask e2e`: the cell under | |
| # `wrangler dev` (workerd), with fakes only at the vendor boundaries. | |
| # The celld e2e on macOS is at tag celld-final. | |
| # | |
| # The e2e runs in five shards (crates/e2e/src/lanes/mod.rs, SHARDS: every | |
| # section in exactly one, a host test checks), each on a runner of its own | |
| # with its own build and node; the `e2e` job is green when every shard is. | |
| # | |
| # A pull request's run is `check` alone: its e2e runs in the merge queue, | |
| # on the very tree that merges, which master's ruleset requires `check` and | |
| # `e2e` of before anything lands (Paul, 2026-10-31). Its `e2e` check passes | |
| # through, saying so. The label `full-e2e` on a pull request runs the | |
| # shards on its runs too (adding it starts one), for a change that deserves | |
| # them before it queues; a run by hand (workflow_dispatch) runs them on a | |
| # branch's head. | |
| # | |
| # Caches are restored on every run and saved only from master's pushes (a | |
| # PR reads master's, and so does production's). `check`'s Rust cache is | |
| # Swatinem/rust-cache's, keyed by the job, rustc, and every Cargo.toml and | |
| # Cargo.lock, and saved at the job's end with only the dependencies' | |
| # builds: a checkout's fresh mtimes rebuild the workspace's own crates | |
| # anyway. It is given the target directories by name (`cache-targets: | |
| # false`), not `target` whole: its cleaning would leave target/tools an | |
| # empty tree, which xtask would take for the pinned Node. The e2e's builds, | |
| # our own crates' with them, are actions/cache's, one entry per master | |
| # commit, with the tree they were built from, so cargo rebuilds only what | |
| # a change touched (.github/actions/e2e-build, saved by the first shard | |
| # or, when a push's shards skip, by `e2e-build`; `prune` keeps the newest | |
| # alone); | |
| # their tools are rust-cache's, and the stub image's layers BuildKit's, | |
| # in the same Actions cache. The pinned Node and the JavaScript caches are | |
| # actions/cache's, each key naming what it holds below, saved right after | |
| # the step that fills it. | |
| # | |
| # A green push deploys (`deploy`, below): master's to the preview, | |
| # production's to production. A push of a tree a green run already | |
| # tested (master's, for production; for master, its merge queue's or its | |
| # pull request's, when master had not moved since) skips the shards | |
| # (`proven`, below), and so does a merge group into master whose tree a | |
| # green run already tested. | |
| name: ci | |
| on: | |
| push: | |
| branches: [master, production] | |
| pull_request: | |
| # the default three, and `labeled`: adding `full-e2e` runs the shards | |
| types: [opened, synchronize, reopened, labeled] | |
| merge_group: | |
| types: [checks_requested] | |
| workflow_dispatch: | |
| # A pull request's run of a new head (`synchronize`) cancels its older | |
| # head's; a push's run is never cancelled, as it may be deploying: a newer | |
| # push's waits for it (and replaces one already waiting). Nothing cancels | |
| # a run of the same head: GitHub counts every run's required checks on a | |
| # commit, a cancelled run's as failed beside the other's passing, so its | |
| # pull request stayed BLOCKED, green and approved, until a new head (#942, | |
| # #944, #956: each opened with `full-e2e`, its `opened` and `labeled` runs | |
| # together, the first cancelled). So a label added waits for the head's | |
| # run under way, and a run that would test nothing new is idle (`check` | |
| # and the shards skip, `e2e` says why), in a group of its own, never | |
| # cancelled: one opened with `full-e2e` on it already, which the `labeled` | |
| # run that comes with it tests, and one for another label. The idle | |
| # expression is the same in `check`, `e2e-shard` and `e2e`. | |
| concurrency: | |
| group: ci-${{ github.ref }}${{ ((github.event.action == 'opened' && contains(github.event.pull_request.labels.*.name, 'full-e2e')) || (github.event.action == 'labeled' && github.event.label.name != 'full-e2e')) && format('-idle-{0}', github.run_id) || '' }} | |
| cancel-in-progress: ${{ github.event.action == 'synchronize' }} | |
| jobs: | |
| # A push to master or production, or a merge group into master: whether | |
| # a run of this workflow (by its id, not its name) already proved what it | |
| # holds. A job can only need | |
| # jobs of its own run, so without this a promotion tested again, for 20 | |
| # minutes, what master's run had (Paul, 2026-10-20), and a merge what its | |
| # pull request's run had (Paul, 2026-10-23). What a run proves is a tree: | |
| # its success is its check's and every shard's, on the tree each checked | |
| # out, which `e2e` records (an artifact named `tested-tree-<tree>`). So a | |
| # commit is proven by a run that succeeded and is one of these: | |
| # | |
| # - on a push to master of a commit whose tree is this one's; for | |
| # production, master's runs of this very SHA too, recorded or not (as | |
| # before trees were). Master's runs test and then deploy the preview, | |
| # or deploy a tree proven so: such a run's success is its deploy's too. | |
| # - on master only, the run of the pull request this merge commit merges, | |
| # whose head is this commit's second parent, that recorded this tree. A | |
| # pull request's run tests GitHub's merge of its head into master as | |
| # master stood then: if master has not moved since, the merge commit's | |
| # tree is that one; if it has, they differ, and the tests run. The head | |
| # binds the run to this merge: another branch's run (a fork's included) | |
| # proves nothing here, whatever tree it names, and this one's workflow | |
| # is the one this merge brings to master. A squash or a rebase merge | |
| # has no such parent. | |
| # - on master only, a merge_group run from this repository's master | |
| # queue (`gh-readonly-queue/master/…`) that recorded this tree, whose | |
| # head really has this tree too. The queue tests the combined changes | |
| # against the base they will land on; moving master rebuilds the | |
| # group. Neither another base's queue nor a fork can prove it. | |
| # | |
| # A merge group into master is held to master's rules: its head is the | |
| # merge commit master will move to (the group's last pull request's head | |
| # its second parent), so it is proven exactly when that push would be. | |
| # Same tree, same files: the proving run ran this very workflow and these | |
| # very tests. | |
| # | |
| # Proven, the e2e's shards skip, `e2e` is green on the proof, and `deploy` | |
| # runs once `check` is green; otherwise (a hotfix pushed straight to | |
| # production, master moved since the pull request's run, a run red or | |
| # still running) they run as on any push. Should this job fail, no shard | |
| # runs, `e2e` is red, and nothing merges or deploys. | |
| # | |
| # `check` runs on every push, proven or not: besides the tree's own | |
| # checks it holds the tree's schema lock to the one at what the push | |
| # moved from (`github.event.before`; xtask/src/schema_lock.rs), which no | |
| # tree can prove ahead of the push. A tree proven before a migration | |
| # step shipped, and pushed again after it (a revert), drops that step, | |
| # and every object that took it refuses code without it (docs/MODEL.md, | |
| # Schemas): its `check` fails, and nothing deploys. On production too: | |
| # its last deploy can hold a step its new commit lost (a hotfix's, lost | |
| # in a merge back), which no run of master's checked against. `check` | |
| # takes about two minutes, beside `proven`; the shards are the 15 to 20. | |
| # A merge group's `check` holds the lock to the group's exact base. | |
| proven: | |
| if: >- | |
| ${{ (github.event_name == 'push' && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/production')) | |
| || (github.event_name == 'merge_group' && github.event.merge_group.base_ref == 'refs/heads/master') }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| permissions: | |
| actions: read # this workflow's runs and their artifacts, through the API | |
| contents: read # the commits' trees and parents | |
| outputs: | |
| proven: ${{ steps.ask.outputs.proven }} | |
| steps: | |
| # the runs the API's filters find, each fetched and held to the rules | |
| # above again; every one found is printed, with its state | |
| - id: ask | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| # the branch this lands on: a push's own, a merge group's base | |
| ONTO: ${{ github.event_name == 'merge_group' && github.event.merge_group.base_ref || github.ref }} | |
| run: | | |
| set -euo pipefail | |
| case "$ONTO" in | |
| refs/heads/master | refs/heads/production) ;; | |
| *) echo "::error::$GITHUB_EVENT_NAME lands on $ONTO, which no rule here proves"; exit 1 ;; | |
| esac | |
| repo=repos/$GITHUB_REPOSITORY | |
| workflow=$(gh api "$repo/actions/runs/$GITHUB_RUN_ID" --jq .workflow_id) | |
| commit=$(gh api "$repo/git/commits/$GITHUB_SHA") | |
| tree=$(jq -r .tree.sha <<< "$commit") | |
| merged=$(jq -r 'if (.parents | length) == 2 then .parents[1].sha else "" end' <<< "$commit") | |
| echo "$GITHUB_SHA's tree is $tree${merged:+; it merges $merged}" | |
| ids=$(gh api -X GET "$repo/actions/artifacts" -f name="tested-tree-$tree" -f per_page=100 --jq '.artifacts[].workflow_run.id') | |
| if [ "$ONTO" = refs/heads/production ]; then | |
| ids+=$'\n'$(gh api -X GET "$repo/actions/workflows/$workflow/runs" -f head_sha="$GITHUB_SHA" -f branch=master -f event=push -f per_page=100 --jq '.workflow_runs[].id') | |
| fi | |
| for id in $(sort -u <<< "$ids"); do | |
| run=$(gh api "$repo/actions/runs/$id") | |
| jq -r '"\(.html_url), attempt \(.run_attempt): \(.event) of \(.head_sha) (\(.head_repository.full_name) \(.head_branch)), \(.status) \(.conclusion // "")"' <<< "$run" | |
| jq -e --argjson w "$workflow" '.workflow_id == $w and .status == "completed" and .conclusion == "success"' <<< "$run" > /dev/null || continue | |
| head=$(jq -r .head_sha <<< "$run") | |
| if jq -e --arg r "$GITHUB_REPOSITORY" '.event == "push" and .head_branch == "master" and .head_repository.full_name == $r' <<< "$run" > /dev/null; then | |
| [ "$head" = "$GITHUB_SHA" ] || [ "$(gh api "$repo/git/commits/$head" --jq .tree.sha)" = "$tree" ] || continue | |
| why="master's run of $head proved this tree" | |
| elif [ "$ONTO" = refs/heads/master ] && [ -n "$merged" ] && jq -e --arg h "$merged" '.event == "pull_request" and .head_sha == $h' <<< "$run" > /dev/null; then | |
| why="the run of the pull request this merges, at its head $head, proved this tree" | |
| elif [ "$ONTO" = refs/heads/master ] && jq -e --arg r "$GITHUB_REPOSITORY" '.event == "merge_group" and .head_repository.full_name == $r and (.head_branch | startswith("gh-readonly-queue/master/"))' <<< "$run" > /dev/null; then | |
| [ "$(gh api "$repo/git/commits/$head" --jq .tree.sha)" = "$tree" ] || continue | |
| why="master's merge queue run of $head proved this tree" | |
| else | |
| continue | |
| fi | |
| echo "$why: $GITHUB_SHA's e2e need not run again" | |
| echo proven=true >> "$GITHUB_OUTPUT" | |
| exit 0 | |
| done | |
| echo "no run of this workflow proved $GITHUB_SHA's tree: testing it first" | |
| echo proven=false >> "$GITHUB_OUTPUT" | |
| # on every run, a proven push's too (`proven`, above, says why), but an | |
| # idle pull request's (`concurrency`, above) | |
| check: | |
| if: ${{ !((github.event.action == 'opened' && contains(github.event.pull_request.labels.*.name, 'full-e2e')) || (github.event.action == 'labeled' && github.event.label.name != 'full-e2e')) }} | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: dtolnay/rust-toolchain@stable | |
| with: | |
| targets: wasm32-unknown-unknown | |
| components: clippy | |
| # the debug, test and clippy builds of the two workspaces | |
| - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 | |
| with: | |
| workspaces: | | |
| . -> target | |
| cell -> target | |
| cache-targets: false | |
| cache-directories: | | |
| target/debug | |
| cell/target | |
| cache-bin: false | |
| save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }} | |
| # the pinned Node `node --check` runs on, unpacked: the e2e's key, | |
| # which its first shard saves from master | |
| - uses: actions/cache/restore@v4 | |
| with: | |
| path: target/tools | |
| key: node-${{ runner.os }}-${{ hashFiles('crates/devstack/src/node_release.rs') }} | |
| # the schema lock master holds, which this tree's keeps step for | |
| # step (xtask/src/schema_lock.rs): a pull request's base, or the | |
| # master a push moved on from, or a merge group's exact base (not | |
| # master, which can move while the group is being tested) | |
| - run: git fetch --no-tags --depth=1 origin "$BASE" && echo "FRAGMENT_SCHEMA_BASE=FETCH_HEAD" >> "$GITHUB_ENV" | |
| env: | |
| BASE: ${{ github.event_name == 'push' && github.event.before || github.event.merge_group.base_sha || github.base_ref || 'master' }} | |
| - run: cargo xtask check | |
| # Also exercise image-input resolution on PRs/queue trees, without | |
| # secrets or registry writes. The real image job is master-only. | |
| - run: cargo xtask image preview --dry-run --output "$RUNNER_TEMP/preview-image-plan.json" | |
| e2e-shard: | |
| name: e2e shard ${{ matrix.shard }}/5 | |
| # on every run but a pull request's without the `full-e2e` label (its | |
| # e2e is the merge queue's) or idle (`concurrency`, above), and a push | |
| # or merge group whose tree a run proved (`proven`, which skips on every | |
| # other run: `!cancelled()` keeps its skip from skipping this), and | |
| # never after `proven` fails | |
| needs: proven | |
| if: >- | |
| ${{ !cancelled() | |
| && (github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'full-e2e')) | |
| && !((github.event.action == 'opened' && contains(github.event.pull_request.labels.*.name, 'full-e2e')) || (github.event.action == 'labeled' && github.event.label.name != 'full-e2e')) | |
| && (needs.proven.result == 'skipped' || (needs.proven.result == 'success' && needs.proven.outputs.proven == 'false')) }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 20 | |
| env: | |
| # xtask/src/build.rs, WORKER_BUILD_VERSION | |
| WORKER_BUILD_VERSION: 0.8.5 | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| shard: [1, 2, 3, 4, 5] | |
| steps: | |
| # first, before any container: no IPv6 address on a link made from | |
| # here on, nor on Docker's bridge. Chrome takes a link's IPv6 address | |
| # coming or going for a network change, and fails every request still | |
| # waiting for a connection (ERR_NETWORK_CHANGED); Docker adds a link | |
| # for each container a computer starts. So a page's module loads queued | |
| # behind its six connections failed, and the page never started | |
| # ("Still loading…", billing-page's `__held`): the module stalls. | |
| - run: | | |
| sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 | |
| if [ -e /proc/sys/net/ipv6/conf/docker0 ]; then sudo sysctl -w net.ipv6.conf.docker0.disable_ipv6=1; fi | |
| - uses: actions/checkout@v4 | |
| # the caches, the stub image and the build (.github/actions/e2e-build); | |
| # the first shard saves them on master's pushes | |
| - id: build | |
| uses: ./.github/actions/e2e-build | |
| with: | |
| save: ${{ matrix.shard == 1 && github.event_name == 'push' && github.ref == 'refs/heads/master' }} | |
| worker-build-version: ${{ env.WORKER_BUILD_VERSION }} | |
| # one shard's sections on a fresh node; the run's scratch (each node | |
| # boot's log, the lanes' screenshots) is kept when a check fails, and | |
| # uploaded below | |
| - run: cargo xtask e2e --no-build --shard ${{ matrix.shard }}/5 | |
| env: | |
| BUILDX_BUILDER: ${{ steps.build.outputs.builder }} | |
| CHROME_BIN: /usr/bin/google-chrome | |
| # after the run, from any shard that shot a preview card: miniflare | |
| # fetches its Chrome for the first one during the run (the first | |
| # shard to save wins; the others find the key taken) | |
| - if: always() && github.event_name == 'push' && github.ref == 'refs/heads/master' && steps.build.outputs.js-cache-hit != 'true' && hashFiles('target/cache/.wrangler/chrome/**/.miniflare-install-complete') != '' | |
| uses: actions/cache/save@v4 | |
| with: | |
| path: target/cache | |
| key: ${{ steps.build.outputs.js-cache-key }} | |
| - if: failure() | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: e2e-scratch-${{ matrix.shard }} | |
| path: target/e2e/ | |
| retention-days: 7 | |
| # The check named `e2e`, which pull requests and merge groups require: | |
| # green when every shard is (a re-run's, by each shard's latest attempt), | |
| # or when `proven` found a run that proved this tree, so none ran, or on | |
| # a pull request without `full-e2e`, whose e2e the merge queue runs (a | |
| # merge group is never one: its `e2e` is the real one). It runs whatever | |
| # came before it, unless the run was cancelled: a required check that | |
| # skips reads as green, so a `proven` that failed, or shards that were to | |
| # run and skipped or failed, make it red, never skipped. | |
| e2e: | |
| needs: [proven, e2e-shard] | |
| if: ${{ !cancelled() }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| permissions: | |
| contents: read # the tested commit's tree, through the API | |
| steps: | |
| - env: | |
| PROVEN: ${{ needs.proven.result == 'success' && needs.proven.outputs.proven == 'true' }} | |
| PROVEN_RESULT: ${{ needs.proven.result }} | |
| SHARDS: ${{ needs.e2e-shard.result }} | |
| QUEUED: ${{ github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'full-e2e') }} | |
| IDLE: ${{ (github.event.action == 'opened' && contains(github.event.pull_request.labels.*.name, 'full-e2e')) || (github.event.action == 'labeled' && github.event.label.name != 'full-e2e') }} | |
| run: | | |
| if [ "$IDLE" = true ]; then echo "this run tests nothing new (\`concurrency\` says why): another run of this head tests it"; exit 0; fi | |
| if [ "$PROVEN" = true ]; then echo "a run that proved this tree ran its e2e (\`proven\`'s log names it): none ran here"; exit 0; fi | |
| if [ "$QUEUED" = true ]; then echo "a pull request's e2e runs in the merge queue, on the tree that merges, before it lands; the label full-e2e runs it here too"; exit 0; fi | |
| test "$SHARDS" = success || { echo "the shards did not all pass (shards: $SHARDS; proven: $PROVEN_RESULT): their logs, or proven's, say why"; exit 1; } | |
| # The tree every shard tested, for `proven`: GITHUB_SHA's (a pull | |
| # request's merge commit, a merge group's head, a push's head), | |
| # which each checked out, | |
| # recorded once they are all green as an artifact named by it (the | |
| # commit and the tree in it). `proven` counts it only for a run that | |
| # succeeded, `check` and every shard of its latest attempt included. | |
| # A re-run's replaces its own. Recording is a saving, not a test: a | |
| # step of it that fails warns and leaves the run as green as its | |
| # shards, unrecorded (a push of its tree is then tested again). | |
| # Only shards that ran here record: a proven run's proof is another's. | |
| - id: tree | |
| if: needs.e2e-shard.result == 'success' | |
| continue-on-error: true | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| set -euo pipefail | |
| tree=$(gh api "repos/$GITHUB_REPOSITORY/git/commits/$GITHUB_SHA" --jq .tree.sha) | |
| [[ $tree =~ ^[0-9a-f]{40}([0-9a-f]{24})?$ ]] || { echo "::warning::the API named no tree for $GITHUB_SHA: '$tree'"; exit 1; } | |
| echo "this run tested $GITHUB_SHA, whose tree is $tree" | |
| echo "$GITHUB_SHA $tree" > "$RUNNER_TEMP/tested-tree" | |
| echo "tree=$tree" >> "$GITHUB_OUTPUT" | |
| - if: steps.tree.outcome == 'success' | |
| continue-on-error: true | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: tested-tree-${{ steps.tree.outputs.tree }} | |
| path: ${{ runner.temp }}/tested-tree | |
| if-no-files-found: error | |
| overwrite: true | |
| retention-days: 30 | |
| # Master follows one image, keyed by its inputs rather than its commit. | |
| # A registry tag preserves the digest across cache eviction and re-runs; | |
| # a changed cell or shell never builds a new image. Prepare it before | |
| # deploy can start. This job never changes the running Worker. The merge | |
| # queue builds and prepares a group's image ahead (preview-image.yml), | |
| # which this job copies to its input tag; it builds only when neither | |
| # is there. | |
| preview-image: | |
| if: github.event_name == 'push' && github.ref == 'refs/heads/master' | |
| runs-on: ubuntu-latest | |
| # Build/push plus prepare.rs's 40-minute preparation limit. | |
| timeout-minutes: 65 | |
| permissions: | |
| actions: read # the merge queue's image build of this commit, through the API | |
| contents: read | |
| environment: preview | |
| # Share the manual Image workflow's publication lock: two cold builds | |
| # must not race to replace the same input tag with different digests. | |
| concurrency: | |
| group: image-preview | |
| cancel-in-progress: false | |
| outputs: | |
| # A whole reference contains the secret account ID, so GitHub would | |
| # suppress it as a secret output. Deploy uses its config's account. | |
| digest: ${{ steps.ready.outputs.digest }} | |
| key: ${{ steps.ready.outputs.key }} | |
| env: | |
| CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} | |
| CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false | |
| - uses: dtolnay/rust-toolchain@stable | |
| - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 | |
| with: | |
| shared-key: preview-image | |
| cache-targets: false | |
| cache-directories: target/debug | |
| cache-bin: false | |
| - uses: actions/cache/restore@v4 | |
| with: | |
| path: target/tools | |
| key: node-${{ runner.os }}-${{ hashFiles('crates/devstack/src/node_release.rs') }} | |
| - run: command -v skopeo > /dev/null || { sudo apt-get update -q && sudo apt-get install -y -q skopeo; } | |
| # A merge group's head is the commit master moves to, so the queue's | |
| # image run for this very commit, when one is still going, is | |
| # building these inputs: wait for it rather than build them beside | |
| # it. It ends within its own 65 minutes; past 30, or if it failed, | |
| # this job resolves and builds as it would without it. | |
| - name: wait for the merge queue's image build of this commit | |
| timeout-minutes: 30 | |
| continue-on-error: true | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| set -euo pipefail | |
| run=$(gh api -X GET "repos/$GITHUB_REPOSITORY/actions/workflows/preview-image.yml/runs" -f head_sha="$GITHUB_SHA" -f event=merge_group -f per_page=1 --jq '.workflow_runs[0] // empty | "\(.id) \(.html_url)"') | |
| [ -n "$run" ] || { echo "the merge queue built no image for $GITHUB_SHA"; exit 0; } | |
| id=${run%% *} | |
| echo "the merge queue's image run for $GITHUB_SHA: ${run#* }" | |
| until [ "$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$id" --jq .status)" = completed ]; do sleep 15; done | |
| echo "it ended: $(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$id" --jq .conclusion)" | |
| - id: probe | |
| run: | | |
| cargo xtask image preview --output "$RUNNER_TEMP/preview-image.json" | |
| echo "build=$(jq -r .build "$RUNNER_TEMP/preview-image.json")" >> "$GITHUB_OUTPUT" | |
| # its disk freed before it builds, as the merge queue's build's is | |
| # (the action says why) | |
| - if: steps.probe.outputs.build == 'true' | |
| uses: ./.github/actions/free-disk | |
| - id: buildx | |
| if: steps.probe.outputs.build == 'true' | |
| uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 | |
| # The CLI stage now copies only its inputs (build.rs's host test). | |
| # This cache saves its unchanged stage, the bridge's, and the large | |
| # base. One master-only scope (~3 GB), not one entry per commit. | |
| # Registry input tags, not this evictable cache, preserve identity. | |
| - if: steps.probe.outputs.build == 'true' | |
| uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 | |
| env: | |
| DOCKER_BUILD_SUMMARY: false | |
| DOCKER_BUILD_RECORD_UPLOAD: false | |
| with: | |
| context: . | |
| file: images/hermes/Dockerfile | |
| platforms: linux/amd64 | |
| provenance: false | |
| cache-from: type=gha,scope=preview-hermes | |
| cache-to: type=gha,scope=preview-hermes,mode=max,ignore-error=true,timeout=5m | |
| - id: ready | |
| env: | |
| BUILDX_BUILDER: ${{ steps.buildx.outputs.name }} | |
| run: | | |
| cargo xtask image preview --build --output "$RUNNER_TEMP/preview-image.json" --summary "$GITHUB_STEP_SUMMARY" | |
| echo "digest=$(jq -er .digest "$RUNNER_TEMP/preview-image.json")" >> "$GITHUB_OUTPUT" | |
| echo "key=$(jq -er .key "$RUNNER_TEMP/preview-image.json")" >> "$GITHUB_OUTPUT" | |
| - if: always() && steps.probe.outputs.build == 'true' | |
| name: disk, after | |
| run: | | |
| docker system df; df -h /; free -h | |
| echo "peak: $(awk '$1 > d { d = $1 } END { print d }' "$RUNNER_TEMP/disk-used") MiB of the disk used, $(awk '$2 > m { m = $2 } END { print m }' "$RUNNER_TEMP/disk-used") MiB of the memory" | |
| # Continuous deployment, on a green push: master's to the preview | |
| # (`--branch preview`), production's to production, after `check` and | |
| # the e2e, or after `check` alone when a run proved its tree (`proven`), | |
| # and never after a red test or a failed `proven`. Only Paul updates | |
| # `production` (a protected branch): his push, or the pull request he | |
| # merges into it, is the approval. Each | |
| # deploys from a GitHub environment of its name, `preview` or | |
| # `production`, which the repo's settings let | |
| # only its branch use, and which holds the secrets DEPLOY_REPO_TOKEN (below), | |
| # CLOUDFLARE_API_TOKEN (wrangler's, and the DNS token), the preview's | |
| # PREVIEW_TEST_SECRET, and, optional, CLOUDFLARE_ACCOUNT_ID (below). | |
| # Its config is its environment's file in the private repo | |
| # finitecomputer/fragment-deploy, `preview.jsonc` or `production.jsonc` | |
| # at the root of its main as it stands when the job runs: names and | |
| # paths, no secret's value, kept out of this public repo and its logs, | |
| # and readable by the team, as a secret is not. Every computer image is | |
| # pinned by digest (`--pinned-only`): this job builds none; preview's | |
| # separate image job resolves its input-keyed pin before it can start | |
| # (xtask/src/deploy.rs, `pinned_only`, says why). | |
| deploy: | |
| needs: [proven, check, e2e, preview-image, cli-version, cli-release] | |
| if: >- | |
| ${{ !cancelled() && github.event_name == 'push' | |
| && (github.ref == 'refs/heads/master' || github.ref == 'refs/heads/production') | |
| && needs.check.result == 'success' | |
| && (github.ref == 'refs/heads/production' || needs.preview-image.result == 'success') | |
| && ((needs.proven.result == 'success' && needs.proven.outputs.proven == 'true') | |
| || needs.e2e.result == 'success') }} | |
| runs-on: ubuntu-latest | |
| # the deploy waits up to 20 minutes for its images to be ready on | |
| # Cloudflare's network before it builds (xtask/src/prepare.rs, | |
| # DEPLOY_LIMIT), and, after wrangler's deploy, up to 10 to finish one | |
| # whose Worker is live and 7 for every object class to answer it, and | |
| # 7 more for the release before when it rolls back (rollout.rs) | |
| timeout-minutes: 60 | |
| permissions: | |
| contents: read | |
| environment: ${{ github.ref_name == 'production' && 'production' || 'preview' }} | |
| # one deploy at a time to each, never cancelled mid-way | |
| concurrency: | |
| group: deploy-${{ github.ref_name }} | |
| cancel-in-progress: false | |
| env: | |
| # xtask/src/build.rs, WORKER_BUILD_VERSION | |
| WORKER_BUILD_VERSION: 0.8.5 | |
| # the environment's config, in fragment-deploy's checkout (below) | |
| CONFIG_FILE: ${{ github.ref_name == 'production' && 'production' || 'preview' }}.jsonc | |
| steps: | |
| # the deploy reads its commit, and needs no git credentials | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false | |
| # the config, from fragment-deploy's main as it stands now (a change | |
| # there deploys at the environment's next deploy), read with | |
| # DEPLOY_REPO_TOKEN, a fine-grained token with Contents · Read on that | |
| # repo alone, which its checkout does not keep. It is checked out | |
| # after this repo, whose checkout cleans the tree, into | |
| # deploy-config/, which .gitignore names: the tree stays clean, so the | |
| # deploy's id is its commit (xtask/src/deploy.rs, `git_head`). The job | |
| # prints the config's commit, so a deploy's config is traceable, and | |
| # never its contents. | |
| - env: | |
| DEPLOY_REPO_TOKEN: ${{ secrets.DEPLOY_REPO_TOKEN }} | |
| run: test -n "$DEPLOY_REPO_TOKEN" || { echo "::error::the environment of $GITHUB_REF_NAME's deploy has no DEPLOY_REPO_TOKEN (a fine-grained token with Contents · Read on finitecomputer/fragment-deploy), so it has no config"; exit 1; } | |
| - uses: actions/checkout@v4 | |
| with: | |
| repository: finitecomputer/fragment-deploy | |
| ref: main | |
| token: ${{ secrets.DEPLOY_REPO_TOKEN }} | |
| path: deploy-config | |
| persist-credentials: false | |
| - run: | | |
| at=$(git -C deploy-config rev-parse --short HEAD) | |
| test -s "deploy-config/$CONFIG_FILE" || { echo "::error::finitecomputer/fragment-deploy has no $CONFIG_FILE at its root on main ($at)"; exit 1; } | |
| echo "$GITHUB_REF_NAME deploys with finitecomputer/fragment-deploy's $CONFIG_FILE at $at" | |
| - id: rust | |
| uses: dtolnay/rust-toolchain@stable | |
| with: | |
| targets: wasm32-unknown-unknown | |
| # the e2e's caches, restored and never saved here: a deploy builds what | |
| # its build does (xtask in target/debug, the cell's wasm32 builds, with | |
| # worker-build and its tools). Every input of each but save-if is the | |
| # e2e build's own (.github/actions/e2e-build), as the paths are part | |
| # of a cache's version. Its builds are this commit's when its first | |
| # shard saved them (a master push the e2e ran on; a proven push's | |
| # `e2e-build` saves them beside this job, and so mostly after its | |
| # restore), and their mtimes are made as there: the cell's | |
| # crates are then all fresh, and the Wasm this ships is made from the | |
| # compile the e2e ran. On production they are master's newest (a | |
| # branch reads the default branch's caches), as its shards save none, | |
| # and a proven push runs none; what changed since then is rebuilt. | |
| - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 | |
| with: | |
| shared-key: e2e-tools-worker-build-${{ env.WORKER_BUILD_VERSION }} | |
| workspaces: | | |
| . -> target | |
| cell -> target | |
| cache-targets: false | |
| cache-directories: ~/.cache/worker-build | |
| save-if: false | |
| - uses: actions/cache/restore@v4 | |
| with: | |
| path: | | |
| target/built-tree | |
| target/debug | |
| target/release | |
| cell/target | |
| key: e2e-builds-${{ runner.os }}-${{ steps.rust.outputs.cachekey }}-${{ github.sha }} | |
| restore-keys: e2e-builds-${{ runner.os }}-${{ steps.rust.outputs.cachekey }}- | |
| # as e2e-shard's | |
| - run: | | |
| rustc --edition 2021 -O -o "$RUNNER_TEMP/stamp" xtask/src/stamp.rs | |
| "$RUNNER_TEMP/stamp" restore target/built-tree "$RUNNER_TEMP/stamp-state" | |
| - uses: actions/cache/restore@v4 | |
| with: | |
| path: target/tools | |
| key: node-${{ runner.os }}-${{ hashFiles('crates/devstack/src/node_release.rs') }} | |
| - run: worker-build --version | grep -qF "$WORKER_BUILD_VERSION" || cargo install worker-build --version "$WORKER_BUILD_VERSION" --locked | |
| # the two files the config reads by path, at the paths a CI config | |
| # names (deploy/example.jsonc): the token, its DNS token too, and the | |
| # preview's test secret, each 0600. printf is the shell's own, so no | |
| # value is ever a process's argument, and nothing here echoes one. | |
| # CLOUDFLARE_ACCOUNT_ID is named only to be masked: the runner masks | |
| # every secret a job names, wherever it appears in the job's logs | |
| # (wrangler's and xtask's registry paths and URLs); unset, it is empty | |
| # and masks nothing. | |
| - env: | |
| CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} | |
| PREVIEW_TEST_SECRET: ${{ secrets.PREVIEW_TEST_SECRET }} | |
| CLOUDFLARE_ACCOUNT_ID_MASKED: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} | |
| run: | | |
| umask 077 | |
| test -n "$CLOUDFLARE_API_TOKEN" || { echo "the environment of $GITHUB_REF_NAME's deploy has no CLOUDFLARE_API_TOKEN"; exit 1; } | |
| mkdir -p ~/.config/fragment/secrets | |
| printf '%s\n' "$CLOUDFLARE_API_TOKEN" > ~/.config/fragment/secrets/cloudflare-api-token | |
| if [ -n "$PREVIEW_TEST_SECRET" ]; then printf '%s\n' "$PREVIEW_TEST_SECRET" > ~/.config/fragment/secrets/preview-test-secret; fi | |
| # The private config remains the source for deployment settings. | |
| # Preview's effective image is resolved by this checkout's inputs, | |
| # and applied in memory; this summary is its auditable image trail. | |
| - if: github.ref_name == 'master' | |
| env: | |
| DIGEST: ${{ needs.preview-image.outputs.digest }} | |
| INPUT_KEY: ${{ needs.preview-image.outputs.key }} | |
| run: | | |
| test -n "$DIGEST" && test -n "$INPUT_KEY" || { echo "::error::preview image resolution is missing; no deploy"; exit 1; } | |
| printf '### Resolved preview image\n\nInputs: `%s`\n\nDigest: `%s`\n\n`hermes` uses this digest.\n' "$INPUT_KEY" "$DIGEST" >> "$GITHUB_STEP_SUMMARY" | |
| # green only when every object class answers this commit's release | |
| # (`git rev-parse --short=12 HEAD`) at /healthz/objects; a deploy that | |
| # cannot be finished or verified rolls back, and fails (rollout.rs) | |
| - run: | | |
| args=() | |
| if [ "$GITHUB_REF_NAME" = master ]; then args=(--branch preview --preview-image-digest "$PREVIEW_IMAGE_DIGEST"); fi | |
| cargo xtask deploy --config "deploy-config/$CONFIG_FILE" --pinned-only "${args[@]}" | |
| env: | |
| CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} | |
| PREVIEW_IMAGE_DIGEST: ${{ needs.preview-image.outputs.digest }} | |
| # Empty after a failed lookup/build: deploy still proceeds, with | |
| # no update header. Only a verified published release is named. | |
| FRAGMENT_RELEASED_CLI_VERSION: ${{ needs.cli-release.result == 'success' && needs.cli-release.outputs.version || (needs.cli-version.result == 'success' && needs.cli-version.outputs.version) || '' }} | |
| # Release new CLI inputs after green tests and before production deploys. | |
| # Unchanged inputs reuse their published release. A failed lookup/build | |
| # never blocks deploy, which then advertises no available CLI version. | |
| cli-version: | |
| needs: [proven, check, e2e] | |
| if: >- | |
| ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/production' | |
| && needs.check.result == 'success' | |
| && ((needs.proven.result == 'success' && needs.proven.outputs.proven == 'true') | |
| || needs.e2e.result == 'success') }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| permissions: | |
| contents: read | |
| outputs: | |
| tag: ${{ steps.next.outputs.tag }} | |
| version: ${{ steps.next.outputs.version }} | |
| publish: ${{ steps.next.outputs.publish }} | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false | |
| - uses: dtolnay/rust-toolchain@stable | |
| - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 | |
| with: | |
| save-if: false | |
| - id: next | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| cargo xtask cli-release plan >> "$GITHUB_OUTPUT" | |
| cli-release: | |
| needs: cli-version | |
| if: ${{ !cancelled() && needs.cli-version.result == 'success' && needs.cli-version.outputs.publish == 'true' }} | |
| permissions: | |
| contents: write # the tag and the release | |
| uses: ./.github/workflows/release.yml | |
| with: | |
| tag: ${{ needs.cli-version.outputs.tag }} | |
| sha: ${{ github.sha }} | |
| # A master push whose shards skip (`proven`) still builds the e2e and | |
| # saves it, as the first shard does on any other master push: pull | |
| # requests and merge groups restore master's newest build, the stub | |
| # image's layers and the tools, and without this they built from the | |
| # last master push that ran its shards (on 2026-10-30, about 160 s more | |
| # for the stub's bridge and 100 s more of Rust in every shard). Nothing | |
| # waits for it: `deploy` restores whatever is newest, as it did. | |
| e2e-build: | |
| needs: proven | |
| if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' && needs.proven.result == 'success' && needs.proven.outputs.proven == 'true' }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 20 | |
| env: | |
| # xtask/src/build.rs, WORKER_BUILD_VERSION | |
| WORKER_BUILD_VERSION: 0.8.5 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: ./.github/actions/e2e-build | |
| with: | |
| save: true | |
| worker-build-version: ${{ env.WORKER_BUILD_VERSION }} | |
| # The e2e's builds are kept one entry per master commit (`e2e-shard`'s | |
| # first shard, or `e2e-build`), each about 0.5 GB of the repository's | |
| # 10 GB of caches, and a run restores the newest: after a master push's | |
| # shards or its `e2e-build`, every older entry is deleted. A job of its own, as deleting one needs `actions: write`, | |
| # which no job that runs the repo's code is given. | |
| prune: | |
| needs: [e2e-shard, e2e-build] | |
| if: ${{ !cancelled() && github.event_name == 'push' && github.ref == 'refs/heads/master' }} | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 5 | |
| permissions: | |
| actions: write | |
| steps: | |
| - env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| set -euo pipefail | |
| entries=$(gh api -X GET "repos/$GITHUB_REPOSITORY/actions/caches" -f key=e2e-builds- -f ref=refs/heads/master \ | |
| -f sort=created_at -f direction=desc -f per_page=100 --jq '.actions_caches[] | "\(.id) \(.key) \(.size_in_bytes / 1e6 | floor) MB"') | |
| echo "$entries" | head -n 1 | sed 's/^/kept: /' | |
| echo "$entries" | tail -n +2 | while read -r id rest; do | |
| [ -n "$id" ] || continue | |
| echo "deleted: $id $rest" | |
| gh api -X DELETE "repos/$GITHUB_REPOSITORY/actions/caches/$id" | |
| done | |
| # A pull request's or a merge group's cache is that ref's alone, which | |
| # no other run reads, and every save here is master's (each cache's | |
| # `save-if`, `cache-to` or `save` says so): an entry such a ref holds | |
| # is waste, as the search image's layers were (2026-10-31: 8.8 GB of | |
| # 12.8, every queue run writing 800 MB more). Deleted here: those | |
| # refs' entries alone, never master's or production's, nor any other | |
| # branch's (xtask/src/ci_caches.rs runs this step against fixtures). | |
| - name: delete pull requests' and merge groups' caches | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| run: | | |
| set -euo pipefail | |
| gh api --paginate -X GET "repos/$GITHUB_REPOSITORY/actions/caches" -f per_page=100 \ | |
| | jq -r '.actions_caches[] | select(.ref | test("^refs/pull/[0-9]+/merge$") or startswith("refs/heads/gh-readonly-queue/")) | "\(.id) \(.ref) \(.key)"' \ | |
| | while read -r id ref key; do | |
| case "$ref" in | |
| refs/pull/*/merge | refs/heads/gh-readonly-queue/*) ;; | |
| *) echo "::error::not a pull request's or merge group's ref: $ref"; exit 1 ;; | |
| esac | |
| echo "deleted: $id $ref $key" | |
| gh api -X DELETE "repos/$GITHUB_REPOSITORY/actions/caches/$id" | |
| done |