Skip to content

ci(runners): move every Blacksmith job to a RunsOn machine of the same size - #1419

Draft
auxesis wants to merge 12 commits into
docs/ci-cost-decisionsfrom
ci/heavy-jobs-runs-on
Draft

auxesis wants to merge 12 commits into
docs/ci-cost-decisionsfrom
ci/heavy-jobs-runs-on

Conversation

@auxesis

@auxesis auxesis commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This pull request moves every CI job that runs on Blacksmith to a RunsOn machine of the same size. Blacksmith and RunsOn both provide Linux machines for GitHub Actions jobs. RunsOn starts them in CipherStash's own AWS account, at a lower price.

At November 2026's pace, CI for this repository costs about $3,470 a month on Blacksmith. ADR 0004 estimates between $1,140 and $1,350 a month on RunsOn. Each job keeps its vCPU count and Ubuntu version. Job names, triggers, and steps stay the same.

Changes

Runner labels:

  • 31 jobs move from blacksmith-16vcpu-ubuntu-2204 to the 16 vCPU RunsOn label, with Ubuntu 22.04.
  • 24 jobs move from blacksmith-4vcpu-ubuntu-2404 to the 4 vCPU RunsOn label, with Ubuntu 24.04.
  • The three Linux builds of the FFI release matrix move to the 4 vCPU label. scripts/ffi-release-matrix.mjs puts the run ID in their label.
  • Earlier commits on this branch moved some jobs to 8 vCPUs. Those jobs go back to their Blacksmith size.
  • .github/actionlint.yaml lists no self-hosted labels now. actionlint then reports any Blacksmith label.

Docker Hub:

  • Each RunsOn job logs in to Docker Hub before it pulls an image. The login reads the DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD secrets.
  • When either secret is empty, as on a pull request from a fork, the login does nothing. The pull then stays anonymous.
  • The musl build in _build-ffi-artifacts.yml pulls Alpine from Docker Hub. release.yml and ffi-preflight.yml now pass that workflow the two secrets, by name.

apt:

  • Each step that runs apt-get on a RunsOn machine waits up to 300 seconds for the apt lock. A new machine can still be running apt from boot.

Tests, run by pnpm run test:scripts:

  • workflow-runs-on.test.mjs fails on a Blacksmith label, and on a RunsOn label that ADR 0004 does not name.
  • It also fails on a matrix runner missing from MATRIX_RUNNERS, and on an apt command that does not wait for the lock.
  • workflow-docker-hub-login.test.mjs fails when a RunsOn job pulls an image before it logs in.
  • It also fails when a caller does not pass the secrets to a reusable workflow that logs in.

Verification

  • pnpm run test:scripts: all 84 test files passed.
  • pnpm run lint:workflow-cache passed. pnpm run code:check passed, with warnings only.
  • actionlint 1.7.7 with shellcheck 0.10.0 found nothing in the 10 release workflows that lint-release.yml checks.
  • main has 31 jobs at 16 vCPUs and 24 at 4 vCPUs. This branch has the same counts on RunsOn.
  • Against main, the diff adds 16 steps named "Log in to Docker Hub". It renames no job.
  • The new apt test failed when one lock timeout was removed, and passed again when it was restored.
  • This pull request's CI on 3b53d88, an earlier head of this branch: all 65 checks passed, and 2 were skipped. No job ran out of memory.
  • CI started again on the current head, ec00e6c, after the rebase described under "Related".
  • ffi-preflight.yml on 00a8083, an earlier commit of this branch: both gnu builds passed on RunsOn. The musl build failed on the musl-dev pin, as The musl build image no longer builds #1380 describes. Its login step ran and skipped, because the secrets do not exist yet.
  • The wrapper job was skipped because the musl build failed, so it has not run on RunsOn yet.
  • Some 16 vCPU jobs do not run on a pull request: bench-eql, macro-expand-eql, the release jobs, _build-eql-sql, and _build-eql-docs. Their memory on RunsOn is untested.

The Docker Hub secrets do not exist yet. Until they do, every pull is anonymous, and Docker Hub may refuse some pulls.

Related

Review notes

  • Start with scripts/__tests__/lib/runs-on.mjs and workflow-runs-on.test.mjs. They hold the label rules.
  • A c8a machine has 2 GB of memory per vCPU, half of Blacksmith's x64 memory. A job that runs out of memory moves to family=m8a, with 4 GB per vCPU. No job uses m8a yet.
  • The publish jobs also get extras=s3-cache. It has no effect there, because those jobs have no cache step.

@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

@changeset-bot

changeset-bot Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 00a8083

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

No mutants were generated for the changed lines.

@auxesis

auxesis commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

First CI run on RunsOn: every moved job passed

On commit b21019d, all 22 jobs that ran with a RunsOn label passed. All 13 workflow runs passed. Two review workflows were skipped, as they are on every draft.

  • Every job got a machine within 18 to 28 seconds of being queued.
  • Mutants ran on c8a.4xlarge (16 vCPUs) as a spot machine.
  • The fallback family is used. fmt, clippy, nextest, doctests, rustdoc in Tests (crates) ran on m8a.2xlarge, also a spot machine.
  • The jobs that ran:
    • Test EQL: 14 jobs, including all 4 Postgres 17 shards and the e2e property suite.
    • CRAP: both crates.
    • Tests (crates): both jobs.
    • Tests (Go): the WASI, guests, and Linux Go job.
    • Mutants.
    • The short and long auth refresh soaks.
  • I have not compared job times with Blacksmith. The mise cache was cold on this run, because the trial ran with it off.

Still not run on RunsOn: Bench EQL, the macro expansion check, and the release jobs. None of them run on a pull request.

@auxesis

auxesis commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Second CI run: Docker Hub refuses some image pulls on RunsOn

Blocker. On commit 5da3bd9, three jobs failed while starting Postgres. Docker Hub answered toomanyrequests: You have reached your unauthenticated pull rate limit. The jobs pull images from Docker Hub without logging in.

  • Test EQL shards 1/4 and 4/4 failed pulling postgres:17. They failed the same way when I re-ran them. Shards 2/4 and 3/4 passed both times. All four shards passed on b21019d.
  • The protect-ffi integration suite failed pulling postgres:latest. It passed when I re-ran it.

The other 25 RunsOn jobs passed. They include the five Drizzle, prisma-next, and Supabase integration jobs. All 28 jobs got a machine within 18 to 25 seconds.

The trial in #1297 did not report this error. I cannot see from the logs whether RunsOn machines share an outbound address. Each machine reports IsPrivate false.

Which jobs pull from Docker Hub

  • Test EQL: every job that starts Postgres. packages/eql/tests/docker-compose.yml names postgres:14 to postgres:17.
  • protect-ffi: postgres:latest, in languages/typescript/packages/protect-ffi/integration-tests/docker-compose.yml.
  • The db=supabase integration jobs: supabase/postgres and postgrest/postgrest, in local/docker-compose.supabase.yml.

The db=postgres integration jobs pull ghcr.io/cipherstash/postgres-eql from GHCR, so Docker Hub's limit does not apply to them.

Ways to clear it

  1. Pull the same images from AWS ECR Public. For example, public.ecr.aws/docker/library/postgres:17 is the official image's mirror. public.ecr.aws/supabase/postgres:17.4.1.048 and public.ecr.aws/supabase/postgrest:v12.2.12 exist there too. This needs no secret, and fork pull requests keep working. It changes test files, including one in the EQL subtree. The Supabase copies need their digests checked against the pinned ones.
  2. Log in to Docker Hub in each job. This needs a Docker Hub account and a new repository secret. Fork pull requests get no secrets, so their jobs would still pull without logging in.
  3. Add a registry mirror to the RunsOn stack. One example is an ECR pull-through cache. It would cover every repository on RunsOn at once. It is a change to the AWS configuration. This pull request does not touch that configuration.

Bench EQL and the macro expansion check

I started both by hand on this branch. Both got an m8a.2xlarge spot machine within 19 seconds, and both passed. Bench EQL took about 23 minutes.

@auxesis auxesis changed the title ci(runners): run the heavy 16 vCPU jobs on RunsOn ci(runners): move every Blacksmith job to a RunsOn machine of the same size Nov 19, 2026
@auxesis
auxesis force-pushed the ci/heavy-jobs-runs-on branch from 3b53d88 to 607cb23 Compare November 19, 2026 15:02
@auxesis
auxesis changed the base branch from main to docs/ci-cost-decisions November 19, 2026 15:02
The mise cache guard keys each cached mise step by the Ubuntu image its
job runs on. Jobs built on one image fail with "GLIBC_2.38 not found"
when they restore tools built on another. The guard found the image by
looking for `ubuntu-NNNN` in the `runs-on` label.

A RunsOn label names its image differently, as `image=ubuntu22-full-x64`.
The heavy jobs are about to move to RunsOn. The guard would then report
every one of them as having no image. So the guard now reads the
`image=` segment of a RunsOn label.

A RunsOn label without that segment uses the RunsOn stack's default
image. The label does not show that image, so the mise cache must stay
off for such a job.
CI on Blacksmith cost about $3,865 a month at October's pace. The budget
for this repository and the others that share it is $1,000 a month.

RunsOn starts a machine in our own AWS account for each job. In the
runner trial in #1297, a week of these jobs cost $194 on RunsOn c8a with
8 vCPUs. The same jobs cost $843 on Blacksmith. They took 0.93 times as
long on RunsOn.

This moves every heavy job that ran on `blacksmith-16vcpu-ubuntu-2204`.
A heavy job compiles Rust, builds a Docker image or runs a Postgres
suite. Each one now runs on c8a with 8 vCPUs. Mutants runs on c8a with
16 vCPUs. It tests one mutant at a time, and in the trial 16 vCPUs ran
it about 25% faster than 8.

Every label also carries these settings:

- `family=c8a+m8a`: m8a has the same CPU as c8a. RunsOn can use it when
  AWS has no spare c8a spot machines, before paying the on-demand price.
- `image=ubuntu22-full-x64`: Ubuntu 22.04, as on the Blacksmith label.
  The mise cache prefixes stay `mise-v1-ubuntu-2204`.
- `volume=80gb`: the default disk fills during a Rust build.
- `extras=s3-cache`: the job cache comes from RunsOn's S3 bucket in our
  account. Fetching GitHub's cache from AWS is slow.

The light jobs on that label stay on Blacksmith for now. They are change
detection, the matrix setup, two bash-only checks, the summary job, and
an image re-tag. They move to GitHub-hosted runners in a later change.

actionlint does not check a label that contains an expression. So the
RunsOn labels need no entry in `.github/actionlint.yaml`.
A heavy job is one that runs too slowly on a GitHub-hosted standard
runner. For the 4 vCPU jobs, the rule comes from the runner trial in
as Blacksmith.

All four integration suites pass that rule. In the trial's warm run,
GitHub's runner took these multiples of Blacksmith 4 vCPU's time:

- Drizzle: 3.42 (db=postgres) and 3.68 (db=supabase)
- prisma-next: 3.00 (db=postgres) and 2.68 (db=supabase)
- Supabase: 2.96
- protect-ffi: 4.78

The warm run is the trial's like-for-like comparison, because every
runner had a warm cache. On the trial's first run some caches were cold.

They move to RunsOn with 8 vCPUs, like the other heavy jobs. In the
trial's warm run, c8a with 8 vCPUs took 0.84 to 1.05 times Blacksmith 4
vCPU's time on these jobs. They keep Ubuntu 24.04 through
`image=ubuntu24-full-x64`, so protect-ffi's `mise-v1-ubuntu-2404` cache
prefix stays valid.

udeps stays on Blacksmith. GitHub's runner took 1.17 times as long in
the warm run.
Docker Hub refuses pulls without a login after too many from one
address. RunsOn machines reach that limit. On this branch, Test EQL
shards failed with `toomanyrequests` before any test ran, and failed
again on a re-run. The protect-ffi suite failed the same way once.

Every RunsOn job that starts or builds a container now logs in first,
with the DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD secrets:

- A `run:` step pipes the password to `docker login --password-stdin`.
  It comes before Postgres starts in Test EQL, Bench EQL, and the four
  integration suites. In the postgres-eql image release, it comes before
  QEMU and Buildx, because both pull their own images from Docker Hub.
- The Postgres service container in Test EQL's `rust-crates` job sets
  `credentials` from the same secrets.

Both forms do nothing when either secret is empty. That is the case on a
fork's pull request, and until the secrets exist. The step exits early,
and the runner skips the service login when either value is empty. Pulls
then stay anonymous, as before.

`workflow-docker-hub-login.test.mjs` fails when a RunsOn job may pull an
image without logging in first. Any step that starts or builds a
container counts as a pull, whatever the registry. So the test cannot
miss a Docker Hub image it does not recognise. It follows `mise run` into task
bodies, `depends`, and nested `mise run` calls, and into local composite
actions.
The previous commit gave Test EQL's Postgres service `username:` and
`password:` keys from the two Docker Hub secrets. The secrets do not
exist yet, so both values were empty. GitHub rejects an empty value for
either key before the runner starts: "The template is not valid ...
Unexpected value ''". The `rust-crates` job failed at "Set up job". A
fork's pull request would fail the same way.

`credentials` now takes one expression. With both secrets set, it builds
the username and password from them, through `toJSON`, so any character
is escaped. With either secret empty, it gives `{}`. The workflow schema
allows that, because neither key is required. The runner then skips the
login and pulls without one.

The test now rejects the key form and requires the `fromJSON('{}')`
fallback.
A 4 vCPU job counts as heavy when GitHub's standard runner takes more
than twice as long as Blacksmith. The runner trial in #1297 did not time
the FFI builds. So they ran on `ubuntu-24.04` three times, through
`ffi-preflight.yml`. That workflow cannot publish. These are the medians:

- linux-x64-gnu: 258 s on GitHub's runner, 125 s on Blacksmith (2.06).
- linux-arm64-gnu: 288 s on GitHub's runner, 143 s on Blacksmith (2.01).

Both are only just over the line. They move to RunsOn with 8 vCPUs and
Ubuntu 24.04, as Blacksmith used.

The runner comes from `runnerFor` in the matrix script, not from the
workflow. A matrix value cannot read `${{ github.run_id }}`, so the
script puts `GITHUB_RUN_ID` into the label. It throws when the run ID is
missing, because a label without it would be shared by every run.

Three Linux jobs stay as they are:

- The musl build could not be timed. Its image pins
  `musl-dev=1.2.6-r2`. Alpine no longer serves that version, so the
  build fails on any runner. That also breaks the next FFI release.
- The WASM and wrapper build needs every binary, so it never ran.
- Miri and tests-rust took 1.35 and 1.36 times as long on GitHub's
  runner, so they are light.
On its first RunsOn run, the linux-arm64-gnu FFI build failed in its
`apt-get install`: "Could not get lock /var/lib/dpkg/lock-frontend. It
is held by process 1720 (apt)". A RunsOn machine boots for each job, and
apt was still running from boot. GitHub's and Blacksmith's images do not
run apt at boot, so the step never met a held lock there.

apt-get fails at once on a held lock unless told to wait. Both RunsOn
jobs that install packages now pass `-o DPkg::Lock::Timeout=300`: the
gnu FFI build's cross toolchain and the EQL docs build's Doxygen.
…e size

Each job keeps its Blacksmith vCPU count and Ubuntu version. A 16 vCPU
job gets the 16 vCPU RunsOn label with Ubuntu 22.04. A 4 vCPU job gets
the 4 vCPU label with Ubuntu 24.04. The jobs this branch had moved to
8 vCPUs go back to their Blacksmith size. The three Linux platforms of
the FFI release matrix move too.

workflow-runs-on.test.mjs fails when a job names a Blacksmith label, or
a RunsOn label that ADR 0004 does not name. It also fails when a job
whose runner comes from a matrix is missing from MATRIX_RUNNERS.

actionlint now lists no self-hosted labels, so it reports any Blacksmith
label. It does not check a label that contains an expression, which
every RunsOn label does.
Five jobs that moved to RunsOn pull images with no login before the
pull: the Prisma Next E2E, bench and e2e Postgres containers, the
image promotion's Buildx setup, and the musl FFI build in Alpine. Each
now logs in first. The login does nothing when either secret is empty.

The FFI build is a reusable workflow, where secrets are empty unless the
caller passes them. release.yml and ffi-preflight.yml now pass the two
Docker Hub secrets by name. Only the RunsOn legs log in, because the
macOS leg has no Docker.

The login test now also checks a job whose runner comes from a matrix,
when MATRIX_RUNNERS marks it as RunsOn. It also fails when a caller of a
workflow whose RunsOn jobs log in does not pass both secrets.
A new RunsOn machine can still be running apt from boot, and apt-get
fails at once on the held lock. The two apt-get steps on RunsOn already
pass -o DPkg::Lock::Timeout=300. This test fails when a step in a RunsOn
job, or in a local action it uses, runs apt without it.
e2e/tests/prisma-example-readme.e2e.test.ts runs the Prisma example's
README, and the README runs `docker compose up -d`, which pulls
postgres:16 from Docker Hub. The login test cannot see a pull made from
test code. Two RunsOn jobs run that test: the README walkthrough job,
and tests.yml's e2e-tests, whose login came after the step.

The walkthrough job now logs in before the step, and e2e-tests logs in
before "Run E2E tests". HIDDEN_PULLS in the login test names both steps
as pulls, and the test fails when an entry names a step that is gone.
@auxesis
auxesis force-pushed the ci/heavy-jobs-runs-on branch from 607cb23 to ec00e6c Compare November 20, 2026 03:58
@auxesis
auxesis added this pull request to stack #1735 November 20, 2026 03:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant