Repository navigation
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true |
|
Mutation testing (cargo-mutants,
|
First CI run on RunsOn: every moved job passedOn commit b21019d, all 22 jobs that ran with a RunsOn label passed. All 13 workflow runs passed. Two review workflows were skipped, as they are on every draft.
Still not run on RunsOn: Bench EQL, the macro expansion check, and the release jobs. None of them run on a pull request. |
Second CI run: Docker Hub refuses some image pulls on RunsOnBlocker. On commit 5da3bd9, three jobs failed while starting Postgres. Docker Hub answered
The other 25 RunsOn jobs passed. They include the five Drizzle, prisma-next, and Supabase integration jobs. All 28 jobs got a machine within 18 to 25 seconds. The trial in #1297 did not report this error. I cannot see from the logs whether RunsOn machines share an outbound address. Each machine reports Which jobs pull from Docker Hub
The Ways to clear it
Bench EQL and the macro expansion checkI started both by hand on this branch. Both got an m8a.2xlarge spot machine within 19 seconds, and both passed. Bench EQL took about 23 minutes. |
3b53d88 to
607cb23
Compare
The mise cache guard keys each cached mise step by the Ubuntu image its job runs on. Jobs built on one image fail with "GLIBC_2.38 not found" when they restore tools built on another. The guard found the image by looking for `ubuntu-NNNN` in the `runs-on` label. A RunsOn label names its image differently, as `image=ubuntu22-full-x64`. The heavy jobs are about to move to RunsOn. The guard would then report every one of them as having no image. So the guard now reads the `image=` segment of a RunsOn label. A RunsOn label without that segment uses the RunsOn stack's default image. The label does not show that image, so the mise cache must stay off for such a job.
CI on Blacksmith cost about $3,865 a month at October's pace. The budget for this repository and the others that share it is $1,000 a month. RunsOn starts a machine in our own AWS account for each job. In the runner trial in #1297, a week of these jobs cost $194 on RunsOn c8a with 8 vCPUs. The same jobs cost $843 on Blacksmith. They took 0.93 times as long on RunsOn. This moves every heavy job that ran on `blacksmith-16vcpu-ubuntu-2204`. A heavy job compiles Rust, builds a Docker image or runs a Postgres suite. Each one now runs on c8a with 8 vCPUs. Mutants runs on c8a with 16 vCPUs. It tests one mutant at a time, and in the trial 16 vCPUs ran it about 25% faster than 8. Every label also carries these settings: - `family=c8a+m8a`: m8a has the same CPU as c8a. RunsOn can use it when AWS has no spare c8a spot machines, before paying the on-demand price. - `image=ubuntu22-full-x64`: Ubuntu 22.04, as on the Blacksmith label. The mise cache prefixes stay `mise-v1-ubuntu-2204`. - `volume=80gb`: the default disk fills during a Rust build. - `extras=s3-cache`: the job cache comes from RunsOn's S3 bucket in our account. Fetching GitHub's cache from AWS is slow. The light jobs on that label stay on Blacksmith for now. They are change detection, the matrix setup, two bash-only checks, the summary job, and an image re-tag. They move to GitHub-hosted runners in a later change. actionlint does not check a label that contains an expression. So the RunsOn labels need no entry in `.github/actionlint.yaml`.
A heavy job is one that runs too slowly on a GitHub-hosted standard runner. For the 4 vCPU jobs, the rule comes from the runner trial in as Blacksmith. All four integration suites pass that rule. In the trial's warm run, GitHub's runner took these multiples of Blacksmith 4 vCPU's time: - Drizzle: 3.42 (db=postgres) and 3.68 (db=supabase) - prisma-next: 3.00 (db=postgres) and 2.68 (db=supabase) - Supabase: 2.96 - protect-ffi: 4.78 The warm run is the trial's like-for-like comparison, because every runner had a warm cache. On the trial's first run some caches were cold. They move to RunsOn with 8 vCPUs, like the other heavy jobs. In the trial's warm run, c8a with 8 vCPUs took 0.84 to 1.05 times Blacksmith 4 vCPU's time on these jobs. They keep Ubuntu 24.04 through `image=ubuntu24-full-x64`, so protect-ffi's `mise-v1-ubuntu-2404` cache prefix stays valid. udeps stays on Blacksmith. GitHub's runner took 1.17 times as long in the warm run.
Docker Hub refuses pulls without a login after too many from one address. RunsOn machines reach that limit. On this branch, Test EQL shards failed with `toomanyrequests` before any test ran, and failed again on a re-run. The protect-ffi suite failed the same way once. Every RunsOn job that starts or builds a container now logs in first, with the DOCKER_HUB_USERNAME and DOCKER_HUB_PASSWORD secrets: - A `run:` step pipes the password to `docker login --password-stdin`. It comes before Postgres starts in Test EQL, Bench EQL, and the four integration suites. In the postgres-eql image release, it comes before QEMU and Buildx, because both pull their own images from Docker Hub. - The Postgres service container in Test EQL's `rust-crates` job sets `credentials` from the same secrets. Both forms do nothing when either secret is empty. That is the case on a fork's pull request, and until the secrets exist. The step exits early, and the runner skips the service login when either value is empty. Pulls then stay anonymous, as before. `workflow-docker-hub-login.test.mjs` fails when a RunsOn job may pull an image without logging in first. Any step that starts or builds a container counts as a pull, whatever the registry. So the test cannot miss a Docker Hub image it does not recognise. It follows `mise run` into task bodies, `depends`, and nested `mise run` calls, and into local composite actions.
The previous commit gave Test EQL's Postgres service `username:` and
`password:` keys from the two Docker Hub secrets. The secrets do not
exist yet, so both values were empty. GitHub rejects an empty value for
either key before the runner starts: "The template is not valid ...
Unexpected value ''". The `rust-crates` job failed at "Set up job". A
fork's pull request would fail the same way.
`credentials` now takes one expression. With both secrets set, it builds
the username and password from them, through `toJSON`, so any character
is escaped. With either secret empty, it gives `{}`. The workflow schema
allows that, because neither key is required. The runner then skips the
login and pulls without one.
The test now rejects the key form and requires the `fromJSON('{}')`
fallback.
A 4 vCPU job counts as heavy when GitHub's standard runner takes more than twice as long as Blacksmith. The runner trial in #1297 did not time the FFI builds. So they ran on `ubuntu-24.04` three times, through `ffi-preflight.yml`. That workflow cannot publish. These are the medians: - linux-x64-gnu: 258 s on GitHub's runner, 125 s on Blacksmith (2.06). - linux-arm64-gnu: 288 s on GitHub's runner, 143 s on Blacksmith (2.01). Both are only just over the line. They move to RunsOn with 8 vCPUs and Ubuntu 24.04, as Blacksmith used. The runner comes from `runnerFor` in the matrix script, not from the workflow. A matrix value cannot read `${{ github.run_id }}`, so the script puts `GITHUB_RUN_ID` into the label. It throws when the run ID is missing, because a label without it would be shared by every run. Three Linux jobs stay as they are: - The musl build could not be timed. Its image pins `musl-dev=1.2.6-r2`. Alpine no longer serves that version, so the build fails on any runner. That also breaks the next FFI release. - The WASM and wrapper build needs every binary, so it never ran. - Miri and tests-rust took 1.35 and 1.36 times as long on GitHub's runner, so they are light.
On its first RunsOn run, the linux-arm64-gnu FFI build failed in its `apt-get install`: "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1720 (apt)". A RunsOn machine boots for each job, and apt was still running from boot. GitHub's and Blacksmith's images do not run apt at boot, so the step never met a held lock there. apt-get fails at once on a held lock unless told to wait. Both RunsOn jobs that install packages now pass `-o DPkg::Lock::Timeout=300`: the gnu FFI build's cross toolchain and the EQL docs build's Doxygen.
…e size Each job keeps its Blacksmith vCPU count and Ubuntu version. A 16 vCPU job gets the 16 vCPU RunsOn label with Ubuntu 22.04. A 4 vCPU job gets the 4 vCPU label with Ubuntu 24.04. The jobs this branch had moved to 8 vCPUs go back to their Blacksmith size. The three Linux platforms of the FFI release matrix move too. workflow-runs-on.test.mjs fails when a job names a Blacksmith label, or a RunsOn label that ADR 0004 does not name. It also fails when a job whose runner comes from a matrix is missing from MATRIX_RUNNERS. actionlint now lists no self-hosted labels, so it reports any Blacksmith label. It does not check a label that contains an expression, which every RunsOn label does.
Five jobs that moved to RunsOn pull images with no login before the pull: the Prisma Next E2E, bench and e2e Postgres containers, the image promotion's Buildx setup, and the musl FFI build in Alpine. Each now logs in first. The login does nothing when either secret is empty. The FFI build is a reusable workflow, where secrets are empty unless the caller passes them. release.yml and ffi-preflight.yml now pass the two Docker Hub secrets by name. Only the RunsOn legs log in, because the macOS leg has no Docker. The login test now also checks a job whose runner comes from a matrix, when MATRIX_RUNNERS marks it as RunsOn. It also fails when a caller of a workflow whose RunsOn jobs log in does not pass both secrets.
A new RunsOn machine can still be running apt from boot, and apt-get fails at once on the held lock. The two apt-get steps on RunsOn already pass -o DPkg::Lock::Timeout=300. This test fails when a step in a RunsOn job, or in a local action it uses, runs apt without it.
e2e/tests/prisma-example-readme.e2e.test.ts runs the Prisma example's README, and the README runs `docker compose up -d`, which pulls postgres:16 from Docker Hub. The login test cannot see a pull made from test code. Two RunsOn jobs run that test: the README walkthrough job, and tests.yml's e2e-tests, whose login came after the step. The walkthrough job now logs in before the step, and e2e-tests logs in before "Run E2E tests". HIDDEN_PULLS in the login test names both steps as pulls, and the test fails when an entry names a step that is gone.
607cb23 to
ec00e6c
Compare
Summary
This pull request moves every CI job that runs on Blacksmith to a RunsOn machine of the same size. Blacksmith and RunsOn both provide Linux machines for GitHub Actions jobs. RunsOn starts them in CipherStash's own AWS account, at a lower price.
At November 2026's pace, CI for this repository costs about $3,470 a month on Blacksmith. ADR 0004 estimates between $1,140 and $1,350 a month on RunsOn. Each job keeps its vCPU count and Ubuntu version. Job names, triggers, and steps stay the same.
Changes
Runner labels:
blacksmith-16vcpu-ubuntu-2204to the 16 vCPU RunsOn label, with Ubuntu 22.04.blacksmith-4vcpu-ubuntu-2404to the 4 vCPU RunsOn label, with Ubuntu 24.04.scripts/ffi-release-matrix.mjsputs the run ID in their label..github/actionlint.yamllists no self-hosted labels now. actionlint then reports any Blacksmith label.Docker Hub:
DOCKER_HUB_USERNAMEandDOCKER_HUB_PASSWORDsecrets._build-ffi-artifacts.ymlpulls Alpine from Docker Hub.release.ymlandffi-preflight.ymlnow pass that workflow the two secrets, by name.apt:
apt-geton a RunsOn machine waits up to 300 seconds for the apt lock. A new machine can still be running apt from boot.Tests, run by
pnpm run test:scripts:workflow-runs-on.test.mjsfails on a Blacksmith label, and on a RunsOn label that ADR 0004 does not name.MATRIX_RUNNERS, and on an apt command that does not wait for the lock.workflow-docker-hub-login.test.mjsfails when a RunsOn job pulls an image before it logs in.Verification
pnpm run test:scripts: all 84 test files passed.pnpm run lint:workflow-cachepassed.pnpm run code:checkpassed, with warnings only.lint-release.ymlchecks.mainhas 31 jobs at 16 vCPUs and 24 at 4 vCPUs. This branch has the same counts on RunsOn.main, the diff adds 16 steps named "Log in to Docker Hub". It renames no job.ffi-preflight.ymlon 00a8083, an earlier commit of this branch: both gnu builds passed on RunsOn. The musl build failed on themusl-devpin, as The musl build image no longer builds #1380 describes. Its login step ran and skipped, because the secrets do not exist yet.bench-eql,macro-expand-eql, the release jobs,_build-eql-sql, and_build-eql-docs. Their memory on RunsOn is untested.The Docker Hub secrets do not exist yet. Until they do, every pull is anonymous, and Docker Hub may refuse some pulls.
Related
mainat 11db0ef.mainas 11db0ef. This branch no longer contains its commits.Review notes
scripts/__tests__/lib/runs-on.mjsandworkflow-runs-on.test.mjs. They hold the label rules.family=m8a, with 4 GB per vCPU. No job uses m8a yet.extras=s3-cache. It has no effect there, because those jobs have no cache step.