Skip to content

Commit 0965b48

Browse files
zzylolclaude
andcommitted
Describe the CI benchmark tiers as implemented
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AENeuitPyaE3fBYGTyP2HV
1 parent 6783c3f commit 0965b48

1 file changed

Lines changed: 10 additions & 9 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -79,17 +79,18 @@ cargo test --features otap-engine,stream-benchmark
7979
## Benchmarks
8080

8181
The benchmark is the evidence that a function is cheap enough to leave on.
82-
Expect three tiers:
82+
Three tiers run in CI:
8383

84-
| Tier | When | Scope | Gate | State |
84+
| Tier | When | Scope | Gate | Workflow |
8585
| --- | --- | --- | --- | --- |
86-
| Smoke | Every merge to `main` | Short run of each scenario with correctness checks | Fails on wrong answers | Planned |
87-
| Nightly sweep | Every day | Full rate sweep per scenario | Alerts on a throughput or CPU regression beyond a set margin | The sweep runs today in `.github/workflows/asap-sketch-benchmark.yml` (nightly, on PRs that touch the code, and on manual dispatch); the regression alert is planned |
88-
| PR comparison | On demand, by a comment on a PR | Same sweep, this PR vs `main` | Posts a comparison table | Planned |
89-
90-
Until the PR comparison exists, run the relevant benchmark yourself for a
91-
performance-related change and paste the numbers into the PR. See
92-
`benchmarks/` and the docs in `docs/`.
86+
| Smoke | Every PR and every merge to `main` | fmt, clippy, tests, then `deploy/telemetry-functions/acceptance.py` on real collector containers | Fails on wrong answers | `ci.yml` |
87+
| Nightly sweep | Every day | Full rate sweep per scenario | Fails when throughput drops more than the margin (default 10%) against the previous night | `asap-sketch-benchmark.yml` |
88+
| PR comparison | A maintainer comments `/benchmark` on a PR | Reduced sweep, this PR vs `main` | Posts a comparison table | `benchmark-on-demand.yml` |
89+
90+
Capacity numbers need dedicated CPUs: the nightly and on-demand tiers run on
91+
the runner named by the `BENCH_RUNNER` repository variable (see
92+
`benchmarks/README.md`). For a performance-related change, also paste the
93+
relevant numbers into the PR.
9394

9495
## Weekly sync
9596

0 commit comments

Comments
 (0)