This directory contains Azure DevOps pipeline YAML files and the pod-scheduler configs that generate them.
The CI pipelines that run the daily benchmark sweep are generated by the pod-scheduler. Each pipeline maps to a JSON config that lists the available pods (machine groups) and scenarios.
| Generated YAML | Source config | Notes |
|---|---|---|
benchmarks-ci-01.yml, benchmarks-ci-02.yml |
benchmarks_ci_pods.json |
On-prem fleet (Gold); split across two YAMLs to spread load |
benchmarks-ci-azure.yml |
benchmarks_ci_azure_pods.json |
Azure cloud pods (includes the merged eastus2 cobalt cloud pods) |
benchmarks-ci-cobalt.yml |
benchmarks_ci_cobalt_pods.json |
Cobalt-hosted pods |
To regenerate, from the repo root:
python ./scripts/pod-scheduler/main.py \
--config ./build/benchmarks_ci_pods.json \
--yaml-output ./build
python ./scripts/pod-scheduler/main.py \
--config ./build/benchmarks_ci_azure_pods.json \
--base-name benchmarks-ci-azure \
--yaml-output ./build
python ./scripts/pod-scheduler/main.py \
--config ./build/benchmarks_ci_cobalt_pods.json \
--base-name benchmarks-ci-cobalt \
--yaml-output ./buildThe header of each generated YAML embeds the exact regen command for that file.
The citrine1, citrine2, and citrine3 names in the on-prem config are
controller queues, not retired benchmark machines or relay targets.
Everything else in this directory is hand-maintained. The most important groups:
*-scenarios.yml— AzDO job templates that the generated pipelines invoke (e.g.proxies-scenarios.yml,grpc-scenarios.yml). One per benchmark family. The pod-scheduler picks which template to run via thetemplatefield on each scenario in the JSON config.prbenchmarks.yml— scheduled PR-benchmark bot pipeline.prbenchmarks.*.config.yml— components, scenarios, and machine choices for individual PR-benchmark requests.job-variables.yml— shared variable definitions consumed by the scenario templates.
When adding a new benchmark family, drop a *-scenarios.yml template in this
directory and reference it from the relevant benchmarks_ci_*_pods.json.
prbenchmarks.yml polls benchmark requests on dotnet/aspnetcore,
dotnet/efcore, and dotnet/runtime every 30 minutes. The controller image
determines where PR components are built, not which machine runs the benchmark.
The first profile in each bot config is the default machine selection.
| Config | Controller | Supported benchmark machines (default first) |
|---|---|---|
prbenchmarks.aspnetcore.config.yml |
Windows | aspnet-gold-lin, aspnet-gold-win (x64, 56 cores) |
prbenchmarks.efcore.config.yml |
Windows | aspnet-gold-win (x64, 56 cores) |
prbenchmarks.runtime.windows_x64.config.yml |
Windows | aspnet-gold-win (x64, 56 cores) |
prbenchmarks.runtime.linux_x64.config.yml |
Linux | aspnet-gold-lin (x64, 56 cores) |
These selections use the existing aspnet-gold-*-relay profiles imported by
the benchmark scenarios. Gold Linux uses goldlin, goldload, and golddb;
Gold Windows uses goldwin, goldload2, and golddb. The disconnected Perf
and Citrine choices are no longer offered. Runtime ARM64 jobs/configs have
been retired because their relay machines are unavailable; Gold is x64 and
does not replace ARM64 coverage.
The ASP.NET Core, EF Core, and runtime scenario/component definitions are unchanged. This migration does not change the polling schedule, controller images, or Crank SDK/build configuration.
Common edits, all done in JSON:
- Add a scenario: append to
scenarios[]in the appropriatebenchmarks_ci_*_pods.json. - Add a pod / machine class: append to
pods[]and reference its name from any scenario you want to target it. - Change cadence or splits: edit
metadata.schedule(cron),metadata.yaml_generation.target_yaml_count, ormetadata.yaml_generation.schedule_offset_hours. - Override a job's timeout: add
"timeout": <minutes>to the scenario.
After any edit, regenerate the YAML and commit both the JSON and the YAML
together. The snapshot tests in
scripts/pod-scheduler/tests/test_snapshots.py will fail in CI if the
generated YAML drifts from the configs.
The container, gRPC, HttpClient, SignalR, and WebSocket templates pass Crank's
reuseBuild, buildKey, and source.sourceKey options directly. Keys include
$(Build.BuildId): each agent builds once per compatible variant in a pipeline
run, then refreshes sources and builds on the next run. Source and build keys use
separate namespaces because their metadata shares a cache directory.
--session alone does not invalidate caches.
Keys distinguish Docker images, gRPC implementations and frameworks, and
WebSocket compression builds. Container scenarios with a database also include
$(containerDatabaseBuildReuse). Docker image names get the build ID appended in
onConfigure hooks so interleaved runs cannot overwrite each other's image tags;
the gRPC hooks leave non-Docker jobs unchanged.
Only builds are reused. Applications and database containers still start fresh, and warmup/measurement durations are unchanged. Caches remain local to each agent and rely on its existing cleanup/restart policy; no cache TTL is added.
minimal-todo-scenarios.yml dispatches one Docker language-by-size matrix
from scenarios/minimal-todo/minimal-todo.benchmarks.yml for the gold-lin
and Azure azure-arm64/cobalt-cloud-lin pods. C#, Go, and Rust share four
CPU/MiB/RPS profiles: 0.25/64/2000, 0.5/128/10000, 1/256/20000,
and 2/512/37500. There is no separate default-size scheduled run;
csharp-project remains available only for manual, non-Docker comparisons.
Every regular slot publishes the 12 conditional matrix cases. Subject to
the existing pod condition, all four C# cases are active every slot; the
eight Go/Rust cases run only when
Math.round(Date.now() / 43200000) % 14 == 0 (weekly per pipeline/pod).
Existing consumers, machine profiles, queues, and cron expressions are
unchanged. Distinct scenario/size properties include the CPU, memory, and
target to keep time series separate.
Minimal Todo's enclosing timeout is 150 minutes: twelve 10-minute case
budgets plus 30 minutes of headroom. Per-case task timeouts remain 10
minutes. See scenarios/minimal-todo/README.md for exact profile names,
byte limits, commands, and execution paths. Generation, admission, and
calibration tooling remain in the separate dotnet-performance-tools
repository; this directory only carries the scheduled dispatch template.