Skip to content

Latest commit

 

History

History
144 lines (113 loc) · 6.72 KB

README.md

File metadata and controls

144 lines (113 loc) · 6.72 KB

build/ — Pipeline definitions

This directory contains Azure DevOps pipeline YAML files and the pod-scheduler configs that generate them.

Generated files (do not edit by hand)

The CI pipelines that run the daily benchmark sweep are generated by the pod-scheduler. Each pipeline maps to a JSON config that lists the available pods (machine groups) and scenarios.

Generated YAML Source config Notes
benchmarks-ci-01.yml, benchmarks-ci-02.yml benchmarks_ci_pods.json On-prem fleet (Gold); split across two YAMLs to spread load
benchmarks-ci-azure.yml benchmarks_ci_azure_pods.json Azure cloud pods (includes the merged eastus2 cobalt cloud pods)
benchmarks-ci-cobalt.yml benchmarks_ci_cobalt_pods.json Cobalt-hosted pods

To regenerate, from the repo root:

python ./scripts/pod-scheduler/main.py \
    --config ./build/benchmarks_ci_pods.json \
    --yaml-output ./build

python ./scripts/pod-scheduler/main.py \
    --config ./build/benchmarks_ci_azure_pods.json \
    --base-name benchmarks-ci-azure \
    --yaml-output ./build

python ./scripts/pod-scheduler/main.py \
    --config ./build/benchmarks_ci_cobalt_pods.json \
    --base-name benchmarks-ci-cobalt \
    --yaml-output ./build

The header of each generated YAML embeds the exact regen command for that file.

The citrine1, citrine2, and citrine3 names in the on-prem config are controller queues, not retired benchmark machines or relay targets.

Hand-maintained scenario templates

Everything else in this directory is hand-maintained. The most important groups:

  • *-scenarios.yml — AzDO job templates that the generated pipelines invoke (e.g. proxies-scenarios.yml, grpc-scenarios.yml). One per benchmark family. The pod-scheduler picks which template to run via the template field on each scenario in the JSON config.
  • prbenchmarks.yml — scheduled PR-benchmark bot pipeline.
  • prbenchmarks.*.config.yml — components, scenarios, and machine choices for individual PR-benchmark requests.
  • job-variables.yml — shared variable definitions consumed by the scenario templates.

When adding a new benchmark family, drop a *-scenarios.yml template in this directory and reference it from the relevant benchmarks_ci_*_pods.json.

PR benchmarks

prbenchmarks.yml polls benchmark requests on dotnet/aspnetcore, dotnet/efcore, and dotnet/runtime every 30 minutes. The controller image determines where PR components are built, not which machine runs the benchmark. The first profile in each bot config is the default machine selection.

Config Controller Supported benchmark machines (default first)
prbenchmarks.aspnetcore.config.yml Windows aspnet-gold-lin, aspnet-gold-win (x64, 56 cores)
prbenchmarks.efcore.config.yml Windows aspnet-gold-win (x64, 56 cores)
prbenchmarks.runtime.windows_x64.config.yml Windows aspnet-gold-win (x64, 56 cores)
prbenchmarks.runtime.linux_x64.config.yml Linux aspnet-gold-lin (x64, 56 cores)

These selections use the existing aspnet-gold-*-relay profiles imported by the benchmark scenarios. Gold Linux uses goldlin, goldload, and golddb; Gold Windows uses goldwin, goldload2, and golddb. The disconnected Perf and Citrine choices are no longer offered. Runtime ARM64 jobs/configs have been retired because their relay machines are unavailable; Gold is x64 and does not replace ARM64 coverage.

The ASP.NET Core, EF Core, and runtime scenario/component definitions are unchanged. This migration does not change the polling schedule, controller images, or Crank SDK/build configuration.

Editing the schedule

Common edits, all done in JSON:

  • Add a scenario: append to scenarios[] in the appropriate benchmarks_ci_*_pods.json.
  • Add a pod / machine class: append to pods[] and reference its name from any scenario you want to target it.
  • Change cadence or splits: edit metadata.schedule (cron), metadata.yaml_generation.target_yaml_count, or metadata.yaml_generation.schedule_offset_hours.
  • Override a job's timeout: add "timeout": <minutes> to the scenario.

After any edit, regenerate the YAML and commit both the JSON and the YAML together. The snapshot tests in scripts/pod-scheduler/tests/test_snapshots.py will fail in CI if the generated YAML drifts from the configs.

Build reuse within a pipeline run

The container, gRPC, HttpClient, SignalR, and WebSocket templates pass Crank's reuseBuild, buildKey, and source.sourceKey options directly. Keys include $(Build.BuildId): each agent builds once per compatible variant in a pipeline run, then refreshes sources and builds on the next run. Source and build keys use separate namespaces because their metadata shares a cache directory. --session alone does not invalidate caches.

Keys distinguish Docker images, gRPC implementations and frameworks, and WebSocket compression builds. Container scenarios with a database also include $(containerDatabaseBuildReuse). Docker image names get the build ID appended in onConfigure hooks so interleaved runs cannot overwrite each other's image tags; the gRPC hooks leave non-Docker jobs unchanged.

Only builds are reused. Applications and database containers still start fresh, and warmup/measurement durations are unchanged. Caches remain local to each agent and rely on its existing cleanup/restart policy; no cache TTL is added.

Minimal Todo

minimal-todo-scenarios.yml dispatches one Docker language-by-size matrix from scenarios/minimal-todo/minimal-todo.benchmarks.yml for the gold-lin and Azure azure-arm64/cobalt-cloud-lin pods. C#, Go, and Rust share four CPU/MiB/RPS profiles: 0.25/64/2000, 0.5/128/10000, 1/256/20000, and 2/512/37500. There is no separate default-size scheduled run; csharp-project remains available only for manual, non-Docker comparisons.

Every regular slot publishes the 12 conditional matrix cases. Subject to the existing pod condition, all four C# cases are active every slot; the eight Go/Rust cases run only when Math.round(Date.now() / 43200000) % 14 == 0 (weekly per pipeline/pod). Existing consumers, machine profiles, queues, and cron expressions are unchanged. Distinct scenario/size properties include the CPU, memory, and target to keep time series separate.

Minimal Todo's enclosing timeout is 150 minutes: twelve 10-minute case budgets plus 30 minutes of headroom. Per-case task timeouts remain 10 minutes. See scenarios/minimal-todo/README.md for exact profile names, byte limits, commands, and execution paths. Generation, admission, and calibration tooling remain in the separate dotnet-performance-tools repository; this directory only carries the scheduled dispatch template.