Move cobalt cloud jobs from benchmarks-ci-azure-eastus2 into benchmarks-ci-azure#2166
Merged
LoopedBard3 merged 1 commit intoaspnet:mainfrom Apr 29, 2026
Merged
Conversation
LoopedBard3
force-pushed
the
MoveEastUS2ToMainAzure
branch
3 times, most recently
from
66cef0b to
0cf702f
Compare
- Merge cobalt cloud machines into benchmarks_ci_azure.json with machine_group support - Regenerate benchmarks-ci-azure.yml using crank-scheduler (14 groups, handles machine conflicts) - Update benchmarks.template.liquid header with scheduler instructions - Remove separate eastus2 pipeline files (benchmarks-ci-azure-eastus2.yml, benchmarks.matrix.azure.eastus2.yml, benchmarks_ci_azure_eastus2.json) - Remove benchmarks.matrix.azure.yml (replaced by JSON + scheduler approach) - Remove cobaltcloud service bus queue (cobalt jobs now use azure/azurearm64 queues) - Remove EAST US 2 MACHINES header from azure.profile.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
force-pushed
the
MoveEastUS2ToMainAzure
branch
from
0cf702f to
ba9a434
Compare
Contributor
Author
|
Verified that the pipeline still runs: https://dev.azure.com/dnceng/internal/_build/results?buildId=2963441&view=results As part of this, I will also have the benchmarks-ci-azure-eastus2 pipeline deleted. |
There was a problem hiding this comment.
Pull request overview
Consolidates the former benchmarks-ci-azure-eastus2 (cobalt cloud) pipeline into the main Azure CI pipeline by moving cobalt machines/scenarios into benchmarks_ci_azure.json, regenerating the combined benchmarks-ci-azure.yml, and removing the now-redundant eastus2/matrix files.
Changes:
- Merge cobalt cloud machines + scenarios into
build/benchmarks_ci_azure.jsonusingmachine_group. - Regenerate
build/benchmarks-ci-azure.ymlto include cobalt jobs and updated scheduling/serialization. - Remove separate eastus2 pipeline/config/matrix YAML files and clean up
azure.profile.ymlsection header.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| build/benchmarks_ci_azure_eastus2.json | Deleted legacy eastus2-only scheduler config. |
| build/benchmarks_ci_azure.json | Adds machine_group to existing machines and introduces cobalt cloud machines + scenario targeting. |
| build/benchmarks.template.liquid | Updates header comments to document crank-scheduler-based YAML generation. |
| build/benchmarks.matrix.azure.yml | Deleted legacy matrix definition (superseded by JSON + scheduler). |
| build/benchmarks.matrix.azure.eastus2.yml | Deleted legacy eastus2 matrix definition. |
| build/benchmarks-ci-azure.yml | Regenerated combined Azure pipeline including cobalt jobs and new dependsOn sequencing. |
| build/benchmarks-ci-azure-eastus2.yml | Deleted legacy eastus2 pipeline. |
| build/azure.profile.yml | Removes the “EAST US 2 MACHINES” section header comment. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sebastienros
approved these changes
Apr 29, 2026
LoopedBard3
added a commit
to LoopedBard3/Benchmarks
that referenced
this pull request
Apr 29, 2026
Reflects main branch changes from PR aspnet#2166: - Merged cobalt-cloud-lin pods (eastus2) into azure config - Removed separate benchmarks_ci_azure_eastus2_pods.json - Fixed IDNA pod load profiles to match updated main: - idna-amd-lin now uses idna-amd-win as load - idna-intel-lin now uses idna-intel-win as load - idna-amd-win now uses idna-intel-lin as load - idna-intel-win now uses idna-amd-lin as load - Added cobalt-cloud-lin-azl3-dual pod for type-2 scenarios (uses cobalt-cloud-lin-db as load instead of client) - Total runs: 26 (matches main azure pipeline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
added a commit
to LoopedBard3/Benchmarks
that referenced
this pull request
Apr 29, 2026
Reflects main branch changes from PR aspnet#2166: - Merged cobalt-cloud-lin pods (eastus2) into azure config - Removed separate benchmarks_ci_azure_eastus2_pods.json - Kept IDNA pod load profiles on linux machines (load jobs require linux), reverting the main branch profile change - Added cobalt-cloud-lin-azl3-dual pod for type-2 scenarios (uses cobalt-cloud-lin-db as load instead of client) - Total runs: 26 (matches main azure pipeline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoopedBard3
added a commit
that referenced
this pull request
May 4, 2026
) * Add pod-based crank scheduler prototype Simplified alternative to PR #2106's full crank-scheduler. Uses a pod model where machines are fixed groups (SUT + load + DB) instead of individual machines with capability scoring and preferred partners. Key simplifications: - Pods define fixed machine groupings (no role priority/scoring) - Shared machines between pods handled via collision detection - Same greedy longest-job-first bin-packing algorithm - Same Liquid template YAML generation - ~570 lines vs ~2000 lines in the full scheduler Includes: - scripts/pod-scheduler/ (5 Python files + README) - build/benchmarks_ci_pods.json (pod-based config for CI benchmarks) * Add azure, azure-eastus2, and cobalt pod configs Pod-based configurations for all three additional CI environments: - benchmarks_ci_azure_pods.json: 6 pods, 14 runs (matches main) - benchmarks_ci_azure_eastus2_pods.json: 2 pods, 12 runs (matches main) - benchmarks_ci_cobalt_pods.json: 4 pods, 44 runs (matches main) Notable pod patterns: - Azure IDNA pods cross-use each other as load machines - Cobalt hosted has 28-core variant pods sharing physical machines with full-core pods (handled by collision detection) - Azure eastus2 pods share load/db, serialized automatically Also fixes unicode bar chars for Windows compatibility. * Update azure pod config: merge eastus2, keep IDNA on linux loads Reflects main branch changes from PR #2166: - Merged cobalt-cloud-lin pods (eastus2) into azure config - Removed separate benchmarks_ci_azure_eastus2_pods.json - Kept IDNA pod load profiles on linux machines (load jobs require linux), reverting the main branch profile change - Added cobalt-cloud-lin-azl3-dual pod for type-2 scenarios (uses cobalt-cloud-lin-db as load instead of client) - Total runs: 26 (matches main azure pipeline) * Regenerate pipeline YAMLs from pod-scheduler configs Generated via: python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_pods.json --template ./build/benchmarks.template.liquid --yaml-output ./build python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_azure_pods.json --template ./build/benchmarks.template.liquid --yaml-output ./build --base-name benchmarks-ci-azure python ./scripts/pod-scheduler/main.py --config ./build/benchmarks_ci_cobalt_pods.json --template ./build/benchmarks.template.liquid --yaml-output ./build --base-name benchmarks-ci-cobalt * Cap timeoutInMinutes at 240 (max 2x old 120 default) Formula is now max(120, min(240, 2 * estimated_runtime)). This prevents scenarios with long runtimes (e.g. Proxies at 150min) from setting unreasonably high timeouts compared to previous values. Resulting timeouts: 120 (default), 140 (Grpc), 180 (PGO/Containers), 240 (Proxies) * Address review feedback - Fix 4 incorrect template filenames in benchmarks_ci_pods.json: crossgen-scenarios -> crossgen2-scenarios, custom-proxies-scenarios -> proxies-custom-scenarios, single-file-scenarios -> singlefile-scenarios, websockets-scenarios -> websocket-scenarios - Fix machine utilization calculation bug (was inflating totals for machines not in current stage) - Remove unused imports (sys, Any, Dict, json, Pod) - Remove dead render_with_liquid function and --template CLI arg - Add guard against empty queues (ZeroDivisionError) - Update README and docstrings to reflect removed template arg Code: - Validate cron schedules at load time and raise on unsupported hour fields instead of silently no-op'ing the offset for split YAMLs - Add optional 'timeout' override per scenario; fall back to the runtime-derived formula when absent - Move pipeline plumbing (pool, service-bus connection/namespace) into JSON metadata.pipeline with the previous hardcoded values as defaults - Strict validation of duplicate pods, duplicate scenario.pods entries, empty queues; default scheduler to fail-fast on unknown/invalid pod references with a --lenient opt-out - Stricter job-id sanitization (handles '.', '/', parens, leading digits, unicode) and explicit duplicate detection in generated YAML - Replace id(stage) bookkeeping in split_schedule with explicit indices; add stable name tie-breaker to create_schedule for deterministic output - Use Run.job_name in the generator instead of duplicating the regex - Drop stale '--template' arg from generated YAML headers and README Tests: - 41 unit + snapshot tests covering models, config loader, scheduler, generator, and YAML parity with the committed *_pods.json configs Cleanup: - Revert benchmarks.template.liquid and benchmarks_ci_azure.json to main; the deleted crank-scheduler does not consume them - Regenerate all four pipeline YAMLs against the new generator * Remove unused benchmarks.template.liquid The Liquid template was only consumed by the deleted crank-scheduler. The pod-scheduler renders pipeline YAML directly via Python, and grep confirms no other script, pipeline, or build step reads this file. * Remove orphaned benchmarks.yml and benchmarks.matrix.0[12].yml These were artifacts of the old hand-driven matrix.yml -> json -> Liquid template -> benchmarks.yml workflow. Their only inbound references were stale documentation comments cross-pointing between each other; nothing in the repo (no script, no pipeline) consumed them. * Document pod-scheduler flow across READMEs and YAML headers - Generated YAML headers now embed the exact regen command (with the source config and base name) and a pointer to scripts/pod-scheduler/README.md, so each file documents how to reproduce itself - New build/README.md maps each *_pods.json config to the YAML it produces, lists the hand-maintained scenario templates, and explains the typical edit/regenerate workflow - Top-level README.md gains a 'Continuous benchmarking pipelines' section linking to the pod-scheduler and build/ docs - pod-scheduler README's Quick Start now uses repo-root-relative commands and points at the snapshot tests for verification - Tests cover the new _format_source_path helper and the snapshot test passes the source config so headers stay verified * Remove orphaned crank-scheduler JSON configs benchmarks_ci.json, benchmarks_ci_azure.json, and benchmarks_ci_cobalt.json used the old 'machines + capabilities' format consumed by the deleted crank-scheduler. Their replacements (benchmarks_ci_pods.json, benchmarks_ci_azure_pods.json, benchmarks_ci_cobalt_pods.json) drive the pod-scheduler. grep finds zero inbound references for any of the three across scripts, pipelines, docs, and tests. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Parker Bibus <parker.bibus@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The cobalt cloud machines were moved from eastus2 to westus2, co-located with the other Azure benchmark machines. This merges the separate eastus2 pipeline back into the main Azure CI pipeline.
Changes
benchmarks_ci_azure.jsonwithmachine_groupsupport for proper schedulingbenchmarks-ci-azure.ymlusing the crank-scheduler (14 groups with intelligentdependsOnto avoid machine conflicts)benchmarks.template.liquidheader with crank-scheduler usage instructionsEAST US 2 MACHINESsection header fromazure.profile.ymlbenchmarks-ci-azure-eastus2.yml,benchmarks.matrix.azure.eastus2.yml,benchmarks_ci_azure_eastus2.jsonbenchmarks.matrix.azure.yml(replaced by JSON + crank-scheduler approach)azure/azurearm64queuesHow the generated pipeline works (Used approach in #2106)
benchmarks_ci_azure.json(machines + scenarios) and producesbenchmarks-ci-azure.ymlvia the liquid templatedependsOnNotes
azure.profile.ymlstill referencecobaltcloudlin*relay endpoints onaspnetperf.servicebus.windows.net— these are Azure Relay Hybrid Connection URLs that require infrastructure-level reconfiguration