Skip to content

Re-work how we select machines for CI... and friends #4619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tuxerrante wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Re-work how we select machines for CI... and friends #4619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a1dcbcb
[ARO-24603] Add centralized vms package for VM size management
tuxerrante Apr 16, 2026
b0a664b
[ARO-24603] Refactor API module to use centralized vms package
tuxerrante Apr 16, 2026
bd95b60
[ARO-24603] Remove FeatureRequireD2sWorkers and update callers to use…
tuxerrante Apr 16, 2026
fafb873
[ARO-24603] Update docs to reference new vms package location
tuxerrante Apr 16, 2026
af23276
[ARO-24603] Add focused VM size regression tests
tuxerrante Apr 16, 2026
032a833
[ARO-24603] Reject malformed versions in VM size validation
tuxerrante Apr 16, 2026
54ca805
[ARO-24603] Skip SP cleanup for workload identity deletes
tuxerrante Apr 16, 2026
5fcf10c
[ARO-24603] Make admin VM size checks CI-aware
tuxerrante Apr 16, 2026
dc6e120
[ARO-24603] Fix copyright header placement in vms/types.go
tuxerrante Apr 17, 2026
4dc239f
[ARO-24603] Remove stale TODO and fix log message typo in cluster.go
tuxerrante Apr 17, 2026
ea55a0e
[ARO-24603] Add OS_CLUSTER_VERSION to env.example for developer disco…
Copilot Apr 17, 2026
72a9075
test(api): pin rebased vm size carry-forwards
tuxerrante May 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions docs/adding-new-instance-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,29 +33,23 @@ The desired instance types should be free of any restrictions. The subscription

### CLI Method

1) Comment out `FeatureRequireD2sWorkers` from the range of features in `pkg/env/dev.go`. This will allow you to create development clusters with other VM sizes.
1) Follow the usual steps to [deploy a development RP](https://github.com/Azure/ARO-RP/blob/master/docs/deploy-development-rp.md), but don't use the hack script to create a cluster.

> __NOTE:__ Please be responsible with your usage of larger VM sizes, as they incur additional cost.

2) Follow the usual steps to [deploy a development RP](https://github.com/Azure/ARO-RP/blob/master/docs/deploy-development-rp.md), but don't use the hack script to create a cluster.

3) Follow steps in https://docs.microsoft.com/en-us/azure/openshift/tutorial-create-cluster to create a cluster, specifying `-worker-vm-size` and/or `--master-vm-size` in the `az aro create` step to specify an alternate sku:
2) Follow steps in https://docs.microsoft.com/en-us/azure/openshift/tutorial-create-cluster to create a cluster, specifying `-worker-vm-size` and/or `--master-vm-size` in the `az aro create` step to specify an alternate sku:

~~~
az aro create --resource-group $RESOURCEGROUP --name $CLUSTER --vnet aro-lseries --master-subnet master-subnet --worker-subnet worker-subnet --worker-vm-size "Standard_L8s_v2"
~~~

4) Once an install with an alternate size is successful, a basic check of cluster health can be conducted, as well as local e2e tests to confirm supportability.
3) Once an install with an alternate size is successful, a basic check of cluster health can be conducted, as well as local e2e tests to confirm supportability.

### Hack scripts method

1) Comment out `FeatureRequireD2sWorkers` from the range of features in `pkg/env/dev.go`.

2) Start your local RP. If it was already running, restart it to take into account commented lines.
1) Start your local RP. If it was already running, restart it to take into account commented lines.

3) Use the [hack script to create a cluster.](https://github.com/cadenmarchese/ARO-RP/blob/master/docs/deploy-development-rp.md#run-the-rp-and-create-a-cluster), with MASTER_VM_SIZE and WORKER_VM_SIZE variables set to desired instance size.
2) Use the [hack script to create a cluster.](https://github.com/cadenmarchese/ARO-RP/blob/master/docs/deploy-development-rp.md#run-the-rp-and-create-a-cluster), with MASTER_VM_SIZE and WORKER_VM_SIZE variables set to desired instance size.

4) Once an install with an alternate size is successful, a basic check of cluster health can be conducted, as well as local e2e tests to confirm supportability.
3) Once an install with an alternate size is successful, a basic check of cluster health can be conducted, as well as local e2e tests to confirm supportability.

### Post-install method

Expand Down
3 changes: 3 additions & 0 deletions env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ export AZURE_EXTENSION_DEV_SOURCES="$(pwd)/python"

export CLUSTER_RESOURCEGROUP="${AZURE_PREFIX}-v4-$LOCATION"
export CLUSTER_NAME="${AZURE_PREFIX}-aro-cluster"
# OS_CLUSTER_VERSION overrides the default OpenShift install version used by hack/cluster create.
# If unset, the default install stream version is used automatically.
# export OS_CLUSTER_VERSION="4.14.1"
# CLUSTER is read by hack/cluster (Go) via mapstructure; CLUSTER_NAME is used
# by hack/setup_resources.sh and az aro CLI commands.
export CLUSTER="${CLUSTER_NAME}"
Expand Down
175 changes: 4 additions & 171 deletions pkg/api/admin/openshiftcluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ package admin

import (
"time"

"github.com/Azure/ARO-RP/pkg/api/util/vms"
)

// OpenShiftClusterList represents a list of OpenShift clusters.
Expand Down Expand Up @@ -267,185 +269,16 @@ const (

// MasterProfile represents a master profile.
type MasterProfile struct {
VMSize VMSize `json:"vmSize,omitempty"`
VMSize vms.VMSize `json:"vmSize,omitempty"`
SubnetID string `json:"subnetId,omitempty"`
EncryptionAtHost EncryptionAtHost `json:"encryptionAtHost,omitempty"`
DiskEncryptionSetID string `json:"diskEncryptionSetId,omitempty"`
}

// VMSize represents a VM size.
type VMSize string

// VMSize constants.
const (
VMSizeStandardD2sV3 VMSize = "Standard_D2s_v3"
VMSizeStandardD4sV3 VMSize = "Standard_D4s_v3"
VMSizeStandardD8sV3 VMSize = "Standard_D8s_v3"
VMSizeStandardD16sV3 VMSize = "Standard_D16s_v3"
VMSizeStandardD32sV3 VMSize = "Standard_D32s_v3"

VMSizeStandardD4sV4 VMSize = "Standard_D4s_v4"
VMSizeStandardD8sV4 VMSize = "Standard_D8s_v4"
VMSizeStandardD16sV4 VMSize = "Standard_D16s_v4"
VMSizeStandardD32sV4 VMSize = "Standard_D32s_v4"
VMSizeStandardD64sV4 VMSize = "Standard_D64s_v4"

VMSizeStandardD4sV5 VMSize = "Standard_D4s_v5"
VMSizeStandardD8sV5 VMSize = "Standard_D8s_v5"
VMSizeStandardD16sV5 VMSize = "Standard_D16s_v5"
VMSizeStandardD32sV5 VMSize = "Standard_D32s_v5"
VMSizeStandardD64sV5 VMSize = "Standard_D64s_v5"
VMSizeStandardD96sV5 VMSize = "Standard_D96s_v5"

VMSizeStandardD4asV4 VMSize = "Standard_D4as_v4"
VMSizeStandardD8asV4 VMSize = "Standard_D8as_v4"
VMSizeStandardD16asV4 VMSize = "Standard_D16as_v4"
VMSizeStandardD32asV4 VMSize = "Standard_D32as_v4"
VMSizeStandardD64asV4 VMSize = "Standard_D64as_v4"
VMSizeStandardD96asV4 VMSize = "Standard_D96as_v4"

VMSizeStandardD4asV5 VMSize = "Standard_D4as_v5"
VMSizeStandardD8asV5 VMSize = "Standard_D8as_v5"
VMSizeStandardD16asV5 VMSize = "Standard_D16as_v5"
VMSizeStandardD32asV5 VMSize = "Standard_D32as_v5"
VMSizeStandardD64asV5 VMSize = "Standard_D64as_v5"
VMSizeStandardD96asV5 VMSize = "Standard_D96as_v5"

VMSizeStandardD4dsV5 VMSize = "Standard_D4ds_v5"
VMSizeStandardD8dsV5 VMSize = "Standard_D8ds_v5"
VMSizeStandardD16dsV5 VMSize = "Standard_D16ds_v5"
VMSizeStandardD32dsV5 VMSize = "Standard_D32ds_v5"
VMSizeStandardD48dsV5 VMSize = "Standard_D48ds_v5"
VMSizeStandardD64dsV5 VMSize = "Standard_D64ds_v5"
VMSizeStandardD96dsV5 VMSize = "Standard_D96ds_v5"

VMSizeStandardD4sV6 VMSize = "Standard_D4s_v6"
VMSizeStandardD8sV6 VMSize = "Standard_D8s_v6"
VMSizeStandardD16sV6 VMSize = "Standard_D16s_v6"
VMSizeStandardD32sV6 VMSize = "Standard_D32s_v6"
VMSizeStandardD64sV6 VMSize = "Standard_D64s_v6"
VMSizeStandardD96sV6 VMSize = "Standard_D96s_v6"

VMSizeStandardD4dsV6 VMSize = "Standard_D4ds_v6"
VMSizeStandardD8dsV6 VMSize = "Standard_D8ds_v6"
VMSizeStandardD16dsV6 VMSize = "Standard_D16ds_v6"
VMSizeStandardD32dsV6 VMSize = "Standard_D32ds_v6"
VMSizeStandardD64dsV6 VMSize = "Standard_D64ds_v6"
VMSizeStandardD96dsV6 VMSize = "Standard_D96ds_v6"

VMSizeStandardE4sV3 VMSize = "Standard_E4s_v3"
VMSizeStandardE8sV3 VMSize = "Standard_E8s_v3"
VMSizeStandardE16sV3 VMSize = "Standard_E16s_v3"
VMSizeStandardE32sV3 VMSize = "Standard_E32s_v3"

VMSizeStandardE2sV4 VMSize = "Standard_E2s_v4"
VMSizeStandardE4sV4 VMSize = "Standard_E4s_v4"
VMSizeStandardE8sV4 VMSize = "Standard_E8s_v4"
VMSizeStandardE16sV4 VMSize = "Standard_E16s_v4"
VMSizeStandardE20sV4 VMSize = "Standard_E20s_v4"
VMSizeStandardE32sV4 VMSize = "Standard_E32s_v4"
VMSizeStandardE48sV4 VMSize = "Standard_E48s_v4"
VMSizeStandardE64sV4 VMSize = "Standard_E64s_v4"

VMSizeStandardE2sV5 VMSize = "Standard_E2s_v5"
VMSizeStandardE4sV5 VMSize = "Standard_E4s_v5"
VMSizeStandardE8sV5 VMSize = "Standard_E8s_v5"
VMSizeStandardE16sV5 VMSize = "Standard_E16s_v5"
VMSizeStandardE20sV5 VMSize = "Standard_E20s_v5"
VMSizeStandardE32sV5 VMSize = "Standard_E32s_v5"
VMSizeStandardE48sV5 VMSize = "Standard_E48s_v5"
VMSizeStandardE64sV5 VMSize = "Standard_E64s_v5"
VMSizeStandardE96sV5 VMSize = "Standard_E96s_v5"

VMSizeStandardE4asV4 VMSize = "Standard_E4as_v4"
VMSizeStandardE8asV4 VMSize = "Standard_E8as_v4"
VMSizeStandardE16asV4 VMSize = "Standard_E16as_v4"
VMSizeStandardE20asV4 VMSize = "Standard_E20as_v4"
VMSizeStandardE32asV4 VMSize = "Standard_E32as_v4"
VMSizeStandardE48asV4 VMSize = "Standard_E48as_v4"
VMSizeStandardE64asV4 VMSize = "Standard_E64as_v4"
VMSizeStandardE96asV4 VMSize = "Standard_E96as_v4"

VMSizeStandardE8asV5 VMSize = "Standard_E8as_v5"
VMSizeStandardE16asV5 VMSize = "Standard_E16as_v5"
VMSizeStandardE20asV5 VMSize = "Standard_E20as_v5"
VMSizeStandardE32asV5 VMSize = "Standard_E32as_v5"
VMSizeStandardE48asV5 VMSize = "Standard_E48as_v5"
VMSizeStandardE64asV5 VMSize = "Standard_E64as_v5"
VMSizeStandardE96asV5 VMSize = "Standard_E96as_v5"

VMSizeStandardE64isV3 VMSize = "Standard_E64is_v3"
VMSizeStandardE80isV4 VMSize = "Standard_E80is_v4"
VMSizeStandardE80idsV4 VMSize = "Standard_E80ids_v4"
VMSizeStandardE96dsV5 VMSize = "Standard_E96ds_v5"
VMSizeStandardE104isV5 VMSize = "Standard_E104is_v5"
VMSizeStandardE104idsV5 VMSize = "Standard_E104ids_v5"

VMSizeStandardF4sV2 VMSize = "Standard_F4s_v2"
VMSizeStandardF8sV2 VMSize = "Standard_F8s_v2"
VMSizeStandardF16sV2 VMSize = "Standard_F16s_v2"
VMSizeStandardF32sV2 VMSize = "Standard_F32s_v2"
VMSizeStandardF72sV2 VMSize = "Standard_F72s_v2"

VMSizeStandardM128ms VMSize = "Standard_M128ms"

VMSizeStandardL4s VMSize = "Standard_L4s"
VMSizeStandardL8s VMSize = "Standard_L8s"
VMSizeStandardL16s VMSize = "Standard_L16s"
VMSizeStandardL32s VMSize = "Standard_L32s"

VMSizeStandardL8sV2 VMSize = "Standard_L8s_v2"
VMSizeStandardL16sV2 VMSize = "Standard_L16s_v2"
VMSizeStandardL32sV2 VMSize = "Standard_L32s_v2"
VMSizeStandardL48sV2 VMSize = "Standard_L48s_v2"
VMSizeStandardL64sV2 VMSize = "Standard_L64s_v2"

VMSizeStandardL8sV3 VMSize = "Standard_L8s_v3"
VMSizeStandardL16sV3 VMSize = "Standard_L16s_v3"
VMSizeStandardL32sV3 VMSize = "Standard_L32s_v3"
VMSizeStandardL48sV3 VMSize = "Standard_L48s_v3"
VMSizeStandardL64sV3 VMSize = "Standard_L64s_v3"

VMSizeStandardL4sV4 VMSize = "Standard_L4s_v4"
VMSizeStandardL8sV4 VMSize = "Standard_L8s_v4"
VMSizeStandardL16sV4 VMSize = "Standard_L16s_v4"
VMSizeStandardL32sV4 VMSize = "Standard_L32s_v4"
VMSizeStandardL48sV4 VMSize = "Standard_L48s_v4"
VMSizeStandardL64sV4 VMSize = "Standard_L64s_v4"
VMSizeStandardL80sV4 VMSize = "Standard_L80s_v4"

VMSizeStandardD4lsV6 VMSize = "Standard_D4ls_v6"
VMSizeStandardD8lsV6 VMSize = "Standard_D8ls_v6"
VMSizeStandardD16lsV6 VMSize = "Standard_D16ls_v6"
VMSizeStandardD32lsV6 VMSize = "Standard_D32ls_v6"
VMSizeStandardD48lsV6 VMSize = "Standard_D48ls_v6"
VMSizeStandardD64lsV6 VMSize = "Standard_D64ls_v6"
VMSizeStandardD96lsV6 VMSize = "Standard_D96ls_v6"

VMSizeStandardD4ldsV6 VMSize = "Standard_D4lds_v6"
VMSizeStandardD8ldsV6 VMSize = "Standard_D8lds_v6"
VMSizeStandardD16ldsV6 VMSize = "Standard_D1l6ds_v6"
VMSizeStandardD32ldsV6 VMSize = "Standard_D32lds_v6"
VMSizeStandardD48ldsV6 VMSize = "Standard_D48lds_v6"
VMSizeStandardD64ldsV6 VMSize = "Standard_D64lds_v6"
VMSizeStandardD96ldsV6 VMSize = "Standard_D96lds_v6"

// GPU VMs
VMSizeStandardNC4asT4V3 VMSize = "Standard_NC4as_T4_v3"
VMSizeStandardNC8asT4V3 VMSize = "Standard_NC8as_T4_v3"
VMSizeStandardNC16asT4V3 VMSize = "Standard_NC16as_T4_v3"
VMSizeStandardNC64asT4V3 VMSize = "Standard_NC64as_T4_v3"

VMSizeStandardNC6sV3 VMSize = "Standard_NC6s_v3"
VMSizeStandardNC12sV3 VMSize = "Standard_NC12s_v3"
VMSizeStandardNC24sV3 VMSize = "Standard_NC24s_v3"
VMSizeStandardNC24rsV3 VMSize = "Standard_NC24rs_v3"
)

// WorkerProfile represents a worker profile.
type WorkerProfile struct {
Name string `json:"name,omitempty"`
VMSize VMSize `json:"vmSize,omitempty"`
VMSize vms.VMSize `json:"vmSize,omitempty"`
DiskSizeGB int `json:"diskSizeGB,omitempty"`
SubnetID string `json:"subnetId,omitempty"`
Count int `json:"count,omitempty"`
Expand Down
12 changes: 6 additions & 6 deletions pkg/api/admin/openshiftcluster_convert.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ func (c openShiftClusterConverter) ToExternal(oc *api.OpenShiftCluster) interfac
}(),
},
MasterProfile: MasterProfile{
VMSize: VMSize(oc.Properties.MasterProfile.VMSize),
VMSize: oc.Properties.MasterProfile.VMSize,
SubnetID: oc.Properties.MasterProfile.SubnetID,
EncryptionAtHost: EncryptionAtHost(oc.Properties.MasterProfile.EncryptionAtHost),
DiskEncryptionSetID: oc.Properties.MasterProfile.DiskEncryptionSetID,
Expand Down Expand Up @@ -133,7 +133,7 @@ func (c openShiftClusterConverter) ToExternal(oc *api.OpenShiftCluster) interfac
for _, p := range oc.Properties.WorkerProfiles {
out.Properties.WorkerProfiles = append(out.Properties.WorkerProfiles, WorkerProfile{
Name: p.Name,
VMSize: VMSize(p.VMSize),
VMSize: p.VMSize,
DiskSizeGB: p.DiskSizeGB,
SubnetID: p.SubnetID,
Count: p.Count,
Expand All @@ -148,7 +148,7 @@ func (c openShiftClusterConverter) ToExternal(oc *api.OpenShiftCluster) interfac
for _, p := range oc.Properties.WorkerProfilesStatus {
out.Properties.WorkerProfilesStatus = append(out.Properties.WorkerProfilesStatus, WorkerProfile{
Name: p.Name,
VMSize: VMSize(p.VMSize),
VMSize: p.VMSize,
DiskSizeGB: p.DiskSizeGB,
SubnetID: p.SubnetID,
Count: p.Count,
Expand Down Expand Up @@ -370,7 +370,7 @@ func (c openShiftClusterConverter) ToInternal(_oc interface{}, out *api.OpenShif
}
}

out.Properties.MasterProfile.VMSize = api.VMSize(oc.Properties.MasterProfile.VMSize)
out.Properties.MasterProfile.VMSize = oc.Properties.MasterProfile.VMSize
out.Properties.MasterProfile.SubnetID = oc.Properties.MasterProfile.SubnetID
out.Properties.MasterProfile.EncryptionAtHost = api.EncryptionAtHost(oc.Properties.MasterProfile.EncryptionAtHost)
out.Properties.MasterProfile.DiskEncryptionSetID = oc.Properties.MasterProfile.DiskEncryptionSetID
Expand All @@ -381,7 +381,7 @@ func (c openShiftClusterConverter) ToInternal(_oc interface{}, out *api.OpenShif
out.Properties.WorkerProfiles = make([]api.WorkerProfile, len(oc.Properties.WorkerProfiles))
for i := range oc.Properties.WorkerProfiles {
out.Properties.WorkerProfiles[i].Name = oc.Properties.WorkerProfiles[i].Name
out.Properties.WorkerProfiles[i].VMSize = api.VMSize(oc.Properties.WorkerProfiles[i].VMSize)
out.Properties.WorkerProfiles[i].VMSize = oc.Properties.WorkerProfiles[i].VMSize
out.Properties.WorkerProfiles[i].DiskSizeGB = oc.Properties.WorkerProfiles[i].DiskSizeGB
out.Properties.WorkerProfiles[i].SubnetID = oc.Properties.WorkerProfiles[i].SubnetID
out.Properties.WorkerProfiles[i].Count = oc.Properties.WorkerProfiles[i].Count
Expand All @@ -394,7 +394,7 @@ func (c openShiftClusterConverter) ToInternal(_oc interface{}, out *api.OpenShif
out.Properties.WorkerProfilesStatus = make([]api.WorkerProfile, len(oc.Properties.WorkerProfilesStatus))
for i := range oc.Properties.WorkerProfilesStatus {
out.Properties.WorkerProfilesStatus[i].Name = oc.Properties.WorkerProfilesStatus[i].Name
out.Properties.WorkerProfilesStatus[i].VMSize = api.VMSize(oc.Properties.WorkerProfilesStatus[i].VMSize)
out.Properties.WorkerProfilesStatus[i].VMSize = oc.Properties.WorkerProfilesStatus[i].VMSize
out.Properties.WorkerProfilesStatus[i].DiskSizeGB = oc.Properties.WorkerProfilesStatus[i].DiskSizeGB
out.Properties.WorkerProfilesStatus[i].SubnetID = oc.Properties.WorkerProfilesStatus[i].SubnetID
out.Properties.WorkerProfilesStatus[i].Count = oc.Properties.WorkerProfilesStatus[i].Count
Expand Down
2 changes: 1 addition & 1 deletion pkg/api/admin/openshiftcluster_validatestatic.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
type openShiftClusterStaticValidator struct{}

// Validate validates an OpenShift cluster
func (sv openShiftClusterStaticValidator) Static(_oc interface{}, _current *api.OpenShiftCluster, location, domain string, requireD2sWorkers bool, installArchitectureVersion api.ArchitectureVersion, resourceID string) error {
func (sv openShiftClusterStaticValidator) Static(_oc interface{}, _current *api.OpenShiftCluster, isCI bool, location string, domain string, installArchitectureVersion api.ArchitectureVersion, resourceID string) error {
if _current == nil {
return api.NewCloudError(http.StatusBadRequest, api.CloudErrorCodeRequestNotAllowed, "", "Admin API does not allow cluster creation.")
}
Expand Down
Loading
Loading