From f24843c5563677afc0c4816d9d620a8ac6f19f66 Mon Sep 17 00:00:00 2001 From: Nic Cope Date: Thu, 3 Sep 2026 15:29:31 -0700 Subject: [PATCH 1/2] Activate cloud managed resources on demand from InferenceCluster The Configuration ships one ManagedResourceActivationPolicy that activates every cloud's managed resource kinds. Crossplane v2.4 scales a safe-start provider's runtime to zero until one of its managed resource definitions is active, but activating every kind up front keeps all cloud providers running on every control plane, even one that only ever provisions a single cloud. compose-inference-cluster now composes a policy activating just the kinds the cloud it provisions needs, so a control plane runs a cloud's providers only once it has an InferenceCluster on that cloud. The policy is cluster scoped, so it has to be composed here: a namespaced XR can't compose a cluster-scoped resource, and the cloud cluster XRs (EKSCluster and the rest) are all namespaced. A managed resource composed before its definition's CRD exists fails the whole reconcile, because the API server rejects a custom resource whose CRD doesn't exist. So the function holds off composing the cloud cluster XR until the policy it composed reports Healthy, read back from the observed policy so no definition (each of which carries a full CRD schema) has to be pulled into the request. Healthy means the policy set its definitions Active rather than that their CRDs exist, so a managed resource composed in the brief window before a CRD is served fails its apply and succeeds on the next reconcile; a TODO tracks tightening this to an Established condition once the policy reports one. Once the cluster is observed the function keeps composing it regardless, so a later activation blip never drops a provisioned cluster from desired state and deletes it. Activation is one-way, so dropping the cloud kinds from the shipped policy doesn't deactivate them on a running control plane; existing control planes keep their providers running after an upgrade. The shipped policy is now down to provider-helm and provider-kubernetes, which serve every control plane. Fixes #427. Signed-off-by: Nic Cope --- apis/mrap.yaml | 71 ++-------- .../compose-inference-cluster/function/fn.py | 116 ++++++++++++++- .../tests/test_fn.py | 132 ++++++++++++++++++ 3 files changed, 256 insertions(+), 63 deletions(-) diff --git a/apis/mrap.yaml b/apis/mrap.yaml index e2e78ddf..e6ef45b1 100644 --- a/apis/mrap.yaml +++ b/apis/mrap.yaml @@ -1,19 +1,22 @@ # Crossplane converts every provider CRD into an inactive # ManagedResourceDefinition and only creates the CRD once a -# ManagedResourceActivationPolicy activates it. This policy activates just the -# managed resource kinds Modelplane's compositions actually compose, so only -# those ~40 kinds become live CRDs instead of the ~900 the provider families -# define between them. +# ManagedResourceActivationPolicy activates it. A provider with the safe-start +# capability, like the upbound provider families, scales its runtime to zero +# until one of its definitions is active, so activating a kind is what runs the +# provider that serves it. +# +# This policy activates only provider-helm and provider-kubernetes, which serve +# every control plane. compose-inference-cluster composes a second, cluster- +# scoped policy activating just the kinds the cloud it provisions needs, so a +# control plane runs a cloud's providers only once it has an InferenceCluster on +# that cloud, rather than all of them from here. # # It ships in the Configuration package, so it installs with the Modelplane # APIs. For it to take effect, install Crossplane with an empty default # activation (provider.defaultActivations=[]); see the installation guide. Its -# default is "*", which activates every CRD and makes this policy a no-op. -# -# Keep this in sync with the compositions. A managed resource kind a -# composition composes but this policy omits stays inactive, and composing it -# fails because its CRD does not exist. Activation is one-way: removing an entry -# here does not deactivate a CRD on a running control plane. +# default is "*", which activates every CRD and makes on-demand activation a +# no-op. Activation is one-way: removing an entry from a policy does not +# deactivate a CRD on a running control plane. apiVersion: apiextensions.crossplane.io/v1alpha1 kind: ManagedResourceActivationPolicy metadata: @@ -23,51 +26,3 @@ spec: # provider-helm and provider-kubernetes (serving stack, gateway, caches). - releases.helm.m.crossplane.io - objects.kubernetes.m.crossplane.io - # AWS (EKS clusters). - - eips.ec2.aws.m.upbound.io - - internetgateways.ec2.aws.m.upbound.io - - launchtemplates.ec2.aws.m.upbound.io - - natgateways.ec2.aws.m.upbound.io - - routes.ec2.aws.m.upbound.io - - routetables.ec2.aws.m.upbound.io - - routetableassociations.ec2.aws.m.upbound.io - - securitygroups.ec2.aws.m.upbound.io - - securitygroupegressrules.ec2.aws.m.upbound.io - - securitygroupingressrules.ec2.aws.m.upbound.io - - subnets.ec2.aws.m.upbound.io - - vpcs.ec2.aws.m.upbound.io - - filesystems.efs.aws.m.upbound.io - - mounttargets.efs.aws.m.upbound.io - - addons.eks.aws.m.upbound.io - - clusters.eks.aws.m.upbound.io - - clusterauths.eks.aws.m.upbound.io - - nodegroups.eks.aws.m.upbound.io - - podidentityassociations.eks.aws.m.upbound.io - - policies.iam.aws.m.upbound.io - - roles.iam.aws.m.upbound.io - - rolepolicyattachments.iam.aws.m.upbound.io - # GCP (GKE clusters). - - projectiammembers.cloudplatform.gcp.m.upbound.io - - projectservices.cloudplatform.gcp.m.upbound.io - - serviceaccounts.cloudplatform.gcp.m.upbound.io - - serviceaccountkeys.cloudplatform.gcp.m.upbound.io - - networks.compute.gcp.m.upbound.io - - subnetworks.compute.gcp.m.upbound.io - - clusters.container.gcp.m.upbound.io - - nodepools.container.gcp.m.upbound.io - # Azure (AKS clusters). - - kubernetesclusters.containerservice.azure.m.upbound.io - - kubernetesclusternodepools.containerservice.azure.m.upbound.io - - subnets.network.azure.m.upbound.io - - virtualnetworks.network.azure.m.upbound.io - - resourcegroups.azure.m.upbound.io - # Nebius (managed Kubernetes clusters). - - filesystems.compute.nebius.m.upbound.io - - gpuclusters.compute.nebius.m.upbound.io - - clusters.mk8s.nebius.m.upbound.io - - nodegroups.mk8s.nebius.m.upbound.io - - networks.vpc.nebius.m.upbound.io - - subnets.vpc.nebius.m.upbound.io - # Vultr (VKE clusters). - - kubernetes.vke.vultr.m.upbound.io - - kubernetesnodepools.vke.vultr.m.upbound.io diff --git a/functions/compose-inference-cluster/function/fn.py b/functions/compose-inference-cluster/function/fn.py index a3ef7788..4b478492 100644 --- a/functions/compose-inference-cluster/function/fn.py +++ b/functions/compose-inference-cluster/function/fn.py @@ -42,6 +42,9 @@ from models.ai.modelplane.infrastructure.nebiuscluster import v1alpha1 as nebiusv1alpha1 from models.ai.modelplane.infrastructure.servingstack import v1alpha1 as ssv1alpha1 from models.ai.modelplane.infrastructure.vultrcluster import v1alpha1 as vultrv1alpha1 +from models.io.crossplane.apiextensions.managedresourceactivationpolicy import ( + v1alpha1 as mrapv1alpha1, +) from models.io.crossplane.m.kubernetes.clusterproviderconfig import ( v1alpha1 as k8scpcv1alpha1, ) @@ -100,6 +103,68 @@ # Identity type for Nebius service account credentials. _IDENTITY_TYPE_NEBIUS = "NebiusServiceAccountCredentials" +# The managed resource kinds each cloud's cluster XR composes, and so the +# ManagedResourceDefinitions its activation policy activates. Only this +# cluster-scoped XR can compose the (cluster-scoped) policy; the namespaced +# cluster XRs it composes cannot. Keep each list in sync with the resources +# compose--cluster composes - a kind composed but missing here never +# gets a CRD, and composing it fails the whole reconcile. provider-helm and +# provider-kubernetes are omitted: the Configuration's own policy keeps those +# active for every control plane. +_ACTIVATE_AWS = ( + "eips.ec2.aws.m.upbound.io", + "internetgateways.ec2.aws.m.upbound.io", + "launchtemplates.ec2.aws.m.upbound.io", + "natgateways.ec2.aws.m.upbound.io", + "routes.ec2.aws.m.upbound.io", + "routetables.ec2.aws.m.upbound.io", + "routetableassociations.ec2.aws.m.upbound.io", + "securitygroups.ec2.aws.m.upbound.io", + "securitygroupegressrules.ec2.aws.m.upbound.io", + "securitygroupingressrules.ec2.aws.m.upbound.io", + "subnets.ec2.aws.m.upbound.io", + "vpcs.ec2.aws.m.upbound.io", + "filesystems.efs.aws.m.upbound.io", + "mounttargets.efs.aws.m.upbound.io", + "addons.eks.aws.m.upbound.io", + "clusters.eks.aws.m.upbound.io", + "clusterauths.eks.aws.m.upbound.io", + "nodegroups.eks.aws.m.upbound.io", + "podidentityassociations.eks.aws.m.upbound.io", + "policies.iam.aws.m.upbound.io", + "roles.iam.aws.m.upbound.io", + "rolepolicyattachments.iam.aws.m.upbound.io", +) +_ACTIVATE_GCP = ( + "projectiammembers.cloudplatform.gcp.m.upbound.io", + "projectservices.cloudplatform.gcp.m.upbound.io", + "serviceaccounts.cloudplatform.gcp.m.upbound.io", + "serviceaccountkeys.cloudplatform.gcp.m.upbound.io", + "networks.compute.gcp.m.upbound.io", + "subnetworks.compute.gcp.m.upbound.io", + "clusters.container.gcp.m.upbound.io", + "nodepools.container.gcp.m.upbound.io", +) +_ACTIVATE_AZURE = ( + "kubernetesclusters.containerservice.azure.m.upbound.io", + "kubernetesclusternodepools.containerservice.azure.m.upbound.io", + "subnets.network.azure.m.upbound.io", + "virtualnetworks.network.azure.m.upbound.io", + "resourcegroups.azure.m.upbound.io", +) +_ACTIVATE_NEBIUS = ( + "filesystems.compute.nebius.m.upbound.io", + "gpuclusters.compute.nebius.m.upbound.io", + "clusters.mk8s.nebius.m.upbound.io", + "nodegroups.mk8s.nebius.m.upbound.io", + "networks.vpc.nebius.m.upbound.io", + "subnets.vpc.nebius.m.upbound.io", +) +_ACTIVATE_VULTR = ( + "kubernetes.vke.vultr.m.upbound.io", + "kubernetesnodepools.vke.vultr.m.upbound.io", +) + def _name(meta: metav1.ObjectMeta | None) -> str: """The object's name, always set on resources read from the API server.""" @@ -170,6 +235,37 @@ def compose(self) -> None: else: response.warning(self.rsp, f"unsupported cluster source: {source}") + def compose_activation(self, kinds: tuple[str, ...]) -> None: + """Activate the cloud managed resource kinds the cluster XR composes. + + The policy is cluster scoped, so only this cluster-scoped XR can compose + it; the namespaced cluster XR it composes cannot. _activation_ready then + gates the cluster XR on the policy taking effect, so its managed + resources aren't composed before the API server knows their kinds. + """ + resource.update( + self.rsp.desired.resources["activation"], + mrapv1alpha1.ManagedResourceActivationPolicy( + spec=mrapv1alpha1.Spec(activate=list(kinds)), + ), + ) + self.rsp.desired.resources["activation"].ready = fnv1.READY_TRUE + + def _activation_ready(self) -> bool: + """Whether the activation policy this function composed has taken effect. + + Read from the policy's own Healthy condition, so no ManagedResource- + Definition (each of which carries a full CRD schema) has to be pulled + into the request. + + TODO(negz): gate on an Established condition instead once the policy + reports one. Healthy means the policy set its definitions Active, not + that their CRDs exist, so a managed resource composed in the window + before a CRD is served can still fail its apply until the next reconcile. + """ + activation = self.req.observed.resources.get("activation") + return resource.get_condition(activation, "Healthy").status == "True" + def compose_replica_guard(self) -> None: """Block deletion of the InferenceCluster while ModelReplicas use it. @@ -267,7 +363,9 @@ def compose_gke(self, gke: v1alpha1.Gke | None) -> None: response.warning(self.rsp, "GKE configuration is required when source is GKE") return - self.compose_gke_cluster(gke) + self.compose_activation(_ACTIVATE_GCP) + if self._activation_ready() or "gke-cluster" in self.req.observed.resources: + self.compose_gke_cluster(gke) gke_ready = resource.get_condition(self.req.observed.resources.get("gke-cluster"), "Ready").status == "True" kubeconfig_secret = self.observed_gke_secret(_SECRET_TYPE_KUBECONFIG) @@ -311,7 +409,9 @@ def compose_eks(self, eks: v1alpha1.Eks | None) -> None: response.warning(self.rsp, "EKS configuration is required when source is EKS") return - self.compose_eks_cluster(eks) + self.compose_activation(_ACTIVATE_AWS) + if self._activation_ready() or "eks-cluster" in self.req.observed.resources: + self.compose_eks_cluster(eks) eks_ready = resource.get_condition(self.req.observed.resources.get("eks-cluster"), "Ready").status == "True" kubeconfig = self.observed_eks_secret(_SECRET_TYPE_KUBECONFIG) @@ -347,7 +447,9 @@ def compose_aks(self, aks: v1alpha1.Aks | None) -> None: response.warning(self.rsp, "AKS configuration is required when source is AKS") return - self.compose_aks_cluster(aks) + self.compose_activation(_ACTIVATE_AZURE) + if self._activation_ready() or "aks-cluster" in self.req.observed.resources: + self.compose_aks_cluster(aks) aks_ready = resource.get_condition(self.req.observed.resources.get("aks-cluster"), "Ready").status == "True" kubeconfig = self.observed_aks_secret(_SECRET_TYPE_KUBECONFIG) @@ -385,7 +487,9 @@ def compose_nebius(self, nebius: v1alpha1.Nebius | None) -> None: response.warning(self.rsp, "Nebius configuration is required when source is Nebius") return - self.compose_nebius_cluster(nebius) + self.compose_activation(_ACTIVATE_NEBIUS) + if self._activation_ready() or "nebius-cluster" in self.req.observed.resources: + self.compose_nebius_cluster(nebius) nebius_ready = ( resource.get_condition(self.req.observed.resources.get("nebius-cluster"), "Ready").status == "True" @@ -429,7 +533,9 @@ def compose_vultr(self, vultr: v1alpha1.Vultr | None) -> None: response.warning(self.rsp, "Vultr configuration is required when source is Vultr") return - self.compose_vultr_cluster(vultr) + self.compose_activation(_ACTIVATE_VULTR) + if self._activation_ready() or "vultr-cluster" in self.req.observed.resources: + self.compose_vultr_cluster(vultr) vultr_ready = resource.get_condition(self.req.observed.resources.get("vultr-cluster"), "Ready").status == "True" kubeconfig = self.observed_vultr_secret(_SECRET_TYPE_KUBECONFIG) diff --git a/functions/compose-inference-cluster/tests/test_fn.py b/functions/compose-inference-cluster/tests/test_fn.py index 79b2ac5a..2d2bc492 100644 --- a/functions/compose-inference-cluster/tests/test_fn.py +++ b/functions/compose-inference-cluster/tests/test_fn.py @@ -14,6 +14,7 @@ """Tests for the compose-inference-cluster function.""" +import copy import dataclasses import unittest @@ -40,6 +41,97 @@ def setUpModule() -> None: logging.configure(level=logging.Level.DISABLED) +_ACTIVATION_API_VERSION = "apiextensions.crossplane.io/v1alpha1" + +# The managed resource kinds each cloud's cluster activates. Independent copies of +# the function's own lists, so the test fails if the two drift. +_ACTIVATE_GCP = ( + "projectiammembers.cloudplatform.gcp.m.upbound.io", + "projectservices.cloudplatform.gcp.m.upbound.io", + "serviceaccounts.cloudplatform.gcp.m.upbound.io", + "serviceaccountkeys.cloudplatform.gcp.m.upbound.io", + "networks.compute.gcp.m.upbound.io", + "subnetworks.compute.gcp.m.upbound.io", + "clusters.container.gcp.m.upbound.io", + "nodepools.container.gcp.m.upbound.io", +) +_ACTIVATE_AWS = ( + "eips.ec2.aws.m.upbound.io", + "internetgateways.ec2.aws.m.upbound.io", + "launchtemplates.ec2.aws.m.upbound.io", + "natgateways.ec2.aws.m.upbound.io", + "routes.ec2.aws.m.upbound.io", + "routetables.ec2.aws.m.upbound.io", + "routetableassociations.ec2.aws.m.upbound.io", + "securitygroups.ec2.aws.m.upbound.io", + "securitygroupegressrules.ec2.aws.m.upbound.io", + "securitygroupingressrules.ec2.aws.m.upbound.io", + "subnets.ec2.aws.m.upbound.io", + "vpcs.ec2.aws.m.upbound.io", + "filesystems.efs.aws.m.upbound.io", + "mounttargets.efs.aws.m.upbound.io", + "addons.eks.aws.m.upbound.io", + "clusters.eks.aws.m.upbound.io", + "clusterauths.eks.aws.m.upbound.io", + "nodegroups.eks.aws.m.upbound.io", + "podidentityassociations.eks.aws.m.upbound.io", + "policies.iam.aws.m.upbound.io", + "roles.iam.aws.m.upbound.io", + "rolepolicyattachments.iam.aws.m.upbound.io", +) +_ACTIVATE_AZURE = ( + "kubernetesclusters.containerservice.azure.m.upbound.io", + "kubernetesclusternodepools.containerservice.azure.m.upbound.io", + "subnets.network.azure.m.upbound.io", + "virtualnetworks.network.azure.m.upbound.io", + "resourcegroups.azure.m.upbound.io", +) +_ACTIVATE_NEBIUS = ( + "filesystems.compute.nebius.m.upbound.io", + "gpuclusters.compute.nebius.m.upbound.io", + "clusters.mk8s.nebius.m.upbound.io", + "nodegroups.mk8s.nebius.m.upbound.io", + "networks.vpc.nebius.m.upbound.io", + "subnets.vpc.nebius.m.upbound.io", +) +_ACTIVATE_VULTR = ( + "kubernetes.vke.vultr.m.upbound.io", + "kubernetesnodepools.vke.vultr.m.upbound.io", +) + + +def _observe_healthy(req: fnv1.RunFunctionRequest) -> None: + """Observe the composed activation policy reporting Healthy, so the function + composes the cluster XR rather than waiting for activation.""" + req.observed.resources["activation"].CopyFrom( + fnv1.Resource( + resource=resource.dict_to_struct( + { + "apiVersion": _ACTIVATION_API_VERSION, + "kind": "ManagedResourceActivationPolicy", + "status": {"conditions": [{"type": "Healthy", "status": "True"}]}, + }, + ), + ), + ) + + +def _want_activation(want: fnv1.RunFunctionResponse, kinds: tuple[str, ...]) -> None: + """Add the activation policy the function composes for a cloud cluster.""" + want.desired.resources["activation"].CopyFrom( + fnv1.Resource( + resource=resource.dict_to_struct( + { + "apiVersion": _ACTIVATION_API_VERSION, + "kind": "ManagedResourceActivationPolicy", + "spec": {"activate": list(kinds)}, + }, + ), + ready=fnv1.READY_TRUE, + ), + ) + + def _eks_ready_extras(want: fnv1.RunFunctionResponse, storage_class: str) -> None: """Apply the EKS-ready deltas on top of the EKS first-pass response: mark the EKSCluster ready and relay the backing cluster's status.cache up to the @@ -2675,6 +2767,42 @@ async def test_compose(self) -> None: # noqa: PLR0915 want_creds.requirements.resources["class-gpu-l4"].CopyFrom(class_selector) want_creds.requirements.resources["model-replicas"].CopyFrom(_replicas_selector("test-cluster")) + # Every cloud cluster composes an activation policy; with the policy + # observed Healthy the cluster XR is composed. + for req, want, kinds in [ + (req2, want2, _ACTIVATE_GCP), + (req_creds, want_creds, _ACTIVATE_GCP), + (req4, want4, _ACTIVATE_AWS), + (req5, want5, _ACTIVATE_AWS), + (req6, want6, _ACTIVATE_GCP), + (req7, want7, _ACTIVATE_AWS), + (req8, want8, _ACTIVATE_AWS), + (req9, want9, _ACTIVATE_AWS), + (req10, want10, _ACTIVATE_NEBIUS), + (req11, want11, _ACTIVATE_NEBIUS), + (req12, want12, _ACTIVATE_AZURE), + (req13, want13, _ACTIVATE_AZURE), + (req14, want14, _ACTIVATE_VULTR), + (req_creds_vultr, want_creds_vultr, _ACTIVATE_VULTR), + (req15, want15, _ACTIVATE_VULTR), + ]: + _observe_healthy(req) + _want_activation(want, kinds) + + # Before the policy reports Healthy, and with no cluster observed, the + # function composes only the activation policy, not the cluster XR. + req_unactivated = copy.deepcopy(req2) + del req_unactivated.observed.resources["activation"] + want_unactivated = copy.deepcopy(want2) + del want_unactivated.desired.resources["gke-cluster"] + + # Once the cluster is observed, the function keeps composing it even + # when the policy momentarily stops reporting Healthy, so an activation + # blip never drops a provisioned cluster from desired state. + req_blip = copy.deepcopy(req6) + del req_blip.observed.resources["activation"] + want_blip = copy.deepcopy(want6) + cases = [ Case(name="existing cluster with secrets composes backend and CPC", req=req1, want=want1), Case(name="existing cluster with a non-GCP identity threads the identity type", req=req1b, want=want1b), @@ -2707,6 +2835,10 @@ async def test_compose(self) -> None: # noqa: PLR0915 req=req13, want=want13, ), + Case( + name="cloud cluster not activated composes only the policy", req=req_unactivated, want=want_unactivated + ), + Case(name="observed cluster keeps composing through an activation blip", req=req_blip, want=want_blip), Case(name="Vultr cluster first pass composes VultrCluster XR only", req=req14, want=want14), Case( name="Vultr credentials pass through to VultrCluster spec", From e5e6b8831225c5b716f66cc4291d11ad3fbebca2 Mon Sep 17 00:00:00 2001 From: Nic Cope Date: Fri, 4 Sep 2026 13:16:17 -0700 Subject: [PATCH 2/2] Bump Crossplane to v2.4.0 and drop the e2e scale-to-zero workaround The project pinned Crossplane 2.3.4, which activates every managed resource kind and runs every provider. Safe-start providers only scale their runtime to zero once their definitions are inactive from v2.4.0, so the e2e faked that behaviour: a scale-to-zero DeploymentRuntimeConfig and an ImageConfig that forced every cloud provider Deployment to zero replicas, since otherwise the BYO scenario would run a dozen idle cloud controllers in kind. Pinning both the local dev control plane and the e2e to v2.4.0 makes that native. The lean control plane's narrowed MRAP already activates only the provider-helm and provider-kubernetes kinds the BYO compositions use, so the cloud providers' definitions stay inactive and Crossplane scales their controllers to zero on its own. The DeploymentRuntimeConfig and ImageConfig go. Towards #427. Signed-off-by: Nic Cope --- e2e/README.md | 14 ++++----- e2e/lean-control-plane.yaml | 59 +++++-------------------------------- e2e/run.sh | 11 +++---- nix/apps.nix | 2 +- 4 files changed, 21 insertions(+), 65 deletions(-) diff --git a/e2e/README.md b/e2e/README.md index c7aff122..ff82c5de 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -54,21 +54,19 @@ server exposes both, so the pod goes Ready without a real model or GPU. ### Why cloud provisioning cannot be tested here `lean-control-plane.yaml` trims the control plane to what a BYO cluster needs, -and both trims stop a cloud `InferenceCluster` from reconciling at all. Neither -announces itself, so this is what to expect if you point this control plane at a +which also stops a cloud `InferenceCluster` from reconciling at all. It doesn't +announce itself, so this is what to expect if you point this control plane at a real cloud: - The MRAP activates only `*.kubernetes.m.crossplane.io` and `*.helm.m.crossplane.io`. A cloud provider's managed resources are then never activated, so they sit with **no status conditions at all** — which reads as nothing happening rather than as an error. -- The `dormant-cloud-providers` `ImageConfig` maps every - `xpkg.upbound.io/upbound/provider-*` to a zero-replica runtime config. Editing - that `ImageConfig` is not enough on its own: it is resolved when a package - revision reconciles, so existing Deployments keep their replica count until - something scales them. +- Because those providers declare the safe-start capability, Crossplane scales + their controllers to zero while their managed resources are inactive, so a + dormant cloud provider runs no pod at all. -Undoing both is possible but leaves a control plane that is no longer the one CI +Undoing this is possible but leaves a control plane that is no longer the one CI runs, so prefer a separate control plane for cloud work. ## Prerequisites diff --git a/e2e/lean-control-plane.yaml b/e2e/lean-control-plane.yaml index 678cc826..d02bea4a 100644 --- a/e2e/lean-control-plane.yaml +++ b/e2e/lean-control-plane.yaml @@ -6,57 +6,14 @@ # run.sh feeds this via `crossplane project run --init-resources`, which applies # it BEFORE the Configuration's dependencies install, so the cloud providers # never activate their managed resources or run their controllers. - -# 1. Scale the unused cloud provider controllers to zero. Every provider - -# including provider-kubernetes and provider-helm, the two this scenario -# uses - now lives under xpkg.upbound.io/upbound/provider-*, so an org-wide -# prefix would scale those two to zero as well (an ImageConfig runtime -# overrides even an explicit runtimeConfigRef on the Provider). Match the -# cloud provider families explicitly instead. A new cloud provider must be -# added here to stay dormant; forgetting one costs an idle controller pod in -# kind, not a broken run. ImageConfig applies the runtime config to matching -# packages including ones installed as dependencies - that's what catches -# the family providers, which install as dependencies of the cloud -# providers (Crossplane docs: packages/image-configs). -apiVersion: pkg.crossplane.io/v1beta1 -kind: DeploymentRuntimeConfig -metadata: - name: scale-to-zero -spec: - deploymentTemplate: - spec: - selector: {} - replicas: 0 - template: {} ---- -apiVersion: pkg.crossplane.io/v1beta1 -kind: ImageConfig -metadata: - name: dormant-cloud-providers -spec: - matchImages: - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-aws- - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-azure- - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-gcp- - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-family- - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-nebius - - type: Prefix - prefix: xpkg.upbound.io/upbound/provider-vultr - runtime: - configRef: - name: scale-to-zero ---- -# 2. Replace the Helm chart's default catch-all MRAP (activate: ["*"]) with one -# that activates only the managed resources the BYO compositions use, so the -# cloud providers' MRs stay dormant (fewer CRDs). The compositions run -# provider-helm/kubernetes v2, whose MRs are namespaced (.m.crossplane.io) — -# the same forms the shipped apis/mrap.yaml activates — so the cluster-scoped -# variants aren't needed here. +# +# Replace the Helm chart's default catch-all MRAP (activate: ["*"]) with one +# that activates only the managed resources the BYO compositions use. The cloud +# providers' managed resources stay inactive, and since those providers declare +# the safe-start capability, Crossplane scales their controllers to zero until +# something activates one. The compositions run provider-helm/kubernetes v2, +# whose MRs are namespaced (.m.crossplane.io) - the same forms the shipped +# apis/mrap.yaml activates - so the cluster-scoped variants aren't needed here. apiVersion: apiextensions.crossplane.io/v1alpha1 kind: ManagedResourceActivationPolicy metadata: diff --git a/e2e/run.sh b/e2e/run.sh index 2f6fb954..5945e98c 100644 --- a/e2e/run.sh +++ b/e2e/run.sh @@ -136,14 +136,15 @@ esac log "Building + running the control plane" cd "$ROOT" -# Install the config with the lean control-plane trims (narrowed MRAP + scale-to-0) -# applied before the providers. prerequisites.yaml is applied afterwards with -# kubectl, not through --init-resources: it opens with a comment-only YAML -# document that `crossplane project run` rejects but kubectl skips. +# Install the config with the lean control-plane's narrowed MRAP applied before +# the providers, so the cloud providers stay dormant (safe-start scales them to +# zero). prerequisites.yaml is applied afterwards with kubectl, not through +# --init-resources: it opens with a comment-only YAML document that `crossplane +# project run` rejects but kubectl skips. crossplane project run \ --control-plane-name "$CP" --cluster-admin --timeout 25m \ --init-resources "$ROOT/e2e/lean-control-plane.yaml" \ - --crossplane-version=2.3.4 + --crossplane-version=2.4.0 # Config healthy. Finish the setup the getting-started flow does by hand (as the # nix run app now does too, PR #375): apply the RBAC prerequisites, then point diff --git a/nix/apps.nix b/nix/apps.nix index d13696d8..92c4cf41 100644 --- a/nix/apps.nix +++ b/nix/apps.nix @@ -151,7 +151,7 @@ # Pin Crossplane to the version e2e/run.sh uses: without a pin the # CLI installs the latest release - version_args=(--crossplane-version=2.3.4) + version_args=(--crossplane-version=2.4.0) for arg in "$@"; do case "$arg" in --crossplane-version | --crossplane-version=*) version_args=() ;;