Activate cloud managed resources on demand from InferenceCluster

apis/mrap.yaml

@@ -1,19 +1,22 @@
 # Crossplane converts every provider CRD into an inactive
 # ManagedResourceDefinition and only creates the CRD once a
-# ManagedResourceActivationPolicy activates it. This policy activates just the
-# managed resource kinds Modelplane's compositions actually compose, so only
-# those ~40 kinds become live CRDs instead of the ~900 the provider families
-# define between them.
+# ManagedResourceActivationPolicy activates it. A provider with the safe-start
+# capability, like the upbound provider families, scales its runtime to zero
+# until one of its definitions is active, so activating a kind is what runs the
+# provider that serves it.
+#
+# This policy activates only provider-helm and provider-kubernetes, which serve
+# every control plane. compose-inference-cluster composes a second, cluster-
+# scoped policy activating just the kinds the cloud it provisions needs, so a
+# control plane runs a cloud's providers only once it has an InferenceCluster on
+# that cloud, rather than all of them from here.
 #
 # It ships in the Configuration package, so it installs with the Modelplane
 # APIs. For it to take effect, install Crossplane with an empty default
 # activation (provider.defaultActivations=[]); see the installation guide. Its
-# default is "*", which activates every CRD and makes this policy a no-op.
-#
-# Keep this in sync with the compositions. A managed resource kind a
-# composition composes but this policy omits stays inactive, and composing it
-# fails because its CRD does not exist. Activation is one-way: removing an entry
-# here does not deactivate a CRD on a running control plane.
+# default is "*", which activates every CRD and makes on-demand activation a
+# no-op. Activation is one-way: removing an entry from a policy does not
+# deactivate a CRD on a running control plane.
 apiVersion: apiextensions.crossplane.io/v1alpha1
 kind: ManagedResourceActivationPolicy
 metadata:
@@ -23,51 +26,3 @@ spec:
   # provider-helm and provider-kubernetes (serving stack, gateway, caches).
   - releases.helm.m.crossplane.io
   - objects.kubernetes.m.crossplane.io
-  # AWS (EKS clusters).
-  - eips.ec2.aws.m.upbound.io
-  - internetgateways.ec2.aws.m.upbound.io
-  - launchtemplates.ec2.aws.m.upbound.io
-  - natgateways.ec2.aws.m.upbound.io
-  - routes.ec2.aws.m.upbound.io
-  - routetables.ec2.aws.m.upbound.io
-  - routetableassociations.ec2.aws.m.upbound.io
-  - securitygroups.ec2.aws.m.upbound.io
-  - securitygroupegressrules.ec2.aws.m.upbound.io
-  - securitygroupingressrules.ec2.aws.m.upbound.io
-  - subnets.ec2.aws.m.upbound.io
-  - vpcs.ec2.aws.m.upbound.io
-  - filesystems.efs.aws.m.upbound.io
-  - mounttargets.efs.aws.m.upbound.io
-  - addons.eks.aws.m.upbound.io
-  - clusters.eks.aws.m.upbound.io
-  - clusterauths.eks.aws.m.upbound.io
-  - nodegroups.eks.aws.m.upbound.io
-  - podidentityassociations.eks.aws.m.upbound.io
-  - policies.iam.aws.m.upbound.io
-  - roles.iam.aws.m.upbound.io
-  - rolepolicyattachments.iam.aws.m.upbound.io
-  # GCP (GKE clusters).
-  - projectiammembers.cloudplatform.gcp.m.upbound.io
-  - projectservices.cloudplatform.gcp.m.upbound.io
-  - serviceaccounts.cloudplatform.gcp.m.upbound.io
-  - serviceaccountkeys.cloudplatform.gcp.m.upbound.io
-  - networks.compute.gcp.m.upbound.io
-  - subnetworks.compute.gcp.m.upbound.io
-  - clusters.container.gcp.m.upbound.io
-  - nodepools.container.gcp.m.upbound.io
-  # Azure (AKS clusters).
-  - kubernetesclusters.containerservice.azure.m.upbound.io
-  - kubernetesclusternodepools.containerservice.azure.m.upbound.io
-  - subnets.network.azure.m.upbound.io
-  - virtualnetworks.network.azure.m.upbound.io
-  - resourcegroups.azure.m.upbound.io
-  # Nebius (managed Kubernetes clusters).
-  - filesystems.compute.nebius.m.upbound.io
-  - gpuclusters.compute.nebius.m.upbound.io
-  - clusters.mk8s.nebius.m.upbound.io
-  - nodegroups.mk8s.nebius.m.upbound.io
-  - networks.vpc.nebius.m.upbound.io
-  - subnets.vpc.nebius.m.upbound.io
-  # Vultr (VKE clusters).
-  - kubernetes.vke.vultr.m.upbound.io
-  - kubernetesnodepools.vke.vultr.m.upbound.io

e2e/README.md

@@ -54,21 +54,19 @@ server exposes both, so the pod goes Ready without a real model or GPU.
 ### Why cloud provisioning cannot be tested here
 
 `lean-control-plane.yaml` trims the control plane to what a BYO cluster needs,
-and both trims stop a cloud `InferenceCluster` from reconciling at all. Neither
-announces itself, so this is what to expect if you point this control plane at a
+which also stops a cloud `InferenceCluster` from reconciling at all. It doesn't
+announce itself, so this is what to expect if you point this control plane at a
 real cloud:
 
 - The MRAP activates only `*.kubernetes.m.crossplane.io` and
   `*.helm.m.crossplane.io`. A cloud provider's managed resources are then never
   activated, so they sit with **no status conditions at all** — which reads as
   nothing happening rather than as an error.
-- The `dormant-cloud-providers` `ImageConfig` maps every
-  `xpkg.upbound.io/upbound/provider-*` to a zero-replica runtime config. Editing
-  that `ImageConfig` is not enough on its own: it is resolved when a package
-  revision reconciles, so existing Deployments keep their replica count until
-  something scales them.
+- Because those providers declare the safe-start capability, Crossplane scales
+  their controllers to zero while their managed resources are inactive, so a
+  dormant cloud provider runs no pod at all.
 
-Undoing both is possible but leaves a control plane that is no longer the one CI
+Undoing this is possible but leaves a control plane that is no longer the one CI
 runs, so prefer a separate control plane for cloud work.
 
 ## Prerequisites

e2e/lean-control-plane.yaml

@@ -6,57 +6,14 @@
 # run.sh feeds this via `crossplane project run --init-resources`, which applies
 # it BEFORE the Configuration's dependencies install, so the cloud providers
 # never activate their managed resources or run their controllers.
-
-# 1. Scale the unused cloud provider controllers to zero. Every provider -
-#    including provider-kubernetes and provider-helm, the two this scenario
-#    uses - now lives under xpkg.upbound.io/upbound/provider-*, so an org-wide
-#    prefix would scale those two to zero as well (an ImageConfig runtime
-#    overrides even an explicit runtimeConfigRef on the Provider). Match the
-#    cloud provider families explicitly instead. A new cloud provider must be
-#    added here to stay dormant; forgetting one costs an idle controller pod in
-#    kind, not a broken run. ImageConfig applies the runtime config to matching
-#    packages including ones installed as dependencies - that's what catches
-#    the family providers, which install as dependencies of the cloud
-#    providers (Crossplane docs: packages/image-configs).
-apiVersion: pkg.crossplane.io/v1beta1
-kind: DeploymentRuntimeConfig
-metadata:
-  name: scale-to-zero
-spec:
-  deploymentTemplate:
-    spec:
-      selector: {}
-      replicas: 0
-      template: {}
----
-apiVersion: pkg.crossplane.io/v1beta1
-kind: ImageConfig
-metadata:
-  name: dormant-cloud-providers
-spec:
-  matchImages:
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-aws-
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-azure-
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-gcp-
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-family-
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-nebius
-  - type: Prefix
-    prefix: xpkg.upbound.io/upbound/provider-vultr
-  runtime:
-    configRef:
-      name: scale-to-zero
----
-# 2. Replace the Helm chart's default catch-all MRAP (activate: ["*"]) with one
-#    that activates only the managed resources the BYO compositions use, so the
-#    cloud providers' MRs stay dormant (fewer CRDs). The compositions run
-#    provider-helm/kubernetes v2, whose MRs are namespaced (.m.crossplane.io) —
-#    the same forms the shipped apis/mrap.yaml activates — so the cluster-scoped
-#    variants aren't needed here.
+#
+# Replace the Helm chart's default catch-all MRAP (activate: ["*"]) with one
+# that activates only the managed resources the BYO compositions use. The cloud
+# providers' managed resources stay inactive, and since those providers declare
+# the safe-start capability, Crossplane scales their controllers to zero until
+# something activates one. The compositions run provider-helm/kubernetes v2,
+# whose MRs are namespaced (.m.crossplane.io) - the same forms the shipped
+# apis/mrap.yaml activates - so the cluster-scoped variants aren't needed here.
 apiVersion: apiextensions.crossplane.io/v1alpha1
 kind: ManagedResourceActivationPolicy
 metadata:

e2e/run.sh

@@ -136,14 +136,15 @@ esac
 
 log "Building + running the control plane"
 cd "$ROOT"
-# Install the config with the lean control-plane trims (narrowed MRAP + scale-to-0)
-# applied before the providers. prerequisites.yaml is applied afterwards with
-# kubectl, not through --init-resources: it opens with a comment-only YAML
-# document that `crossplane project run` rejects but kubectl skips.
+# Install the config with the lean control-plane's narrowed MRAP applied before
+# the providers, so the cloud providers stay dormant (safe-start scales them to
+# zero). prerequisites.yaml is applied afterwards with kubectl, not through
+# --init-resources: it opens with a comment-only YAML document that `crossplane
+# project run` rejects but kubectl skips.
 crossplane project run \
 	--control-plane-name "$CP" --cluster-admin --timeout 25m \
 	--init-resources "$ROOT/e2e/lean-control-plane.yaml" \
-	--crossplane-version=2.3.4
+	--crossplane-version=2.4.0
 
 # Config healthy. Finish the setup the getting-started flow does by hand (as the
 # nix run app now does too, PR #375): apply the RBAC prerequisites, then point

functions/compose-inference-cluster/function/fn.py

@@ -42,6 +42,9 @@
 from models.ai.modelplane.infrastructure.nebiuscluster import v1alpha1 as nebiusv1alpha1
 from models.ai.modelplane.infrastructure.servingstack import v1alpha1 as ssv1alpha1
 from models.ai.modelplane.infrastructure.vultrcluster import v1alpha1 as vultrv1alpha1
+from models.io.crossplane.apiextensions.managedresourceactivationpolicy import (
+    v1alpha1 as mrapv1alpha1,
+)
 from models.io.crossplane.m.kubernetes.clusterproviderconfig import (
     v1alpha1 as k8scpcv1alpha1,
 )
@@ -100,6 +103,68 @@
 # Identity type for Nebius service account credentials.
 _IDENTITY_TYPE_NEBIUS = "NebiusServiceAccountCredentials"
 
+# The managed resource kinds each cloud's cluster XR composes, and so the
+# ManagedResourceDefinitions its activation policy activates. Only this
+# cluster-scoped XR can compose the (cluster-scoped) policy; the namespaced
+# cluster XRs it composes cannot. Keep each list in sync with the resources
+# compose-<cloud>-cluster composes - a kind composed but missing here never
+# gets a CRD, and composing it fails the whole reconcile. provider-helm and
+# provider-kubernetes are omitted: the Configuration's own policy keeps those
+# active for every control plane.
+_ACTIVATE_AWS = (
+    "eips.ec2.aws.m.upbound.io",
+    "internetgateways.ec2.aws.m.upbound.io",
+    "launchtemplates.ec2.aws.m.upbound.io",
+    "natgateways.ec2.aws.m.upbound.io",
+    "routes.ec2.aws.m.upbound.io",
+    "routetables.ec2.aws.m.upbound.io",
+    "routetableassociations.ec2.aws.m.upbound.io",
+    "securitygroups.ec2.aws.m.upbound.io",
+    "securitygroupegressrules.ec2.aws.m.upbound.io",
+    "securitygroupingressrules.ec2.aws.m.upbound.io",
+    "subnets.ec2.aws.m.upbound.io",
+    "vpcs.ec2.aws.m.upbound.io",
+    "filesystems.efs.aws.m.upbound.io",
+    "mounttargets.efs.aws.m.upbound.io",
+    "addons.eks.aws.m.upbound.io",
+    "clusters.eks.aws.m.upbound.io",
+    "clusterauths.eks.aws.m.upbound.io",
+    "nodegroups.eks.aws.m.upbound.io",
+    "podidentityassociations.eks.aws.m.upbound.io",
+    "policies.iam.aws.m.upbound.io",
+    "roles.iam.aws.m.upbound.io",
+    "rolepolicyattachments.iam.aws.m.upbound.io",
+)
+_ACTIVATE_GCP = (
+    "projectiammembers.cloudplatform.gcp.m.upbound.io",
+    "projectservices.cloudplatform.gcp.m.upbound.io",
+    "serviceaccounts.cloudplatform.gcp.m.upbound.io",
+    "serviceaccountkeys.cloudplatform.gcp.m.upbound.io",
+    "networks.compute.gcp.m.upbound.io",
+    "subnetworks.compute.gcp.m.upbound.io",
+    "clusters.container.gcp.m.upbound.io",
+    "nodepools.container.gcp.m.upbound.io",
+)
+_ACTIVATE_AZURE = (
+    "kubernetesclusters.containerservice.azure.m.upbound.io",
+    "kubernetesclusternodepools.containerservice.azure.m.upbound.io",
+    "subnets.network.azure.m.upbound.io",
+    "virtualnetworks.network.azure.m.upbound.io",
+    "resourcegroups.azure.m.upbound.io",
+)
+_ACTIVATE_NEBIUS = (
+    "filesystems.compute.nebius.m.upbound.io",
+    "gpuclusters.compute.nebius.m.upbound.io",
+    "clusters.mk8s.nebius.m.upbound.io",
+    "nodegroups.mk8s.nebius.m.upbound.io",
+    "networks.vpc.nebius.m.upbound.io",
+    "subnets.vpc.nebius.m.upbound.io",
+)
+_ACTIVATE_VULTR = (
+    "kubernetes.vke.vultr.m.upbound.io",
+    "kubernetesnodepools.vke.vultr.m.upbound.io",
+)
+
 
 def _name(meta: metav1.ObjectMeta | None) -> str:
     """The object's name, always set on resources read from the API server."""
@@ -170,6 +235,37 @@ def compose(self) -> None:
         else:
             response.warning(self.rsp, f"unsupported cluster source: {source}")
 
+    def compose_activation(self, kinds: tuple[str, ...]) -> None:
+        """Activate the cloud managed resource kinds the cluster XR composes.
+
+        The policy is cluster scoped, so only this cluster-scoped XR can compose
+        it; the namespaced cluster XR it composes cannot. _activation_ready then
+        gates the cluster XR on the policy taking effect, so its managed
+        resources aren't composed before the API server knows their kinds.
+        """
+        resource.update(
+            self.rsp.desired.resources["activation"],
+            mrapv1alpha1.ManagedResourceActivationPolicy(
+                spec=mrapv1alpha1.Spec(activate=list(kinds)),
+            ),
+        )
+        self.rsp.desired.resources["activation"].ready = fnv1.READY_TRUE
+
+    def _activation_ready(self) -> bool:
+        """Whether the activation policy this function composed has taken effect.
+
+        Read from the policy's own Healthy condition, so no ManagedResource-
+        Definition (each of which carries a full CRD schema) has to be pulled
+        into the request.
+
+        TODO(negz): gate on an Established condition instead once the policy
+        reports one. Healthy means the policy set its definitions Active, not
+        that their CRDs exist, so a managed resource composed in the window
+        before a CRD is served can still fail its apply until the next reconcile.
+        """
+        activation = self.req.observed.resources.get("activation")
+        return resource.get_condition(activation, "Healthy").status == "True"
+
     def compose_replica_guard(self) -> None:
         """Block deletion of the InferenceCluster while ModelReplicas use it.
 
@@ -267,7 +363,9 @@ def compose_gke(self, gke: v1alpha1.Gke | None) -> None:
             response.warning(self.rsp, "GKE configuration is required when source is GKE")
             return
 
-        self.compose_gke_cluster(gke)
+        self.compose_activation(_ACTIVATE_GCP)
+        if self._activation_ready() or "gke-cluster" in self.req.observed.resources:
+            self.compose_gke_cluster(gke)
 
         gke_ready = resource.get_condition(self.req.observed.resources.get("gke-cluster"), "Ready").status == "True"
         kubeconfig_secret = self.observed_gke_secret(_SECRET_TYPE_KUBECONFIG)
@@ -311,7 +409,9 @@ def compose_eks(self, eks: v1alpha1.Eks | None) -> None:
             response.warning(self.rsp, "EKS configuration is required when source is EKS")
             return
 
-        self.compose_eks_cluster(eks)
+        self.compose_activation(_ACTIVATE_AWS)
+        if self._activation_ready() or "eks-cluster" in self.req.observed.resources:
+            self.compose_eks_cluster(eks)
 
         eks_ready = resource.get_condition(self.req.observed.resources.get("eks-cluster"), "Ready").status == "True"
         kubeconfig = self.observed_eks_secret(_SECRET_TYPE_KUBECONFIG)
@@ -347,7 +447,9 @@ def compose_aks(self, aks: v1alpha1.Aks | None) -> None:
             response.warning(self.rsp, "AKS configuration is required when source is AKS")
             return
 
-        self.compose_aks_cluster(aks)
+        self.compose_activation(_ACTIVATE_AZURE)
+        if self._activation_ready() or "aks-cluster" in self.req.observed.resources:
+            self.compose_aks_cluster(aks)
 
         aks_ready = resource.get_condition(self.req.observed.resources.get("aks-cluster"), "Ready").status == "True"
         kubeconfig = self.observed_aks_secret(_SECRET_TYPE_KUBECONFIG)
@@ -385,7 +487,9 @@ def compose_nebius(self, nebius: v1alpha1.Nebius | None) -> None:
             response.warning(self.rsp, "Nebius configuration is required when source is Nebius")
             return
 
-        self.compose_nebius_cluster(nebius)
+        self.compose_activation(_ACTIVATE_NEBIUS)
+        if self._activation_ready() or "nebius-cluster" in self.req.observed.resources:
+            self.compose_nebius_cluster(nebius)
 
         nebius_ready = (
             resource.get_condition(self.req.observed.resources.get("nebius-cluster"), "Ready").status == "True"
@@ -429,7 +533,9 @@ def compose_vultr(self, vultr: v1alpha1.Vultr | None) -> None:
             response.warning(self.rsp, "Vultr configuration is required when source is Vultr")
             return
 
-        self.compose_vultr_cluster(vultr)
+        self.compose_activation(_ACTIVATE_VULTR)
+        if self._activation_ready() or "vultr-cluster" in self.req.observed.resources:
+            self.compose_vultr_cluster(vultr)
 
         vultr_ready = resource.get_condition(self.req.observed.resources.get("vultr-cluster"), "Ready").status == "True"
         kubeconfig = self.observed_vultr_secret(_SECRET_TYPE_KUBECONFIG)