Restrict Flex Node bootstrap token RBAC

.github/workflows/e2e-tests.yml

@@ -9,6 +9,14 @@ name: E2E Tests
 on:
   workflow_dispatch:
     inputs:
+      suite:
+        description: "E2E suite to run"
+        required: false
+        default: all
+        type: choice
+        options:
+          - all
+          - historical-rbac-migration
       skip_cleanup:
         description: "Skip cleanup (keep resources for debugging)"
         required: false
@@ -57,6 +65,7 @@ env:
   AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
   GITHUB_RUN_ID: ${{ github.run_id }}
   E2E_WORK_DIR: /tmp/aks-flex-node-e2e-${{ github.run_id }}
+  E2E_SUITE: ${{ inputs.suite || 'all' }}
 
 jobs:
   e2e:
@@ -95,7 +104,7 @@ jobs:
           E2E_SKIP_CLEANUP: "1"  # Cleanup handled in dedicated step below
         run: |
           set -euo pipefail
-          ./hack/e2e/run.sh all
+          ./hack/e2e/run.sh "${E2E_SUITE}"
 
       - name: Collect logs
         if: always()

docs/usages/aks-flex-config.md

@@ -9,7 +9,7 @@ The helper does not install anything on the target host. It uses Azure CLI and,
 - Azure CLI authenticated to the subscription that contains the AKS cluster.
 - `python3` on the workstation.
 - `kubectl` on the workstation for `setup-node-rbac` and `--bootstrap-token` config generation.
-- Permission to run `az aks get-credentials --admin` and create Kubernetes `ClusterRoleBinding` and bootstrap token `Secret` objects.
+- Permission to run `az aks get-credentials --admin`, create Kubernetes `ClusterRoleBinding` and bootstrap token `Secret` objects, and remove the obsolete `aks-flex-node-role` binding when present.
 
 ## Save The Helper
 
@@ -46,7 +46,42 @@ Run this once per cluster for bootstrap-token joins:
   --subscription "$SUBSCRIPTION_ID"
 ```
 
-This applies the bootstrap-related `ClusterRoleBinding` objects for the `system:bootstrappers:aks-flex-node` group.
+This applies only the CSR creation and approval `ClusterRoleBinding` objects for the `system:bootstrappers:aks-flex-node` group. If any binding still grants that group the obsolete `system:node` role, the command stops after applying the safe bindings and explains how to migrate. It does not silently remove the binding because older and development-mode agents may still use their bootstrap token after joining.
+
+`v0.1.1` introduced a separate daemon client certificate, but the version alone does not prove that certificate was issued successfully. Upgrade every bootstrap-token agent to `v0.1.1` or later (preferably the latest release), and on every host verify that the certificate exists, is unexpired, and the agent remains healthy after a restart:
+
+```bash
+sudo test -s /etc/aks-flex-node/daemon-credentials/daemon-controller-current.pem
+sudo openssl x509 \
+  -in /etc/aks-flex-node/daemon-credentials/daemon-controller-current.pem \
+  -noout -subject -enddate -checkend 0
+sudo systemctl restart aks-flex-node-agent.service
+sudo systemctl is-active aks-flex-node-agent.service
+```
+
+Then explicitly remove the obsolete binding:
+
+```bash
+./aks-flex-config setup-node-rbac \
+  --resource-group "$RESOURCE_GROUP" \
+  --cluster-name "$CLUSTER_NAME" \
+  --subscription "$SUBSCRIPTION_ID" \
+  --remove-legacy-node-role-binding
+```
+
+This migration is idempotent. It automatically deletes only the canonical `aks-flex-node-role` object created by older helpers. If another binding grants the same unsafe edge, or that object has extra subjects, the helper refuses to guess and identifies the objects for manual review. Bootstrap-token config generation refuses to create a token while any such binding exists, rather than either issuing an over-privileged token or unexpectedly breaking an old daemon.
+
+To verify the obsolete binding is gone, run:
+
+```bash
+kubectl get clusterrolebinding aks-flex-node-role
+```
+
+The expected result is `NotFound`. Once certificate issuance has been verified, both the kubelet and long-running Flex daemon use issued client certificates, so removing this binding does not interrupt joined nodes. New and in-progress joins retain the CSR permissions installed above.
+
+Do not roll back a migrated host to an older or development-mode agent that still uses the bootstrap token for ordinary Kubernetes API requests. After this binding is removed, those requests correctly receive `403 Forbidden`. Restore a supported certificate-using agent instead of restoring the broad binding.
+
+Finally, delete bootstrap-token Secrets that are no longer needed. In particular, tokens made by helpers before `v0.1.1` had no expiration. Removing the broad binding limits them to bootstrap permissions, but does not revoke them; do not delete a token that is still being used by an in-progress join.
 
 ## Generate Node Config
 

docs/usages/joining-nodes.md

@@ -16,7 +16,7 @@ Bootstrap token mode is the recommended quickstart path. It uses Kubernetes TLS
 
 High-level flow:
 
-1. Run [`scripts/aks-flex-config setup-node-rbac`](../../scripts/aks-flex-config) to setup required node bootstrap RBAC permissions.
+1. Run [`scripts/aks-flex-config setup-node-rbac`](../../scripts/aks-flex-config) to set up the least-privilege node bootstrap RBAC permissions. Clusters configured by an older helper require the explicit compatibility migration documented in the helper guide before another token can be generated.
 2. Run `scripts/aks-flex-config generate-node-config --bootstrap-token` to create a bootstrap token, fetch AKS cluster metadata, and render the host config.
 3. Copy the generated config to `/etc/aks-flex-node/config.json` on the target host.
 4. Run `aks-flex-node preflight --config /etc/aks-flex-node/config.json` to validate host, cluster, rootfs, and artifact prerequisites without mutating the node.

hack/e2e/README.md

@@ -12,6 +12,7 @@ The E2E suite provisions a no-CNI AKS cluster, installs Unbounded-Net as the clu
 | `python3` | Local registry port readiness checks and helper scripts. |
 | `ssh` / `scp` | VM access and artifact copy. |
 | `openssl` | Bootstrap token generation. |
+| `curl` / `sha256sum` / `tar` | Download and verify pinned historical release artifacts. |
 | `docker` | Build and push the controller image into the in-cluster local registry. |
 | `git` / `make` | Fetch and render Unbounded-Net manifests. |
 | `go` | Build the agent binary unless `--binary` is supplied. |
@@ -65,6 +66,7 @@ The default `all` command runs:
 | Command | Description |
 |---------|-------------|
 | `all` | Full flow: build, infra, join, validate, unjoin, validate absent, rejoin, validate, lifecycle, agent upgrade, repave, logs, cleanup. |
+| `historical-rbac-migration` | On a fresh real AKS cluster and token VM, join with the official v0.1.0 helper/binary, upgrade that host to HEAD, migrate legacy bootstrap RBAC, revoke the old token, and validate restarts. It does not run the other join modes. |
 | `infra` | Deploy AKS, four standard VMs, the Arc VM, Unbounded-Net CNI, the local registry, and the in-cluster controller. |
 | `join` | Join all Flex Node VMs. |
 | `join-msi` | Join only the managed-identity node. |
@@ -138,6 +140,45 @@ Additional environment variables:
 | `AZURE_SUBSCRIPTION_ID` | auto-detected | Azure subscription. |
 | `AZURE_TENANT_ID` | auto-detected | Azure tenant. |
 
+## Historical RBAC Migration Validation
+
+Run the focused compatibility suite with:
+
+```bash
+./hack/e2e/run.sh historical-rbac-migration
+```
+
+For a manual GitHub Actions run, select `historical-rbac-migration` in the
+`suite` workflow input. Infrastructure provisioning, the test, log upload, and
+cleanup stay in the same job; the suite deliberately does not call the
+Arc-inclusive parallel join path.
+
+The scenario downloads and verifies the official v0.1.0 release archive,
+extracted binary, helper, and installer. It then uses the pinned helper and
+binary to create the original broad bootstrap RBAC, a non-expiring token, the
+legacy config shape, and a real Ready node on the token VM. The historical
+daemon runs its production no-op path, not the v0.1.0 file-backed E2E machine
+client. On that same host it verifies the HEAD helper fails closed without the
+explicit migration flag, activates the HEAD binary through `agent-upgrade`,
+removes the legacy binding twice to prove idempotency, checks token access
+changes from HTTP 200 to 403 while CSR creation remains authorized, revokes the
+token and waits for HTTP 401, and restarts both kubelet and the daemon while
+checking the Node UID, Lease, readiness, and certificate-backed API access.
+v0.1.0 transitively pins the non-GPU rootfs
+`ghcr.io/azure/agent-ubuntu2404:v20260427`.
+
+There are two intentional compatibility boundaries:
+
+- The test creates a new AKS control plane and reproduces the v0.1.0
+  cluster-side state. It validates a historical node/config/RBAC migration, not
+  an AKS control plane that has itself been retained since v0.1.0.
+- v0.1.0 tokens lack the `kubernetes.azure.com/managedby=aks` label required by
+  the production managed CSR approver. The suite explicitly adopts its known
+  token with that label before the HEAD daemon requests a certificate. The
+  repository E2E approver does not enforce this label, so this test validates
+  the host/config/RBAC migration but is not independent proof of the production
+  approver's ownership check.
+
 ## Join Modes
 
 The suite validates five join paths. The E2E subscription must have `Microsoft.HybridCompute`, `Microsoft.HybridConnectivity`, and `Microsoft.GuestConfiguration` registered before the run.