Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions docs/usages/aks-flex-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The helper does not install anything on the target host. It uses Azure CLI and,
- Azure CLI authenticated to the subscription that contains the AKS cluster.
- `python3` on the workstation.
- `kubectl` on the workstation for `setup-node-rbac` and `--bootstrap-token` config generation.
- Permission to run `az aks get-credentials --admin` and create Kubernetes `ClusterRoleBinding` and bootstrap token `Secret` objects.
- Permission to run `az aks get-credentials --admin`, create Kubernetes `ClusterRoleBinding` and bootstrap token `Secret` objects, and remove the obsolete `aks-flex-node-role` binding when present.

## Save The Helper

Expand Down Expand Up @@ -46,7 +46,17 @@ Run this once per cluster for bootstrap-token joins:
--subscription "$SUBSCRIPTION_ID"
```

This applies the bootstrap-related `ClusterRoleBinding` objects for the `system:bootstrappers:aks-flex-node` group.
This applies only the CSR creation and approval `ClusterRoleBinding` objects for the `system:bootstrappers:aks-flex-node` group. It also removes the obsolete `aks-flex-node-role` binding created by older versions of the helper; that binding granted bootstrap tokens the broad legacy `system:node` role. The command is safe to rerun, and existing clusters should run it once after updating the helper.

Bootstrap-token config generation performs the same legacy-binding cleanup before creating a token. If cleanup fails, token generation stops rather than issuing a token into a cluster that may still grant it broad node permissions.

To remove only the obsolete binding from an existing cluster, run:

```bash
kubectl delete clusterrolebinding aks-flex-node-role --ignore-not-found=true
```

Removing this binding does not interrupt joined nodes: they authenticate with their issued certificates rather than the bootstrap group. New and in-progress joins retain the CSR permissions installed above.

## Generate Node Config

Expand Down
2 changes: 1 addition & 1 deletion docs/usages/joining-nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Bootstrap token mode is the recommended quickstart path. It uses Kubernetes TLS

High-level flow:

1. Run [`scripts/aks-flex-config setup-node-rbac`](../../scripts/aks-flex-config) to setup required node bootstrap RBAC permissions.
1. Run [`scripts/aks-flex-config setup-node-rbac`](../../scripts/aks-flex-config) to set up the least-privilege node bootstrap RBAC permissions. Rerunning the command also removes the broad legacy `aks-flex-node-role` binding from clusters configured by older helper versions.
2. Run `scripts/aks-flex-config generate-node-config --bootstrap-token` to create a bootstrap token, fetch AKS cluster metadata, and render the host config.
3. Copy the generated config to `/etc/aks-flex-node/config.json` on the target host.
4. Run `aks-flex-node preflight --config /etc/aks-flex-node/config.json` to validate host, cluster, rootfs, and artifact prerequisites without mutating the node.
Expand Down
35 changes: 17 additions & 18 deletions hack/e2e/lib/node-join-kubeadm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ _kubeadm_ensure_rbac() {
# - ClusterRole/ClusterRoleBinding for bootstrappers to GET nodes
# - ConfigMaps: cluster-info (kube-public), kubeadm-config and
# kubelet-config (kube-system) consumed by kubeadm join
kubectl apply -f - <<EOF
if ! kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
Expand Down Expand Up @@ -86,22 +86,6 @@ subjects:
name: system:nodes
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: aks-flex-node-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:aks-flex-node
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: ${kubeadmBootstrapGroup}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: kube-system
Expand Down Expand Up @@ -182,11 +166,22 @@ subjects:
apiGroup: rbac.authorization.k8s.io
name: ${kubeadmBootstrapGroup}
EOF
then
log_error "Failed to apply bootstrap RBAC"
return 1
fi

# The CSR and narrowly scoped preflight bindings above replace this legacy,
# overly broad binding. Explicit deletion also migrates reused E2E clusters.
if ! kubectl delete clusterrolebinding aks-flex-node-role --ignore-not-found=true; then
log_error "Failed to remove legacy aks-flex-node-role binding"
return 1
fi

# Publish the ConfigMaps that kubeadm join reads during its preflight phase.
# cluster-info goes into kube-public (publicly readable).
# kubeadm-config and kubelet-config go into kube-system (bootstrapper-readable).
kubectl apply -f - <<EOF
if ! kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
Expand Down Expand Up @@ -229,6 +224,10 @@ data:
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
EOF
then
log_error "Failed to apply kubeadm bootstrap ConfigMaps"
return 1
fi

log_success "Bootstrap RBAC and ConfigMaps configured"
}
Expand Down
38 changes: 23 additions & 15 deletions scripts/aks-flex-config
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ from urllib.parse import urlsplit

RESOURCE_MANAGER_ENDPOINT = "https://management.azure.com"
DEFAULT_AGENT_POOL_NAME = "aksflexnodes"
LEGACY_NODE_ROLE_BINDING = "aks-flex-node-role"


def log_info(message: str) -> None:
Expand Down Expand Up @@ -67,11 +68,32 @@ def setup_node_rbac(args: argparse.Namespace) -> None:

log_info("applying bootstrap token RBAC bindings")
run(["kubectl", "apply", "-f", "-"], input_text=RBAC_MANIFEST)
remove_legacy_node_role_binding()


def remove_legacy_node_role_binding() -> None:
# Older helpers granted bootstrap credentials the broad legacy node role.
# Remove it after the least-privilege CSR bindings are safely in place.
log_info("removing legacy bootstrap node role binding")
run(
[
"kubectl",
"delete",
"clusterrolebinding",
LEGACY_NODE_ROLE_BINDING,
"--ignore-not-found=true",
]
)


def generate_bootstrap_token(args: argparse.Namespace) -> str:
require_command("kubectl")

# Existing clusters may skip setup-node-rbac after updating this helper.
# Never mint another token while the legacy broad binding may still exist.
remove_legacy_node_role_binding()
log_info("creating bootstrap token")
Comment thread
wenhug marked this conversation as resolved.

token_id = secrets.token_hex(3)
token_secret = secrets.token_hex(8)
token = f"{token_id}.{token_secret}"
Expand Down Expand Up @@ -177,7 +199,6 @@ def render_config(args: argparse.Namespace, mode: str, metadata: dict[str, str])

if mode == "bootstrap-token":
load_admin_kubeconfig(args)
log_info("creating bootstrap token")
token = generate_bootstrap_token(args)
server_url = run(
["kubectl", "config", "view", "--minify", "-o", "jsonpath={.clusters[0].cluster.server}"],
Expand Down Expand Up @@ -258,7 +279,7 @@ def build_parser() -> argparse.ArgumentParser:
subparser.add_argument("--cluster-name", required=True)
subparser.add_argument("--subscription")

rbac = subparsers.add_parser("setup-node-rbac", help="Apply node bootstrap RBAC bindings.")
rbac = subparsers.add_parser("setup-node-rbac", help="Reconcile node bootstrap RBAC bindings.")
add_cluster_args(rbac)
rbac.set_defaults(func=setup_node_rbac)

Expand Down Expand Up @@ -305,19 +326,6 @@ roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:aks-flex-node
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: aks-flex-node-role
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:bootstrappers:aks-flex-node
Expand Down
Loading
Loading