OSAC-3616: Netris MaaS/CaaS infra feats & fixes

infra/netris/CLAUDE.md

@@ -1,6 +1,6 @@
 # netris-test-infra
 
-OSAC test infrastructure that deploys and tests OpenShift Assisted Cluster on a simulated Netris Spectrum-X GPU cluster. Tests three service models: VMaaS, BMaaS (planned), and CaaS.
+OSAC test infrastructure that deploys and tests OpenShift Assisted Cluster on a simulated Netris Spectrum-X GPU cluster. Tests three service models: VMaaS, BMaaS (planned), CaaS, and MaaS.
 
 Uses [netris-lab](netris-lab/) as a git submodule for the underlying network infrastructure.
 
@@ -16,12 +16,13 @@ roles/                          # Ansible roles (each has tasks/main.yml)
   cache-snapshot/               # Pull + cache flavor OCI image (skopeo)
   prep-osac/                    # Clone mono-repo, patch Helm values (fresh install)
   prep-refresh-osac/            # Clone mono-repo, patch values, rebuild CLI (snapshot refresh)
-  patch-osac-refresh/           # Post-refresh: patch Netris config + SSH keys into running cluster
+  patch-osac-refresh/           # Post-refresh: Netris/SSH + AAP project pin sync + CaC
   discover-caas/                # Boot discovery VMs with InfraEnv ISO
   setup-caas/                   # Label agents, register host type
-  create-caas/                  # Create CaaS cluster via fulfillment API
+  create-caas/                  # Create CaaS/MaaS cluster via fulfillment API
   destroy-infra/                # Teardown netris-lab
-  destroy-caas/                 # Teardown CaaS resources
+  destroy-caas/                 # Teardown CaaS/MaaS (osac delete + agents/InfraEnv/VMs)
+  force-destroy-caas/           # Force-clean stuck orders/namespaces + Netris orphans
 playbooks/                      # Ansible playbooks (one per workflow phase)
 inventory/
   local.yml                     # Inventory (localhost, local connection)
@@ -34,23 +35,30 @@ vendor/                         # Vendored Ansible collections
 
 ```
 make deploy                 # Full pipeline: deploy-infra + deploy-ocp + deploy-osac (fresh Helm install)
-make deploy-fast            # Snapshot pipeline: deploy-infra + deploy-ocp + deploy-osac (snapshot refresh)
+make deploy-fast            # Snapshot pipeline: same steps with OSAC_DEPLOY_MODE=snapshot
 make setup-infra            # Install prerequisites, cache images + snapshot flavor
 make deploy-infra           # Deploy netris-lab
 make deploy-ocp             # Configure Netris networking + restore OCP SNO from snapshot
 make deploy-osac            # Deploy OSAC (fresh Helm install or snapshot refresh based on OSAC_DEPLOY_MODE)
 make connectivity           # Re-run lab connectivity (VPN, BGP, softgate agents)
 make setup-caas             # CaaS setup: discover hosts, label agents, register host type
 make deploy-caas            # CaaS: create cluster
+make setup-maas             # MaaS setup (wrappers over setup-caas with MaaS overrides)
+make deploy-maas            # MaaS: create cluster (ocp_4_20_ai_maas + -p params)
 make destroy-full           # Teardown all infrastructure
 make destroy-osac           # Teardown OSAC only
 make destroy-ocp            # Reset OCP for reinstall
 make destroy-infra          # Teardown netris-lab
-make destroy-caas           # Teardown CaaS resources
+make destroy-caas           # Teardown CaaS/MaaS cluster + discovery
+make force-destroy-caas     # destroy-caas + strip stuck leftovers / Netris orphans
+make destroy-maas           # destroy-caas with MaaS overrides
+make force-destroy-maas     # force-destroy-caas with MaaS overrides
+make redeploy-fresh         # destroy-full + full BM pipeline (SUITE / OSAC_DEPLOY_MODE)
 make vendor-update          # Refresh vendored Ansible collections
 make gather-infra           # Gather diagnostic info from the cluster
 # Override variables: make <target> EXTRA_VARS="key=value"
 # Image overrides: make deploy-osac EXTRA_VARS="fulfillment_service_image=quay.io/..."
+# Suite / mode: make redeploy-fresh SUITE=maas OSAC_DEPLOY_MODE=snapshot
 ```
 
 ## Workflow Order
@@ -61,6 +69,11 @@ make gather-infra           # Gather diagnostic info from the cluster
 
 **CaaS:** deploy or deploy-fast → setup-caas → deploy-caas
 
+**MaaS:** deploy or deploy-fast → setup-maas → deploy-maas
+(or `make redeploy-fresh SUITE=maas OSAC_DEPLOY_MODE=snapshot`)
+
+**VMaaS / BMaaS:** not yet implemented
+
 ## Ansible Configuration
 
 - Inventory: `inventory/local.yml`
@@ -79,7 +92,7 @@ All variables in `inventory/group_vars/all.yml`. Key sections:
 - **OSAC**: `osac_repo/branch`, `osac_namespace`, `osac_values_file`
 - **Component images**: `osac_operator_image`, `fulfillment_service_image` (empty = defaults)
 - **Snapshot**: `snapshot_flavor_image`, `snapshot_flavor_dir`, `snapshot_recert_image`, `snapshot_osac_namespace`, `snapshot_osac_values_file`
-- **CaaS**: `caas_discovery_vm_patterns`, `caas_host_type_id`, `caas_cluster_name`, `caas_agents`
+- **CaaS / MaaS**: `caas_discovery_vm_patterns`, `caas_host_type_id`, `caas_cluster_name`, `caas_agents`, `caas_resource_class_hostnames` (empty = all agents); per-VM sizing via `caas_discovery_vcpu_overrides` / `caas_discovery_memory_mb_overrides`; MaaS Makefile `MAAS_*` (incl. `MAAS_DISCOVERY_MEMORY_MB_OVERRIDES`, default h01→16GiB)
 
 ## External Dependencies
 

infra/netris/Makefile

@@ -5,6 +5,11 @@ EXTRA_VARS ?=
 ANSIBLE_EXTRA = $(if $(EXTRA_VARS),-e '$(EXTRA_VARS)') -e osac_deploy_mode=$(OSAC_DEPLOY_MODE) -e osac_values_file=$(OSAC_VALUES_FILE)
 ENV_INFRA := .env.infra
 
+# Suite selects CaaS vs MaaS for restore disk grow + redeploy-server.sh flow.
+# SUITE=caas (default) — setup-caas / deploy-caas.
+# SUITE=maas — grow h00 to ocp_snapshot_disk_gb; setup-maas / deploy-maas.
+SUITE ?= caas
+
 OSAC_NAMESPACE ?= osac-e2e-ci
 OSAC_VALUES_FILE ?= values/caas-ci/values.yaml
 OSAC_PULL_SECRET_PATH ?= /root/pull-secret
@@ -39,7 +44,7 @@ deploy-infra:
 	ansible-playbook playbooks/deploy-infra.yml $(ANSIBLE_EXTRA)
 
 deploy-ocp:
-	ansible-playbook playbooks/deploy-ocp.yml $(ANSIBLE_EXTRA)
+	ansible-playbook playbooks/deploy-ocp.yml -e suite=$(SUITE) $(ANSIBLE_EXTRA)
 
 deploy-osac:
 	@if [ "$(OSAC_DEPLOY_MODE)" = "snapshot" ]; then \
@@ -81,6 +86,9 @@ destroy-setup:
 destroy-caas:
 	ansible-playbook playbooks/destroy-caas.yml $(ANSIBLE_EXTRA)
 
+force-destroy-caas:
+	ansible-playbook playbooks/force-destroy-caas.yml $(ANSIBLE_EXTRA)
+
 gather-infra:
 	ansible-playbook playbooks/gather-infra.yml $(ANSIBLE_EXTRA)
 
@@ -102,6 +110,55 @@ destroy-bmaas:
 gather-bmaas:
 	ansible-playbook playbooks/gather-bmaas.yml $(ANSIBLE_EXTRA)
 
+# ---------------------------------------------------------------------------
+# MaaS — thin wrappers over CaaS playbooks with MaaS-specific variable overrides
+# Override any default with MAAS_CLUSTER_NAME=..., MAAS_HOST_TYPE_ID=..., etc.
+# ---------------------------------------------------------------------------
+MAAS_CLUSTER_NAME     ?= maas-cluster
+MAAS_HOST_TYPE_ID     ?= g5
+MAAS_CLUSTER_TEMPLATE ?= osac.templates.ocp_4_20_ai_maas
+MAAS_CATALOG_ITEM     ?= maas-ocp
+# Discovery VM sizing for setup-maas only (CaaS keeps inventory defaults: 4 vCPU / 16GiB).
+# Default MaaS: 10 vCPU / 20GiB for h02–h03; h01 stays at 4 vCPU / 16GiB (unused for MaaS workers).
+MAAS_DISCOVERY_VCPU      ?= 10
+MAAS_DISCOVERY_MEMORY_MB ?= 20480
+MAAS_DISCOVERY_VCPU_OVERRIDES ?= {"hgx-pod00-su0-h01":4}
+MAAS_DISCOVERY_MEMORY_MB_OVERRIDES ?= {"hgx-pod00-su0-h01":16384}
+# Only these inventory hostnames get resource_class=g5 (and approval). Empty = all agents.
+# Comma-separated. Keeps h01 out of the MaaS claim pool; setup-caas still labels all as ci-worker.
+MAAS_LABEL_HOSTNAMES ?= hgx-pod00-su0-h02,hgx-pod00-su0-h03
+# Template -p flags for osac create (deploy-maas only). Comma-separated key=value.
+MAAS_CLUSTER_PARAMS   ?= enable_maas=true,enable_keycloak=true,enable_observability=true,enable_dashboard=true
+MAAS_PARAMS_JSON = $(shell python3 -c 'import json; print(json.dumps([p.strip() for p in """$(MAAS_CLUSTER_PARAMS)""".split(",") if p.strip()]))')
+MAAS_LABEL_HOSTNAMES_JSON = $(shell python3 -c 'import json; print(json.dumps([h.strip() for h in """$(MAAS_LABEL_HOSTNAMES)""".split(",") if h.strip()]))')
+
+MAAS_EXTRA = \
+  -e 'caas_cluster_name=$(MAAS_CLUSTER_NAME)' \
+  -e 'caas_host_type_id=$(MAAS_HOST_TYPE_ID)' \
+  -e 'caas_host_type_title=MaaS Node' \
+  -e 'caas_host_type_description=Bare-metal nodes for MaaS testing' \
+  -e 'caas_cluster_template=$(MAAS_CLUSTER_TEMPLATE)' \
+  -e 'caas_catalog_item=$(MAAS_CATALOG_ITEM)' \
+  -e 'caas_catalog_item_title=MaaS OCP' \
+  -e 'caas_catalog_item_description=OCP cluster for MaaS testing' \
+  -e 'caas_discovery_vcpu=$(MAAS_DISCOVERY_VCPU)' \
+  -e 'caas_discovery_memory_mb=$(MAAS_DISCOVERY_MEMORY_MB)' \
+  -e '{"caas_discovery_vcpu_overrides":$(MAAS_DISCOVERY_VCPU_OVERRIDES)}' \
+  -e '{"caas_discovery_memory_mb_overrides":$(MAAS_DISCOVERY_MEMORY_MB_OVERRIDES)}' \
+  -e '{"caas_resource_class_hostnames":$(MAAS_LABEL_HOSTNAMES_JSON)}'
+
+setup-maas:
+	ansible-playbook playbooks/setup-caas.yml $(MAAS_EXTRA) $(ANSIBLE_EXTRA)
+
+deploy-maas:
+	ansible-playbook playbooks/deploy-caas.yml $(MAAS_EXTRA) -e '{"caas_cluster_params":$(MAAS_PARAMS_JSON)}' $(ANSIBLE_EXTRA)
+
+destroy-maas:
+	ansible-playbook playbooks/destroy-caas.yml $(MAAS_EXTRA) $(ANSIBLE_EXTRA)
+
+force-destroy-maas:
+	ansible-playbook playbooks/force-destroy-caas.yml $(MAAS_EXTRA) $(ANSIBLE_EXTRA)
+
 # ─── Internal sub-targets (deploy-osac implementation) ─────────────
 
 # Fresh install: clone mono-repo, patch Helm values, run make install
@@ -120,6 +177,7 @@ post-osac:
 	ansible-playbook playbooks/post-osac.yml $(ANSIBLE_EXTRA)
 
 # Snapshot refresh: clone mono-repo, patch values, run refresh-after-snapshot.py
+# post-refresh-osac → patch-osac-refresh (incl. AAP project pin sync + CaC)
 refresh-osac: prep-refresh-osac run-refresh-osac post-refresh-osac
 
 prep-refresh-osac:
@@ -157,15 +215,23 @@ cleanup-dns:
 destroy-full:
 	ansible-playbook playbooks/destroy-full.yml $(ANSIBLE_EXTRA)
 
+# OSAC_DEPLOY_MODE=fresh (default) or snapshot — same meaning as deploy / deploy-fast.
+# SUITE=caas (default) or maas — selects setup/deploy flow in redeploy-server.sh.
 redeploy-fresh:
 	@echo "=== Destroying all infrastructure ==="
 	-$(MAKE) destroy-full
 	@sync
-	@echo "=== Starting fresh deploy ==="
-	scripts/redeploy-server.sh --fresh "$(EXTRA_VARS)" 2>&1 | tee -a /root/deploy.log
+	@echo "=== Starting deploy (SUITE=$(SUITE) OSAC_DEPLOY_MODE=$(OSAC_DEPLOY_MODE)) ==="
+	SUITE=$(SUITE) OSAC_DEPLOY_MODE=$(OSAC_DEPLOY_MODE) \
+		scripts/redeploy-server.sh --fresh \
+		$(if $(filter snapshot,$(OSAC_DEPLOY_MODE)),--snapshot) \
+		"$(EXTRA_VARS)" 2>&1 | tee -a /root/deploy.log
 
 redeploy-continue:
-	scripts/redeploy-server.sh "$(EXTRA_VARS)" 2>&1 | tee -a /root/deploy.log
+	SUITE=$(SUITE) OSAC_DEPLOY_MODE=$(OSAC_DEPLOY_MODE) \
+		scripts/redeploy-server.sh \
+		$(if $(filter snapshot,$(OSAC_DEPLOY_MODE)),--snapshot) \
+		"$(EXTRA_VARS)" 2>&1 | tee -a /root/deploy.log
 
 deploy-jump:
 	@scripts/deploy-jump.sh
@@ -187,7 +253,9 @@ health-check:
 .PHONY: deploy deploy-fast \
         deploy-setup setup-infra deploy-infra deploy-ocp deploy-osac \
         setup-caas deploy-caas \
-        destroy-osac destroy-ocp destroy-infra destroy-setup destroy-caas destroy-full \
+		destroy-osac destroy-ocp destroy-infra destroy-setup destroy-full \
+		destroy-caas force-destroy-caas \
+        setup-maas deploy-maas destroy-maas force-destroy-maas \
         gather-infra gather-mgmt-cluster gather-caas \
         setup-bmc setup-bmaas destroy-bmaas gather-bmaas \
         install-osac prep-osac run-osac-setup post-osac \

infra/netris/README.md

@@ -34,7 +34,7 @@ Internet access for OCP image pulls flows through: hgx-00 → NS VNet → softga
 ## Prerequisites
 
 - **Bare-metal host** running RHEL 9.x or Rocky Linux 9.x with KVM support
-- **System packages** — `dnf install -y git make python3-pip ansible-core && pip3 install ansible` (all other tools are installed automatically by `make setup-infra`)
+- **System packages** — `dnf install -y git make python3-pip ansible-core && pip3 install ansible`. `make setup-infra` then installs the rest of the BM bootstrap (`sshpass`, `tmux`, `policycoreutils-python-utils`), EPEL from the Fedora URL (required on RHEL), and the Python `kubernetes` client.
 - **Resources**: ~32+ CPU cores, 128+ GB RAM (lab VMs + OCP SNO VM)
 - **Storage**: ~400GB+ for OCP disk images, K3s, and containers. If the root partition is smaller, use a secondary data disk and enable automatic provisioning with `disk_setup_enabled=true` (via `EXTRA_VARS` or in the config). Run `make disk-setup` standalone or pass the variable during setup: `make setup-infra EXTRA_VARS="disk_setup_enabled=true"`
 - **Netris license key** — place at repo root as `license.key`
@@ -101,6 +101,8 @@ After deployment, the kubeconfig is at `/root/.kube/config`.
 |--------|-------------|------|
 | `make setup-caas` | Discover hosts, label agents, register host type, configure osac CLI | ~30 min |
 | `make deploy-caas` | Create CaaS cluster using `ocp_ci_small` template | ~60 min |
+| `make setup-maas` | Same as setup-caas with MaaS host type / sizing; labels only `MAAS_LABEL_HOSTNAMES` (default h02+h03) with `g5` | ~30 min |
+| `make deploy-maas` | Create MaaS cluster (`ocp_4_20_ai_maas` + `-p` template params) | ~60 min |
 
 ### Destroy
 
@@ -111,6 +113,10 @@ After deployment, the kubeconfig is at `/root/.kube/config`.
 | `make destroy-ocp` | Reset OCP for reinstall: delete cluster, recreate disk, boot VM |
 | `make destroy-infra` | Tear down netris-lab (VMs, K3s, topology) |
 | `make destroy-caas` | CaaS teardown: stop discovery VMs, remove disks/ISO, delete namespace, clean DNS |
+| `make force-destroy-caas` | destroy-caas + strip stuck orders/namespaces, wipe VMs, Netris orphans |
+| `make destroy-maas` | destroy-caas with MaaS cluster/host-type overrides |
+| `make force-destroy-maas` | force-destroy-caas with MaaS overrides |
+| `make destroy-bmaas` | BMaaS teardown: remove BMH/K8s resources, stop BMaaS VMs, clean disks, delete BMH namespace |
 
 ### Recovery and Utilities
 
@@ -174,12 +180,30 @@ make setup-caas     # discover hosts, label agents, register host type
 make deploy-caas    # create cluster
 ```
 
+**Deploy MaaS after OSAC is up:**
+```bash
+make setup-maas     # same discovery path; labels only MAAS_LABEL_HOSTNAMES (default h02+h03) with g5
+make deploy-maas    # create cluster with ocp_4_20_ai_maas + enable_* params
+```
+
 **Rebuild from scratch:**
 ```bash
-make destroy        # tear down everything
+make destroy-full   # tear down everything
 make deploy-fast    # full redeploy (snapshot path)
 ```
 
+**BM full pipeline (redeploy-server.sh):**
+```bash
+# CaaS + fresh OSAC install (default)
+make redeploy-fresh
+
+# CaaS + snapshot OSAC refresh (same mode as deploy-fast)
+make redeploy-fresh OSAC_DEPLOY_MODE=snapshot
+
+# MaaS + snapshot refresh (grows h00 disk; setup-maas / deploy-maas)
+make redeploy-fresh SUITE=maas OSAC_DEPLOY_MODE=snapshot
+```
+
 ### Bare-Metal Lab Deployment (from laptop)
 
 For persistent bare-metal servers behind NAT (e.g., Red Hat lab infrastructure), use the remote deploy workflow. This handles image caching, bootstrapping, and resilient multi-step deploys from your laptop.
@@ -222,12 +246,15 @@ source scripts/env-mylab.sh && make deploy-jump
 
 | Target | Description |
 |--------|-------------|
-| `make redeploy-fresh` | Destroy + wipe progress + full fresh deploy |
+| `make redeploy-fresh` | Destroy + wipe progress + full pipeline (`SUITE`, `OSAC_DEPLOY_MODE`) |
+| `make redeploy-continue` | Resume failed redeploy from last incomplete step |
 | `make disk-setup` | Auto-detect and mount data disk |
 | `make post-install` | Fix Keycloak/UI + generate access doc |
 | `make access-doc` | Generate handover documentation only |
 | `make health-check` | Quick status verification |
 
+`redeploy-server.sh` steps: `setup-infra` → `deploy-infra` → `deploy-ocp` → `deploy-osac` → `setup-caas|setup-maas` → `deploy-caas|deploy-maas` → `post-install`. Pass `OSAC_DEPLOY_MODE=snapshot` (or `--snapshot`) for refresh instead of Helm install; default is `fresh`.
+
 See the PR description for known issues and workarounds specific to BM/RHEL environments.
 
 ## Accessing OCP Routes
@@ -258,10 +285,12 @@ The flow runs three Ansible roles in sequence:
 
 1. **`configure-netris`** — creates VPC, VNet (DHCP disabled), subnet, SNAT/DNAT rules via Netris API
 2. **`configure-dns`** — creates Route 53 DNS records and local dnsmasq config for the cluster domain
-3. **`restore-ocp`** — creates copy-on-write disk overlays backed by the cached flavor, mounts the OS disk via qemu-nbd to write pre-boot config (hostname, nodeip hint, dnsmasq overrides, nmstate config for static IP, OVN/OVS cleanup), runs recert (via JSON config file, matching LCA's approach) to regenerate all TLS certificates and cluster identity, then waits for cluster health
+3. **`restore-ocp`** — creates copy-on-write disk overlays backed by the cached flavor, mounts the OS disk via qemu-nbd to write pre-boot config (hostname, nodeip hint, dnsmasq overrides, nmstate config for static IP, OVN/OVS cleanup), runs recert (via JSON config file, matching LCA's approach) to regenerate all TLS certificates and cluster identity, then waits for cluster health. When `suite=maas` (via `SUITE=maas`), also grows the h00 OS disk to `ocp_snapshot_disk_gb` before pre-boot config.
 
 The snapshot flavor is pulled and cached during `make setup-infra` (one-time ~60GB download). Subsequent deploys use copy-on-write overlays, so only changed blocks are written.
 
+OSAC itself is handled by `make deploy-osac` (not part of `deploy-ocp`). In `OSAC_DEPLOY_MODE=snapshot`, that runs `refresh-osac` → `post-refresh-osac` → **`patch-osac-refresh`**, which wires Netris/SSH/socat and then applies `config-as-code-ig`’s `AAP_PROJECT_GIT_*` pin onto the live AAP Project and launches `osac-config-as-code` once (snapshot AAP DB keeps the bake-time `scm_branch`; waiting for the hourly CaC schedule is too late for `setup-maas` / `publish-templates`).
+
 ## How deploy-osac Works
 
 `make deploy-osac` runs in three phases:
@@ -314,6 +343,10 @@ make deploy-osac EXTRA_VARS='{"osac_branch": "feature-x"}'
 ```bash
 make deploy-ocp EXTRA_VARS="ocp_version=4.18"
 make setup-caas EXTRA_VARS="caas_cluster_name=my-cluster caas_discovery_vcpu=8"
+
+# MaaS: default 10 vCPU / 20GiB for h02–h03; h01 stays 4 vCPU / 16GiB via overrides.
+# Grow workers only (e.g. real model headroom) without resizing h01:
+make setup-maas MAAS_DISCOVERY_MEMORY_MB=49152
 ```
 
 #### Lab & Identity
@@ -396,7 +429,9 @@ dns_server: "10.0.0.1"
 | `caas_cluster_template` | `osac.templates.ocp_ci_small` | Cluster template for CaaS | defaults only |
 | `caas_host_type_id` | `ci-worker` | Resource class label for CaaS agents | defaults only |
 | `caas_discovery_vcpu` | `4` | Discovery VM vCPUs | yes (8) |
+| `caas_discovery_vcpu_overrides` | `{}` | Per-VM vCPU map (VM name → count); empty = use `caas_discovery_vcpu` | defaults only |
 | `caas_discovery_memory_mb` | `16384` | Discovery VM memory in MB | yes (32768) |
+| `caas_discovery_memory_mb_overrides` | `{}` | Per-VM memory map (VM name → MB); empty = use `caas_discovery_memory_mb` | defaults only |
 | `caas_discovery_disk_gb` | `100` | Discovery VM disk in GB | yes (150) |
 | `caas_discovery_vm_patterns` | `[hgx-pod00-su0-h01..03]` | VM names for CaaS discovery | defaults only |