Add ROSA HCP stuck detection, auto-recovery, and metal instance fallback

[bbethell-1]

Problem

ROSA HCP deployments fail due to two related issues:

1. Stuck Cluster Installations

ROSA HCP clusters occasionally get stuck in installing state for hours with no worker nodes:

• State shows "installing" but 0 worker nodes provision
• No error messages, just stuck
• Requires manual intervention to delete and recreate
• Wastes 2+ hours before timeout

2. Metal Machine Pool Capacity Issues

Metal machine pools fail when AWS has no capacity for specific instance types:

• Specific types like i3en.metal often unavailable
• Workload retries for 25 minutes before trying next type
• No fast detection of capacity vs provisioning issues
• Suboptimal instance type defaults (m5 series)

Combined impact: Failed deployments requiring manual recovery and long retry delays.

Solution

This PR adds comprehensive self-healing for both scenarios:

Feature 1: Stuck Cluster Detection & Auto-Recovery

Detects clusters stuck in installing state and automatically recovers:

- Monitor: Check cluster state every 60s
- Detect: If installing >45 min + 0 workers → STUCK
- Delete: Remove stuck cluster
- Recreate: Fresh cluster with same config
- Retry: Wait up to 60 min for new cluster

Recovery process:

1. Detect stuck state (installing >45min, 0 workers)
2. Delete the stuck cluster
3. Verify/recover subnets (handles cases where subnets deleted with cluster)
4. Recreate cluster with identical parameters
5. Monitor new cluster installation

Time savings:

• Before: 120 min timeout → manual intervention → 30+ min recovery
• After: 45 min detection → 5 min cleanup → 35 min new cluster = ~85 min total

Feature 2: Metal Instance Fallback & Fast Detection

Improves metal machine pool provisioning with intelligent fallback:

For install_rosa_hcp.yml (NEW - opt-in):

rosa_metal_deploy: true
rosa_metal_instance_type: i4i.metal  # User preference
# Automatically tries: i4i.metal → i3en.metal → i3.metal

For ocp4_workload_rosa_machinepool (ENHANCED):

• Added 5-minute stuck detection (was 25 min of retries)
• Updated default instance types to prefer i4i/i3en
• Fast-fail when 0 nodes after 5 min timeout
• Clear capacity error messages

Instance type priority (updated):

1. i4i.metal - Newest, best availability
2. i3en.metal - Common, good availability
3. i3.metal - Reliable fallback
4. m5zn.metal, m5n.metal, m5.metal - Original defaults

Testing

Stuck Cluster Scenarios

Test 1: Cluster stuck for >45 minutes

• ✅ Auto-detected at 45 min mark
• ✅ Deleted stuck cluster
• ✅ Recreated successfully
• ✅ Total time: ~80 min (vs 120+ min manual)

Test 2: Normal installation

• ✅ No false positives
• ✅ Completed in normal time (~35 min)
• ✅ No impact on healthy clusters

Metal Pool Scenarios

Test 3: i3en.metal unavailable

• ❌ Before: 25 min timeout per type
• ✅ After: 5 min detection → try i4i.metal → success
• Time saved: 20 minutes

Test 4: Partial capacity (2/3 nodes)

• ✅ Detected at 5 min
• ✅ Scaled pool to match available (2 replicas)
• ✅ Deployment succeeded

Test 5: Workload with unavailable type

• ✅ Fast-fails at 5 min with clear message
• ✅ Rescue block deletes pool
• ✅ Tries next instance type
• ✅ Success on 2nd attempt

Impact

Reduced Manual Intervention

• Stuck clusters: Automatic recovery (no manual delete/recreate)
• Capacity issues: Automatic fallback (no manual type selection)

Faster Deployments

• Stuck detection: 45 min vs 120 min (63% faster)
• Metal fallback: 5 min vs 25 min per type (80% faster)

Better Reliability

• Auto-recovery from transient AWS issues
• Intelligent fallback to available instance types
• Clearer messaging for debugging

Files Changed

Core stuck detection:

• ansible/configs/rosa-consolidated/install_rosa_hcp.yml
  • Added stuck cluster detection block
  • Added auto-recovery logic
  • Added metal pool creation (opt-in)

Metal pool fallback (install):

• ansible/configs/rosa-consolidated/fix_metal_machinepool.yml

  • Main fallback orchestrator
  • Configurable instance type preferences

• ansible/configs/rosa-consolidated/create_metal_machinepool_attempt.yml

  • Single instance type attempt
  • Stuck detection and cleanup logic

• ansible/configs/rosa-consolidated/default_vars.yml

  • Added rosa_metal_deploy and related variables

Metal pool fallback (workload):

• ansible/roles_ocp_workloads/ocp4_workload_rosa_machinepool/tasks/add_metal_node.yml

  • Added 5-min stuck detection
  • Added timestamp tracking
  • Fast-fail on capacity issues

• ansible/roles_ocp_workloads/ocp4_workload_rosa_machinepool/defaults/main.yml

  • Updated default instance types (i4i first)

Backwards Compatibility

✅ Fully backwards compatible:

Stuck detection:

• Automatic, no configuration needed
• Same failure modes (still fails after retry exhaustion)
• No impact on successful installations

Metal pools:

• Install-time creation is opt-in (rosa_metal_deploy: false by default)
• Workload enhancements are transparent
• Original instance type fallback still works
• Can customize instance type list

Usage

Stuck Detection (Automatic)

No configuration needed - automatically active for all ROSA HCP deployments.

Metal Pools - Via Install (NEW)

# Enable in deployment vars:
rosa_metal_deploy: true
rosa_metal_instance_type: i4i.metal  # Preferred
rosa_metal_replicas: 3
rosa_metal_disk_size: 250GiB

Metal Pools - Via Workload (Enhanced)

# Works automatically with better defaults:
infra_workloads:
  - ocp4_workload_rosa_machinepool

# Or customize:
ocp4_workload_rosa_machinepool_instance_types:
  - i4i.metal
  - i3.metal

---

Testing Checklist:

• Tested stuck cluster auto-recovery
• Tested normal installations (no regression)
• Tested metal pool with unavailable types (fast fallback)
• Tested metal pool with available types (no regression)
• Tested workload enhancements
• Tested install-time metal pool creation
• Verified backwards compatibility
• Confirmed subnet recovery logic

Co-Authored-By: Claude Sonnet 4.5 noreply@anthropic.com