RHOAI Deploy

Simplified deployment of Red Hat OpenShift AI (RHOAI) platform using GitOps.

Overview

This repository provides a streamlined approach to deploying the RHOAI platform:

Platform: RHOAI operator and dependencies (NFD, Kueue, NVIDIA GPU Operator)
GitOps: ArgoCD-based deployment automation
Optional storage: MinIO (S3-compatible) via a separate Argo CD Application with manual sync

Note: These deployment steps are specifically for RHOAI 3.x. RHOAI 3.x requires OpenShift 4.19+ and uses the fast-3.x subscription channel.

For GPU infrastructure (MachineSets), see the openshift-infra repository.

Prerequisites

OpenShift 4.19+ with cluster-admin access
oc CLI installed and configured
GPU nodes (optional, for model serving and training)
- See openshift-infra for GPU node deployment

Prerequisites Check:

# Verify OpenShift version (4.19+ required)
oc version

# Verify cluster-admin access
oc whoami
oc auth can-i create namespace

Deployment Steps

Step 1: Install OpenShift GitOps (2-3 minutes)

./bootstrap.sh

Note: If GitOps is already installed (e.g., from deploying another repository), the bootstrap script will detect it and skip installation.

Step 2: Deploy GPU Nodes (Optional, 10-15 minutes)

GPU node deployment is managed in a separate repository. See the openshift-infra repository for:

Multi-GPU instance type support (g4dn, g6)
Automated deployment scripts
Cost optimization guidance

Step 3: Deploy RHOAI Platform (5-10 minutes)

Option A: Deploy as one Application (Recommended)

# Single command deploys RHOAI + dependencies + GPU Operator
oc apply -f gitops/platform/rhoai-platform.yaml

Option B: Step-by-step

# 1. Deploy RHOAI dependencies (NFD + Kueue)
oc apply -f gitops/platform/rhoai-dependencies.yaml

# Wait for dependencies
oc wait --for=condition=Ready \
  pod -l app=nfd-master -n openshift-nfd --timeout=300s
oc wait --for=condition=Ready \
  pod -l control-plane=controller-manager \
  -n openshift-kueue --timeout=300s

# 2. Deploy NVIDIA GPU Operator (required for GPU support)
oc apply -f gitops/platform/nvidia-gpu-operator.yaml

# Wait for GPU operator (3-5 minutes, only if GPU nodes exist)
oc wait --for=condition=Ready \
  pod -l app=gpu-operator -n nvidia-gpu-operator --timeout=300s

# 3. Deploy RHOAI Operator
oc apply -f gitops/platform/rhoai-operator.yaml

Step 4: MinIO object storage (Optional)

MinIO is not included in gitops/platform/rhoai-platform.yaml. The gitops/platform/minio.yaml Application uses manual sync only (no automated sync). Root credentials are not committed to Git; you must create the Secret before syncing.

Manifests live under platform/storage/minio/ (Kustomize base in base/, large PVC overlay in overlays/large/). Images are pinned (quay.io/minio/minio and quay.io/minio/mc, same release tag). A Job creates buckets models, data, and pipelines after the server is up.

1. Create the minio namespace and root credentials Secret

Generate a root user and password (or set literals yourself). Keys in the Secret must be exactly MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.

MINIO_ROOT_USER="minioadmin"
MINIO_ROOT_PASSWORD="$(openssl rand -hex 24)"

oc create namespace minio --dry-run=client -o yaml | oc apply -f -

oc create secret generic minio-secret -n minio \
  --from-literal=MINIO_ROOT_USER="${MINIO_ROOT_USER}" \
  --from-literal=MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD}"

# Save the password — you need it for the MinIO console and S3 clients
echo "MinIO root password: ${MINIO_ROOT_PASSWORD}"

For a fixed lab password instead of random bytes, assign MINIO_ROOT_PASSWORD yourself before oc create secret (use a strong value outside demos).

2. Register the Argo CD Application

oc apply -f gitops/platform/minio.yaml

3. Sync manually

In the Argo CD UI (or CLI), open the minio application and run Sync. Until the Secret exists, the MinIO pods and bucket Job will not succeed.

4. Larger PVC (optional)

For a 180Gi PVC instead of the default 60Gi, set spec.source.path in gitops/platform/minio.yaml to platform/storage/minio/overlays/large before the PVC is first bound. Expanding a bound RWO PVC depends on your storage class; planning size up front avoids migration.

Endpoints

In-cluster S3 API: http://minio-service.minio.svc:9000
OpenShift Routes: oc get routes -n minio (API and console UI)

Step 5: External Secrets Operator and pipeline credentials (Optional)

platform/access/ installs the External Secrets Operator for Red Hat OpenShift and creates the minio-dspa-connection Secret in demo by syncing from minio/minio-secret. This is required if you plan to use the Data Science Pipelines server from rhoai-distributed.

The access Application is intentionally separate from minio so it can be ordered after MinIO is healthy. When using rhoai-demo-foundations as an app-of-apps, this ordering is handled automatically via sync waves (minio wave 4, access wave 5).

# 1. Install the External Secrets Operator (OLM subscription + namespace)
oc apply -f gitops/platform/access-operator.yaml

# 2. Wait for the CSV to reach Succeeded before applying the instance
oc wait csv -n external-secrets-operator \
  -l operators.coreos.com/openshift-external-secrets-operator.external-secrets-operator \
  --for=jsonpath='{.status.phase}'=Succeeded --timeout=300s

# 3. Deploy the ESO instance and minio-dspa-connection ExternalSecret
oc apply -f gitops/platform/access.yaml

Then Sync the access Application in Argo CD. Once access-operator is healthy and the external-secrets-operator CSV shows Succeeded, the ExternalSecretsConfig operand and the minio-dspa-connection ExternalSecret will reconcile automatically.

Note: access-operator must be applied and healthy before access. Applying only access.yaml without access-operator.yaml first will cause the access Application to fail with CRD-not-found errors until it retries.

Note: The Red Hat ESO (openshift-external-secrets-operator) cannot co-exist with the community ESO. If you have the community operator installed, uninstall it first.

Verification

Check ArgoCD Applications

# List all applications
oc get applications -n openshift-gitops

# Check specific application status
oc describe application <name> -n openshift-gitops

# Watch applications sync
watch oc get applications -n openshift-gitops

Check RHOAI Status

# Check RHOAI operator
oc get csv -n redhat-ods-operator

# Check DataScienceCluster
oc get datasciencecluster

# Check RHOAI pods
oc get pods -n redhat-ods-applications

# Get RHOAI Dashboard URL
RHOAI_URL=$(oc get route rhods-dashboard \
  -n redhat-ods-applications -o jsonpath='{.spec.host}')
echo "RHOAI Dashboard: https://${RHOAI_URL}"

Check GPU Status (if GPU nodes deployed)

# List GPU nodes
oc get nodes -l nvidia.com/gpu.present=true

# Check GPU allocatable resources
oc describe node <gpu-node-name> | grep -A 5 "Allocatable:"

# Check NVIDIA GPU Operator pods
oc get pods -n nvidia-gpu-operator

# Verify GPU device plugin is running
oc get daemonset nvidia-device-plugin-daemonset -n nvidia-gpu-operator

Check MinIO (if deployed)

oc get pods,routes,job -n minio
oc logs job/minio-create-buckets -n minio

Repository Structure

rhoai-deploy/
├── README.md              # This file
├── bootstrap.sh           # GitOps installer script
├── bootstrap/             # GitOps operator manifests
│   └── gitops-operator/
├── gitops/               # ArgoCD Application manifests
│   └── platform/        # RHOAI, GPU Operator, dependencies, optional MinIO
└── platform/            # Platform component definitions
    ├── gitops-operator/ # OpenShift GitOps/ArgoCD (legacy)
    ├── rhoai-operator/  # RHOAI with dependencies (NFD, Kueue)
    ├── nvidia-gpu-operator/  # NVIDIA GPU Operator
    └── storage/minio/   # Optional MinIO (base + overlays/large)

llm-d Distributed Inference

llm-d is a Kubernetes-native framework for serving large language models at scale with prefix-cache-aware routing and disaggregated serving. The following components are included in this repository to enable llm-d on RHOAI.

Prerequisites for llm-d

cert-manager must be installed on the cluster before deploying the LeaderWorkerSet operator (LWS requires cert-manager for webhook TLS certificates). cert-manager is not included here and is assumed to be pre-installed.

# Verify cert-manager is running
oc get pods -n cert-manager

LeaderWorkerSet (LWS) Operator

Located at platform/rhoai-operator/dependencies/lws-operator/. Required by llm-d for LLMInferenceService controller and multi-pod inference workloads (e.g., tensor parallelism across multiple GPUs).

On redhat-operators the package name is leader-worker-set (not lws), channel stable-v1.0, and the operator installs into namespace openshift-lws-operator with its own OperatorGroup — matching what OperatorHub creates when you install manually.

# Verify LWS is installed after ArgoCD sync
oc get subscription,csv -n openshift-lws-operator
oc get crd leaderworkersets.leaderworkerset.x-k8s.io

If you previously synced an incorrect Subscription (name: lws in openshift-operators), delete it so it does not conflict with catalog expectations:

oc delete subscription lws -n openshift-operators --ignore-not-found

Gateway API for llm-d Inference

Located at platform/rhoai-operator/dependencies/gateway-api/. Creates a dedicated openshift-ai-inference Gateway in openshift-ingress that llm-d uses to expose inference endpoints. This is separate from the data-science-gateway (which serves RHOAI dashboard and workbenches).

Reuses the existing data-science-gateway-class GatewayClass (provisioned by RHOAI 3.x)
Uses data-science-gateway-service-tls (same Secret as RHOAI's data-science-gateway) — present in openshift-ingress on typical RHOAI 3.x installs. Many clusters do not have router-certs-default; the default ingress cert is often a cert-manager Secret such as cert-manager-ingress-cert (see IngressController spec.defaultCertificate).
Allows routes from all namespaces so any project's LLMInferenceService can attach

# Verify the inference gateway is programmed after ArgoCD sync
oc get gateway openshift-ai-inference -n openshift-ingress

# If Argo CD shows Degraded but Programmed=True, check listener conditions (ResolvedRefs / Accepted must be True)
oc get gateway openshift-ai-inference -n openshift-ingress -o jsonpath='{.status.listeners}' | jq .

# Pick the correct TLS Secret name for your cluster (then patch gateway-api/gateway.yaml if different):
oc get secret -n openshift-ingress | grep -E 'tls|TLS'
oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='Default ingress cert Secret: {.spec.defaultCertificate.name}{"\n"}'

Both components are included in platform/rhoai-operator/dependencies/kustomization.yaml and will be picked up automatically by the rhoai-platform ArgoCD Application.

Components

Platform Layer

RHOAI Operator - Red Hat OpenShift AI platform

Dashboard, Workbenches, Model Serving, Pipelines
Requires OpenShift 4.19+ for RHOAI 3.x
Currently using fast-3.x subscription channel

RHOAI Dependencies

Node Feature Discovery (NFD) - Hardware feature detection
Red Hat Build for Kueue - Job queuing and resource management
LeaderWorkerSet (LWS) Operator - Multi-pod inference workloads for llm-d
Gateway API (openshift-ai-inference) - Inference endpoint routing for llm-d

NVIDIA GPU Operator - GPU infrastructure

GPU drivers, CUDA runtime, device plugin
DCGM monitoring and metrics

MinIO (optional) - S3-compatible object storage in namespace minio

Separate Argo CD Application gitops/platform/minio.yaml, manual sync
Root Secret created out-of-band; buckets models, data, pipelines via post-deploy Job

See platform/rhoai-operator/README.md and platform/nvidia-gpu-operator/README.md for details.

GPU Infrastructure

For GPU node provisioning and management, see the openshift-infra repository, which provides:

Multi-GPU instance type support (g4dn, g6)
Automated deployment scripts
Cost optimization guidance
GitOps-ready manifests

Customization

Forking the Repository

If you fork this repository, update the repoURL in all GitOps manifests:

# Update all ArgoCD Application manifests
find gitops/ -name "*.yaml" -type f -exec sed -i '' \
  's|repoURL: .*|repoURL: https://github.com/YOUR-ORG/rhoai-deploy|g' {} \;

GPU Instance Configuration

For GPU instance configuration, see the openshift-infra repository.

Troubleshooting

ArgoCD "forbidden" Errors on System Namespaces

Problem: ArgoCD reports errors like:

serviceaccounts "nvidia-dcgm-dashboard-sa" is forbidden: User "system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application-controller" cannot patch resource "serviceaccounts" in API group "" in the namespace "openshift-config-managed"

Root Cause: The openshift-gitops-admin ClusterRoleBinding that grants cluster-admin to the ArgoCD service account was never applied. This happens when OpenShift GitOps was pre-installed by another repository — bootstrap.sh detected it was already running and exited early without applying the binding.

Solution:

# Apply the ClusterRoleBinding (idempotent — safe to run at any time)
oc apply -f bootstrap/gitops-operator/instance/clusterrolebinding.yaml

# Verify
oc get clusterrolebinding openshift-gitops-admin

# Force ArgoCD to re-sync (selfHeal will also trigger it automatically)
oc patch application rhoai-platform -n openshift-gitops \
  --type merge -p '{"operation":{"sync":{}}}'

GPU Scheduling Issues

Problem: Pods fail to schedule with error: 0/N nodes are available: X Insufficient nvidia.com/gpu, Y node(s) didn't match Pod's node affinity/selector

Root Cause: NVIDIA GPU Operator not deployed or GPU device plugin not running.

Solution:

# 1. Verify GPU Operator is deployed
oc get applications -n openshift-gitops | grep nvidia-gpu-operator

# If not deployed, deploy it:
oc apply -f gitops/platform/nvidia-gpu-operator.yaml

# 2. Check GPU Operator pods
oc get pods -n nvidia-gpu-operator

# 3. Verify GPU device plugin is running on GPU nodes
oc get daemonset nvidia-device-plugin-daemonset -n nvidia-gpu-operator

# 4. Check GPU resources are advertised
oc get nodes -o json | jq '.items[] | {name: .metadata.name, gpuAllocatable: .status.allocatable["nvidia.com/gpu"]}'

# 5. If GPU nodes exist but show 0 allocatable GPUs, restart device plugin
oc rollout restart daemonset/nvidia-device-plugin-daemonset -n nvidia-gpu-operator

GPU Nodes Issues

For GPU node provisioning and troubleshooting, see the openshift-infra repository.

# Check GPU nodes exist
oc get nodes -l nvidia.com/gpu.present=true

# Check GPU node labels
oc get nodes --show-labels | grep gpu

RHOAI Not Ready

# Check operator logs
oc logs -l name=rhods-operator -n redhat-ods-operator

# Check DataScienceCluster status
oc describe datasciencecluster default-dsc

# Check RHOAI component pods
oc get pods -n redhat-ods-applications
oc get pods -n redhat-ods-monitoring

ArgoCD Application OutOfSync

# Force sync
oc patch application <name> -n openshift-gitops \
  --type merge -p '{"operation":{"sync":{}}}'

# Check sync status
oc get application <name> -n openshift-gitops -o yaml

MinIO pods or bucket Job failing

Cause: minio-secret missing, wrong keys, or MinIO not ready when the Job ran.

Solution:

oc get secret minio-secret -n minio
oc describe pod -l app=minio -n minio
oc logs job/minio-create-buckets -n minio
# After fixing the Secret or waiting for MinIO to be Ready:
oc delete job minio-create-buckets -n minio --ignore-not-found
oc patch application minio -n openshift-gitops --type merge -p '{"operation":{"sync":{}}}'

Resources

Next Steps

After successful deployment:

Explore RHOAI Dashboard
- Access the dashboard using the URL from verification steps
- Login with your OpenShift credentials
- Create a workbench for data science work
- Access Jupyter notebooks
Deploy AI Models (see rhoai-app-demos repository)
- Download models to storage
- Configure model serving with KServe
- Test inference endpoints
Build Applications (see rhoai-app-demos repository)
- Deploy AnythingLLM for RAG applications
- Set up n8n for workflow automation
- Create custom AI applications

Related Repositories

openshift-infra - GPU infrastructure and MachineSets
rhoai-app-demos - Application-level demos and examples
rhoai-model-serving - Model serving configurations

Name		Name	Last commit message	Last commit date
bootstrap/gitops-operator		bootstrap/gitops-operator
gitops/platform		gitops/platform
platform		platform
.gitignore		.gitignore
README.md		README.md
bootstrap.sh		bootstrap.sh

Latest commit

History

Folders and files

Repository files navigation

RHOAI Deploy

Overview

Prerequisites

Deployment Steps

Step 1: Install OpenShift GitOps (2-3 minutes)

Step 2: Deploy GPU Nodes (Optional, 10-15 minutes)

Step 3: Deploy RHOAI Platform (5-10 minutes)

Step 4: MinIO object storage (Optional)

Step 5: External Secrets Operator and pipeline credentials (Optional)

Verification

Check ArgoCD Applications

Check RHOAI Status

Check GPU Status (if GPU nodes deployed)

Check MinIO (if deployed)

Repository Structure

llm-d Distributed Inference

Prerequisites for llm-d

LeaderWorkerSet (LWS) Operator

Gateway API for llm-d Inference

Components

Platform Layer

GPU Infrastructure

Customization

Forking the Repository

GPU Instance Configuration

Troubleshooting

ArgoCD "forbidden" Errors on System Namespaces

GPU Scheduling Issues

GPU Nodes Issues

RHOAI Not Ready

ArgoCD Application OutOfSync

MinIO pods or bucket Job failing

Resources

Next Steps

Related Repositories

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages