docs: add CI Medic Guide and fix stale CI documentation

.ci/pipelines/README.md

@@ -2,17 +2,16 @@
 
 ## Overview
 
-The RHDH deployment for end-to-end (e2e) tests in CI has been updated to use **ephemeral clusters**
-on OpenShift Container Platform (OCP) instead of persistent clusters.
+The RHDH deployment for end-to-end (e2e) tests in CI uses **ephemeral clusters** on OpenShift
+Container Platform (OCP).
 
-### Key Updates
+### Key Details
 
-- Starting from version **1.5**, ephemeral clusters are used for:
-  - OCP nightly jobs (v4.17, v4.16, and v4.14).
-  - PR checks on the main branch.
-- Previously, RHDH PR checks utilized persistent clusters created on IBM Cloud.
-- Now, ephemeral clusters are provisioned using the **OpenShift CI cluster claim** on AWS via the
+- Ephemeral OCP clusters are provisioned using the **OpenShift CI cluster claim** on AWS via the
   RHDH-QE account in the `us-east-2` region.
+- Used for OCP nightly jobs (multiple OCP versions) and PR checks on the main branch.
+- Non-OCP platforms (AKS, EKS) use ephemeral clusters provisioned by
+  [Mapt](https://github.com/redhat-developer/mapt). GKE uses a long-running shared cluster.
 
 ---
 
@@ -28,23 +27,12 @@ To access ephemeral clusters, you must:
 
 ## Cluster Pools
 
-The following cluster pools are available for different OCP versions:
+RHDH uses dedicated Hive cluster pools with the `rhdh` prefix. Pool versions rotate as new OCP
+releases come out.
 
-- **RHDH-4-19-US-EAST-2**
-  - Usage: OCP v4.19 nightly jobs.
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-19-0-amd64-aws-us-east-2_clusterpool.yaml).
-
-- **RHDH-4-18-US-EAST-2**
-  - Usage: OCP v4.18 nightly jobs.
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-18-0-amd64-aws-us-east-2_clusterpool.yaml).
-
-- **RHDH-4-17-US-EAST-2**
-  - Usage: PR checks on the main branch and OCP v4.17 nightly jobs.
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-17-0-amd64-aws-us-east-2_clusterpool.yaml).
-
-- **RHDH-4-16-US-EAST-2**
-  - Usage: OCP v4.16 nightly jobs.
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-16-0-amd64-aws-us-east-2_clusterpool.yaml).
+To find the current list of available pools, filter for `rhdh` in the
+[existing cluster pools](https://docs.ci.openshift.org/how-tos/cluster-claim/#existing-cluster-pools)
+page.
 
 ---
 
@@ -87,7 +75,7 @@ ephemeral environment credentials.
    .ci/pipelines/ocp-cluster-claim-login.sh
    ```
 2. Provide the Prow log URL when prompted, for example:
-   `https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/janus-idp_backstage-showcase/2089/pull-ci-janus-idp-backstage-showcase-main-e2e-tests/1866766753132974080 `
+   `https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-redhat-developer-rhdh-main-e2e-ocp-helm-nightly/<BUILD_ID>`
 3. The script will:
    - Log in to the hosted-mgmt cluster, which manages ephemeral cluster creation.
    - Retrieve admin credentials and log in to the ephemeral cluster.
@@ -153,20 +141,8 @@ yarn shellcheck
 
 ### Modular Architecture
 
-Pipeline utilities are organized into modules in `.ci/pipelines/lib/`:
-
-- `log.sh` - Logging functions
-- `common.sh` - Common utilities (oc_login, sed_inplace, etc.)
-- `k8s-wait.sh` - Kubernetes wait/polling operations
-- `operators.sh` - Operator installations
-
-Usage example:
-
-```bash
-# Using modular functions
-k8s_wait::deployment "namespace" "deployment"
-common::oc_login
-operator::install_pipelines
-```
+Pipeline utilities are organized into modules in `.ci/pipelines/lib/`. See
+[`lib/README.md`](lib/README.md) for the full list of modules, function signatures, and conventions.
 
-See `lib/README.md` for module details.
+For detailed triage and failure investigation, see the
+[CI Medic Guide](../../docs/e2e-tests/CI-medic-guide.md).

.claude/memories/ci-e2e-testing.md

@@ -297,7 +297,7 @@ Available cluster pools for different OCP versions:
 #### Nightly Tests
 - **Schedule**: Automated nightly runs
 - **Environments**: Multiple OCP versions, AKS, GKE
-- **Reporting**: Slack notifications to `#rhdh-e2e-test-alerts`
+- **Reporting**: Slack notifications to `#rhdh-e2e-alerts`
 
 ### Test Execution Environment
 

.claude/rules/ci-e2e-testing.md

@@ -77,6 +77,7 @@ test.beforeAll(async ({ }, testInfo) => {
    - `showcase-operator`: General functionality tests with base deployment using Operator
    - `showcase-operator-rbac`: General functionality tests with RBAC-enabled deployment using Operator
    - `showcase-runtime`: Runtime environment tests
+   - `showcase-runtime-db`: Runtime database tests
    - `showcase-sanity-plugins`: Plugin sanity checks
    - `showcase-upgrade`: Upgrade scenario tests
    - `showcase-localization-fr`: French localization tests
@@ -125,18 +126,18 @@ test.beforeAll(async ({ }, testInfo) => {
 
 #### CI/CD Pipeline Execution
 
-In the CI/CD pipeline, tests are executed directly using Playwright's `--project` flag via the `run_tests()` function in `.ci/pipelines/utils.sh`:
+In the CI/CD pipeline, tests are executed directly using Playwright's `--project` flag via `testing::run_tests()` in `.ci/pipelines/lib/testing.sh`:
 
 ```bash
 yarn playwright test --project="${playwright_project}"
 ```
 
-The namespace and Playwright project are decoupled, allowing flexible reuse. The `check_and_test()` and `run_tests()` functions accept an explicit `playwright_project` argument:
+The namespace and Playwright project are decoupled, allowing flexible reuse. The `testing::check_and_test()` and `testing::run_tests()` functions accept an explicit `playwright_project` argument:
 
 ```bash
 # Function signatures:
-check_and_test "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}" [max_attempts] [wait_seconds]
-run_tests "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}"
+testing::check_and_test "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}" [max_attempts] [wait_seconds]
+testing::run_tests "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}"
 ```
 
 #### Local Development Scripts
@@ -266,25 +267,7 @@ Check the readme at `.ci/pipelines/README.md`
 
 ### Cluster Pools
 
-Available cluster pools for different OCP versions:
-
-- **RHDH-4-19-US-EAST-2**
-  - Usage: OCP v4.19 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-19-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-18-US-EAST-2**
-  - Usage: OCP v4.18 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-18-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-17-US-EAST-2**
-  - Usage: PR checks on main branch and OCP v4.17 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-17-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-16-US-EAST-2**
-  - Usage: OCP v4.16 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-16-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-**Note:** This is subject to change. Use `.ci/pipelines/README.md` as a source of truth.
+RHDH uses dedicated Hive cluster pools with the `rhdh` prefix on AWS `us-east-2`. Pool versions rotate as new OCP releases come out. Find the current list by filtering for `rhdh` in the [existing cluster pools](https://docs.ci.openshift.org/how-tos/cluster-claim/#existing-cluster-pools) page. See also `.ci/pipelines/README.md`.
 
 ### CI Job Types
 
@@ -297,7 +280,7 @@ Available cluster pools for different OCP versions:
 #### Nightly Tests
 - **Schedule**: Automated nightly runs
 - **Environments**: Multiple OCP versions, AKS, GKE
-- **Reporting**: Slack notifications to `#rhdh-e2e-test-alerts`
+- **Reporting**: Slack notifications to `#rhdh-e2e-alerts`
 
 ### Test Execution Environment
 
@@ -347,14 +330,15 @@ yarn prettier:fix            # Fix formatting for shell, markdown, and YAML file
 - Functions may temporarily disable/re-enable error handling with `set +e` / `set -e` pattern
 
 #### Job Handlers
-The main script handles different job types:
-- `handle_aks_helm`: AKS Helm deployment
-- `handle_eks_helm`: EKS Helm deployment
-- `handle_gke_helm`: GKE Helm deployment
-- `handle_ocp_operator`: OCP Operator deployment
-- `handle_ocp_nightly`: OCP nightly tests
-- `handle_ocp_pull`: OCP PR tests
+The main script dispatches to job-specific handlers in `.ci/pipelines/jobs/`:
+- `handle_aks_helm` / `handle_aks_operator`: AKS Helm/Operator deployment
+- `handle_eks_helm` / `handle_eks_operator`: EKS Helm/Operator deployment
+- `handle_gke_helm` / `handle_gke_operator`: GKE Helm/Operator deployment
+- `handle_ocp_nightly`: OCP Helm nightly tests (also handles OSD-GCP Helm)
+- `handle_ocp_operator`: OCP Operator nightly tests (also handles OSD-GCP Operator)
+- `handle_ocp_pull`: OCP PR checks
 - `handle_auth_providers`: Auth provider tests
+- `handle_ocp_helm_upgrade`: Upgrade scenario tests
 
 #### Special Case: showcase-auth-providers Deployment
 
@@ -544,5 +528,3 @@ The choice of config map depends on the **Playwright test project** being execut
 ### **Configuration Deployment Process**
 
 The config maps are deployed as Kubernetes ConfigMaps during CI/CD pipeline execution and are mounted into the RHDH pods to provide runtime configuration. The pipeline selects the appropriate config map based on the test project being executed.
-
----

.cursor/rules/ci-e2e-testing.mdc

@@ -80,6 +80,7 @@ test.beforeAll(async ({ }, testInfo) => {
    - `showcase-operator`: General functionality tests with base deployment using Operator
    - `showcase-operator-rbac`: General functionality tests with RBAC-enabled deployment using Operator
    - `showcase-runtime`: Runtime environment tests
+   - `showcase-runtime-db`: Runtime database tests
    - `showcase-sanity-plugins`: Plugin sanity checks
    - `showcase-upgrade`: Upgrade scenario tests
    - `showcase-localization-fr`: French localization tests
@@ -128,18 +129,18 @@ test.beforeAll(async ({ }, testInfo) => {
 
 #### CI/CD Pipeline Execution
 
-In the CI/CD pipeline, tests are executed directly using Playwright's `--project` flag via the `run_tests()` function in `.ci/pipelines/utils.sh`:
+In the CI/CD pipeline, tests are executed directly using Playwright's `--project` flag via `testing::run_tests()` in `.ci/pipelines/lib/testing.sh`:
 
 ```bash
 yarn playwright test --project="${playwright_project}"
 ```
 
-The namespace and Playwright project are decoupled, allowing flexible reuse. The `check_and_test()` and `run_tests()` functions accept an explicit `playwright_project` argument:
+The namespace and Playwright project are decoupled, allowing flexible reuse. The `testing::check_and_test()` and `testing::run_tests()` functions accept an explicit `playwright_project` argument:
 
 ```bash
 # Function signatures:
-check_and_test "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}" [max_attempts] [wait_seconds]
-run_tests "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}"
+testing::check_and_test "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}" [max_attempts] [wait_seconds]
+testing::run_tests "${RELEASE_NAME}" "${NAMESPACE}" "${PLAYWRIGHT_PROJECT}" "${URL}"
 ```
 
 #### Local Development Scripts
@@ -269,25 +270,7 @@ Check the readme at `.ci/pipelines/README.md`
 
 ### Cluster Pools
 
-Available cluster pools for different OCP versions:
-
-- **RHDH-4-19-US-EAST-2**
-  - Usage: OCP v4.19 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-19-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-18-US-EAST-2**
-  - Usage: OCP v4.18 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-18-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-17-US-EAST-2**
-  - Usage: PR checks on main branch and OCP v4.17 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-17-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-- **RHDH-4-16-US-EAST-2**
-  - Usage: OCP v4.16 nightly jobs
-  - [Cluster Pool Configuration](https://github.com/openshift/release/blob/master/clusters/hosted-mgmt/hive/pools/rhdh/rhdh-ocp-4-16-0-amd64-aws-us-east-2_clusterpool.yaml)
-
-**Note:** This is subject to change. Use `.ci/pipelines/README.md` as a source of truth.
+RHDH uses dedicated Hive cluster pools with the `rhdh` prefix on AWS `us-east-2`. Pool versions rotate as new OCP releases come out. Find the current list by filtering for `rhdh` in the [existing cluster pools](https://docs.ci.openshift.org/how-tos/cluster-claim/#existing-cluster-pools) page. See also `.ci/pipelines/README.md`.
 
 ### CI Job Types
 
@@ -300,7 +283,7 @@ Available cluster pools for different OCP versions:
 #### Nightly Tests
 - **Schedule**: Automated nightly runs
 - **Environments**: Multiple OCP versions, AKS, GKE
-- **Reporting**: Slack notifications to `#rhdh-e2e-test-alerts`
+- **Reporting**: Slack notifications to `#rhdh-e2e-alerts`
 
 ### Test Execution Environment
 
@@ -350,14 +333,15 @@ yarn prettier:fix            # Fix formatting for shell, markdown, and YAML file
 - Functions may temporarily disable/re-enable error handling with `set +e` / `set -e` pattern
 
 #### Job Handlers
-The main script handles different job types:
-- `handle_aks_helm`: AKS Helm deployment
-- `handle_eks_helm`: EKS Helm deployment
-- `handle_gke_helm`: GKE Helm deployment
-- `handle_ocp_operator`: OCP Operator deployment
-- `handle_ocp_nightly`: OCP nightly tests
-- `handle_ocp_pull`: OCP PR tests
+The main script dispatches to job-specific handlers in `.ci/pipelines/jobs/`:
+- `handle_aks_helm` / `handle_aks_operator`: AKS Helm/Operator deployment
+- `handle_eks_helm` / `handle_eks_operator`: EKS Helm/Operator deployment
+- `handle_gke_helm` / `handle_gke_operator`: GKE Helm/Operator deployment
+- `handle_ocp_nightly`: OCP Helm nightly tests (also handles OSD-GCP Helm)
+- `handle_ocp_operator`: OCP Operator nightly tests (also handles OSD-GCP Operator)
+- `handle_ocp_pull`: OCP PR checks
 - `handle_auth_providers`: Auth provider tests
+- `handle_ocp_helm_upgrade`: Upgrade scenario tests
 
 #### Special Case: showcase-auth-providers Deployment
 
@@ -547,5 +531,3 @@ The choice of config map depends on the **Playwright test project** being execut
 ### **Configuration Deployment Process**
 
 The config maps are deployed as Kubernetes ConfigMaps during CI/CD pipeline execution and are mounted into the RHDH pods to provide runtime configuration. The pipeline selects the appropriate config map based on the test project being executed.
-
----

.gitignore

@@ -42,6 +42,7 @@ site
 
 # Local configuration files
 *.local.yaml
+*.local.md
 
 # Sensitive credentials
 *-credentials.local.yaml