NO-JIRA: Fix flaky CI unit tests: reduce parallelism

[stefanonardo]

Summary

Unit tests have been frequently failing in CI due to envtest resource contention. Analysis of 19 recent failed builds (across PRs #577, #591, #622, #627, #634, #635, #636, #637, #638, #640, #642) shows the root cause: --procs=4 runs 4 test suites simultaneously, each starting its own etcd + kube-apiserver (8 heavy processes total), causing widespread timeouts on resource-constrained CI nodes.

This PR:

• Reduces CI parallelism from --procs=4 to --procs=2, halving resource contention
• Increases the machinesetsync Eventually timeout from 2s to 5s (the most frequently failing suite: 22 of 57 failures)

Why not just increase timeouts?

We tested both approaches on CI:

Approach	Runs	Pass rate	Test step time
procs=2 + machinesetsync 5s	2	2/2	11.9-15.3m
procs=4 + all timeout increases	3	3/3	9.1-11.9m
Baseline procs=4 (no changes)	28	~64%	9.5-20.6m

We chose procs=2 because it addresses the root cause (resource contention) rather than masking symptoms with larger timeouts, and is more robust against future test additions. The wall-clock cost (~3-5 min) is small compared to the cost of frequent CI retries.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
  • Increased the default test wait timeout from 2 to 5 seconds.
  • Updated OpenShift CI test execution to use four parallel processes for end-to-end tests and two for other test suites.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 28da1f03-35ad-4454-bd4c-2d862896189f

📥 Commits

Reviewing files that changed from the base of the PR and between d67e0da and 7133b07.

📒 Files selected for processing (1)

• hack/test.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• hack/test.sh

---

Walkthrough

The pull request increases the machine set sync test timeout from 2 to 5 seconds. OpenShift CI uses four Ginkgo processes for e2e tests and two for other test suites.

Changes

Test execution settings

Layer / File(s)	Summary
Timeout and parallelism settings pkg/controllers/machinesetsync/suite_test.go, hack/test.sh	The machine set sync suite uses a 5-second eventual timeout. OpenShift CI uses four Ginkgo processes for e2e tests and two for other test suites.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Mergeability Score: ⚪ Minimal · up to 7133b

This PR reduces CI test contention and adjusts a flaky test timeout; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: nrb, radekmanak

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The diff changes only the Eventually timeout and CI process selection; it adds no Ginkgo title declarations or dynamic title construction.
Test Structure And Quality	✅ Passed	The PR only raises an existing default Eventually timeout and adjusts CI process count; it introduces no new test behavior, resource creation, waits, or assertions violating the stated quality requ...
Microshift Test Compatibility	✅ Passed	The diff adds no Ginkgo e2e tests or API references; it only changes CI parallelism and an existing machinesetsync timeout, so this check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The HEAD^→HEAD diff changes only CI parallelism and a machinesetsync timeout; it adds no It, Describe, Context, or When e2e tests, so SNO compatibility is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff changes only test timeout configuration and Ginkgo CI process counts; it adds no deployment, operator, controller, affinity, topology spread, node selector, toleration, replica, or PDB sch...
Ote Binary Stdout Contract	✅ Passed	The diff only changes Ginkgo process count and a timeout; it adds no stdout writes. Existing BeforeSuite redirects klog to GinkgoWriter.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The diff only changes CI Ginkgo parallelism and a unit-suite timeout; it adds no new e2e Ginkgo tests, IPv4 assumptions, or external connectivity.
No-Weak-Crypto	✅ Passed	The diff only changes Ginkgo process selection and a test timeout; it adds no weak crypto, custom crypto implementation, or secret/token comparison.
Container-Privileges	✅ Passed	The commit changes only Ginkgo process selection and a test timeout; no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or root setting is introduced.
No-Sensitive-Data-In-Logs	✅ Passed	The diff only changes Ginkgo process counts, comments, and a timeout; it adds no logging or output of passwords, tokens, PII, hostnames, or customer data.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: reducing CI test parallelism and addressing flaky unit tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[stefanonardo]

/test unit

[stefanonardo]

/test unit

[pmeida]

/lgtm

Looks like a suitable fix without much more time cost

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-capi-disconnected-techpreview
/test e2e-aws-capi-techpreview
/test e2e-aws-capi-techpreview-post-install
/test e2e-aws-ovn-techpreview
/test e2e-aws-ovn-techpreview-upgrade
/test e2e-azure-capi-techpreview
/test e2e-azure-ovn-techpreview
/test e2e-azure-ovn-techpreview-upgrade
/test e2e-gcp-capi-techpreview
/test e2e-gcp-ovn-techpreview
/test e2e-metal3-capi-techpreview
/test e2e-openstack-capi-techpreview
/test e2e-vsphere-capi-techpreview
/test regression-clusterinfra-aws-ipi-techpreview-capi

[damdo]

/override ci/prow/e2e-aws-capi-disconnected-techpreview ci/prow/e2e-aws-capi-techpreview ci/prow/e2e-aws-capi-techpreview-post-install ci/prow/e2e-aws-ovn-techpreview ci/prow/e2e-aws-ovn-techpreview-upgrade ci/prow/e2e-azure-ovn-techpreview ci/prow/e2e-azure-ovn-techpreview-upgrade ci/prow/e2e-gcp-capi-techpreview ci/prow/e2e-gcp-ovn-techpreview ci/prow/e2e-metal3-capi-techpreview ci/prow/e2e-openstack-capi-techpreview ci/prow/regression-clusterinfra-aws-ipi-techpreview-capi

[damdo]

/skip

[openshift-ci]

@damdo: Overrode contexts on behalf of damdo: ci/prow/e2e-aws-capi-disconnected-techpreview, ci/prow/e2e-aws-capi-techpreview, ci/prow/e2e-aws-capi-techpreview-post-install, ci/prow/e2e-aws-ovn-techpreview, ci/prow/e2e-aws-ovn-techpreview-upgrade, ci/prow/e2e-azure-ovn-techpreview, ci/prow/e2e-azure-ovn-techpreview-upgrade, ci/prow/e2e-gcp-capi-techpreview, ci/prow/e2e-gcp-ovn-techpreview, ci/prow/e2e-metal3-capi-techpreview, ci/prow/e2e-openstack-capi-techpreview, ci/prow/regression-clusterinfra-aws-ipi-techpreview-capi

Details

In response to this:

/override ci/prow/e2e-aws-capi-disconnected-techpreview ci/prow/e2e-aws-capi-techpreview ci/prow/e2e-aws-capi-techpreview-post-install ci/prow/e2e-aws-ovn-techpreview ci/prow/e2e-aws-ovn-techpreview-upgrade ci/prow/e2e-azure-ovn-techpreview ci/prow/e2e-azure-ovn-techpreview-upgrade ci/prow/e2e-gcp-capi-techpreview ci/prow/e2e-gcp-ovn-techpreview ci/prow/e2e-metal3-capi-techpreview ci/prow/e2e-openstack-capi-techpreview ci/prow/regression-clusterinfra-aws-ipi-techpreview-capi

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[damdo]

/approve
/lgtm

/skip

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-capi-disconnected-techpreview
/test e2e-aws-capi-techpreview
/test e2e-aws-capi-techpreview-post-install
/test e2e-aws-ovn-techpreview
/test e2e-aws-ovn-techpreview-upgrade
/test e2e-azure-capi-techpreview
/test e2e-azure-ovn-techpreview
/test e2e-azure-ovn-techpreview-upgrade
/test e2e-gcp-capi-techpreview
/test e2e-gcp-ovn-techpreview
/test e2e-metal3-capi-techpreview
/test e2e-openstack-capi-techpreview
/test e2e-vsphere-capi-techpreview
/test regression-clusterinfra-aws-ipi-techpreview-capi

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: damdo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [damdo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[damdo]

/skip

[openshift-ci-robot]

@stefanonardo: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

Unit tests have been frequently failing in CI due to envtest resource contention. Analysis of 19 recent failed builds (across PRs #577, #591, #622, #627, #634, #635, #636, #637, #638, #640, #642) shows the root cause: --procs=4 runs 4 test suites simultaneously, each starting its own etcd + kube-apiserver (8 heavy processes total), causing widespread timeouts on resource-constrained CI nodes.

This PR:

• Reduces CI parallelism from --procs=4 to --procs=2, halving resource contention
• Increases the machinesetsync Eventually timeout from 2s to 5s (the most frequently failing suite: 22 of 57 failures)

Why not just increase timeouts?

We tested both approaches on CI:

Approach	Runs	Pass rate	Test step time
procs=2 + machinesetsync 5s	2	2/2	11.9-15.3m
procs=4 + all timeout increases	3	3/3	9.1-11.9m
Baseline procs=4 (no changes)	28	~64%	9.5-20.6m

We chose procs=2 because it addresses the root cause (resource contention) rather than masking symptoms with larger timeouts, and is more robust against future test additions. The wall-clock cost (~3-5 min) is small compared to the cost of frequent CI retries.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
• Increased the default test wait timeout from 2 to 5 seconds.
• Updated OpenShift CI test execution to use four parallel processes for end-to-end tests and two for other test suites.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[stefanonardo]

/retest

[openshift-ci]

@stefanonardo: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[stefanonardo]

/verified by @stefanonardo

[openshift-ci-robot]

@stefanonardo: This PR has been marked as verified by @stefanonardo.

Details

In response to this:

/verified by @stefanonardo

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.