Skip to content

fix(e2e): improve reliability of operator nightly E2E tests #4521

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zdrapela wants to merge 2 commits into
base: main
Choose a base branch
from
+19 −12
Draft

fix(e2e): improve reliability of operator nightly E2E tests #4521

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9fa6589
fix(e2e): improve reliability of operator nightly E2E tests
zdrapela Apr 1, 2026
2cbf1d9
Merge branch 'main' into fix/e2e-operator-nightly-failures
zdrapela Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions e2e-tests/playwright/e2e/audit-log/log-utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,11 +185,11 @@ export class LogUtils {
static async getPodLogsWithGrep(
filterWords: string[] = [],
namespace: string = process.env.NAME_SPACE || "showcase-ci-nightly",
maxRetries: number = 4,
retryDelay: number = 2000,
maxRetries: number = 6,
retryDelay: number = 3000,
): Promise<string> {
const deploySelector = getBackstageDeploySelector();
const tailNumber = 100;
const tailNumber = 500;

// Resolve the deployment by its metadata labels, then fetch logs from it.
// This works for both Helm and Operator since both set app.kubernetes.io/name
Expand Down
4 changes: 2 additions & 2 deletions e2e-tests/playwright/e2e/plugins/bulk-import.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ spec:
"Ready to import",
]);
}).toPass({
intervals: [5_000],
timeout: 40_000,
intervals: [5_000, 10_000, 15_000],
timeout: 90_000,
});

await bulkimport.selectRepoInTable(catalogRepoDetails.name);
Expand Down
1 change: 1 addition & 0 deletions e2e-tests/playwright/e2e/plugins/notifications/mark-notifications.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ test.describe("Mark notification tests", () => {
};
await notificationsApi.createNotification(notification);
await notificationPage.clickNotificationsNavBarItem();
await notificationPage.notificationContains(`${notificationTitle}-${r}`);
await notificationPage.selectNotification();
await notificationPage.saveSelected();
await notificationPage.viewSaved();
Expand Down
20 changes: 13 additions & 7 deletions e2e-tests/playwright/support/pages/adoption-insights.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,12 +122,15 @@ export class TestHelper {
}

async expectTopEntriesToBePresent(panelTitle: string) {
const panel = this.page.locator(".v5-MuiPaper-root", {
hasText: panelTitle,
});
const entries = panel.locator("tbody").locator("tr");
// Use auto-retrying assertion instead of instant count check
await expect(entries.first()).toBeVisible({ timeout: 30000 });
await expect(async () => {
await this.page.reload();
await this.page.waitForLoadState("domcontentloaded");
const panel = this.page.locator(".v5-MuiPaper-root", {
hasText: panelTitle,
});
const entries = panel.locator("tbody").locator("tr");
await expect(entries.first()).toBeVisible({ timeout: 10000 });
}).toPass({ intervals: [5000, 10000], timeout: 60000 });
Comment on lines +125 to +133
Copy link
Copy Markdown

@rhdh-qodo-merge rhdh-qodo-merge bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Reload in retry loop 🐞 Bug ☼ Reliability

TestHelper.expectTopEntriesToBePresent reloads the page on every toPass() retry and only waits
for domcontentloaded before asserting table rows, which can invalidate state set by prior steps
and can race panel data loading. This makes the Adoption Insights E2E less deterministic and can
still fail (or hide real regressions) depending on network/backend timing.
Agent Prompt 
### Issue description
`expectTopEntriesToBePresent()` currently reloads the page inside a `toPass()` retry and only waits for `domcontentloaded`. This introduces repeated navigations and does not wait for Adoption Insights panel API calls, making the assertion non-deterministic and potentially invalidating state established earlier in the test (filters/date range).

### Issue Context
The Adoption Insights spec already uses `waitForPanelApiCalls(page)` as the readiness signal. The helper should align with that approach rather than forcing reloads.

### Fix Focus Areas
- e2e-tests/playwright/support/pages/adoption-insights.ts[124-134]

### What to change
- Remove the unconditional `page.reload()` from the retry loop.
- Replace it with a deterministic wait:
  - either call `await this.waitForPanelApiCalls(this.page)` before asserting rows, or
  - wait for a specific loading indicator to disappear / panel rows to render with an appropriate timeout.
- If you still need a reload as a last resort, make it conditional (e.g., only if the panel is missing) and then await `waitForPanelApiCalls()` after reload before asserting.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}

async clickAndVerifyText(
Expand All @@ -141,7 +144,10 @@ export class TestHelper {
// Wait for the expected API call to succeed
await this.waitUntilApiCallSucceeds(newpage);

await newpage.getByText(expectedText).first().waitFor({ state: "visible" });
await newpage
.getByText(expectedText)
.first()
.waitFor({ state: "visible", timeout: 30000 });
await newpage.waitForTimeout(5000); // wait for the flush interval to be sure
await newpage.close();
}
Expand Down
Loading