Flaky test report: committed-code failures on 2026-06-19

[andrross]

Flaky test report: committed-code failures on 2026-06-19

Window scanned: builds in the last 24 hours (2026-06-18 ~10:00 UTC → 2026-06-19 ~10:00 UTC) where invoke_type=Timer & git_reference=main or invoke_type=Post Merge Action & pull_request_title prefix "push trigger main". Source: gradle-check-* indices on metrics.opensearch.org.

Summary table (sorted by historical unique builds affected)

#	Test	Recent build	Reproduced locally?	First failure seen	Unique builds (all)	Unique builds (committed-code only)	Trend
1	org.opensearch.discovery.ClusterDisruptionIT.classMethod	80834	No (full class run, all 6 methods passed, afterClass span validation passed)	2024-04-29	71	28	Chronic, stable. Steady 1–3/month for 2+ years
2	org.opensearch.remotestore.RemoteStorePinnedTimestampsGarbageCollectionIT.testIndexDeletionWithPinnedTimestamps	80846	No (test is @AwaitsFix muted in local repo via opensearch-project#16088, so skipped)	2024-09-17	62	5	Improving overall: peaked at 15 in 2025-04, has tapered to 1/month in 2026
3	org.opensearch.indices.IndicesRequestCacheCleanupIT.testCacheCleanupOnEqualStalenessAndThreshold	80847	No (test passed with original seed)	2024-06-20	11	3	Chronic, low rate. Sporadic (≤2/month), slight uptick in 2026-06 (2 so far)
4	org.opensearch.index.store.remote.utils.TransferManagerBlobContainerReaderTests.testDownloadFailsAsyncDownload	80809	No (test passed with original seed)	2025-09-18	4	1	Stable / low. ~1/month, first committed-code failure was today
5	org.opensearch.storage.action.tiering.TransportPrepareTieringActionTests.testShardOperationAsync_Timeout_FiresMergeDrainTimeoutException	80831	Not attempted — test method does not exist in this fork's main (it was added by upstream commit f5c42dadeed "Harden hot-to-warm tiering and file registry warm read routing", merged 2026-06-17)	2026-06-17	3	3	Worsening / brand new. Failing on every committed-code Timer build since the test method was introduced

Plus one infra-style failure that did not register as a test-case failure:

#	Build	Symptom
–	80828 (Post Merge Action)	:distribution:tools:fips-demo-installer-cli:test — Gradle test executor process finished with non-zero exit value 1. No JUnit case recorded. Treat as a possible JVM/test-process crash; not analyzed further.

---

Per-test detail

1. ClusterDisruptionIT.classMethod — afterClass span-validation flake

• Recent failing build: 80834 (Post Merge Action, push trigger main c5623b162d2e3ef9f8e285b7c31c69086e731b08).

• Failure: AssertionError: SpanData validation failed for validator AllSpansAreEndedProperly. Three dispatchedShardOperationOnPrimary spans on index=restart_while_indexing did not end. Class-level seed: [B311A8DD86D6EF8B].

• Existing autocut issue: #14308 (open since 2024-06-13).

• Local reproduction: ran the entire ClusterDisruptionIT class with -Dtests.seed=B311A8DD86D6EF8B. All 6 methods passed and the afterClass span validation passed. Failure did not reproduce on first attempt.

• Historical pattern (all builds, monthly unique-builds-affected):

  2024-04: 1   2024-06: 7   2024-07: 4   2024-08: 1   2024-09: 3   2024-10: 4
  2025-01: 2   2025-02: 2   2025-04: 3   2025-05: 2   2025-06: 5   2025-07: 4
  2025-08: 4   2025-09: 2   2025-10: 4   2025-11: 4   2025-12: 3
  2026-01: 3   2026-02: 2   2026-03: 1   2026-04: 2   2026-05: 3   2026-06: 5 (so far)
  

• Pattern: chronic and stable. Persistent low-rate flake in PR + committed-code builds for over 2 years. Note that the failure is in afterClass (StrictCheckSpanProcessor.validateTracingStateOnShutdown) rather than in any specific test method, so the seed is class-level only and reproducibility in a single run is expected to be poor (per the steering note about disruption tests).

2. RemoteStorePinnedTimestampsGarbageCollectionIT.testIndexDeletionWithPinnedTimestamps

• Recent failing build: 80846 (Timer, git_reference=main).

• Failure: AssertionError: expected:<3> but was:<4> inside assertBusy at RemoteStorePinnedTimestampsGarbageCollectionIT.java:483. Seed: [D6126B97FB838CC0:418E8E743B933CDD].

• Existing autocut issue: #16088 (open since 2024-09-26).

• Local reproduction: skipped — the test is annotated @AwaitsFix(bugUrl = "https://github.com/opensearch-project/OpenSearch/issues/16088") in the local fork, so it does not execute under internalClusterTest. The CI build that failed is running upstream code where the mute is not present (or has been removed).

• Historical pattern (monthly unique-builds-affected, all build types):

  2024-09: 1   2024-10: 9   2024-11: 1   2024-12: 4
  2025-01: 1   2025-02: 3   2025-03: 11  2025-04: 15  2025-05: 7
  2025-07: 1   2025-08: 1   2025-09: 1   2025-10: 2   2025-12: 1
  2026-02: 1   2026-03: 1   2026-04: 1   2026-06: 1
  

• Pattern: improving / declining. Peaked at 15 unique builds in 2025-04, then steadily tapered to ~1/month in 2026. Today's failure (80846) is the first committed-code failure since 2025-04.

3. IndicesRequestCacheCleanupIT.testCacheCleanupOnEqualStalenessAndThreshold

• Recent failing build: 80847 (Timer, git_reference=main).

• Failure: AssertionError: expected:<1> but was:<2> inside assertBusy at IndicesRequestCacheCleanupIT.java:240. Seed: [17D758AB69C632A4:4303AE945153784A].

• Existing autocut issues: #15822 (open since 2024-09), and #21397 (closed 2026-04).

• Local reproduction: ran with original seed (-Dtests.seed=17D758AB69C632A4:4303AE945153784A). Test passed (1 test, 0 failures, 4.6 s). Did not reproduce on first attempt.

• Historical pattern (monthly unique-builds-affected, all build types):

  2024-06: 1   2024-09: 2   2025-01: 2   2025-06: 1
  2025-09: 1   2025-11: 1   2026-05: 1   2026-06: 2 (so far)
  

• Pattern: chronic, low-rate. About 1–2 unique builds per affected month for 2 years. 2026-06 has 2 builds already — slight uptick — but volume is low and the test class has been refactored once (was IndicesRequestCacheIT.testCacheCleanupOnEqualStalenessAndThreshold).

4. TransferManagerBlobContainerReaderTests.testDownloadFailsAsyncDownload

• Recent failing build: 80809 (Timer, git_reference=main).
• Failure: AssertionFailedError: Unexpected exception type, expected IOException but got AlreadyClosedException. The test body is in the parent TransferManagerTestCase.testDownloadFailsAsyncDownload. Seed: [EBD9EAA60A7FE85F:EB62B46DFB2584F5].
• Existing autocut issue: #18872 (open since 2025-07).
• Local reproduction: ran with original seed (-Dtests.seed=EBD9EAA60A7FE85F:EB62B46DFB2584F5). Test passed. Did not reproduce on first attempt.
• Historical pattern (monthly unique-builds-affected): 2025-09: 1, 2026-01: 1, 2026-05: 1, 2026-06: 1.
• Pattern: stable, very low rate (~1/month). The 2026-06 failure is the first committed-code failure for this test; earlier failures were all on PR builds.

5. TransportPrepareTieringActionTests.testShardOperationAsync_Timeout_FiresMergeDrainTimeoutException

• Recent failing builds: 80831 and 80809 (both Timer, git_reference=main).
• Failure: Mockito WantedButNotInvoked — releasable.close() expected but never called (TransportPrepareTieringActionTests.java:562). Seeds: [2B0132309D995608:B255A8EAC4478F2E] (80831), [EBD9EAA60A7FE85F:728D707C53A13179] (80809).
• Existing autocut issue: #22216 (open, created 2026-06-17 — same day as the first failure).
• Local reproduction: not attempted in this fork. The test method does not exist in the local repository's main. It was added by upstream commit f5c42dadeed ("Harden hot-to-warm tiering and file registry warm read routing", Harden hot-to-warm tiering and file registry warm read routing opensearch-project/OpenSearch#22103) on 2026-06-17. The local fork has only the original 8 methods of TransportPrepareTieringActionTests — none of the 13 new testShardOperationAsync_* methods. Reproducing would require checking out upstream main (or the PR branch).
• Historical pattern: all 3 known failures are committed-code Timer builds, all from 2026-06. The test has been failing on every committed-code run since it was introduced.
• Pattern: brand-new / worsening. The test broke immediately upon introduction. This looks more like a genuine concurrency bug in the new code or in the new test than a pre-existing flake. Worth prioritizing over the chronic flakes.

---

Method notes

• "Reproduced locally?" reflects only a single run with the exact seed reported by Jenkins. Per the project's flaky-test guidance, a single passing run with the original seed is not proof that the failure is non-reproducible — RandomizedRunner seeds the Random stream but cannot control thread scheduling, GC pauses, or network/disruption timing. For tests Bump com.diffplug.spotless from 5.6.1 to 6.2.0 #1–Bump opensearch-core from current to 1.2.4 in /buildSrc/src/testKit/opensearch.build #4, the more honest statement is "did not reproduce on first attempt with the original seed; given the failure modes (assertBusy, span-validation in afterClass, mock-vs-async ordering), seed-determinism is unlikely."
• The local repository (andrross/OpenSearch working tree at HEAD 90fdbf95a0f) is a fork that diverges from upstream opensearch-project/OpenSearch main (current 128652a16011). One test (Bump com.diffplug.spotless from 5.6.1 to 6.3.0 #5) is missing entirely in the fork, and one (Bump com.diffplug.spotless from 5.6.1 to 6.2.1 #2) is muted via @AwaitsFix in the fork but not in the upstream tree the CI build was running against. Local reproduction results should be read in light of that.
• All 5 distinct failing tests already have open [AUTOCUT] tracking issues in opensearch-project/OpenSearch. No new tracking issues are needed; this report is informational.