Flaky test report: committed-code failures on 2026-06-08

[andrross]

Flaky test report: committed-code failures on 2026-06-08

Summary

7 test failures were observed across 5 builds in the past 24 hours against committed code (Timer and Post Merge Action runs against main). These failures span 5 distinct test classes. None of the failures reproduced locally with the original seed, indicating they are timing/scheduling-dependent flakes rather than deterministic logic bugs.

Failing Tests

1. MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/90_sig_text/Dedup noise}

Property	Value
Build	#79684
Seed	7894CF9411B24A5C
Error	Aggregation result mismatch: aggregations.class.buckets.0.sig_text.buckets.0.key didn't match expected value
Reproduced locally	❌ No
First failure	2024-03-25
Total unique builds affected	192
Pattern	Chronic flake. Major spike in Sep 2024 (92 builds), then settled to low background rate (1-6/month). Recent months stable at low rate.

2. FullRollingRestartIT.testFullRollingRestart_withNoRecoveryPayloadAndSource {SEGMENT replication}

Property	Value
Build	#79684
Seed	7894CF9411B24A5C
Error	replica shards haven't caught up with primary expected:<16> but was:<13>
Reproduced locally	❌ No
First failure	2024-10-11
Total unique builds affected	138
Pattern	Worsening. Massive spike Jul-Aug 2025 (47+24 builds), quiescent Sep 2025-Jan 2026, then resurgence Feb-Jun 2026 (9-19/month). Likely affected by the Apr 2026 runner migration to m7a.8xlarge.

3. RestoreShallowSnapshotV2IT.testRestoreShallowSnapshotIndexAfterSnapshot {writable_warm_index=true}

Property	Value
Builds	#79696, #79708
Seed	14D504855B045307 (build 79708; build 79696 had no seed in stack trace)
Error	UnavailableShardsException: primary shard is not active Timeout: [1m] / Expected: is <SUCCESS> but: was <PARTIAL>
Reproduced locally	❌ No
First failure	2024-09-03
Total unique builds affected	59
Pattern	Stable low-rate flake (2-6 builds/month). Spike in Apr 2026 (6 builds) possibly correlated with runner change.

4. IndicesRequestCacheCleanupIT.testStaleKeysCleanupWithLowThreshold

Property	Value
Build	#79681
Seed	F01A6594C986D5F6
Error	expected:<1> but was:<2>
Reproduced locally	❌ No
First failure	2024-06-25
Total unique builds affected	8
Pattern	Very rare flake. ~1 failure every 2-3 months. Stable low rate.

5. DiskThresholdDeciderIT.testDiskMonitorAppliesBlockBackWhenUserRemovesIndexCreateBlock {writable_warm_index=false}

Property	Value
Build	#79709
Seed	CB04CCFC9697CA09
Error	java.lang.AssertionError (no message)
Reproduced locally	❌ No
First failure	2025-04-12
Total unique builds affected	7
Pattern	Rare flake. Appeared Apr 2025, sporadic since. Slight uptick May-Jun 2026 (3+1 builds).

6. RemoteStoreMergedSegmentWarmerIT.testConcurrentMergeSegmentWarmerRemote

Property	Value
Build	#79681
Seed	F01A6594C986D5F6
Error	java.lang.AssertionError (no message)
Reproduced locally	❌ No
First failure	2025-07-30
Total unique builds affected	6
Pattern	Rare flake. ~1 failure every 1-2 months since Jul 2025. Stable low rate.

Summary Table (sorted by builds affected)

Test	Builds Affected	First Seen	Trend	Module
MixedClusterClientYamlTestSuiteIT (sig_text/Dedup noise)	192	2024-03	Stable (low)	qa/mixed-cluster
FullRollingRestartIT (SEGMENT replication)	138	2024-10	Worsening	server
RestoreShallowSnapshotV2IT	59	2024-09	Stable (low)	server
IndicesRequestCacheCleanupIT	8	2024-06	Stable (rare)	server
DiskThresholdDeciderIT	7	2025-04	Slightly worsening	server
RemoteStoreMergedSegmentWarmerIT	6	2025-07	Stable (rare)	server

Notes

• None of the failures reproduced locally with their original seeds. This is expected for timing-dependent integration tests. The seeds control Random streams but not thread scheduling, GC pauses, or network timing.
• The MixedClusterClientYamlTestSuiteIT test requires a full BWC cluster setup and cannot be run with a simple ./gradlew command targeting a single test class.
• The FullRollingRestartIT test shows the most concerning trend — it has worsened significantly since the CI runners moved to m7a.8xlarge in April 2026, suggesting CPU-speed amplification of an existing race condition.
• All 5 server-module tests run in internalClusterTest source set and involve multi-node cluster operations, making them inherently sensitive to scheduling and timing.