From 485ae56df68e8d2d66dbcbeb341819bb99a188ee Mon Sep 17 00:00:00 2001 From: Gautier DI FOLCO Date: Mon, 10 Aug 2026 16:54:55 +0200 Subject: [PATCH 1/3] fix: gate federated conversation create on assertFullyConnected Conversation.testAddUnreachableUserFromFederatingBackend[domain=other] flaked in CI: the conversation create (Test/Conversation.hs) returned HTTP 533 ({unreachable_backends: []}) instead of 201. The 7s test duration (vs sub-second normally) is consistent with a federation RPC timeout. The freshly-started dynamic backend C was reachable; only the always-on static fed2 was reported unreachable -- a transient reachability blip, not a correctness bug. Root cause: this test's conversation spans two remote backends concurrently -- bob on the static fed2 (otherDomain) and charlie on the dynamic backend C. Conversation creation pings every remote backend concurrently and fails closed with no retry: registerRemoteConversationMemberships -> ensureNoUnreachableBackends =<< runFederatedConcurrentlyEither (Wire/ConversationSubsystem/Util.hs:833-835) which throws UnreachableBackends (-> 533, Wire/API/Error/Galley.hs:583-584) and deletes the half-created conversation (deleteOnUnreachable, Util.hs:897-905). Creation staying strict is deliberate (WPB-5208 relaxed reachability only for adding members), so the fix belongs at the test layer. Add a reusable readiness gate `assertFullyConnected` in API.GalleyInternal (polls getFederationStatus until HTTP 200 + "fully-connected" via `eventually`, which retries on AssertionFailure every 100ms up to Env.timeOutSeconds) and call it before the create. getFederationStatus pings the same backends and reuses the same reachability verdict (ensureNoUnreachableBackends) the create will run, and the gate covers exactly the create's backend set {ownDomain, otherDomain, cDom.berDomain}. The downstream addMembers 533 assertion (targeting the stopped dynamic C) is deterministic and unaffected. A genuine fed2 outage is not masked: if fed2 stays down the gate never reaches fully-connected within eventually's window and the test fails loudly at assertFullyConnected (issuing no postConversation, so no create/delete churn). A residual sub-second TOCTOU race between gate and create remains; if CI shows it still bites, wrap gate+create together with retryT (not eventually, since the create can throw a non-AssertionFailure exception). Runtime trace excerpt: assertion failure: Actual: 533 / Expected: 201 response body: { unreachable_backends: [ ] } --- integration/test/API/GalleyInternal.hs | 18 ++++++++++++++++++ integration/test/Test/Conversation.hs | 4 ++++ 2 files changed, 22 insertions(+) diff --git a/integration/test/API/GalleyInternal.hs b/integration/test/API/GalleyInternal.hs index e6354cf1959..8daebaafe5d 100644 --- a/integration/test/API/GalleyInternal.hs +++ b/integration/test/API/GalleyInternal.hs @@ -85,6 +85,24 @@ getFederationStatus user domains = $ req & addJSONObject ["domains" .= domainList] +-- | Poll (via 'eventually') until the user's backend is fully connected to all +-- the given domains — every domain reachable AND the federation graph among them +-- connected. Absorbs transient reachability blips under dynamic-backend churn +-- (WPB-3797). +assertFullyConnected :: + ( HasCallStack, + MakesValue user + ) => + user -> + [String] -> + App () +assertFullyConnected user domains = + eventually + $ bindResponse (getFederationStatus user domains) + $ \resp -> do + resp.status `shouldMatchInt` 200 + resp.json %. "status" `shouldMatch` "fully-connected" + -- | https://staging-nginz-https.zinfra.io/api-internal/swagger-ui/galley/#/galley/put_i_legalhold_whitelisted_teams__tid_ legalholdWhitelistTeam :: (HasCallStack, MakesValue uid, MakesValue tid) => tid -> uid -> App Response legalholdWhitelistTeam tid uid = do diff --git a/integration/test/Test/Conversation.hs b/integration/test/Test/Conversation.hs index 3dd5615c667..681df7f59fa 100644 --- a/integration/test/Test/Conversation.hs +++ b/integration/test/Test/Conversation.hs @@ -526,6 +526,10 @@ testAddUnreachableUserFromFederatingBackend domain = do otherDomain <- make domain & asString [alice, bob, charlie, chad] <- createAndConnectUsers [ownDomain, otherDomain, cDom.berDomain, cDom.berDomain] + -- Wait for ownDomain <-> {fed2, dynamicC} to be fully connected before + -- creating the conversation, so a transient fed2 reachability blip under + -- dynamic-backend churn does not turn the create into a 533 (WPB-3797). + assertFullyConnected alice [otherDomain, cDom.berDomain] conv <- withWebSockets [bob, charlie] $ \wss -> do conv <- From 6b2c72c70a17e30c04085839865f5cc7079500fd Mon Sep 17 00:00:00 2001 From: Gautier DI FOLCO Date: Mon, 10 Aug 2026 17:07:50 +0200 Subject: [PATCH 2/3] Hello CI From 3dd6dadeae16a1cf672bbb71f47da916a2237150 Mon Sep 17 00:00:00 2001 From: Gautier DI FOLCO Date: Tue, 11 Aug 2026 11:02:58 +0200 Subject: [PATCH 3/3] Update integration/test/API/GalleyInternal.hs Co-authored-by: Sven Tennie --- integration/test/API/GalleyInternal.hs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/integration/test/API/GalleyInternal.hs b/integration/test/API/GalleyInternal.hs index 8daebaafe5d..5df6c94d549 100644 --- a/integration/test/API/GalleyInternal.hs +++ b/integration/test/API/GalleyInternal.hs @@ -85,10 +85,7 @@ getFederationStatus user domains = $ req & addJSONObject ["domains" .= domainList] --- | Poll (via 'eventually') until the user's backend is fully connected to all --- the given domains — every domain reachable AND the federation graph among them --- connected. Absorbs transient reachability blips under dynamic-backend churn --- (WPB-3797). +-- | Poll until the user's backend is fully connected to all given domains assertFullyConnected :: ( HasCallStack, MakesValue user