fix(meta): never reap the single SQLite meta store connection#26125
Closed
yuhao-su wants to merge 1 commit into
Closed
fix(meta): never reap the single SQLite meta store connection#26125yuhao-su wants to merge 1 commit into
yuhao-su wants to merge 1 commit into
Conversation
Move `.idle_timeout(MAX_DURATION).max_lifetime(MAX_DURATION)` from the in-memory branch into `sqlite_common()` so the file-backed SQLite meta store gets them too. Previously the file-backed store kept the config default `idle_timeout_sec = 30`, which keeps sqlx's pool reaper task running. sqlx 0.8.2's `num_idle` counter can transiently underflow to ~2^64 (transact-rs/sqlx#3645, fix pending in transact-rs/sqlx#4289); if the reaper samples the underflow as its sweep-loop bound, it spins `try_acquire` inside a single poll without ever yielding and can permanently wedge the single-connection pool, silently hanging the meta node (#24991). With exactly one connection, idle/lifetime reaping is pointless anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Closing in favor of fixing the root cause directly in the pinned madsim sqlx fork (cherry-picking transact-rs/sqlx#4289's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Mitigation for the CI silent meta hang #24991 (full root-cause analysis: #24991 (comment)).
The file-backed SQLite meta store previously kept the config default
idle_timeout_sec = 30(only the in-memory branch overrode it withMAX_DURATION), which keeps sqlx's background pool-reaper task alive. sqlx 0.8.2 has anum_idleaccounting race —release()publishes the connection and releases the semaphore permit before incrementingnum_idle, so a concurrentpop_idlecan decrement first — that transiently underflows the counter to ~2^64. If the reaper's 30s tick happens to sample the underflow as its sweep-loop bound (for _ in 0..pool.num_idle()), it spinstry_acquireinside a single poll without ever yielding (the loop body has no await;try_acquireconsumes no coop budget). During the spin, oneadd_permitshands the only permit to a queued waiter whose wakeup lands in the spinning worker's LIFO slot — not stealable on tokio 1.49 (tokio-rs/tokio#4941) — so the waiter never runs, the permit stays assigned forever, and the single-connection pool is permanently wedged. Every meta DB access then queues forever (acquire_timeoutis effectively infinite per #18966), presenting as the silent hang:commit_epochstuck ~150s+, DDL frozen, heartbeat timeouts.Upstream status: the same bug is independently reported in transact-rs/sqlx#3645 ("100% CPU in
spawn_maintenance_tasks") with a fix pending in transact-rs/sqlx#4289; it is unmerged and no released sqlx version contains it.This PR removes the ignition site by never expiring the single SQLite connection: move
.idle_timeout(MAX_DURATION).max_lifetime(MAX_DURATION)intosqlite_common()so both the in-memory and file-backed branches get them (the in-memory branch already did this for its own reason — releasing the connection clears the in-memory database). Withmax_connections = 1against a local SQLite file, idle/lifetime reaping serves no purpose anyway.A follow-up will fix the underlying
num_idlerace in the pinned madsim sqlx fork (mirroring transact-rs/sqlx#4289).Note: file-backed SQLite is also the meta store for standalone/single-node deployments, so this hazard is not CI-only; this change is a low-risk cherry-pick candidate for release branches.
Checklist
Documentation
Release note