Skip to content

fix(meta): never reap the single SQLite meta store connection#26125

Closed
yuhao-su wants to merge 1 commit into
mainfrom
yuhao/sqlite-meta-disable-pool-reaper
Closed

fix(meta): never reap the single SQLite meta store connection#26125
yuhao-su wants to merge 1 commit into
mainfrom
yuhao/sqlite-meta-disable-pool-reaper

Conversation

@yuhao-su

@yuhao-su yuhao-su commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Mitigation for the CI silent meta hang #24991 (full root-cause analysis: #24991 (comment)).

The file-backed SQLite meta store previously kept the config default idle_timeout_sec = 30 (only the in-memory branch overrode it with MAX_DURATION), which keeps sqlx's background pool-reaper task alive. sqlx 0.8.2 has a num_idle accounting race — release() publishes the connection and releases the semaphore permit before incrementing num_idle, so a concurrent pop_idle can decrement first — that transiently underflows the counter to ~2^64. If the reaper's 30s tick happens to sample the underflow as its sweep-loop bound (for _ in 0..pool.num_idle()), it spins try_acquire inside a single poll without ever yielding (the loop body has no await; try_acquire consumes no coop budget). During the spin, one add_permits hands the only permit to a queued waiter whose wakeup lands in the spinning worker's LIFO slot — not stealable on tokio 1.49 (tokio-rs/tokio#4941) — so the waiter never runs, the permit stays assigned forever, and the single-connection pool is permanently wedged. Every meta DB access then queues forever (acquire_timeout is effectively infinite per #18966), presenting as the silent hang: commit_epoch stuck ~150s+, DDL frozen, heartbeat timeouts.

Upstream status: the same bug is independently reported in transact-rs/sqlx#3645 ("100% CPU in spawn_maintenance_tasks") with a fix pending in transact-rs/sqlx#4289; it is unmerged and no released sqlx version contains it.

This PR removes the ignition site by never expiring the single SQLite connection: move .idle_timeout(MAX_DURATION).max_lifetime(MAX_DURATION) into sqlite_common() so both the in-memory and file-backed branches get them (the in-memory branch already did this for its own reason — releasing the connection clears the in-memory database). With max_connections = 1 against a local SQLite file, idle/lifetime reaping serves no purpose anyway.

A follow-up will fix the underlying num_idle race in the pinned madsim sqlx fork (mirroring transact-rs/sqlx#4289).

Note: file-backed SQLite is also the meta store for standalone/single-node deployments, so this hazard is not CI-only; this change is a low-risk cherry-pick candidate for release branches.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

Move `.idle_timeout(MAX_DURATION).max_lifetime(MAX_DURATION)` from the
in-memory branch into `sqlite_common()` so the file-backed SQLite meta
store gets them too. Previously the file-backed store kept the config
default `idle_timeout_sec = 30`, which keeps sqlx's pool reaper task
running. sqlx 0.8.2's `num_idle` counter can transiently underflow to
~2^64 (transact-rs/sqlx#3645, fix pending in transact-rs/sqlx#4289); if
the reaper samples the underflow as its sweep-loop bound, it spins
`try_acquire` inside a single poll without ever yielding and can
permanently wedge the single-connection pool, silently hanging the meta
node (#24991).

With exactly one connection, idle/lifetime reaping is pointless anyway.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

yuhao-su commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@github-actions github-actions Bot added type/fix Type: Bug fix. Only for pull requests. ci/run-e2e-single-node-tests labels Jul 1, 2026
@yuhao-su yuhao-su marked this pull request as ready for review July 1, 2026 22:37
@yuhao-su yuhao-su added the A-meta Area: Meta node. label Jul 1, 2026
@yuhao-su yuhao-su requested review from hzxa21 and wenym1 July 1, 2026 22:39
@yuhao-su

yuhao-su commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of fixing the root cause directly in the pinned madsim sqlx fork (cherry-picking transact-rs/sqlx#4289's num_idle underflow fix), which covers all backends rather than only removing the SQLite reaper trigger.

@yuhao-su yuhao-su closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-meta Area: Meta node. ci/run-e2e-single-node-tests type/fix Type: Bug fix. Only for pull requests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant