fix(store): serialize snapshots and compactions per database

[corylanou]

Summary

Store.Open runs one goroutine per compaction level plus one for the snapshot level, and CompactDB did nothing to serialize work per database, so an L9 snapshot and an L1 compaction can run at the same time on the same database. Each holds a database-sized LTX page index, so peak memory multiplies with database size; a 130 GiB database was OOM-killed in a 12 GiB container this way (#1477).

• Store.CompactDB now takes a per-database TryLock. A refused operation returns the new ErrMaintenanceBusy instead of queueing, so one database's long snapshot cannot stall the level monitor for every other database.
• The level monitor treats busy like "no compaction" but wakes early to retry: 1s after the first refusal, doubling up to MaintenanceBusyRetryInterval (10s) while the database stays busy. An early pass only re-attempts the databases that were refused (plus any registered since the last regular pass) and skips snapshot retention; regular passes stay anchored to the level's schedule. Without the early retry a snapshot that lost startup contention to L1 would wait for the next snapshot boundary (up to 24h); without the 1s start, levels whose boundaries coincide (e.g. L1 15s / L2 30s in the LTX behavioral gate) would drift by a full 10s on every collision.
• The snapshot reader's Close now cancels and joins the producer goroutine, so a snapshot whose replica write fails early cannot leave its WAL page map and encoder alive after the lock is released.
• Compaction across different databases is unaffected (covered by the new test).

This bounds concurrent maintenance memory to one operation per database. The page indexes themselves still scale with database size; shrinking the encoder's index is superfly/ltx#95, and this PR pins go.mod to that branch's head so it builds and soaks as the intended final combination. The pin must be replaced with the tagged ltx release before merge.

Fixes #1477

Evidence

Measured with the litestream-soak snapshot-compaction-overlap rig (corylanou/litestream-soak#196, corylanou/litestream-soak#197): it runs snapshot and L1 compaction sequentially as a baseline, then holds the snapshot's replica stream at 95% so the encoder's page index stays resident while the L1 compaction runs, sampling runtime.MemStats (GOGC=25) and writing a heap profile at each phase's peak. "Overlap" is the peak heap growth during that phase; the pass limit is 1.10× the larger sequential phase plus one multipart upload's fixed buffers. The "this PR" rows were built by the rig straight from this PR's head (go.mod as committed, ltx pinned to superfly/ltx#95), not via a local replace.

build	DB	seq snapshot	seq compact-l1	overlap	limit	result
main 3b468c9 + ltx v0.5.2	1 GiB / 262,803 pages	69.0 MB	101.3 MB	170.0 MB	145.0 MB	fail (overlap/sum = 1.00)
serialization only (ltx v0.5.2)	1 GiB	68.6 MB	99.3 MB	99.1 MB	142.8 MB	pass (ratio 1.00)
this PR (serialization + ltx#95)	1 GiB	48.7 MB	79.2 MB	81.3 MB	120.7 MB	pass (ratio 1.03)
main 3b468c9 + ltx v0.5.2	4 GiB / 1,051,216 pages	168.6 MB	306.3 MB	470.6 MB	370.5 MB	fail (overlap/sum = 0.99)
serialization only (ltx v0.5.2)	4 GiB	173.1 MB	300.7 MB	286.5 MB	364.3 MB	pass (ratio 0.95)
this PR (serialization + ltx#95)	4 GiB	68.8 MB	203.4 MB	199.6 MB	257.3 MB	pass (ratio 0.98, gate_release_reason=timeout, 364 busy retries)

On main the overlap's memory is the sum of the two operations; with serialization it is the larger of the two; with ltx#95 the encoder index itself shrinks from ~95 B/page to ~18 B/page. Peak heap profiles (inuse_space, 4 GiB run):

main, overlap phase:        233.81MB ltx.(*Encoder).EncodePage   (two live encoders)
                             64.00MB s3/manager.(*maxSlicePool).newSlice (two in-flight uploads)
                             48.03MB ltx.DecodePageIndex
serialization only:         114.54MB ltx.(*Encoder).EncodePage   (one encoder)
                             32.00MB s3/manager.(*maxSlicePool).newSlice
                             13.20MB ltx.DecodePageIndex
this PR, snapshot phase:     32.00MB s3/manager.(*maxSlicePool).newSlice
                             15.70MB ltx.(*pageIndex).append
this PR, overlap phase:      84.45MB ltx.DecodePageIndex        (now the largest term; follow-up in ltx)
                             32.00MB s3/manager.(*maxSlicePool).newSlice

Overlap cost: ~450 B/page on main, ~270 B/page with serialization alone, ~190 B/page for this PR as pinned — at the reporter's 34.2M pages that is ≈15 GB → ≈9.3 GB → ≈6.5 GB. The decoder-side DecodePageIndex map is now the biggest remaining per-page term (~80 B/page) and is called out as follow-up work in superfly/ltx#95.

The same head is also deployed as the pr-1479 litestream-soak fleet (15 workers incl. two 100-database workers that upload hourly heap/alloc/goroutine/CPU pprof captures) for a longer-running comparison against main.

Known limits

• TryLock is not fair; under continuous contention on one database a level could repeatedly lose to another. Defaults make that unlikely (snapshots are daily, compactions minutes apart).
• Only Store.CompactDB takes the lock. Direct DB.Snapshot/DB.Compact callers (e.g. replicate -force-snapshot, which runs with monitors disabled) are not serialized.
• The pre-existing race between the monitor's IsOpen check and DisableDB/UnregisterDB is unchanged; a re-registered path gets a fresh DB and therefore a fresh lock.

Test plan

• tests/integration LTX behavioral gate (-short) passes locally with the 1s→10s backoff (it failed on the first push, where a 10s retry after every L1/L2 boundary collision fell outside the ±50% timing window)
• TestStore_CompactDB_SerializesPerDBMaintenance: same-DB compaction refused with ErrMaintenanceBusy while a snapshot is held, other DB unaffected, lock available after release
• TestBusyRetryDelay for the monitor delay cap
• go test ./ root package, go vet ./..., pre-commit (goimports, vet, staticcheck)
• Rig A/B above, 1 GiB and 4 GiB, including a build from this PR's exact head
• Five adversarial Codex review passes; all confirmed findings addressed (producer join + cancellation, busy retry cap, early passes scoped to refused/not-ready/newly registered DBs, schedule anchored at the regular boundary, test cleanup ordering); final pass approved with no confirmed regressions

[github-actions]

PR Build Metrics

✅ All clear — no issues detected

Check	Status	Summary
Binary size	✅	37.17 MB (+8.0 KB / +0.02%)
Dependencies	ℹ️	1 added, 1 removed
Vulnerabilities	✅	None detected
Go toolchain	✅	1.25.14 (latest)
Module graph	✅	1230 edges (0)

Binary Size

	Size	Change
Base (3b468c9)	37.16 MB
PR (3353f6f)	37.17 MB	+8.0 KB (+0.02%)

Dependency Changes

Added:

• github.com/superfly/ltx v0.5.3-0.20260827162011-d457a1ab7844

Removed:

• github.com/superfly/ltx v0.5.2

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 3
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric	Value
Build time	3s
Go version	go1.25.14
Commit	3353f6f

History (2 previous)

Commit	Updated	Status	Summary
4afa037	2026-08-27 19:12 UTC	✅	37.17 MB (+8.0 KB / +0.02%)
9fd7da4	2026-08-27 18:47 UTC	✅	37.17 MB (+8.0 KB / +0.02%)

---

_{🤖 Updated on each push.}