Skip to content

fix(wal): stop snapshot WAL reads at the advertised end offset - #1491

Open
yasyf wants to merge 1 commit into
benbjohnson:mainfrom
yasyf:fix-snapshot-wal-read-exceeded-bound
Open

fix(wal): stop snapshot WAL reads at the advertised end offset#1491
yasyf wants to merge 1 commit into
benbjohnson:mainfrom
yasyf:fix-snapshot-wal-read-exceeded-bound

Conversation

@yasyf

@yasyf yasyf commented Aug 30, 2026

Copy link
Copy Markdown

Problem

Level-9 snapshot compaction can enter a permanent failure state:

compaction failed ... extract timestamp from LTX header: snapshot wal read exceeded bound: max offset 24752 > end offset 16512

Once triggered, every subsequent compaction attempt for that database fails identically, so the snapshot chain never advances and cold restores must replay an ever-growing LTX chain. Observed on v0.5.16 across many databases in a multi-database deployment; the bug is unchanged on main. Full analysis in #1490.

Root cause

snapshotReader bounds the WAL scan with pageMap(ctx, maxBytes) where maxBytes = pos.walEndOffset - WALHeaderSize, but pageMap's bound check only fires on commit frames. A transaction that straddles the advertised WAL end is therefore read through to its commit frame past the bound, and the pages land in the returned map. snapshotReader then hard-fails on its absolute check maxOffset > pos.walEndOffset.

The overshoot in the error above is exactly the tail of the straddling transaction (24752 − 16512 = 2 × 4120), and the regression test below reproduces the identical error on unpatched main.

While investigating we confirmed the advertised end offset can land mid-transaction: snapshotWALEndOffset's lastSyncedWALOffset fast path returns the stored offset without the WAL-salt validation its own LTX fallback performs, so after a WAL restart the stored offset (commit-aligned in the old WAL) can point inside a transaction of the new WAL. The sync paths themselves always record commit-aligned offsets within a generation (finalOffset = info.offset + sz with sz derived from pageMap's committed-frame end), so the mid-transaction case is a bound from a previous WAL generation, and discarding the partial transaction — rather than widening the read past the bound — is the correct response. It preserves the #1281 invariant that a snapshot includes only WAL content synced through the advertised position.

Fix

pageMap gains a second bound, endOffset, that is hard: the scan stops before reading any frame that would end past it, so a transaction whose commit frame lies past the bound is discarded entirely (pages are only transferred to the map at commit frames, so no restructuring is needed).

  • snapshotReader now passes pos.walEndOffset as the hard bound (and no byte budget). The existing maxOffset > pos.walEndOffset check is kept as an invariant guard.
  • The sync path keeps the existing soft byte budget (maxSyncWALBytes) unchanged: a transaction larger than the budget must still be read through to its commit so sync always makes progress and L0 files end on commit boundaries.

In scope: the snapshot read bound in pageMap/snapshotReader, plus regression tests.
Not in scope: salt-validating the lastSyncedWALOffset fast path in snapshotWALEndOffset (a possible follow-up hardening; with this fix a stale bound now degrades to a slightly older, still-consistent snapshot instead of a permanent failure), and any change to sync batching semantics.

Tests

  • TestWALReaderPageMapEndOffsetStopsBeforeStraddlingTransaction — unit test on the existing testdata/wal-reader/ok/wal fixture: a bound inside the first transaction discards it (MidTransaction), a bound exactly on a commit-frame end includes it (AtCommitBoundary).
  • TestDB_SnapshotReaderWALEndOffsetMidTransaction — end-to-end repro: a WAL ending in a multi-frame transaction with the advertised end offset rewound one frame into it. On unpatched main this fails with the exact production error (snapshot wal read exceeded bound: max offset 41232 > end offset 37112); with the fix the snapshot succeeds, decodes, and its header WAL range stays within the advertised bound.
go test -race -v -run 'TestWALReaderPageMapEndOffsetStopsBeforeStraddlingTransaction|TestDB_SnapshotReaderWALEndOffsetMidTransaction' .
go test ./...

Full suite passes (destination integration tests that need cloud credentials skip as usual).

Fixes #1490

The snapshot WAL scan bounded pageMap with a byte budget that only
breaks on commit frames, so a transaction straddling the advertised
WAL end offset was read through to its commit frame past the bound.
snapshotReader then rejected the result with 'snapshot wal read
exceeded bound' and level-9 snapshot compaction failed permanently.

Add a hard end-offset bound to pageMap that stops before reading any
frame past the bound, discarding a partial transaction instead of
completing it, and use it for snapshot reads. The sync path keeps the
existing soft byte budget so a transaction larger than the budget is
still read through to its commit and sync always makes progress.

Fixes benbjohnson#1490
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snapshot compaction permanently stalls on "snapshot wal read exceeded bound" due to a frame-offset bug in pageMap

1 participant