Skip to content

fix(vfs): tolerate L1 files that straddle the poll watermark - #1464

Open
darkgnotic wants to merge 2 commits into
benbjohnson:mainfrom
rocicorp:fix-vfs-compaction-boundary-handling
Open

fix(vfs): tolerate L1 files that straddle the poll watermark#1464
darkgnotic wants to merge 2 commits into
benbjohnson:mainfrom
rocicorp:fix-vfs-compaction-boundary-handling

Conversation

@darkgnotic

@darkgnotic darkgnotic commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description

The VFS live-tailer's incremental poll (pollLevel) and every ReplicaClient
backend's LTXFiles(seek) implementation assumed a resumed watermark always
falls exactly on a compaction-level file boundary (file.MinTXID == watermark+1).
That assumption breaks whenever a file's range straddles the watermark —
MinTXID <= watermark < MaxTXID — which happens:

  • on open, maxTXID1 is seeded from pos.TXID (an L0/snapshot position) when
    L1 is still empty. That seed is correct and necessary (L1 files get pruned by
    retention, so L1 doesn't generally start at TXID 1) — but it need not land on
    a future L1 file's boundary.

In both cases, every backend's seek filter (MinTXID < seek -> skip) makes the
straddling file invisible, and pollLevel's exact-match contiguity check
(MinTXID == watermark+1) then rejects the next file as non-contiguous. At L1
and above this returns a hard error; nothing retries or recovers, so a follower
wedges permanently at the stale watermark — polling forever, logging
"non-contiguous ltx file", and serving stale reads with no indication to
callers.

This PR makes the seek + contiguity logic straddle-tolerant:

  • pollLevel (vfs.go) now classifies each candidate file against the current
    watermark instead of requiring an exact match:
    1. Already covered (MaxTXID <= watermark) -> skip (defends against within-level
      overlap; not produced by litestream today)).
    2. Covers the next TXID, including straddling (MinTXID <= watermark+1 <= MaxTXID) -> apply and advance the watermark to MaxTXID. Re-applying the
      already-covered portion of the page index is an idempotent overwrite, so
      this is safe.
    3. Real gap (MinTXID > watermark+1) -> unchanged: defer at L0, error above.
  • Every backend's LTXFiles seek filter changes from MinTXID < seek to
    MaxTXID < seek, so a straddling file is returned instead of silently
    dropped: s3, file, sftp, webdav, nats, oss.
  • gs and abs additionally used seek as a list-key prefix
    (prefix += seek.String()), which excludes a straddling file at the listing
    level regardless of the post-filter. Both now list the full level directory
    and filter by MaxTXID < seek in the iterator, matching s3/oss. This
    trades a narrower list for correctness — same list volume s3/oss already
    have.
  • mock needs no change; it delegates entirely to a caller-supplied func.

The LTXFiles seek-semantics change is a no-op for every existing caller except pollLevel: seek==0 callers are unaffected by definition, L0 callers only ever see single-TXID files (MinTXID==MaxTXID), and compaction always seeks on a level-aligned boundary. Only pollLevel passes a seek that can fall inside a file's range.

Motivation and Context

Found via a production incident: a read-only VFS follower tailing S3 wedged
permanently on

poll L1: non-contiguous ltx file: level=1, current=...6, next=...c-...1d

even though the backup data in object storage was fully intact and contiguous.
Root-caused to the seed-vs-boundary mismatch above (case 1) — the L1 file that
would have bridged the gap (1-b, covering the missing range) was present in
storage the whole time but permanently invisible to the seek.

Fixes Issue #1460

How Has This Been Tested?

  • Added TestVFSFile_StraddlingL1FileAfterSeed, which reproduces the exact
    production sequence (snapshot seeds maxTXID1=6, first L1 compaction emits a
    straddling 1-b, next L1 file is c-1d) — fails on the old code
    (maxTXID1 stuck at 6) and passes with this fix.
  • Added TestVFSFile_MergedWiderL1FileAfterConsume, a defensive test that
    pollLevel doesn't wedge on within-level overlap; not a state litestream produces today.
  • Updated mockReplicaClient.LTXFiles in tests to mirror the fixed backend
    seek semantics (MaxTXID >= seek).
  • Full existing -tags vfs suite passes with no regressions:
    go test -tags vfs ./....
  • go build -tags vfs ./..., go vet clean on all backend packages.
  • Per-backend package tests (s3, gs, oss, nats, webdav, file) pass;
    abs/sftp/mock have no existing test files to run.
  • gofmt -l clean on all touched files.

Note: this also fixes a latent, unrelated compile break in the -tags vfs test
suite — ReplicaClient gained SetLogger in a prior commit but the VFS test
doubles (mockReplicaClient, countingReplicaClient, writeTestReplicaClient)
were never updated, so go test -tags vfs ./... hasn't built since. Added
no-op SetLogger implementations to unblock running (and adding to) these
tests; worth wiring -tags vfs into CI separately so this doesn't recur.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (would cause existing functionality to not work as expected)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes (go test ./...)
  • I have updated the documentation accordingly (if needed)

@darkgnotic

Copy link
Copy Markdown
Contributor Author

@corylanou, I am somewhat uneasy about changing the semantics of the LTXFiles() iterator, though it makes sense for the pollLevel() logic. Claude seems to think that it is safe for all other call sites, but let me know if we need to parameterize the iterator for different behaviors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant