Skip to content

fix(vfs): recover straddling L1 polls - #1478

Open
corylanou wants to merge 4 commits into
mainfrom
issue-1460-vfs-l1-straddle-wedge
Open

fix(vfs): recover straddling L1 polls#1478
corylanou wants to merge 4 commits into
mainfrom
issue-1460-vfs-l1-straddle-wedge

Conversation

@corylanou

@corylanou corylanou commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Description

  • Track whether the level-1 polling watermark came from an observed L1 file boundary while preserving the existing position seed used for retention-pruned replicas.
  • Accept contiguous or straddling LTX ranges, skip fully covered ranges, and perform one bounded level-1 relist from TXID zero when an unanchored watermark makes a covering file invisible or a normal seek reports a gap.
  • Prefer page-index entries with newer MaxTXIDs and, when MaxTXIDs are equal, prefer the higher compaction level across L0/L1 combination, active and pending reader indexes, completed hydration, and VFS write mode. Commit shrink still replaces the discarded index with an empty target.
  • Run the tagged VFS race suite in CI and update the VFS test clients for the current replica-client interface.

The recovery remains local to the VFS poller. It does not change ReplicaClient.LTXFiles seek semantics or the GCS/Azure prefix behavior.

Incremental hydration failures disable hydrated reads and return the polling error because leaving a stale hydrated file enabled could serve wrong pages; disabling hydration falls back to remote page-index reads.

Motivation and Context

A follower can open while L1 is empty and seed its L1 watermark from an L0/snapshot position such as TXID 6. When the first L1 compaction later produces 1-b, the existing MinTXID-based seek for TXID 7 hides that covering file. The next visible file, c-1d, then produces this permanent polling error even though storage is contiguous:

poll L1: non-contiguous ltx file: level=1, current=0000000000000006, next=000000000000000c-000000000000001d

The regression test for the related freshness defect also failed on main before this fix:

TestVFSFile_PollLevel1DoesNotRegressNewerL0Pages
L1 poll regressed page 1 to older version 'b', want 'c'

That stale page-index behavior predates this issue: an older L1 page entry could overwrite a newer L0 entry in the combined, active, pending, or hydrated indexes. Recovering straddling files makes that existing path more reachable, so this PR fixes both behaviors together.

Equal-TXID entries need a level tie-break as well. Once L1 covers TXID N, L0 retention can delete files with MaxTXID less than or equal to N. Repointing equal-TXID pages to the higher L1 level keeps reads valid after that inclusive cleanup, while a strictly newer L0 entry still wins.

PR #1464 and PR #1465 were evaluated. This differs from #1464 by retaining the backend seek contract and containing recovery in the VFS caller, avoiding changes to all storage backends and the GCS/Azure listing prefixes. It differs from #1465 by retaining the current seed because a first L1 file can start above TXID 1; the new anchored fallback recovers the covering range without relying on a TXID-1 invariant. A freshness guard is included for the pre-existing stale-index defect that neither alternative fully addresses.

Fixes #1460

How Has This Been Tested?

go build ./...
go build -tags vfs ./...
go vet -tags vfs ./...
go test -race -count=1 .
go test -tags vfs -race -count=1 .
pre-commit run --all-files

The tagged race suite covers the reported two-poll timeline, a straddler and its successor visible in one poll, already-covered L1 ranges, a true L1 gap, a first L1 range starting above TXID 1, aligned and straddling freshness, equal-TXID promotion to retained L1 storage, greater-TXID L0 preservation, pending readers, completed hydration, hydration update failure with remote fallback, VFS writer self-compaction, and commit-shrink replacement.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (would cause existing functionality to not work as expected)

Checklist

  • My code follows the code style of this project (go fmt, go vet)
  • I have tested my changes (go test ./...)
  • I have updated the documentation accordingly (not needed; this restores intended polling and freshness behavior)

Recover L1 compaction files that straddle an unanchored polling
watermark without changing replica client seek semantics.

Keep page index entries from regressing when older L1 files arrive,
including pending readers, hydration, and VFS write mode. Add tagged
VFS race tests to CI.
@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown

PR Build Metrics

All clear — no issues detected

Check Status Summary
Binary size 37.16 MB (0.0 KB / 0.00%)
Dependencies No changes
Vulnerabilities None detected
Go toolchain 1.25.14 (latest)
Module graph 1230 edges (0)

Binary Size

Size Change
Base (3b468c9) 37.16 MB
PR (983b19e) 37.16 MB 0.0 KB (0.00%)

Dependency Changes

No dependency changes.

govulncheck Output

=== Symbol Results ===

No vulnerabilities found.

Your code is affected by 0 vulnerabilities.
This scan also found 1 vulnerability in packages you import and 3
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.

Build Info

Metric Value
Build time 44s
Go version go1.25.14
Commit 983b19e

History (3 previous)

Commit Updated Status Summary
29ff1d7 2026-08-26 23:14 UTC 37.16 MB (0.0 KB / 0.00%)
388bfd6 2026-08-26 22:51 UTC 37.16 MB (0.0 KB / 0.00%)
017edc8 2026-08-26 22:42 UTC 37.16 MB (0.0 KB / 0.00%)

🤖 Updated on each push.

Prefer higher compaction levels when page index entries share a
MaxTXID so inclusive L0 retention cannot strand page reads.

Cover equal-TXID promotion, greater-TXID L0 preservation, and the
hydration failure fallback.
- ingest unanchored L1 files at the seeded TXID boundary
- record streamed VFS writes in the page index
- strengthen equal-TXID and write-mode regressions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vfs gets wedged on non-contiguous ltx file when an L1 compaction straddles maxTXID1

1 participant