fix(replica): snapshot missing local L0 gaps - #1463
Draft
corylanou wants to merge 1 commit into
Draft
Conversation
Gap healing cannot re-upload a remote L0 gap after its local source files are removed. Force a current-position snapshot for that specific missing local-open error and retain the snapshot as a recalculation baseline so replication can continue without repeated fallbacks. The snapshot lets latest restore bypass the gapped lower-level history. Fixes #1458
PR Build Metrics
Binary Size
Dependency ChangesNo dependency changes. govulncheck OutputBuild Info
🤖 Updated on each push. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When L0 gap healing finds that the missing remote range is also absent locally, force a snapshot at the current database position and advance the replica position to the snapshot TXID. Position recalculation recognizes a snapshot that covers the gap, so later compaction invalidations do not force duplicate snapshots.
Latest restore planning starts from the forced snapshot and does not select the gapped L0 history below it. Other local-open errors and backend write errors retain their existing failure behavior.
This PR stacks on #1462, which stacks on #1155. Its base must remain
issue-1457-gap-heal-upload-only-missinguntil the stack lands.Motivation and Context
The regression test reproduced the failure before the fix: deleting a two-TXID range from remote L0 and its matching local files caused
Replica.Syncto fail at the first local open withno such file or directory. Position cleanup then made every retry repeat the same failure.The fallback is limited to
os.IsNotExistfrom the expected local L0 path while a later remote L0 file proves an interior gap. It emits a WARN with the full gap range, creates one current-position snapshot, and resumes replication above it.Fixes #1458
Related: #1462, #1155
How Has This Been Tested?
GOTOOLCHAIN=go1.25.13 go test -race -run "^(TestReplica_InvalidatePos_(HealsL0Gap|MissingLocalL0FallsBackToSnapshot)|TestDB_Compact/L0GapHealsViaReplica|TestReplica_CalcRestorePlan)$" -count=1 .GOTOOLCHAIN=go1.25.13 go test -race ./...GOTOOLCHAIN=go1.25.13 pre-commit run --all-filesLOG_LEVEL=DEBUG GOTOOLCHAIN=go1.25.13 go test -tags "integration,soak" -run "^TestLTXBehavior$" -short -v -timeout 15m ./tests/integration/Types of changes
Checklist
go fmt,go vet)go test ./...)