Skip to content

fix: skip unchanged L0 retention checks - #1481

Open
UniversePeak wants to merge 2 commits into
benbjohnson:mainfrom
UniversePeak:fix/1468-retention-list
Open

fix: skip unchanged L0 retention checks#1481
UniversePeak wants to merge 2 commits into
benbjohnson:mainfrom
UniversePeak:fix/1468-retention-list

Conversation

@UniversePeak

Copy link
Copy Markdown

Summary

Avoid repeated replica LIST requests from the L0 retention monitor when a database's local L0 watermark and cached L1 watermark have not changed. The retention pass now serializes per database, reuses the existing L1 max-file cache, and retries time-only eligibility changes at the next eligible timestamp.

Problem

Store.monitorL0Retention runs every L0RetentionCheckInterval (15 seconds by default) and DB.EnforceL0RetentionByTime previously listed the replica at level 1 and level 0 on every tick, even when an idle database had not changed. That creates up to 5,760 unnecessary LIST requests per database per day.

Solution

  • Track the local L0 max TXID and last successful L1 watermark in DB state.
  • Skip the remote retention scan when those inputs are unchanged and no file is waiting to become old enough.
  • Reuse MaxLTXFileInfo for the cached L1 max file instead of listing level 1 directly on every pass.
  • Reset the retention state when local state or the configured L0 retention window changes.
  • Add a regression test that counts level LIST calls across unchanged and new-L0-file passes.

Scope

In scope:

  • Avoiding redundant L0 retention LIST operations for unchanged databases.
  • Preserving retention checks when a new L0 file appears or a time threshold is reached.

Not in scope:

  • Changing retention policy, intervals, or replica layouts.
  • Altering compaction or replication behavior outside the retention check.

Test Plan

  • CGO_ENABLED=0 go.exe test -run TestDB_EnforceL0RetentionByTime_SkipsUnchangedDatabase -count=10 . (pass)
  • CGO_ENABLED=0 go.exe test -run '^$' ./... (pass; all packages compile)
  • CGO_ENABLED=0 go.exe vet ./... (pass)
  • go test -race ... was attempted but this Windows host has no CGO compiler/toolchain; Go reported -race requires cgo.
  • pre-commit run --files db.go db_test.go store.go was attempted; hook environment setup was blocked by a network read timeout downloading setuptools. gofmt and git diff --check pass.

The full package test suite was also attempted, but this Windows environment's file-backend tests fail with Access is denied while syncing temporary directories; this is independent of the focused regression test, which passes repeatedly.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

L0 retention monitor LISTs the replica for every database on every tick, with no change-detection guard

1 participant