Core: ensure snapshot timestamp is monotonically increasing for V4 tables#16293
Open
stevenzwu wants to merge 2 commits into
Open
Core: ensure snapshot timestamp is monotonically increasing for V4 tables#16293stevenzwu wants to merge 2 commits into
stevenzwu wants to merge 2 commits into
Conversation
… tables. Made-with: Cursor Model: claude-4.6-opus-high-thinking Co-authored-by: Cursor <cursoragent@cursor.com>
2 tasks
…arget branch Pins that SnapshotProducer.snapshotTimestampMillis uses the parent on the target branch (via SnapshotUtil.latestSnapshot(base, targetBranch)) rather than the table's globally newest snapshot. The test commits to a separate branch with a clock pinned far in the future, then commits to main with a clock pinned just slightly past main's head. Main's new timestamp must equal the pinned wall-clock value (no fast-forward) and remain well below the sibling branch's head timestamp. Without this test, a regression that read base.currentSnapshot() (or any max-over-branches) for the parent would silently pass the existing single-branch monotonicity tests. Made-with: Cursor Model: claude-4.7-opus
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For format version 4 and above, this change makes
SnapshotProducerproduce snapshot timestamps that are strictly greater than the parent snapshot'stimestamp-ms, using a Lamport-clock style fast-forward when the wall clock has drifted backward.TableMetadata.MIN_FORMAT_VERSION_MONOTONIC_TIMESTAMPS = 4.SnapshotProducer.commit(), replacesSystem.currentTimeMillis()withsnapshotTimestampMillis(parentSnapshot). The new helper returnsclock.millis()for V1\u2013V3 (and for V4 root snapshots with no parent), andmax(clock.millis(), parentSnapshot.timestampMillis() + 1)for V4+ with a parent on the target branch. This preserves wall-clock timestamps in the steady state and only fast-forwards by the minimum amount needed when the clock has drifted backward.@VisibleForTestingsetClock(Clock)hook so the Lamport behavior can be exercised deterministically.This is a behavior change for V4 only. V1\u2013V3 commits continue to use the wall clock unchanged.
Why
The format-v4 row-timestamp feature exposes each manifest entry's
commit_timestamp_msas the_last_updated_timestamp_msmetadata column. Time-travel and "latest update" queries on that column become incoherent if a backward jump in the writer's wall clock produces snapshots whosetimestamp-msdoes not strictly increase with sequence number. Enforcing monotonicity at write time keeps the per-row "last updated" timestamp consistent with snapshot order.The change is also defensive against clock skew between writers in distributed environments: a stale clock that briefly reports a past time no longer rewrites history.
Test plan
./gradlew :iceberg-core:test --tests org.apache.iceberg.TestSnapshotProducer(29 tests, 6 skipped for pre-V4 byassumeThat, 0 failures)./gradlew :iceberg-core:spotlessCheckNew tests in
TestSnapshotProducer:testSnapshotTimestampsAreMonotonicallyIncreasing\u2014 three consecutive V4 appends produce strictly increasingtimestamp-ms.testV4LamportClockFastForwardsDriftedClock\u2014 with the producer'sClockpinned 10s in the past, the second V4 snapshot'stimestamp-msequalsfirstTs + 1, confirming the fast-forward branch is exercised.Related
A separate PR will propose the corresponding table spec change documenting the monotonic-
timestamp-msrequirement for V4.Made with Cursor