Skip to content

materialize-s3-iceberg: sort tables by collection key on create#4377

Draft
jacobmarble wants to merge 1 commit into
mainfrom
jgm-ice-sort
Draft

materialize-s3-iceberg: sort tables by collection key on create#4377
jacobmarble wants to merge 1 commit into
mainfrom
jgm-ice-sort

Conversation

@jacobmarble
Copy link
Copy Markdown
Contributor

@jacobmarble jacobmarble commented May 7, 2026

Description:

When materialize-s3-iceberg creates a new Iceberg table, set its write sort order to the collection-key fields (identity transform, ascending, nulls-first). Previously tables were created unsorted.

Workflow steps:

New tables created after this lands will carry a sort order in their Iceberg metadata; existing tables are untouched.

Documentation links affected:

None.

Notes for reviewers:

  • Identifier fields are deliberately not set: this connector is delta-updates only and append-only, so it cannot guarantee row-level uniqueness, which Iceberg's identifier_field_ids implies.
  • Field IDs are assigned 1..N by iceberg-ctl over FieldSelection.AllFields() (Keys ++ Values ++ Document), so the key field IDs are simply 1..len(Keys).
  • Verified end-to-end via the existing TestIntegration against the dockerized Polaris REST catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant