Skip to content

Add support for lock-free commit in Iceberg HMS catalog#25445

Merged
ebyhr merged 1 commit into
trinodb:masterfrom
ebyhr:ebi/iceberg-hms-lock
Mar 31, 2025
Merged

Add support for lock-free commit in Iceberg HMS catalog#25445
ebyhr merged 1 commit into
trinodb:masterfrom
ebyhr:ebi/iceberg-hms-lock

Conversation

@ebyhr
Copy link
Copy Markdown
Member

@ebyhr ebyhr commented Mar 28, 2025

Description

Fixes #22182

Request from Iceberg community: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1743095891453949
Iceberg documentation: https://iceberg.apache.org/docs/nightly/configuration/#hadoop-configuration

Release notes

## Iceberg
* Add support for lock-free commit in Hive catalog. ({issue}`issuenumber`)

@cla-bot cla-bot Bot added the cla-signed label Mar 28, 2025
@github-actions github-actions Bot added docs iceberg Iceberg connector labels Mar 28, 2025
Comment thread docs/src/main/sphinx/object-storage/metastores.md Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'a take that asserting no lock has been acquired is unfeasible/impossible

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is necessary, but if you fetch /tmp/root/hive.log from the metastore container and look for lines with HiveMetaStore.audit, you should see all of the metastore Thrift calls.

Copy link
Copy Markdown
Member

@electrum electrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? Based on my reading of the HMS change, it seems like we need to pass the “expectedParameter” value to the metastore so that it can do the check.

Comment thread docs/src/main/sphinx/object-storage/metastores.md Outdated
Comment thread docs/src/main/sphinx/object-storage/metastores.md Outdated
Comment thread docs/src/main/sphinx/object-storage/metastores.md Outdated
@ebyhr ebyhr force-pushed the ebi/iceberg-hms-lock branch 2 times, most recently from 1074ef8 to da96ac3 Compare March 31, 2025 01:25
@github-actions github-actions Bot added delta-lake Delta Lake connector hive Hive connector labels Mar 31, 2025
@ebyhr ebyhr force-pushed the ebi/iceberg-hms-lock branch from da96ac3 to 21a9e9c Compare March 31, 2025 01:31
@electrum electrum requested a review from Copilot March 31, 2025 02:22
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds configurable support for lock-free commits in the Hive catalog when using Iceberg. It introduces a new lockingEnabled configuration option in the Iceberg Hive catalog, integrates optional Hive lock acquisition in commit operations, and updates method signatures across metastore interfaces with a new environmentContext parameter.

Reviewed Changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated no comments.

Show a summary per file
File Description
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/HiveMetastoreTableOperationsProvider.java Loads and propagates the new lockingEnabled configuration.
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/HiveMetastoreTableOperations.java Integrates conditional Hive lock acquisition and release in table commit operations.
plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/file/FileMetastoreTableOperations.java Updates the replaceTable call to include an empty environment context.
plugin/trino-hive/* Updates the method signatures of replaceTable and alterTable to accept an environmentContext parameter.
docs/src/main/sphinx/object-storage/metastores.md Documents the new Iceberg Hive catalog property for controlling locking.
Comments suppressed due to low confidence (1)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/hms/HiveMetastoreTableOperations.java:95

  • [nitpick] Consider renaming the local variable 'lockingEnabled' (e.g., to 'tableLockingEnabled') to clearly distinguish it from the class member.
boolean lockingEnabled = parseBoolean(table.getParameters().getOrDefault(HIVE_LOCK_ENABLED, Boolean.toString(this.lockingEnabled)));

@ebyhr ebyhr force-pushed the ebi/iceberg-hms-lock branch from 21a9e9c to 4570e2e Compare March 31, 2025 02:25
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is necessary, but if you fetch /tmp/root/hive.log from the metastore container and look for lines with HiveMetaStore.audit, you should see all of the metastore Thrift calls.

@ebyhr ebyhr force-pushed the ebi/iceberg-hms-lock branch from 4570e2e to 70fcab3 Compare March 31, 2025 02:59
@ebyhr ebyhr merged commit 2cf9749 into trinodb:master Mar 31, 2025
61 checks passed
@ebyhr ebyhr deleted the ebi/iceberg-hms-lock branch March 31, 2025 04:02
@github-actions github-actions Bot added this to the 475 milestone Mar 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed delta-lake Delta Lake connector docs hive Hive connector iceberg Iceberg connector

Development

Successfully merging this pull request may close these issues.

Support lock-free commit for Iceberg using HMS

6 participants