Add Hudi MOR snapshot query and metadata table support by codope · Pull Request #25599 · trinodb/trino

codope · 2025-04-16T12:52:00Z

Description

Re-enable metadata table support - bring back the defunct config.
Add support for reading log files to support snapshot querying of Hudi Merge-on-Read (MOR) tables.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Hudi
* Support for snapshot query of Hudi Merge-on-Read (MOR) tables.
* Support for Hudi metadata table.

ebyhr

Please fix CI failures.

cla-bot · 2025-04-21T09:50:04Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-04-21T11:36:18Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

codope · 2025-05-02T06:04:57Z

@ebyhr @mxmarkovics Could you please take a final pass and help in merging this PR? I have two other PRs stackedon top of this one.

codope · 2025-05-07T08:26:37Z

@ebyhr @mxmarkovics @mosabua gentle reminder for this PR. All comments are addressed and CI is successful.

mxmarkovics

LGTM

ebyhr

Just skimmed.

ebyhr · 2025-05-07T20:04:18Z

Why did we remove the usage of isIgnoreAbsentPartitions?

The filesystem view API of Hudi internally handles non-matching partitions. In other words the directory lister will return empty list (from listStatus call) when called for absent partitions. So, we don't need this config separately. I have removed it from hudi.md as well now.

codope · 2025-05-09T15:28:02Z

@ebyhr All comments addressed. Could you help me with fixing maven-checks 25-ea failure? I don't think it is related to the changes. Logs just show:

The JAVA_HOME environment variable is not defined correctly,
this environment variable is needed to run this program.
Error: Process completed with exit code 1.

codope · 2025-05-10T01:41:17Z

@ebyhr All comments addressed. Could you help me with fixing maven-checks 25-ea failure? I don't think it is related to the changes. Logs just show:
The JAVA_HOME environment variable is not defined correctly,
this environment variable is needed to run this program.
Error: Process completed with exit code 1.

Never mind, I saw that this CI check has been disabled recently - #25759. I've rebased.

codope · 2025-05-12T12:15:44Z

@ebyhr gentle ping for review.

codope · 2025-05-22T00:17:41Z

@ebyhr @yihua @mosabua @wendigo Please help merge this PR.

wendigo · 2025-05-22T11:17:04Z

@codope I'd need to go through it in full extent. @ebyhr did you review it already?

codope · 2025-05-22T15:16:03Z

Would appreciate if this can be reviewed and merged sooner. I have another PR for adding data skipping support stacked on these changes - #25601

ebyhr · 2025-05-23T01:04:05Z

I don't think we can merge this PR soon as the change isn't so simple. Please be patient.

codope · 2025-05-27T16:13:02Z

I don't think we can merge this PR soon as the change isn't so simple. Please be patient.

I understand. Let me know how I can help to accelerate the review. This PR is very crucial for the Hudi community as it restores metadata support and adds MOR snapshot query.

ebyhr · 2025-05-29T00:12:53Z

Is it possible to separate into 2 PRs? Why do we handle both MoR and metadata-tables in a commit?

codope · 2025-06-09T07:35:00Z

Is it possible to separate into 2 PRs? Why do we handle both MoR and metadata-tables in a commit?

@ebyhr Hudi's metadata table is MoR table so we need to handle both.

Read log files using filegroup reader and fix tests Add test table and a test with metadata enabled Fix SqlDate type not able to cast to Number error

github-actions · 2025-07-02T17:03:54Z

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.

liujinhui1994 · 2025-07-04T06:58:29Z

We need this PR. When can it be merged into the trino community? @codope @ebyhr
We switched from presto to trino, and we are all based on hudi's lake warehouse table.

voonhous · 2025-07-11T02:59:25Z

Hello all, i can try picking this up. @ebyhr what are the pending items remaining here?

github-actions · 2025-08-01T17:05:45Z

This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack.

github-actions · 2025-08-25T17:02:52Z

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

cla-bot Bot added the cla-signed label Apr 16, 2025

codope requested a review from ebyhr April 16, 2025 12:52

github-actions Bot added hudi Hudi connector hive Hive connector labels Apr 16, 2025

codope mentioned this pull request Apr 16, 2025

Add support for data skipping using column stats #25601

Closed

ebyhr reviewed Apr 16, 2025

View reviewed changes

codope force-pushed the hudi-mor-snapshot branch from 0cfa118 to 941d35d Compare April 17, 2025 13:55

github-actions Bot added the docs label Apr 17, 2025

cla-bot Bot removed the cla-signed label Apr 21, 2025

codope force-pushed the hudi-mor-snapshot branch from 0e495c0 to 935c8c0 Compare April 21, 2025 11:36

codope force-pushed the hudi-mor-snapshot branch from 935c8c0 to d9200d3 Compare April 21, 2025 15:49

cla-bot Bot added the cla-signed label Apr 21, 2025

codope force-pushed the hudi-mor-snapshot branch from d9200d3 to b5bbebd Compare April 23, 2025 15:52

mxmarkovics reviewed Apr 23, 2025

View reviewed changes

Comment thread plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/util/HudiAvroSerializer.java Outdated

mxmarkovics reviewed Apr 23, 2025

View reviewed changes

Comment thread plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/util/HudiAvroSerializer.java Outdated

mxmarkovics reviewed Apr 23, 2025

View reviewed changes

Comment thread plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiUtil.java Outdated

codope force-pushed the hudi-mor-snapshot branch 3 times, most recently from e4e5630 to a060c18 Compare May 2, 2025 05:20

codope force-pushed the hudi-mor-snapshot branch 2 times, most recently from ce716a4 to 33f70f9 Compare May 7, 2025 07:10

mxmarkovics approved these changes May 7, 2025

View reviewed changes

ebyhr reviewed May 7, 2025

View reviewed changes

codope force-pushed the hudi-mor-snapshot branch 3 times, most recently from a771840 to 8d007ff Compare May 9, 2025 14:28

codope force-pushed the hudi-mor-snapshot branch from 8d007ff to fe98198 Compare May 10, 2025 01:39

codope force-pushed the hudi-mor-snapshot branch from fe98198 to 278ad2d Compare May 12, 2025 12:15

codope force-pushed the hudi-mor-snapshot branch from 278ad2d to 88f1a4c Compare May 14, 2025 07:13

codope force-pushed the hudi-mor-snapshot branch 3 times, most recently from 042a4dd to 20a13ac Compare May 21, 2025 23:24

Add Hudi MOR snapshot query and metadata table support

4f2a8f8

Read log files using filegroup reader and fix tests Add test table and a test with metadata enabled Fix SqlDate type not able to cast to Number error

codope force-pushed the hudi-mor-snapshot branch from 20a13ac to 4f2a8f8 Compare June 11, 2025 03:10

github-actions Bot added the stale label Jul 2, 2025

github-actions Bot removed the stale label Jul 4, 2025

github-actions Bot added the stale label Aug 1, 2025

liujinhui1994 mentioned this pull request Aug 20, 2025

support trino query Hudi MOR snapshot query #26449

Closed

github-actions Bot closed this Aug 25, 2025

Conversation

codope commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Additional context and related issues

Release notes

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cla-bot Bot commented Apr 21, 2025

Uh oh!

cla-bot Bot commented Apr 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codope commented May 2, 2025

Uh oh!

codope commented May 7, 2025

Uh oh!

mxmarkovics left a comment

Choose a reason for hiding this comment

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebyhr May 7, 2025

Choose a reason for hiding this comment

Uh oh!

codope May 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codope commented May 9, 2025

Uh oh!

codope commented May 10, 2025

Uh oh!

codope commented May 12, 2025

Uh oh!

codope commented May 22, 2025

Uh oh!

wendigo commented May 22, 2025

Uh oh!

codope commented May 22, 2025

Uh oh!

ebyhr commented May 23, 2025

Uh oh!

codope commented May 27, 2025

Uh oh!

ebyhr commented May 29, 2025

Uh oh!

codope commented Jun 9, 2025

Uh oh!

github-actions Bot commented Jul 2, 2025

Uh oh!

liujinhui1994 commented Jul 4, 2025

Uh oh!

voonhous commented Jul 11, 2025

Uh oh!

github-actions Bot commented Aug 1, 2025

Uh oh!

codope commented Apr 16, 2025 •

edited

Loading