Output metrics about remove_orphan_files execution#26661
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the Iceberg connector's remove_orphan_files procedure to output execution metrics. Instead of returning no information, the procedure now returns a table with metrics including manifest file processing count, valid files found, files scanned, and files deleted.
- Modified the
executeTableExecutemethod signature across the SPI to returnMap<String, Long>instead ofvoid - Enhanced the
remove_orphan_filesimplementation to collect and return execution metrics - Updated query result handling to display metrics as a two-column table with metric names and values
Reviewed Changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| core/trino-spi/src/main/java/io/trino/spi/connector/ConnectorMetadata.java | Changed executeTableExecute return type from void to Map<String, Long> |
| plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java | Implemented metrics collection and return for remove_orphan_files procedure |
| core/trino-main/src/main/java/io/trino/operator/SimpleTableExecuteOperator.java | Updated operator to build result pages from returned metrics |
| core/trino-main/src/main/java/io/trino/sql/planner/plan/SimpleTableExecuteNode.java | Changed from single output symbol to list of symbols for metric name/value columns |
| docs/src/main/sphinx/connector/iceberg.md | Added documentation for the new metrics output format |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
3577a9d to
b63db28
Compare
| List<MaterializedRow> expectedRows = expectedResults.getMaterializedRows(); | ||
|
|
||
| if (compareUpdate) { | ||
| if (compareUpdate && !actualResults.getUpdateType().equals(Optional.of("ALTER TABLE EXECUTE"))) { |
There was a problem hiding this comment.
Instead of skipping here, can we introduce something assertAlter or a dedicated method - which would set compareUpdate to false ?
There was a problem hiding this comment.
There should be no need for a special assertAlter method. We should be able to assert a more detailed output with assertUpdate itself.
This branch is written with the assumption that an update query can only produce a row count output and nothing else. Since ALTER commands did not output anything before, this branch was never executed before.
Now with the more detailed output, we need to skip over this branch to allow the assertion on detailed results.
There was a problem hiding this comment.
Maybe add boolean compareExecute boolean flag and special assertExecute method?
c75a156 to
0a72e6b
Compare
| List<MaterializedRow> expectedRows = expectedResults.getMaterializedRows(); | ||
|
|
||
| if (compareUpdate) { | ||
| if (compareUpdate && !actualResults.getUpdateType().equals(Optional.of("ALTER TABLE EXECUTE"))) { |
There was a problem hiding this comment.
Maybe add boolean compareExecute boolean flag and special assertExecute method?
0a72e6b to
c8e5c41
Compare
c8e5c41 to
41883de
Compare
alter table lineitem execute remove_orphan_files(retention_threshold => '0d');
metric_name | metric_value
----------------------------+--------------
processed_manifests_count | 2
active_files_count | 98
scanned_files_count | 97
deleted_files_count | 0
41883de to
420b8ed
Compare
Description
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: