Add IcebergTableCredentials back to IcebergSplit#29263
Add IcebergTableCredentials back to IcebergSplit#29263JohnFitzpatrick44 wants to merge 5 commits into
Conversation
|
7de4fcd to
c5d7c4e
Compare
|
How this is more secure that the current implementation? What problem does this PR solve? |
This is less about security and more about use cases which require the fileIoProperties in the One such use case which would benefit of having the fileIoProperties in the |
|
@findinpath security comes first, not a single feature use case should impose security hole. I prefer worse but secure API |
|
"Add IcebergTableCredentials back to IcebergSplit" Does it mean we send potentially same table credential info over and over again with each split? Should it be called |
Fair point, but this approach is at least as secure as the current state. We no longer indiscriminately dump splits to query json (from PRs here and here), so the only remaining risk is As for motivation, being able to vend split-level credentials makes us significantly more flexible moving forward, and this approach moves us in a much better direction than trying to find a workaround to provide creds to tasks after scheduling.
Yes, although this was the norm before #28537
The interface is |
4b88e10 to
61e7260
Compare
https://github.com/trinodb/trino/actions/runs/25135388739/job/73672490397?pr=29263 Related failure? |
|
@findinpath passes locally, I'll try rerunning tests |
bed96ba to
61e7260
Compare
b41536b to
45a237f
Compare
b726046 to
bf49fd4
Compare
chenjian2664
left a comment
There was a problem hiding this comment.
Adding credentials back to IcebergSplit would be more of an enhancement for connectors that support split-level specific credentials. Am I understanding that correctly?
| Map<Integer, String> partitionSpecsByIdJson, | ||
| Optional<Type> partitionColumnType) | ||
| Optional<Type> partitionColumnType, | ||
| IcebergTableCredentials tableCredentials) |
There was a problem hiding this comment.
Do we need the IcebergTableCredentials in FilesTableSplit?
| } | ||
|
|
||
| @Override | ||
| public Void visitTableScan(TableScanNode node, Void context) |
There was a problem hiding this comment.
I'm thinking about connectors that require, or can operate with, table-level credentials, in that case we probably still need this method
|
Closing to reconsider this approach |
Description
Previously,
fileIoPropertieswere removed fromIcebergSplitas part of the response to #28537.Passing credentials through the split is not inherently risky, however. We currently do not surface any split info that isn't explicitly selected through APIs like
ConnectorSplit#getSplitInfo(here) orSplitCompletedEvent(here).Now that credentials are wrapped in a very clearly named
IcebergTableCredentialsobject, it should be fine to pass this through the split. We may have been overzealous with what was removed. It may still be useful to be able to handle vended credentials from splits, rather than only at the table level.I considered an alternative approach where we do dynamically update credentials using a similar and simplified version of how we handle dynamic filters. This proved to be quite complicated and invasive, however, and I would like to limit the amount of changes outside of the Iceberg connector.
This approach is much simpler, and shouldn't result in any credential leaks that we were initially responding to from the above issue.
Additional context and related issues
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: