Encryption for REST catalog by smaheshwar-pltr · Pull Request #13225 · apache/iceberg

smaheshwar-pltr · 2025-06-03T21:08:13Z

This PR implements client-side support for REST catalog encryption. With it, clients interacting with a REST catalog can read and write encrypted data.

It is similar to #13066, that integrates encryption with the Hive catalog.

cc @rdblue @RussellSpitzer @ggershinsky

github-actions · 2025-07-07T00:19:57Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

smaheshwar-pltr · 2025-07-07T12:37:19Z

(Bump to remove staleness)

github-actions · 2025-08-07T00:19:41Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

smaheshwar-pltr · 2025-08-07T13:06:57Z

(Bump to remove staleness)

github-actions · 2025-09-08T00:17:56Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

smaheshwar-pltr · 2025-09-11T17:35:59Z

(Bump to remove staleness)

smaheshwar-pltr · 2025-09-26T13:48:06Z

Given #7770 is merged, curious for thoughts on this PR.
REST integration sounds on the cards. Also happy for this PR to be superseded if other folks have worked on it.

cc @huaxingao @RussellSpitzer @ggershinsky @rdblue

RussellSpitzer · 2025-09-27T00:04:50Z

Could you elaborate on the api additions? I think it would help to have some more description on the general direction of this or

huaxingao · 2025-10-25T01:46:02Z

@smaheshwar-pltr Could you please resolve the conflicts?

ggershinsky · 2025-10-27T13:43:36Z

@huaxingao @smaheshwar-pltr Our team has a person who works on encryption with the REST catalog. If @smaheshwar-pltr does not object, we can follow up on this patch.

ggershinsky · 2025-10-27T13:50:30Z

+    return encryptionManager;
+  }
+
+  private void encryptionPropsFromMetadata(TableMetadata metadata) {


Is this method applied on the TableMetadata that is fetched directly from the REST catalog, and not from the metadata.json file? Both are possible, but the former must override (and check) the latter, to protect against the key removal and other attacks.

Yes this method will always be applied on a metadata field of a LoadTableResponse object received directly from the REST catalog (its only usage within this class is as such, and you can check the constructor usages within RESTSessionCatalog to confirm that the metadata coming in from there is as such too.

Question (not sure if there have been discussions here or if your team have thoughts): we want the key ID to come from the REST catalog service directly for security reasons.

It's typical for REST catalogs to provide metadata that corresponds to the metadata file in storage and not modify it apart from that. Given this, would it be preferable to have this field returned within the LoadTableResponse itself, to encourage catalogs to track it explicitly?

The concrete proposal here might be: ENCRYPTION_TABLE_KEY and ENCRYPTION_DEK_LENGTH become properties on the LoadTableResponse's config (mentioned in the REST spec here).

@ggershinsky @huaxingao LMKWYT!

Well I see two scenarios when thinking about this:

metadata.json is something that both the server and the clients can read (although clients wouldn't need to, given they get the metadata with the LoadTableResponse )

metadata.json can only be accessed on the server side and clients are not given FS credentials (either vended or not) to reach it

For case (1) I totally agree, we can't rely on just metadata.json to store these encryption properties, and the catalog should store it separately too, and eventually populating (i.e. doing the override logic referred by @ggershinsky) the properties in the LoadTableResponse to be created.
For case (2) I'm not 100% sure, but still leaning toward the catalog taking on this responsibility.

Either way, for the client side there's not much we can do other than recommending clients to consider the metadata from LoadTableResponse only. The rest (no pun intended) is on the server side to be decided and will be implementation-specific. For this code snippet above, irrelevant IMHO.

Let me know your thoughts.

Afaik, HMS is not optimized for JSON storage. But maybe someone in the community will take on storing the full metadata object there, to improve table security. Having only the table properties is barely sufficient. I think we should recommend REST catalogs for encrypted tables.

It's not. But would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered @ggershinsky ?

would storing a hash suffice? If so we could generate the hash of the whole JSON content and store it via an additional (Hive) table property. Then during table loading we can verify that the TableMetadata we just read in from a potentially untrusted storage (and yet metadata.json is not encrypted) is original or has been tampered with.

I think it's a good idea

One aspect of having an encrypted metadata.json is when the table schema is also considered a sensitive piece of information. I haven't found this in the discussions but do you know if this has ever been considered

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

I think it's a good idea

Sounds good, I can take this on and will produce a PR shortly.

Not sure. Though, it should be possible to have a REST implementation that hides the metadata.json file from the storage.

Yes, with REST that's true, I just meant it in a general sense, e.g. it's not currently possible to hide the schema of an encrypted table with Hive catalog. It may just be one more thing to note/document as a limitation of encryption wrt. Hive catalog - just merely wanted to highlight this though.

ggershinsky · 2025-10-27T14:00:05Z

Also, it would be good to refactor (if possible) a code common to this PR and to #13066 , so that other catalogs will be able to re-use it.

smaheshwar-pltr · 2026-03-12T16:43:06Z

Will need a change similar to the HiveCatalog addition in #15272

Great catch @ggershinsky, addressed in 3c055ce.

smaheshwar-pltr · 2026-03-12T18:41:05Z

@huaxingao @singhpk234 @ggershinsky, my apologies for the delay here, I was caught up in other work. I believe all comments are now be addressed or responded to, PTAL! 🙏

adp2201 · 2026-03-12T20:46:37Z

Thanks for the update on this PR. I reviewed the recent discussion and this looks close.

I have one suggestion that might help with merge confidence:

Could we make the security/behavior contract explicit in RESTOperationsBuilder Javadocs (or nearby API docs), especially for:

Expected metadata access path for encrypted tables (what is trusted / not trusted),
How encryption.key-id expectations should be handled by custom table-ops implementations,
What guarantees Iceberg provides vs what is caller/implementation responsibility when replacing default REST table operations.

If possible, adding one focused negative test (or equivalent assertion) for custom/unsafe table-ops behavior would also make this easier to reason about for downstream users.

Overall, appreciate the work here. This is an important capability for REST catalog users.

huaxingao · 2026-04-15T23:54:12Z

@singhpk234 Do you have more comments? Are you OK with deferring the remote scan planning test to a follow-up? #13225 (comment)

smaheshwar-pltr · 2026-04-22T20:05:18Z

Friendly ping @singhpk234 🙏

smaheshwar-pltr · 2026-04-27T08:27:32Z

Friendly ping, @singhpk234!

singhpk234 · 2026-04-28T06:46:55Z

Apologies I lost track of this PR :(
I think its fine to take scan planning as follow-up can you add a TODO in code for that ?

Also one thing which i am unclear of is how is the rest-catalog giving creds for encryption ? is it implictly assumed storage config will have creds for encryption too like KMS ? what happens in case such as vault ? how does the rest server give those creds back ?

singhpk234 · 2026-04-28T06:30:46Z

+  }
+
+  @Override
+  public EncryptionManager encryption() {


can this be called in multi-threaded fashion in that case we might create more than one instance ?

Thanks for flagging - I'm mirroring here what we did for Hive

iceberg/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java

Line 141 in 53f1f1a

public EncryptionManager encryption() {

cc @ggershinsky. And I think this is fine as I believe that TableOperations are inherently not thread-safe:

E.g. BaseMetastoreTableOperations has unsynchronized mutable fields e.g. currentMetadata

iceberg/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java

Line 54 in 53f1f1a

private TableMetadata currentMetadata = null;

and this very RESTTableOperations class too similarly:

iceberg/core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java

Line 65 in 53f1f1a

private TableMetadata current;

I don't think concurrent use of a single table operations class is supported. LMKWYT!

smaheshwar-pltr · 2026-05-01T23:09:03Z

I think its fine to take scan planning as follow-up can you add a TODO in code for that ?

Thanks, @singhpk234 - 2092454.

smaheshwar-pltr · 2026-05-02T00:20:20Z

Also one thing which i am unclear of is how is the rest-catalog giving creds for encryption ? is it implictly assumed storage config will have creds for encryption too like KMS ? what happens in case such as vault ? how does the rest server give those creds back ?

Thanks for flagging this @singhpk234, it's a great point.

This PR integrates with the existing KeyManagementClient wiring as in #13066 - it's initialised from merged catalog properties and clients can configure it as integration tests do. The catalog properties would include the server-returned config, but that's still at a per-client(/catalog) level and not a per-table level.

I think that per-table KMS credential vending is valuable - I can imagine REST servers wanting to follow the principle of least privilege and subscope KMS credentials where that's possible. I do think there's a discussion to be had - the AWS KMS client doesn't pick up the s3.access-key-id vended properties, and even if it did, it's debatable whether those should be used; we'd maybe want some KMS prefix, as you could imagine a setup with Azure storage and an AWS KMS, in which case it doesn't feel great to vend those KMS credentials as if they're S3 storage credentials. Thinking out loud, I wonder about KMS credential refresh if that's desirable.

On this PR: I'd prefer to split PRs and discussions (we've already discussed a lot here) - in my opinion, it makes sense to ship catalog-property KMS configuration first because I think this should be supported anyway (in the same way that you can initialise a RESTCatalog with your own storage credentials that will be used if there's no access delegation) and is valuable as is. Introducing per-table KMS clients as a follow up makes sense to, me and shouldn't break this behaviour. I'll open an issue for the KMS credential vending discussion, and maybe we can proceed this PR, how does that sound?

(+ cc @ggershinsky as I know you folks have been thinking about REST integration too, would love to hear your thoughts on this)

Edit: I've put up #16194

…cherry-pick # Conflicts: # spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java # spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java

ggershinsky · 2026-05-04T07:57:43Z

Basically, the lifecycle of a KMS client object is driven by these events:

initialize call - done once. Passes the catalog properties map. A pointer to this map can be kept. Also, other potential channels for credential passing (such as system env or files) can be queried at this stage. Plus, if needed, a per-table credential map can be fetched and stored.
wrap/unwrap calls. Pass the table key id. Credentials can be refreshed (and, if needed, per-table credentials can be fetched or extracted locally) at this stage, using the catalog property map, system env, files or other channels.

Also, it's possible to run a thread in the KMS client that would asynchronously handle the credentials.

So the mechanism seems to be quite flexible. But if it doesn't cover important usecases, we can consider changes.

singhpk234 · 2026-05-04T15:44:21Z

Thanks for the response @smaheshwar-pltr @ggershinsky !
Unlike other catalogs, REST catalog works on minting creds at table which it expects the client to use to do subsequent operations, we have a dedicated /credentials endpoint which works on refreshing this creds so this machinery is in place already.
Now the concern for me the following :

How do we handle things like VAULT, given rest catalog wants to return per table level creds to talk to vault store (this came up recently), so far this PR is advocating storage creds contains the creds of KMS too, but if we end up supporting vault, this can't be mixed with storage creds right (side note : catalogs have been mixing KMS creds with this obj for things like SSE which entirely an object store level thing), i believe we should conclude this dicussion, let me start a thread for this
If catalog returns you cred, it overrides the client side creds, so this means the catalog which is not aware of encryption and is just treating metadata as is will have no clue it needs to vend creds (naively thinking) also client should put additional assertion that the creds returned are valid since there is encryption enabled on the table otherwise fail directly rather than during the execution.
Backward compatibility : lets say we release this client and we expect storage will always contain this creds for kms too but later in protocol we introduce a new dedidcate field in loadTable response, we will not need to think of backward compatibilty to support both the ways of returning creds.

Will that being said i don't wanna block the progress in this pr, appreciate all the work, but i would really like to conclude on this before we mark REST catalog ready for supporting encryption.

singhpk234 · 2026-05-04T16:23:01Z

started a dev thread to discuss this : https://lists.apache.org/thread/z48t5wgx778j17pzto9kqxwysw4ysxxo

Also please free to move forward with this PR ! For now lets not conclude for saying REST Catalog is ready to support encryption

smaheshwar-pltr · 2026-05-05T00:25:09Z

Thanks, I've responded to the mailing list thread here: https://lists.apache.org/thread/t9sj6nlxgxyl9k5cbmf70gnmrhgxz1xg. I think there was some confusion as this PR does not implement REST KMS credential vending which is a larger issue that definitely needs more discussion if we want to support it.

smaheshwar-pltr · 2026-05-05T00:25:40Z

@huaxingao, all comments here have been now addressed, might we be able to get this in for the upcoming release?

ggershinsky · 2026-05-05T07:02:52Z

+1 to including this PR in 1.11. The patch does not handle credentials (delegating this to KMS clients) - instead, it provides scaffolding for table encryption in the REST catalog client, similar to the HMS catalog client (and other catalogs in the future).
Also, the built-in encryption is designed to protect tables in any storage backend, including untrusted storage. The storage credentials and KMS credentials don't have the same purpose/scope. But again, this is basically orthogonal to the current PR.

smaheshwar-pltr · 2026-05-05T23:01:39Z

I agree with Gidon's comment above. Friendly ping @huaxingao (sorry to pester) in case you have thoughts!

szlta · 2026-05-06T08:55:56Z

Agreed with @ggershinsky. Let's have this in and we can address the other questions later as they don't seem to be that tightly connected, this PR has been dragging for quite long now.

github-actions Bot added spark core labels Jun 3, 2025

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

Comment thread spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java Outdated

smaheshwar-pltr marked this pull request as ready for review June 3, 2025 21:24

smaheshwar-pltr mentioned this pull request Jun 3, 2025

Encryption integration and test #13066

Merged

smaheshwar-pltr commented Jun 3, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java Outdated

smaheshwar-pltr mentioned this pull request Jun 8, 2025

AWS: KeyManagementClient implementation that works with AWS KMS #13136

Merged

github-actions Bot added the stale label Jul 7, 2025

github-actions Bot removed the stale label Jul 8, 2025

github-actions Bot added the stale label Aug 7, 2025

github-actions Bot removed the stale label Aug 8, 2025

github-actions Bot added the stale label Sep 8, 2025

github-actions Bot removed the stale label Sep 12, 2025

smaheshwar-pltr force-pushed the rest-encrypt-on-main-cherry-pick branch from d007aa2 to 889cbbf Compare September 23, 2025 15:23

smaheshwar-pltr mentioned this pull request Sep 23, 2025

Encryption for REST catalog #13224

Closed

smaheshwar-pltr commented Sep 23, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

smaheshwar-pltr commented Sep 23, 2025

View reviewed changes

Comment thread spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestTableEncryption.java Outdated

ggershinsky reviewed Oct 27, 2025

View reviewed changes

github-actions Bot added the build label Oct 27, 2025

smaheshwar-pltr mentioned this pull request Mar 12, 2026

[POC] JDBC encryption to test encrypted scan planning #15603

Draft

smaheshwar-pltr requested review from huaxingao and singhpk234 March 26, 2026 00:24

This was referenced Apr 7, 2026

Repro: Concurrent replace V3 failure #15905

Draft

POC: Encryption read support apache/iceberg-python#3221

Draft

Feature Request: Encryption apache/iceberg-python#3222

Open

singhpk234 reviewed Apr 28, 2026

View reviewed changes

Add TODO for encryption <> scan planning

2092454

smaheshwar-pltr mentioned this pull request May 1, 2026

Encryption integration with REST scan planning #16192

Open

3 tasks

smaheshwar-pltr mentioned this pull request May 2, 2026

KMS credential vending #16194

Open

3 tasks

smaheshwar-pltr requested a review from singhpk234 May 2, 2026 00:32

smaheshwar-pltr added 2 commits May 2, 2026 01:35

Fixes

cfb7345

Merge remote-tracking branch 'origin/main' into rest-encrypt-on-main-…

0af4ef4

…cherry-pick # Conflicts: # spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java # spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java

Conversation

smaheshwar-pltr commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2025

Uh oh!

smaheshwar-pltr commented Jul 7, 2025

Uh oh!

github-actions Bot commented Aug 7, 2025

Uh oh!

smaheshwar-pltr commented Aug 7, 2025

Uh oh!

github-actions Bot commented Sep 8, 2025

Uh oh!

smaheshwar-pltr commented Sep 11, 2025

Uh oh!

Uh oh!

Uh oh!

smaheshwar-pltr commented Sep 26, 2025

Uh oh!

RussellSpitzer commented Sep 27, 2025

Uh oh!

huaxingao commented Oct 25, 2025

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky commented Oct 27, 2025

Uh oh!

smaheshwar-pltr commented Mar 12, 2026

Uh oh!

smaheshwar-pltr commented Mar 12, 2026

Uh oh!

adp2201 commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huaxingao commented Apr 15, 2026

Uh oh!

smaheshwar-pltr commented Apr 22, 2026

Uh oh!

smaheshwar-pltr commented Apr 27, 2026

Uh oh!

singhpk234 commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smaheshwar-pltr commented May 1, 2026

Uh oh!

smaheshwar-pltr commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggershinsky commented May 4, 2026

Uh oh!

singhpk234 commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singhpk234 commented May 4, 2026

smaheshwar-pltr commented Jun 3, 2025 •

edited

Loading

adp2201 commented Mar 12, 2026 •

edited

Loading

smaheshwar-pltr commented May 2, 2026 •

edited

Loading

singhpk234 commented May 4, 2026 •

edited

Loading