Skip to content

feat: Support anonymous S3 access for public buckets #27758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
0eu wants to merge 1 commit into from
Closed

feat: Support anonymous S3 access for public buckets #27758

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/src/main/sphinx/object-storage/file-system-s3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,11 @@ and secret keys, STS, or an IAM role:
`trino-filesystem`.
* - `s3.external-id`
- External ID for the IAM role trust policy when connecting to S3.
* - `s3.anonymous-access`
- Use anonymous credentials for accessing public S3 buckets without
authentication. When set to `true`, no credentials are sent with S3 requests.
This takes priority over other authentication methods when enabled.
Defaults to `false`.
:::

## Security mapping
Expand Down
29 changes: 29 additions & 0 deletions lib/trino-filesystem-s3/src/main/java/io/trino/filesystem/s3/S3FileSystemConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ public static RetryStrategy getRetryStrategy(RetryMode retryMode)
private String sseKmsKeyId;
private String sseCustomerKey;
private boolean useWebIdentityTokenCredentialsProvider;
private boolean anonymousAccess;
private SignerType signerType;
private DataSize streamingPartSize = DataSize.of(32, MEGABYTE);
private boolean requesterPays;
Expand Down Expand Up @@ -397,6 +398,19 @@ public S3FileSystemConfig setUseWebIdentityTokenCredentialsProvider(boolean useW
return this;
}

public boolean isAnonymousAccess()
{
return anonymousAccess;
Copy link
Copy Markdown
Contributor

@findinpath findinpath Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at:

I see the use of an "authentication type" field to distinguish between the many potential ways to configure the authentication for s3 access.
This can be done though EVENTUALLY in a follow-up PR.

Or if we decide to add it - then we'd be working with

s3.auth-type=ANONYMOUS

Copy link
Copy Markdown
Author

@0eu 0eu Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion! I really like this direction to be honest. s3.auth-type would be more consistent with Azure/GCS and eventually, if there would be more types, they could be easily supported. I went with a boolean flag for simplicity, but see the value in the proposed idea.

However, before we implement this (now or in the follow-up PR), we need to clarify how this will interact with existing config parameters. For example, if the user doesn't specify an auth type, should we fail with an error or use the old logic (default credentials chain)?

I would be happy moving s3.auth-type to a separate PR. But let's create an issue or track it somewhere so that S3 isn't an exception when it comes to authentication configuration. In the meantime, I will see, how this can be implemented -- I'm relatively new to the codebase, but willing to learn more and contribute :D

Copy link
Copy Markdown
Contributor

@chenjian2664 chenjian2664 Dec 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the user doesn't specify an auth type, should we fail with an error or use the old logic (default credentials chain)?

When introducing the new configuration, we should deprecate the existing one. This preserves backward compatibility while providing users with a warning, giving them a transition period to adjust if necessary. For reference, see #26681 for details.

Copy link
Copy Markdown
Author

@0eu 0eu Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing this PR, it's helpful 👍

}

@Config("s3.anonymous-access")
@ConfigDescription("Use anonymous credentials for accessing public S3 buckets")
Copy link
Copy Markdown
Member

@findepi findepi Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add config validation. when anonymous access is set, other authentication option should be left unset, as they are ignored (eg access/secret key)

see here for example validation:

trino/core/trino-main/src/test/java/io/trino/sql/TestSqlEnvironmentConfig.java

Lines 64 to 73 in b892a70

@Test
public void testInvalidPath()
{
assertFailsValidation(
new SqlEnvironmentConfig()
.setPath("too.many.parts"),
"sqlPathValid",
"sql.path must be a valid SQL path",
AssertTrue.class);
}

Copy link
Copy Markdown
Author

@0eu 0eu Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good point! I've added validation to ensure that certain options are mutually exclusive.

public S3FileSystemConfig setAnonymousAccess(boolean anonymousAccess)
{
this.anonymousAccess = anonymousAccess;
return this;
}

public String getSseCustomerKey()
{
return sseCustomerKey;
Expand All @@ -420,6 +434,21 @@ public boolean isSseWithCustomerKeyConfigValid()
return true;
}

@AssertTrue(message = "s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)")
public boolean isAnonymousAccessConfigValid()
{
if (anonymousAccess) {
return awsAccessKey == null &&
awsSecretKey == null &&
iamRole == null &&
externalId == null &&
stsEndpoint == null &&
stsRegion == null &&
!useWebIdentityTokenCredentialsProvider;
}
return true;
}

public Optional<SignerType> getSignerType()
{
return Optional.ofNullable(signerType);
Expand Down
13 changes: 11 additions & 2 deletions lib/trino-filesystem-s3/src/main/java/io/trino/filesystem/s3/S3FileSystemLoader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import io.trino.filesystem.TrinoFileSystemFactory;
import io.trino.filesystem.s3.S3Context.S3SseContext;
import jakarta.annotation.PreDestroy;
import software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
Expand Down Expand Up @@ -162,6 +163,7 @@ private static S3ClientFactory s3ClientFactory(SdkHttpClient httpClient, OpenTel
Optional<String> staticEndpoint = Optional.ofNullable(config.getEndpoint());
boolean pathStyleAccess = config.isPathStyleAccess();
boolean useWebIdentityTokenCredentialsProvider = config.isUseWebIdentityTokenCredentialsProvider();
boolean anonymousAccess = config.isAnonymousAccess();
Optional<String> staticIamRole = Optional.ofNullable(config.getIamRole());
String staticRoleSessionName = config.getRoleSessionName();
String externalId = config.getExternalId();
Expand Down Expand Up @@ -189,7 +191,10 @@ private static S3ClientFactory s3ClientFactory(SdkHttpClient httpClient, OpenTel
endpoint.map(URI::create).ifPresent(s3::endpointOverride);
s3.forcePathStyle(pathStyleAccess);

if (useWebIdentityTokenCredentialsProvider) {
if (anonymousAccess) {
s3.credentialsProvider(AnonymousCredentialsProvider.create());
}
else if (useWebIdentityTokenCredentialsProvider) {
s3.credentialsProvider(WebIdentityTokenFileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build());
Expand Down Expand Up @@ -219,6 +224,7 @@ private static S3Presigner s3PreSigner(SdkHttpClient httpClient, OpenTelemetry o
Optional<String> staticEndpoint = Optional.ofNullable(config.getEndpoint());
boolean pathStyleAccess = config.isPathStyleAccess();
boolean useWebIdentityTokenCredentialsProvider = config.isUseWebIdentityTokenCredentialsProvider();
boolean anonymousAccess = config.isAnonymousAccess();
Optional<String> staticIamRole = Optional.ofNullable(config.getIamRole());
String staticRoleSessionName = config.getRoleSessionName();
String externalId = config.getExternalId();
Expand All @@ -233,7 +239,10 @@ private static S3Presigner s3PreSigner(SdkHttpClient httpClient, OpenTelemetry o
.pathStyleAccessEnabled(pathStyleAccess)
.build());

if (useWebIdentityTokenCredentialsProvider) {
if (anonymousAccess) {
s3.credentialsProvider(AnonymousCredentialsProvider.create());
}
else if (useWebIdentityTokenCredentialsProvider) {
s3.credentialsProvider(WebIdentityTokenFileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build());
Expand Down
175 changes: 153 additions & 22 deletions lib/trino-filesystem-s3/src/test/java/io/trino/filesystem/s3/TestS3FileSystemConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import org.junit.jupiter.api.Test;

import java.util.Map;
import java.util.Set;

import static io.airlift.configuration.testing.ConfigAssertions.assertFullMapping;
import static io.airlift.configuration.testing.ConfigAssertions.assertRecordedDefaults;
Expand Down Expand Up @@ -60,6 +61,7 @@ public void testDefaults()
.setMaxErrorRetries(20)
.setSseKmsKeyId(null)
.setUseWebIdentityTokenCredentialsProvider(false)
.setAnonymousAccess(false)
.setSseCustomerKey(null)
.setStreamingPartSize(DataSize.of(32, MEGABYTE))
.setRequesterPays(false)
Expand All @@ -79,20 +81,12 @@ public void testDefaults()
.setApplicationId("Trino"));
}

@Test
public void testExplicitPropertyMappings()
private static void addCommonProperties(ImmutableMap.Builder<String, String> builder)
{
Map<String, String> properties = ImmutableMap.<String, String>builder()
.put("s3.aws-access-key", "abc123")
.put("s3.aws-secret-key", "secret")
.put("s3.endpoint", "endpoint.example.com")
builder.put("s3.endpoint", "endpoint.example.com")
.put("s3.region", "eu-central-1")
.put("s3.path-style-access", "true")
.put("s3.iam-role", "myrole")
.put("s3.role-session-name", "mysession")
.put("s3.external-id", "myid")
.put("s3.sts.endpoint", "sts.example.com")
.put("s3.sts.region", "us-west-2")
.put("s3.storage-class", "STANDARD_IA")
.put("s3.signer-type", "Aws4Signer")
.put("s3.canned-acl", "BUCKET_OWNER_FULL_CONTROL")
Expand All @@ -101,7 +95,6 @@ public void testExplicitPropertyMappings()
.put("s3.sse.type", "KMS")
.put("s3.sse.kms-key-id", "mykey")
.put("s3.sse.customer-key", "customerKey")
.put("s3.use-web-identity-token-credentials-provider", "true")
.put("s3.streaming.part-size", "42MB")
.put("s3.requester-pays", "true")
.put("s3.max-connections", "42")
Expand All @@ -117,20 +110,16 @@ public void testExplicitPropertyMappings()
.put("s3.http-proxy.password", "test")
.put("s3.http-proxy.preemptive-basic-auth", "true")
.put("s3.application-id", "application id")
.put("s3.cross-region-access", "true")
.buildOrThrow();
.put("s3.cross-region-access", "true");
}

S3FileSystemConfig expected = new S3FileSystemConfig()
.setAwsAccessKey("abc123")
.setAwsSecretKey("secret")
private static S3FileSystemConfig setCommonConfig(S3FileSystemConfig config)
{
return config
.setEndpoint("endpoint.example.com")
.setRegion("eu-central-1")
.setPathStyleAccess(true)
.setIamRole("myrole")
.setRoleSessionName("mysession")
.setExternalId("myid")
.setStsEndpoint("sts.example.com")
.setStsRegion("us-west-2")
.setStorageClass(STANDARD_IA)
.setSignerType(Aws4Signer)
.setCannedAcl(ObjectCannedAcl.BUCKET_OWNER_FULL_CONTROL)
Expand All @@ -139,7 +128,6 @@ public void testExplicitPropertyMappings()
.setMaxErrorRetries(12)
.setSseType(S3SseType.KMS)
.setSseKmsKeyId("mykey")
.setUseWebIdentityTokenCredentialsProvider(true)
.setSseCustomerKey("customerKey")
.setRequesterPays(true)
.setMaxConnections(42)
Expand All @@ -156,8 +144,53 @@ public void testExplicitPropertyMappings()
.setHttpProxyPreemptiveBasicProxyAuth(true)
.setCrossRegionAccessEnabled(true)
.setApplicationId("application id");
}

@Test
public void testExplicitPropertyMappings()
{
ImmutableMap.Builder<String, String> builder = ImmutableMap.<String, String>builder()
.put("s3.aws-access-key", "abc123")
.put("s3.aws-secret-key", "secret")
.put("s3.iam-role", "myrole")
.put("s3.external-id", "myid")
.put("s3.sts.endpoint", "sts.example.com")
.put("s3.sts.region", "us-west-2")
.put("s3.use-web-identity-token-credentials-provider", "true");
addCommonProperties(builder);
Map<String, String> properties = builder.buildOrThrow();

S3FileSystemConfig expected = setCommonConfig(new S3FileSystemConfig()
.setAwsAccessKey("abc123")
.setAwsSecretKey("secret")
.setIamRole("myrole")
.setExternalId("myid")
.setStsEndpoint("sts.example.com")
.setStsRegion("us-west-2")
.setUseWebIdentityTokenCredentialsProvider(true));

assertFullMapping(properties, expected, Set.of("s3.anonymous-access"));
}

@Test
public void testAnonymousAccessMapping()
{
ImmutableMap.Builder<String, String> builder = ImmutableMap.<String, String>builder()
.put("s3.anonymous-access", "true");
addCommonProperties(builder);
Map<String, String> properties = builder.buildOrThrow();

S3FileSystemConfig expected = setCommonConfig(new S3FileSystemConfig()
.setAnonymousAccess(true));

assertFullMapping(properties, expected);
assertFullMapping(properties, expected, Set.of(
"s3.aws-access-key",
"s3.aws-secret-key",
"s3.iam-role",
"s3.external-id",
"s3.sts.endpoint",
"s3.sts.region",
"s3.use-web-identity-token-credentials-provider"));
}

@Test
Expand All @@ -169,4 +202,102 @@ public void testSSEWithCustomerKeyValidation()
"s3.sse.customer-key has to be set for server-side encryption with customer-provided key",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithAccessKey()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setAwsAccessKey("test-key"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithSecretKey()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setAwsSecretKey("test-secret"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithIamRole()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setIamRole("arn:aws:iam::123456789012:role/test"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithExternalId()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setExternalId("external-id"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithStsEndpoint()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setStsEndpoint("sts.example.com"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithStsRegion()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setStsRegion("us-west-2"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithWebIdentityToken()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setUseWebIdentityTokenCredentialsProvider(true),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}

@Test
public void testAnonymousAccessWithMultipleAuthMethods()
{
assertFailsValidation(
new S3FileSystemConfig()
.setAnonymousAccess(true)
.setAwsAccessKey("test-key")
.setAwsSecretKey("test-secret")
.setIamRole("arn:aws:iam::123456789012:role/test"),
"anonymousAccessConfigValid",
"s3.anonymous-access cannot be used with other authentication methods (s3.aws-access-key, s3.aws-secret-key, s3.iam-role, s3.external-id, s3.sts.endpoint, s3.sts.region, s3.use-web-identity-token-credentials-provider)",
AssertTrue.class);
}
}
Loading