Custom S3 repositories

[PacoDu]

We have a use case that is not covered by this extension:

We manage custom S3 repositories (via minio), if we intent to use this extension we should use the storage:platform OTHER, but this doesn't allow us to distinguish which minio instance is hosting the data. Could we imagine something like storage:host or storage:endpoint in case storage:platform is set to OTHER ?

This is a feature available in aws s3 cli, for exemple we could use the following command aws --endpoint <storage:endpoint> ls <href>

Example:

"CO_GEOTIFF_MINIO_RGB": {
      "href": "s3://naip-visualization/tx/2016/100cm/rgb/30097/m_3009743_sw_14_1_20160928.tif",
      "type": "image/vnd.stac.geotiff; cloud-optimized=true",
      "storage:platform": "OTHER",
      "storage:region": "region-1",
      "storage:endpoint": "https://s3-compliant-endpoint.com"
}

The issue is similar for providers not covered by the spec (Scaleway for exemple) and I assume many other "smaller" providers.

[kylebarron]

Seems also relevant for storage providers that aim to be S3-compliant, like Digital Ocean Spaces.

[fabricebrito]

Yes, having the S3 service URL is a must if you want to address any S3 provider

[m-mohr]

I agree that supporting more S3 providers would be great. It seems rather weird that the endpoint is not part of the URI though, but I guess that ship has sailed.

The endpoint seems to be a field that is not generic across storage providers. Maybe we can encode this in storage:platform?

[jdries]

Indeed, hitting this as well. There's also the question of path style vs virtual hosted style:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access

By default, S3 libraries often assume virtual hosting is possible, but a lot of providers don't support it.
Why not just have an extra property for the S3 endpoint url?

[m-mohr]

Because this would be an S3 specific field :) Naturally it would ideally be in a S3 extension.

But there is consensus that the way it is, is not good enough.

So who has the time to work on this? Ideally this could also work on the deduplication in assets, similar to the authentication extension.

[jdries]

also note that other properties like 'region' and 'requester pays' seem to come from an S3 context.
Most of the storage providers mentioned here in fact offer some form of S3 compatible interface. So this starts to look more like an 'object storage' extension rather than a general 'storage' extension.

For instance, if I were to point to POSIX file paths, most of the options in this extension would not really apply.

[m-mohr]

also note that other properties like 'region' and 'requester pays' seem to come from an S3 context.

Region is not just an S3 concept, see e.g. https://cloud.google.com/storage/docs/locations?hl=de
Same for requester_pays, see e.g https://cloud.google.com/storage/docs/requester-pays?hl=de

So this starts to look more like an 'object storage' extension rather than a general 'storage' extension.

That is true, indeed, the extension name is not ideal. But the he extension describes itself:

It allows adding details related to cloud storage access [...] This extension does not cover NFS solutions

[m-mohr]

Please review the PR for version 2 that is meant to solve this issue: #24 @jdries @PacoDu @kylebarron @fabricebrito