Use S3Proxy to access object storage so we can swap out cloud storage engines#29
Merged
chrisghill merged 6 commits intomainfrom Jan 24, 2026
Merged
Use S3Proxy to access object storage so we can swap out cloud storage engines#29chrisghill merged 6 commits intomainfrom
chrisghill merged 6 commits intomainfrom
Conversation
chrisghill
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request introduces S3Proxy as an abstraction layer for object storage access in Massdriver, enabling support for multiple cloud storage backends (MinIO, AWS S3, Google Cloud Storage, and Azure Blob Storage) through a unified interface. The change replaces direct storage access with a proxy-based approach that simplifies multi-cloud deployments.
Changes:
- Introduced S3Proxy deployment with support for MinIO, S3, GCS, and Azure Blob Storage backends
- Refactored Massdriver application to connect to S3Proxy instead of directly to storage backends
- Consolidated storage credentials management and added configuration for multiple cloud providers
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 19 comments.
Show a summary per file
| File | Description |
|---|---|
| charts/massdriver/values.yaml | Added S3Proxy configuration section with credentials, log levels, and provider-specific settings for minio, s3, gcs, and azureblob |
| charts/massdriver/templates/s3proxy/serviceaccount.yaml | Created ServiceAccount for S3Proxy with configurable annotations for IAM role bindings |
| charts/massdriver/templates/s3proxy/service.yaml | Created Kubernetes Service to expose S3Proxy on port 80 |
| charts/massdriver/templates/s3proxy/secret-envs.yaml | Created Secret containing S3Proxy and JClouds configuration environment variables for all supported storage backends |
| charts/massdriver/templates/s3proxy/deployment.yaml | Created S3Proxy Deployment with 2 replicas, health probes, and security context |
| charts/massdriver/templates/massdriver/secret-minio.yaml | Simplified to remove duplicate auth secret; now only creates MinIO root credentials |
| charts/massdriver/templates/massdriver/secret-envs.yaml | Updated to use S3Proxy credentials instead of backend-specific credentials |
| charts/massdriver/templates/massdriver/job-db-migration.yaml | Removed MinIO auth secret reference as authentication now goes through S3Proxy |
| charts/massdriver/templates/massdriver/deployment.yaml | Removed MinIO secret checksum annotation and MinIO auth secret reference |
| charts/massdriver/templates/massdriver/configmap-envs.yaml | Updated blob storage endpoint to point to S3Proxy service instead of backend-specific endpoints |
| charts/massdriver/templates/_helpers.tpl | Removed minio.username and minio.password helper functions as they're no longer needed |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
coryodaniel
approved these changes
Jan 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces S3Proxy as an abstraction layer for object storage access in Massdriver, enabling support for multiple cloud storage backends (MinIO, AWS S3, Google Cloud Storage, and Azure Blob Storage) through a unified interface. The change replaces direct storage access with a proxy-based approach that simplifies multi-cloud deployments.
Changes: