Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion deploy/chorus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,10 @@ storage:

### Credentials

Stored in Kubernetes Secret, separate from storage config:
Stored in Kubernetes Secrets, separate from storage config. Worker and proxy
use different credential models:

**Worker** (`credentials`) - one credential per user per storage:

```yaml
credentials:
Expand All @@ -58,6 +61,28 @@ credentials:
secretAccessKey: "..."
```

**Proxy** (`proxyCredentials`) - multiple credentials per user, keyed by alias
(`user -> alias -> credential`). The proxy authenticates S3 clients by alias
access key and re-signs forwarded requests with the target storage's credential
of the same alias:

```yaml
proxyCredentials:
storages:
main:
user1:
laptop:
accessKeyID: "..."
secretAccessKey: "..."
ci:
accessKeyID: "..."
secretAccessKey: "..."
```

Required when the proxy is enabled with `proxy.config.auth.useStorage`. Routing
and replication policies stay keyed by user; aliases only affect proxy
authentication and request signing.

### Dynamic Credentials

Manage credentials via API instead of config files:
Expand Down
15 changes: 14 additions & 1 deletion deploy/chorus/examples/values-dynamic-credentials.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,25 @@ credentials:
accessKeyID: "AKIAIOSFODNN7EXAMPLE"
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Initial proxy credentials (user -> alias -> credential)
proxyCredentials:
storages:
main:
initial-admin:
default:
accessKeyID: "BKIAIOSFODNN7EXAMPLE"
secretAccessKey: "xJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# After deployment, add more credentials via chorctl:
#
# Add S3 credentials:
# Add S3 worker credentials (used by the worker for replication):
# chorctl set-user --storage main --user new-user --type s3 \
# --access-key NEWAKEYID --secret-key NEWSECRETKEY
#
# Add S3 proxy alias credentials (accepted by the proxy s3 endpoint):
# chorctl set-user --storage main --user new-user --type s3 --alias laptop \
# --access-key ALIASKEYID --secret-key ALIASSECRETKEY
#
# Add Swift credentials:
# chorctl set-user --storage swift-main --user projectId123 --type swift \
# --swift-username myuser --swift-password mypassword \
Expand Down
15 changes: 15 additions & 0 deletions deploy/chorus/examples/values-external-redis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,28 @@ storage:
provider: Ceph
isSecure: true

# Worker credentials
credentials:
storages:
main:
user1:
accessKeyID: "AKIAIOSFODNN7EXAMPLE"
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Proxy credentials (user -> alias -> credential)
proxyCredentials:
storages:
main:
user1:
default:
accessKeyID: "BKIAIOSFODNN7EXAMPLE"
secretAccessKey: "xJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

proxy:
config:
auth:
useStorage: main

---
# Example with Redis Sentinel
# externalRedis:
Expand Down
31 changes: 29 additions & 2 deletions deploy/chorus/examples/values-s3.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ storage:
isSecure: true
defaultRegion: eu-west-1

# Storage credentials (stored in Kubernetes Secret)
# Worker storage credentials (stored in Kubernetes Secret)
# One credential per user per storage, used by the worker for replication
credentials:
storages:
main:
Expand All @@ -42,13 +43,39 @@ credentials:
accessKeyID: "BKIAI44QH8DHBEXAMPLE"
secretAccessKey: "ke7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY"

# Proxy storage credentials (stored in a separate Kubernetes Secret)
# user -> alias -> credential: the proxy authenticates S3 clients by alias
# access key and re-signs forwarded requests with the target storage's
# credential of the same alias
proxyCredentials:
storages:
main:
admin:
default:
accessKeyID: "CKIAIOSFODNN7EXAMPLE"
secretAccessKey: "yJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
backup-service:
default:
accessKeyID: "CKIAI44QH8DHBEXAMPLE"
secretAccessKey: "le7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY"

follower:
admin:
default:
accessKeyID: "DKIAIOSFODNN7EXAMPLE"
secretAccessKey: "zJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
backup-service:
default:
accessKeyID: "DKIAI44QH8DHBEXAMPLE"
secretAccessKey: "me7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY"

# Proxy configuration
proxy:
enabled: true
replicas: 2
config:
auth:
# Use credentials from main storage for S3 proxy authentication
# Authenticate S3 clients against main storage proxy credentials
useStorage: main

# Worker configuration
Expand Down
23 changes: 16 additions & 7 deletions deploy/chorus/examples/values-swift.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ storage:
provider: Ceph
isSecure: true

# Credentials (stored in Kubernetes Secret)
# Worker credentials (stored in Kubernetes Secret)
credentials:
storages:
# Swift credentials - key must be OpenStack Project ID
Expand All @@ -53,18 +53,27 @@ credentials:
accessKeyID: "AKIAIOSFODNN7EXAMPLE"
secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Proxy S3 credentials (stored in a separate Kubernetes Secret)
# user -> alias -> credential. Only needed for the S3 storage: for Swift the
# proxy does not perform authentication itself
proxyCredentials:
storages:
s3-backup:
swift-admin:
default:
accessKeyID: "BKIAIOSFODNN7EXAMPLE"
secretAccessKey: "xJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Proxy configuration
proxy:
enabled: true
replicas: 2
config:
auth:
# For Swift, you typically use custom credentials or disable auth
# since Swift handles its own authentication
custom:
swift-admin:
accessKeyID: "custom-access-key"
secretAccessKey: "custom-secret-key"
# S3 clients authenticate against the S3 storage proxy credentials.
# Swift requests are not authenticated by the proxy - Swift handles
# its own authentication
useStorage: s3-backup

# Worker configuration
worker:
Expand Down
20 changes: 20 additions & 0 deletions deploy/chorus/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,17 @@ Credentials secret name
{{- end }}
{{- end }}

{{/*
Proxy credentials secret name
*/}}
{{- define "chorus.proxyCredentialsSecretName" -}}
{{- if .Values.existingProxyCredentialsSecret }}
{{- .Values.existingProxyCredentialsSecret }}
{{- else }}
{{- printf "%s-proxy-credentials" (include "chorus.fullname" .) }}
{{- end }}
{{- end }}

{{/*
Redis secret name
*/}}
Expand Down Expand Up @@ -158,6 +169,15 @@ true
{{- end }}
{{- end }}

{{/*
Check if proxy credentials secret should be created
*/}}
{{- define "chorus.createProxyCredentialsSecret" -}}
{{- if and (not .Values.existingProxyCredentialsSecret) .Values.proxyCredentials.storages }}
true
{{- end }}
{{- end }}

{{/*
Dynamic credentials secret name
*/}}
Expand Down
8 changes: 4 additions & 4 deletions deploy/chorus/templates/proxy/deployment-proxy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
{{- end }}
annotations:
checksum/config: {{ include (print $.Template.BasePath "/proxy/config.yaml") . | sha256sum }}
{{- if include "chorus.createCredentialsSecret" . }}
{{- if include "chorus.createProxyCredentialsSecret" . }}
checksum/secret: {{ include (print $.Template.BasePath "/secrets/secret.yaml") . | sha256sum }}
{{- end }}
{{- if .Values.metrics.enabled }}
Expand Down Expand Up @@ -108,7 +108,7 @@ spec:
- name: config
mountPath: /bin/config/config.yaml
subPath: config.yaml
{{- if or .Values.existingCredentialsSecret (include "chorus.createCredentialsSecret" .) }}
{{- if or .Values.existingProxyCredentialsSecret (include "chorus.createProxyCredentialsSecret" .) }}
- name: credentials
mountPath: /bin/config/override.yaml
subPath: config.yaml
Expand All @@ -117,10 +117,10 @@ spec:
- name: config
configMap:
name: {{ include "chorus.fullname" . }}-proxy
{{- if or .Values.existingCredentialsSecret (include "chorus.createCredentialsSecret" .) }}
{{- if or .Values.existingProxyCredentialsSecret (include "chorus.createProxyCredentialsSecret" .) }}
- name: credentials
secret:
secretName: {{ include "chorus.credentialsSecretName" . }}
secretName: {{ include "chorus.proxyCredentialsSecretName" . }}
{{- end }}
{{- $nodeSelector := include "chorus.nodeSelector" (list .Values.proxy.nodeSelector .Values.global.nodeSelector) }}
{{- if $nodeSelector }}
Expand Down
24 changes: 23 additions & 1 deletion deploy/chorus/templates/secrets/secret.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- /* Storage credentials secret */}}
{{- /* Worker storage credentials secret (user -> credential) */}}
{{- if include "chorus.createCredentialsSecret" . }}
apiVersion: v1
kind: Secret
Expand All @@ -24,6 +24,28 @@ stringData:
{{- end }}
{{- end }}
---
{{- /* Proxy storage credentials secret (user -> alias -> credential) */}}
{{- if include "chorus.createProxyCredentialsSecret" . }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "chorus.proxyCredentialsSecretName" . }}
labels:
{{- include "chorus.labels" . | nindent 4 }}
type: Opaque
stringData:
config.yaml: |
{{- if .Values.proxyCredentials.storages }}
storage:
storages:
{{- range $storageName, $users := .Values.proxyCredentials.storages }}
{{ $storageName }}:
credentials:
{{- $users | toYaml | nindent 12 }}
{{- end }}
{{- end }}
{{- end }}
---
{{- /* Dynamic credentials master password secret */}}
{{- if include "chorus.createDynamicCredentialsSecret" . }}
apiVersion: v1
Expand Down
46 changes: 39 additions & 7 deletions deploy/chorus/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,10 @@ storage:
# =============================================================================
# CREDENTIALS (Secret)
# =============================================================================
# Storage credentials - stored in Kubernetes Secret
# Worker storage credentials - stored in Kubernetes Secret
# One credential per user per storage. Used by the worker for replication.
credentials:
# Per-storage credentials
# Per-storage credentials: user -> credential
storages: {}
# S3 credentials example:
# main:
Expand All @@ -105,9 +106,38 @@ credentials:
# tenantName: admin

# Use existing secret instead of creating one
# Secret must have 'config' key with YAML content matching override.yaml format
# Secret must have 'config.yaml' key with YAML content matching worker override.yaml format
existingCredentialsSecret: ""

# =============================================================================
# PROXY CREDENTIALS (Secret)
# =============================================================================
# Proxy S3 storage credentials - stored in a separate Kubernetes Secret.
# Unlike the worker, the proxy supports multiple credentials per user:
# user -> alias -> credential. The proxy authenticates callers by alias access
# key, keeps routing/replication policy keyed by user, and re-signs forwarded
# requests with the target storage's credential of the same alias name.
proxyCredentials:
# Per-storage S3 credentials: user -> alias -> credential
storages: {}
# main:
# user1:
# laptop: # alias name, joins credentials across storages
# accessKeyID: "AKIAIOSFODNN7EXAMPLE"
# secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
# ci:
# accessKeyID: "AKIAI44QH8DHBEXAMPLE"
# secretAccessKey: "je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY"
# follower:
# user1:
# laptop:
# accessKeyID: "BKIAIOSFODNN7EXAMPLE"
# secretAccessKey: "xJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Use existing secret instead of creating one
# Secret must have 'config.yaml' key with YAML content matching proxy override.yaml format
existingProxyCredentialsSecret: ""

# =============================================================================
# FEATURES
# =============================================================================
Expand Down Expand Up @@ -242,11 +272,13 @@ proxy:

auth:
allowV2Signature: false
useStorage: "" # Use credentials from this storage
custom: {} # Or define custom credentials
useStorage: "" # Use proxy credentials of this storage (see proxyCredentials)
custom: {} # Or define custom credentials: user -> alias -> credential.
# Each (user, alias) pair must exist in the main storage proxy credentials.
# user1:
# accessKeyID: xxx
# secretAccessKey: xxx
# alias1:
# accessKeyID: xxx
# secretAccessKey: xxx

# =============================================================================
# WORKER
Expand Down
5 changes: 3 additions & 2 deletions docker-compose/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
│   ├── s3cmd-follower.conf # s3cmd credentials for follower storage
│   ├── s3cmd-main.conf # s3cmd credentials for main storage
│   ├── s3cmd-proxy.conf # s3cmd credentials for proxy storage
│   ├── s3-credentials.yaml # chorus common config with S3 credentials
│   ├── s3-credentials.yaml # chorus worker config with S3 credentials
│   ├── proxy-s3-credentials.yaml # chorus proxy config with S3 alias credentials
│   └── worker-conf.yaml # example config for chorus-worker
```

Expand Down Expand Up @@ -113,7 +114,7 @@ Try to add more worker instances to speed up replication process and observe how
```
The same can be done for `worker3`, `worker4`, etc. And for `proxy` service.

Replace S3 credentials in [./s3-credentials.yaml](./s3-credentials.yaml) with your own s3 storages and start docker-compose without fake backends:
Replace S3 credentials in [./s3-credentials.yaml](./s3-credentials.yaml) and [./proxy-s3-credentials.yaml](./proxy-s3-credentials.yaml) with your own s3 storages and start docker-compose without fake backends:
```shell
docker-compose -f ./docker-compose/docker-compose.yml --profile proxy up
```
Expand Down
4 changes: 2 additions & 2 deletions docker-compose/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ services:
- type: bind
source: ./proxy-conf.yaml
target: /bin/config/config.yaml
# common config: s3 credentials
# proxy s3 credentials (user -> alias -> credential)
- type: bind
source: ./s3-credentials.yaml
source: ./proxy-s3-credentials.yaml
target: /bin/config/override.yaml
ports:
- "9669:9669" # expose proxy s3 api
Expand Down
Loading