Harden secrets at rest using envelope encryption#6211
Conversation
There was a problem hiding this comment.
Pull request overview
Hardens persisted secrets in Chronograf's KV store by encrypting Source.Password, Source.SharedSecret, Source.ManagementToken, Source.DatabaseToken, and Server.Password using AES‑256‑GCM envelope encryption (a DEK wrapped by a master key). Adds startup migration of legacy plaintext records, fail‑closed behavior when a master key is missing but encrypted records exist, and three new chronoctl commands for key generation, rotation, and rollback. Also redacts databaseToken and managementToken from API responses and updates Swagger descriptions.
Changes:
- New envelope crypto in
kv/internal(versioned[v1|nonce|ciphertext]format) plus per‑fieldSecretEncodingenum onSource/Serverproto messages. - New
kv/secrets.gowithInitializeSecretDEK,RewrapSecretDEK,DisableSecretEncryption, and migration of legacy plaintext records on startup. - New server flags (
--secrets-master-key,--secrets-master-key-file), wiring intoopenServicewithdefer clear(...)of the master key, response redaction of new token fields, and three newchronoctlsubcommands.
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| kv/internal/crypto.go | New AES‑GCM seal/open primitives and DEK wrap/unwrap helpers. |
| kv/internal/internal.go | DEK process‑global state and per‑field marshal/unmarshal that transparently encrypt/decrypt secret strings. |
| kv/internal/internal.proto / internal.pb.go | Adds SecretEncoding enum and per‑secret encoding fields on Source/Server. |
| kv/internal/internal_test.go, crypto_test.go | Unit tests for crypto and marshaling roundtrips. |
| kv/secrets.go | Wrapped‑DEK lifecycle, fail‑closed checks, migration, and rollback to plaintext. |
| kv/kv_test.go | Integration tests for migration and DEK lifecycle. |
| server/server.go | Adds master key flags, loadSecretsMasterKey, wires DEK init into openService, clears key bytes after use. |
| server/server_test.go | Unit tests for loadSecretsMasterKey covering both flag and file inputs. |
| server/sources.go | Redacts DatabaseToken and ManagementToken in API responses. |
| server/sources_test.go | Updates expected response bodies to reflect redaction. |
| server/swagger.json | Updates secret field descriptions to indicate write‑only/redacted behavior. |
| cmd/chronoctl/gen_secrets_master_key.go | New command to generate base64 32‑byte master keys (contains "non-existant" typo). |
| cmd/chronoctl/rewrap_secrets_master_key.go | New command to rotate the master key by rewrapping the stored DEK. |
| cmd/chronoctl/disable_secrets_encryption.go | New command to decrypt persisted secrets and remove the wrapped DEK. |
| cmd/chronoctl/main_test.go | Test helper updates for new commands. |
| CHANGELOG.md, cmd/chronoctl/README.md | Documents new feature and commands. |
Files not reviewed (1)
- kv/internal/internal.pb.go: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Manual test plan:
|
|
After running some exploratory tests I've uncovered some minor issues. General
Using a non-default database and files for key storage reveals two issues. See the Exploratory section below.
|
|
@karel-rehor please copy tests results also here … of course without confidential information 😉 |
Provided test plan results1) Baseline2026-05-25 17:00 PASSED
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
In second terminal $ ls -al chronograf-v1.db
-rw------- 1 karl karl 32768 May 25 14:50 chronograf-v1.dbOK.
2) Basic startup/migration with SECRETS_MASTER_KEY2026-05-26 10:00 PASSED
$ ./chronoctl gen-secrets-master-key
rCFXqeLS2kKdOJgtitIlk3l4nZZfQEuzQRIialLVT0s=
3) Basic restart with SECRETS_MASTER_KEY2026-05-26 10:15 PASSED
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key="rCFXqeLS2kKdOJgtitIlk3l4nZZfQEuzQRIialLVT0s="
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
4) Basic restart without SECRETS_MASTER_KEY2026-05-26 10:15 PASSED
$ ./chronograf --influxdb-v3-support-enabled
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
ERRO[0000] Unable to initialize secret encryptionwrapped DEK exists but no secrets master key is configured component=Secrets
5) Basic rewrap and then restart2026-05-26 10:45 PASSED
$ ./chronoctl rewrap-secrets-master-key
old secrets master key is required
$ ./chronoctl gen-secrets-master-key
IRd88h8Q6mGjPG0jnP8Z5N2KpHI2NWH7R0BsKJqs3SY=
$ ./chronoctl rewrap-secrets-master-key --old-secrets-master-key="rCFXqeLS2kKdOJgtitIlk3l4nZZfQEuzQRIialLVT0s=" --new-secrets-master-key="IRd88h8Q6mGjPG0jnP8Z5N2KpHI2NWH7R0BsKJqs3SY="
Successfully rewrapped DEK with new secrets master key
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key="IRd88h8Q6mGjPG0jnP8Z5N2KpHI2NWH7R0BsKJqs3SY="
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
$ strings chronograf-v1.db | grep "apiv3"
$ strings chronograf-v1.db | grep "changeit"
$6) Disable secrets encryption2026-05-26 11:00 PASSED
$ ./chronoctl disable-secrets-encryption --help
Usage:
chronoctl [OPTIONS] disable-secrets-encryption [disable-secrets-encryption-OPTIONS]
...
$ ./chronoctl disable-secrets-encryption
current secrets master key is required
$ ./chronoctl disable-secrets-encryption --secrets-master-key="IRd88h8Q6mGjPG0jnP8Z5N2KpHI2NWH7R0BsKJqs3SY="
Successfully disabled secrets encryption and removed wrapped DEK
$ ./chronograf --influxdb-v3-support-enabled
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] Moving from version 1.8.0
INFO[0000] Moving to version 202605251210~2bb6296
INFO[0000] Successfully created backup/chronograf-v1.db.1.8.0
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
$ strings chronograf-v1.db | grep "apiv3"
\apiv3_kbyT09fqW464tDApPghYZyeX4I_AbInpdeAbIZyufaWxXEY1rpeqW7UPjVYtNBsDWh0B0muyIuJGZpfzSowwDAUsersV2
$ strings chronograf-v1.db | grep "changeit"
changeit*7) Check chronograf endpoints2026-05-26 11:00 PASSED
8) Check kapacitor endpoints2026-05-26 14:00 PASSED
$ curl -s -u admin:changeit http://localhost:9092/kapacitor/v1/config | jq | grep "password"
"password": false,
"password"
"password": false,
"password",
"password": false,
"password"
"password": false,
"password",
"password": false,
"password"
"password": false,
"password"
"password": false,
"password"
### unknown user
$ curl -s -u foo:wumpus http://localhost:9092/kapacitor/v1/users
{"error":"authorization failed","message":"authorization failed"}
### actual admin
$ curl -s -u admin:changeit http://localhost:9092/kapacitor/v1/users
{
"users": [
{
"link": {
"rel": "self",
"href": "/kapacitor/v1/users/admin"
},
"name": "admin",
"permissions": [],
"type": "admin"
}
]
}
$ curl -s -u admin:changeit http://localhost:9092/kapacitor/v1/users/admin
{
"link": {
"rel": "self",
"href": "/kapacitor/v1/users/admin"
},
"name": "admin",
"type": "admin",
"permissions": []
} |
Exploratory testing notesInterim Summary Using a non-default database and files for key storage reveals one potential issue.
Fresh non-default database and SECRETS_MASTER_KEY in file
$ ./chronoctl gen-secrets-master-key --out=super-key.dek
Secrets master key generated and saved at super-key.dek
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key-file=super-key.dek --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
INFO[0000] Response: OK component=server method=GET remote_addr="[::1]:57338" response_time="195.249µs" status=200
$ strings my-chronograf.db | grep "apiv3"
karl@bannock:~/bonitoo/qa/testing/chronograf/PR6211$ strings my-chronograf.db | grep "changeit"
karl@bannock:~/bonitoo/qa/testing/chronograf/PR6211$ strings my-chronograf.db | grep "admin"
admin"4AauTmcT7lCa6+EiE1fTz9R3QuknpNrOAqtzK7AjTVmOXOiJh3A==*
$ ./chronograf --influxdb-v3-support-enabled --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] Moving from version 1.8.0
INFO[0000] Moving to version 202605251210~2bb6296
INFO[0000] Successfully created backup/my-chronograf.db.1.8.0
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
ERRO[0000] Unable to initialize secret encryptionwrapped DEK exists but no secrets master key is configured component=Secrets
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key-file=super-key.dek --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com"
$ ./chronoctl gen-secrets-master-key --out=meta-key.dek
Secrets master key generated and saved at meta-key.dek
### see what happens if db file is not found
$ ./chronoctl rewrap-secrets-master-key --old-secrets-master-key-file=super-key.dek --new-secrets-master-key-file=meta-key.dek
wrapped DEK not found
### !!! odd message
### !!! it appears a default deb chronograf-v1.db was suddenly added by the above command.
$ ls -al
total 209100
drwxrwxr-x 3 karl karl 4096 May 26 16:19 .
drwxrwxr-x 10 karl karl 4096 May 25 13:51 ..
drwx------ 2 karl karl 4096 May 26 16:10 backup
-rwxrwxr-x 1 karl karl 83812552 May 25 14:30 chronoctl
-rwxrwxr-x 1 karl karl 130221224 May 25 14:29 chronograf
-rw------- 1 karl karl 32768 May 26 16:19 chronograf-v1.db
-rw------- 1 karl karl 45 May 26 16:15 meta-key.dek
-rw------- 1 karl karl 32768 May 26 16:11 my-chronograf.db
-rw------- 1 karl karl 45 May 26 15:07 super-key.dek
-rw-rw-r-- 1 karl karl 1157 May 25 13:51 testing_notes.md
### continuing with param for non-default db.
$ ./chronoctl rewrap-secrets-master-key --old-secrets-master-key-file=super-key.dek --new-secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
Successfully rewrapped DEK with new secrets master key
### ^^^ OK
### start with wrapper key
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] Moving from version 1.8.0
INFO[0000] Moving to version 202605251210~2bb6296
INFO[0000] Successfully created backup/my-chronograf.db.1.8.0
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
$ strings my-chronograf.db | grep "apiv3"
$ strings my-chronograf.db | grep "changeit"
$ strings my-chronograf.db | grep "admin"
admin"4AauTmcT7lCa6+EiE1fTz9R3QuknpNrOAqtzK7AjTVmOXOiJh3A==*
$ ./chronoctl disable-secrets-encryption --secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
Successfully disabled secrets encryption and removed wrapped DEK
$ ./chronograf --influxdb-v3-support-enabled --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] Moving from version 1.8.0
INFO[0000] Moving to version 202605251210~2bb6296
INFO[0000] Successfully created backup/my-chronograf.db.1.8.0
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
$ strings my-chronograf.db | grep "changeit"
changeit*
$ strings my-chronograf.db | grep "apiv3"
\apiv3_kbyT09fqW464tDApPghYZyeX4I_AbInpdeAbIZyufaWxXEY1rpeqW7UPjVYtNBsDWh0B0muyIuJGZpfzSowwDAUsersV2
|
|
@karel-rehor not working bash completion in |
|
@karel-rehor Thank you for the thorough testing and report. The issue of |
|
In CircleCI I rebuilt the nightly-build binaries based on commit 80ef3d78 Using the build for linux amd64 I repeated the exploratory test that uncovered the
$ ./chronoctl gen-secrets-master-key --out=meta-key.dek
Secrets master key generated and saved at meta-key.dek
### see what happens if db file is not found
$ ./chronoctl rewrap-secrets-master-key --old-secrets-master-key-file=super-key.dek --new-secrets-master-key-file=meta-key.dek
bolt db "chronograf-v1.db" not found; use --bolt-path to target an existing chronograf database
### nice message - issue fixed
### no new default deb chronograf-v1.db was added by the above command.
$ ls -al
total 418188
drwxrwxr-x 3 karl karl 4096 May 27 15:05 .
drwxrwxr-x 10 karl karl 4096 May 25 13:51 ..
drwx------ 2 karl karl 4096 May 26 16:23 backup
-rwxrwxr-x 1 karl karl 83841104 May 27 14:56 chronoctl
-rwxrwxr-x 1 karl karl 83812552 May 25 14:30 chronoctl-2bb629
-rwxrwxr-x 1 karl karl 130274888 May 27 14:56 chronograf
-rwxrwxr-x 1 karl karl 130221224 May 25 14:29 chronograf-2bb629
-rw------- 1 karl karl 45 May 27 15:05 meta-key.dek
-rw------- 1 karl karl 32768 May 27 15:03 my-chronograf.db
-rw------- 1 karl karl 45 May 27 14:58 super-key.dek
-rw-rw-r-- 1 karl karl 1157 May 25 13:51 testing_notes.md
### continuing with param for non-default db.
$ ./chronoctl rewrap-secrets-master-key --old-secrets-master-key-file=super-key.dek --new-secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
Successfully rewrapped DEK with new secrets master key
### ^^^ OK
### start with wrapper key
$ ./chronograf --influxdb-v3-support-enabled --secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
INFO[0000] Starting Chronograf 202605251210~2bb62962bb6296a0c99ad862ad83fb954849517388d175e
INFO[0000] Moving from version 1.8.0
INFO[0000] Moving to version 202605251210~2bb6296
INFO[0000] Successfully created backup/my-chronograf.db.1.8.0
INFO[0000] InfluxDB v3 time condition validated and configured component=server time_condition="time > now() - 1d"
INFO[0000] Serving chronograf at http://[::]:8888 component=server
INFO[0000] Reporting usage stats component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime"
also
### repeat check of missing bolt path - OK
$ ./chronoctl disable-secrets-encryption --secrets-master-key-file=meta-key.dek
bolt db "chronograf-v1.db" not found; use --bolt-path to target an existing chronograf database
### Now disable encryption
$ ./chronoctl disable-secrets-encryption --secrets-master-key-file=meta-key.dek --bolt-path=my-chronograf.db
Successfully disabled secrets encryption and removed wrapped DEKI'll now continue with code review. |
This PR hardens secret handling in Chronograf by encrypting persisted secret fields in the KV store using envelope encryption (DEK wrapped by a master key).
It covers
Source.Password,Source.SharedSecret,Source.ManagementToken,Source.DatabaseToken, andServer.Password, while keeping runtime/API models unchanged.Migration and Rollback
disable-secrets-encryptiondecrypts persisted secrets back to plaintext and removes the wrapped DEK.❗ Note for reviewers: If operational preference is to make migration explicit as well / or only, a dedicated
chronoctlmigration command can be added.New
chronoctlcommandsgen-secrets-master-key— generate a base64 32-byte master key (stdout or file).rewrap-secrets-master-key— rotate master key wrapping for the stored DEK (does not re-encrypt secret records).disable-secrets-encryption— decrypt persisted secrets and remove wrapped DEK.