Skip to content

fix(helm): Generate strong Temporal payload encryption key - #4453

Open
kfelternv wants to merge 2 commits into
NVIDIA:mainfrom
kfelternv:temporal-payload-key
Open

fix(helm): Generate strong Temporal payload encryption key#4453
kfelternv wants to merge 2 commits into
NVIDIA:mainfrom
kfelternv:temporal-payload-key

Conversation

@kfelternv

Copy link
Copy Markdown
Contributor

Fresh NICo REST installs currently use a fixed Temporal payload encryption key. This change generates the key from 32 random bytes while preserving existing Secrets on upgrades and honoring explicit values.

Related issues

None.

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

Existing installations keep their current key during upgrades so encrypted Temporal payloads remain readable.

@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features

    • Automatically generates a secure 32-byte Temporal encryption key when none is configured during a fresh installation.
    • Preserves explicitly configured keys and reuses existing keys during upgrades.
  • Bug Fixes

    • Prevents replacement of an existing encryption key during chart updates.
    • Fails upgrades when no configured or existing key is available.
    • Honors the setting to disable Secret creation.

Walkthrough

The Helm chart resolves the Temporal encryption key from configured values, an existing Kubernetes Secret, or generated 32-byte random data. Upgrade behavior, failure cases, and disabled Secret creation are covered by Helm tests.

Changes

Temporal encryption key handling

Layer / File(s) Summary
Key resolution and Secret validation
helm/rest/nico-rest/charts/nico-rest-common/values.yaml, helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml, helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml
The chart documents key precedence, reuses an existing key during upgrades, generates a key on fresh installs, and fails upgrades when no key is available. Tests cover these cases and disabled Secret creation.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HelmValues
  participant SecretsTemplate
  participant KubernetesSecret
  HelmValues->>SecretsTemplate: provide configured key and create setting
  SecretsTemplate->>KubernetesSecret: read existing key during upgrade
  KubernetesSecret-->>SecretsTemplate: return decoded key
  SecretsTemplate-->>KubernetesSecret: render resolved Secret or fail
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the primary change: generating a strong Temporal payload encryption key in the Helm chart.
Description check ✅ Passed The description directly explains key generation, upgrade preservation, explicit values, testing, and the related fix.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@kfelternv
kfelternv force-pushed the temporal-payload-key branch from 4bbc7a3 to a31f289 Compare July 31, 2026 19:17
@kfelternv
kfelternv requested review from a team and thossain-nv July 31, 2026 19:18

Copy link
Copy Markdown
Contributor Author

What changed

Fresh NICo REST installs now generate the Temporal payload encryption key from 32 random bytes. Explicit values remain supported, Secret creation can still be disabled, and upgrades reuse the existing key.

Scenario and setup

Tested commit a31f289da21badec5c076e857a481f1dccf425c1 with Helm v3.19.0 against an isolated local Kind Kubernetes v1.34.0 cluster. Each fresh or explicit-value scenario used a new namespace. The parent nico-rest chart supplied the required global values to the nico-rest-common dependency. The six focused chart tests passed, and make rest-helm-lint linted all three REST charts with zero failures.

Verification

Step 1: Generate a 32-byte key for a fresh install

Why this step exists: It verifies the empty default value reaches the random-key path and creates a usable Kubernetes Secret.

Runnable command:

kubectl create namespace temporal-key-fresh
helm template temporal-key-pr4453 helm/rest/nico-rest \
  --namespace temporal-key-fresh \
  --set nico-rest-api.config.keycloak.enabled=true \
  --show-only charts/nico-rest-common/templates/secrets.yaml |
  kubectl apply -f -
kubectl -n temporal-key-fresh get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' |
  base64 -D | base64 -D | wc -c

Observed result:

namespace/temporal-key-fresh created
secret/keycloak-client-secret created
secret/temporal-encryption-key created
secret/db-creds created
secret/image-pull-secret created
32

Why this proves the behavior: Decoding the Kubernetes Secret value and then the value returned by Helm's randBytes produced exactly 32 bytes.

Step 2: Generate a different key for another fresh install

Why this step exists: It verifies fresh installs do not reuse a fixed or deterministic default.

Runnable command:

FRESH_ONE_HASH=$(kubectl -n temporal-key-fresh get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' | shasum -a 256 | awk '{print $1}')
kubectl create namespace temporal-key-fresh-two
helm template temporal-key-pr4453 helm/rest/nico-rest \
  --namespace temporal-key-fresh-two \
  --set nico-rest-api.config.keycloak.enabled=true \
  --show-only charts/nico-rest-common/templates/secrets.yaml |
  kubectl apply -f -
FRESH_TWO_HASH=$(kubectl -n temporal-key-fresh-two get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' | shasum -a 256 | awk '{print $1}')
if [ "$FRESH_ONE_HASH" != "$FRESH_TWO_HASH" ]; then
  printf '%s\n' 'fresh key hashes differ'
else
  printf '%s\n' 'fresh key hashes matched unexpectedly'
  false
fi

Observed result:

namespace/temporal-key-fresh-two created
secret/keycloak-client-secret created
secret/temporal-encryption-key created
secret/db-creds created
secret/image-pull-secret created
fresh key hashes differ

Why this proves the behavior: Two independent fresh renders stored different key values.

Step 3: Honor an explicitly configured key

Why this step exists: It verifies operators can still provide a specific value.

Runnable command:

kubectl create namespace temporal-key-explicit
helm template temporal-key-pr4453 helm/rest/nico-rest \
  --namespace temporal-key-explicit \
  --set nico-rest-api.config.keycloak.enabled=true \
  --set-string nico-rest-common.secrets.temporalEncryptionKey.value=configured-key \
  --show-only charts/nico-rest-common/templates/secrets.yaml |
  kubectl apply -f -
kubectl -n temporal-key-explicit get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' | base64 -D

Observed result:

namespace/temporal-key-explicit created
secret/keycloak-client-secret created
secret/temporal-encryption-key created
secret/db-creds created
secret/image-pull-secret created
configured-key

Why this proves the behavior: The stored Secret contains the exact configured value instead of a generated replacement.

Step 4: Keep Secret creation disabled when requested

Why this step exists: It verifies the existing secrets.create=false contract remains unchanged.

Runnable command:

helm template temporal-key-pr4453 helm/rest/nico-rest \
  --namespace temporal-key-disabled \
  --set nico-rest-api.config.keycloak.enabled=true \
  --set nico-rest-common.secrets.create=false \
  --show-only charts/nico-rest-common/templates/secrets.yaml |
  awk '$0 == "kind: Secret" {count++} END {print count+0}'

Observed result:

0

Why this proves the behavior: The common Secret template rendered zero Secret objects when creation was disabled.

Step 5: Reuse the existing key during an upgrade

Why this step exists: It verifies the live-cluster lookup path preserves the key that protects existing Temporal payloads.

Runnable command:

UPGRADE_BEFORE=$(kubectl -n temporal-key-fresh get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' | shasum -a 256 | awk '{print $1}')
helm template temporal-key-pr4453 helm/rest/nico-rest \
  --dry-run=server \
  --is-upgrade \
  --namespace temporal-key-fresh \
  --set nico-rest-api.config.keycloak.enabled=true \
  --show-only charts/nico-rest-common/templates/secrets.yaml |
  kubectl apply -f -
UPGRADE_AFTER=$(kubectl -n temporal-key-fresh get secret temporal-encryption-key \
  -o jsonpath='{.data.temporal-encryption-key}' | shasum -a 256 | awk '{print $1}')
if [ "$UPGRADE_BEFORE" = "$UPGRADE_AFTER" ]; then
  printf '%s\n' 'upgrade preserved key hash'
else
  printf '%s\n' 'upgrade changed key hash unexpectedly'
  false
fi

Observed result:

secret/keycloak-client-secret configured
secret/temporal-encryption-key configured
secret/db-creds configured
secret/image-pull-secret unchanged
upgrade preserved key hash

Why this proves the behavior: The server-connected upgrade render read the existing Secret, and the stored key hash remained identical after applying that render.

@kfelternv
kfelternv marked this pull request as ready for review July 31, 2026 22:17

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a31f289da2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml
@github-actions

Copy link
Copy Markdown

🔐 TruffleHog Secret Scan

No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

🕐 Last updated: 2026-07-31 22:23:04 UTC | Commit: a31f289

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml (1)

8-13: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Add regression coverage for Secret reuse and upgrade failure.

This test checks only the generated value shape. It does not exercise the existing-Secret lookup branch or verify that an upgrade fails instead of generating a replacement key when the Secret is unavailable.

Add tests for both cases to protect the key-persistence contract.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml` around
lines 8 - 13, Add regression tests alongside the existing
temporal-encryption-key test to cover both Secret reuse when an existing Secret
is found and upgrade failure when that Secret is unavailable. Exercise the
chart’s existing-Secret lookup path, assert the reused key remains unchanged,
and verify an upgrade returns an error rather than generating a replacement key.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml`:
- Around line 5-14: The Temporal encryption key resolution in secrets.yaml must
generate a random key only during installation; on upgrades or rollbacks, fail
when neither the configured value nor the existing Secret key is available.
Preserve reuse of an existing Secret key, using Helm release state to
distinguish installs from upgrades. Update secrets_test.yaml to cover existing
Secret reuse and failure for an upgrade with a missing Secret.

In `@helm/rest/nico-rest/charts/nico-rest-common/values.yaml`:
- Around line 19-21: Update the comment above value in the Temporal payload
encryption configuration to identify value: "" as the chart default and document
resolution precedence: a configured value overrides Secret reuse and random
generation, while an empty value reuses the existing Secret on upgrade or
generates 32 random bytes on install; state that secrets.create: false disables
this resolution.

---

Nitpick comments:
In `@helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml`:
- Around line 8-13: Add regression tests alongside the existing
temporal-encryption-key test to cover both Secret reuse when an existing Secret
is found and upgrade failure when that Secret is unavailable. Exercise the
chart’s existing-Secret lookup path, assert the reused key remains unchanged,
and verify an upgrade returns an error rather than generating a replacement key.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fde99b05-9f56-4d13-a8fd-04948ca1a7c3

📥 Commits

Reviewing files that changed from the base of the PR and between 8b9e44f and a31f289.

📒 Files selected for processing (3)
  • helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml
  • helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml
  • helm/rest/nico-rest/charts/nico-rest-common/values.yaml

Comment thread helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml
Comment thread helm/rest/nico-rest/charts/nico-rest-common/values.yaml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml (1)

7-12: 🗄️ Data Integrity & Integration | 🔵 Trivial

Do not use this lookup path with client-side renderers.

Client-side renderers cannot read the existing Secret. On install, this generates a new key; on upgrade, it fails. Use server-side rendering with get access to the Secret, or use an externally managed Secret with secrets.create=false.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml` around
lines 7 - 12, The temporal-encryption-key lookup in the secrets template is
incompatible with client-side rendering. Update the secret-generation flow
around $existingSecret and $existingKey to use server-side rendering with get
access, or support an externally managed Secret when secrets.create=false, while
preserving reuse of the existing key and generation only for installs without
one.

Sources: Path instructions, MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml`:
- Around line 7-12: The temporal-encryption-key lookup in the secrets template
is incompatible with client-side rendering. Update the secret-generation flow
around $existingSecret and $existingKey to use server-side rendering with get
access, or support an externally managed Secret when secrets.create=false, while
preserving reuse of the existing key and generation only for installs without
one.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f2280dcb-4084-4e49-9d61-a7c55c7e49cc

📥 Commits

Reviewing files that changed from the base of the PR and between a31f289 and 577bb43.

📒 Files selected for processing (3)
  • helm/rest/nico-rest/charts/nico-rest-common/templates/secrets.yaml
  • helm/rest/nico-rest/charts/nico-rest-common/tests/secrets_test.yaml
  • helm/rest/nico-rest/charts/nico-rest-common/values.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • helm/rest/nico-rest/charts/nico-rest-common/values.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant