Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/infrastructure-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ permissions:
jobs:
test-infrastructure:
runs-on: staging

steps:
- name: Checkout code
uses: actions/checkout@v7
Expand Down
49 changes: 31 additions & 18 deletions helm/fuzeinfra/values-contabo.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -178,44 +178,57 @@ overprovisioning:
# Secret the consumer repo seals FOR the `fuzeinfra` namespace. Helm replaces
# (does NOT deep-merge) lists, so the full list is restated here.
#
# GITOPS GATE: each service stays `enabled: false` until its SealedSecret is
# present in the `fuzeinfra` namespace. Flipping `enabled: true` before the
# secret exists makes the hook Job fail to start (unresolvable secretKeyRef),
# degrading the Argo sync. Flip each to `true` in the SAME change that lands its
# secret. As of this PR none of the three secrets exist yet (verified via
# read-only kubectl) — cross-repo hand-off issues track sealing them into
# `fuzeinfra`:
# - fuzesales -> Secret `fuzesales-db-credentials` key `password`
# - fuzecontact -> Secret `fuzecontact-db-credentials` key `password`
# - fuzeservice -> Secret `fuzeservice-db-credentials` key `password`
# - mendys -> Secret `mendys-db-credentials` key `password`
# (SealedSecret must be committed to izzywdev/MendysRobotics
# sealed for namespace=fuzeinfra before enabling here)
# GITOPS GATE: each service stays `enabled: false` until its per-service
# credentials Secret is present in the `fuzeinfra` namespace. This is a HARD
# gate: the provisioning hook is a SINGLE Job pod that mounts every enabled
# service's password via `secretKeyRef` (Optional: false), so ONE missing
# secret fails the whole pod with CreateContainerConfigError, blocks ALL
# entries (including ones whose secret DOES exist), and degrades the Argo
# PostSync hook. (Observed: the Job sat in CreateContainerConfigError for
# ~3.7 days because fuzesales/fuzecontact/fuzeservice were flipped on before
# their secrets landed.) Flip each to `true` in the SAME change that lands its
# secret — never before.
# - mendys -> Secret `mendys-db-credentials` key `password`
# (DONE, issue #183: applied as a hand-off Secret in the
# `fuzeinfra` namespace via the FuzeKeys #136 precedent, so
# `mendys` below is enabled)
# - fuzesales -> Secret `fuzesales-db-credentials` key `password` (pending seal)
# - fuzecontact -> Secret `fuzecontact-db-credentials` key `password` (pending seal)
# - fuzeservice -> Secret `fuzeservice-db-credentials` key `password` (pending seal)
serviceDatabases:
# Disabled: `fuzesales-db-credentials` is NOT yet present in `fuzeinfra`
# (verified via kubectl); leaving it enabled wedges the shared provisioning
# Job for every service. Re-enable in the change that seals its secret.
- name: fuzesales
enabled: true
enabled: false

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude why disable those services? they are already now on fuzeinfra

@claude claude Bot Jul 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude finished @izzywdev's task in 1m 28s —— View job


Why fuzesales / fuzecontact / fuzeservice are disabled

Short version: their credentials secrets aren't on the cluster yet, and the provisioning hook is a single shared pod — so leaving them enabled: true wedges provisioning for every service, including mendys. I re-verified against prod just now:

1. The three secrets do not exist in fuzeinfra (kubectl -n fuzeinfra get secrets):

fuzeinfra-secrets
fuzeinfra-tunnel-secrets
fuzekeys-db-credentials
fuzesocial-app-db-credentials
mendys-db-credentials          ← the only consumer secret added recently

No fuzesales-db-credentials, fuzecontact-db-credentials, or fuzeservice-db-credentials.

2. Their Postgres roles don't exist either (SELECT rolname ... LIKE '%_svc' on fuzeinfra-postgres-0):

billing_svc
mendys_svc

No fuzesales_svc / fuzecontact_svc / fuzeservice_svc, and no fuzesales/fuzecontact/fuzeservice databases.

3. The live provisioning Job is currently wedged for exactly this reason. Because main still has those three enabled without their secrets, the hook pod has been in CreateContainerConfigError for ~4 days:

fuzeinfra-service-db-provision-sf64q   0/1   CreateContainerConfigError   (x26410 over 3d23h)
Warning  Failed  kubelet  Error: secret "fuzesales-db-credentials" not found

That single pod mounts every enabled service's password via a non-optional secretKeyRef, so one missing secret fails the whole pod → nothing gets provisioned, mendys included. That's the whole reason this PR's mendys enablement wouldn't actually take effect until those three are turned back off.

So "they're already on fuzeinfra"?

The apps may well be deployed, but their per-service DB credentials haven't been sealed into the fuzeinfra namespace, and their roles/dbs were never created. The enabled flag here doesn't track "is the app running" — it tracks "has this service's password Secret landed in fuzeinfra so the shared Job can provision its role." For these three, it hasn't. Flipping them on before the secret exists is what caused the 4-day wedge in the first place, and it's what the file's own GITOPS GATE comment warns against.

To enable them (the right way)

Same path we used for mendys (FuzeKeys #136 precedent): land each service's *-db-credentials Secret in fuzeinfra first, then flip enabled: true in that same PR. If you want, I can apply the hand-off Secrets for the three now (given their passwords) and re-enable all three in this PR — that would provision fuzesales_svc/fuzecontact_svc/fuzeservice_svc too. Just say the word and provide/point me at the passwords (or confirm you want them generated + sealed).

Leaving them disabled here is deliberate and non-destructive: it unblocks mendys without regressing anything (those roles/dbs don't exist to lose).

· claude/issue-183-20260707-1704

role: fuzesales_svc
database: fuzesales
passwordSecret:
name: fuzesales-db-credentials
key: password
# Disabled: `fuzecontact-db-credentials` not yet present in `fuzeinfra`.
- name: fuzecontact
enabled: true
enabled: false
role: fuzecontact_svc
database: fuzecontact
passwordSecret:
name: fuzecontact-db-credentials
key: password
# Disabled: `fuzeservice-db-credentials` not yet present in `fuzeinfra`.
- name: fuzeservice
enabled: true
enabled: false
role: fuzeservice_svc
database: fuzeservice
passwordSecret:
name: fuzeservice-db-credentials
key: password
# Enable once izzywdev/MendysRobotics seals mendys-db-credentials for fuzeinfra.
# Enabled: `mendys-db-credentials` (key `password`) is present in the
# `fuzeinfra` namespace as a hand-off Secret (issue #183, FuzeKeys #136
# precedent — annotated `fuzeinfra.dev/consumer: izzywdev/MendysRobotics`).
# The role/db were also provisioned imperatively with the same password, so
# this hook Job idempotently adopts and re-syncs them on the next Argo sync.
- name: mendys
enabled: false
enabled: true
role: mendys_svc
database: mendys
passwordSecret:
Expand Down
8 changes: 8 additions & 0 deletions runners/arc/runner-scale-set-values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,14 @@ template:
operator: Exists
effect: NoSchedule

# ---- Docker-in-Docker -------------------------------------------------------
# Enables the ARC-native DinD sidecar so runner pods have a real Docker daemon.
# The chart injects a `dind` sidecar container and sets DOCKER_HOST automatically;
# workflows that call docker/docker-compose work without any extra setup.
# The runner container needs to be privileged to run DinD.
containerMode:
type: dind

# ---- ARC controller reference -----------------------------------------------
controllerServiceAccount:
namespace: arc-systems
Expand Down
Loading