Having Trouble Figuring out layout/design for Initial Cluster Bootstrapping

[jseiser]

I have alot of experience managing Flux cluster, but we have a new environment where ArgoCD was the request.

Im having a ton of trouble finding.. clear guidance on a normal repo structure. This is currently once cluster, but will grow into more. A Cluster may be qa and prod or just one or the other. This repo is strictly for infrastructure resources on these clusters, end-user deployments would come form a different repo.

The main issue im having is around, how to handle dissimilar parts of an Application

In Flux, I would define kustomize that would

1. Install the Cert Manager CRD's
2. Install the Helm Chart using the HelmController
3. Create all the Cluster issuers etc.

If I needed to make one cluster have 2 more Cluster issuers than another, its just adding a overlay for that cluster and moving on. Im having issues wrapping my head around the expected ArgoCD way of doing this.

I currently have this

Layout

.
├── appsets
│   ├── 00-cluster-crds.yaml
│   └── 01-cluster-apps.yaml
├── cluster-apps
│   ├── cert-manager
│   │   ├── appset.yaml
│   │   ├── base
│   │   │   └── values.yaml
│   │   └── clusters
│   │       └── narmer-demo-alpha
│   │           └── values.yaml
│   ├── external-secrets
│   │   ├── appset.yaml
│   │   ├── base
│   │   │   ├── kustomization.yaml
│   │   │   ├── ssm-secretstore.yaml
│   │   │   └── values.yaml
│   │   └── clusters
│   │       └── narmer-demo-alpha
│   │           ├── kustomization.yaml
│   │           └── values.yaml
│   ├── karpenter
│   │   ├── appset.yaml
│   │   ├── base
│   │   │   ├── kustomization.yaml
│   │   │   ├── nodeclass.yaml
│   │   │   └── nodepool.yaml
│   │   └── clusters
│   │       └── narmer-demo-alpha
│   │           ├── kustomization.yaml
│   │           └── nodeclass.yaml
│   └── trust-manager
│       ├── appset.yaml
│       ├── base
│       │   └── values.yaml
│       └── clusters
│           └── narmer-demo-alpha
│               └── values.yaml
├── README.md
└── root-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-services-root
  namespace: argocd
spec:
  project: platform-services
  source:
    repoURL: https://gitab.something.com/something/the_platform/platform_services.git
    targetRevision: develop
    path: appsets
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Which picks up an app set

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: 01-cluster-apps
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://gitab.something.com/something/the_platform/platform_services.git
        revision: develop
        directories:
          - path: cluster-apps/*

  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: default
      source:
        repoURL: https://gitab.something.com/something/the_platform/platform_services.git
        targetRevision: develop
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Which loops through application sets

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cert-manager-cluster
  namespace: argocd
spec:
  goTemplate: true

  generators:
    - clusters:
        selector:
          matchLabels:
            workload: "platform"
  template:
    metadata:
      name: "{{.name}}-cert-manager"

    spec:
      project: platform-services

      sources:
        - repoURL: https://charts.jetstack.io
          chart: cert-manager
          targetRevision: 1.20.0
          helm:
            valueFiles:
              - $values/cluster-apps/cert-manager/base/values.yaml
              - $values/cluster-apps/cert-manager/clusters/{{.name}}/values.yaml

        - repoURL: https://gitab.something.com/something/the_platform/platform_services.git
          targetRevision: develop
          ref: values

        - repoURL: https://gitab.something.com/something/the_platform/platform_services.git
          targetRevision: develop
          path: cluster-apps/cert-manager/clusters/{{.name}}

      destination:
        server: '{{.server}}'
        namespace: cert-manager
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
        automated:
          prune: true
          selfHeal: true

This looks incredibly messy in the UI. With apps that deploys apps that deploys Applications. I also read you should not have multiple sources defined.. So can anyone please offer some guidance to get back on track?

im much more used to just definine kustomizations, and being able to reference raw manifests and helm charts without this additional abstraction so I know im definitely doing something wrong.

[mg1986jp]

Coming from Flux, the main mental shift is: Flux is "define what to deploy inline with kustomizations" while ArgoCD is "point Applications at paths/charts." You don't need the nested ApplicationSet-spawning-ApplicationSet pattern — that's the source of your UI mess.

Here's a cleaner layout that gets you the same per-cluster overlay capability:

.
├── apps/
│   ├── cert-manager/
│   │   ├── base/
│   │   │   ├── kustomization.yaml      # references the Helm chart + raw manifests
│   │   │   ├── values.yaml
│   │   │   └── cluster-issuers.yaml    # your ClusterIssuer manifests
│   │   └── overlays/
│   │       └── narmer-demo-alpha/
│   │           ├── kustomization.yaml   # patches base, adds extra issuers
│   │           └── extra-issuer.yaml
│   ├── external-secrets/
│   │   ├── base/
│   │   └── overlays/
│   │       └── narmer-demo-alpha/
│   └── karpenter/
│       ├── base/
│       └── overlays/
│           └── narmer-demo-alpha/
└── appset.yaml    # ONE ApplicationSet, that's it

The single ApplicationSet uses a matrix generator — cluster selector × git directory:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-services
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  workload: "platform"
          - git:
              repoURL: https://gitlab.something.com/something/platform_services.git
              revision: develop
              directories:
                - path: apps/*
  template:
    metadata:
      name: "{{.name}}-{{.path.basename}}"
    spec:
      project: platform-services
      source:
        repoURL: https://gitlab.something.com/something/platform_services.git
        targetRevision: develop
        path: "apps/{{.path.basename}}/overlays/{{.name}}"
      destination:
        server: "{{.server}}"
        namespace: "{{.path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true

For Helm charts with kustomize overlays, use the kustomization.yaml in each overlay to pull the Helm chart via the helmCharts field (Kustomize 5.0+):

# apps/cert-manager/base/kustomization.yaml
helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io
    version: 1.20.0
    releaseName: cert-manager
    namespace: cert-manager
    valuesFile: values.yaml
resources:
  - cluster-issuers.yaml

# apps/cert-manager/overlays/narmer-demo-alpha/kustomization.yaml
resources:
  - ../../base
  - extra-issuer.yaml
patches:
  - path: values-patch.yaml  # if you need cluster-specific helm values

This gives you:

• One ApplicationSet instead of 3 layers of apps-deploying-apps
• Clean UI — one Application per app per cluster
• Same overlay model you know from Flux — base + per-cluster patches
• No multi-source needed — single source with kustomize handles both Helm and raw manifests

For the "multiple sources" thing you mentioned — it's fine to use multi-source, it's just not needed here since kustomize can render Helm charts natively now.

[fedeam]

I only run a single cluster, so I can't speak to the per-cluster overlay part. For the "install the chart, then create the resources that depend on it" part, what works for me with plain Applications (no ApplicationSets, no sync waves) is one Application per piece:

root (app-of-apps)
├── longhorn         -> Helm chart (also creates the namespace)
└── longhorn-config  -> a plain directory in the repo with the custom
                        resources that depend on the chart's CRDs

The same applies to cert-manager: its issuers live in their own directory and have their own Application.

Each Application has its own sync policy, which turned out to be the useful part. For the config Application I started with automated but no prune and no selfHeal, so Argo CD could adopt resources that already existed in the cluster without touching or deleting them. The plan is to enable selfHeal once kubectl diff is clean.

The chart is Helm-rendered by Argo CD, and the config is a directory. That avoids having to bridge Helm and Kustomize in the same Application.

If you need a different set of issuers per cluster, the config Application would be the one to vary (a separate path or a Kustomize overlay per cluster), while the chart Application stays identical. I haven't done the multi-cluster part, so take that as a design suggestion rather than something I've run.