App-of-Apps: Preventing Cascade Sync Failures

[fetidbell]

App-of-Apps: Preventing Cascade Sync Failures

Question

Is there a way to prevent a single malformed manifest from blocking synchronization of all applications in an app-of-apps setup?

Context

We operate a multi-tenant GitOps platform using the app-of-apps pattern with directory.recurse: true:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
spec:
  source:
    repoURL: https://github.com/org/repo.git
    path: manifests
    directory:
      recurse: true

When one of our tenants introduces a malformed YAML (e.g., unquoted annotation value), the entire root Application fails to sync:

Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = FailedPrecondition desc = Failed to unmarshal "application.yaml": <nil>

This blocks all sibling Applications from syncing — one tenant's mistake impacts the entire platform.

What We've Tried

• CI validation — catches most issues but can't prevent all edge cases
• ApplicationSet — provides isolation but has trade-offs (rigid structure, uniform generation, debugging complexity)

Question

Are there any built-in mechanisms or recommended patterns in ArgoCD that allow app-of-apps to:

1. Skip unparseable files and continue syncing valid manifests?
2. Isolate failures to individual child resources?
3. Provide clearer attribution of which file caused the failure?

Or is switching to ApplicationSet the only way to achieve fault isolation?

Reproduction: https://github.com/fetidbell/app-of-apps-issue

Thanks!

[sunnypatell]

@fetidbell there's no built-in "skip invalid" option. the repo-server's manifest generation is fail-fast by design.

looking at the source, findManifests() calls getObjsFromYAMLOrJSON() for each file, and if any file contains apiVersion:, kind:, and metadata: but fails to parse, it returns a FailedPrecondition error immediately, aborting the entire directory source. there's no recovery path.

your best option: ApplicationSet with the git directories generator

this gives you per-tenant fault isolation because each directory becomes its own Application resource. if tenants/teamA/ has broken YAML, only tenant-teamA errors while everything else syncs fine:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenants
spec:
  goTemplate: true
  generators:
  - git:
      repoURL: https://github.com/org/repo.git
      revision: HEAD
      directories:
      - path: manifests/*
  template:
    metadata:
      name: 'tenant-{{.path.basename}}'
    spec:
      project: '{{.path.basename}}'
      source:
        repoURL: https://github.com/org/repo.git
        targetRevision: HEAD
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{.path.basename}}'

re: your concerns about ApplicationSet:

• "rigid structure" is actually a feature for multi-tenancy since it enforces consistent layout
• "debugging complexity" is mitigated by each tenant getting their own Application resource with isolated sync status and error messages
• you can add exclude patterns and use values for per-directory customization

for error attribution (your question 3): even with the current app-of-apps setup, #3295 and #15521 have been open since 2019 requesting filepath info in errors. still unresolved.

layered defense: pair the ApplicationSet migration with kubeconform or yamllint in CI as a pre-merge gate. argocd app diff --local can also dry-run manifest generation before merge.

ref: ApplicationSet git generator docs | directory source docs