Recommended patterns for a deployment pipeline on top of ArgoCD that needs to wait for an App to be "deployed succesfully"

[gaeljw]

Hello folks,

Opening a discussion to get feedbacks or informations regarding the recommended pattern(s) for a deployment pipeline that sits on top of ArgoCD and needs to know an App is synced + healthy for a given commit.

That's a follow-up of a discussion I started on Reddit some time ago. I'm copying the relevant pieces here.

---

Original question:

Context: you have some kind of CD pipeline that updates manifests in Git repo for the staging environment then production.

Question: how to prevent starting the deployment in production if there's an error in staging?

By the nature of ArgoCD, it works asynchronously, even if using webhooks, there may be a few seconds before a manifests is updated in Git and the ArgoCD apps starts its rollout. Moreover the App can appear Healthy for a very short period of time after rollout before switching to Degraded mode for whatever reason.

How to detect that staging App is degraded before deploying the prod one?

Ideas:

• use ArgoCD CLI to kinda bypass the async nature (argocd app sync + wait). I don't really like this and I'm not even sure it would work.

• add arbitrary delay (at least a few minutes) between env promotion and do not start Prod if staging is not Healthy. Sounds reasonable, adds unnecessary delay when everything works just fine though.

• ???

In a similar context, how do you handle this?

Discussion with @todaywasawesome:

D:

Remember that Argo CD does not handle environment promotion. There are lots of tools out there to take this on and they have strenghts and weaknesses. I did a full talk on this at ArgoCon London - https://www.youtube.com/watch?v=iCTgRC3AQQk&list=PLj6h78yzYM2N9MWCsU_4upn64NDtHGv6i&index=13 this gives an overview of all the different solutions. (...)

G:

Great vid! I'm still a bit like Victor at the end: how to workaround the async/flapiness nature of the ArgoCD health status. There's obviously no definite answer.

D:

Something I think we need to build into Argo CD is to attach commits to statuses. So you can verify if your commit was deployed and if that commit reached healthy for some period of time, or through some round of updates.

We built this logic in Codefresh and it's now coming into Octoups Deploy but it's actually a lot more complex than it would seem because Argo CD doesn't yet do this. Might be something we contribute this year.

G:

I was thinking about this. That'd be really useful indeed.

Should I open an issue on GitHub to track this and see if how much/many people would find this useful?

D:

There's discussion about it in this issue and a proposal that I haven't reviewed indepth yet.

• [ApplicationSet/ProgressiveSync] ProgressiveSync syncing out of order #22558
• docs(proposal): handling progressive sync's syncing out of order due to cache delays. #24757

---

The referenced GH issues/PR are more about the "Progressive Sync" feature whick makes sense in a "environment promotion" context. However I'm not using these and I don't think they are the answer.

Let's forget about environment promotion, just think of a single App in a single environment, how would you go for checking that a commit you just pushed to a GitOps repo has been taken into account by ArgoCD and the App is actually Synced & Healthy for "long enough"?

We have a bunch of code on top the ArgoCD API to do that by waiting an arbitrary amount of time + checking the App status + checking some resources statuses (typically Deployments) but it feels heavy and not the right way.

A concrete scenario we hit sometimes and made us implement this "glue code" on top of ArgoCD API is that an App goes Synced and Healthy for a few seconds but the pods of Deployment are rolled out progressively and fail. Depending on what and when we look at in the ArgoCD API, we may not be able to detect this failure.

Obviously a part of the answer is to detect such failures "after" rather than during the deployment which is by nature async in a GitOps/ArgoCD context but I'm looking for way to catch most of these failures sooner: typically when a new version doesn't start at all.

[XNinety9]

For the single-App case, most of the "Healthy for a few seconds, then fails" window can be closed on the Kubernetes side rather than with glue code, because Argo CD's Deployment health just mirrors the rollout status (health_deployment.go, borrowed from kubectl rollout status):

• It stays Progressing while availableReplicas < updatedReplicas, and goes Degraded only when the Progressing condition has reason ProgressDeadlineExceeded.
• So set minReadySeconds on the Deployment (e.g. 60-120). A new pod only counts as available after being Ready that long without restarting, so a pod that starts, passes readiness once and crashes 20 s later keeps the App Progressing instead of flashing Healthy.
• Set a realistic progressDeadlineSeconds. A rollout that never gets enough available pods then turns into a real Degraded state you can wait on, instead of staying Progressing forever.
• Make sure the readiness probe actually exercises the app; with a trivial probe, "available" means very little.

Then the pipeline step becomes:

argocd app get "$APP" --refresh >/dev/null           # don't wait for polling/webhook
# poll until Argo has picked up your commit
until [ "$(argocd app get "$APP" -o json | jq -r .status.sync.revision)" = "$SHA" ]; do sleep 5; done
argocd app wait "$APP" --sync --health --timeout 900  # fails as soon as a resource turns Degraded, or at timeout

(.status.sync.revisions for multi-source apps.) Combined with minReadySeconds, "Synced + Healthy at $SHA" really means every new pod stayed up for that long.

Failures that show up minutes later (memory leak, a bad dependency) will still get past this; that's where metric-based analysis comes in (Argo Rollouts AnalysisTemplate, or Kargo verification if you go the promotion-tool route).

(I used an AI assistant to go through the health-check code; I checked the logic and links myself.)

[gaeljw]

I missed that minReadySeconds could be really useful in these scenario indeed. I'll give it a try, thanks!

To be fair, we use ArgoCD API rather than CLI and a part of our glue code is reimplementing argocd app wait. We may reconsider using the CLI to reduce our own code footprint.

Failures that show up minutes later (memory leak, a bad dependency) will still get past this (...)

Sure. My scope is only pods that "immediately" fail (probes are never OK).

[XNinety9]

Glad I could help! Dont' forget to mark as answer for future reference

[gaeljw]

Okay, so I reviewed this a bit more.

As expected, minReadySeconds is not helpful in the scenario of a pod immediately failing (the pods is never ready), it could help for scenarios where the pod may be ready for a short duration before going not ready though.

The loop to wait for status.sync.revision = $SHA feels a bit fragile in a context where there can be several commits arriving. If a second commit arrives and supersedes the 1st one, you'd wait indefinitely whereas we should consider it ok. Ideally we'd have status.sync.revision >= $SHA where >= is interpreted in a Git history meaning. Also I'm not sure if status.sync.revision is updated when a commit that does not affect the App is pushed to the repo? If it is, it's even worse. For that reason, our glue code look at the datetimes instead.

argocd app wait works fine as long as the loop to wait for a Sync to be "started" correctly waits.

[XNinety9]

Fair point on minReadySeconds: for a pod that never becomes Ready it adds nothing. In that case the Deployment never counts as available, so the App stays Progressing until progressDeadlineSeconds turns it Degraded, and argocd app wait --health fails there. The only way to get a false "Healthy" is reading the old state before the sync starts, so the revision gate is what matters.

On that gate, you're right on both counts:

• status.sync.revision does move on commits that don't touch the App. On each refresh it's set to the resolved SHA of targetRevision (state.go). With manifest-generate-paths, UpdateRevisionForPaths only skips regenerating the manifests; the revision still moves to the new commit.
• So strict equality can hang when newer commits land.

Your ">= in Git history" works directly with git merge-base --is-ancestor, since the pipeline has the repo anyway:

until rev=$(argocd app get "$APP" --refresh -o json | jq -r .status.sync.revision) &&
      git fetch -q origin "$rev" && git merge-base --is-ancestor "$SHA" "$rev"; do
  sleep 5
done
argocd app wait "$APP" --sync --health --timeout 900

Once Argo has compared at $SHA or any descendant, its desired state includes your change. "Synced" is then checked against that revision. Health can't flash Healthy before the rollout really starts, because the Deployment check returns Progressing while observedGeneration < generation. A superseding commit just ends the loop sooner. A commit that doesn't touch the App ends it too, which is fine because it contains yours. (.status.sync.revisions[i] for multi-source apps; git fetch by SHA needs a server that allows it, which GitHub and GitLab do.)

The timestamp approach also works. The ancestor check just avoids clock and polling-interval assumptions.

(Same as before: I used an AI assistant to go through the controller code, and checked the logic and links myself.)

[gaeljw]

I think this is quite nice indeed.

One last thing I'd need but I don't expect ArgoCD to provide it is to wait for a fixed number of retries in case of a Deployment failure rather than a fixed duration. The reason is that in some limit cases, some containers can take a significant long time to start compared to others. And I'd have to use a large timeout value to cope with these pods whereas for regular pods I could use a shorter timeout.

I could extract the ArgoCD wait timeout to be n times the probe timeout though to keep it generic.

Thanks for the suggestions anyways :)

[keemgdeok]

For your last point, probe failures and container restarts are different counters. A failed readiness probe keeps the container unready; it does not restart it. A startupProbe lets each container have its own startup allowance, roughly failureThreshold * periodSeconds, before Kubernetes restarts it. That is useful for the slow-starting workloads, but the threshold applies to one startup attempt, not to N container restarts across the rollout.

Argo CD's Deployment health check uses rollout progress and availability, not a restart-count budget. I would configure startup allowances per workload, make progressDeadlineSeconds compatible with those allowances, and keep argocd app wait --timeout as the pipeline's outer limit. Multiplying N by the probe timeout alone misses probe intervals and restart backoff. If failing after exactly N restarts is a hard requirement, that needs an additional policy outside the built-in Deployment health check. Check the settings against both a slow successful startup and a container that never becomes ready.

AI-generated; checked against the linked documentation and health-check code, without a live rollout reproduction.

[DYNOSuprovo]

This is one of the most critical challenges when orchestrating multi-environment promotions (Staging -> Production) on top of an asynchronous GitOps engine like Argo CD.

The core difficulty stems from three distinct timing gaps:

1. The Git-to-Argo Polling Delay: Argo CD has not yet detected the new commit, so a naive health check immediately returns Healthy based on the previous deployment.
2. The "Premature Healthy" Kubernetes Window: A new Pod passes its readiness probe once, Kubernetes marks the Deployment healthy, but the process crash-loops 10 seconds later under real load.
3. The Multi-Stage Gate: Ensuring staging stays healthy before advancing production.

Here are the industry-standard patterns to solve this reliably:

---

1. Close the "Premature Healthy" Window at the Kubernetes Layer (minReadySeconds)

Before adding external glue code, configure your Deployment specs with minReadySeconds:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  minReadySeconds: 30  # Keeps the rollout in progress until pods are healthy for at least 30s
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: app
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 3

Why this matters:
Without minReadySeconds, Kubernetes considers a Pod ready the instant its readiness probe succeeds once. Argo CD's health evaluator (health_deployment.go) mirrors Kubernetes Deployment status (Progressing: Complete). With minReadySeconds: 30, Kubernetes refuses to mark the replica set rollout complete until the pods have stayed healthy continuously for 30 seconds, preventing transient boot successes from prematurely flipping Argo CD to Healthy.

---

2. The Deterministic CLI Sync & Wait Pattern (Targeting Git Revision)

If your CD pipeline runs a script (e.g. GitHub Actions, GitLab CI) waiting for staging to succeed before triggering prod, you must bind the wait command to the exact target Git commit SHA, not just generic health:

TARGET_SHA=$(git rev-parse HEAD)

# 1. Force Argo CD to refresh immediately against Git (bypassing 3-minute poll interval)
argocd app get staging-app --refresh

# 2. Trigger sync (if not automated)
argocd app sync staging-app --revision "${TARGET_SHA}"

# 3. Wait for BOTH Sync and Health matching the target revision
argocd app wait staging-app \
  --revision "${TARGET_SHA}" \
  --health \
  --sync \
  --operation \
  --timeout 600

# 4. Defensive post-check: verify status didn't regress
STATUS=$(argocd app get staging-app -o json | jq -r '.status.health.status')
if [ "$STATUS" != "Healthy" ]; then
  echo "Staging deployment degraded! Halting production rollout."
  exit 1
fi

Binding --revision "${TARGET_SHA}" ensures the pipeline never passes based on the health of the previous commit while the new rollout is still queuing.

---

3. Use PostSync Hooks for Automated Smoke / E2E Verification

Rather than relying purely on HTTP readiness probes, attach an Argo CD PostSync Job that runs integration smoke tests against staging:

apiVersion: batch/v1
kind: Job
metadata:
  name: staging-smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: smoke-test
          image: my-registry/smoke-tests:v1
          command: ["npm", "run", "test:smoke"]
          env:
            - name: APP_URL
              value: "http://staging-service.internal"
      restartPolicy: Never

Argo CD holds the entire application sync operation in a Progressing state until the PostSync hook Job completes with exit code 0. If the smoke test fails, Argo CD marks the sync operation as Failed, immediately stopping your CD pipeline.

---

4. The Decoupled GitOps Approach: Argo Rollouts with AnalysisTemplates

For zero-glue-code promotion, pair Argo CD with Argo Rollouts:

• Define an AnalysisTemplate that queries your telemetry provider (Prometheus, Datadog) for error rates ($5xx > 1%$) and latency over a 5-minute window.
• The rollout automatically pauses or aborts before promoting.
• In multi-cluster setups, tools like Kargo or GitOps PR-promotion bots watch for the green status check on staging before submitting the automated pull request to update the production manifest repository.