Intermittent 504 Gateway Timeout During Application Refresh – application refresh deadline exceeded (v3.5.3)

[3LOG1]

Description

Hi Argo CD Team,

We are experiencing intermittent 504 Gateway Timeout errors when running Argo CD application operations through our CI/CD pipelines.

{"level":"fatal","msg":"rpc error: code = Unknown desc = POST https://**********:443/argo-cd/application.ApplicationService/Get failed with status code 504","time":"2026-10-06T13:58:20Z"}

During our investigation, we observed that the Argo CD application-server request fails with application refresh deadline exceeded after approximately 60 seconds.
We have checked the argocd-server pod logs during the time as below:

"level":"info","msg":"received unary call /application.ApplicationService/Get","time":"2026-10-01T14:37:32Z"} {"level":"info","msg":"Requested app 'demo-app' refresh","time":"2026-10-01T14:37:33Z"}

After the deadline exceeded : "grpc.code":"Unknown","grpc.component":"server","grpc.error":"application refresh deadline exceeded","grpc.method":"Get","grpc.method_type":"unary","grpc.service":"application.ApplicationService","grpc.start_time":"2026-10-01T14:37:32Z","grpc.time_ms":"59999.465","level":"error","msg":"finished

While checking the argocd-repo and controller logs we also noticed multiple applications undergoing refresh and reconciliation operations around the same period.

We would like to understand whether concurrent application reconciliation workload or application-controller processing delays could be contributing to these failures, and whether increasing the application-controller replica count would help.

As a workaround, we increased the replica count to 2 for both argocd-application-controller and argocd-repo-server.

Sometimes rerunning the pipeline resolves the issue, but in other cases, even the second or third retry does not help.
Since this issue occurs intermittently, has anyone experienced similar behavior? We would appreciate your suggestions and guidance on identifying the root cause and finding a permanent solution.

Environment

Argo CD version: v3.5.3

Platform: Kubernetes

Application-controller replicas: 2

argocd-repo-server replicas : 2

Application-controller CPU usage (observed): ~969m

Application-controller memory usage (observed): ~4922Mi

Controller resource requests/limits: Not explicitly configured (resources: {})

Controller status/operation processors: Not explicitly configured (using defaults)

Reconciliation timeout/jitter: Not explicitly configured (using defaults)

Application management: Multiple applications managed through Argo CD, including applications with automated sync and self-heal enabled

Error observed

The pipeline encountered an HTTP 504 error while calling the Argo CD Application API:

rpc error: code = Unknown desc =
POST https:///argo-cd/application.ApplicationService/Get
failed with status code 504

From the argocd-server logs:

14:37:32 - Received Get request for application demo-app
14:37:33 - Requested app 'demo-app' refresh

14:38:32 - grpc.code=Unknown
grpc.error="application refresh deadline exceeded"
grpc.time_ms=59999.465

The application refresh request took approximately 60 seconds before failing.

Current structure of the application running in the platform.

Additional information

We can provide further sanitized controller/server logs, configuration details, and relevant metrics if needed.

Thanks in advance for your guidance!

[gowchin]

The ~60s is the clue. Two things are combining here:

1. What Get with refresh actually does

When the pipeline calls argocd app get <app> --refresh (or the API with ?refresh=normal), argocd-server does not refresh anything itself. In server/application/application.go it sets the argocd.argoproj.io/refresh annotation on the Application and then blocks until the application-controller picks the app up and removes that annotation. If the request context is cancelled before that happens, you get exactly application refresh deadline exceeded.

There is no server-side timeout on that loop — the context is cancelled because the client side gave up. grpc.time_ms: 59999 + a 504 on /argo-cd/application.ApplicationService/Get means your ingress / load balancer in front of argocd-server is closing the request at 60s (nginx proxy-read-timeout and the AWS ALB idle timeout both default to 60s).

2. Why the controller sometimes takes >60s to pick it up

The refresh request goes into the controller's app refresh queue behind everything else being reconciled. With many auto-sync/self-heal apps, default processors and no resource requests (your controller is already at ~1 CPU / ~5 GiB), that queue can back up — which explains why retries sometimes don't help.

What I'd do

• Raise the proxy timeout for the Argo CD API path, e.g. for ingress-nginx:

  nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
  nginx.ingress.kubernetes.io/proxy-send-timeout: "300"

  (or idle_timeout.timeout_seconds=300 on an ALB).
• In CI, avoid blocking on get --refresh. argocd app sync <app> already fetches latest manifests, then use argocd app wait <app> --sync --health --timeout 600 so the long wait isn't a single unary call.
• Give the controller real resources and more workers (argocd-cmd-params-cm):

  controller.status.processors: "50"
  controller.operation.processors: "25"

  and set resources.requests so it isn't CPU-starved.
• Double-check sharding actually took effect with 2 controller replicas: ARGOCD_CONTROLLER_REPLICAS must match the replica count (the Helm chart sets it when you use controller.replicas). Otherwise the second replica doesn't help.
• If apps are refreshed mainly by polling, configure Git webhooks and/or raise timeout.reconciliation (with timeout.reconciliation.jitter) in argocd-cm to reduce the background load.

To confirm the cause, watch workqueue_depth{name="app_reconciliation_queue"} and argocd_app_reconcile on the controller, plus argocd_repo_pending_request_total on the repo-server, around the time of a failure. If the queue depth spikes when the 504s happen, it's controller backlog; if the repo-server pending requests climb, manifest generation is the bottleneck (raise reposerver.parallelism.limit / repo-server replicas).

[3LOG1]

Hi @gowchin

Thanks for the detailed explanation. One additional detail I missed in the original post is that our pipeline is not explicitly running argocd app get --refresh.

The pipeline first runs:

Shell

argocd app sync --auth-token ${ARGOCD_ACCESS_TOKEN} --server ${ARGOCD_SERVER} --grpc-web-root-path /argo-cd --prune ${CI_ENVIRONMENT_NAME}-${APP_NAME}

and then:

argocd app wait --auth-token ${ARGOCD_ACCESS_TOKEN} --server ${ARGOCD_SERVER} --grpc-web-root-path /argo-cd --timeout 300 ${CI_ENVIRONMENT_NAME}-${APP_NAME}

For applications using Argo Rollouts, we run argocd app wait with --sync --operation.

The resources are shown as Synced/Healthy, and afterwards we receive:

rpc error: code = Unknown desc = POST .../argo-cd/application.ApplicationService/Get failed with status code 504

Could argocd app wait itself be triggering the ApplicationService/Get request that is hitting the ~60-second timeout?

If so, would you still recommend increasing the ingress/ALB timeout to 300 seconds in this case?

[jmrplens]

Yes, it can, and so can the argocd app sync before it. Both commands end by fetching the app with refresh=normal whenever they saw a sync operation in progress while waiting, and that is the ApplicationService/Get that hits your 60 s limit.

In v3.5.3, waitOnApplicationStatus sets refresh = true as soon as a watch event shows app.Operation (app.go#L2507-L2508). Before printing the final status it calls Get with that refresh (app.go#L2378-L2386). argocd app sync uses the same function with watchOpts{operation: true} unless you pass --async (app.go#L2041). On the server that Get is the wait gowchin described. When your proxy cuts it at 60 s, the server returns application refresh deadline exceeded (application.go#L880-L881), which is the line in your log.

I checked it with Argo CD v3.5.3 on k3s, with the CLI going through an nginx set to proxy_read_timeout 60s:

Run	What argocd-server logged
app sync --prune, then app wait --timeout 300	Requested app 'guestbook' refresh at the end of app sync (that Get took 436 ms); nothing at the end of app wait, because the operation had already finished
app sync --async, then app wait --timeout 300 or app wait --sync --operation	Requested app 'guestbook' refresh at the end of app wait
controller scaled to 0, then a Get with refresh	after 60 s: the CLI prints POST .../application.ApplicationService/Get failed with status code 504, the server logs application refresh deadline exceeded with grpc.time_ms 60063, and nginx logs upstream timed out

So with your pipeline the 504 most likely comes at the end of argocd app sync. app wait only does the same when it sees an operation still in progress: after app sync --async, or when auto-sync or self-heal starts one while it waits. Adding --sync --operation doesn't change that. You can tell from the CI log: app sync runs that Get before printing its Name: / Operation: / Phase: summary, so if the summary is there, the sync's Get went through. Either way it's the same call, so yes, raise the ingress read timeout (the idle timeout on an ALB). 300 s is enough: app wait --timeout 300 gives up by then anyway, and app sync has no limit of its own unless you pass --timeout. How often the refresh takes that long comes down to the controller backlog gowchin described.