Syncs failing. Is there a limit to number of resources in the application?

[lstyles]

I'm at a bit of a loss as we're experiencing some syncing issues in (fortunately) our non-production cluster.

We have 3 applications with ~40 microservices (deployment + service) and few ingress resources. The total of resources (I'm assuming) watched by ArgoCD is ~450 per application. (Counted by applying filter and seeing how many hidden resources it thinks it has).

I am able to manually Sync by selecting Apply Out of Sync Only. Automated sync always ends up in application controller restarting.
I've increased the repo server timeout to 180s and resources on the controller pod.

argocd-application-controller
--status-processors
20
--operation-processors
5
--repo-server-timeout-seconds
180
--loglevel
debug

but I'm not even seeing a lot of activity on the repo server.
We're now deploying any helm charts or kustomize manifests. Just plain YAML resources.

When a full sync is triggered, I get the following few log lines:

time="2021-07-05T15:54:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-07-05T15:54:55Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: test02)" application=test02-services
time="2021-07-05T15:54:55Z" level=debug msg="https://github.com/********/EnvironmentsTest has credentials"
time="2021-07-05T15:54:55Z" level=info msg="getRepoObjs stats" application=test02-services build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=26 unmarshal_ms=26 version_ms=0
time="2021-07-05T15:54:55Z" level=debug msg="Retrieved lived manifests" application=test02-services
time="2021-07-05T15:54:55Z" level=debug msg="built managed objects list" application=test02-services
time="2021-07-05T15:54:55Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-07-05T15:54:55Z" level=info msg=Syncing application=test02-services skipHooks=false started=false syncId=00005-AoxJn
time="2021-07-05T15:54:55Z" level=info msg="Tasks from managed resources" application=test02-services resourceTasks="[Sync/0 resource extensions/Ingress:test02/cassiopeia obj->obj (,,), Sync/0 resource apps/Deployment:test02/orders obj->obj (,,), Sync/0 resource /Service:test02/withdrawals obj->obj (,,), Sync/0 resource apps/Deployment:test02/addresses obj->obj (,,),...

the last log line is repeated a few times and each contains long list of resources. Then the controller pod restarts.

Any suggestions where to look next please?

[jannfis]

Hi @lstyles, do you have any resource limits on your application controller workload?

Also, 450 resources for a given app is not a small amount of resources, and I'd really suggest that you think about cutting your applications in more slices. E.g. you wrote that you have 40 microservices per application. I think the best practice here would be to cut each microservice into its own application.

[lstyles]

Thanks.

Yes I do have resource limits set up but they're really high compared to observed usage on the pod.

If I cut every microservice in a separate application and use app of apps pattern to group them in environment I will quickly have 200+ apps. Will it not cause other issues?

The other thing I've just noticed while testing is that I can't see an easy way to pause a sync of children apps from the top level.