argocd pods lose IAM access for the first few minutes

[balusarakesh]

Description

• we recently updated argocd to 2.1.3 from v2.0.0 and also the helm chart to 3.24.0 from version 3.0.0
• We are seeing a weird and peculiar issue with argocd version 2.1.3
• we use AWS IRSA for the pods to gain access to AWS resources like KMS keys which will allow argocd to decrypt helm secrets using sops
• for some reason, access to KMS keys is denied by aws IAM for the first few minutes like 40-50 minutes at the beginning and after some time later the pods can decrypt the secrets just fine
• as you can see from the FIRST screenshot, AccessDenied errors stop after sometime for the same set of pods
• we've also grabbed AWS IAM events (SECOND screenshot) and noticed that AccessDenied events suddenly stop from the same serviceaccount argocd-repo-server

Tried the following steps to fix the issue:

• I've tried re-creating the service account
• I also exec into the pod and able to decrypt the secrets using AWS KMS key and helm secrets dec command but the argocd-repo-server pods were not able to decrypt the secrets

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [] I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

To Reproduce

• we don't 100% know how to reproduce the issue
• we have a total of 6-8 environments, this happened in the 2 dev environments we upgraded to new argocd (when it happened in the first environment, I thought it was a glitch and ignored it)
• I've also created a new IAM role, ServiceAccount and a Deployment (argocd-repo-server) and it worked just fine for these new ones, this issue seems to happen only when we upgrade the helm chart OR maybe it happens when we upgrade argocd version

Expected behavior

• argocd-repo-server should have access to AWS resources as soon as the pods start

Screenshots

Version

argocd: v2.1.3+d855831
  BuildDate: 2021-09-29T21:51:21Z
  GitCommit: d855831540e51d8a90b1006d2eb9f49ab1b088af
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified 

[jannfis]

I'm not sure whether this could qualify as a bug in Argo CD. We do not provide the repo server deployment with a ServiceAccount mount by default.

When you use the default installation manifests for e.g. upgrade from one version to another, chances are that your manifests in the cluster (with a ServiceAccount mounted to repo-server pod) is overwritten with the vanilla version (which do not mount a ServiceAccount to the pod).

[balusarakesh]

overwritten with the vanilla version
this is not possible because I can see the ServiceAccount token inside the pod and also the same pod didn't show any AccessDenied errors after a few minutes