etcdserver: leader changed

[auswells]

We run a flux helm controller in multiple AKS clusters across different regions and occasionally encounter these errors:

reconciliation failed: Helm install failed: etcdserver: leader changed

I've stumbled across this post https://serverfault.com/questions/1097410/etcd-snapshots-causing-etcdserver-leader-changed which mentions:
"From what we've learned there is an ETCD snapshot performed every 2h on AKS resulting in this leader change"

However, I have not been able to find any information about this in the AKS docs, nor any confirmation that this is true from any of the Azure support cases I have opened.

Looking for additional information if there is periodic etcd maintenance so that we may plan accordingly.

Thanks in advance

[Jorge Palma (palma21)]

Yes snapshots are performed frequently. How frequent depends on multiple factors and the environment.
We don't typically share any implementation details of the Azure managed sides which if why you won't find this in any docs. So yes ideally you should plan for maintenance (which goes beyond snapshots) that can occur and is possible, within the 99.95 SLA of a Standard cluster to cause things like leader re-elections. Any client retry should work, we can also seek to see why API server itself didn't if you want to open a ticket or are using the supported Gitops Agent.

[benchwang]

Jorge Palma (@palma21) , compact and defragmentation are required by etcd, fyi: https://etcd.io/docs/v3.4/op-guide/maintenance/

[Pascal Bourdier (PascalBourdier)]

it is a very good reason to stop use etcd (like GKE or EKS) !