Repository navigation
ArgoCD repo-server serves stale Helm chart when same version is re-uploaded to repository #29194
Replies: 3 comments 4 replies
|
The setting you tuned governs a different cache from the one holding you back, which is why dropping it to one minute changed nothing. There are two layers.
func (c *nativeHelmChart) getCachedChartPath(chart string, version string) (string, error) {
keyData, err := json.Marshal(map[string]string{"url": c.repoURL, "chart": chart, "version": version})
...
}No digest, no content hash. Worth checking your version against this, because it changes what you can do:
So a TTL on the chart cache exists on HelmChartCacheExpiration: repoCacheExpiration,which means once that ships, your existing Until then: Bump the chart version. This is the real fix, not a workaround. Helm's ecosystem treats To clear what's already cached, restart the repo-server pods. The cache lives under If you're stuck with same-version overwrites for reasons outside your control, that's worth stating explicitly in a feature request, because the |
|
Hi @MLuc24 , Thank you for the thorough and precise response — this is exactly the clarity we needed. The To add some context on our use case: this requirement primarily comes from the development and integration testing stage, where engineers frequently iterate on chart content without bumping the version. In production, we follow proper versioning and this isn't an issue. We agree that bumping the chart version is the correct approach and will adopt that going forward. Setting We'll watch for the master change to land in a release. If it eventually supports a per-repo or per-application override (rather than a global TTL), that would be ideal for our case — we could set a short TTL only for the development Helm repo without affecting others. Thanks again for pointing us to the exact source code and saving us significant debugging time. Regards, |
|
I need to correct something I told you, and the correction is good news: hard refresh does clear the chart cache. I said it wouldn't. That was wrong.
} else if source.IsHelm() {
if settings.noCache {
err = helmClient.CleanChartCache(source.Chart, revision)
if err != nil {
return err
}
}and compareResult, err := ctrl.appStateManager.CompareAppState(
ctx, app, project, revisions, sources,
refreshType == appv1.RefreshTypeHard, // -> noCache
...
)So a hard refresh deletes the cached tarball for that chart before re-rendering, rather than only invalidating the manifest cache as I claimed. That matters for the thing you said would be ideal:
You have the per-application version of that today. Hard refresh is scoped to the application you run it on — no global TTL, no repo-server restart, no effect on any other application's cached charts: (the same button exists in the UI under Refresh). For your described workflow — engineers iterating on chart content during integration testing — that's a one-command cache bust on exactly the app being worked on, and the cost falls only on that app's next reconcile. So the ordering I'd suggest is different from what I said before: hard refresh for the dev loop, version bumps for anything that leaves it, and the Sorry for the wrong steer on the first pass — I'd read the |
|
Hi @MLuc24, Again thanks for your input. I did try Below logs show when I triggered app hard refresh. Below are logs from repo server pod. So looks like there was a cache miss seen at Not sure why even with Regards, |
|
@MLuc24 has the mechanism right, and one of your own verification steps is what's hiding it from you. You checked kubectl exec -n argocd deploy/argocd-repo-server -- ls -la /tmp/_argocd-repo/The tarball is one of those UUIDs, with no chart name and no version in the path. The key that maps to it is identity only ( keyData, err := json.Marshal(map[string]string{"url": c.repoURL, "chart": chart, "version": version})
// ...
return c.chartCachePaths.GetPath(string(keyData))and if val, ok := p.paths[key]; ok {
return val, nil
}
uniqueId, err := uuid.NewRandom()
// ...
repoPath := filepath.Join(p.root, uniqueId.String())
p.paths[key] = repoPathNo digest anywhere. The cheap unblock is a Hard Refresh, not a pod restart. That cache does have an invalidator: That also fixes the test you'd want. A normal refresh will keep serving A repo-server restart works too, for a different reason worth knowing: the key-to-path map lives on the The durable fix is to stop republishing a version. There's no digest in that key by design, and Helm treats a chart version as immutable content. Bump to 1.0.1, or point |
|
The delay isn't randomness, it's that a hard refresh is a request rather than an action, and your logs happen to show a second render beating it to the cache.
The other half is visible in your own timeline, and it's why the old chart kept getting served in the meantime. Your repo-server logs a manifest cache miss at That also explains why deleting the repo-server pod looked instant while For your automation, the fix is to stop treating the API return as completion. The annotation is removed after the controller processes it, so poll for its absence, or watch kubectl get app ab-cp1 -n argocd \
-o jsonpath='{.metadata.annotations.argocd\.argoproj\.io/refresh}'Empty means it's been consumed and the fresh chart is in play. Waiting on that instead of on the call returning will make the timing deterministic, and it'll stay correct whatever you set |
This is Argo CD's manifest cacheThe repo-server caches generated manifests keyed on the source revision (git commit / chart version). If you re-upload the same version to the Helm repo, the source revision doesn't change, so Argo CD keeps serving the cached manifests until the cache entry is invalidated. Ways to force it to pick up the new chart
For charts sourced from a git repo, add the If you re-upload same-version charts regularly, bump the patch version per upload — same-version re-upload is exactly the case the cache is designed to avoid re-processing. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Problem
When a Helm chart is re-uploaded to a Helm repository with the same version but different content, ArgoCD's repo-server continues to render manifests from the previously cached chart indefinitely. The Helm repository correctly serves the updated chart, but ArgoCD does not pick up the change.
Environment
Scenario
Verification
What we tried
Repo-server logs timeline (UTC)
T+0m00s manifest cache miss ← app created, first render uses old chartT+0m03s manifest cache hit ← cached old manifests servedT+4m40s manifest cache miss ← hard-refresh triggered programmaticallyT+4m43s manifest cache hit ← re-cached, still old chart contentT+6m05s manifest cache miss ← 1m TTL expiry, still old chartT+11m37s manifest cache miss ← new chart eventually picked upQuestions
All reactions