Repository navigation
[Backups] Count backup attempts by outcome in the operator - #127
Draft
andrewkroh wants to merge 1 commit into
Draft
andrewkroh wants to merge 1 commit into
andrewkroh wants to merge 1 commit into
Conversation
A Backup object lives two to four hours. A cleaner CronJob deletes terminal and orphaned Backups after that, and the BackupOwnerReference on Cluster collects them sooner. kube-state-metrics reports only objects that exist, so a success ratio computed from kube_customresource_cnpg_backup_info over days silently shows only the last few hours, and it counts objects, not events. Add a Prometheus counter, cnpg_backup_attempts_total, with labels phase, method, and failure_reason. Register it on controller-runtime's global registry, so the operator serves it on the metrics endpoint it already exposes. Increment it at the terminal-phase early return in the Backup reconciler, because a long-lived operator with leader election observes every Backup transition. The instance manager cannot hold this counter: it dies with its pod, so a scale-to-zero wake resets it. Do not label by cluster, namespace, or backup name: cell and region already arrive as resource attributes from the collector, and the label set needs to stay bounded regardless of how many clusters exist. A terminal Backup stays around until garbage collection removes it, so the reconciler visits it many times. Add backupAttemptDeduper, an in-memory set of already-counted Backup UIDs, so each Backup counts once. Age an entry out 24 hours after the reconciler first counts it, not by a field read off the object, so the bound does not depend on CreationTimestamp being set. An operator restart starts the set empty, so a terminal Backup still present at that point counts again once. Add a unit test that reconciles the same terminal Backup three times and asserts the counter increments once. The test needs prometheus/client_golang/prometheus/testutil, so run go mod tidy to add its one new indirect dependency to go.mod.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Backup object lives two to four hours. The cleaner CronJob deletes terminal Backups after that, and the
BackupOwnerReferenceon the Cluster collects them sooner. A success ratio computed from Backup objects over days therefore shows only the last few hours, and it counts objects, not attempts. Today the only durable record of a backup attempt and itsfailureReason(#117) is the operator's log line.Add a Prometheus counter,
cnpg_backup_attempts_total{phase, method, failure_reason}. It is registered on controller-runtime's global registry, so the operator serves it on the metrics endpoint it already exposes. No scrape configuration changes. The Backup reconciler increments it at the terminal-phase return. The operator holds the counter, not the instance manager, because the instance manager dies with its pod and a scale-to-zero wake would reset it.The labels stay bounded:
failure_reasonis the closed enum from #117, empty for a completed backup. There is no cluster, namespace, or backup name label; cell and region come from the collector as resource attributes.The reconciler visits a terminal Backup many times before garbage collection removes it.
backupAttemptDeduperkeeps an in-memory set of counted Backup UIDs so that each Backup counts once. An entry expires 24 hours after it is first counted. An operator restart starts with an empty set, so a terminal Backup still present at that time counts once more.