fix: re-handshake in place so a dropped session costs 0 polls, not 1 - #2
Merged
Conversation
Every plug was answering only ~half its polls in production:
avg_over_time(tapo_plug_up[3h])
ASUS Ascent GX10 0.503
DGX Spark 0.503
k3s Cluster 0.503
Not flaky hardware — the alternation was built into poll(). A Tapo session here
survives exactly one full read; the next one answers 403 Forbidden /
Tapo(SessionTimeout). The old code responded by setting up=0, clearing
self.device and waiting for the NEXT cycle to handshake. So the loop could only
ever go success, expiry, re-handshake, expiry, forever.
poll() now retries once in place: on any failure it drops the session,
re-handshakes and reads again, and only reports up=0 if that also fails. A
recoverable expiry now costs nothing. test_poller.py models the observed plug
(session dies after one read) and pins the old behaviour at 50% versus 100% now.
Each poller also gets its own ApiClient. Sharing one across plugs let their
handshakes race, which is the likeliest reason sessions were being invalidated
after a single use in the first place.
Three metrics so this class of failure is visible from outside:
tapo_plug_last_success_seconds gauges keep their last good values on failure,
so nothing looks missing — this is how a
consumer measures the true age of a reading
tapo_plug_reauth_total recoverable expiries, absorbed silently
tapo_plug_poll_failures_total polls that failed even after a re-handshake
CI now runs the test on pull requests as well, and only publishes :latest from
main.
Consumer-side alerting added separately in argocd-app-of-apps
(manifests/observability/monitoring/tapo-alerts.yaml): TapoPlugPollsFailing
fires below 90% poll success over 30m.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Every plug was answering only about half its polls in production:
Three plugs, three IPs, identical
0.503— that is not flaky hardware. The alternation was built intopoll().A Tapo session here survives exactly one full read; the next request answers
403 Forbidden. The old code reacted by settingup=0, clearingself.device, and waiting for the next cycle to handshake. So the loop could only ever go: success → expiry → re-handshake → expiry → forever.The fix
poll()retries once in place: on failure it drops the session, re-handshakes, and reads again — reportingup=0only if that also fails. A recoverable expiry now costs zero polls.Each poller also gets its own
ApiClient. Sharing one across plugs let their handshakes race, which is the likeliest reason a session was being invalidated after a single use to begin with.Verified
test_poller.pymodels the plug that was actually observed — a session that dies after one complete read — and runs the realPlugPoller:A second case checks the retry does not paper over a plug that is genuinely gone — an unreachable plug still reports
up=0and increments the failure counter.Three new metrics
This failure was near-invisible from outside: the process stayed up, Prometheus scraped it happily, and the power gauges kept moving from the successful half of the polls. Readings looked live — just arriving at half the intended rate, with energy totals quietly drifting.
tapo_plug_last_success_secondstime() - thisis the true age of every reading.tapo_plug_reauth_totaltapo_plug_poll_failures_totalCI
The test runs on pull requests as well as pushes;
:lateststill publishes only frommain.Consumer side
Alerting added separately in
argocd-app-of-apps(manifests/observability/monitoring/tapo-alerts.yaml) so a half-dead exporter cannot hide again —TapoPlugPollsFailingfires below 90% poll success over 30m.Closes #1