Skip to content

fix: re-handshake in place so a dropped session costs 0 polls, not 1 - #2

Merged
agjs merged 1 commit into
mainfrom
fix/session-timeout-retry
Aug 16, 2026
Merged

fix: re-handshake in place so a dropped session costs 0 polls, not 1#2
agjs merged 1 commit into
mainfrom
fix/session-timeout-retry

Conversation

@agjs

@agjs agjs commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

The bug

Every plug was answering only about half its polls in production:

avg_over_time(tapo_plug_up[3h])
  ASUS Ascent GX10  0.503
  DGX Spark         0.503
  k3s Cluster       0.503
WARNING Response error: 403 Forbidden
WARNING poll failed for plug DGX Spark (192.168.10.217): Tapo(SessionTimeout)
WARNING poll failed for plug ASUS Ascent GX10 (192.168.10.216): Tapo(SessionTimeout)

Three plugs, three IPs, identical 0.503 — that is not flaky hardware. The alternation was built into poll().

A Tapo session here survives exactly one full read; the next request answers 403 Forbidden. The old code reacted by setting up=0, clearing self.device, and waiting for the next cycle to handshake. So the loop could only ever go: success → expiry → re-handshake → expiry → forever.

The fix

poll() retries once in place: on failure it drops the session, re-handshakes, and reads again — reporting up=0 only if that also fails. A recoverable expiry now costs zero polls.

Each poller also gets its own ApiClient. Sharing one across plugs let their handshakes race, which is the likeliest reason a session was being invalidated after a single use to begin with.

Verified

test_poller.py models the plug that was actually observed — a session that dies after one complete read — and runs the real PlugPoller:

old code:  up per poll  : [1, 0, 1, 0, 1, 0, 1, 0, 1, 0]     success rate: 50%
new code:  up per poll  : [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]     success rate: 100%
           reauths: 9    hard failures: 0

A second case checks the retry does not paper over a plug that is genuinely gone — an unreachable plug still reports up=0 and increments the failure counter.

Three new metrics

This failure was near-invisible from outside: the process stayed up, Prometheus scraped it happily, and the power gauges kept moving from the successful half of the polls. Readings looked live — just arriving at half the intended rate, with energy totals quietly drifting.

Metric Why
tapo_plug_last_success_seconds gauges keep their last good values on failure, so nothing ever looks missing. time() - this is the true age of every reading.
tapo_plug_reauth_total recoverable expiries, absorbed silently — a rising rate means the session lifetime is degrading
tapo_plug_poll_failures_total polls that failed even after a re-handshake

CI

The test runs on pull requests as well as pushes; :latest still publishes only from main.

Consumer side

Alerting added separately in argocd-app-of-apps (manifests/observability/monitoring/tapo-alerts.yaml) so a half-dead exporter cannot hide again — TapoPlugPollsFailing fires below 90% poll success over 30m.

Closes #1

Every plug was answering only ~half its polls in production:

    avg_over_time(tapo_plug_up[3h])
      ASUS Ascent GX10  0.503
      DGX Spark         0.503
      k3s Cluster       0.503

Not flaky hardware — the alternation was built into poll(). A Tapo session here
survives exactly one full read; the next one answers 403 Forbidden /
Tapo(SessionTimeout). The old code responded by setting up=0, clearing
self.device and waiting for the NEXT cycle to handshake. So the loop could only
ever go success, expiry, re-handshake, expiry, forever.

poll() now retries once in place: on any failure it drops the session,
re-handshakes and reads again, and only reports up=0 if that also fails. A
recoverable expiry now costs nothing. test_poller.py models the observed plug
(session dies after one read) and pins the old behaviour at 50% versus 100% now.

Each poller also gets its own ApiClient. Sharing one across plugs let their
handshakes race, which is the likeliest reason sessions were being invalidated
after a single use in the first place.

Three metrics so this class of failure is visible from outside:

  tapo_plug_last_success_seconds  gauges keep their last good values on failure,
                                  so nothing looks missing — this is how a
                                  consumer measures the true age of a reading
  tapo_plug_reauth_total          recoverable expiries, absorbed silently
  tapo_plug_poll_failures_total   polls that failed even after a re-handshake

CI now runs the test on pull requests as well, and only publishes :latest from
main.

Consumer-side alerting added separately in argocd-app-of-apps
(manifests/observability/monitoring/tapo-alerts.yaml): TapoPlugPollsFailing
fires below 90% poll success over 30m.
@agjs
agjs merged commit 65d72e2 into main Aug 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tapo session expires mid-poll: ~50% of polls fail with 403 Forbidden / SessionTimeout

1 participant