fix: recover from a stuck legacy (cert-based) auth/refresh request

[bjornmage]

Symptom

A cert-based (non-OIDC) identity lost its API session and never recovered on
its own -- it looped on UNAUTHORIZED/no api session token set for ziti_controller continuously for hours, well past the point the underlying
disruption had cleared, until the process was manually restarted. This is the
same defect class as #990 ("OIDC token refresh never recovers from network
disruption, permanent UNAUTHORIZED state requires restart"), but #990's fix
(#1025) is scoped entirely to oidc.c/oidc.h/credentials.c -- the OIDC
token-refresh path. The identity in this report uses cert-based auth
(--identity <file>.json, not OIDC enrollment), so #1025's fix does not
cover this code path. #572 (2023, general controller-API "stuck, never
reconnects" pattern, not OIDC-specific, closed without an evident code fix)
looks like the same still-open gap in the non-OIDC path.

Environment: ziti-edge-tunnel/ziti-sdk-c 1.18.1, tlsuv v0.41.4, 3-node
Raft HA controller cluster, 30-minute API session timeout. Client process
never crashed or exited (systemctl showed active (running) throughout),
so nothing triggered Restart=always; only an explicit process restart
recovered it, and the fresh process authenticated immediately.

# Steady-state failure loop, repeating every ~10s for hours:
ERROR ziti-sdk:ziti.c:1728 update_identity_data() ztx[1] failed to get identity_data: no api session token set for ziti_controller[UNAUTHORIZED]
ERROR ziti-sdk:ziti.c:1647 edge_routers_cb() ztx[1] failed to get current edge routers: code[0] UNAUTHORIZED/no api session token set for ziti_controller
ERROR ziti-sdk:connect.c:524 process_connect() conn[...] ziti context is not authenticated, cannot connect to service[...]
ERROR tunnel-cbs:ziti_tunnel_cbs.c:128 on_ziti_connect() ziti dial failed: invalid state

Root cause

tlsuv_http exposes a connect timeout (tlsuv_http_connect_timeout()) but
no idle/response timeout. In legacy_auth.c, once a refresh/auth request
completes its TCP handshake, there is no bound on how long it can wait for a
response. If the peer accepts the connection and then never responds (e.g. a
controller instance that disappears mid-request during a rolling restart,
which is what coincided with the session invalidation in this report),
legacy_session_cb() is simply never invoked. auth->refreshing stays
latched true forever, and the existing retry/backoff logic in
legacy_session_cb() -- which does work correctly for every other failure
mode -- never gets a chance to run, because it's never reached.

Fix

Add a bounded watchdog timer (AUTH_REQUEST_TIMEOUT_SECONDS, 30s) around
each outstanding legacy refresh/auth request (library/legacy_auth.c). On
expiry it calls tlsuv_http_cancel_all() on the auth context's HTTP client,
which routes the stuck request back through legacy_session_cb() with
UV_ECANCELED -- the same transport-failure path already exercised for
other network errors, and it re-arms the existing retry/backoff calculation
unchanged. The watchdog timer is started alongside every outstanding
request and stopped as soon as a response (successful or not) arrives, so it
never fires for healthy sessions.

This mirrors the recovery semantics #1025 added for the OIDC auth path,
applied to the legacy/cert-based path that #1025 did not cover.

Handle lifecycle: req_timer is closed and its close callback chains into
closing the existing timer handle (whose close callback frees auth), so
auth is only freed after both embedded uv_timer_t handles have finished
closing.

No behavior change for healthy sessions: the watchdog only fires when a
request has gone unanswered past AUTH_REQUEST_TIMEOUT_SECONDS; the
existing delay/backoff calculations (refresh_delay(), next_backoff())
are untouched.

Test evidence

• Full existing unit test suite (ctest, Catch2, 83 tests / ~100k
  assertions) passes unchanged against the patched build -- no regressions.
• Added a standalone regression harness (not part of this PR, used for local
  verification) that opens a mock controller which accepts a connection and
  then goes silent, confirming: the watchdog fires at ~30s, cancels the
  stuck request, the existing backoff path schedules a retry, the retry
  opens a fresh connection, and a subsequent valid response drives the auth
  state machine to ZitiAuthStateFullyAuthenticated.
• Not verified: end-to-end recovery against a live controller cluster
  reproducing the exact rolling-restart timing from the original incident
  (tests/integ requires a live quickstart Ziti network and was out of
  scope for this change's local verification).

Related

• OIDC token refresh never recovers from network disruption, permanent UNAUTHORIZED state requires restart #990 -- same "permanent UNAUTHORIZED, restart required" symptom, fixed by
  Fix/OIDC reconnect refresh #1025, but scoped to OIDC auth only.
• Fix/OIDC reconnect refresh #1025 -- the OIDC-path fix this PR mirrors for the legacy/cert-based path.
• unable to recover API session #572 -- closed 2023, no evident code fix; same "stuck, controller API
  handle never reconnects" pattern via the general controller-API path,
  likely the same defect class as OIDC token refresh never recovers from network disruption, permanent UNAUTHORIZED state requires restart #990 in the non-OIDC path.

[github-actions]

Thank you for your submission! Please read and sign our Contributor License Agreement before we can accept your contribution.

You can retrigger this check by commenting recheck cla on this PR.

---

I have read the CLA Document and I hereby sign the CLA

---

Björn Mage seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{Posted by the CLA Assistant Lite bot.}

[ekoby]

recheck cla