fix: bound auto-update network load so a bad release can't spam GitHub/gateways

[sanity]

Problem

An aggregate-load audit of the auto-updater found an unbounded failure loop. When a release's install can never succeed (for example a checksum/signature that does not verify, so the fail-closed gate added in #4586 refuses every attempt), a node can loop indefinitely:

1. the node detects the newer version and exits 42 (update needed),
2. the supervisor runs freenet update, which fails the verification gate, so nothing is installed,
3. that failure is classified as "no change" (not a crash, not a lockout), so nothing counts it,
4. the supervisor restarts the node, which re-detects the same version — back to step 1.

Each iteration costs GitHub REST polls and a gateway reconnect, and the on-crash freenet update is a fresh process with no persistent rate limit, so the loop is unbounded fleet-wide. The bad-binary crash loop is already handled by #4591 (auto-rollback); this is the distinct install-failure loop, which #4586's fail-closed gate converted from a self-terminating path into a non-terminating one.

Solution

Three layered bounds (none touch the #4591 rollback state machine, #4586 checksum verify, #4587 signing, or #4588 exit-code machinery):

1. Persistent GitHub-poll rate limit (token buckets). Every GitHub release poll goes through an on-disk token bucket in state_dir(), so the limit holds across the fresh one-shot freenet update process and restarts. Two independent buckets — one for the node's get_latest_version, one for the supervisor-side get_latest_release — because the node always polls first in a restart cycle and a shared bucket would let it starve the installer. Capacity 8, refill 1 token / 10 min ⇒ ≤ ~6 polls/hour/node per path. Missing bucket = full (first boot works); corrupt or unpersistable = deny (fail closed); time-injectable for tests.

2. Per-target-version install-failure gate (mirrors the feat: crash-loop auto-rollback for the auto-updater (#4073) #4591 known-bad pin, but for install failures). After 3 deterministic install failures of the same version (checksum mismatch or invalid signature), the node's update detection and the installer stop acting on that version until a strictly-newer release appears. This is what actually stops the restart loop. Transient failures (download/network outage, a still-propagating manifest, extraction errors) are deliberately not counted, so a brief outage can't permanently suppress a good release. Atomic writes, degrade-safe reads.

3. Parity gaps. The macOS launchd wrapper gains a consecutive-failure cap (exits the loop; targets a tight crash loop via a healthy-runtime reset; treats a benign "nothing to do" updater exit as not-a-failure; exits 0 so launchd does not respawn). The systemd units gain RestartSteps + RestartMaxDelaySec restart backoff (systemd ≥ 254; ignored gracefully on older), reducing reconnect churn during any residual loop.

Testing

New unit/regression tests cover: the token bucket (capacity, refill, refill cap, deny-on-corrupt, fail-closed-on-unpersistable, missing-is-full, backwards-clock and dip-then-recover give no credit, the two buckets are independent); the install-failure gate (engages after N, allows a strictly-newer version, resets on target change, clears on success, degrade-safe, atomic, and an end-to-end "loop is bounded by the gate" test); the verification-vs-transient classification; and source-scrape pins that get_latest_version/get_latest_release consult their buckets, that RateLimited and PinnedKnownBad arms never exit 42, that a rate-limited poll records no check-time/backoff, that the install gate only counts verification failures, and the macOS wrapper cap/healthy-reset/exit-2 behavior. cargo fmt, cargo clippy --locked -- -D warnings, and cargo test -p freenet --bin freenet (307 tests) are green.

External review: codex review --base origin/main was run iteratively; all findings were addressed (separate buckets, fail-closed persistence, backwards-clock timestamp, macOS exit-0/healthy-reset/exit-2, rate-limited result not feeding the exit-42 fallback, check-time not recorded for denied polls, transient-vs-verification gating) and the final pass reported no actionable regressions.

Scope note

This per-node load-bounding targets the per-node auto-update crash-loop path (#4073). It intentionally does not cover the gateway-only release-agent's own GitHub fetch (crates/release-agent/src/github.rs, GET /repos/{repo}/releases/latest): that is the release-workflow gateway-push path, invoked once per release, not the per-node restart loop, so it is not an aggregate-load amplification vector and is out of scope here.

Refs #4073.

[AI-assisted - Claude]

[github-actions]

I have all the information needed to complete the review. Let me write up the findings.

---

Rule Review: No blocking issues — thorough change with excellent test coverage

Rules checked: git-workflow.md, code-style.md, testing.md
Files reviewed: 7 (auto_update.rs, rollback.rs, service.rs, service/linux.rs, service/macos.rs, update.rs, freenet.rs)

Warnings

None.

---

Key findings on each checked category:

fix: regression tests (testing.md): All three loop-bounding mechanisms have direct regression tests:

• Token bucket: 9 tests covering burst, refill, cap-at-capacity, corrupt/NaN/negative files, backwards clock, clock dip, first-boot (missing = full), and fail-closed on unpersistable write.
• Install-failure gate: 7 tests covering threshold engagement, version isolation, version-change reset, success clear, degrade-safe on corrupt/missing, atomic-write invariant, and end-to-end loop bounding.
• RateLimited result routing: rate_limited_never_triggers_exit_42 source-scrapes all three match sites to assert no update_tx.send in any RateLimited arm.

Edge-case coverage (testing.md): Excellent — boundary conditions (zero/capacity), error paths (corrupt, unpersistable), clock anomalies (backwards, dip-then-recover), and first-boot edge case are all covered explicitly.

Unwrap in production code (code-style.md): None found. All new production code uses unwrap_or(0), is_ok(), is_err(), and if let Err patterns.

Cleanup exemptions without TTL (AGENTS.md): The install-failure gate is not a GC exemption — it clears on success or version change. Does not fall under the AGENTS.md "Cleanup exemptions MUST be time-bounded" rule.

Retry loops / jitter (code-style.md): No new Rust retry loops were added. The token bucket is a gate, not a loop.

Fire-and-forget spawns (code-style.md): No new async spawns.

Commit messages (git-workflow.md): All commits (fix: and docs:) follow conventional commits and are appropriately scoped. Each fix: is a focused, single-concern change.

Info

• crates/core/src/bin/commands/update.rs (install outcome match) — The Err(_) => {} arm is a necessary catch-all (anyhow errors can't be exhaustively matched) and is clearly documented with a multi-line comment explaining the transient-vs-deterministic distinction. Not a style error, just flagging the pattern.

---

Rule review against .claude/rules/. WARNING findings block merge.

[sanity]

Reviewed and approving after a full review (this bounds the auto-update network-load failure mode an aggregate-load audit surfaced — a bad-manifest/sig release could otherwise loop forever, ~100-170 GitHub req/hr/node + a gateway reconnect/min fleet-wide, made infinite by #4586's correct fail-closed change).

Review: external Codex (8 iterative rounds, final pass clean) + 3 adversarial lenses — rate-limit bypass/amplification, install-gate false-positives, and end-to-end loop-termination + cross-platform parity + no-regression. No Must-fix. Key things verified:

• Rate-limit: on-disk token bucket (cap 8, refill 1/10min) at both GitHub-fetch choke points; fail-closed on corrupt/unpersistable; the fresh on-crash freenet update honors it; no bypass path; sound refill math (clock-rewind safe); normal ops have comfortable headroom.
• Install-gate (the loop-stopper): only DETERMINISTIC verification failures (checksum/sig) count — transient download/IO/missing-manifest errors deliberately do NOT, so a brief outage can't wrongly gate a good release; a strictly-newer version supersedes (no permanent freeze); degrade-safe.
• Parity + no-regression: macOS wrapper failure cap works; systemd RestartSteps backoff documented honestly re: the systemd unit: StartLimitBurst/StartLimitIntervalSec are in [Service], where systemd ignores them #4551 tradeoff (fast crashes still terminal-stop via StartLimit, confirmed in the live-systemd verify; medium crashes slow-flap gently); feat: crash-loop auto-rollback for the auto-updater (#4073) #4591 rollback / fix: make auto-update checksum verification fail-closed on Linux/Windows #4586 checksum / feat: add ed25519 release-artifact signing and verification to auto-update #4587 signing / fix: distinguish fast-crash from update exit to bound #4551 crash loops #4588 exit-codes all intact (separate counters, no collision).

307 tests pass (incl. token-bucket edge cases, gate classification, end-to-end loop-bound, source-scrape pins). CI green on 3d770e3. Merging.

[AI-assisted - Claude]