fix(opal-server): freeze publishes during broadcaster backbone gaps to keep the fleet consistent

[Zivxx]

The problem

Follow-up to #915. That PR made a fleet of OPAL servers survive a broadcast-backbone outage (reconnect with backoff, buffer outbound broadcasts, resync clients on recovery). Multi-server validation of it surfaced a consistency gap during the outage:

A server-side publish fans out two independent ways — local in-process delivery to that worker's own clients, and the broadcaster to peer workers. Only the outbound path is buffered when the backbone is down; local delivery still fires. So an update that reaches one server mid-outage is applied to its clients while every other server's clients hold old data — a fleet split that lasts the whole outage. In a 3-server / 9-client test against a real managed-Postgres backbone stop, a data update published during the outage left the fleet at 3/9 new vs 6/9 old for the entire ~13-minute gap. For an authorization system, that means the same request can be allowed by one PDP and denied by another until the backbone returns.

The fix

FreezablePubSubEndpoint gates client-facing publishes on live backbone connectivity: while the ReconnectingBroadcaster is mid-gap (an established backbone session was lost and is being re-acquired — see is_in_backbone_gap()), publish is skipped entirely. Clients stay connected but frozen on the last consistent state; the write still lands in its datasource; and the existing reconnect resync (from #915) makes every worker's clients refetch on recovery — so the whole fleet moves together instead of diverging. Skipping the whole publish also keeps the replay buffer out of the recovery path for these updates: convergence is purely the resync refetch.

Configurable via OPAL_BROADCAST_FREEZE_ON_DISCONNECT (default true; set false for the previous behavior).

Scope & semantics (please read if you rely on runtime data updates)

• Covered: updates whose data clients re-fetch on reconnect — the configured data sources (OPAL_DATA_CONFIG_SOURCES / scope config) and the policy bundle. These are reconciled fleet-wide on recovery.
• Dropped, not deferred: one-off updates published during a gap that fall outside that set — inline data payloads, or fetch URLs not part of the configured sources. Consistency is chosen over freshness for these; if you rely on them, set the flag to false.
• Exempt (keep the old deliver-locally + buffer-for-replay behavior): internal coordination topics — __-prefixed channels (statistics protocol, broadcaster keepalive) and the git-webhook trigger topic. These target server-side subscribers, not clients; freezing them would break statistics state and silently drop repo-pull triggers that no resync re-issues.
• Guardrail: the resync is the freeze's only recovery path, so enabling the freeze with OPAL_BROADCAST_RESYNC_ON_RECONNECT=false is refused with a warning (freeze disabled) rather than silently losing updates.

Hardening (second commit)

An adversarial review of the first commit caught real issues, all addressed:

• Gap-only gating — freeze keys off is_in_backbone_gap() (had a session, lost it, retrying), not mere "not subscribed". Freezing while the reader was never started (idle worker — the reader only runs while a listening context is held) or never yet connected (boot) would silently drop publishes on a healthy backbone with nothing to ever reconcile them; those states delegate to the pre-freeze behavior. Gap state is scoped to the reader task's own lifetime, mirroring the resync's semantics.
• notify = publish alias re-bound — the upstream endpoint aliases notify = publish at class level, which binds the base publish; without re-binding, endpoint.notify(...) bypassed the gate entirely.
• Log hygiene — one WARNING per freeze episode (subsequent hits at DEBUG) plus a suppressed-count summary on recovery, instead of a WARNING per frozen keepalive for the whole outage.
• Known limitations (documented in the class docstring; each degrades to the pre-freeze behavior, never worse): an outbound broadcast can fail while the reader subscription is alive (delivered locally + buffered, as before); the gate reopens on re-subscribe before the session proves sustained (a rare connect-flap can slip a publish through); client-originated RPC publishes bypass the endpoint override.

Validation

• Unit: 63 opal-server tests green (35 in the reconnecting-broadcaster suite, 8 of them new), including a fault-injectable in-memory backbone driving the full gap lifecycle, the freeze matrix (never-started / never-connected / gap / reconnected / exempt topics / disabled / no broadcaster / stock broadcaster), suppression + episode-counter behavior, reader-restart state hygiene, and the notify alias pin.
• Multi-server staging (3 servers × 9 clients, shared Postgres backbone, real stop/start and reboot outages):
  • Without the fix: update during the outage → 3/9 clients diverge for the entire gap.
  • With the fix: same choreography → 0/9 leak (write-target server's own clients included), held across a ~20-minute outage; all 9 clients stayed connected, 0 worker restarts.
  • Source-based end-to-end: a value written to a datasource during the gap, with its update notification provably frozen at the gate (and nothing buffered), converged 9/9 after recovery via the resync refetch alone.
  • The final run also confirmed the exemptions live (internal topics buffered-for-replay during the gap, as before this PR) and the episode logging (single WARNING + recovery summary).

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for opal-docs canceled.

Name	Link
🔨 Latest commit	4082c58
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a4ce424b8d05900088a6312

[Copilot]

Pull request overview

This PR introduces a “fleet-consistency freeze” for OPAL Server publishes during broadcaster backbone gaps so that writes received by a single worker during an outage are not delivered locally (and thus can’t split the fleet) until recovery via reconnect resync.

Changes:

• Add backbone subscription/gap tracking to ReconnectingBroadcaster and a new FreezablePubSubEndpoint that suppresses publish() during gaps (with exemptions).
• Wire the new endpoint into server pubsub initialization with a safety guardrail when resync is disabled.
• Add config flag BROADCAST_FREEZE_ON_DISCONNECT (default true) and new unit tests covering gap lifecycle + freeze behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
packages/opal-server/opal_server/tests/reconnecting_broadcaster_test.py	Adds tests for is_backbone_connected / is_in_backbone_gap lifecycle and freeze gating behavior.
packages/opal-server/opal_server/pubsub.py	Switches server endpoint to FreezablePubSubEndpoint and applies freeze + exemptions + resync guardrail.
packages/opal-server/opal_server/pubsub_resilience.py	Adds backbone connectivity state, gap detection API, and implements FreezablePubSubEndpoint.
packages/opal-server/opal_server/config.py	Introduces BROADCAST_FREEZE_ON_DISCONNECT configuration option and documentation text.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeevmoney]

See comments

[zeevmoney]

Re-reviewed at a2a01fc. All findings from the previous round are verified fixed in the actual code, not just marked resolved:

• Freeze episodes no longer merge across back-to-back gaps (_gap_generation bumped on the connected→disconnected edge; publish flushes the prior episode's summary on a generation change; covered by test_back_to_back_gaps_get_separate_episodes).
• The guardrail now keys off the broadcaster actually built — full truth table traced: silent when no broadcaster, INFO no-op note for the stock broadcaster, WARNING+disable only where the freeze could actually engage; config description matches the code.
• The exempt list now carries the seven configured channel names — all verified to exist on the right config objects (no startup risk) and complete against every server-consumed publish site.
• E2E: fixed sleep replaced by wait_for_backbone_drop polling (grep patterns match the reader's real log lines), summary assertion polls across the keepalive interval, negative assertions capture logs and fail on empty, _should_freeze is now the single predicate publish uses, empty/None topic lists can't slip past the gate, get_running_loop, awaited dummy tasks, FakeNotifier.unsubscribe — all in place.
• Tests: 64/64 opal-server tests green from this branch's sources; no new warnings.

Two remaining issues, neither blocking:

[MEDIUM] app-tests/run.sh:486, 496 — the new positive freeze assertions cannot actually fail the E2E run. main is invoked as main && break (line 519), which suppresses set -e inside it, and check_servers_logged only returns grep's status without exiting. So "freezing publish…" and "resyncing this worker's clients" are swallowed if they miss. The core no-fleet-split property is still enforced (check_clients_not_logged exits explicitly), but the freeze/resync-fired checks are decorative until check_servers_logged hard-fails on miss (mirror the negative helpers' exit 1).

[LOW] app-tests/run.sh:304-320 — residual race in wait_for_backbone_drop: the poll returns on the first drop observed across all 8 workers, while publish_data lands on one specific worker whose own reader may not be mid-gap yet. Much narrower than the old sleep 3 (SIGKILL severs all backbone connections at once) and absorbed by the retry wrapper, but a miss is still possible; polling until the count rises by the worker count, or retrying the publish until the freeze log appears, would close it.