Serialize server election with a flock so only one server spawns - #15
Open
agigante80 wants to merge 1 commit into
Open
Serialize server election with a flock so only one server spawns#15agigante80 wants to merge 1 commit into
agigante80 wants to merge 1 commit into
Conversation
…ses fork #5) The election was meant to be bind()-atomic but wasn't: with SO_REUSEADDR set, two clients racing to start a server could both bind() the port before either listened, so both spawned a server. The loser's listen() then got EADDRINUSE, and its cleanup deleted the pidfile — wiping the winning server's identity — so both clients failed verify_server_identity and refused to connect. This is what made the four two-listener integration tests fail. Fix: serialize the election with a per-endpoint advisory flock (server.<port>.election.lock). ensure_server_running acquires it (non-blocking poll that short-circuits on is_server_up and never wedges on a stuck holder), the holder re-checks then binds + spawns + waits, and losers observe the server come up and return. Only one server ever spawns. Both prior invariants are preserved: server.py still writes identity before listen(), and SO_REUSEADDR stays (needed for fast rebind after a SIGKILLed server whose connections sit in TIME_WAIT). Tests: the four two-listener tests go green (verified across repeated runs), plus a new deterministic concurrency test (N threads → exactly one server). Full suite now 200 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WwNCo3qLamBCzhMVAHGuH9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14.
Problem
The server election is meant to be
bind()-atomic but isn't.spawn.pysetsSO_REUSEADDRon the client's bind socket, andSO_REUSEADDRpermits a secondbind()on the same port while neither socket has calledlisten()yet. So two sessions connecting at the same moment bothbind()and both spawn a server:listen()s, and serves.listen()raisesEADDRINUSE; its cleanup (_unlink_own_identity) sees B's own pid and deletes it, wiping the live server A's identity.verify_server_identityand printserver identity check failed … refusing to connect.This is what made the four two-listener integration tests fail (
test_two_clients_exchange_messages,test_send_routes_via_control,test_broadcast,test_list_shows_all_agents).Fix
Serialize the election with a per-endpoint advisory
flock(server.<port>.election.lock).ensure_server_runningacquires it via a non-blocking poll that short-circuits onis_server_up(so it never wedges on a stuck holder); the holder re-checksis_server_up, then binds + spawns + waits; losers observe the server come up and return. Only one server ever spawns.Both prior invariants are preserved:
server.pystill writes identity beforelisten()(closes the different race where a client's TCP probe succeeds before the pidfile exists);SO_REUSEADDRstays (needed for fast rebind after a SIGKILLed server whose connections sit inTIME_WAIT) — it's safe now that the flock guarantees a single binder.A dead lock holder releases the flock automatically (kernel-released fd lock), so there's no deadlock risk.
Tests
ensure_server_runningat once → exactly one server pidfile, identity-verified, all callers see it up (8/8 across runs).Full suite: 200 passed, 0 failed (previously 3–4 failing from this race).
Notes
Independent of the label/security PRs (branches off
main), so it can land standalone.