Add shared gateway health contract - #112
Merged
Merged
Conversation
Three separate gateway integrations hit the same hazard independently today --
dronecot (6 capture workers), aprscot (APRSWorker + SensorWorker) and
sikw00fcot (three entry points, only two of which Conflicts= each other). Each
integrator worked it out for themselves, which is a good sign the library
should say it rather than leaving it to be rediscovered a fourth time.
The hazard: every write serialises a WHOLE document, so two processes sharing
an app_name do not merge. The file alternates between two disjoint sets of
counters once a second, and every reader sees whichever wrote last. A UI shows
figures flickering between two gateways' worth of traffic and nothing anywhere
reports a fault. It is not a crash, which is exactly what makes it nasty.
So the rule is now stated in the module docstring -- give the writer to the
single choke point the workers feed, not to each worker -- and, because a
docstring only helps someone who reads it, StatusWriter now detects the case:
before replacing the file it checks whether the document there carries a
different pid that is still alive, and if so warns ONCE and records
`contended_with` in its own output so a UI can report that the figures may not
be this gateway's.
Deliberately narrow, to avoid crying wolf:
* a dead pid is NOT contention -- that is our own previous run, and taking
the file over is the correct behaviour after a restart;
* our own pid is not contention;
* an absent or half-written file concludes nothing;
* the check is rate-limited to every 30s and stops entirely once it has
something to say, so the common correct case costs nothing.
Liveness uses os.kill(pid, 0), which sends no signal. A pid that exists but is
not ours to signal still counts as live.
6 new tests, 29 in the module, 131 in the suite. All five mutations tried were
caught: never detecting, warning per-write instead of once, treating a dead pid
as live, flagging our own pid, and dropping the field from the document.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0197da7dhvcPoHxYKamrYqyM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coordinated CoT naming, health, packaging migration, and AryaOS cockpit cutover.