debug_utils: async buffer sanity inspector + --inspect-buffers-on-shutdown#683
Open
DLWoodruff wants to merge 10 commits into
Open
debug_utils: async buffer sanity inspector + --inspect-buffers-on-shutdown#683DLWoodruff wants to merge 10 commits into
DLWoodruff wants to merge 10 commits into
Conversation
mpisppy/debug_utils/buffer_inspect.py is a passive content checker for SendArray / RecvArray that detects the SHUTDOWN-stomp signature plus a small set of per-Field invariants. Producers are not touched; callers pass an optional InspectContext (or an SPBase via duck-typing). _BoundSpoke.got_kill_signal gets an env-gated hook (MPISPPY_INSPECT_BUFFERS=1) that runs the inspector on each shutdown poll and prints findings only when the report is non-OK. See doc/designs/async_buffer_sanity_design.md for invariants leaned on (NaN-padding init from communicator_array, buf.id() as recv baseline), the check tables, and the wiring rationale. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the prior MPISPPY_INSPECT_BUFFERS env-var gate with a Config flag --inspect-buffers-on-shutdown (popular_args). The flag is propagated through cfg_vanilla.shared_options into opt.options and read by _BoundSpoke.got_kill_signal only when a shutdown is detected; when the report is non-OK the spoke prints rank info plus the verbose Report dump. The check fires at the moment of detection because a spurious shutdown's buffer state has not yet been overwritten by later activity. Public surface: mpisppy.debug_utils now re-exports inspect_buffer, InspectContext, and Report. Adds mpisppy/tests/test_buffer_inspect.py (34 cases covering generic checks, all per-field checkers, SPBase fallback, verbose dump, and the config-flag wiring). Wired into the unit-tests CI job. User-facing RST page (doc/src/debug_utils.rst) added under For Developers and Contributors; design doc updated to reflect the CLI-flag wiring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #683 +/- ##
==========================================
+ Coverage 70.14% 70.45% +0.31%
==========================================
Files 154 157 +3
Lines 19257 19474 +217
==========================================
+ Hits 13507 13721 +214
- Misses 5750 5753 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Switches the inspector's failure path at the shutdown call site from print() to warnings.warn(..., RuntimeWarning). This makes the signal filterable, capturable in tests, and escalatable to a hard error via python -W. We don't raise at this site because got_kill_signal runs during the collective shutdown path; aborting one rank would deadlock the others on the next barrier. Tests: - Unit: TestSpokeGotKillSignalWarning drives Spoke.got_kill_signal on a hand-stomped buffer (asserts the warning fires), on a legit shutdown (asserts silence), and with the flag off (asserts the inspector is skipped entirely). - Smoke: the existing multi-stage Aircond cylinder invocation in straight_tests.py now also passes --inspect-buffers-on-shutdown and -W error::RuntimeWarning:mpisppy.cylinders.spoke, so a healthy run doubles as a no-false-positives check; a regression that produces a buffer warning will fail the smoke. Design doc updated to reflect the warn/no-raise rationale and the new coverage layers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI runs straight_tests.py with python_args="-m coverage run ..." prepended. With -W placed after python_args, the resulting command was "python -m coverage run ... -W error::... -m mpi4py ...", which made coverage parse -W as one of its own flags and fail with "no such option: -W". Moving -W before python_args yields "python -W ... -m coverage run ... -m mpi4py ...", which Python parses first; coverage then runs the target in-process under the installed filter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the SHUTDOWN call site only inspected the SHUTDOWN recv
buffer, so six of seven per-field checkers (_check_nonant,
_check_lower_bounds, _check_upper_bounds, _check_objective_scalar,
_check_best_xhat) saw only synthetic buffers in unit tests. A false
positive in any of them would not have been caught by CI.
Spoke.got_kill_signal now delegates to _inspect_buffers_on_shutdown,
which inspects every registered receive buffer and every send buffer
through inspect_buffer. SHUTDOWN goes first and verbose; the rest are
non-verbose. InspectContext(spbase=self.opt) is threaded so checkers
that need nonant length pick it up via the spbase fallback. One
RuntimeWarning per bad buffer, tagged with field name and direction.
With the existing -W escalation in the straight_tests Aircond smoke,
this turns that smoke into a real-buffer no-false-positives guard for
every checker whose field is used by the run (SHUTDOWN, NONANT,
OBJECTIVE_INNER_BOUND, OBJECTIVE_OUTER_BOUND, BEST_XHAT). The bounds
checkers (NONANT_LOWER_BOUNDS / NONANT_UPPER_BOUNDS) stay uncovered by
smoke because Aircond doesn't run a spoke that produces those.
Tests:
- Two new unit tests: multi-buffer sweep (stomped OBJECTIVE_INNER_BOUND
among healthy buffers -> exactly one warning) and the healthy-only
sweep (no warnings under warnings.simplefilter("error", RuntimeWarning)).
- Existing tests adjusted: stub now exposes send_buffers, _split_key,
optional spbase nonant_length, and binds the new helper methods.
Design doc updated with the sweep description, what smoke covers, and
what remains uncovered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CI sweep on multi-stage Aircond surfaced a real false positive in _check_nonant: it compared len(data) against ctx.get_nonant_count(), but NONANT buffers are sized to the *publisher's* sum of per-scenario nonants across that publisher's local scenarios -- not per-scenario. With 24 leaf scenarios on the hub and 6 nonants per scenario, the buffer is 144 wide while spbase.nonant_length is 6, so the check spuriously fired. The recv buffer's logical length is already validated at registration in _validate_recv_field, so the hard-equality check in _check_nonant was redundant on the recv side. Relax it to "len(data) must be a positive multiple of nonant_count" -- still catches a torn or truncated buffer, no false positives in the multi-scenario case. Tests: - New: multi-scenario buffer (24 = 4*6) passes. - New: non-multiple length (10 not divisible by 6) is still caught. - Existing length-mismatch tests still pass; they use 5 vs 7, and 5 % 7 != 0 still trips the relaxed check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…init)
The previous check accepted {0.0, 1.0}, but Hub.send_terminate (the only
producer) only ever writes 1.0. So 0.0 -- and any other non-1.0 value --
can only appear via a cross-field stomp, an RMA race, or a producer bug.
Accepting 0.0 was a silent false negative on exactly the kind of stomp
signature the inspector is meant to catch.
The check now rejects anything that isn't (1.0 with write_id>=1) or
(NaN with write_id==0, the initial state from communicator_array).
Tests:
- New test_shutdown_zero_is_rejected: data[0]=0.0 is now flagged.
- Existing test_shutdown_value_not_in_set updated to match the new
error message; 0.5 is still flagged.
- Existing test_initial_state_passes (NaN + write_id==0) still passes.
Design doc updated to lead with the general rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Under numpy 2.x, repr(np.float64(0.0)) is 'np.float64(0.0)', not '0.0', so the substring check on "data[0]=0.0" failed in CI (which runs numpy 2.4.4) even though the rejection itself was working. Switch the match to the stable message text "only 1.0 is ever published". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A small barrier-style sanity check on an MPI comm: each rank Allreduces a uint8 0 with MPI.LOR, rank 0 announces START/STOP with the comm name, and any rank seeing a non-zero reduction prints the value, comm name, and a stack trace. The non-zero branch is exercised via a fake comm in the test so the failure-print path is covered without adding a test-only flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two things: 0. adds a test for mpi itself failing to do an LOR and
mpisppy.debug_utilscontaining a passive content-check utility for the MPI-RMA send and receive buffers used by the hub-and-spoke system, plus an opt-in CLI flag that runs the check at cylinder shutdown.Motivation. An xhat spoke was observed receiving a
SHUTDOWNsignal thatHub.send_terminatehad not produced — suggesting that the SHUTDOWN buffer region was being written to from somewhere unexpected. mpi-sppy had no runtime sanity check on buffer contents;_validate_recv_fieldonly checks layout at registration time. This PR adds the missing layer.What's in it
mpisppy/debug_utils/buffer_inspect.py— the inspector. Single entry pointinspect_buffer(buf, fld, ctx, *, send, verbose).InspectContextis duck-typed (accepts SPBase viagetattr); explicit fields take precedence.buf.id(); recv-side id does not regress belowbuf.id()or optionalctx.last_write_id; noinfin data; noNaNoncewrite_id >= 1; padding region (betweenlogical_lenandpadded_len) staysNaN.SHUTDOWN,NONANT,NONANT_LOWER_BOUNDS,NONANT_UPPER_BOUNDS,OBJECTIVE_INNER_BOUND,OBJECTIVE_OUTER_BOUND,BEST_XHAT. Adding a new checker is one function and one dict entry.--inspect-buffers-on-shutdownCLI flag inpopular_args, propagated throughcfg_vanilla.shared_options._BoundSpoke.got_kill_signalreads it and runs the inspector only when shutdown is detected (the diagnostic moment). Off by default; no overhead when unset.Tests (34 new)
mpisppy/tests/test_buffer_inspect.pycovers: every generic check, every per-field check, SPBase fallback for nonant length, verbose dump, severity ladder, and the config-flag wiring (default false, propagation throughshared_options). Wired into the existingunit-testsCI job so codecov picks it up.Documentation
doc/src/debug_utils.rst— user-facing page under For Developers and Contributors.doc/designs/async_buffer_sanity_design.md— internal design doc with invariants, check tables, and explicit non-goals (cross-cylinder consensus, module-level history state).Draft because
Opened as a draft so the design can be reviewed before any further hot-path integrations (e.g., wiring the inspector into
update_nonantsorsync_bounds) are added. The current scope is intentionally narrow: prove out the SHUTDOWN check, then expand.Test plan
python -m pytest mpisppy/tests/test_config.py mpisppy/tests/test_buffer_inspect.py— 112 passedruff checkclean on all touched files--inspect-buffers-on-shutdownpropagates throughshared_optionsfor hub and spoke; unset default propagates as False🤖 Generated with Claude Code