add feasible_xhat_creator convention for per-scenario-feasible candidates#677
Draft
DLWoodruff wants to merge 6 commits into
Draft
add feasible_xhat_creator convention for per-scenario-feasible candidates#677DLWoodruff wants to merge 6 commits into
DLWoodruff wants to merge 6 commits into
Conversation
…ates
Downstream consumers (e.g. findW's pin-dual algorithm) need a candidate
first-stage point that is feasible to fix in every real scenario's
per-scenario subproblem. Jensen's xhat path tolerates infeasibility by
silently skipping; the pin-dual path cannot. This is a separate
contract.
Convention: example modules expose feasible_xhat_creator(*, solver_name,
solver_options=None, **kwargs) -> {nodename: np.ndarray}. The rounding
rule that turns a deterministic-proxy solution into a feasible
candidate is model-specific (depends on monotonicity of recourse
feasibility in each first-stage variable) and lives in each module,
not in mpi-sppy.
mpisppy/utils/xhat_helpers.py provides two engines for the common
cases: average_xhat_nonants (solves the average scenario, optionally
LP-relaxed) and lp_xbar_nonants (solves each real scenario's LP
relaxation, returns probability-weighted xbar). The second is what
"feasible everywhere" callers actually want, since LP-relaxing the
average scenario can underestimate which first-stage variables need
to be active across real scenarios.
netdes_auxiliary.feasible_xhat_creator = lp_xbar_nonants + np.ceil,
relying on the y[e] - u[e]*x[e] <= 0 monotonicity (more arcs open
never tightens recourse). sslp_auxiliary rolls the same pattern with
np.round; sslp ships no average_scenario_creator, so the convention
explicitly does not require one.
Both files live in examples/<model>/<model>_auxiliary.py rather than
in <model>.py to keep introductory examples uncluttered for first-time
users; see Pyomo#676 for the broader cleanup proposal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #677 +/- ##
==========================================
+ Coverage 71.02% 71.16% +0.13%
==========================================
Files 154 155 +1
Lines 19248 19351 +103
==========================================
+ Hits 13671 13771 +100
- Misses 5577 5580 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The convention's call site goes through feasible_xhat_creator, not through average_xhat_nonants directly, so swapping in a model with integer first-stage doesn't require changes at the consumer. To make that point load-bearing, farmer needs a feasible_xhat_creator too -- the trivial continuous-first-stage no-rounding case. farmer_auxiliary delegates to average_xhat_nonants and returns its result unrounded. doc/src/feasible_xhat.rst documents the contract, the two helpers in xhat_helpers.py, the choose-an-engine and choose-a-rounding-rule decisions that belong to the caller, and three worked examples (farmer continuous, netdes lp_xbar+ceil, sslp lp_xbar+round). Listed under Advanced Topics in index.rst, with a see-also note added to the top-of-file warning admonition in jensens.rst pointing readers who need a per-scenario-feasible candidate (vs. a silently-skip-on- infeasibility candidate) to the new doc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add test_feasible_xhat.py to both coverage entry points so codecov sees
the lines that test_feasible_xhat already exercises locally:
- run_coverage.bash: new test_feasible_xhat (serial) phase, mirroring
the existing test_jensens phase.
- .github/workflows/test_pr_and_main.yml:
- "Basic regression tests" job (has cplex/xpress) gains a Test
feasible_xhat step alongside Test xhat from file, so the solver-
needing tests run and produce coverage data for xhat_helpers.py.
- "unit tests (no solver required)" pytest list gains
test_feasible_xhat.py alongside test_jensens.py, so the contract
tests count in the no-solver job.
Without these the test runs fine locally and in any plain `pytest`
invocation but is invisible to codecov, so codecov/patch reports 0%
patch coverage on the new lines in mpisppy/utils/xhat_helpers.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds --<xhatter>-try-feasible-xhat-first per xhat spoke (xhatshuffle, xhatxbar, xhatlooper, xhatspecific), parallel to the existing --*-try-jensens-first surface. When set, the spoke calls the module's feasible_xhat_creator once before its main loop, fixes the returned ROOT cache as the candidate first-stage, evaluates the expected objective across all real scenarios, and -- if feasible -- sends that as its first inner bound. Implementation lives in _JensensMixin._try_feasible_xhat alongside _try_average_scenario_xhat. Per spoke, the two pre-loop candidates are mutually exclusive: cfg_vanilla._maybe_attach_feasible_xhat raises with a message naming both CLI options if a user enables both --<xhatter>-try-jensens-first and --<xhatter>-try-feasible-xhat-first on the same spoke. The docstring spells out why -- jensens often gives a tighter incumbent when its candidate is feasible, feasible_xhat is guaranteed feasible but can be looser, and per-spoke the right pick depends on model structure. Mixing across spokes is fine. Discovery is flag-gated and lives in cfg_vanilla._find_feasible_xhat_creator: tries the main scenario module first, falls back to importlib-importing <module>_auxiliary, and raises if neither defines feasible_xhat_creator. Auxiliary import is skipped entirely when no flag is set. Tests: extends test_feasible_xhat.py from 10 to 22 tests covering the cfg_vanilla wiring (no-op, raise on missing creator, mutual-exclusion raise, install path), the discovery helper (flag-off no-op, main- module hit, auxiliary fallback, both-missing raise), and the mixin method (flag off, feasible-update, infeasible-skip, bad return-shape). Docs: feasible_xhat.rst gains a Discovery admonition and a "In- cylinder use" section with the four flags and a mutual-exclusion warning admonition. jensens.rst's see-also note is refined to mention the new flags and the per-spoke mutual exclusion. examples/run_all.py gains a netdes smoke entry exercising --xhatshuffle-try-feasible- xhat-first end-to-end via the auxiliary-discovery path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When average_scenario_creator is not directly usable, the underscore helpers (_scenario_data, _average_scenario_data, _build_model) can still be worth shipping: feasible_xhat_creator may want to build an averaged-data model internally -- including with relaxed Var domains -- to derive a starting first-stage point, since its output is projected and verified feasible per scenario rather than handed off as a Jensen's bound. The data-only-averaging principle binds average_scenario_creator, not feasible_xhat_creator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
feasible_xhat_creator(*, solver_name, ..., **kwargs) -> {nodename: np.ndarray}convention for example modules, returning a candidate first-stage point feasible to fix in every real scenario's per-scenario subproblem. Distinct contract from Jensen's xhat (which silently skips infeasibility); needed by downstream consumers like findW's pin-dual algorithm where the pin cannot fall back.Adds
mpisppy/utils/xhat_helpers.pywith two stateless primitives:average_xhat_nonantsbuilds and solves the average scenario; returns ROOT nonants.lp_xbar_nonantsbuilds each scenario, LP-relaxes, solves, and returns the probability-weighted average of ROOT nonants across scenarios.Each caller composes one of these with its own rounding/repair rule -- which rule is appropriate (e.g.
np.ceil,np.round, identity, something else) is a model-specific decision that depends on monotonicity of recourse feasibility in each first-stage variable. The choice between the two helpers is also the caller's. The two are not interchangeable for a model like netdes: averaging data and averaging solutions don't commute when first-stage is binary, soaverage_xhat_nonantscan produce a candidate that's infeasible to pin in real scenarios whose individually needed arcs the averaged-data problem missed. (Specifically whatnetdes_auxiliaryuses below:lp_xbar_nonantsfollowed bynp.ceil, where the ceil is justified because in netdes opening more arcs only loosens recourse. A different model could pair either helper with a different rounding rule.)Prototypes the convention in
examples/netdes/netdes_auxiliary.pyandexamples/sslp/sslp_auxiliary.py. Auxiliary files keep the intro examples uncluttered; see Move example auxiliary functions into <model>_auxiliary.py to declutter intro examples #676 for the broader cleanup proposal that would also moveaverage_scenario_creatorout offarmer.pyandnetdes.py.Not added to farmer (continuous first stage, no rounding needed).
Test plan
mpisppy/tests/test_feasible_xhat.py(8 tests, all pass) -- including a per-scenario pin-feasibility check on netdes that exercises the actual contract: build each real scenario, fix nonants to the candidate, solve, expect optimal/feasible.test_jensens.pyregression (33/33 pass) -- the helper refactor doesn't touch the Jensen's mixin.test_scenario_lpwriter_extensionfailure that reproduces onorigin/main, unrelated).ruff check .clean.