Skip to content

add feasible_xhat_creator convention for per-scenario-feasible candidates#677

Draft
DLWoodruff wants to merge 6 commits into
Pyomo:mainfrom
DLWoodruff:feasible-xhat-creator
Draft

add feasible_xhat_creator convention for per-scenario-feasible candidates#677
DLWoodruff wants to merge 6 commits into
Pyomo:mainfrom
DLWoodruff:feasible-xhat-creator

Conversation

@DLWoodruff
Copy link
Copy Markdown
Collaborator

@DLWoodruff DLWoodruff commented May 4, 2026

Summary

  • Adds a feasible_xhat_creator(*, solver_name, ..., **kwargs) -> {nodename: np.ndarray} convention for example modules, returning a candidate first-stage point feasible to fix in every real scenario's per-scenario subproblem. Distinct contract from Jensen's xhat (which silently skips infeasibility); needed by downstream consumers like findW's pin-dual algorithm where the pin cannot fall back.

  • Adds mpisppy/utils/xhat_helpers.py with two stateless primitives:

    • average_xhat_nonants builds and solves the average scenario; returns ROOT nonants.
    • lp_xbar_nonants builds each scenario, LP-relaxes, solves, and returns the probability-weighted average of ROOT nonants across scenarios.

    Each caller composes one of these with its own rounding/repair rule -- which rule is appropriate (e.g. np.ceil, np.round, identity, something else) is a model-specific decision that depends on monotonicity of recourse feasibility in each first-stage variable. The choice between the two helpers is also the caller's. The two are not interchangeable for a model like netdes: averaging data and averaging solutions don't commute when first-stage is binary, so average_xhat_nonants can produce a candidate that's infeasible to pin in real scenarios whose individually needed arcs the averaged-data problem missed. (Specifically what netdes_auxiliary uses below: lp_xbar_nonants followed by np.ceil, where the ceil is justified because in netdes opening more arcs only loosens recourse. A different model could pair either helper with a different rounding rule.)

  • Prototypes the convention in examples/netdes/netdes_auxiliary.py and examples/sslp/sslp_auxiliary.py. Auxiliary files keep the intro examples uncluttered; see Move example auxiliary functions into <model>_auxiliary.py to declutter intro examples #676 for the broader cleanup proposal that would also move average_scenario_creator out of farmer.py and netdes.py.

  • Not added to farmer (continuous first stage, no rounding needed).

Test plan

  • mpisppy/tests/test_feasible_xhat.py (8 tests, all pass) -- including a per-scenario pin-feasibility check on netdes that exercises the actual contract: build each real scenario, fix nonants to the candidate, solve, expect optimal/feasible.
  • test_jensens.py regression (33/33 pass) -- the helper refactor doesn't touch the Jensen's mixin.
  • Focused serial subset: test_sputils, test_config, test_solver_spec, test_scenario_tree, test_extensions, test_ph_main, test_ef_ph (352 pass; one pre-existing test_scenario_lpwriter_extension failure that reproduces on origin/main, unrelated).
  • ruff check . clean.
  • CI on PR.

…ates

Downstream consumers (e.g. findW's pin-dual algorithm) need a candidate
first-stage point that is feasible to fix in every real scenario's
per-scenario subproblem. Jensen's xhat path tolerates infeasibility by
silently skipping; the pin-dual path cannot. This is a separate
contract.

Convention: example modules expose feasible_xhat_creator(*, solver_name,
solver_options=None, **kwargs) -> {nodename: np.ndarray}. The rounding
rule that turns a deterministic-proxy solution into a feasible
candidate is model-specific (depends on monotonicity of recourse
feasibility in each first-stage variable) and lives in each module,
not in mpi-sppy.

mpisppy/utils/xhat_helpers.py provides two engines for the common
cases: average_xhat_nonants (solves the average scenario, optionally
LP-relaxed) and lp_xbar_nonants (solves each real scenario's LP
relaxation, returns probability-weighted xbar). The second is what
"feasible everywhere" callers actually want, since LP-relaxing the
average scenario can underestimate which first-stage variables need
to be active across real scenarios.

netdes_auxiliary.feasible_xhat_creator = lp_xbar_nonants + np.ceil,
relying on the y[e] - u[e]*x[e] <= 0 monotonicity (more arcs open
never tightens recourse). sslp_auxiliary rolls the same pattern with
np.round; sslp ships no average_scenario_creator, so the convention
explicitly does not require one.

Both files live in examples/<model>/<model>_auxiliary.py rather than
in <model>.py to keep introductory examples uncluttered for first-time
users; see Pyomo#676 for the broader cleanup proposal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 4, 2026

Codecov Report

❌ Patch coverage is 96.11650% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.16%. Comparing base (40aadef) to head (c113f8d).

Files with missing lines Patch % Lines
mpisppy/utils/cfg_vanilla.py 93.93% 2 Missing ⚠️
mpisppy/cylinders/xhatspecific_bounder.py 0.00% 1 Missing ⚠️
mpisppy/utils/xhat_helpers.py 97.82% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
+ Coverage   71.02%   71.16%   +0.13%     
==========================================
  Files         154      155       +1     
  Lines       19248    19351     +103     
==========================================
+ Hits        13671    13771     +100     
- Misses       5577     5580       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The convention's call site goes through feasible_xhat_creator, not
through average_xhat_nonants directly, so swapping in a model with
integer first-stage doesn't require changes at the consumer. To make
that point load-bearing, farmer needs a feasible_xhat_creator too --
the trivial continuous-first-stage no-rounding case. farmer_auxiliary
delegates to average_xhat_nonants and returns its result unrounded.

doc/src/feasible_xhat.rst documents the contract, the two helpers in
xhat_helpers.py, the choose-an-engine and choose-a-rounding-rule
decisions that belong to the caller, and three worked examples
(farmer continuous, netdes lp_xbar+ceil, sslp lp_xbar+round). Listed
under Advanced Topics in index.rst, with a see-also note added to the
top-of-file warning admonition in jensens.rst pointing readers who
need a per-scenario-feasible candidate (vs. a silently-skip-on-
infeasibility candidate) to the new doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@DLWoodruff DLWoodruff marked this pull request as draft May 4, 2026 18:05
DLWoodruff and others added 4 commits May 4, 2026 11:05
Add test_feasible_xhat.py to both coverage entry points so codecov sees
the lines that test_feasible_xhat already exercises locally:

- run_coverage.bash: new test_feasible_xhat (serial) phase, mirroring
  the existing test_jensens phase.
- .github/workflows/test_pr_and_main.yml:
  - "Basic regression tests" job (has cplex/xpress) gains a Test
    feasible_xhat step alongside Test xhat from file, so the solver-
    needing tests run and produce coverage data for xhat_helpers.py.
  - "unit tests (no solver required)" pytest list gains
    test_feasible_xhat.py alongside test_jensens.py, so the contract
    tests count in the no-solver job.

Without these the test runs fine locally and in any plain `pytest`
invocation but is invisible to codecov, so codecov/patch reports 0%
patch coverage on the new lines in mpisppy/utils/xhat_helpers.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds --<xhatter>-try-feasible-xhat-first per xhat spoke
(xhatshuffle, xhatxbar, xhatlooper, xhatspecific), parallel to the
existing --*-try-jensens-first surface. When set, the spoke calls the
module's feasible_xhat_creator once before its main loop, fixes the
returned ROOT cache as the candidate first-stage, evaluates the
expected objective across all real scenarios, and -- if feasible --
sends that as its first inner bound. Implementation lives in
_JensensMixin._try_feasible_xhat alongside _try_average_scenario_xhat.

Per spoke, the two pre-loop candidates are mutually exclusive:
cfg_vanilla._maybe_attach_feasible_xhat raises with a message naming
both CLI options if a user enables both --<xhatter>-try-jensens-first
and --<xhatter>-try-feasible-xhat-first on the same spoke. The
docstring spells out why -- jensens often gives a tighter incumbent
when its candidate is feasible, feasible_xhat is guaranteed feasible
but can be looser, and per-spoke the right pick depends on model
structure. Mixing across spokes is fine.

Discovery is flag-gated and lives in
cfg_vanilla._find_feasible_xhat_creator: tries the main scenario
module first, falls back to importlib-importing <module>_auxiliary,
and raises if neither defines feasible_xhat_creator. Auxiliary import
is skipped entirely when no flag is set.

Tests: extends test_feasible_xhat.py from 10 to 22 tests covering the
cfg_vanilla wiring (no-op, raise on missing creator, mutual-exclusion
raise, install path), the discovery helper (flag-off no-op, main-
module hit, auxiliary fallback, both-missing raise), and the mixin
method (flag off, feasible-update, infeasible-skip, bad return-shape).

Docs: feasible_xhat.rst gains a Discovery admonition and a "In-
cylinder use" section with the four flags and a mutual-exclusion
warning admonition. jensens.rst's see-also note is refined to mention
the new flags and the per-spoke mutual exclusion. examples/run_all.py
gains a netdes smoke entry exercising --xhatshuffle-try-feasible-
xhat-first end-to-end via the auxiliary-discovery path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When average_scenario_creator is not directly usable, the underscore
helpers (_scenario_data, _average_scenario_data, _build_model) can
still be worth shipping: feasible_xhat_creator may want to build an
averaged-data model internally -- including with relaxed Var domains
-- to derive a starting first-stage point, since its output is
projected and verified feasible per scenario rather than handed off
as a Jensen's bound. The data-only-averaging principle binds
average_scenario_creator, not feasible_xhat_creator.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant