Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator by Tyler-lc · Pull Request #517 · iiasa/message-ix-models

Tyler-lc · 2026-05-26T18:45:35Z

This is a branch related to work in the Energy Foundation China, for coordination between @yiyi1991 and I.

Further details TBA.

Summary description from Claude

Summary

Add Yoga's meth_h2 mode-parity port for the hyway electrolyser parents (h2_elec_alk/pem/soe). Without it, ADDON_ACTIVITY_UP for meth_h2 collapses against the removed h2_elec and meth_h2 silently zeros across every EFC scenario.
Rename the hydrogen reporting YAMLs to IAMC-compliant form (Production|... → Secondary Energy|...) and add the Hydrogen|Fossil aggregator family.
Fix two pre-existing upstream bugs that surfaced while wiring this end-to-end (stale transact kwarg + empty IamDataFrame constructor incompatible with current pyam).

Commit structure

hydrogen reporting: IAMC renames + Hydrogen|Fossil aggregator family — 8 YAMLs in data/hydrogen/reporting/ + .gitignore patch for debug artifacts.
hydrogen: fix add_hydrogen_techs transact call, empty IamDataFrame in _filter_years — two narrow upstream fixes (data_hydrogen.py, report/hydrogen/h2_reporting.py).
efc: add Yoga meth_h2 mode-parity port + wire into EFC workflow — new model/hydrogen/yoga_modes.py module + new build_hydrogen step in project/efc/workflow.py (replaces the placeholder at "hydrogen added").

Background — why the port is needed

Upstream Yoga's update_meth_h2_modes (in project/ssp/script/util/functions.py) hardcodes h2_elec as the methanol_synthesis_addon parent and broadcasts its feedstock/fuel modes into six split modes ({feedstock, fuel}_{bic, dac, fic}). In the hydrogen module, add_hydrogen_techs removes h2_elec and adds the hyway techs, each populated from per-tech CSVs that still carry the pre-Yoga mode shape: addon_conversion/addon_up in feedstock/fuel, input/output/var_cost in M1. After the swap, ADDON_ACTIVITY_UP for meth_h2 has no parent to bind against — meth_h2 silently zeros.

apply_meth_h2_mode_parity reconstructs the binding on the chosen electrolyser parents:

Broadcasts each parent's M1 mode-indexed rows (input/output/var_cost/...) into the six split modes. Mode-invariance is intentional — the split modes are reporting/tracking labels in the hyway scope.
Replicates feedstock/fuel addon_conversion/addon_up rows onto the six split modes per Yoga's pattern; drops the originals.
Registers each parent in map_tec_addon under methanol_synthesis_addon.
M1 is retained — it remains the tech's primary operating mode for hydrogen unrelated to meth_h2.

Verification (on a fresh `baseline_BMT` clone)

Fingerprint: each parent has 7 input/output/var_cost modes (M1 + 6 split); addon_conversion and addon_up cover the 6 split modes; all 3 parents registered in map_tec_addon under methanol_synthesis_addon. meth_h2 retains its 6 split modes unchanged.
Solve: 6.5 min on R12, MESSAGE, status optimal, objective 7,377,529.
meth_h2 ACT: total 80.86 GWa across 25 nonzero rows (vs guaranteed-zero pre-port). Dominant in R12_CHN (72.8 GWa) with growth in 2090-2110 — the China hydrogen-derived-fuels framing EFC is meant to study.
Reporting (via EFC_2026/utils/reporting.py helpers): legacy + Genno H2 coexistence runs cleanly; 949 hydrogen-related IAMC tuples uploaded.

Bugfixes (commit 2)

data_hydrogen.py — add_hydrogen_techs used with scenario.transact(commit_message="..."). Current ixmp.TimeSeries.transact takes message, not commit_message — version drift. Also dropped the outer transact wrapper: the inner load_* helpers in utils.py each do their own check_out → add_* → commit cycle per CSV file, so the outer was double-managing transactions and colliding ("Scenario is currently locked" Java exception).
h2_reporting.py:_filter_years — empty-fallback constructed pyam.IamDataFrame() (no args). Current pyam rejects that; pass the already-empty df so the IAMC schema is preserved.

Test plan

Code review
Confirm fingerprint test passes on reviewer's machine (or in CI)
Confirm scenario solve produces non-zero meth_h2 ACT in CHN
Exercise the workflow's report() end-to-end (transport / materials / legacy / Genno H2) — partial verification done locally via the utils-helper path; workflow-path test was in progress at PR time

Handle items from MARK via @pytest.mark.key, where possible.

- Use via @pytest.mark.key_of_item, instead of importing MARK.

- Extend top-level testing.MARK using transport.testing.MARK, instead of making the latter a ChainMap of the former. - Adjust usage.

- 2 → "R12_only". - 3 → "ISR_no_data". - 4 → "GEA_not_implemented", "SHAPE_not_implemented". - 9 → "R14_no_data". - 10 → "transport_build_data". - Remove unused key 0. - Use "from pytest import mark" for brevity. - Expand type hints.

- Use in message_ix_models.testing hooks.

- Add "no_data", "non_public_data" package-wide marks. - Use factories through @pytest.mark.no_data("…").

- Picked from #462.

- Picked from #474.

- Consolidate options handling in .Options subclass. - Improve .transform(): - Handle broadcast to transport "F ROAD" technologies. - Always convert units. - Let parent class handle mapping to target key. - Add units to iea-2017-t4-2.csv.

- Add to .build.add_exogenous_data() - Add to docs.

- Update checks/tests.

- Use current K.t["F usage"]. - Change missed in #471.

- Address "MissingKeyError: required keys ('MERtoPPP',) not defined" in .report.prepare_reporter(). - Base scenario(s) such as: ixmp://ixmp-dev/SSP_SSP2_v6.5/baseline_DEFAULT_step_13 …appear not to have this parameter.

- Move IAMC_VAR_REPLACE out of convert_iamc(); document; expand: - Use "Transportation|Vehicle" for final energy total. - Expand "F" and "P". - Relabel: - "Passenger rail" (group) → "Passenger|Rail" to match individual techs. - "Rail|Freight" → "Freight|Rail"

- Deduplicate resid_cook_lightoil in buildings/commodity.yaml - Occurred during review of #433.

Per #433 (review)

- Add test.

…so that legacy reporting can merge its output.

- Add Hydrogen|Geological entry to h2_prod.yaml filtering on white_hydrogen_dist technology (mode M1, commodity hydrogen, level secondary) - Include prod_h2_geological in the level_2 Hydrogen aggregate in h2_prod_aggregates.yaml

… root

- Rename Production|... → Secondary Energy|... prefix in h2_prod and h2_transient family YAMLs (and their _hist counterparts) per IAMC convention. - Singularize Hydrogen|Gas → Hydrogen|Gases at family level (Q3 plural). - Add Hydrogen|Fossil aggregator family to h2_prod_aggregates.yaml and h2_prod_hist_aggregates.yaml: level_1 adds Hydrogen|Fossil|w/ CCS (prod_h2_smr_w_ccs + prod_h2_coal_w_ccs) and Hydrogen|Fossil|w/o CCS (prod_h2_smr_wo_ccs + prod_h2_coal_wo_ccs); level_2 rolls up both into Hydrogen|Fossil. h2_pyro_elec deliberately excluded — turquoise H2 stays only under Gases|Pyrolysis since carbon_black is a solid co-product, not atmospheric CO2. - .gitignore: cover /_*.xlsx and /*_check.xlsx debug artifacts at repo root.

… _filter_years Two pre-existing bugs surfaced while wiring the EFC workflow end-to-end: - data_hydrogen.py: add_hydrogen_techs wrapped its body in `with scenario.transact(commit_message="...")`. Current ixmp.TimeSeries.transact takes `message` (positional or keyword), not `commit_message` — version drift. Also, the inner load_commodity_sets / load_emission_sets / load_hydrogen_sets / load_hydrogen_parameters / remove_deprecated_sets helpers in utils.py each do their own check_out → add_* → commit cycle per CSV file, so the outer transact was double-managing transactions and collided with the inner check_out ("Scenario is currently locked" Java exception). Dropped the outer wrapper — inner functions manage their own transactions. - h2_reporting.py: _filter_years returned `pyam.IamDataFrame()` (no args) as the empty-fallback when year filtering produced zero rows. Old pyam accepted that; current pyam requires the `data` arg. Pass the already-empty `df` so the IAMC schema (model/scenario/region/variable/unit/year/value) is preserved.

Upstream Yoga's update_meth_h2_modes (message_ix_models.project.ssp.script.util.functions) hardcodes h2_elec as the methanol_synthesis_addon parent and broadcasts its feedstock/fuel modes into the six split modes ({feedstock,fuel}_{bic,dac,fic}). In the hydrogen module add_hydrogen_techs removes h2_elec and adds the hyway techs (h2_elec_alk/pem/soe + h2_pyro_elec + h2_ct), each populated from per-tech CSVs that still carry the pre-Yoga mode shape: addon_conversion/addon_up in feedstock/fuel modes, input/output/var_cost in M1. Without a port, ADDON_ACTIVITY_UP for meth_h2 collapses against the missing parent and meth_h2 silently zeros across every EFC scenario. New module model/hydrogen/yoga_modes.py: apply_meth_h2_mode_parity(scenario, parents) — for each parent electrolyser, broadcasts M1 mode-indexed rows (input/output/var_cost/etc.) into the 6 split modes and replicates feedstock/fuel addon_conversion/addon_up entries per Yoga's pattern, removing the originals. Registers parents in map_tec_addon under methanol_synthesis_addon. M1 is retained — it remains the tech's primary operating mode for hydrogen unrelated to meth_h2. EFC workflow integration (project/efc/workflow.py): - New build_hydrogen step composes add_hydrogen_techs (now-fixed) with apply_meth_h2_mode_parity. Replaces the placeholder at the "hydrogen added" workflow step. - Methanol-addon parents: h2_elec_alk, h2_elec_pem, h2_elec_soe. h2_pyro_elec excluded (separate turquoise framing); h2_ct excluded (consumes H2 rather than producing it). - report() extended with Genno H2 reporter coexistence step after the legacy IAMC reporter; tech-presence guard skips Genno cleanly on pre- hydrogen-added scenarios. Verification on a fresh baseline_BMT clone: - Fingerprint passes: each parent has 7 input/output/var_cost modes (M1 + 6 split); addon_conversion and addon_up cover the 6 split modes; all 3 parents registered in map_tec_addon under methanol_synthesis_addon. - Solved in 6.5 min on R12 (MESSAGE, status optimal, objective 7,377,529). - meth_h2 ACT total 80.86 GWa across 25 nonzero rows (vs guaranteed-zero pre-port). Dominant in R12_CHN (72.8 GWa) growing 2090-2110.

New config-driven reporting families for the hydrogen Genno reporter, auto-discovered by fetch_variables(): - h2_flow_out.yaml (var: out): out|<tech>|hydrogen for the 10 production routes in the scenario. - h2_flow_in.yaml (var: in): in|<tech>|<commodity> feedstock/energy flows (biomass/coal/gas/electricity/ht_heat). Display labels follow EFC mapping_clean; filters use model commodity names. - h2_flow_in_aggregates.yaml: in|hydrogen|total electricity, summing electricity over electrolysers + electric pyrolysis + SMR-CCS capture. Per EFC stakeholder request to report concrete per-technology energy flows alongside the aggregated Secondary Energy| variables. Electrolyser names normalised to the hyway canonical set; hard-zero electr placeholders and absent techs (h2_cog) excluded; pyro gas, SOEC ht_heat added as real flows.

generate() set ssp/regions but never called apply_bmt_config, so context.transport was unbuilt and the CLI report path silently skipped transport. Also log the skip with exc_info.

khaeru and others added 30 commits May 8, 2026 16:51

Add pytest_{collection_modifyitems,configure} hooks

fec72b3

Handle items from MARK via @pytest.mark.key, where possible.

Add .testing.MARK["ci_linux_only"]

f2040b8

Mark 6 tests "ci_linux_only" to lower GHA run time

ab97437

Rename 9 entries in MARK; adjust usage

e0e6256

- Use via @pytest.mark.key_of_item, instead of importing MARK.

Reverse relationship of {,transport}.testing.MARK

9af5565

- Extend top-level testing.MARK using transport.testing.MARK, instead of making the latter a ChainMap of the former. - Adjust usage.

Add .util.pytest.MarkFactory

4bdfc6e

- Use in message_ix_models.testing hooks.

Replace transport.testing.make_mark w/ MarkFactory

ebf403a

- Add "no_data", "non_public_data" package-wide marks. - Use factories through @pytest.mark.no_data("…").

Add growth_{activity,new_cap}_lo for 2 LDV techs

ccd11b8

- Picked from #462.

Reduce R12_EEU LDV lifetime 28 → 22 years

2031f60

- Picked from #474.

Improve IEA_Future_of_Trucks

a9407e1

- Consolidate options handling in .Options subclass. - Improve .transform(): - Handle broadcast to transport "F ROAD" technologies. - Always convert units. - Let parent class handle mapping to target key. - Add units to iea-2017-t4-2.csv.

Add .transport.data.InputVehicle

bb8e27a

- Add to .build.add_exogenous_data() - Add to docs.

Merge energy input data for F RAIL and F ROAD

72f3c56

- Update checks/tests.

Check input values are generated for F usage techs

bb232c6

Update reference to "t::F usage"

0d9b6e8

- Use current K.t["F usage"]. - Change missed in #471.

Remove gdp_ppp key from report/global.yaml

112d3bc

- Address "MissingKeyError: required keys ('MERtoPPP',) not defined" in .report.prepare_reporter(). - Base scenario(s) such as: ixmp://ixmp-dev/SSP_SSP2_v6.5/baseline_DEFAULT_step_13 …appear not to have this parameter.

Add #497 to doc/whatsnew

f615190

Partly revert d86843e

b7fc7cc

- Deduplicate resid_cook_lightoil in buildings/commodity.yaml - Occurred during review of #433.

Sort .cli.submodules

e1b8d1f

Per #433 (review)

Add TODO reduce complexity of prepare_data_B()

62407f1

Per #433 (review)

Use .transport.workflow.add_steps in .bmt.workflow

7618e5d

- Add test.

Update BMT workflow

c7983a9

Update material haircut vetting

e41f17d

Update T in the BMT workflow

aa55249

Update T report TODO

2be870c

Add add_macro and clean BMT config

218bebe

Update add_macro for BMT

56bca1b

Update add_macro_materials

f392d96

Replace transport.report 0 with blanks in y<y₀

d378b8e

…so that legacy reporting can merge its output.

Tyler-lc and others added 7 commits May 13, 2026 13:19

Add geological (white) hydrogen to h2_prod reporting

e68454c

- Add Hydrogen|Geological entry to h2_prod.yaml filtering on white_hydrogen_dist technology (mode M1, commodity hydrogen, level secondary) - Include prod_h2_geological in the level_2 Hydrogen aggregate in h2_prod_aggregates.yaml

Track hydrogen module/reporting docs; ignore scenario outputs at repo…

129cb66

… root

Add EFC cli

9b20b04

Add initial EFC workflow structure

fa521e8

Tyler-lc requested review from OFR-IIASA, adrivinca, khaeru, macflo8, r-aneeque and ywpratama as code owners May 26, 2026 18:45

Tyler-lc temporarily deployed to publish May 26, 2026 18:45 — with GitHub Actions Inactive

Tyler-lc marked this pull request as draft May 26, 2026 18:51

Tyler-lc added 3 commits May 29, 2026 13:43

fix modes for h2_elec_*

f5ea67f

aggregation across R12_GLB naming

cf99cfd

Tyler-lc temporarily deployed to publish May 29, 2026 13:55 — with GitHub Actions Inactive

Tyler-lc temporarily deployed to publish May 29, 2026 14:04 — with GitHub Actions Inactive

Tyler-lc added 2 commits June 1, 2026 15:42

remove "commit_message" as argument, use positional

ccb0aff

add hydrogen sets first

a6a7bb5

Tyler-lc temporarily deployed to publish June 1, 2026 13:52 — with GitHub Actions Inactive

fix stale dictionary nesting

dcf69d8

Tyler-lc temporarily deployed to publish June 1, 2026 15:06 — with GitHub Actions Inactive

efc: build BMT config in workflow generate() so transport reporting runs

e57d595

generate() set ssp/regions but never called apply_bmt_config, so context.transport was unbuilt and the CLI report path silently skipped transport. Also log the skip with exc_info.

yiyi1991 temporarily deployed to publish June 3, 2026 07:39 — with GitHub Actions Inactive

Tyler-lc self-assigned this Jun 5, 2026

Tyler-lc added the p:EFC_2026 label Jun 5, 2026

Tyler-lc changed the title ~~EFC: Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator~~ Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator Jun 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator#517

Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator#517
Tyler-lc wants to merge 99 commits into
mainfrom
project/EFC_2026

Tyler-lc commented May 26, 2026 •

edited by khaeru

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Tyler-lc commented May 26, 2026 • edited by khaeru Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Commit structure

Background — why the port is needed

Verification (on a fresh baseline_BMT clone)

Bugfixes (commit 2)

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tyler-lc commented May 26, 2026 •

edited by khaeru

Loading

Verification (on a fresh `baseline_BMT` clone)