Refactor MESSAGEix-Materials build by macflo8 · Pull Request #475 · iiasa/message-ix-models

macflo8 · 2026-02-10T16:36:31Z

This PR refactors the model building code of MESSAGEix-Materials and updates some parameter sets. It does not alter the model structure of MESSAGEix-Materials.

The following changes have been made:

Remove deprecated functions and CLI commands
Add type hints and docstrings to all functions
Drop .model.material.util pycountry functions in favour of .model.util.pycountry
Reduce set of parameters with year_act & year_vtg dimension by entries where: $y^A - y^V > \text{technical\_lifetime}$
Add separate input files for emission factors and relations
Convert .xlsx input files to .csv
Move all "other industry" parametrisation functions to .model.material.data_other_industry
Refactor share_constraints.py

Effective outcome:

Refactoring reduced build run time by ~35% (~6.5 vs ~4 mins) without power sector build and 75% with power sector build (~16 vs ~4 mins)
Parameter data set row count of aluminum, cement and steel models reduced by ~40%

How to review

Read the diff and note that the CI checks all pass.
Look at the materials doc in the ReadTheDocs preview build of the documentation —in particular, the entry in “What's New” and linked entries.
Ensure that changes/additions are self-documenting, i.e. that another
developer (someone like the reviewer) will be able to understand what the code
does in the future.

PR checklist

Continuous integration checks all ✅
Add or expand tests; coverage checks both ✅
Add, expand, or update documentation.
Update doc/whatsnew.

codecov · 2026-02-11T12:58:07Z

Codecov Report

❌ Patch coverage is 77.08333% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.3%. Comparing base (df6c5dc) to head (ac48376).
⚠️ Report is 232 commits behind head on main.

Files with missing lines	Patch %	Lines
...ge_ix_models/model/material/data_other_industry.py	17.3%	38 Missing ⚠️
message_ix_models/model/material/data_generic.py	30.7%	27 Missing ⚠️
...sage_ix_models/model/material/data_power_sector.py	67.9%	17 Missing ⚠️
...sage_ix_models/model/material/share_constraints.py	72.5%	14 Missing ⚠️
message_ix_models/model/material/build.py	50.0%	6 Missing ⚠️
message_ix_models/model/material/data_steel.py	94.2%	4 Missing ⚠️
message_ix_models/model/material/data_aluminum.py	97.9%	1 Missing ⚠️
message_ix_models/model/material/data_ammonia.py	50.0%	1 Missing ⚠️
message_ix_models/model/material/data_methanol.py	0.0%	1 Missing ⚠️
message_ix_models/model/material/data_petro.py	98.7%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #475     +/-   ##
=======================================
+ Coverage   69.6%   73.3%   +3.6%     
=======================================
  Files        300     301      +1     
  Lines      24771   24427    -344     
=======================================
+ Hits       17260   17912    +652     
+ Misses      7511    6515    -996

Files with missing lines	Coverage Δ
message_ix_models/model/material/cli.py	`23.5% <100.0%> (-0.5%)`	⬇️
message_ix_models/model/material/data_cement.py	`95.2% <100.0%> (-0.1%)`	⬇️
message_ix_models/model/material/data_util.py	`82.5% <100.0%> (+41.0%)`	⬆️
...ssage_ix_models/model/material/demand/math_util.py	`83.3% <ø> (ø)`
message_ix_models/model/material/util.py	`80.9% <100.0%> (+6.1%)`	⬆️
...ssage_ix_models/tests/model/material/test_build.py	`91.3% <100.0%> (ø)`
..._models/tests/model/material/test_data_aluminum.py	`100.0% <100.0%> (ø)`
...ix_models/tests/model/material/test_data_cement.py	`100.0% <ø> (ø)`
..._ix_models/tests/model/material/test_data_petro.py	`100.0% <ø> (ø)`
...els/tests/model/material/test_data_power_sector.py	`100.0% <100.0%> (ø)`
... and 12 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Replace get_pycountry_iso with .util.pycountry - Speed up region mapping function

- Reorder functions - Replace t.id with getattr(t, "id", t) - Remove deprecated functions

- Convert xlsx mappings to csv and yaml - Move deprecated files to archive - Replace loops with vectorized computations - Generate parameter data dynamically based on vintage capacity of scenario

- Add csv copy of NTNU_LCA_coefficients.xlsx sheet environmentalImpacts - Adjust parser to use csv copy

Drop in favor of .util.common.load_package_data()

Drop historical countries from aluminum statistics

khaeru

Even aside from the documented speedups, overall this looks great!

Most removed lines (-ve diff numbers) in body code streamlined and deduplicated.
Most added lines (+ve numbers) in docs, tests, and data files.

I left some stylistic comments inline, but none of these are blockers. I rebased after #432 was merged and the checks appear to all pass, so will merge now.

khaeru · 2026-03-09T10:23:24Z

+   :local:
+
+Binary/raw data files
+---------------------


The hierarchy for headings should be * = - ^ ", per here. We omit the overline at the top of a file.

khaeru · 2026-03-09T10:23:40Z

+
+The code relies on the following input files, stored in :file:`data/material/`:
+
+**Cement**


Could be a third-level heading.

khaeru · 2026-03-09T10:24:12Z

+   :local:
+
+Sets (:file:`material/set.yaml`)
+==========================================


Sphinx gives a warning if the underline is shorter/longer than the underlined text.

khaeru · 2026-03-09T10:25:10Z

+This release of MESSAGEix-Materials refactors the model building code of MESSAGEix-Materials and updates some parameter sets.
+It does **not** alter the model structure of MESSAGEix-Materials.
+
+Summary of changes:


Trailing colon isn't necessary in a headline.

khaeru · 2026-03-09T10:25:21Z

+Summary of changes:
+===================
+
+- Remove deprecated functions and CLI commands


Periods to end sentences.

khaeru · 2026-03-09T10:26:57Z

+CLI changes
+===========
+
+- The ``--power-sector`` option added to ``material-ix build`` command, to allow building the model with or without the power sector materials module.


Generally the reST/Sphinx 'program' directive can be used for full CLI commands or parts thereof (docs).

khaeru · 2026-03-09T10:28:27Z

-  title = {MESSAGEix-Materials v1.0.0: Representation of Material Flows and Stocks in an Integrated Assessment Model},
-  url = {https://egusphere.copernicus.org/preprints/2024/egusphere-2023-3035/},
-  year = {2024}
+  doi = {https://doi.org/10.5194/gmd-17-8321-2024},


In BibTeX/BibLaTeX, the contents of the 'doi' field should just be e.g. 10.5194/gmd-17-8321-2024. When the entry is formatted into an output document, the URL is constructed automatically.

The URL is not necessary when the DOI is supplied.

khaeru · 2026-03-09T10:39:23Z

+
+    # add 25% total trade bound
+    df_dem = results["demand"]
+    df_dem = df_dem.groupby("year").sum(numeric_only=True) * 0.25


Operations like this will be faster if chained.

khaeru · 2026-03-09T10:41:09Z

+        columns=lambda c: int(c) if isinstance(c, str) and c.isdigit() else c
+    )
+    data = overwrite_hydro_intensities(data)
+    data = (


data = ( pd.read_csv(...) .pipe(overwrite_hydro_intensities) .assign(technology=...) ... )

macflo8 temporarily deployed to publish February 10, 2026 16:36 — with GitHub Actions Inactive

macflo8 self-assigned this Feb 10, 2026

macflo8 added material MESSAGEix-Materials variant enh New features or functionality labels Feb 10, 2026

macflo8 temporarily deployed to publish February 10, 2026 17:19 — with GitHub Actions Inactive

macflo8 force-pushed the enh/materials-clean-up branch from 75a4f2a to f83d20d Compare February 10, 2026 18:52

macflo8 temporarily deployed to publish February 10, 2026 18:52 — with GitHub Actions Inactive

macflo8 force-pushed the enh/materials-clean-up branch from f83d20d to 6b9418b Compare February 10, 2026 20:05

macflo8 temporarily deployed to publish February 10, 2026 20:05 — with GitHub Actions Inactive

macflo8 added a commit that referenced this pull request Feb 11, 2026

Adjust test expectations for #475

3487ae2

macflo8 temporarily deployed to publish February 11, 2026 10:54 — with GitHub Actions Inactive

macflo8 temporarily deployed to publish February 11, 2026 12:05 — with GitHub Actions Inactive

macflo8 added a commit that referenced this pull request Feb 11, 2026

Adjust test expectations for #475

bdab1cc

macflo8 force-pushed the enh/materials-clean-up branch from 293cf41 to 64fca17 Compare February 11, 2026 12:34

macflo8 temporarily deployed to publish February 11, 2026 12:34 — with GitHub Actions Inactive

macflo8 added a commit that referenced this pull request Feb 11, 2026

Adjust test expectations for #475

8801e3b

macflo8 force-pushed the enh/materials-clean-up branch from 64fca17 to 10aee8c Compare February 11, 2026 14:55

macflo8 temporarily deployed to publish February 11, 2026 14:55 — with GitHub Actions Inactive

macflo8 added a commit that referenced this pull request Feb 11, 2026

Adjust test expectations for #475

81b30ac

macflo8 force-pushed the enh/materials-clean-up branch from 10aee8c to 59e5534 Compare February 11, 2026 15:59

macflo8 temporarily deployed to publish February 11, 2026 16:00 — with GitHub Actions Inactive

macflo8 force-pushed the enh/materials-clean-up branch from 59e5534 to fb59d37 Compare February 12, 2026 12:13

macflo8 temporarily deployed to publish February 12, 2026 12:13 — with GitHub Actions Inactive

macflo8 force-pushed the enh/materials-clean-up branch from fb59d37 to 4c4ec18 Compare February 17, 2026 21:14

macflo8 temporarily deployed to publish February 17, 2026 21:14 — with GitHub Actions Inactive

macflo8 added a commit that referenced this pull request Feb 18, 2026

Adjust test expectations for #475

8d925ae

macflo8 force-pushed the enh/materials-clean-up branch from 4c4ec18 to a12464b Compare February 18, 2026 13:43

macflo8 temporarily deployed to publish February 18, 2026 13:43 — with GitHub Actions Inactive

macflo8 marked this pull request as ready for review February 18, 2026 13:44

macflo8 added 23 commits March 9, 2026 09:46

Improve node mapping in materials build

f805f2e

- Replace get_pycountry_iso with .util.pycountry - Speed up region mapping function

Extend typing in .model.material.data_aluminum

827678d

Clean up materials data modules

a49a449

- Reorder functions - Replace t.id with getattr(t, "id", t) - Remove deprecated functions

Improve power sector material module

936903c

- Convert xlsx mappings to csv and yaml - Move deprecated files to archive - Replace loops with vectorized computations - Generate parameter data dynamically based on vintage capacity of scenario

Use type union operator in .model.materials

1799a33

Use csv copy for power material coefficients

8cf4e58

- Add csv copy of NTNU_LCA_coefficients.xlsx sheet environmentalImpacts - Adjust parser to use csv copy

Add missing docstrings in model.material

58b6901

Make materials power_sector build optional

7d2ba9b

Silence pandas FutureWarnings

e793220

Satisfy ruff check of .model.material

44c945b

Update documentation for Materials release

c1ad213

Satisfy mypy in .model.material

e45e813

Adjust test expectations for #475

7e5b685

Replace .material.util.read_yaml_file()

8f37527

Drop in favor of .util.common.load_package_data()

Skip test if message_ix < 3.9

d327e56

Make aluminum test pass

b4fe178

Drop historical countries from aluminum statistics

Update material build options

4f5bece

Fix bug in drop_redundant_rows()

5cd4988

Reactivate data_steel tests for GHA

7bd6fef

Extend tests for materials data modules

49286f6

Adjust gen_data_alu_trade() for pandas 3.0

9188193

Restructure doc/material

f464b79

Update materials paper citation

ac48376

khaeru force-pushed the enh/materials-clean-up branch from a12464b to ac48376 Compare March 9, 2026 08:46

khaeru temporarily deployed to publish March 9, 2026 08:46 — with GitHub Actions Inactive

khaeru approved these changes Mar 9, 2026

View reviewed changes

khaeru added this to the 2026-03 milestone Mar 9, 2026

khaeru merged commit 87d4167 into main Mar 9, 2026
25 checks passed

khaeru deleted the enh/materials-clean-up branch March 9, 2026 10:50

macflo8 mentioned this pull request Mar 9, 2026

Address review comments of PR #475 #484

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor MESSAGEix-Materials build#475

Refactor MESSAGEix-Materials build#475
khaeru merged 40 commits into
mainfrom
enh/materials-clean-up

macflo8 commented Feb 10, 2026 •

edited by khaeru

Loading

Uh oh!

codecov Bot commented Feb 11, 2026 •

edited

Loading

Uh oh!

khaeru left a comment

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

khaeru Mar 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		The code relies on the following input files, stored in :file:`data/material/`:

		Cement

Uh oh!

Conversation

macflo8 commented Feb 10, 2026 • edited by khaeru Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to review

PR checklist

Uh oh!

codecov Bot commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

khaeru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

macflo8 commented Feb 10, 2026 •

edited by khaeru

Loading

codecov Bot commented Feb 11, 2026 •

edited

Loading