Yoga meth_h2 mode-parity port + IAMC reporting renames + Hydrogen|Fossil aggregator

.gitignore

@@ -153,7 +153,6 @@ cache/
 
 # Scratch files
 *scratch*
-
 # GDX and input files
 scenario_parameters.pkl
 */data/bilateralize/gdx/*
@@ -163,3 +162,19 @@ scenario_parameters.pkl
 */data/bilateralize/*/bare_files/*
 */data/bilateralize/*/GEM/*
 /message_ix_models/project/newpathways/diagnostics/data
+
+# SSP driver data files (large)
+message_ix_models/data/ssp/*.csv.xz
+message_ix_models/data/ssp/*.csv.gz
+message_ix_models/data/ssp/*.csv.zip
+message_ix_models/data/ssp/*.txt
+
+# Hydrogen debug utilities
+message_ix_models/data/hydrogen/debug_utils/
+
+# Scenario output artifacts at repo root
+/hyway_*.xlsx
+/SSP_*.csv
+/SSP_*.xlsx
+/_*.xlsx
+/*_check.xlsx

HYDROGEN_REPORTING_CHANGES.md

@@ -0,0 +1,293 @@
+# Hydrogen Reporting Implementation - Changes Summary
+
+## Date: 2025-10-06
+
+## Changes Made
+
+### 1. Fixed YAML Configuration Files
+
+#### `h2_fgt_emi.yaml`
+- ✅ **Fixed line 70:** Removed double comma in technology list
+  - Before: `[furnace_h2_refining, fc_h2_refining, hydro_cracking_ref, , h2_i, h2_fc_I]`
+  - After: `[furnace_h2_refining, fc_h2_refining, hydro_cracking_ref, h2_i, h2_fc_I]`
+
+#### `h2_fgt_emi_aggregates.yaml`
+- ✅ **Fixed level_1:** Removed incorrect electricity production aggregate
+  - The base file defines `emi_h2_fgt_prod_elec` as a single aggregated variable (all 3 electrolyzer types)
+  - The aggregates file was trying to re-aggregate it from components that don't exist as base variables
+  - Now level_2 correctly references the existing `emi_h2_fgt_prod_elec` directly
+
+- ✅ **Fixed level_3:** Added missing methanol component
+  - Before: Industry aggregate was missing `emi_h2_fgt_dem_ind_meth`
+  - After: Includes all 6 industry subsectors (steel, cement, alu, chem, meth, other)
+
+### 2. Created Missing File
+
+#### `h2_prod_aggregates.yaml` (NEW)
+- Created complete aggregation hierarchy for hydrogen production
+- Defines 2 levels of aggregation:
+  - **Level 1:** By fuel type (Gas, Coal, Biomass, Electricity)
+  - **Level 2:** Total hydrogen production
+- All short names match those defined in `h2_prod.yaml`
+
+### 3. Updated Python Code
+
+#### Deleted: `message_ix_models/model/material/report/h2_reporting.py`
+- This file was in the wrong location (material module instead of hydrogen module)
+- It was trying to import from material reporting which uses a different Config class
+- Hydrogen reporting should be separate from material reporting
+
+#### Updated: `message_ix_models/report/hydrogen/h2_reporting.py`
+- ✅ **Added `run_h2_prod_reporting()` function** for hydrogen production reporting
+- ✅ **Updated `run_h2_reporting()` function** to include production reporting
+- Now generates 3 types of reports:
+  1. Hydrogen production
+  2. H2 fugitive emissions
+  3. LH2 fugitive emissions
+
+#### Updated: `run_h2_reporting_example.py`
+- ✅ **Fixed Reporter import**
+  - Before: `from message_ix_models.report import Reporter` (incorrect)
+  - After: `from message_ix.report import Reporter` (correct)
+
+### 4. Created Documentation
+
+#### `HYDROGEN_REPORTING_GUIDE.md` (NEW)
+- Comprehensive guide covering:
+  - File structure and organization
+  - How configuration files work
+  - Complete technology coverage
+  - Aggregation hierarchies
+  - Usage instructions
+  - Verification and quality checks
+  - Troubleshooting common issues
+  - Maintenance procedures
+  - Best practices
+
+## Verification Checklist
+
+### ✅ Configuration Files
+
+- [ ] **No double-counting in base files**
+  - Each technology-mode-commodity-level combination appears only once
+  - No overlapping filters
+
+- [ ] **Complete coverage**
+  - All production technologies have:
+    - Entry in `h2_prod.yaml`
+    - Entry in `h2_fgt_emi.yaml` (for emissions)
+  - All demand sectors covered in `h2_fgt_emi.yaml`
+
+- [ ] **Valid aggregations**
+  - All `components` reference existing `short` names
+  - No circular references
+  - Proper hierarchy (level_1 → level_2 → level_3 → level_4)
+
+### ✅ File Structure
+
+```
+✅ data/hydrogen/reporting/
+   ✅ h2_prod.yaml                    (exists, corrected)
+   ✅ h2_prod_aggregates.yaml         (CREATED)
+   ✅ h2_fgt_emi.yaml                 (exists, corrected)
+   ✅ h2_fgt_emi_aggregates.yaml      (exists, corrected)
+   ✅ lh2_fgt_emi.yaml                (exists, unchanged)
+
+✅ report/hydrogen/
+   ✅ __init__.py                     (exists)
+   ✅ config.py                       (exists, unchanged)
+   ✅ h2_reporting.py                 (updated)
+   ✅ cli.py                          (exists, unchanged)
+
+✅ Root directory:
+   ✅ run_h2_reporting_example.py     (updated)
+   ✅ HYDROGEN_REPORTING_GUIDE.md     (CREATED)
+   ✅ HYDROGEN_REPORTING_CHANGES.md   (this file)
+```
+
+## Testing Instructions
+
+### 1. Basic Functionality Test
+
+```python
+import ixmp
+import message_ix
+from message_ix.report import Reporter
+from message_ix_models.report.hydrogen.h2_reporting import run_h2_reporting
+import pyam
+
+# Load scenario
+mp = ixmp.Platform(name="ixmp-dev")
+scenario = message_ix.Scenario(mp, model="your_model", scenario="your_scenario")
+
+# Create reporter and run
+rep = Reporter.from_scenario(scenario)
+dfs = run_h2_reporting(rep, scenario.model, scenario.scenario)
+
+# Check results
+py_df = pyam.concat(dfs)
+assert not py_df.empty, "No data generated!"
+print(f"Generated {len(py_df.variable)} variables")
+print(py_df.variable)
+```
+
+### 2. Verify Production Reporting
+
+```python
+# Check production variables exist
+prod_vars = py_df.filter(variable="Production|Hydrogen*")
+print("Production variables:")
+print(prod_vars.variable)
+
+# Should include:
+# - Production|Hydrogen (total)
+# - Production|Hydrogen|Gas (aggregate)
+# - Production|Hydrogen|Gas|SMR w/o CCS (base)
+# - Production|Hydrogen|Gas|SMR w/ CCS (base)
+# - Production|Hydrogen|Gas|Pyrolysis (base)
+# - Production|Hydrogen|Coal (aggregate)
+# - etc.
+```
+
+### 3. Verify Emission Reporting
+
+```python
+# Check emission variables exist
+emi_vars = py_df.filter(variable="Emissions|H2|Fugitive*")
+print(f"Found {len(emi_vars.variable)} emission variables")
+
+# Check for key aggregates
+assert "Emissions|H2|Fugitive|Energy|Supply|Hydrogen|Production" in emi_vars.variable
+assert "Emissions|H2|Fugitive|Energy|Demand|Industry" in emi_vars.variable
+```
+
+### 4. Verify Aggregations
+
+```python
+# Test that aggregates equal sum of components
+# Example: Total hydrogen production should equal sum of fuel types
+
+total_h2 = py_df.filter(
+    variable="Production|Hydrogen",
+    level=0  # Only the exact variable, not children
+).timeseries()
+
+h2_gas = py_df.filter(variable="Production|Hydrogen|Gas", level=0).timeseries()
+h2_coal = py_df.filter(variable="Production|Hydrogen|Coal", level=0).timeseries()
+h2_bio = py_df.filter(variable="Production|Hydrogen|Biomass", level=0).timeseries()
+h2_elec = py_df.filter(variable="Production|Hydrogen|Electricity", level=0).timeseries()
+
+calculated_total = h2_gas.add(h2_coal, fill_value=0).add(h2_bio, fill_value=0).add(h2_elec, fill_value=0)
+
+# Check they match (within small tolerance for floating point)
+diff = (total_h2 - calculated_total).abs()
+assert (diff < 0.001).all().all(), "Aggregation mismatch found!"
+print("✅ Aggregations verified correctly")
+```
+
+### 5. Run Example Script
+
+```bash
+# Edit run_h2_reporting_example.py with your platform/scenario details
+# Then run:
+python run_h2_reporting_example.py
+
+# Should produce:
+# - Console output with summary
+# - Excel file: {model}_{scenario}_h2_report.xlsx
+```
+
+## Known Issues and Notes
+
+### 1. Technology Mode Variations
+
+Some technologies in `h2_fgt_emi.yaml` have comments noting mode differences:
+- `fc_h2_steel` does not have `high_temp` mode
+- `fc_h2_aluminum` does not have `high_temp` mode
+- Other `fc_x` technologies do not have `high_temp` mode
+
+**Current solution:** Filters include both `[high_temp, low_temp]` modes. The Reporter will simply not find data for non-existent modes, which is acceptable.
+
+**Alternative solution (if issues arise):** Create separate filter entries for furnace vs. fuel cell technologies.
+
+### 2. Electrolyzer Aggregation
+
+The base file `h2_fgt_emi.yaml` aggregates all electrolyzer types into a single variable:
+```yaml
+Energy|Supply|Hydrogen|Production|Electricity:
+  filter: { technology: [h2_elec_pem, h2_elec_soe, h2_elec_alkaline], ... }
+  short: emi_h2_fgt_prod_elec
+```
+
+This means:
+- ✅ No separate emissions by electrolyzer type
+- ✅ Only total electricity-based H2 fugitive emissions
+- ℹ️ Different from production reporting, which has separate variables per type
+
+**Rationale:** Fugitive emissions from electrolysis are likely similar across types, so aggregation is acceptable.
+
+### 3. Methanol Production
+
+Methanol production using hydrogen (`meth_h2`) has multiple modes:
+- `feedstock_dac`, `feedstock_bic`, `feedstock_fic` (using different carbon sources)
+- `fuel_dac`, `fuel_bic`, `fuel_fic` (if methanol is used as fuel)
+
+The filter captures all modes to ensure complete coverage.
+
+## Integration with Existing Reporting
+
+### Separation from Material Reporting
+
+The hydrogen reporting system is **intentionally separate** from material reporting:
+
+**Reasons:**
+1. Different scope (energy commodity vs. materials)
+2. Different stakeholders and use cases
+3. Allows independent evolution
+4. Prevents complexity in material reporting
+
+**How they coexist:**
+- Hydrogen: `message_ix_models/report/hydrogen/`
+- Material: `message_ix_models/model/material/report/`
+- Shared utilities: Config class structure is similar but separate
+- Both use same Reporter infrastructure
+
+### Running Both Systems
+
+You can run both reporting systems independently:
+
+```python
+# Material reporting
+from message_ix_models.model.material.report.run_reporting import run
+material_df = run(scenario, upload_ts=False)
+
+# Hydrogen reporting  
+from message_ix_models.report.hydrogen.h2_reporting import run_h2_reporting
+hydrogen_dfs = run_h2_reporting(rep, scenario.model, scenario.scenario)
+
+# Combine if needed
+combined_df = pyam.concat([material_df] + hydrogen_dfs)
+```
+
+## Next Steps
+
+1. **Test with real scenario data** - Run on actual MESSAGEix scenario
+2. **Validate outputs** - Check that numbers make sense
+3. **Add CLI command** - Create `mix-models hydrogen report` command (optional)
+4. **Add unit tests** - Test Config loading and aggregations
+5. **Benchmark performance** - Ensure reporting runs efficiently
+
+## Questions for Review
+
+1. ❓ Should we add more detailed breakdowns (e.g., separate electrolyzer emissions)?
+2. ❓ Are there other hydrogen demand sectors missing?
+3. ❓ Should we add capacity reporting (installed capacity by technology)?
+4. ❓ Do we need investment cost reporting?
+
+## Contact
+
+For questions or issues with hydrogen reporting:
+- Check `HYDROGEN_REPORTING_GUIDE.md` for detailed documentation
+- Review this changes summary for what was modified
+- Test with example script: `run_h2_reporting_example.py`
+