Prototest manual input USA

MANIFEST.in

@@ -1,6 +1,7 @@
 include src/technologydata/utils/*.txt
-include src/technologydata/package_data/dea_energy_storage/*.json
-include src/technologydata/package_data/raw/*
+include src/technologydata/parsers/dea_energy_storage/*.json
+include src/technologydata/parsers/manual_input_usa/*.json
+include src/technologydata/parsers/raw/*
 include test/test_data/currency_conversion/WB_CNY_2020/*
 include test/test_data/currency_conversion/WB_EUR_2020/*
 include test/test_data/currency_conversion/WB_USD_2020/*

REUSE.toml

@@ -8,22 +8,21 @@ path = ["uv.lock", "*.yaml", "docs/**", "*.md", "*.in"]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "MIT"
 
-
 [[annotations]]
-path = ["test/test_data/**",]
+path = ["test/test_data/**"]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "CC-BY-4.0"
 
 [[annotations]]
 path = [
-    "src/technologydata/package_data/dea_energy_storage/*.json", "src/technologydata/package_data/schemas/*.json"
+    "src/technologydata/parsers/dea_energy_storage/*.json", "src/technologydata/parsers/schemas/*.json", "src/technologydata/parsers/raw/manual_input_usa.csv", "src/technologydata/parsers/manual_input_usa/*.json",
 ]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "CC-BY-4.0"
 
 [[annotations]]
 path = [
-    "src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx"
+    "src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx"
 ]
 SPDX-FileCopyrightText = "The Danish Energy Agency"
 SPDX-License-Identifier = "CC-BY-4.0"

docs/examples/dea_storage.md

@@ -2,11 +2,11 @@
 
 ## Overview
 
-The Danish Energy Agency (DEA) data parser `dea_energy_storage.py` demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py`.
+The Danish Energy Agency (DEA) data parser `dea_energy_storage.py` demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py`.
 
 ## Dataset Description
 
-The original dataset is available at this [link](https://ens.dk/media/6589/download). A full description of the dataset is available at this [link](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx`.
+The original dataset is available at this [link](https://ens.dk/media/6589/download). A full description of the dataset is available at this [link](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`.
 
 The dataset is in Excel format, and it includes, under the data sheet `alldata_flat`, a flat table of technology parameters for a range of energy storage technologies. Columns include `Technology`, `ws`, `par` (parameter name), `val` (value), `unit`, `year`, `est` (case/estimate), `priceyear`, plus metadata columns such as `cat`, `ref`, `note`. Rows are individual parameter records (parameter value + unit + context) for technologies and estimation cases.
 
@@ -25,7 +25,7 @@ Function `parse_input_arguments()` defines and parses the command-line arguments
 
 ### Read the raw data
 
-The script reads the raw data available at `src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially.
+The script reads the raw data available at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially.
 
 ### Data cleaning, validation and dealing with missing/null values
 
@@ -64,21 +64,21 @@ Function `build_technology_collection()`:
 - for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `units`, `sources`, `provenance`).
 - creates a `Technology` object for each group, with `name` = `ws`, `detailed_technology` = `Technology`, `year`=`year`, `region` = `EU`, `case` = `est` and collects them into a `TechnologyCollection` object.
 - writes the `TechnologyCollection` object to a `technologies.json`.
-- if `--export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/package_data/schemas`.
+- if `--export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/parsers/schemas`.
 
 ## Running the parser
 
 ### Execution instructions
 
 From repository root:
 
-- Basic run: `python src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py`
+- Basic run: `python src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py`
 - Example with options: `--num_digits 3 --store_source --filter_params --export_schema`
 
 ### Outputs
 
 The parser generates the following outputs:
 
-- `src/technologydata/package_data/dea_energy_storage/technologies.json`.
-- `src/technologydata/package_data/dea_energy_storage/sources.json`.
-- Optional schema files moved to `src/technologydata/package_data/schemas` when `--export_schema` is used.
+- `src/technologydata/parsers/dea_energy_storage/technologies.json`.
+- `src/technologydata/parsers/dea_energy_storage/sources.json`.
+- Optional schema files moved to `src/technologydata/parsers/schemas` when `--export_schema` is used.

docs/examples/manual_input_usa.md

@@ -0,0 +1,78 @@
+# Manual Input USA Parser Documentation
+
+## Overview
+
+The Manual Input USA data parser `manual_input_usa.py` demonstrates a data-cleaning and transformation pipeline for converting manually curated, USA-specific tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/manual_input_usa/manual_input_usa.py`.
+
+## Dataset Description
+
+The original dataset is a manually curated CSV file containing USA-specific technology parameters available at this [link](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). The raw source file is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`.
+
+The dataset is in CSV format and includes a flat table of technology parameters for various energy technologies relevant to the USA context. Columns include `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case`, and `scenario`. Rows are individual parameter records (parameter value + unit + context) for technologies with different scenarios and financial cases.
+
+## Parser description
+
+The parser is articulated in the following steps.
+
+### Command line argument parsing
+
+Function `CommonsParser.parse_input_arguments()` defines and parses the command-line arguments:
+
+- `--num_digits` (int, default 4) — number of decimals used when rounding numeric values. The default value is 4.
+- `--store_source` (boolean flag) — whether to store the source on the Wayback Machine. The default value is `false`.
+
+### Read the raw data
+
+The script reads the raw data available at `src/technologydata/parsers/raw/manual_input_usa.csv` in a `pandas` dataframe. It uses `pandas.read_csv(..., dtype=str, na_values="None")`. All entries are handled as strings initially except for the `value` column which is converted to float.
+
+### Data cleaning, validation and dealing with missing/null values
+
+The data cleaning and validation happens with the following steps.
+
+Function `extract_units_carriers_heating_value()` extracts standardized units, carriers, and heating values from input unit strings. This function maps complex unit representations to simplified unit, carrier, and heating value combinations using a predefined dictionary of special patterns. Examples include:
+
+- `USD_2022/MW_FT` → unit: `USD_2022/MW`, carrier: `1/FT`, heating_value: `1/LHV`
+- `MWh_H2/MWh_FT` → unit: `MWh/MWh`, carrier: `H2/FT`, heating_value: `LHV`
+- `MWh_el/MWh_FT` → unit: `MWh/MWh`, carrier: `el/FT`, heating_value: `LHV`
+- `t_CO2/MWh_FT` → unit: `t/MWh`, carrier: `CO2/FT`, heating_value: `LHV`
+- `USD_2022/kWh_H2` → unit: `USD_2022/kWh`, carrier: `1/H2`, heating_value: `LHV`
+- `USD_2023/t_CO2/h` → unit: `USD_2023/t/h`, carrier: `1/CO2`, heating_value: `None`
+- `MWh_el/t_CO2` → unit: `MWh/t`, carrier: `el/CO2`, heating_value: `LHV`
+- `MWh_th/t_CO2` → unit: `MWh/t`, carrier: `thermal/CO2`, heating_value: `LHV`
+
+The parser also fills missing values in the `scenario` column with `"not_available"`.
+
+The parser applies the following unit conversions:
+
+- Convert `per unit` to `%` and multiply the corresponding `value` by 100.0, rounding to `num_digits` decimals.
+
+Function `Commons.update_unit_with_currency_year(unit, currency_year)` appends `currency_year` information to currency units when present. This is because `technologydata` follows the currency pattern `\b(?P<cu_iso3>[A-Z]{3})_(?P<year>\d{4})\b`, as for example `USD_2022`.
+
+### Populate and export the source and technology collections
+
+Function `build_technology_collection()`:
+
+- if `store_source` is set, constructs a `Source` object for the manual input USA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`.
+- groups the cleaned DataFrame by `scenario`, `year`, `technology`.
+- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `sources`, and optionally `carrier`, `heating_value`, `units`, `note`).
+- captures the `financial_case` value from rows within each group to combine with `scenario`.
+- creates a `case` value by combining `scenario` and `financial_case` in the format `"{scenario} - {financial_case}"` when `financial_case` is present; otherwise uses `scenario` alone.
+- creates a `Technology` object for each group, with `name` = `technology`, `detailed_technology` = `technology`, `year` = `year`, `region` = `USA`, `case` = combined case value, and collects them into a `TechnologyCollection` object.
+- writes the `TechnologyCollection` object to a `technologies.json`.
+
+## Running the parser
+
+### Execution instructions
+
+From repository root:
+
+- Basic run: `python src/technologydata/parsers/manual_input_usa/manual_input_usa.py`
+- Example with options: `--num_digits 3 --store_source`
+
+### Outputs
+
+The parser generates the following outputs:
+
+- `src/technologydata/parsers/manual_input_usa/technologies.json`.
+- `src/technologydata/parsers/manual_input_usa/sources.json`.
+- Optional schema files moved to `src/technologydata/parsers/schemas` when `--export_schema` is used.

mkdocs.yaml

@@ -32,6 +32,7 @@ nav:
 
 - Examples:
   - Danish Energy Agency Parser: examples/dea_storage.md
+  - Manual Input USA Parser: examples/manual_input_usa.md
 
 - API Reference:
   - Data Package: api/datapackage.md

src/technologydata/parsers/commons.py

@@ -0,0 +1,136 @@
+# SPDX-FileCopyrightText: technologydata contributors
+#
+# SPDX-License-Identifier: MIT
+
+"""Classes for Commons methods for the data parsers."""
+
+import argparse
+from typing import Annotated, Any
+
+import pydantic
+from pydantic import BaseModel, ConfigDict
+
+
+class ArgumentConfig(BaseModel):
+    """
+    Pydantic model for defining argument configurations.
+
+    Allows flexible configuration of command-line arguments with type checking
+    and validation.
+    """
+
+    name: Annotated[str, pydantic.Field(description="Name of the argument config")]
+    arg_type: Annotated[
+        type | None,
+        pydantic.Field(
+            description="The type to which the command-line argument should be converted."
+        ),
+    ] = None
+    default: Annotated[
+        Any | None, pydantic.Field(description="Default value of the argument config")
+    ] = None
+    help: Annotated[
+        str | None,
+        pydantic.Field(description="A brief description of what the argument does."),
+    ] = None
+    action: Annotated[
+        str | None,
+        pydantic.Field(
+            description="Specification of how the command-line arguments should be handled"
+        ),
+    ] = None
+    required: Annotated[
+        bool, pydantic.Field(description="Flag to check whether field is mondatory")
+    ] = False
+
+    # Allow extra fields for maximum flexibility
+    model_config = ConfigDict(extra="allow")
+
+
+class CommonsParser:
+    """Commons methods for the data parsers."""
+
+    @staticmethod
+    @pydantic.validate_call
+    def parse_input_arguments(
+        additional_arguments: list[ArgumentConfig] | None = None,
+        description: str = "Flexible command line argument parser",
+    ) -> argparse.Namespace:
+        """
+        Parse command line arguments with robust configuration.
+
+        Parameters
+        ----------
+        additional_arguments : Optional[List[ArgumentConfig]]
+            A list of ArgumentConfig objects defining extra arguments.
+        description : str
+            Description for the argument parser. Defaults to a generic message.
+
+        Returns
+        -------
+        argparse.Namespace
+            Parsed command line arguments
+
+        Examples
+        --------
+        >>> extra_args = [
+        ...     ArgumentConfig(
+        ...         name="--input_file",
+        ...         arg_type=str,
+        ...         required=True,
+        ...         help="Path to input CSV file"
+        ...     ),
+        ...     ArgumentConfig(
+        ...         name="--verbose",
+        ...         action="store_true",
+        ...         help="Enable verbose output"
+        ...     )
+        ... ]
+        >>> args = CommonsParser.parse_input_arguments(additional_arguments=extra_args)
+
+        """
+        # Create parser with provided or default description
+        parser = argparse.ArgumentParser(
+            description=description,
+            formatter_class=argparse.RawTextHelpFormatter,
+        )
+
+        # Default arguments
+        default_args = [
+            ArgumentConfig(
+                name="--num_digits",
+                arg_type=int,
+                default=4,
+                help="Number of significant digits to round the values.",
+            ),
+            ArgumentConfig(
+                name="--store_source",
+                action="store_true",
+                help="Store_source, store the source object on the wayback machine. Default: false",
+            ),
+        ]
+
+        # Combine default and additional arguments
+        all_arguments = default_args + (additional_arguments or [])
+
+        # Add arguments to parser (Option 1)
+        for arg_config in all_arguments:
+            # Convert Pydantic model to argparse-compatible dictionary
+            arg_dict = {
+                k: v
+                for k, v in arg_config.model_dump().items()
+                if v is not None and k != "name"
+            }
+
+            if arg_dict.get("arg_type") is not None:
+                arg_dict["type"] = arg_dict.pop("arg_type")
+
+            print("arg_dict", arg_dict)
+
+            # Add argument to parser
+            parser.add_argument(arg_config.name, **arg_dict)
+
+        # Parse arguments
+        args = parser.parse_args()
+
+        return args