diff --git a/docs/home/faq.md b/docs/home/faq.md index f108a77b..050fe603 100644 --- a/docs/home/faq.md +++ b/docs/home/faq.md @@ -6,11 +6,54 @@ See the dedicated page on [Citing](citing.md). ## How do I add a new data source? -TODO +To add a new data source to the `technologydata` package: + +1. **Create a Parser Class**: Implement a parser class that inherits from `DataParserBase` in `src/technologydata/parsers/` +2. **Register the Data Source**: Add your data source to the `DataSourceName` enumeration in `src/technologydata/parsers/data_accessor.py` +3. **Implement Version Support**: Create version-specific subdirectories under your parser directory (e.g., `v1.0/`, `v2.0/`) +4. **Place Raw Data**: Store your raw data files in `src/technologydata/parsers/raw/` +5. **Update DataAccessor**: Add your parser to the `parse()` method logic in `DataAccessor` +6. **Document Your Parser**: Create documentation in `docs/examples/` following the pattern of existing parser documentation + +For detailed contribution guidelines, see the [contributing instructions](../contributing/instructions.md). + +## How can I access data from a remote source? + +You can use the `download()` method of the `DataAccessor` class to load technology data directly from remote URLs: + +```python +from technologydata.parsers.data_accessor import DataAccessor + +accessor = DataAccessor( + data_source="dea_energy_storage", + version="v10" +) + +# Download from a remote URL +base_url = "https://example.com/data/dea_energy_storage/v10/" +dp = accessor.download(base_url) +``` + +The method will download both `technologies.json` and `sources.json` (if available) from the specified URL. ## If I use the package today, will the data change when I update the package later? -TODO +The data versioning system ensures reproducibility: + +- Each data source has explicit version identifiers (e.g., `v10`, `v0.13.4`) +- When you specify a version in `DataAccessor`, you lock to that specific dataset +- Updating the package may add new versions but will not modify existing versioned data + +Example for reproducible usage: + +```python +# Pin specific versions +accessor = DataAccessor( + data_source="dea_energy_storage", + version="v10" # Explicitly specify version +) +dp = accessor.load() +``` ## How can I support your work? diff --git a/docs/user_guide/data_accessor.md b/docs/user_guide/data_accessor.md index 4b6d328d..1e0c9f3c 100644 --- a/docs/user_guide/data_accessor.md +++ b/docs/user_guide/data_accessor.md @@ -18,9 +18,10 @@ It is designed to work with a specific directory structure where datasets are or - **Data Source Selection**: Easily specify which dataset to load using the `DataSourceName` enumeration. - **Version Management**: Load a specific version of a dataset or automatically detect and use the latest available version. - **Standardized Loading**: Loads a `DataPackage` from a versioned folder, which is expected to contain `technologies.json` and optionally `sources.json`. +- **Remote Data Download**: Download and load technology data directly from remote URLs using the `download()` method. - **Parsing Interface**: Provides a `parse()` method to run the appropriate parser for a given data source and version to generate the data files. - **Error Handling**: Raises `FileNotFoundError` if the specified data source or version directories cannot be found, and `ValueError` for unsupported sources or versions. -- **Seamless Integration**: The `load()` method returns a `DataPackage` object, ready for use with other components of the `technologydata` library. +- **Seamless Integration**: The `load()` and `download()` methods return a `DataPackage` object, ready for use with other components of the `technologydata` library. - **Path Safety Helper**: Provides `ensure_path_exists()` to create missing directories (including parents) safely. ## Usage Examples @@ -61,6 +62,37 @@ print(type(dp_v1)) # ``` +### Downloading Data from Remote URLs + +The `download()` method enables you to download and load technology data directly from a remote URL. This is useful when you want to access datasets hosted on external servers without manually downloading files. + +```python +from technologydata.parsers.data_accessor import DataAccessor + +# Create an accessor for remote data +remote_accessor = DataAccessor( + data_source="dea_energy_storage", + version="v10" +) + +# Download and load data from a remote URL +base_url = "https://example.com/data/dea_energy_storage/v10/" +dp_remote = remote_accessor.download(base_url) + +# dp_remote is now a DataPackage object containing the downloaded data +print(type(dp_remote)) +# +``` + +The `download()` method will: + +1. Download `technologies.json` from the specified base URL +2. Attempt to download `sources.json` (optional - if not found, sources will be extracted from technologies) +3. Save the files to the configured data path +4. Load and return a `DataPackage` object + +**Note**: The base URL should point to the directory containing the JSON files. The method will automatically append the file names to construct the full URLs. + ### Parsing Raw Data The `parse()` method is used to execute the data processing pipeline for a specific data source and version. It takes the raw data file as input and generates the structured `technologies.json` and `sources.json` files. diff --git a/src/technologydata/parsers/data_accessor.py b/src/technologydata/parsers/data_accessor.py index fe71a5df..a15ce5d3 100644 --- a/src/technologydata/parsers/data_accessor.py +++ b/src/technologydata/parsers/data_accessor.py @@ -12,6 +12,7 @@ from typing import Annotated import pydantic +import requests from packaging.version import parse from pydantic import Field, field_validator @@ -37,7 +38,8 @@ class DataAccessor(pydantic.BaseModel): This class provides a standardized interface to locate and load technology datasets from predefined data sources. It can either load a specific version - or automatically determine and load the latest available version. + from the local storage, automatically determining and loading the latest available + version, or download data from a remote URL. Attributes ---------- @@ -172,6 +174,102 @@ def load(self) -> DataPackage: dp = DataPackage.from_json(self.data_source, self.version, data_path) return dp + def download(self, base_url: str) -> DataPackage: + """ + Download and load technology data from a remote URL. + + This method downloads technologies.json and sources.json files from the + specified base URL and loads them into a DataPackage instance. The sources.json + file is optional; if not found, sources will be extracted from technologies. + + Parameters + ---------- + base_url : str + Base URL where the JSON files are hosted. The method will attempt to download + technologies.json and sources.json from this location. The URL should point + to the directory containing these files. + + Returns + ------- + DataPackage + An instance of DataPackage initialized with the downloaded data. + + Raises + ------ + requests.HTTPError + If the HTTP request to download technologies.json fails. + requests.ConnectionError + If there is a network connectivity issue. + + Examples + -------- + >>> accessor = DataAccessor(data_source="dea_energy_storage", version="v1.0") + >>> dp = accessor.download("https://example.com/data") + + """ + # Ensure base_url ends with / + if not base_url.endswith("/"): + base_url += "/" + + # Download technologies.json + technologies_url = f"{base_url}technologies.json" + sources_url = f"{base_url}sources.json" + if self.version: + technologies_path = pathlib.Path( + self.data_path, self.data_source, self.version, "technologies.json" + ) + sources_path = pathlib.Path( + self.data_path, self.data_source, self.version, "sources.json" + ) + else: + technologies_path = pathlib.Path( + self.data_path, self.data_source, "technologies.json" + ) + sources_path = pathlib.Path( + self.data_path, self.data_source, "sources.json" + ) + + # Ensure parent directories exist + self.ensure_path_exists(technologies_path.parent) + self.ensure_path_exists(sources_path.parent) + + try: + logger.info(f"Downloading technologies.json from {technologies_url}") + response = requests.get(technologies_url, timeout=30) + response.raise_for_status() + + with open(technologies_path, "w", encoding="utf-8") as f: + f.write(response.text) + except requests.HTTPError as e: + logger.error(f"Failed to download technologies.json: {e}") + raise + + try: + logger.info(f"Downloading sources.json from {sources_url}") + response = requests.get(sources_url, timeout=30) + response.raise_for_status() + + with open(sources_path, "w", encoding="utf-8") as f: + f.write(response.text) + except requests.HTTPError as e: + logger.error(f"Failed to download sources.json: {e}") + raise + + # Load using DataPackage.from_json + # Construct path to folder containing the downloaded files + if self.version: + path_to_folder = pathlib.Path( + self.data_path, self.data_source, self.version + ) + else: + path_to_folder = pathlib.Path(self.data_path, self.data_source) + + data_package = DataPackage.from_json( + name=self.data_source, version=self.version, path_to_folder=path_to_folder + ) + + return data_package + def parse( self, input_file_name: str, diff --git a/test/conftest.py b/test/conftest.py index 07d6ba9f..df8f1eb4 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -14,8 +14,11 @@ reusability and organization. """ +import json import pathlib import sys +from collections.abc import Callable +from typing import Any import pytest @@ -161,3 +164,14 @@ def example_parameter(request: pytest.FixtureRequest) -> technologydata.Paramete note=request.param.get("parameter_note"), sources=technologydata.SourceCollection(sources=source_list), ) + + +@pytest.fixture(scope="function") # type: ignore +def load_json() -> Callable[[pathlib.Path], Any]: + """Fixture to load json files to unit tests.""" + + def _load(filepath: pathlib.Path) -> Any: + with open(filepath) as f: + return json.load(f) + + return _load diff --git a/test/test_parser_data_accessor.py b/test/test_parser_data_accessor.py index fd6af51f..d5e90dfc 100644 --- a/test/test_parser_data_accessor.py +++ b/test/test_parser_data_accessor.py @@ -5,11 +5,15 @@ """Test the DataAccessor class.""" import pathlib +from collections.abc import Callable +from typing import Any import pytest from technologydata.parsers.data_accessor import DataAccessor, DataSourceName +path_cwd = pathlib.Path.cwd() + class TestDataAccessor: """Test suite for the DataAccessor class in the technologydata module.""" @@ -112,3 +116,41 @@ def test_ensure_path_exists_is_idempotent(self, tmp_path: pathlib.Path) -> None: DataAccessor.ensure_path_exists(target) assert target.exists() assert target.is_dir() + + def test_download( + self, load_json: Callable[[pathlib.Path], Any], tmp_path: pathlib.Path + ) -> None: + """Test downloading a DataPackage from URL by mocking HTTP requests.""" + base_url = ( + "https://raw.githubusercontent.com/open-energy-transition/technology-data/" + ) + branch = "prototype-2/" + data_source = DataSourceName.MANUAL_INPUT_USA + version = "v0.13.4" + target_url = f"src/technologydata/parsers/{data_source}/{version}/" + url = base_url + branch + target_url + data_accessor = DataAccessor( + data_source="manual_input_usa", version="v0.13.4", data_path=tmp_path + ) + dp = data_accessor.download(url) + assert dp is not None + assert dp.sources is not None + assert dp.technologies is not None + assert dp.name == "manual_input_usa" + assert dp.version == "v0.13.4" + assert len(dp.sources) == 1 + assert len(dp.technologies) == 85 + sources_reference_path = pathlib.Path(path_cwd, target_url, "sources.json") + technologies_reference_path = pathlib.Path( + path_cwd, target_url, "technologies.json" + ) + sources_reference = load_json(sources_reference_path) + technologies_reference = load_json(technologies_reference_path) + sources_download = load_json( + pathlib.Path(tmp_path, data_source, version, "sources.json") + ) + technologies_download = load_json( + pathlib.Path(tmp_path, data_source, version, "technologies.json") + ) + assert sources_reference == sources_download + assert technologies_reference == technologies_download