Issue 99 from url

docs/home/faq.md

@@ -6,11 +6,54 @@ See the dedicated page on [Citing](citing.md).
 
 ## How do I add a new data source?
 
-TODO
+To add a new data source to the `technologydata` package:
+
+1. **Create a Parser Class**: Implement a parser class that inherits from `DataParserBase` in `src/technologydata/parsers/`
+2. **Register the Data Source**: Add your data source to the `DataSourceName` enumeration in `src/technologydata/parsers/data_accessor.py`
+3. **Implement Version Support**: Create version-specific subdirectories under your parser directory (e.g., `v1.0/`, `v2.0/`)
+4. **Place Raw Data**: Store your raw data files in `src/technologydata/parsers/raw/`
+5. **Update DataAccessor**: Add your parser to the `parse()` method logic in `DataAccessor`
+6. **Document Your Parser**: Create documentation in `docs/examples/` following the pattern of existing parser documentation
+
+For detailed contribution guidelines, see the [contributing instructions](../contributing/instructions.md).
+
+## How can I access data from a remote source?
+
+You can use the `download()` method of the `DataAccessor` class to load technology data directly from remote URLs:
+
+```python
+from technologydata.parsers.data_accessor import DataAccessor
+
+accessor = DataAccessor(
+    data_source="dea_energy_storage",
+    version="v10"
+)
+
+# Download from a remote URL
+base_url = "https://example.com/data/dea_energy_storage/v10/"
+dp = accessor.download(base_url)
+```
+
+The method will download both `technologies.json` and `sources.json` (if available) from the specified URL.
 
 ## If I use the package today, will the data change when I update the package later?
 
-TODO
+The data versioning system ensures reproducibility:
+
+- Each data source has explicit version identifiers (e.g., `v10`, `v0.13.4`)
+- When you specify a version in `DataAccessor`, you lock to that specific dataset
+- Updating the package may add new versions but will not modify existing versioned data
+
+Example for reproducible usage:
+
+```python
+# Pin specific versions
+accessor = DataAccessor(
+    data_source="dea_energy_storage",
+    version="v10"  # Explicitly specify version
+)
+dp = accessor.load()
+```
 
 ## How can I support your work?
 

docs/user_guide/data_accessor.md

@@ -18,9 +18,10 @@ It is designed to work with a specific directory structure where datasets are or
 - **Data Source Selection**: Easily specify which dataset to load using the `DataSourceName` enumeration.
 - **Version Management**: Load a specific version of a dataset or automatically detect and use the latest available version.
 - **Standardized Loading**: Loads a `DataPackage` from a versioned folder, which is expected to contain `technologies.json` and optionally `sources.json`.
+- **Remote Data Download**: Download and load technology data directly from remote URLs using the `download()` method.
 - **Parsing Interface**: Provides a `parse()` method to run the appropriate parser for a given data source and version to generate the data files.
 - **Error Handling**: Raises `FileNotFoundError` if the specified data source or version directories cannot be found, and `ValueError` for unsupported sources or versions.
-- **Seamless Integration**: The `load()` method returns a `DataPackage` object, ready for use with other components of the `technologydata` library.
+- **Seamless Integration**: The `load()` and `download()` methods return a `DataPackage` object, ready for use with other components of the `technologydata` library.
 - **Path Safety Helper**: Provides `ensure_path_exists()` to create missing directories (including parents) safely.
 
 ## Usage Examples
@@ -61,6 +62,37 @@ print(type(dp_v1))
 # <class 'technologydata.datapackage.DataPackage'>
 ```
 
+### Downloading Data from Remote URLs
+
+The `download()` method enables you to download and load technology data directly from a remote URL. This is useful when you want to access datasets hosted on external servers without manually downloading files.
+
+```python
+from technologydata.parsers.data_accessor import DataAccessor
+
+# Create an accessor for remote data
+remote_accessor = DataAccessor(
+    data_source="dea_energy_storage",
+    version="v10"
+)
+
+# Download and load data from a remote URL
+base_url = "https://example.com/data/dea_energy_storage/v10/"
+dp_remote = remote_accessor.download(base_url)
+
+# dp_remote is now a DataPackage object containing the downloaded data
+print(type(dp_remote))
+# <class 'technologydata.datapackage.DataPackage'>
+```
+
+The `download()` method will:
+
+1. Download `technologies.json` from the specified base URL
+2. Attempt to download `sources.json` (optional - if not found, sources will be extracted from technologies)
+3. Save the files to the configured data path
+4. Load and return a `DataPackage` object
+
+**Note**: The base URL should point to the directory containing the JSON files. The method will automatically append the file names to construct the full URLs.
+
 ### Parsing Raw Data
 
 The `parse()` method is used to execute the data processing pipeline for a specific data source and version. It takes the raw data file as input and generates the structured `technologies.json` and `sources.json` files.

src/technologydata/parsers/data_accessor.py

@@ -12,6 +12,7 @@
 from typing import Annotated
 
 import pydantic
+import requests
 from packaging.version import parse
 from pydantic import Field, field_validator
 
@@ -37,7 +38,8 @@ class DataAccessor(pydantic.BaseModel):
 
     This class provides a standardized interface to locate and load technology
     datasets from predefined data sources. It can either load a specific version
-    or automatically determine and load the latest available version.
+    from the local storage, automatically determining and loading the latest available
+    version, or download data from a remote URL.
 
     Attributes
     ----------
@@ -172,6 +174,102 @@ def load(self) -> DataPackage:
         dp = DataPackage.from_json(self.data_source, self.version, data_path)
         return dp
 
+    def download(self, base_url: str) -> DataPackage:
+        """
+        Download and load technology data from a remote URL.
+
+        This method downloads technologies.json and sources.json files from the
+        specified base URL and loads them into a DataPackage instance. The sources.json
+        file is optional; if not found, sources will be extracted from technologies.
+
+        Parameters
+        ----------
+        base_url : str
+            Base URL where the JSON files are hosted. The method will attempt to download
+            technologies.json and sources.json from this location. The URL should point
+            to the directory containing these files.
+
+        Returns
+        -------
+        DataPackage
+            An instance of DataPackage initialized with the downloaded data.
+
+        Raises
+        ------
+        requests.HTTPError
+            If the HTTP request to download technologies.json fails.
+        requests.ConnectionError
+            If there is a network connectivity issue.
+
+        Examples
+        --------
+        >>> accessor = DataAccessor(data_source="dea_energy_storage", version="v1.0")
+        >>> dp = accessor.download("https://example.com/data")
+
+        """
+        # Ensure base_url ends with /
+        if not base_url.endswith("/"):
+            base_url += "/"
+
+        # Download technologies.json
+        technologies_url = f"{base_url}technologies.json"
+        sources_url = f"{base_url}sources.json"
+        if self.version:
+            technologies_path = pathlib.Path(
+                self.data_path, self.data_source, self.version, "technologies.json"
+            )
+            sources_path = pathlib.Path(
+                self.data_path, self.data_source, self.version, "sources.json"
+            )
+        else:
+            technologies_path = pathlib.Path(
+                self.data_path, self.data_source, "technologies.json"
+            )
+            sources_path = pathlib.Path(
+                self.data_path, self.data_source, "sources.json"
+            )
+
+        # Ensure parent directories exist
+        self.ensure_path_exists(technologies_path.parent)
+        self.ensure_path_exists(sources_path.parent)
+
+        try:
+            logger.info(f"Downloading technologies.json from {technologies_url}")
+            response = requests.get(technologies_url, timeout=30)
+            response.raise_for_status()
+
+            with open(technologies_path, "w", encoding="utf-8") as f:
+                f.write(response.text)
+        except requests.HTTPError as e:
+            logger.error(f"Failed to download technologies.json: {e}")
+            raise
+
+        try:
+            logger.info(f"Downloading sources.json from {sources_url}")
+            response = requests.get(sources_url, timeout=30)
+            response.raise_for_status()
+
+            with open(sources_path, "w", encoding="utf-8") as f:
+                f.write(response.text)
+        except requests.HTTPError as e:
+            logger.error(f"Failed to download sources.json: {e}")
+            raise
+
+        # Load using DataPackage.from_json
+        # Construct path to folder containing the downloaded files
+        if self.version:
+            path_to_folder = pathlib.Path(
+                self.data_path, self.data_source, self.version
+            )
+        else:
+            path_to_folder = pathlib.Path(self.data_path, self.data_source)
+
+        data_package = DataPackage.from_json(
+            name=self.data_source, version=self.version, path_to_folder=path_to_folder
+        )
+
+        return data_package
+
     def parse(
         self,
         input_file_name: str,

test/conftest.py

@@ -14,8 +14,11 @@
 reusability and organization.
 """
 
+import json
 import pathlib
 import sys
+from collections.abc import Callable
+from typing import Any
 
 import pytest
 
@@ -161,3 +164,14 @@ def example_parameter(request: pytest.FixtureRequest) -> technologydata.Paramete
         note=request.param.get("parameter_note"),
         sources=technologydata.SourceCollection(sources=source_list),
     )
+
+
+@pytest.fixture(scope="function")  # type: ignore
+def load_json() -> Callable[[pathlib.Path], Any]:
+    """Fixture to load json files to unit tests."""
+
+    def _load(filepath: pathlib.Path) -> Any:
+        with open(filepath) as f:
+            return json.load(f)
+
+    return _load

test/test_parser_data_accessor.py

@@ -5,11 +5,15 @@
 """Test the DataAccessor class."""
 
 import pathlib
+from collections.abc import Callable
+from typing import Any
 
 import pytest
 
 from technologydata.parsers.data_accessor import DataAccessor, DataSourceName
 
+path_cwd = pathlib.Path.cwd()
+
 
 class TestDataAccessor:
     """Test suite for the DataAccessor class in the technologydata module."""
@@ -112,3 +116,41 @@ def test_ensure_path_exists_is_idempotent(self, tmp_path: pathlib.Path) -> None:
         DataAccessor.ensure_path_exists(target)
         assert target.exists()
         assert target.is_dir()
+
+    def test_download(
+        self, load_json: Callable[[pathlib.Path], Any], tmp_path: pathlib.Path
+    ) -> None:
+        """Test downloading a DataPackage from URL by mocking HTTP requests."""
+        base_url = (
+            "https://raw.githubusercontent.com/open-energy-transition/technology-data/"
+        )
+        branch = "prototype-2/"
+        data_source = DataSourceName.MANUAL_INPUT_USA
+        version = "v0.13.4"
+        target_url = f"src/technologydata/parsers/{data_source}/{version}/"
+        url = base_url + branch + target_url
+        data_accessor = DataAccessor(
+            data_source="manual_input_usa", version="v0.13.4", data_path=tmp_path
+        )
+        dp = data_accessor.download(url)
+        assert dp is not None
+        assert dp.sources is not None
+        assert dp.technologies is not None
+        assert dp.name == "manual_input_usa"
+        assert dp.version == "v0.13.4"
+        assert len(dp.sources) == 1
+        assert len(dp.technologies) == 85
+        sources_reference_path = pathlib.Path(path_cwd, target_url, "sources.json")
+        technologies_reference_path = pathlib.Path(
+            path_cwd, target_url, "technologies.json"
+        )
+        sources_reference = load_json(sources_reference_path)
+        technologies_reference = load_json(technologies_reference_path)
+        sources_download = load_json(
+            pathlib.Path(tmp_path, data_source, version, "sources.json")
+        )
+        technologies_download = load_json(
+            pathlib.Path(tmp_path, data_source, version, "technologies.json")
+        )
+        assert sources_reference == sources_download
+        assert technologies_reference == technologies_download