Read zarr v2 NWB files via a dedicated NWBZarrV2IO

.github/workflows/test_backward_compat.yml

@@ -0,0 +1,54 @@
+name: Test zarr v2 backward compatibility
+on:
+  push:
+    branches:
+      - dev
+  pull_request:
+
+jobs:
+  test-v2-backward-compat:
+    name: zarr v2 → v3 backward compat
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash
+
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+
+      # --- Step 1: generate a zarr v2 NWB file with old deps ---
+      - name: Create zarr v2 environment
+        run: |
+          python -m venv /tmp/zarr_v2_env
+          /tmp/zarr_v2_env/bin/pip install --upgrade pip
+          /tmp/zarr_v2_env/bin/pip install "hdmf-zarr<1.0" "zarr>=2.18,<3" pynwb
+
+      - name: Generate zarr v2 NWB test file
+        run: |
+          /tmp/zarr_v2_env/bin/python tests/unit/helpers/generate_v2_nwb_zarr.py \
+            --nwb-output-path tests/unit/helpers/v2_test_file.nwb.zarr \
+            --expectations-output-path tests/unit/helpers/v2_expected.json
+          echo "--- generated file tree ---"
+          find tests/unit/helpers/v2_test_file.nwb.zarr -maxdepth 2 -type d | head -30
+          echo "--- expected metadata ---"
+          cat tests/unit/helpers/v2_expected.json
+
+      # --- Step 2: read the v2 file with current code + zarr v3 ---
+      - name: Install current hdmf-zarr + zarr v3
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[full]"
+          python -m pip install pytest
+          python -m pip list
+
+      - name: Run backward compatibility tests
+        run: |
+          pytest tests/unit/test_v2_backward_compat.py -v

.gitignore

@@ -151,5 +151,7 @@ _version.py
 zarr_test*.zarr
 *.nwb.zarr
 
+# Generated backward compat test artifacts
+tests/unit/helpers/v2_expected.json
 # uv
 uv.lock

CHANGELOG.md

@@ -11,6 +11,7 @@
 ### Added
 - Added `ZarrIO.generate_dataset_html` method to generate rich HTML representations of Zarr arrays for display in Jupyter notebooks. @rly [#306](https://github.com/hdmf-dev/hdmf-zarr/pull/306)
 - Added S3 streaming tutorial with consolidated metadata guidance. @rly @oruebel [#308](https://github.com/hdmf-dev/hdmf-zarr/pull/308) [#330](https://github.com/hdmf-dev/hdmf-zarr/pull/330)
+- Added `NWBZarrV2IO.export_to_v3` and the one-shot `NWBZarrV2IO.convert_to_v3` static helper to convert NWB Zarr v2 files to Zarr v3. @alejoe91 [#XXX](https://github.com/hdmf-dev/hdmf-zarr/pull/XXX)
 
 ### Fixed
 - Fixed zarr array `.info` property to display compression data ("No. bytes stored" and "Storage ratio") when using consolidated metadata stores by patching `ConsolidatedMetadataStore.getsize()` to query the underlying chunk store. @rly [#305](https://github.com/hdmf-dev/hdmf-zarr/pull/305)
@@ -19,6 +20,7 @@
 - Fixed bug where the `specifications` group and its contents were read during `ZarrIO.read_builder`. @rly [#322](https://github.com/hdmf-dev/hdmf-zarr/pull/322)
 - Fixed issue with `tox.ini` configuration. @rly [#326](https://github.com/hdmf-dev/hdmf-zarr/pull/326)
 - Fixed `ZarrIO.is_remote()` returning `False` for remote stores using consolidated metadata, which caused `resolve_ref` to mangle HTTPS URLs and fail with `PathNotFoundError` when reading links. @rly [#328](https://github.com/hdmf-dev/hdmf-zarr/pull/328)
+- Fixed `ZarrIO._copy_array` failing to export arrays compressed with Zarr v2 numcodecs (e.g. `numcodecs.Blosc`) by mapping them to their `numcodecs.zarr3` equivalents, preserving compression/filters when exporting from Zarr v2 to v3. @alejoe91 [#XXX](https://github.com/hdmf-dev/hdmf-zarr/pull/XXX)
 - Fixed `ZarrIO.resolve_ref` failing with `PathNotFoundError: nothing found at path ''` when resolving self-references (`source == "."`) over fsspec-backed stores. The function now short-circuits the `"."` case and reuses the already-open file instead of re-opening the URL as if it were an external store. Affects every non-trivial NWB Zarr file read over HTTPS, S3, GCS, or any other fsspec scheme. @h-mayorquin [#348](https://github.com/hdmf-dev/hdmf-zarr/pull/348)
 
 ### Changed

docs/source/overview.rst

@@ -30,7 +30,7 @@ Known Limitations
 -----------------
 
 - The Zarr backend is currently experimental and may still change.
-- **Zarr v3 only:** This version of hdmf-zarr writes zarr v3 format only and cannot read files written in zarr v2 format. To read zarr v2 files, use hdmf-zarr 0.x with zarr-python v2. The `zarr CLI migration tool <https://zarr.readthedocs.io/en/stable/user-guide/cli/>`_ may also be used to convert v2 files to v3 format.
+- **Zarr v3 only for write:** This version of hdmf-zarr writes zarr v3 format only. Zarr v2 folders can only be read in read-only mode with the :py:class:`~hdmf_zarr.nwb_v2.NWBZarrV2IO` class. Zarr v2 folders can be converted to v3 using the :py:meth:`~hdmf_zarr.nwb_v2.NWBZarrV2IO.convert_to_v3` function.
 - Attributes are stored as JSON documents in Zarr (using the :py:class:`~zarr.storage.LocalStore`). As such, all attributes must be JSON serializable. The :py:class:`~hdmf_zarr.backend.ZarrIO` backend attempts to cast types to JSON serializable types as much as possible.
 - Currently the :py:class:`~hdmf_zarr.backend.ZarrIO` backend supports Zarr's :py:class:`~zarr.storage.LocalStore` for local storage and :py:class:`~zarr.storage.FsspecStore` for remote read-only access. Other `Zarr stores <https://zarr.readthedocs.io/en/stable/user-guide/storage/>`_ could be added but will require proper treatment of links and references for those backends as links are not supported in Zarr (see `zarr-python issues #389 <https://github.com/zarr-developers/zarr-python/issues/389>`_).
 - **Compound dtypes with strings:** String and reference fields in compound data types are stored as fixed-length Unicode strings (``FixedLengthUTF32``) in zarr v3. The ``FixedLengthUTF32`` data type is currently not part of the Zarr V3 specification and may not be supported by other zarr implementations.

src/hdmf_zarr/__init__.py

@@ -1,6 +1,8 @@
 from .backend import ZarrIO
 from .utils import ZarrDataIO
 from .nwb import NWBZarrIO
+from .backend_zarrv2 import ZarrV2IO, is_zarr_v2_file
+from .nwb_zarrv2 import NWBZarrV2IO
 
 try:
     # see https://effigies.gitlab.io/posts/python-packaging-2023/
@@ -12,7 +14,7 @@
     __version__ = version("hdmf")
     del version
 
-__all__ = ["ZarrIO", "ZarrDataIO", "NWBZarrIO"]
+__all__ = ["ZarrIO", "ZarrDataIO", "NWBZarrIO", "ZarrV2IO", "NWBZarrV2IO", "is_zarr_v2_file"]
 
 # Duecredit definitions
 from ._due import due, BibTeX  # noqa: E402