Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
060d121
Add V2 backward-compatibility to Zarr v3 implementation
alejoe91 Apr 2, 2026
d31c9a9
fix ruff
alejoe91 Apr 2, 2026
1e5efad
feat: support ZarrV2 in read mode with NWBZarrV2IO
alejoe91 May 29, 2026
bbbccee
fix: ruff
alejoe91 May 29, 2026
c05d039
feat: add convert_to_v3 helper function
alejoe91 Jun 23, 2026
312eb6e
fix: conflicts
alejoe91 Jun 24, 2026
60b51c9
fix: rename v2_backend->backend_v2; docstrings; static methods
alejoe91 Jun 24, 2026
0fb9d13
fix: v2 check for remote assets
alejoe91 Jun 24, 2026
7cda419
refac: rename v2 files to zarrv2
alejoe91 Jun 25, 2026
3498950
refac: rename v2 files to zarrv2 2
alejoe91 Jun 25, 2026
6bce5f3
refac: rename v2 files to zarrv2 3
alejoe91 Jun 25, 2026
67afc56
docs: convert numpy to sphynx-style
alejoe91 Jun 25, 2026
523e9c5
fix: fallback for remote v2 data
alejoe91 Jun 25, 2026
f007f28
fix: monkey-patch zarr.Array.__iter__
alejoe91 Jun 25, 2026
393cb66
Merge branch 'zarr-v3-migration' into nwb-zarr-v2-io
alejoe91 Jun 25, 2026
5e56808
Apply suggestions from code review
alejoe91 Jun 26, 2026
c32aa25
suggestions from code review
alejoe91 Jun 26, 2026
d0eef2d
fix: zarrv2 renaming in tests
alejoe91 Jun 26, 2026
77381ed
fix: remove zarr version dispatc in NWBZarrIO.read_nwb and more docs
alejoe91 Jun 26, 2026
0675eb0
test: remove sniffer test
alejoe91 Jun 26, 2026
4908eb7
fix: codespell
alejoe91 Jun 26, 2026
0c0538f
docs: update streaming example
alejoe91 Jun 26, 2026
3743564
test: add nwb zarrv2 test files
alejoe91 Jun 30, 2026
0dc34d9
fix: bump up minimal to pynwb 3.0 and fix windows relpath
alejoe91 Jun 30, 2026
ed13100
fix: test file and expected json
alejoe91 Jun 30, 2026
ae7f535
build: bump up minimal to 3.1.0
alejoe91 Jun 30, 2026
f5b5ac9
build: revert minimal to 2.8.3 and regenerate test files
alejoe91 Jun 30, 2026
e0e325a
Fix v2 backend docs/naming and add clear error for v2-on-v3 reads
rly Jul 19, 2026
493eda4
Keep the Zarr v2 read-error message within the HDMF layer
rly Jul 19, 2026
4c13a69
Merge branch 'zarr-v3-migration' into nwb-zarr-v2-io
rly Jul 20, 2026
17378f2
Capture the intermittent py3.14 CI deadlock instead of spinning for 6…
rly Jul 22, 2026
56e269e
Revert "Capture the intermittent py3.14 CI deadlock instead of spinni…
rly Jul 22, 2026
9c14265
Document anonymous S3 read behavior in NWBZarrV2IO helpers
rly Jul 24, 2026
d32ca77
Fix and harden the Zarr v2 read-error hint on the read_builder path
rly Jul 24, 2026
3c901c9
Apply black formatting to files flagged by the pinned black hook
rly Jul 24, 2026
b9f6a4a
Add changelog entry for the zarr Array __iter__ patch
rly Jul 24, 2026
4001cf0
Point the eager-load docstring at the tracking issue #356
rly Jul 24, 2026
e7d2111
Document the 1-D assumption in the v2 object-array chunk decode
rly Jul 24, 2026
f0c963d
Restore the #348 changelog entry to the released 0.13.0 section
rly Jul 24, 2026
7236877
Clean up ZarrV2IO open fallbacks
rly Jul 24, 2026
efc70e5
Clean up tutorial text
rly Jul 24, 2026
3c5bb28
Correct consolidated-metadata and pynwb read references in the S3 tut…
rly Jul 24, 2026
2bf250b
Correct zarr.copy comment and align v2 legacy-version references to 0.14
rly Jul 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,14 @@

## Upcoming Release (TBD)

### Added
- Added read-only `ZarrV2IO` and `NWBZarrV2IO` backends for reading legacy Zarr v2 files, with `NWBZarrV2IO.export_to_v3` and the one-shot `NWBZarrV2IO.convert_to_v3` static helper to convert NWB Zarr v2 files to Zarr v3. @alejoe91 [#349](https://github.com/hdmf-dev/hdmf-zarr/pull/349)
- Added a clear error when reading a Zarr v2 file with the Zarr v3 `ZarrIO`/`NWBZarrIO` that directs the user to `ZarrV2IO`/`NWBZarrV2IO`, replacing an opaque zarr-python parse error. @alejoe91 [#349](https://github.com/hdmf-dev/hdmf-zarr/pull/349)

### Fixed
- Fixed bug where `ZarrIO.generate_dataset_html` would raise an error when called with a non-Zarr object. @oruebel [#355](https://github.com/hdmf-dev/hdmf-zarr/pull/355)
- Fixed `ZarrIO._copy_array` failing to export arrays compressed with Zarr v2 numcodecs (e.g. `numcodecs.Blosc`) by mapping them to their `numcodecs.zarr3` equivalents, preserving compression/filters when exporting from Zarr v2 to v3. @alejoe91 [#349](https://github.com/hdmf-dev/hdmf-zarr/pull/349)
- Fixed lazily-loaded Zarr arrays not being recognized as `collections.abc.Iterable`, which caused hdmf/pynwb type checks (e.g. `ImageSeries.dimension`) to reject them. `zarr.Array` gains an `__iter__` that yields elements lazily via `__getitem__`. This applies to arrays from both Zarr v2 and Zarr v3 files, since both are read through zarr-python v3. @alejoe91 [#349](https://github.com/hdmf-dev/hdmf-zarr/pull/349)


## 0.13.0 (June 22, 2026)
Expand Down
43 changes: 28 additions & 15 deletions docs/gallery/plot_s3_streaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,23 +31,33 @@
# Streaming from a Public S3 Bucket
# ----------------------------------
#
# To read an NWB Zarr file from a public S3 bucket, you can provide the S3 URL
# to :py:class:`~hdmf_zarr.nwb.NWBZarrIO`. For HTTPS URLs (``https://``), no
# additional configuration is needed. For ``s3://`` protocol URLs, you need to
# specify ``storage_options=dict(anon=True)`` to enable anonymous access.
# To read an NWB Zarr file from a public S3 bucket, you provide the S3 URL to the
# appropriate IO class. Files written with current hdmf-zarr use Zarr v3 and are
# read with :py:class:`~hdmf_zarr.nwb.NWBZarrIO`, while older files that
# use Zarr v2 must be read with the read-only
# :py:class:`~hdmf_zarr.nwb_zarrv2.NWBZarrV2IO`. You can detect which format a file
# uses with :py:func:`~hdmf_zarr.backend_zarrv2.is_zarr_v2_file` and pick the right
# class accordingly.
#
# For HTTPS URLs (``https://``), no additional configuration is needed. For
# ``s3://`` protocol URLs, you need to specify ``storage_options=dict(anon=True)``
# to enable anonymous access.
#
# Here we demonstrate reading from a public dataset in the DANDI Archive using
# an HTTPS URL:
# an HTTPS URL. Since this file predates Zarr v3, it is a Zarr v2 file:

from hdmf_zarr import NWBZarrIO
from hdmf_zarr import NWBZarrIO, NWBZarrV2IO, is_zarr_v2_file

# Public S3 URL from DANDI Archive (DANDISET 000719)
# Path: sub-R6_ses-20200206T210000_behavior+ophys_DirectoryStore_rechunked.nwb.zarr
s3_url = "https://dandiarchive.s3.amazonaws.com/zarr/c8c6b848-fbc6-4f58-85ff-e3f2618ee983/"

# Pick the IO class based on the Zarr format of the file
io_class = NWBZarrV2IO if is_zarr_v2_file(s3_url) else NWBZarrIO

# Open the file from S3
try:
with NWBZarrIO(s3_url, mode="r") as io:
with io_class(s3_url, mode="r") as io:
nwbfile = io.read()
print(f"Session Description: {nwbfile.session_description}")
print(f"Identifier: {nwbfile.identifier}")
Expand Down Expand Up @@ -110,8 +120,9 @@
# significantly slow down file opening and data access.
#
# **Consolidated metadata** addresses this by storing all metadata in a single
# ``.zmetadata`` file at the root of the Zarr store. This helps improve read performance
# by reducing the number of S3 requests needed to open a file.
# consolidated document at the root of the Zarr store (``zarr.json`` for Zarr v3,
# ``.zmetadata`` for Zarr v2). This helps improve read performance by reducing the
# number of S3 requests needed to open a file.
#
# By default, :py:class:`~hdmf_zarr.nwb.NWBZarrIO` consolidates metadata when
# writing files, and automatically uses consolidated metadata when available
Expand Down Expand Up @@ -150,22 +161,24 @@
# Using the Convenience Method
# ----------------------------
#
# :py:class:`~hdmf_zarr.nwb.NWBZarrIO` provides a convenience static method
# :py:meth:`~hdmf_zarr.nwb.NWBZarrIO.read_nwb` for quick read access:
# Both :py:class:`~hdmf_zarr.nwb.NWBZarrIO` and
# :py:class:`~hdmf_zarr.nwb_zarrv2.NWBZarrV2IO` provide a convenience static method
# ``read_nwb`` for quick read access. Each reads its own Zarr format, so use the
# class that matches the file (here, a Zarr v2 file):

# Read file directly using the convenience static method
try:
nwbfile = NWBZarrIO.read_nwb(s3_url)
nwbfile = io_class.read_nwb(s3_url)
print(f"Session Start Time: {nwbfile.session_start_time}")
except Exception as e:
print(f"Note: Could not access S3 file (network access may be required): {e}")

###############################################################################
# .. note::
#
# PyNWB also provides a more general :py:func:`~pynwb.NWBHDF5IO.read` method that
# can automatically detect and use the appropriate IO class (HDF5 or Zarr) based
# on the file path or URL.
# PyNWB also provides a more general :py:func:`~pynwb.read_nwb` function that
# automatically detects and uses the appropriate IO class (HDF5 or Zarr) based
# on the file.

###############################################################################
# Best Practices for S3 Streaming
Expand Down
4 changes: 2 additions & 2 deletions docs/source/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Supported features
- Links
- Object references
- Writing/loading namespaces/specifications
- Iterative data write using :py:class:`~hdmf.data_utils.AbstractDataChunkIterator`
- Iterative data write using :py:class:`~hdmf.data_utils.AbstractDataChunkIterator`
- Parallel write with :py:class:`~hdmf.data_utils.GenericDataChunkIterator` (since v0.4)
- Lazy load of datasets
- Lazy load of datasets containing object references (since v0.4)
Expand All @@ -30,7 +30,7 @@ Known Limitations
-----------------

- The Zarr backend is currently experimental and may still change.
- **Zarr v3 only:** This version of hdmf-zarr writes zarr v3 format only and cannot read files written in zarr v2 format. To read zarr v2 files, use hdmf-zarr 0.x with zarr-python v2. The `zarr CLI migration tool <https://zarr.readthedocs.io/en/stable/user-guide/cli/>`_ may also be used to convert v2 files to v3 format.
- **Zarr v2 read-only:** Zarr v2 folders can only be read in read-only mode with the :py:class:`~hdmf_zarr.backend_zarrv2.ZarrV2IO` and corresponding :py:class:`~hdmf_zarr.nwb_zarrv2.NWBZarrV2IO` classes. Writing of new Zarr v2 files is not supported, i.e., new files can only be written in Zarr v3 via :py:class:`~hdmf_zarr.backend.ZarrIO` and :py:class:`~hdmf_zarr.nwb.NWBZarrIO`. Zarr v2 folders can be converted to v3 using the :py:meth:`~hdmf_zarr.nwb_zarrv2.NWBZarrV2IO.convert_to_v3` function. (since v0.14)
- Attributes are stored as JSON documents in Zarr (using the :py:class:`~zarr.storage.LocalStore`). As such, all attributes must be JSON serializable. The :py:class:`~hdmf_zarr.backend.ZarrIO` backend attempts to cast types to JSON serializable types as much as possible.
- Currently the :py:class:`~hdmf_zarr.backend.ZarrIO` backend supports Zarr's :py:class:`~zarr.storage.LocalStore` for local storage and :py:class:`~zarr.storage.FsspecStore` for remote read-only access. Other `Zarr stores <https://zarr.readthedocs.io/en/stable/user-guide/storage/>`_ could be added but will require proper treatment of links and references for those backends as links are not supported in Zarr (see `zarr-python issues #389 <https://github.com/zarr-developers/zarr-python/issues/389>`_).
- **Compound dtypes with strings:** String and reference fields in compound data types are stored as fixed-length Unicode strings (``FixedLengthUTF32``) in zarr v3. The ``FixedLengthUTF32`` data type is currently not part of the Zarr V3 specification and may not be supported by other zarr implementations.
Expand Down
4 changes: 3 additions & 1 deletion src/hdmf_zarr/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
from .backend import ZarrIO
from .utils import ZarrDataIO
from .nwb import NWBZarrIO
from .backend_zarrv2 import ZarrV2IO, is_zarr_v2_file
from .nwb_zarrv2 import NWBZarrV2IO

try:
# see https://effigies.gitlab.io/posts/python-packaging-2023/
Expand All @@ -12,7 +14,7 @@
__version__ = version("hdmf")
del version

__all__ = ["ZarrIO", "ZarrDataIO", "NWBZarrIO"]
__all__ = ["ZarrIO", "ZarrDataIO", "NWBZarrIO", "ZarrV2IO", "NWBZarrV2IO", "is_zarr_v2_file"]
Comment thread
oruebel marked this conversation as resolved.

# Duecredit definitions
from ._due import due, BibTeX # noqa: E402
Expand Down
Loading
Loading