Migrate hdmf-zarr from zarr-python v2 to v3

[bendichter]

Summary

Migrate hdmf-zarr from zarr-python v2 (zarr>=2.18.0, <3.0) to zarr-python v3 (zarr>=3.1.3), writing zarr v3 format only while maintaining the ability to read existing v2 format files.

Closes #202

Key changes

• Storage classes: Replace DirectoryStore/NestedDirectoryStore/TempStore with LocalStore; FSStore with FsspecStore
• Object codecs removed: Replace object_codec=numcodecs.Pickle()/JSON() with StringDType + JSON serialization for references, specs, and compound datasets
• zarr.copy() not implemented in v3: Add custom _copy_array() helper
• Synchronizers removed: Deprecate synchronizer parameter with warning
• API renames: require_dataset/create_dataset → require_array/create_array
• zarr v3 Array quirks: Monkey-patch __len__ (not implemented) and scalar indexing (returns 0-d ndarrays)
• StringDType handling: Eagerly load StringDType arrays as Python lists for cross-backend compatibility; decode bytes before writing
• Codec categorization: numcodecs.Shuffle is BytesBytesCodec (compressor), not ArrayArrayCodec (filter)
• LocalStore paths: Use store.root.resolve() for filesystem paths (str(LocalStore) returns URI)
• Consolidated metadata: Updated for v3 API (removed ConsolidatedMetadataStore wrapper)
• Documentation: Updated intersphinx links, zarr version references, and broken docs links
• Minimum dependency bumps: hdmf>=4.2.0, zarr>=3.1.3, numpy>=1.26.0, numcodecs>=0.14.0, Python >=3.11

Files changed

File	Changes
src/hdmf_zarr/backend.py	Core ZarrIO migration (stores, codecs, API, monkey-patches, dtype guards)
src/hdmf_zarr/utils.py	ZarrDataIO codec handling, Shuffle categorization
src/hdmf_zarr/zarr_utils.py	Remove monkey-patches, update imports
src/hdmf_zarr/nwb.py	Synchronizer deprecation passthrough
pyproject.toml	zarr>=3.1.3, hdmf>=4.2.0, remove numcodecs upper bound, Python >=3.11
requirements-min.txt	Updated minimum versions
tox.ini	Fix wheelinstall env (extras = null → empty), update Python versions
docs/source/conf.py	Intersphinx link update
docs/source/index.rst	Remove "Zarr v2" references
docs/source/storage.rst	DirectoryStore → LocalStore link
docs/source/integrating_data_stores.rst	DirectoryStore → LocalStore link
docs/gallery/plot_s3_streaming.py	Fix broken consolidated metadata docs link
tests/unit/base_tests_zarrio.py	Store class updates, v3 API
tests/unit/test_zarrio.py	Store assertions, consolidated metadata
tests/unit/test_io_convert.py	Shuffle as compressor, store updates
tests/unit/test_zarrdataio.py	Updated codec assertions
tests/unit/test_compression_info.py	Remove monkey-patch tests
tests/unit/test_fsspec_streaming.py	Skip tests (S3 files use v2 object_codec unsupported in v3)

Test plan

• All 454 unit tests pass locally (pytest tests/)
• Domain-specific roundtrip verified (ecephys, ophys, icephys, behavior, ogen, images, TimeSeries)
• CI pipeline passes on all platforms (Linux, macOS, Windows)
• Manual test reading existing v2-format NWB zarr files
• Fsspec streaming tests (skipped: S3 test files use zarr v2 object_codec)

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.00000% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.00%. Comparing base (0a537fa) to head (b3dbd30).

Files with missing lines	Patch %	Lines
src/hdmf_zarr/backend.py	75.41%	27 Missing and 17 partials ⚠️
src/hdmf_zarr/zarr_utils.py	54.16%	8 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #325      +/-   ##
==========================================
- Coverage   85.72%   84.00%   -1.72%     
==========================================
  Files           5        5              
  Lines        1261     1388     +127     
  Branches      239      278      +39     
==========================================
+ Hits         1081     1166      +85     
- Misses        123      146      +23     
- Partials       57       76      +19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[oruebel]

@bendichter thanks for taking a crack at this. I'll take a look.

Do you know if this is compatible with reading existing Zarr v2 files? I think it is fine to drop Zarr v2 support for writing, but it would be great to have a pathway to maintain read for existing v2 data.

[bendichter]

@oruebel Thanks for the review! Here's a summary of the changes in the latest commit:

Re: zarr v2 read compatibility - zarr-python v3 can transparently read zarr v2 format files as long as they don't use object_codec (pickle/JSON codecs for object dtype arrays). Files with object_codec (which hdmf-zarr v2 used for references and specs) cannot be read by zarr v3 directly - those would need a migration tool. The vast majority of simple NWB-Zarr files should read fine. I've skipped the fsspec streaming tests since the S3 test files use v2 object_codec format.

Changes in this commit addressing your review:

1. Removed synchronizer parameter entirely from ZarrIO and NWBZarrIO (not just deprecated)
2. Removed object_codec_class parameter entirely from ZarrIO (same logic)
3. Removed _numcodec_to_zarr_v3 compat wrapper - ZarrDataIO now accepts only zarr v3 codecs (BytesBytesCodec for compressors, ArrayArrayCodec for filters). Users should use zarr.codecs (e.g., zarr.codecs.BloscCodec()) or zarr.codecs.numcodecs wrappers instead of raw numcodecs objects.
4. Updated hdf5_to_zarr_filters to produce zarr v3 wrapped codecs directly
5. Restored removed comment in zarr_utils.py about orphaned containers

[oruebel]

Files with object_codec (which hdmf-zarr v2 used for references and specs) cannot be read by zarr v3 directly - those would need a migration tool. The vast majority of simple NWB-Zarr files should read fine. I

Just to clarify, does Zarr v3 fail to read 1) any v2 file that contains object_codec datasets or 2) "just" not support reading the respective datasets within the v2 file?

If it is 1) then I think that would mean that no NWB Zarr file would be readable, because of the cached spec. I think in that case we may need some form of a migration tool.

If it is 2) then I think the main problem cases are:

• spec JSON dataset
• compound dataset with variable-length strings
• compound dataset with references (which are encoded a variable-length JSONs)
  I am wondering if those cases could be handled here?

[bendichter]

what NWB datasets currently use Zarr v2?

[bendichter]

I tested this systematically. It's scenario 2 — zarr v3 fails on individual object_codec datasets, not the entire file. However, the practical impact is closer to 1, because .groups() and .arrays() iterate all children and fail if any child has object_codec:

# These work fine:
root = zarr.open_group(store, mode='r')  # opening the group succeeds
root['normal_data'][:]                    # accessing non-object arrays by name works
root['subgroup']                          # accessing subgroups by name works

# These fail:
root['spec_dataset']    # accessing an object_codec array by name fails
root.arrays()           # iterating arrays fails if ANY array has object_codec
root.groups()           # iterating groups also fails (same underlying iteration)
root.members()          # same

Since hdmf-zarr's read_builder uses .groups() and .arrays() to walk the hierarchy, and every NWB-Zarr v2 file has object_codec datasets in /specifications, no existing NWB-Zarr file can be read with this branch as-is.

We have a few options:

1. Skip for now — v2 read support is a separate concern from the v3 write migration. A migration tool or a separate read_v2 pathway could be added later.
2. Replace .arrays()/.groups() with filesystem-level listing — manually scan for .zarray/.zgroup files, catch errors on individual arrays, and handle object_codec datasets specially (read the raw JSON chunks and decode them ourselves).
3. Try-catch during iteration — wrap individual array access in try/except and log warnings for unreadable datasets.

I think option 1 is reasonable for this PR since it's already a major release. We could file a follow-up issue for v2 read compatibility. What do you think?

[oruebel]

what NWB datasets currently use Zarr v2?

@bendichter Any of the Allen Institute for Neural Dynamics (AIND) datasets should be Zarr v2. However, I am not sure if they have published those on DANDI. If there are no NWB Zarr files on DANDI, then I think the migration problem may be mainly an AIND-specific issue.

1. Skip for now — v2 read support is a separate concern

I agree that this doesn't necessarily need to be part of this PR. However, we should have some answer for the migration question for the major release.

[bendichter]

@oruebel I know AIND is one of the main groups pushing for Zarr v3, so I think from their perspective they would rather us push forward with this

[oruebel]

I know AIND is one of the main groups pushing for Zarr v3, so I think from their perspective they would rather us push forward with this

At the very least, I think it would be useful to add some discussion in the docs on:

• Zarr v2/v3 incompatibility to clarify why this is an issue
• Mention previous version of hdmf-zarr to read V2 files (and when it will stop support)
• It looks like there is a CLI tool for migrating V2 to V3. It would be useful to test whether migration with that tool works https://zarr.readthedocs.io/en/stable/user-guide/cli/ . If that works, then I think that would a good option, because it would at least provide a useful path without introducing a lot of extra burden on hdmf-zarr

[oruebel]

May be something to add to the https://hdmf-zarr.readthedocs.io/en/stable/overview.html#known-limitations

[bendichter]

Added to known limitations in 8a0195d — documented both the FixedLengthUTF32 compound dtype limitation and the zarr v3-only write format.

[oruebel]

@bendichter we should add a Zarr V3 test dataset to DANDI to test streaming read. Maybe as part of https://dandiarchive.org/dandiset/001083. We should at least file an issue for this.

In addition, as part of this PR, is there a way to check that a file is V2 when we try to access it (probably in ZarrIO.open or ZarrIO.__init__) to check that the file is V2 and if it is V2 raise an error. If reading V2 is not supported then we should raise an error as early as possible to avoid confusion.

[bendichter]

Zarr 3.1.3 doesn't support Python 3.10, so the required checks will need to be bumped to 3.11 for this PR

[yarikoptic]

note, that it seems that hdmf-zarr restriction of numcodes seems to forbid dandi-cli to come to the bright future of python 3.14:

• feat: add support for Python 3.14 in workflows and classifiers dandi/dandi-cli#1838

Is there a time line for this PR or should we look into alternative approaches on how to unblock us? (e.g. proposing hdmf-zarr to become an optional dependency, or alike)

[rly]

Is there a time line for this PR or should we look into alternative approaches on how to unblock us? (e.g. proposing hdmf-zarr to become an optional dependency, or alike)

We discussed this on Monday. NeurodataWithoutBorders/nwbinspector#698 should unblock dandi

[alejoe91]

This PR is tested against also in SpikeInterface (all green!): SpikeInterface/spikeinterface#4260

Also, here is a WIP PR in neuroconv to configure a Zarr v3 backend with new codecs and sharding: catalystneuro/neuroconv#1749

[bendichter]

Reading any non-scalar string dataset now does data = list(zarr_obj[:]). For a large 1-D text column (e.g. millions of entries) this pulls the whole dataset into a Python list on open, defeating zarr's chunked/lazy access and risking OOM. Worse, for a 2-D string dataset list(zarr_obj[:]) yields a list of row ndarrays, so a consumer doing builder.data[i, j] raises TypeError: list indices must be integers or slices, not tuple, and .shape/.dtype are gone. Consider a lazy decoding wrapper (like the reference-dataset wrappers just above) instead of full materialization.

[bendichter]

Good catch — fixed in b23a49a.

Replaced the eager data = list(zarr_obj[:]) with a lazy ZarrStringDataset wrapper (in zarr_utils.py, mirroring the reference-dataset wrappers just above), so string datasets are no longer materialized on open:

• Lazy — wrapping reads nothing; elements are decoded to native str only on access, preserving zarr's chunked access and avoiding the OOM risk on large text columns.
• 2-D correctness — data[i, j] now returns the decoded scalar (previously TypeError: list indices must be integers or slices, not tuple), and .shape/.dtype/len() are preserved.
• Export — reports an object dtype so HDMF infers utf8 on re-build; Zarr→HDF5 and Zarr→Zarr round-trips verified.

(Root cause of why the eager cast was there: for generic dtype=None specs like VectorData, HDMF's __check_edgecases doesn't recognize np.dtypes.StringDType and falls through to value.dtype.type = python str, which HDF5IO can't write. The wrapper's object dtype steers inference to utf8 instead.)

Added TestZarrStringDataset covering laziness, shape/dtype/len preservation, scalar/slice decoding, and 2-D [i, j] indexing.

[bendichter]

Correction / follow-up (b3dbd30): the lazy ZarrStringDataset wrapper I described above broke export of typed text columns (spec dtype text, e.g. electrodes.location) and failed the plot_convert_nwb_hdf5 gallery job. Root cause: HDMF's convert_dtype only recognizes zarr.Array / ndarray / StrDataset / list / DataChunkIterator for string specs, so a custom wrapper falls through to _unicode(wrapper) and raises Expected unicode or ascii string. A raw zarr StringDType array doesn't work either: its dtype kind is "T", which HDMF's generic (dtype=None) inference doesn't recognize, so it infers python str and HDF5IO can't create the dataset.

I've replaced the wrapper with decoding to a numpy object array of native str on read. That is recognized by HDMF for both typed and generic specs, preserves shape/dtype, and supports N-D data[i, j] indexing (which list(zarr_obj[:]) did not). It is still materialized on open — genuinely lazy access isn't achievable without upstream HDMF support for StringDType (or wrapping as a DataChunkIterator, which is a much larger change and unsuitable as the read representation). Left a code comment to that effect; happy to file an HDMF issue for the upstream fix if you'd like.

[bendichter]

Any other fsspec protocol (gcs://, gs://, abfs://, az://) fails the check and gets rewritten to /cwd/gcs:/bucket/file.zarr, so the subsequent FsspecStore.from_url receives a corrupted URL and the open fails or points at the wrong location. A GCS/Azure user passing storage_options cannot open their file.

[bendichter]

Fixed in ed79899. The check now detects any protocol URL generically via "://" not in path instead of an s3/http allowlist, so gcs://, gs://, abfs://, az://, and chained fsspec protocols (e.g. simplecache::s3://) are left untouched while local/relative paths are still made absolute. Added TestPathNormalization covering all these protocols.

[bendichter]

The replaced zarr.copy() streamed chunk-by-chunk. During export of a multi-GB dataset (e.g. a large ElectricalSeries), this now decompresses the whole array into one in-memory ndarray before writing, which can OOM. Iterate over chunk regions (for sl in ...: dest[sl] = source[sl]) to bound memory to a single chunk. (zarr v3's zarr.from_array(...) also does this and would copy codecs/fill_value for you.)

[bendichter]

Fixed in ed79899. _copy_array now copies one chunk region at a time (itertools.product over chunk-aligned offsets → dest[region] = source[region]) so peak memory is bounded to a single chunk instead of decompressing the whole array into one ndarray. I kept the explicit copy rather than zarr.from_array since this method already handles the object→StringDType conversion and copies compressors/filters/fill_value/attrs; the chunk loop reuses dest.chunks (== source.chunks). Added TestCopyArray covering multi-chunk numeric, object→StringDType, and scalar (0-d) copies.