Zarr v2 backward compatibility: read v2 NWB files with zarr-python v3

[alejoe91]

Motivation

zarr-python v3 cannot parse certain zarr v2 array metadata:

• Object-dtype arrays (|O with pickle, json2, vlen-utf8 codecs) raise ValueError
• Datetime/bytes arrays with fill_value: 0 raise TypeError (expects a string)
• Consolidated metadata containing any of the above causes the entire file open to fail

This breaks reading existing zarr v2 NWB files written by previous versions of hdmf-zarr.

Disclaimer

Most of it was generated with agents

TODO:

• carefully check implementation details
• fix HDMF-dev tests

Solution

Instead of skipping unreadable arrays, we now read them via a raw store fallback — decoding .zarray metadata and chunk bytes directly through numcodecs, bypassing zarr v3's metadata parser entirely.

Core changes (src/hdmf_zarr/backend.py)

• __read_v2_dataset_from_fs() — Reads .zarray + .zattrs + raw chunk data from any store, decodes via numcodecs, returns a DatasetBuilder
• __decode_v2_dataset() / __decode_v2_chunk() — Handle decompression + filter application for both object and regular dtypes
• __read_store_bytes() / __store_key_exists() / __store_list_dir() — Store-agnostic helpers that work with both LocalStore and FsspecStore
• __iter_children() — Unified local/remote iteration via store listing + individual access; falls back to v2 raw reading on failure
• __open_file() / load_namespaces() — Catch (TypeError, ValueError) from consolidated metadata and fall back to use_consolidated=False

Remote store support (src/hdmf_zarr/utils.py)

• ZarrSpecReader.__read_v2_object_array() — Extended to support FsspecStore via async store.get()

Backward compatibility test

• tests/unit/helpers/generate_v2_nwb_zarr.py — Generates a representative v2 NWB file (electrodes with pickle-encoded group column, datetime fields, numerical TimeSeries) using hdmf-zarr<1.0 + zarr<3
• tests/unit/test_v2_backward_compat.py — 25 tests validating metadata, datetime fields, object-dtype columns, references, and numerical data
• .github/workflows/test_backward_compat.yml — CI workflow: generates v2 file in a temp venv with old deps, then reads it with current code

Other fixes

• tests/unit/test_fsspec_streaming.py — Fixed mode="-r" typo → mode="r-"

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.35741% with 99 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.80%. Comparing base (8f63d2d) to head (d31c9a9).

Files with missing lines	Patch %	Lines
src/hdmf_zarr/backend.py	63.59%	60 Missing and 19 partials ⚠️
src/hdmf_zarr/utils.py	56.52%	14 Missing and 6 partials ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           zarr-v3-migration     #340      +/-   ##
=====================================================
- Coverage              83.73%   80.80%   -2.93%     
=====================================================
  Files                      5        5              
  Lines                   1371     1615     +244     
  Branches                 274      325      +51     
=====================================================
+ Hits                    1148     1305     +157     
- Misses                   147      213      +66     
- Partials                  76       97      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[oruebel]

@alejoe91 Quick question, to what extend and for how long do we expect that Zarr V2 support will be needed? Also is V2 support required only for read of existing data or do we expect that users also need to continue to create new V2 files? If we expect that support for ZarrV2 is limited in time and for read only, then I'm wondering whether it may be simpler to have a V2 and V3 backend side-by-side vs. maintaining a lot of extra code in the V3 backend.

[alejoe91]

Hi @oruebel

We have tons of V2 data already produced on our end, so having a reader for V2 would be enough. Once we switch to V3, we won't produce V2 data anymore but we still want to read existing ones.

If we go for a V2-specific backend (which is probably the best option), I don't see why support should be limited in time. What's the argument for that? Since carry V2 is not going to change anymore, I don't think it would take a lot of effort to keep it and maintain it.

[alejoe91]

Superseeded by #349