Read zarr v2 NWB files via a dedicated NWBZarrV2IO

[alejoe91]

That PR baked the zarr v2 fallbacks directly into ZarrIO / NWBZarrIO — so every v3 read went through try / except / use_consolidated=False, every group walk used a chunk-decoding iterator, and ~440 lines of v2-specific code lived in backend.py. It worked but the v3 path paid the complexity tax for files it would never see.

This PR splits the two:

• ZarrIO and NWBZarrIO go back to their clean v3 shape.
• A new ZarrV2IO(ZarrIO) (read-only) and NWBZarrV2IO(ZarrV2IO) live in backend_zarrv2.py / nwb_zarrv2.py and contain all the v2 logic — consolidated-metadata fallbacks, raw-chunk decoding for |O arrays, dual path resolution for old-style references, the spec-reader fallback.
• To make this possible, ZarrIO grew a few protected hooks (_open_file, _open_file_consolidated, _iter_children, _resolve_ref_source, _make_spec_reader, _open_for_namespaces). No behavior change for v3 users; ZarrV2IO overrides them.

Usage

from hdmf_zarr import NWBZarrIO, NWBZarrV2IO

# explicit:
with NWBZarrV2IO("old_v2_file.nwb.zarr", mode="r") as io:
    nwbfile = io.read()

# read_nwb zarrv2
nwbfile = NWBZarrV2IO.read_nwb("old_v2_file.nwb.zarr")

NWBZarrV2IO is read-only by design — writes always produce v3.

The NWBZarrV2IO has helper functions to export/convert to Zarr v3:

io = NWBZarrV2IO("old_v2_file.nwb.zarr", mode="r")

io.export_to_v3(output="new_v3_file.nwb.zarr")

# or directly
NWBZarrV2IO.convert_to_v3(source_path="old_v2_file.nwb.zarr", dest_path="new_v3_file.nwb.zarr")

Examples

Updated plot_s3_streaming.py to use the NWBZarrV2IO reader.

Tests

• test_zarrv2_backward_compat.py now uses NWBZarrV2IO.
• The remote S3 v2 fixtures in test_fsspec_streaming.py are switched to NWBZarrV2IO
• Three test_zarrio.py tests that poked _ZarrIO__* mangled attributes were updated to the new protected names.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 69.98158% with 163 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.31%. Comparing base (b3dbd30) to head (2bf250b).

Files with missing lines	Patch %	Lines
src/hdmf_zarr/backend_zarrv2.py	63.10%	92 Missing and 29 partials ⚠️
src/hdmf_zarr/backend.py	80.43%	18 Missing and 9 partials ⚠️
src/hdmf_zarr/nwb_zarrv2.py	80.43%	4 Missing and 5 partials ⚠️
src/hdmf_zarr/nwb.py	70.00%	3 Missing and 3 partials ⚠️

Additional details and impacted files

@@                  Coverage Diff                  @@
##           zarr-v3-migration     #349      +/-   ##
=====================================================
- Coverage              84.00%   80.31%   -3.70%     
=====================================================
  Files                      5        7       +2     
  Lines                   1388     1854     +466     
  Branches                 278      357      +79     
=====================================================
+ Hits                    1166     1489     +323     
- Misses                   146      252     +106     
- Partials                  76      113      +37     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[alejoe91]

@bendichter @oruebel let me know what you guys think about this approach

[oruebel]

That PR baked the zarr v2 fallbacks directly into ZarrIO / NWBZarrIO — so every v3 read went through try / except / use_consolidated=False, every group walk used a chunk-decoding iterator, and ~440 lines of v2-specific code lived in backend.py. It worked but the v3 path paid the complexity tax for files it would never see.

I think the general approach of separating support for V2 files from the V3 backend makes sense. The overall code may be a bit longer right now but it'll make maintenance easier and keep the V3 code cleaner and more focused. I'll take a closer look at the actual code in the next couple of days.

[oruebel]

Overall the approach looks good. I still need to take a closer look at v2_backend.py and the tests, but added comments to the other files.

[alejoe91]

Thanks for the review @oruebel

I'll tackle them in the next days but happy to hear that the approach is fine!

[alejoe91]

Done. Added test file generated with pynwb 2.8.3 and Zarr v2.18

[oruebel]

Thanks @alejoe91 . From my end this looks good. @rly I think it would be good if you could also take a look since this is part of the larger Zarr V3 upgrade.

[d-v-b]

which raises for zarr v2 arrays
that v3 cannot parse (most commonly object-dtype arrays stored with v2-only codecs such as pickle, json2, or vlen-utf8)

vlen-utf8 should work, but more generally this is behavior I would like to fix in zarr-python. We should support gradual levels of support for arrays with unknown data types, so e.g. you could see that the array is there, but reading / writing chunks would be disallowed.

[alejoe91]

Thanks @d-v-b

I think we should keep an eye on this and defer the logic to zarr-python once it's implemented. I opened an issue on our end as a reminder: #358

[alejoe91]

@oruebel @rly can we move forward with this?

[oruebel]

@oruebel @rly can we move forward with this?

Looks fine from my end. I leave final review to @rly

[rly]

Thanks. I'm starting to review this now. I made some minor fixes:

• I renamed the test file (combat → compat)
• I fixed broken Sphinx refs in backend.py, nwb_zarrv2.py, and overview.rst.
• I corrected the backward-compat test docstring.
• I moved the unreleased Fix resolve_ref self-reference resolution over fsspec stores #348 and this PR CHANGELOG entries to Upcoming Release.
• I reflowed/formatted the new v2 backend modules with black

And I added a more informative error message if a user tries to read a Zarr v2 with the new ZarrIO/NWBZarrIO class:

Failed to read '…' with NWBZarrIO, which reads Zarr v3 files, but this path is a Zarr v2 file. Open it read-only with the NWBZarrV2IO. Or convert it to Zarr v3 with NWBZarrV2IO.convert_to_v3(source_path, dest_path).

[rly]

A Zarr to HDF5 test is timing out sporadically due to a tricky deadlocking issue between the HDF5 global lock, Python's cyclic garbage collector, and zarr v3's async→sync execution model. This should be resolved in HDMF: hdmf-dev/hdmf#1547

[rly]

I made a few more minor changes:

• Fixed the "this is a Zarr v2 file, use NWBZarrV2IO" hint, which wasn't showing up when the error came from read_builder.
• Made ZarrV2IO fall back to a non-consolidated open more reliably, and removed an unused import.
• Cleaned up the changelog (moved the Fix resolve_ref self-reference resolution over fsspec stores #348 entry to 0.13.0, added one for the zarr.Array.__iter__ patch).
• Small docstring/comment clarifications (s3 anonymous reads, the 1-D object-array assumption, link to Support lazy reading for zarr v2 arrays that zarr v3 cannot parse #356).
• Ran black on the files flagged by the pinned hook.