Avoid Zarr-to-HDF5 export deadlock and make the IO finalizer non-blocking

[rly]

Two related changes to remove a deadlock when exporting a Zarr v3 file to HDF5. Context: hdmf-dev/hdmf-zarr#362.

Materialize before the HDF5 write. HDF5IO.__list_fill__/__scalar_fill__ read a zarr array into memory before create_dataset/dset[:] =, so the read does not run while h5py's global lock is held. zarr v3 dispatches reads to a background event-loop thread; reading under that lock can deadlock against a GC finalizer that acquires the same lock on that thread. Scoped to zarr arrays via the existing is_zarr_array helper; RAM-neutral (h5py already materializes on assignment).

Non-blocking IO finalizer. HDMFIO.__del__ no longer runs a lock-acquiring close() (finalizers run at unpredictable times on unpredictable threads, so a blocking close can deadlock or hit shutdown-ordering issues). It now emits a ResourceWarning for an unclosed IO, as CPython does for files; still-open IOs are flushed and closed by an atexit handler on the main thread.

The materialize change alone resolves the deadlock; the finalizer change removes the unsafe blocking-close-in-a-finalizer pattern for the whole IO class.

Verified against a concurrent Zarr→HDF5 export reproducer (10/10 processes deadlock without the materialize change, 0/10 with it). Adds regression tests for both changes. No new failures in the existing suite (pre-existing zarr-v3 API failures in TestWriteHDF5withZarrInput etc. are unrelated).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.15%. Comparing base (f1b2eb4) to head (98d5f06).

Files with missing lines	Patch %	Lines
src/hdmf/backends/hdf5/h5tools.py	50.00%	1 Missing and 1 partial ⚠️
src/hdmf/backends/io.py	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1547      +/-   ##
==========================================
- Coverage   93.22%   93.15%   -0.08%     
==========================================
  Files          41       41              
  Lines       10221    10245      +24     
  Branches     2115     2119       +4     
==========================================
+ Hits         9529     9544      +15     
- Misses        415      422       +7     
- Partials      277      279       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR addresses a Zarr→HDF5 export deadlock by ensuring Zarr arrays are materialized before HDF5 writes, and makes HDMFIO finalization non-blocking by warning on unclosed IOs and deferring best-effort cleanup to an atexit handler.

Changes:

• Materialize Zarr arrays (np.asarray) before create_dataset / dset[:] = to avoid reads occurring while the HDF5 global lock is held.
• Replace HDMFIO.__del__ blocking close() with a ResourceWarning and add an atexit cleanup hook that closes still-tracked IO instances.
• Add regression tests covering the new finalizer behavior and atexit cleanup registration.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
tests/unit/test_io_hdf5_h5tools.py	Adds tests for HDMFIO.__del__ warning behavior and atexit cleanup registration.
src/hdmf/backends/io.py	Introduces WeakSet tracking of open IOs, atexit cleanup, and changes __del__ to warn instead of closing.
src/hdmf/backends/hdf5/h5tools.py	Materializes Zarr arrays before HDF5 dataset creation/assignment to avoid deadlocks.
CHANGELOG.md	Documents the finalizer/atexit behavior change and the Zarr→HDF5 deadlock fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.