Fix Zarr-to-HDF5 export deadlock - #363
Conversation
Patch zarr.Array.__array__ so numpy coercion of a zarr Array reads on the calling thread. During HDF5IO.export, h5py copies the array while holding its global "phil" lock; zarr's default read blocks on the shared "zarr_io" loop thread, and a cyclic-GC finalizer that acquires "phil" on that thread then deadlocks against the caller. Reading on the calling thread keeps the loop thread out of the read, so the lock cycle cannot form. Fixes #362 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## zarr-v3-migration #363 +/- ##
=====================================================
+ Coverage 84.00% 84.10% +0.09%
=====================================================
Files 5 5
Lines 1388 1403 +15
Branches 278 279 +1
=====================================================
+ Hits 1166 1180 +14
- Misses 146 147 +1
Partials 76 76 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Superseding this in favor of hdmf-dev/hdmf#1547, which fixes the deadlock in the right place. This PR patched Closing here; the fix will land via the next HDMF release (bump the |
Fixes #362.
Patches
zarr.Array.__array__so numpy coercion of a zarr Array reads on the calling thread. DuringHDF5IO.export, h5py copies the array while holding its globalphillock; zarr's default read blocks on the sharedzarr_ioloop thread, and a cyclic-GC finalizer that acquiresphilon that thread deadlocks against the caller. Reading on the calling thread keeps the loop thread out of the read, so the lock cycle cannot form.Verified: an end-to-end ZarrIO→HDF5 export reproducer deadlocks 10/10 processes without this change and 0/10 with it. Adds a deterministic (non-racy) regression test.
🤖 Generated with Claude Code