Windows ros3 test jobs hang on process exit (since 4.0.0)

[rly]

The scheduled Run all tests workflow has two Windows ros3 jobs that hang and get cancelled at the 6-hour job limit:

• conda-windows-python3.14-ros3 (python test.py --ros3)
• conda-windows-gallery-python3.14-ros3 (python test.py --example-ros3)

The ros3 tests themselves pass. The suite logs its final Ran N tests - OK and then the Python process never exits, so the job runs until GitHub cancels it ~6 hours later. The equivalent Linux and macOS ros3 jobs on Python 3.14 pass in ~2 minutes, so this is Windows-specific.

This started with the 4.0.0 release.

Example run: https://github.com/NeurodataWithoutBorders/pynwb/actions/runs/29983760208

[rly]

Root cause

The ros3 tests pass; the process then deadlocks during interpreter/process shutdown. A native stack captured from a minidump of the hung Windows process (procdump + cdb) shows the main thread stuck here:

ntdll!NtWaitForAlertByThreadId
ntdll!RtlSleepConditionVariableSRW
KERNELBASE!SleepConditionVariableSRW
aws_c_common!aws_condition_variable_wait
aws_c_common!aws_condition_variable_wait_pred
aws_c_common!aws_thread_join_all_managed      <- blocks joining AWS worker threads
aws_c_s3!aws_s3_library_clean_up              <- AWS-C-S3 library teardown
hdf5!...s3comms...  (ros3 S3 VFD cleanup)
ucrtbase!execute_onexit_table                 <- hdf5.dll CRT atexit table
ntdll!LdrpCallInitRoutine -> ntdll!LdrShutdownProcess   <- DLL_PROCESS_DETACH (loader lock held)
ntdll!RtlExitUserProcess -> kernel32!ExitProcess -> python314!Py_Exit -> Py_RunMain

During ExitProcess, the Windows loader runs DLL_PROCESS_DETACH for hdf5.dll while holding the loader lock. HDF5's ros3 S3 VFD has registered a CRT atexit cleanup that calls aws_s3_library_clean_up() -> aws_thread_join_all_managed(), which blocks forever waiting to join AWS background threads that cannot complete while the loader lock is held. This is a loader-lock deadlock: threads must never be joined inside DllMain/detach.

(The hdf5!... frame symbols are approximate because the conda-forge hdf5.dll ships without PDBs, but the aws_c_s3/aws_c_common and ntdll!LdrShutdownProcess frames are exported and unambiguous.)

Why it started with 4.0.0

4.0.0 moved the ros3 stack to h5py 3.16.0 / HDF5 2.1.0 (#2201). HDF5 2.x's ros3 VFD is backed by the AWS C SDK (aws-c-s3 / aws-c-common), which spawns managed worker threads and registers the join-on-exit cleanup. The pre-4.0.0 libcurl-based ros3 VFD spawned no such threads, so exit was clean. Python 3.14 is incidental; the matrix simply runs ros3 only on 3.14.

Why it is Windows-only

Linux and macOS join threads at exit without the Windows loader-lock-during-detach constraint. Their ros3 jobs finish in ~2 minutes.

Additional observations

• threading.enumerate() shows no extra Python threads at shutdown; the stuck threads are native AWS threads.
• os._exit() does not help: on Windows it calls ExitProcess, which still runs DLL_PROCESS_DETACH and hits the same deadlock.
• Terminating the process from an external parent process does work (it skips the exiting process's loader teardown).

This is fundamentally an HDF5 + aws-c-s3 defect (joining threads during DLL detach), not a pynwb bug; pynwb only triggers it by exercising the ros3 driver.

Mitigation (CI)

The ros3 CI jobs now run through run_ros3_hang_safe.py, which runs test.py as a child, and once the child reports its exit code but fails to exit, terminates it from the parent and propagates the reported code. The ros3 jobs also get a timeout-minutes backstop. A fix belongs upstream in HDF5 (do not join AWS threads during process-exit DLL detach on Windows).

[rly]

Update: filed upstream at HDFGroup/hdf5#6560. The fix belongs in HDF5, which must not join AWS worker threads during exit-time DLL_PROCESS_DETACH. Also linked the existing symptom report at h5py/h5py#2850.

CI mitigation implemented and verified green on windows-latest: the ros3 jobs now run through run_ros3_hang_safe.py, which runs test.py as a child process and terminates it from the parent once it reports its exit code but hangs in shutdown, plus a timeout-minutes backstop on the ros3 jobs. This keeps CI green while the upstream deadlock is unresolved.