Fix netcdf4/h5py dimension scales on emscripten#5951
Open
MMesch wants to merge 5 commits into
Open
Conversation
MMesch
force-pushed
the
fix-netcdf4-dimscales
branch
from
July 20, 2026 09:36
cb26276 to
f4548c4
Compare
- bump 1.12.3 -> 1.14.6, update source URL template - patches/0001-fenv-fe-invalid-guard.patch: guard FE_INVALID use in H5Tinit_float.c; emscripten's <bits/fenv.h> defines only rounding modes, FE_INVALID is not declared - patches/0002-drop-soversion.patch: skip VERSION/SOVERSION properties under EMSCRIPTEN so libhdf5_hl.so records bare "libhdf5.so" as its NEEDED entry. Otherwise it records "libhdf5.so.310.5.1" while downstream packages link with -lhdf5 and record "libhdf5.so"; the emscripten loader dedupes by string identity (not realpath) and ends up loading libhdf5.so twice with independent bss segments -- two copies of H5CX_head_g etc., which breaks the metadata cache when calls cross between the instances - variant.yaml: pin hdf5 to 1.14.6 - 1.14 auto-generates settings/ so the old workaround is dropped
- HDF5_VERSION 1.12.3 -> 1.14.6 - drop the ctypes.CDLL preload prepended to __init__.py; the hdf5 SOVERSION fix makes the emscripten loader dedupe libhdf5.so properly, so the manual dlopen is no longer needed and was masking the double-load bug during diagnosis - add version check and a dimension-scale round-trip test that guards the HL path (H5DSset_scale + attach_scale + is_scale after reopen)
netcdf-c's isdaoscontainer() in libdispatch/dinfermodel.c uses listxattr/getxattr to sniff a path for DAOS marker attributes. Emscripten's <sys/xattr.h> declares these but the runtime does not implement them, so the built .so fails to load with "cannot resolve symbol listxattr". There is no --disable-daos configure option (the check is hard-coded), so patch the source to early-return NC_ENOTNC under __EMSCRIPTEN__. Also bump the variant.yaml pin to 4.10.1 to match the recipe version; the previous 4.9.3 pin was forcing consumers to resolve against upstream 4.9.3 which had been built against HDF5 1.12.3 headers and aborted at runtime against a 1.14.6 runtime.
netcdf-4 file access walks the HDF5 tree with H5DSis_scale to distinguish coordinate dims from data variables. Without -lhdf5_hl in the extension LDFLAGS the built _netCDF4.so has no libhdf5_hl.so in its NEEDED entries, so the loader never brings it in and calls to H5DSis_scale die with "Dynamic linking error: cannot resolve symbol H5DSis_scale" the moment xarray/netCDF4 opens a NETCDF4 file. Also replace the smoke test with two focused cases: a NETCDF3_CLASSIC round-trip and a NETCDF4 round-trip that exercises dim + variable + attribute together.
CI lint requires a tests section on every recipe.
MMesch
force-pushed
the
fix-netcdf4-dimscales
branch
from
July 20, 2026 15:17
f4548c4 to
5495139
Compare
Contributor
Author
|
@IsabelParedes , waiting for your thoughts on this now. After discussing with @DerThorsten these are some points to consider:
|
Member
This would be nice. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the runtime failure
xarray.open_dataset(...)→netCDF4.Dataset→nc_open→H5DSis_scale→Dynamic linking error: cannot resolve symbol H5DSis_scalein the browser, and the follow-onRuntimeError: ring type mismatch occurred for cache entrythat surfaces onceH5DSis_scaleis reachable.Four package commits along the dependency chain:
ctypes.CDLLpreload in__init__.py.__EMSCRIPTEN__.-lhdf5_hlto LDFLAGS.variant.yamlbumped:hdf5: 1.14.6,libnetcdf: 4.10.1.<This was assisted by: Opus 4.7>
Complications
HDF5 1.14.6 uses
feclearexcept(FE_INVALID)inH5Tinit_float.c. Emscripten's<bits/fenv.h>defines only rounding modes andFE_ALL_EXCEPT=0;FE_INVALIDis not declared because WASM has no FP exception flags. Guarded with#ifdef FE_INVALID.Emscripten's dynamic loader dedupes shared libraries by string identity of the NEEDED entry, not by realpath. With SOVERSION set,
libhdf5_hl.sorecords NEEDEDlibhdf5.so.310.5.1, while-lhdf5in consumers records NEEDEDlibhdf5.so. Both paths resolve to the same file via symlinks, but the loader treats them as distinct and loads libhdf5.so twice, each with its own bss segment. That produces two copies ofH5CX_head_g(a file-static inH5CX.c) and every other libhdf5 global. When a call chain crosses between the two instances the metadata-cache "ring" invariant fires and any HL call afterH5Aexistsaborts. Confirmed by tracing patchedH5CX_set_ring/push/pop/get_ringand observing two distinct&head_gaddresses in the same process (0x2e31be0and0x38c9f40). Fix: skipset_target_properties(... VERSION SOVERSION)underEMSCRIPTENinconfig/cmake/HDF5Macros.cmakeso every consumer sees the same barelibhdf5.soNEEDED string. Same class of hazard may affect other C libraries in emscripten-forge that ship a versioned SONAME.netcdf-c 4.10.1 references
listxattrfromlibdispatch/dinfermodel.cinside the DAOS-container probe, guarded byHAVE_SYS_XATTR_H. Emscripten's<sys/xattr.h>declareslistxattrbut the runtime doesn't implement it, so the built.sofails to load.configure.achas zero DAOS knobs (no--disable-daos), so patch the source to early-returnNC_ENOTNCunder__EMSCRIPTEN__. Attemptingac_cv_header_sys_xattr_h=noproduced garbled compile errors in the surroundingisdaoscontainer()function under-std=gnu23and was abandoned.The
libnetcdfvariant pin was 4.9.3 even though the recipe carries version 4.10.1. Locally, that forced netcdf4's solver to reject 4.10.1 and pull 4.9.3 from upstream — which had been compiled against HDF5 1.12.3 headers and aborted at runtime against the 1.14.6 library ("Headers are 1.12.3, library is 1.14.6"). Bumped the pin to 4.10.1.Test plan