Skip to content

Fix netcdf4/h5py dimension scales on emscripten#5951

Open
MMesch wants to merge 5 commits into
emscripten-forge:mainfrom
MMesch:fix-netcdf4-dimscales
Open

Fix netcdf4/h5py dimension scales on emscripten#5951
MMesch wants to merge 5 commits into
emscripten-forge:mainfrom
MMesch:fix-netcdf4-dimscales

Conversation

@MMesch

@MMesch MMesch commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the runtime failure xarray.open_dataset(...)netCDF4.Datasetnc_openH5DSis_scaleDynamic linking error: cannot resolve symbol H5DSis_scale in the browser, and the follow-on RuntimeError: ring type mismatch occurred for cache entry that surfaces once H5DSis_scale is reachable.

Four package commits along the dependency chain:

  • hdf5: 1.12.3 → 1.14.6, plus two patches (see complications).
  • h5py: rebuild against 1.14.6, drop the ctypes.CDLL preload in __init__.py.
  • libnetcdf: skip the DAOS-container check under __EMSCRIPTEN__.
  • netcdf4: add -lhdf5_hl to LDFLAGS.

variant.yaml bumped: hdf5: 1.14.6, libnetcdf: 4.10.1.

<This was assisted by: Opus 4.7>

Complications

  1. HDF5 1.14.6 uses feclearexcept(FE_INVALID) in H5Tinit_float.c. Emscripten's <bits/fenv.h> defines only rounding modes and FE_ALL_EXCEPT=0; FE_INVALID is not declared because WASM has no FP exception flags. Guarded with #ifdef FE_INVALID.

  2. Emscripten's dynamic loader dedupes shared libraries by string identity of the NEEDED entry, not by realpath. With SOVERSION set, libhdf5_hl.so records NEEDED libhdf5.so.310.5.1, while -lhdf5 in consumers records NEEDED libhdf5.so. Both paths resolve to the same file via symlinks, but the loader treats them as distinct and loads libhdf5.so twice, each with its own bss segment. That produces two copies of H5CX_head_g (a file-static in H5CX.c) and every other libhdf5 global. When a call chain crosses between the two instances the metadata-cache "ring" invariant fires and any HL call after H5Aexists aborts. Confirmed by tracing patched H5CX_set_ring/push/pop/get_ring and observing two distinct &head_g addresses in the same process (0x2e31be0 and 0x38c9f40). Fix: skip set_target_properties(... VERSION SOVERSION) under EMSCRIPTEN in config/cmake/HDF5Macros.cmake so every consumer sees the same bare libhdf5.so NEEDED string. Same class of hazard may affect other C libraries in emscripten-forge that ship a versioned SONAME.

  3. netcdf-c 4.10.1 references listxattr from libdispatch/dinfermodel.c inside the DAOS-container probe, guarded by HAVE_SYS_XATTR_H. Emscripten's <sys/xattr.h> declares listxattr but the runtime doesn't implement it, so the built .so fails to load. configure.ac has zero DAOS knobs (no --disable-daos), so patch the source to early-return NC_ENOTNC under __EMSCRIPTEN__. Attempting ac_cv_header_sys_xattr_h=no produced garbled compile errors in the surrounding isdaoscontainer() function under -std=gnu23 and was abandoned.

  4. The libnetcdf variant pin was 4.9.3 even though the recipe carries version 4.10.1. Locally, that forced netcdf4's solver to reject 4.10.1 and pull 4.9.3 from upstream — which had been compiled against HDF5 1.12.3 headers and aborted at runtime against the 1.14.6 library ("Headers are 1.12.3, library is 1.14.6"). Bumped the pin to 4.10.1.

Test plan

  • hdf5 builds cleanly with both patches
  • h5py builds against 1.14.6, all 4 tests pass (import, usage, version, dimscales)
  • libnetcdf builds cleanly
  • netcdf4 builds cleanly, all 3 tests pass (versions, netcdf3 round-trip, netcdf4 round-trip)

@MMesch
MMesch force-pushed the fix-netcdf4-dimscales branch from cb26276 to f4548c4 Compare July 20, 2026 09:36
MMesch added 5 commits July 20, 2026 15:17
- bump 1.12.3 -> 1.14.6, update source URL template
- patches/0001-fenv-fe-invalid-guard.patch: guard FE_INVALID use in
  H5Tinit_float.c; emscripten's <bits/fenv.h> defines only rounding
  modes, FE_INVALID is not declared
- patches/0002-drop-soversion.patch: skip VERSION/SOVERSION properties
  under EMSCRIPTEN so libhdf5_hl.so records bare "libhdf5.so" as its
  NEEDED entry. Otherwise it records "libhdf5.so.310.5.1" while
  downstream packages link with -lhdf5 and record "libhdf5.so"; the
  emscripten loader dedupes by string identity (not realpath) and ends
  up loading libhdf5.so twice with independent bss segments -- two
  copies of H5CX_head_g etc., which breaks the metadata cache when
  calls cross between the instances
- variant.yaml: pin hdf5 to 1.14.6
- 1.14 auto-generates settings/ so the old workaround is dropped
- HDF5_VERSION 1.12.3 -> 1.14.6
- drop the ctypes.CDLL preload prepended to __init__.py; the
  hdf5 SOVERSION fix makes the emscripten loader dedupe libhdf5.so
  properly, so the manual dlopen is no longer needed and was masking
  the double-load bug during diagnosis
- add version check and a dimension-scale round-trip test that guards
  the HL path (H5DSset_scale + attach_scale + is_scale after reopen)
netcdf-c's isdaoscontainer() in libdispatch/dinfermodel.c uses
listxattr/getxattr to sniff a path for DAOS marker attributes.
Emscripten's <sys/xattr.h> declares these but the runtime does not
implement them, so the built .so fails to load with "cannot resolve
symbol listxattr". There is no --disable-daos configure option (the
check is hard-coded), so patch the source to early-return NC_ENOTNC
under __EMSCRIPTEN__.

Also bump the variant.yaml pin to 4.10.1 to match the recipe version;
the previous 4.9.3 pin was forcing consumers to resolve against upstream
4.9.3 which had been built against HDF5 1.12.3 headers and aborted at
runtime against a 1.14.6 runtime.
netcdf-4 file access walks the HDF5 tree with H5DSis_scale to
distinguish coordinate dims from data variables. Without -lhdf5_hl
in the extension LDFLAGS the built _netCDF4.so has no libhdf5_hl.so
in its NEEDED entries, so the loader never brings it in and calls
to H5DSis_scale die with "Dynamic linking error: cannot resolve
symbol H5DSis_scale" the moment xarray/netCDF4 opens a NETCDF4 file.

Also replace the smoke test with two focused cases: a NETCDF3_CLASSIC
round-trip and a NETCDF4 round-trip that exercises dim + variable +
attribute together.
CI lint requires a tests section on every recipe.
@MMesch
MMesch force-pushed the fix-netcdf4-dimscales branch from f4548c4 to 5495139 Compare July 20, 2026 15:17
@MMesch

MMesch commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

@IsabelParedes , waiting for your thoughts on this now. After discussing with @DerThorsten these are some points to consider:

  • the feclearexcept(FE_INVALID) issue is solved upstream in hdf5 2.x.
  • the deduplication problem should be solved generally. But we can use the patch here as hot fix and move ahead with this recipe more quickly.
  • the netcdf updates could be split out in a separate PR

@IsabelParedes

Copy link
Copy Markdown
Member

the netcdf updates could be split out in a separate PR

This would be nice. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants