Update blosc2 tests

[t20100]

closes #377

[t20100]

Running tests with blosc2_j2k works, but there is some issues with running tests

• blosc2_grok:
  • When running tests with a locally built wheel of blsoc2_grok PR #23, I get OSError: /home/tvincent/miniforge3/envs/py314/lib/python3.14/site-packages/blosc2_grok/libblosc2_grok.so: undefined symbol: blosc2_error_string
  • When running on macos CI:

  OSError: dlopen(/private/var/folders/8d/778wjbv96mq1760tv6gk374m0000gn/T/cibw-run-masqg2hv/cp313-macosx_arm64/venv-test-arm64/lib/python3.13/site-packages/blosc2_grok/libblosc2_grok.so, 0x0006): Library not loaded: @rpath/libblosc2.7.dylib
    Referenced from: <E92C9D24-3C0C-320D-B445-845253CE1BA6> /private/var/folders/8d/778wjbv96mq1760tv6gk374m0000gn/T/cibw-run-masqg2hv/cp313-macosx_arm64/venv-test-arm64/lib/python3.13/site-packages/blosc2_grok/libblosc2_grok.so
    Reason: tried: '/private/var/folders/8d/778wjbv96mq1760tv6gk374m0000gn/T/cibw-run-masqg2hv/cp313-macosx_arm64/venv-test-arm64/lib/python3.13/site-packages/blosc2_grok/../blosc2/lib/libblosc2.7.dylib' (no such file), '/private/var/folders/8d/778wjbv96mq1760tv6gk374m0000gn/T/cibw-run-masqg2hv/cp313-macosx_arm64/venv-test-arm64/lib/python3.13/site-packages/blosc2_grok/../blosc2/lib/libblosc2.7.dylib' (no such file)

• blosc2_htj2k:

• When running tests locally, I get a seg fault for a stack of images:

double free or corruption (!prev)
Aborted (core dumped)

[t20100]

Updates:

• blosc2_grok issue can be fixed with comments in Use new b2nd_deserialize_meta_inline() in forthcoming c-blosc2 3.1.0 Blosc/blosc2_grok#23 (comment)

• blosc2_htj2k issue, running tests with python 3.14 on linux and:

  blosc2                        4.5.1
  blosc2_grok                   0.3.6.dev0  # Installed with previous patch
  blosc2_htj2k                  0.4.1
  blosc2_j2k                    0.4.0
  

  • this error and segfaut:

    Read blosc2 dataset written with jpeg2000-related external codec plugins ... 
      test_blosc2_j2k_codecs (__main__.TestBlosc2Plugins.test_blosc2_j2k_codecs) (plugin='blosc2_htj2k', shape=(10, 128, 128))
    Read blosc2 dataset written with jpeg2000-related external codec plugins ... ERROR
    
    ======================================================================
    ERROR: test_blosc2_j2k_codecs (__main__.TestBlosc2Plugins.test_blosc2_j2k_codecs) (plugin='blosc2_htj2k', shape=(10, 128, 128))
    Read blosc2 dataset written with jpeg2000-related external codec plugins
    ----------------------------------------------------------------------
    concurrent.futures.process._RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "python3.14/concurrent/futures/process.py", line 254, in _process_worker
        r = call_item.fn(*call_item.args, **call_item.kwargs)
      File "hdf5plugin/build/lib.linux-x86_64-cpython-314/hdf5plugin/test.py", line 1041, in _direct_chunk_write_blosc2
        blosc_array = blosc2.asarray(
            data,
        ...<2 lines>...
            cparams=cparams,
        )
      File "python3.14/site-packages/blosc2/ndarray.py", line 6673, in asarray
        return blosc2_ext.asarray(array, chunks, blocks, **kwargs)
               ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "blosc2/blosc2_ext.pyx", line 4361, in blosc2.blosc2_ext.asarray
      File "blosc2/blosc2_ext.pyx", line 3378, in blosc2.blosc2_ext._check_rc
        raise RuntimeError(message)
    RuntimeError: Error while creating the NDArray
    """
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "hdf5plugin/build/lib.linux-x86_64-cpython-314/hdf5plugin/test.py", line 1160, in test_blosc2_j2k_codecs
        self._write_blosc2_dataset(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^
            filename,
            ^^^^^^^^^
        ...<7 lines>...
            splitmode=blosc2.SplitMode.NEVER_SPLIT,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "hdf5plugin/build/lib.linux-x86_64-cpython-314/hdf5plugin/test.py", line 1089, in _write_blosc2_dataset
        _ = future.result()
      File "python3.14/concurrent/futures/_base.py", line 450, in result
        return self.__get_result()
               ~~~~~~~~~~~~~~~~~^^
      File "python3.14/concurrent/futures/_base.py", line 395, in __get_result
        raise self._exception
    RuntimeError: Error while creating the NDArray
    
    ----------------------------------------------------------------------
    Ran 61 tests in 1.620s
    
    FAILED (errors=1, skipped=28)
    Test suite failed
    double free or corruption (!prev)
    Aborted (core dumped)
    
    

  • When patching the test to only run with shape (128, 128) by commenting (10, 128, 128), there is still a segfault at the end:

    Read blosc2 dataset written with jpeg2000-related external codec plugins ... ok
    
    ----------------------------------------------------------------------
    Ran 61 tests in 1.166s
    
    OK (skipped=28)
    Test suite succeeded
    double free or corruption (!prev)
    Aborted (core dumped)
    

[t20100]

@alemirone there is still some issue with hdf5plugin tests and blosc2_htj2k (v0.4.1).

[alemirone]

Hi Thomas,

I reproduced the issue. The problem was specific to the OpenHTJ2K backend.

The failing case was a stack written as one Blosc2 chunk with several 2D blocks, for example:

shape = (10, 128, 128)
chunks = (10, 128, 128)
blocks = (1, 128, 128)

In this situation Blosc2 can call the codec several times in parallel, once per 2D block. OpenHTJ2K itself has internal multithreading, but its C++ API does not seem to be safe when several encoder/decoder instances are called concurrently from outside. This produced the failure and the crash at process shutdown.

I fixed it in blosc2_htj2k by serializing only the calls to the OpenHTJ2K backend. This keeps the internal multithreading of OpenHTJ2K, but avoids concurrent external calls from Blosc2 to multiple OpenHTJ2K instances.

Kakadu does not show this problem. I tested the same case with the Kakadu backend and Blosc2 multithreading enabled:

blosc2.set_nthreads(1, 2, 4, 8)
shape = (10, 128, 128)
chunks = (10, 128, 128)
blocks = (1, 128, 128)
backend = kakadu

All cases passed.

So the summary is:

OpenHTJ2K: internal backend multithreading is kept, but external concurrent Blosc2 calls are serialized.
Kakadu: external Blosc2 multithreading works normally.

I prepared this as blosc2_htj2k 0.4.2.

[alemirone]

One more clarification: we should not confuse Blosc2 external threading with the internal threading/tuning of each backend.

Blosc2 threading controls how many Blosc2 blocks can be processed in parallel:

blosc2.set_nthreads(n)

Backend threading is a separate layer. For now, these backend-specific settings are mostly controlled through environment variables.

For Kakadu:

BLOSC2_HTJ2K_KAKADU_THREADS=N

This controls Kakadu's internal kdu_thread_env. Values 0 or 1 mean no Kakadu thread environment; values >1 enable Kakadu internal threading. In practice, N is the intended total Kakadu thread count, including the calling thread.

Other Kakadu tuning variables currently supported are:

BLOSC2_HTJ2K_KAKADU_PRECISE=0/1
BLOSC2_HTJ2K_KAKADU_FAST=0/1
BLOSC2_HTJ2K_CLEVELS=N
BLOSC2_HTJ2K_KAKADU_DECODE_STRIPE_HEIGHT=N
BLOSC2_HTJ2K_KAKADU_PARAMS="Clevels=5;Qstep=..."

For OpenHTJ2K:

there is currently no public environment variable exposed by our wrapper for the internal thread count.

The wrapper currently calls OpenHTJ2K with num_threads=0:

openhtj2k_encoder(..., num_threads=0)
openhtj2k_decoder(..., num_threads=0)

In OpenHTJ2K, 0 means the default/all available internal threads.

So the current situation is:

Blosc2 external threads:
blosc2.set_nthreads(n)

Kakadu internal threads:
BLOSC2_HTJ2K_KAKADU_THREADS=N

OpenHTJ2K internal threads:
fixed to OpenHTJ2K default, i.e. num_threads=0

If you think it would be useful, we can expose these backend thread controls through the Python API as well, for example through configure() or a backend-specific settings function. For the moment they remain backend-specific environment variables.

[t20100]

Tests pass with blosc2_htj2k 0.4.2, thanks for the fix