Skip to content

ValueError: mmap length is greater than file size for 2d models #2732

Open

Description

@yzluka

Receiving the error message: "ValueError: mmap length is greater than file size" when training 2d model with multiple GPUs (or DDP). This error won't occur if only one GPU is used.

Activity

  1. mrokuss commented on May 9, 2025

    @mrokuss
    Contributor

    Hey,

    could you provide more details on what you are training or any stacktrace?

  2. yzluka commented on May 29, 2025

    @yzluka
    Author

    I am using nnUNet through a docker container base on one prebuilt by pyTorch team. When training 2d models, had the -num_gpus flag be set to anything other than 1 (or even set at all), during the data unpacking stage, it will show the error:

    mmap length is greater than file size
    Traceback (most recent call last):
    File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
    item = next(data_loader)
    ^^^^^^^^^^^^^^^^^
    File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in next
    return self.generate_train_batch()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/data_loader_2d.py", line 21, in generate_train_batch
    data, seg, properties = self._data.load_case(current_key)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
    data = np.load(entry['data_file'][:-4] + ".npy", 'r')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/numpy/lib/npyio.py", line 453, in load
    return format.open_memmap(file, mode=mmap_mode,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/numpy/lib/format.py", line 945, in open_memmap
    marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/lib/python3.11/site-packages/numpy/core/memmap.py", line 268, in new
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ValueError: mmap length is greater than file size

  3. added a commit that references this issue on Oct 29, 2026
  4. mrokuss commented on Nov 20, 2026

    @mrokuss
    Contributor

    @yzluka Could your issue be resolved in the meantime?

  5. FabianIsensee commented on Nov 20, 2026

    @FabianIsensee
    Member

    FYI this could be an older blosc2 issue. We recently contributed heavily to version 4.15.0. Please install that - it should solve your problems

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions