You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ValueError: mmap length is greater than file size for 2d models #2732
Receiving the error message: "ValueError: mmap length is greater than file size" when training 2d model with multiple GPUs (or DDP). This error won't occur if only one GPU is used.
I am using nnUNet through a docker container base on one prebuilt by pyTorch team. When training 2d models, had the -num_gpus flag be set to anything other than 1 (or even set at all), during the data unpacking stage, it will show the error:
mmap length is greater than file size
Traceback (most recent call last):
File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
^^^^^^^^^^^^^^^^^
File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in next
return self.generate_train_batch()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/data_loader_2d.py", line 21, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/lib/npyio.py", line 453, in load
return format.open_memmap(file, mode=mmap_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/lib/format.py", line 945, in open_memmap
marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/core/memmap.py", line 268, in new
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: mmap length is greater than file size
Activity
mrokuss commentedon May 9, 2025
Hey,
could you provide more details on what you are training or any stacktrace?
yzluka commentedon May 29, 2025
I am using nnUNet through a docker container base on one prebuilt by pyTorch team. When training 2d models, had the -num_gpus flag be set to anything other than 1 (or even set at all), during the data unpacking stage, it will show the error:
mmap length is greater than file size
Traceback (most recent call last):
File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 53, in producer
item = next(data_loader)
^^^^^^^^^^^^^^^^^
File "/home/yz696/.local/lib/python3.11/site-packages/batchgenerators/dataloading/data_loader.py", line 126, in next
return self.generate_train_batch()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/data_loader_2d.py", line 21, in generate_train_batch
data, seg, properties = self._data.load_case(current_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/nnUNet_workflow/nnUNet/nnunetv2/training/dataloading/nnunet_dataset.py", line 86, in load_case
data = np.load(entry['data_file'][:-4] + ".npy", 'r')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/lib/npyio.py", line 453, in load
return format.open_memmap(file, mode=mmap_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/lib/format.py", line 945, in open_memmap
marray = numpy.memmap(filename, dtype=dtype, shape=shape, order=order,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/numpy/core/memmap.py", line 268, in new
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: mmap length is greater than file size
Merge #3058: create the dataloaders after unpacking, copy image crops
mrokuss commentedon Nov 20, 2026
@yzluka Could your issue be resolved in the meantime?
FabianIsensee commentedon Nov 20, 2026
FYI this could be an older blosc2 issue. We recently contributed heavily to version 4.15.0. Please install that - it should solve your problems