You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
there are so many question like "RuntimeError: Some background workers are no longer alive", BUT no one can resove it #2659
My computing resources are i9 14900K, 96GB RAM, 4090GPU 24GB. When I perform Inflance on a CT image, it cannot be executed and displays GPU OOM. After transferring to the CPU, there is a memory crash, and my file: shape (576, 768, 768)
Direction: (1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0)
Image starting point: (-384.0, -384.0, -72.5)
Image slice spacing: (0.30000001192092896, 0.30000001192092896, 0.30000001192092896)
Similar bugs in nnunetv2 with pytorch 2.5.0, cuda 12.1, python 3.12.8.
When I am running the nnunetv2 framework, I am keeping to get the similar errors reporting that
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Traceback (most recent call last):
File "/home/ziyang/anaconda3/envs/nnunet/bin/nnUNetv2_train", line 8, in <module>
sys.exit(run_training_entry())
^^^^^^^^^^^^^^^^^^^^
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/run/run_training.py", line 275, in run_training_entry
run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/run/run_training.py", line 211, in run_training
nnunet_trainer.run_training()
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1370, in run_training
train_outputs.append(self.train_step(next(self.dataloader_train)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
item = self.__get_next_item()
^^^^^^^^^^^^^^^^^^^^^^
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
Exception in thread Thread-3 (results_loop):
Traceback (most recent call last):
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
self.run()
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/threading.py", line 1012, in run
self._target(*self._args, **self._kwargs)
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
raise e
File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
But I cannot figure it out where the bug is. When I add nnUNet_n_proc_DA=0 to run the training command, the bug disappears, but the cost may be the slower running time.
Based on my experience, the "RuntimeError: Some background workers are no longer alive" usually comes from running out of memory, either on your GPU or system RAM. That's why setting a lower nnUNet_n_proc_DA usually helps. If it still doesn't, you should check your patch and batch sizes, as we all know that volumetric data tends to consume a lot of space. :/
FYI: My RTX 4090 can fit only a patch size of (96, 224, 224) with batch size of 2 for nnUNetResEncUNetLPlans, which fully utilizes the 24GB VRAM available on the 4090. So, if you're using a patch size of (576, 768, 768), I don't think it will fit.
did @oonatmai 's comment already help? you should use the same patchsize for inference that you used for training and set the batchsize to the maximum your gpu can handle. If the input image is larger than the patchsize it will be sliced into patches that match the patchsize and combined to a prediction of the input dimensions automatically.
Hi everyone,
I guess this issue is resolved, so I will close it now. In case you are still having problems, feel free to reopen this issue or open a new one.
Best,
Yannick
Activity
Diting-li commentedon Dec 28, 2024
My computing resources are i9 14900K, 96GB RAM, 4090GPU 24GB. When I perform Inflance on a CT image, it cannot be executed and displays GPU OOM. After transferring to the CPU, there is a memory crash, and my file: shape (576, 768, 768)
Direction: (1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0)
Image starting point: (-384.0, -384.0, -72.5)
Image slice spacing: (0.30000001192092896, 0.30000001192092896, 0.30000001192092896)
dongdongtong commentedon Jan 4, 2025
Similar bugs in nnunetv2 with pytorch 2.5.0, cuda 12.1, python 3.12.8.
When I am running the nnunetv2 framework, I am keeping to get the similar errors reporting that
But I cannot figure it out where the bug is. When I add
nnUNet_n_proc_DA=0to run the training command, the bug disappears, but the cost may be the slower running time.oonatmai commentedon Feb 18, 2025
Based on my experience, the "RuntimeError: Some background workers are no longer alive" usually comes from running out of memory, either on your GPU or system RAM. That's why setting a lower nnUNet_n_proc_DA usually helps. If it still doesn't, you should check your patch and batch sizes, as we all know that volumetric data tends to consume a lot of space. :/
FYI: My RTX 4090 can fit only a patch size of (96, 224, 224) with batch size of 2 for nnUNetResEncUNetLPlans, which fully utilizes the 24GB VRAM available on the 4090. So, if you're using a patch size of (576, 768, 768), I don't think it will fit.
dojoh commentedon Feb 24, 2025
did @oonatmai 's comment already help? you should use the same patchsize for inference that you used for training and set the batchsize to the maximum your gpu can handle. If the input image is larger than the patchsize it will be sliced into patches that match the patchsize and combined to a prediction of the input dimensions automatically.
LinkinPony commentedon Mar 25, 2025
In my situation, these errors are caused by memory stability. Try to run Memtest86+ and make sure you can pass all these tests.
ykirchhoff commentedon Nov 20, 2026
Hi everyone,
I guess this issue is resolved, so I will close it now. In case you are still having problems, feel free to reopen this issue or open a new one.
Best,
Yannick