Skip to content

there are so many question like "RuntimeError: Some background workers are no longer alive", BUT no one can resove it #2659

Closed

Description

@Diting-li
No description provided.

Activity

  1. Diting-li commented on Dec 28, 2024

    @Diting-li
    Author

    My computing resources are i9 14900K, 96GB RAM, 4090GPU 24GB. When I perform Inflance on a CT image, it cannot be executed and displays GPU OOM. After transferring to the CPU, there is a memory crash, and my file: shape (576, 768, 768)
    Direction: (1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0)
    Image starting point: (-384.0, -384.0, -72.5)
    Image slice spacing: (0.30000001192092896, 0.30000001192092896, 0.30000001192092896)

  2. dongdongtong commented on Jan 4, 2025

    @dongdongtong

    Similar bugs in nnunetv2 with pytorch 2.5.0, cuda 12.1, python 3.12.8.

    When I am running the nnunetv2 framework, I am keeping to get the similar errors reporting that

    RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
    Traceback (most recent call last):
      File "/home/ziyang/anaconda3/envs/nnunet/bin/nnUNetv2_train", line 8, in <module>
        sys.exit(run_training_entry())
                 ^^^^^^^^^^^^^^^^^^^^
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/run/run_training.py", line 275, in run_training_entry
        run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/run/run_training.py", line 211, in run_training
        nnunet_trainer.run_training()
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1370, in run_training
        train_outputs.append(self.train_step(next(self.dataloader_train)))
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 196, in __next__
        item = self.__get_next_item()
               ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 181, in __get_next_item
        raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
    RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
    Exception in thread Thread-3 (results_loop):
    Traceback (most recent call last):
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
        self.run()
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/threading.py", line 1012, in run
        self._target(*self._args, **self._kwargs)
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 125, in results_loop
        raise e
      File "/home/ziyang/anaconda3/envs/nnunet/lib/python3.12/site-packages/batchgenerators/dataloading/nondet_multi_threaded_augmenter.py", line 103, in results_loop
        raise RuntimeError("One or more background workers are no longer alive. Exiting. Please check the "
    RuntimeError: One or more background workers are no longer alive. Exiting. Please check the print statements above for the actual error message
    

    But I cannot figure it out where the bug is. When I add nnUNet_n_proc_DA=0 to run the training command, the bug disappears, but the cost may be the slower running time.

    nnUNet_n_proc_DA=0 nnUNetv2_train 162 3d_fullres 0
    
  3. oonatmai commented on Feb 18, 2025

    @oonatmai

    Based on my experience, the "RuntimeError: Some background workers are no longer alive" usually comes from running out of memory, either on your GPU or system RAM. That's why setting a lower nnUNet_n_proc_DA usually helps. If it still doesn't, you should check your patch and batch sizes, as we all know that volumetric data tends to consume a lot of space. :/

    FYI: My RTX 4090 can fit only a patch size of (96, 224, 224) with batch size of 2 for nnUNetResEncUNetLPlans, which fully utilizes the 24GB VRAM available on the 4090. So, if you're using a patch size of (576, 768, 768), I don't think it will fit.

  4. dojoh commented on Feb 24, 2025

    @dojoh

    did @oonatmai 's comment already help? you should use the same patchsize for inference that you used for training and set the batchsize to the maximum your gpu can handle. If the input image is larger than the patchsize it will be sliced into patches that match the patchsize and combined to a prediction of the input dimensions automatically.

  5. LinkinPony commented on Mar 25, 2025

    @LinkinPony

    In my situation, these errors are caused by memory stability. Try to run Memtest86+ and make sure you can pass all these tests.

  6. ykirchhoff commented on Nov 20, 2026

    @ykirchhoff
    Member

    Hi everyone,
    I guess this issue is resolved, so I will close it now. In case you are still having problems, feel free to reopen this issue or open a new one.
    Best,
    Yannick

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions