Skip to content

There is a typing error in the "nnUNetTrainer.py" #2971

Closed

Description

@huangliang-666
Image Hello authors ! I think there is a typing error in **the line 956 of the "nnUNetTrainer.py"** as show in uploaded picture. I think there should be as follows: isinstance(**self.dataloader_val**, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter))

I'm not sure if what I think is correct.

Thank you, looking forward to your reply.

Activity

  1. FabianIsensee commented on Apr 8, 2026

    @FabianIsensee
    Member

    @claude let's try to solve this issue

  2. claude commented on Apr 8, 2026

    @claude

    🤖 nnU-Net Assistant

    Thanks for the report — you are correct, and this is a genuine bug!

    The bug is in nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py (around line 983–985 in the current version). Inside on_train_end, the code correctly checks whether dataloader_val is not None, but then mistakenly passes self.dataloader_train to isinstance instead of self.dataloader_val:

    # Current (buggy):
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_train, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()
    
    # Correct fix:
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_val, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()

    Effect of the bug: When dataloader_val is a NonDetMultiThreadedAugmenter or MultiThreadedAugmenter but dataloader_train is not (or is None), the validation dataloader's background workers will not be properly shut down at the end of training — potentially leaving zombie threads/processes behind.

    The fix is exactly what you suggested: replace self.dataloader_train with self.dataloader_val in the isinstance call.

    A fix PR will be created for this.

  3. FabianIsensee commented on Apr 8, 2026

    @FabianIsensee
    Member

    @claude let's try again

  4. claude commented on Apr 8, 2026

    @claude

    🤖 nnU-Net Assistant

    Thanks for the careful report! You are correct — this is a genuine bug.

    Confirmed bug in nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, line 984:

    # Current (wrong):
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_train, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()
    
    # Should be:
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_val, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()

    What's wrong: The if self.dataloader_val is not None guard on line 983 correctly targets the validation dataloader, but the isinstance check on line 984 mistakenly uses self.dataloader_train instead of self.dataloader_val. This means:

    • If dataloader_train is not a MultiThreadedAugmenter (e.g. a plain DataLoader), dataloader_val._finish() is never called even when it should be.
    • If dataloader_train is a MultiThreadedAugmenter but dataloader_val is not, _finish() would be called on a non-augmenter object incorrectly.

    This could cause worker threads not to be cleaned up properly after training ends.

    A fix PR will be opened shortly.

  5. FabianIsensee commented on Apr 8, 2026

    @FabianIsensee
    Member

    @claude AGAIN

  6. claude commented on Apr 8, 2026

    @claude

    🤖 nnU-Net Assistant

    Great catch! This is a confirmed bug.

    What's wrong

    In nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, inside on_train_end(), the second isinstance check (line 984) mistakenly references self.dataloader_train instead of self.dataloader_val:

    # Current (buggy):
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_train, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()
    
    # Should be:
    if self.dataloader_val is not None and \
            isinstance(self.dataloader_val, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
        self.dataloader_val._finish()

    Impact

    Because of this typo, self.dataloader_val._finish() is never called when dataloader_train happens to not be a MultiThreadedAugmenter (even if dataloader_val is), meaning the validation dataloader's background worker threads may not be properly shut down at the end of training. This could cause resource leaks or hangs in certain configurations.

    This is a trivial one-line fix. A maintainer will review and issue a patch.

  7. added a commit that references this issue on Apr 8, 2026
    d843a12
  8. added
    has-fix-prMaintainer or agent has created a PR for this issue
    and removed on Apr 8, 2026
  9. FabianIsensee commented on Apr 8, 2026

    @FabianIsensee
    Member

    Thanks a lot for this, and sorry for misusing your issue as a PoC for claude :-)

  10. added a commit that references this issue on Jun 15, 2026
    a3014d4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghas-fix-prMaintainer or agent has created a PR for this issueneeds-maintainerRequires maintainer attention

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions