You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
There is a typing error in the "nnUNetTrainer.py" #2971
Hello authors !
I think there is a typing error in **the line 956 of the "nnUNetTrainer.py"** as show in uploaded picture.
I think there should be as follows:
isinstance(**self.dataloader_val**, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter))
Thanks for the report — you are correct, and this is a genuine bug!
The bug is in nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py (around line 983–985 in the current version). Inside on_train_end, the code correctly checks whether dataloader_val is not None, but then mistakenly passes self.dataloader_train to isinstance instead of self.dataloader_val:
Effect of the bug: When dataloader_val is a NonDetMultiThreadedAugmenter or MultiThreadedAugmenter but dataloader_train is not (or is None), the validation dataloader's background workers will not be properly shut down at the end of training — potentially leaving zombie threads/processes behind.
The fix is exactly what you suggested: replace self.dataloader_train with self.dataloader_val in the isinstance call.
Thanks for the careful report! You are correct — this is a genuine bug.
Confirmed bug in nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, line 984:
# Current (wrong):ifself.dataloader_valisnotNoneand \
isinstance(self.dataloader_train, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
self.dataloader_val._finish()
# Should be:ifself.dataloader_valisnotNoneand \
isinstance(self.dataloader_val, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
self.dataloader_val._finish()
What's wrong: The if self.dataloader_val is not None guard on line 983 correctly targets the validation dataloader, but the isinstance check on line 984 mistakenly uses self.dataloader_train instead of self.dataloader_val. This means:
If dataloader_train is not a MultiThreadedAugmenter (e.g. a plain DataLoader), dataloader_val._finish() is never called even when it should be.
If dataloader_trainis a MultiThreadedAugmenter but dataloader_val is not, _finish() would be called on a non-augmenter object incorrectly.
This could cause worker threads not to be cleaned up properly after training ends.
In nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, inside on_train_end(), the second isinstance check (line 984) mistakenly references self.dataloader_train instead of self.dataloader_val:
# Current (buggy):ifself.dataloader_valisnotNoneand \
isinstance(self.dataloader_train, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
self.dataloader_val._finish()
# Should be:ifself.dataloader_valisnotNoneand \
isinstance(self.dataloader_val, (NonDetMultiThreadedAugmenter, MultiThreadedAugmenter)):
self.dataloader_val._finish()
Impact
Because of this typo, self.dataloader_val._finish() is never called when dataloader_train happens to not be a MultiThreadedAugmenter (even if dataloader_val is), meaning the validation dataloader's background worker threads may not be properly shut down at the end of training. This could cause resource leaks or hangs in certain configurations.
This is a trivial one-line fix. A maintainer will review and issue a patch.
Activity
FabianIsensee commentedon Apr 8, 2026
@claude let's try to solve this issue
claude commentedon Apr 8, 2026
🤖 nnU-Net Assistant
Thanks for the report — you are correct, and this is a genuine bug!
The bug is in
nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py(around line 983–985 in the current version). Insideon_train_end, the code correctly checks whetherdataloader_val is not None, but then mistakenly passesself.dataloader_traintoisinstanceinstead ofself.dataloader_val:Effect of the bug: When
dataloader_valis aNonDetMultiThreadedAugmenterorMultiThreadedAugmenterbutdataloader_trainis not (or isNone), the validation dataloader's background workers will not be properly shut down at the end of training — potentially leaving zombie threads/processes behind.The fix is exactly what you suggested: replace
self.dataloader_trainwithself.dataloader_valin theisinstancecall.A fix PR will be created for this.
FabianIsensee commentedon Apr 8, 2026
@claude let's try again
claude commentedon Apr 8, 2026
🤖 nnU-Net Assistant
Thanks for the careful report! You are correct — this is a genuine bug.
Confirmed bug in
nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, line 984:What's wrong: The
if self.dataloader_val is not Noneguard on line 983 correctly targets the validation dataloader, but theisinstancecheck on line 984 mistakenly usesself.dataloader_traininstead ofself.dataloader_val. This means:dataloader_trainis not aMultiThreadedAugmenter(e.g. a plain DataLoader),dataloader_val._finish()is never called even when it should be.dataloader_trainis aMultiThreadedAugmenterbutdataloader_valis not,_finish()would be called on a non-augmenter object incorrectly.This could cause worker threads not to be cleaned up properly after training ends.
A fix PR will be opened shortly.
FabianIsensee commentedon Apr 8, 2026
@claude AGAIN
claude commentedon Apr 8, 2026
🤖 nnU-Net Assistant
Great catch! This is a confirmed bug.
What's wrong
In
nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py, insideon_train_end(), the secondisinstancecheck (line 984) mistakenly referencesself.dataloader_traininstead ofself.dataloader_val:Impact
Because of this typo,
self.dataloader_val._finish()is never called whendataloader_trainhappens to not be aMultiThreadedAugmenter(even ifdataloader_valis), meaning the validation dataloader's background worker threads may not be properly shut down at the end of training. This could cause resource leaks or hangs in certain configurations.This is a trivial one-line fix. A maintainer will review and issue a patch.
fix: correct isinstance check for dataloader_val in on_train_end (fixes
FabianIsensee commentedon Apr 8, 2026
Thanks a lot for this, and sorry for misusing your issue as a PoC for claude :-)
fix: correct isinstance check for dataloader_val in on_train_end (fixes