nnUNetv2_predict_streaming --save_probabilities and save_probabilities=True on the nnUNetPredictorStreaming
predict_* methods write the predicted probabilities slice by slice to <case>.b2nd (a blosc2 array, uint16
p * 65535; --float32 / probability_dtype = 'float32' for lossless float32 probabilities, with a warning about file
size and speed) plus <case>.pkl; see streaming inference and
nnunetv2/inference/streaming_probabilities.py. The new keyword arguments come last, so existing calls are unchanged;
_predict_and_export_streaming and _predict_case_2d take an optional probability_writer, and
nnUNetPredictor.probabilities_file_ending (.npz; .b2nd for the streaming predictor) names the file that
--continue_prediction looks for. nnUNetv2_ensemble does not read .b2nd files.
nnU-Net now requires blosc2>=4.15.0 (was >=4.1.0): it writes these files 2-3x faster than 4.14.1. blosc2 needs
Python 3.11 or newer since 4.4.1, so nnU-Net now requires Python >= 3.11 (was 3.10).
nnUNetDataLoader.get_bbox draws patch positions only where the final patch lies inside the case, and pads a case
that is smaller than the patch on both sides, centered. Up to 2.8.1, foreground patches were clamped to that range at
the lower border only (GitHub PR #3033): near the upper border they reached into the padding, and a case smaller than
the patch was pushed off-center in foreground patches. With that fixed (clamped at both borders), a case smaller than
the patch sat at exactly the same place in every training sample, because all spatial augmentation acts about the
patch center, and a case only a little larger than the patch hardly moved. The network can then tie its decisions to
positions in the patch. That cost Dice on datasets with such cases.
Now the final patch can always move by at least half its size along every axis: where the range of positions inside
the case is shorter than final_patch_size // 2 (cases smaller than, as large as, or up to 1.5 times the patch along an
axis), it is widened to that, equally on both sides, and the patch reaches into the padding. Cases at least 1.5 times
the patch are sampled as before. This holds for random and foreground patches, in the training and validation
dataloaders. The slice axis of 2D configurations (final patch size 1) is unchanged.
Measured with ResEnc-L, 3d_fullres, 5-fold cross-validation on 10 datasets, against the fix above alone: ACDC +0.0060 and Prostate +0.0153 foreground Dice, better on every fold of both; over the 9 datasets other than Colon +0.0034 on average and better on 8; no dataset worse beyond run-to-run noise (BraTS -0.0011). Against earlier runs with the lower-border clamp, the fix alone was -0.003 on ACDC and -0.010 on Prostate; with the minimum range both end above those runs. 2d and 3d_lowres use the same code but were not measured.
nnUNetDataLoader has a new keyword argument min_patch_range_div (default 2; the minimum range per axis is
final_patch_size // min_patch_range_div, kept in self.min_patch_range). min_patch_range_div=None keeps patches
inside the case and centers small cases, as with the fix alone. Custom trainers that build nnUNetDataLoader get the
new default; a subclass that overrides get_bbox without calling it keeps its own placement.
nnunetv2/training/nnUNetTrainer/variants/patch_size/nnUNetTrainer_PGPS.py adds PGPSEfficiencyMixin, the progressive
patch size schedule of Progressive Growing of Patch Size (PGPS-Efficiency), and the
trainers nnUNetTrainer_PGPS, nnUNetTrainer_PGPS_2700epochs, nnUNetTrainer_DASegOrd0_NoMirroring_PGPS_2700epochs_SV
and nnUNetTrainer_DASegOrd0_NoMirroring_PGPS_5400epochs_SV (see Train models). Mix it in first
to give your own trainer PGPS. Nothing changes for existing trainers.
- The stage patch size is written into the trainer's configuration manager;
plans.jsonkeeps the configured one. The dataloaders are built again at every stage switch, and only the last stage writescheckpoint_best. - With
torch.compileon CUDA, the training steps of stages with up topgps_cudagraph_max_patch_voxelsvoxels per patch run as CUDA graphs (same numerics).pgps_cudagraphs = Falseswitches them off. - While a PGPS trainer runs, all compiled code is dropped at every stage switch (
torch._dynamo.reset()), and in the stage whose bottleneck has a single voxel, InstanceNorm's check for more than one spatial element is skipped inside the steps of that stage only. No global setting of torch.compile is changed.
nnunetv2/training/nnUNetTrainer/variants/validation/nnUNetTrainer_streamingValidation.py adds
StreamingValidationMixin, which replaces perform_actual_validation with one through nnUNetPredictorStreaming.
The stock final validation resamples the full (num_classes, *original_shape) probability volume of every case on the
CPU and runs out of RAM with many classes on large images; the streaming one never builds that volume. Training is
unchanged. Resampling follows the plans as in nnUNetv2_predict_streaming, so the validation segmentations match those
of the stock validation up to float rounding at argmax boundaries; plans whose resampling streaming cannot reproduce
(e.g. cubic) get torch trilinear and a warning in the training log. 3d configurations only; no cascade, next-stage
prediction or probability export; the trainers check this at initialize, so an unsupported configuration fails
before training, not after it.
New trainers: nnUNetTrainer_SV, nnUNetTrainer_2000epochs_NoMirroring_SV, and
nnUNetTrainer_DASegOrd0_NoMirroring_SV with _2000epochs_SV / _5000epochs_SV versions (order-0 segmentation
resampling in the spatial augmentation, no mirroring, ChunkedLossMixin). Mix StreamingValidationMixin in first
to give your own trainer streaming validation. Nothing changes for existing trainers.
NibabelIO / NibabelIOWithReorient gained read_nibabel_images and seg_to_nibabel, SimpleITKIO /
SimpleITKIOWithReorient gained read_sitk_images and seg_to_sitk. They take images that are already in memory
(nibabel / SimpleITK image objects, for example handed over by another tool) and return exactly what read_images
returns for the same images stored as files; seg_to_* return the image that write_seg would write.
read_images and write_seg now go through them. A custom reader that overrides read_images or write_seg works
as before; the new in-memory methods do not go through those overrides. SimpleITKIOWithReorient no longer defines
its own write_seg: it inherits SimpleITKIO.write_seg, which writes seg_to_sitk's (reoriented) image, so the
written files are unchanged.
NibabelIO, NibabelIOWithReorient, SimpleITKIO and SimpleITKIOWithReorient return the same arrays and properties
as before, bit for bit, but make fewer full-size copies of the image on the way. Reading a 512 x 512 x 1940 CT (int16
with scl_slope / scl_inter; 1.9 GB as float32) used to raise the peak memory by 7.6 GB with either nibabel reader. Now
it is 2.9 GB, and the read is about 10 % faster. With the SimpleITK readers it went from 5.7 GB (SimpleITKIO) and
7.6 GB (SimpleITKIOWithReorient) to 4.7 GB, which is the peak of SimpleITK's own ReadImage for this file.
- The nibabel readers no longer create
get_fdata()'s float64 image. They read a stored image once in its stored dtype, reorient it as a view and convert it to float32 in small blocks, with nibabel's own scaling in float64. read_nibabel_imagesdoes not fill theget_fdata()cache of the images it is given, so an image you keep does not hold a float64 copy afterwards. It checks shapes, affines and spacings before it reads any voxels.- The SimpleITK readers take views of the ITK images' buffers instead of copying them.
SimpleITKIOWithReorientreorients a 3d image as a view of its buffer and takes the geometry fromDICOMOrientapplied to a one-byte stand-in, instead of reorienting a copy.
Creating an nnUNetPredictor for a CUDA device used to set torch.backends.cudnn.benchmark = True for the whole
process. It now leaves the setting alone. cuDNN's algorithm search took 1.2 s and 2.2 GB of extra device memory on the
first case of a 3d ResEnc model (192^3 patches, RTX 4090) and made the following cases about 1 % faster, so it only
paid off over a few dozen cases. The algorithms it picks, and with them the predictions, can also differ between runs.
To get the old behaviour, set torch.backends.cudnn.benchmark = True yourself. Training (nnUNetv2_train) still
enables it, so the final validation is unaffected.
predict_logits_from_preprocessed_data (and everything that predicts through it, such as predict_from_files and
predict_single_npy_array) used to load the state dicts in list_of_parameters into the network for every case, even
with a single fold. It now loads a state dict only if the network does not hold it already: it remembers which state
dict object it loaded into which network module. Fold ensembles still load every fold for every case, and replacing
predictor.network or predictor.list_of_parameters loads the weights again, as before. Predictions are unchanged.
What is different: if your code changes the network's weights or buffers in place between predictions, the next case
no longer resets them to list_of_parameters, and changes made in place to the tensors of a state dict in
list_of_parameters are no longer loaded. Call predictor.network.load_state_dict(...) yourself in that case.
DefaultPreprocessor.run_case and run_case_npy take a new argument return_seg (default True, which changes
nothing). nnU-Net's own inference code passes return_seg=False when no segmentation of a previous stage is
preprocessed along: the segmentation that would only hold the nonzero mask is then not resampled and None is returned
in its place, and the mask is not built at all unless the normalization needs it. Data and properties are identical.
- Custom preprocessors whose
run_case/run_case_npydo not takereturn_segare called as before. - A custom preprocessor that overrides
_normalizealways gets the mask. - Custom normalization schemes always get the mask, as before. If your scheme's
run()looks atsegonly whenuse_mask_for_normis set (like all of nnU-Net's own schemes), setuses_seg_only_if_use_mask_for_norm = Trueon the class to let inference skip it. Subclasses do not inherit this setting.
Three changes make every training iteration cheaper on the host. Training results are unchanged: fused SGD computes the same update as the foreach SGD it replaces, and the logged training loss is the same mean of the same values. The gain is largest where the host and not the GPU limits the speed (small patches or networks, fast GPUs, slow CPUs).
configure_optimizersbuilds fused SGD on CUDA (fused=True, if the installed torch has it), with its momentum buffers initialised to zero (zero_missing_momentum_buffers). Other devices keep foreach SGD. Customconfigure_optimizersare not affected.load_checkpointkeeps the optimizer implementation thatconfigure_optimizersbuilt.Optimizer.load_state_dictalso restores thefused/foreachflags stored in a checkpoint; nnU-Net now puts back the ones of its own optimizer (_load_optimizer_state). Trainings started before this change continue with--con fused SGD. An override ofload_checkpointthat callsself.optimizer.load_state_dict(...)itself should callself._load_optimizer_state(...)instead: continuing a foreach checkpoint with fused SGD otherwise fails in the first step (AssertionErrorinGradScaler.step).train_stepandvalidation_stepreturn their outputs as detached tensors on the training device (the loss;tp_hard,fp_hard,fn_hardof the validation step), no longer as numpy arrays, so that the host does not wait for the GPU in every iteration.run_trainingmoves an epoch's outputs to the host at once (outputs_to_host, one transfer per key), soon_train_epoch_endandon_validation_epoch_end, and overrides of them, get the same numpy arrays as before. Only code that calls the steps itself and reads their outputs gets tensors: a customrun_trainingshould pass them throughoutputs_to_hosttoo (collate_outputsaccepts tensors as well). Step overrides that return numpy arrays keep working.- progress.png is written at most every
self.plot_progress_every_sseconds (default 60), and always after the first epoch of a run, after the last one and before an exit on a termination signal. Plotting takes 0.4 - 1.2 s per epoch and grows with the number of epochs.self.plot_progress_every_s = 0restores a plot after every epoch. - The exit on a termination signal (SIGUSR1) stops the data augmentation workers first (
_shutdown_dataloaders, whichon_train_enduses as well). Before, the workers were killed under the augmenters' results threads, which printed "One or more background workers are no longer alive" tracebacks although the exit was clean.
on_epoch_end writes checkpoint_best through the new method _maybe_save_checkpoint_best, with the same rule as
before (a new best EMA pseudo Dice). A trainer that wants only some epochs to compete for checkpoint_best overrides
that method instead of copying on_epoch_end. Trainers that override on_epoch_end are not affected.
DDP: oversample_foreground_percent is the value for the whole batch again after _set_batch_size_and_oversample
Under DDP, _set_batch_size_and_oversample used to overwrite oversample_foreground_percent with the share of the
current rank. A trainer that called it a second time (for example after changing the batch size) then split that
share again, and the fraction of foreground-forced samples changed. oversample_foreground_percent is now a property:
setting it sets the value for the whole batch, and after the split reading it returns the rank's share, as before.
Every call splits the batch's value, and setting the attribute again discards the share. Nothing changes for trainers
that call the method once (initialize does) or that set the attribute and then call it. A subclass that defines
oversample_foreground_percent as a class attribute replaces the property, and under DDP the split then raises an
AttributeError. Set it in __init__ instead (nnUNetTrainer.__init__ overrode such a class attribute before, too).
nnU-Net writes checkpoint_final, checkpoint_best and checkpoint_latest as .safetensors instead of .pth. A
.pth is a pickle, and torch.load(..., weights_only=False) runs any code the file contains; a safetensors file is a
JSON header plus raw tensor bytes. The checkpoint holds the same dict as before (network weights, optimizer state,
logging, init args, ...): tensors are stored as tensors, shared ones once, and everything else as typed JSON, so
loading returns the same values with the same types. nnunetv2/utilities/checkpoints.py describes the format.
checkpoint_finalno longer containsoptimizer_stateandgrad_scaler_state. Training never continues from it, and the optimizer state is as large as the weights. Continuing a finished training for more epochs therefore starts with a fresh optimizer (nnU-Net prints a warning).checkpoint_bestandcheckpoint_latestkeep everything.- Old checkpoints keep working. Everything that reads a checkpoint (inference,
--c,--val,--val_best,-pretrained_weights, export) reads both formats, and.pthfiles are loaded exactly as before. A checkpoint name finds the file with the other extension if the named one does not exist, so the new default-chk checkpoint_final.safetensorsfinds thecheckpoint_final.pthof an old model. - Continuing an old training loads its
.pth. Whenever the trainer then writesX.safetensors, it deletes theX.pthnext to it, the checkpoint the new file replaces. nnUNetv2_convert_checkpoints_to_safetensors -i FOLDERconverts the.pthcheckpoints of finished trainings belowFOLDER, verifies each new file against the.pth, and deletes the.pthunless--keep_pthis set.checkpoint_finalloses its optimizer state on the way. Only convert checkpoints you trust: loading a.pthruns code.- nnU-Net versions before this change cannot read
.safetensorscheckpoints.
What custom trainers and scripts need to know:
save_checkpoint(filename)writes<name>.safetensors, whatever extensionfilenamehas, and nnU-Net itself now passes.safetensorsnames. An override that callstorch.save(checkpoint, filename)therefore writes a pickle with a.safetensorsname, which nnU-Net refuses to load (the error says how to rename it). Old:torch.save(checkpoint, filename). New:save_checkpoint_file(checkpoint, filename)fromnnunetv2.utilities.checkpoints.- An override of
load_checkpointthat callstorch.load(filename, ...)cannot read the new files. Old:torch.load(filename, map_location=self.device, weights_only=False). New:load_checkpoint_file(filename, map_location=self.device), which reads both formats and, like everything else, finds the file with the other extension if the named one does not exist. Expectoptimizer_stateandgrad_scaler_stateto be missing incheckpoint_final. - Scripts that open checkpoints with
torch.loadshould useload_checkpoint_fileas well.torch.loadcannot read a.safetensorsfile. - A checkpoint can only hold plain data: tensors, numpy scalars and arrays,
None,bool,int,float,str, lists, tuples, dicts,torch.deviceandtorch.dtype. That covers everythingnnUNetTrainerstores. A custom trainer whose__init__arguments (they are stored asinit_args) or whose modules'get_extra_state()hold other objects fails when it writes its first checkpoint,checkpoint_bestafter epoch 1, with aValueErrornaming the entries. Under DDP every rank raises it:save_checkpointbuilds the checkpoint on all ranks and the ones that do not write check that it can be stored, so the job ends instead of leaving ranks waiting for one that crashed.
The compound loss is compiled as a whole, and _build_loss picks its loss classes from class attributes
nnUNetTrainer._build_loss used to compile only the dice term (loss.dc = torch.compile(loss.dc)); it now
compiles the whole compound loss. The dice and the CE both start from the same logits (softmax vs
log_softmax), so one graph lets inductor share that work and lets AOTAutograd recompute instead of saving.
Measured on a 4090 at 119 classes / 128^3 / bs 2 / AMP, fwd+bwd peak over the inputs and time for the
compound loss: 6.07 GiB / 38.2 ms with only loss.dc compiled, 3.78 GiB / 18.9 ms compiled as a whole
(eager: 9.30 GiB / 77.6 ms). At 160^3 the same change measured -35 % peak and 1.65x; on a 14 class
3d_fullres it is 0.59 -> 0.39 GiB.
This changes results very slightly, in a direction that is an improvement. The dice term is
bit-identical. Eager F.cross_entropy under autocast dispatches the fused ATen kernel, which is not
autocast-listed, so its internal log_softmax runs in fp16; torch.compile decomposes it and the
log_softmax gets the fp32 treatment, which is 4.3x closer to an fp64 reference. The memory saving and the
accuracy change have the same cause and cannot be separated. Magnitude: ~1e-7 relative on the loss, about
one ulp on the fp16 gradient.
Checked against Dice, not only against the loss. The previous and the new wiring were trained from scratch against each other on 8 datasets (ResEnc-L, fold 0, 1000 dense epochs, everything else identical): mean difference -0.14 Dice points, sd 0.53, p = 0.48, 95 % CI -0.59 .. +0.31, 3 better / 5 worse. That bounds any systematic shift well under ±0.6 points. End to end the change is worth ~1 % on few-class datasets and 11 % on a 127 class one, which is where the loss is a large part of the step.
nnUNetTrainer also gained two class attributes, dice_class and dc_and_ce_class, which
_build_loss instantiates instead of naming MemoryEfficientSoftDiceLoss / DC_and_CE_loss directly. The
defaults reproduce the previous behaviour exactly. They exist so a mixin can swap the loss implementation
without overriding _build_loss - see ChunkedLossMixin
(variants/loss/nnUNetTrainer_chunkedLoss.py), which selects the batch-chunked CE and dice needed once
batch * num_classes * voxels reaches PyTorch's int32 indexing limit in the nll_loss2d backward. If your
trainer overrides _build_loss, nothing changes; if you want the swap to reach it, read the attributes.
With compilation on, the compound loss that _build_loss returns (or wraps in DeepSupervisionWrapper) is now the
compiled module (torch._dynamo.OptimizedModule) instead of the DC_and_CE_loss itself, whose dc used to be
the compiled part. Attribute access is forwarded to the loss, but isinstance checks have to look at
._orig_mod.
For softmax training, nnUNetTrainer.validation_step computed the green line by scattering the argmax into
a (b, num_classes, *patch_size) one-hot and handing that to get_tp_fp_fn_tn. That one-hot is ~3.3 GB at
118 classes / 192^3 / bs 2, before get_tp_fp_fn_tn allocates on top, and it was the memory ceiling of the
validation step on datasets with many classes. It is replaced by get_tp_fp_fn_from_argmax
(nnunetv2/training/loss/dice.py), which accumulates the (num_classes, num_classes) confusion matrix over
(prediction, target) in one pass - 133 KB at 129 classes. It is exactly equivalent, including the ignore
label; region-based training is untouched and still uses get_tp_fp_fn_tn. Tests:
nnunetv2/tests/test_online_evaluation.py.
on_validation_epoch_end now wraps the Dice division in np.errstate instead of letting it warn. A class
absent from both prediction and target still gives 0/0 = nan and is still dropped by the np.nanmean, so
mean_fg_dice is unchanged - only the warning spam is gone. Custom trainers that read
dice_per_class_or_region should keep expecting nan there. For softmax training, tp_hard, fp_hard and
fn_hard are now integer counts (int64) instead of floats, with the same values.
Segmentation resampling: ties are broken differently, and the memory efficient branch is now the default
resample_torch_simple and resample_torch_fornnunet resample a segmentation by interpolating each
label's indicator and picking a winner per voxel. So do resample_data_or_seg and its two wrappers on
the default (non-torch) path, by way of batchgenerators' resize_segmentation; they take the same new
seg_tiebreak argument, with the same default. Whenever an axis is resampled by an even integer
factor, every boundary voxel is an exact 50/50 tie between two labels - in plane (512 -> 256) just as
much as out of plane. On the torch path those ties used to be resolved by argmax, which always returns
the first maximum and therefore always picked the smallest label, i.e. background for a normal
segmentation. That eroded foreground systematically and annihilated structures one voxel thick.
nnU-Net now requires batchgenerators >= 0.25.5 (resize_segmentation gained seg_tiebreak in 0.25.4,
and 0.25.5 makes it handle an empty segmentation again), and batchgeneratorsv2 >= 0.3.6, whose
SpatialTransform changes the augmented segmentations every training sees in two ways:
- The same tie-break,
seg_tiebreak='nearest'. nnU-Net passesbg_style_seg_sampling=False, whose ties used to go to the smallest label. Exact ties are rarer here than in preprocessing (about one voxel in a million under random rotation and scaling), but they are not zero. - Segmentations are padded with the padding label, then resampled, so they end at the outer edge of the
border pixels, like the image. A rotated or scaled patch that reaches past the loaded crop used to cut
labels half a pixel short there. Under nnU-Net's settings (
border_mode_seg='constant',padding_value_seg=-1) this turns -1 (later background) into the real label at crop edges: 0.64 % of the voxels of a patch that is both rotated and scaled, none in a patch that stays inside the crop.
Custom trainers that build their own SpatialTransform get both changes. seg_tiebreak='lowest' restores
the old tie direction; the padding change has no switch. bg_style_seg_sampling now defaults to False
there, which gives the same output as True and is faster.
Two new defaults:
seg_tiebreak='nearest'(new parameter). Undecided voxels now take the nearest neighbour label, which is the decision the separate-z path already made and is symmetric in the labels - provided it is one of the tied labels. Where three or more labels meet in 3D it can be a label that lost, and such a voxel keeps the smallest of the tied labels instead.'lowest'and'highest'keep the smallest or the largest label instead; all three are accepted on both paths, and an unknown value is rejected up front.memefficient_seg_resampling=True. The two seg implementations are now bit-identical, so this is purely a time/memory trade: the previous default scores all labels into onen_labels x voxels x 2byte buffer, this one holds a single running score. It is also the faster of the two now. Measured on CPU withseg_tiebreak='nearest', upsampling 128x256x256 to 160x320x320: 1.5x faster at 3 labels at the same peak memory, 2.8x faster at 20 labels (peak RSS 2.3 GB -> 0.9 GB), and 2.4x faster at 100 labels from 96x192x192 (7.3 GB -> 0.9 GB). Downsampling 200x512x512 to 100x256x256 it is 1.2x faster at 3 labels and 2.3x at 20. It handles every interpolation modeF.interpolatedoes.
This changes preprocessed segmentations, and with them what a model is trained on. Exported predictions of a single-stage configuration are not affected: they are resampled as probabilities and only argmaxed afterwards. The cascade is affected at inference: the low resolution stage's prediction is resampled to the full resolution grid as a segmentation, with this code, and fed to the next stage. Measured over 100 segmentations from 18 datasets, roundtripping each through its target grid and back: mean foreground Dice +0.010 at an even integer factor on the anisotropic axis (+0.012 at 4x, +0.005 in plane), foreground volume ratio 0.912 -> 0.998, and +0.027 Dice on structures under 1000 voxels. Where no exact ties occur - odd factors, and most real target spacings - the two are identical or within 0.0002 Dice.
Plans files do not pin seg_tiebreak and will therefore pick up the new tie-break. Plans from the
torch planners do pin memefficient_seg_resampling, so they keep the value they were created with; the
default planners' plans resample with resample_data_or_seg_to_shape, which has no such option. To get the
old results back, set "seg_tiebreak" in resampling_fn_seg_kwargs of your plans file:
- torch planners:
"lowest". Together with"memefficient_seg_resampling": false- what older plans files already say - this reproduces the old results bit for bit. - default planners (
nnUNetPlans, the ResEnc presets):"highest"is the closest, but nothing reproduces the old results exactly, because what they did was a bug. The old code compared each label against 0.5 rather than against the other labels, and left voxels that no label claimed at 0 (see below). Where no voxel went unclaimed,"highest"gives the old result at all but very rare voxels: floating-point rounding can let two labels both reach 0.5 with slightly different scores, and there the old code took the larger label while the argmax takes the higher score.
The memefficient_seg_resampling branch also had a bug worth knowing about if you ever enabled it: it
assigned labels with interp > 0.5, so a voxel no label claimed kept the zero the result was
initialized with. With labels {1, 2} that produced a 0 that does not occur in the input at all.
The default path had that same bug, in batchgenerators rather than here (fixed in 0.25.4). It bit harder
there: resize_segmentation defaults to spline order 3, and skimage's clip=True breaks the sum-to-one
property that the 0.5 threshold relies on, so on random four-label 2d data 6.2 % of output voxels had no
label at 0.5 or above. Ties among the labels that did pass 0.5 went to the largest label, not the
smallest, because the loop assigned in ascending label order and the last writer won - the opposite
direction from the bug above, and the reason the default path did not show the same foreground erosion.
The separate-z pass along the anisotropic axis did the same thing in nnU-Net itself whenever order_z was
not 0. All of these now take the per-voxel argmax with seg_tiebreak. The default plans use order_z=0
for segmentations, and that pass is unchanged.
seg_tiebreak is keyword-only everywhere, so it can be reordered later without breaking positional
callers.
resample_torch_fornnunet now honours its mode argument when it does not resample separately along z.
It used to drop it there and always interpolate linearly. No shipped planner sets mode, so this only
changes plans files that set it in their resampling kwargs.
In DDP, nnUNetTrainer.do_split now lets only global rank 0 read or create splits_final.json and broadcasts the
splits to the other ranks (dist.broadcast_object_list). Previously every rank could create the file at the same
time, and a rank could read a file that another rank was still writing. Because of the broadcast, do_split is now a
collective: if a custom trainer calls it on some ranks only (for example inside if self.global_rank == 0:), the job
hangs. Call it on all ranks. Trainers that override do_split entirely are not affected.
NaturalImage2DIO.write_seg now writes .tif / .tiff predictions with PackBits via tifffile instead of uncompressed skimage.io.imsave. Pixel values and dtypes are unchanged (still lossless). PNG/BMP writes and Tiff3DIO (zlib) are unchanged. Downstream tools that digest the raw TIFF byte stream (not just the label map) will see a different container.
NaturalImage2DIO also claims .tiff and is listed before Tiff3DIO. With an example file, registry selection still falls through to Tiff3DIO for non-RGB 3D volumes. Calling determine_reader_writer_from_file_ending('.tiff', None) (no example file) now returns NaturalImage2DIO instead of Tiff3DIO; pass an example file when you need 3D disambiguation.
nnU-Net now supports DDP across several nodes (see Multi-GPU training). To make that work,
nnUNetTrainer distinguishes three quantities where it previously only had self.local_rank:
| attribute | meaning |
|---|---|
self.local_rank |
index of this process within its node. This is the CUDA device index. |
self.global_rank |
index of this process within the whole job. Exactly one process in the job has 0. |
self.world_size |
number of processes in the job |
self.local_world_size |
number of processes on this node |
On a single node local_rank == global_rank, so nothing changes. Across nodes they differ, and every check that
decides who writes to nnUNet_results must use global_rank — there is one local_rank == 0 per node, and on a
shared filesystem they would all write the same log file, checkpoints and progress plot. All in-tree trainers were
converted. If you maintain a custom trainer, replace if self.local_rank == 0: with if self.global_rank == 0:
wherever it guards output, and keep local_rank only where you mean the GPU.
AllGatherGrad (nnunetv2.utilities.ddp_allgather) is deprecated and will be removed in a future release.
nnU-Net's losses now use AllReduceGrad (nnunetv2.utilities.ddp), which computes the same forward and
backward while moving world_size times less data: AllGatherGrad.apply(x).sum(0) is exactly
AllReduceGrad.apply(x). Importing AllGatherGrad still works but raises a DeprecationWarning.
-
We now support hierarchical labels (named regions in nnU-Net). For example, instead of training BraTS with the 'edema', 'necrosis' and 'enhancing tumor' labels you can directly train it on the target areas 'whole tumor', 'tumor core' and 'enhancing tumor'. See here for a detailed description + also have a look at the BraTS 2021 conversion script.
-
Cross-platform support. Cuda, mps (Apple M1/M2) and of course CPU support! Simply select the device with
-deviceinnnUNetv2_trainandnnUNetv2_predict. -
Unified trainer class: nnUNetTrainer. No messing around with cascaded trainer, DDP trainer, region-based trainer, ignore trainer etc. All default functionality is in there!
-
Supports more input/output data formats through ImageIO classes.
-
I/O formats can be extended by implementing new Adapters based on
BaseReaderWriter. -
The nnUNet_raw_cropped folder no longer exists -> saves disk space at no performance penalty. magic! (no jk the saving of cropped npz files was really slow, so it's actually faster to crop on the fly).
-
Preprocessed data and segmentation are stored in different files when unpacked. Seg is stored as int8 and thus takes 1/4 of the disk space per pixel (and I/O throughput) as in v1.
-
Native support for multi-GPU (DDP) TRAINING. Multi-GPU INFERENCE should still be run with
CUDA_VISIBLE_DEVICES=X nnUNetv2_predict [...] -num_parts Y -part_id X. There is no cross-GPU communication in inference, so it doesn't make sense to add additional complexity with DDP. -
All nnU-Net functionality is now also accessible via API. Check the corresponding entry point in
setup.pyto see what functions you need to call. -
Dataset fingerprint is now explicitly created and saved in a json file (see nnUNet_preprocessed).
-
Complete overhaul of plans files (read also this:
- Plans are now .json and can be opened and read more easily
- Configurations are explicitly named ("3d_fullres" , ...)
- Configurations can inherit from each other to make manual experimentation easier
- A ton of additional functionality is now included in and can be changed through the plans, for example normalization strategy, resampling etc.
- Stages of the cascade are now explicitly listed in the plans. 3d_lowres has 'next_stage' (which can also be a list of configurations!). 3d_cascade_fullres has a 'previous_stage' entry. By manually editing plans files you can now connect anything you want, for example 2d with 3d_fullres or whatever. Be wild! (But don't create cycles!)
- Multiple configurations can point to the same preprocessed data folder to save disk space. Careful! Only configurations that use the same spacing, resampling, normalization etc. should share a data source! By default, 3d_fullres and 3d_cascade_fullres share the same data
- Any number of configurations can be added to the plans (remember to give them a unique "data_identifier"!)
Folder structures are different and more user-friendly:
- nnUNet_preprocessed
- By default, preprocessed data is now saved as:
nnUNet_preprocessed/DATASET_NAME/PLANS_IDENTIFIER_CONFIGURATIONto clearly link them to their corresponding plans and configuration - Name of the folder containing the preprocessed images can be adapted with the
data_identifierkey.
- By default, preprocessed data is now saved as:
- nnUNet_results
- Results are now sorted as follows: DATASET_NAME/TRAINERCLASS__PLANSIDENTIFIER__CONFIGURATION/FOLD
- Integration into MONAI (together with our friends at Nvidia)
- New pretrained weights for a large number of datasets (coming very soon))