UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation

TLDR

Medical image segmentation models are dominated by U‑Net and FCN variants, yet they suffer from an unknown optimal depth and overly restrictive skip‑connection fusion that limits multi‑scale feature aggregation. This work introduces UNet++ to address these issues by enabling efficient depth ensemble, flexible multi‑scale skip connections, and inference acceleration. UNet++ achieves this through a nested U‑Net architecture that shares an encoder across varying depths with deep supervision, redesigns skip connections to fuse features across scales in the decoder, and applies a pruning strategy to speed inference. Across six CT, MRI, and EM datasets, UNet++ consistently surpasses baseline U‑Nets for semantic segmentation, improves segmentation of objects of different sizes, enhances Mask R‑CNN performance for instance segmentation, and pruned models deliver significant speedups with only modest accuracy loss, with code available on GitHub.

Abstract

The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance segmentation, by (1) alleviating the unknown network depth with an efficient ensemble of U-Nets of varying depths, which partially share an encoder and co-learn simultaneously using deep supervision; (2) redesigning skip connections to aggregate features of varying semantic scales at the decoder sub-networks, leading to a highly flexible feature fusion scheme; and (3) devising a pruning scheme to accelerate the inference speed of UNet++. We have evaluated UNet++ using six different medical image segmentation datasets, covering multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and electron microscopy (EM), and demonstrating that (1) UNet++ consistently outperforms the baseline models for the task of semantic segmentation across different datasets and backbone architectures; (2) UNet++ enhances segmentation quality of varying-size objects-an improvement over the fixed-depth U-Net; (3) Mask RCNN++ (Mask R-CNN with UNet++ design) outperforms the original Mask R-CNN for the task of instance segmentation; and (4) pruned UNet++ models achieve significant speedup while showing only modest performance degradation. Our implementation and pre-trained models are available at https://github.com/MrGiovanni/UNetPlusPlus.