Combining Fully Convolutional and Recurrent Neural Networks for 3D Biomedical Image Segmentation

TLDR

Segmentation of 3D biomedical images is a fundamental problem, and deep learning methods have achieved state‑of‑the‑art performance, yet existing 3D, 2D‑plane, and LSTM approaches struggle with the highly anisotropic dimensions common in such data. The authors aim to develop a new deep‑learning framework that combines a fully convolutional network and a recurrent neural network to exploit intra‑slice and inter‑slice context for 3D segmentation. The framework integrates a fully convolutional network for intra‑slice context and a recurrent neural network for inter‑slice context, jointly modeling 3D structure to address anisotropy. This first DL framework explicitly leverages 3D anisotropy and, evaluated on the ISBI Neuronal Structure Segmentation Challenge and in‑house fungal stacks, achieves promising results compared to existing DL‑based 3D segmentation methods.

Abstract

Segmentation of 3D images is a fundamental problem in biomedical image analysis. Deep learning (DL) approaches have achieved state-of-the-art segmentation perfor- mance. To exploit the 3D contexts using neural networks, known DL segmentation methods, including 3D convolution, 2D convolution on planes orthogonal to 2D image slices, and LSTM in multiple directions, all suffer incompatibility with the highly anisotropic dimensions in common 3D biomedical images. In this paper, we propose a new DL framework for 3D image segmentation, based on a com- bination of a fully convolutional network (FCN) and a recurrent neural network (RNN), which are responsible for exploiting the intra-slice and inter-slice contexts, respectively. To our best knowledge, this is the first DL framework for 3D image segmentation that explicitly leverages 3D image anisotropism. Evaluating using a dataset from the ISBI Neuronal Structure Segmentation Challenge and in-house image stacks for 3D fungus segmentation, our approach achieves promising results comparing to the known DL-based 3D segmentation approaches.