Fully convolutional structured LSTM networks for joint 4D medical image segmentation

Abstract

Longitudinal medical image analysis has great potential to reveal developmental trajectories and monitor disease progression. This process relies on consistent and robust joint 4D segmentation. Traditional methods highly depend on the similarity of images over time and either build a template or assume the images could be co-registered. This process may fail when image sequences present major appearance changes. Recently, deep learning (DL) approaches have achieved state-of-the-art results for related challenges in computer vision. These approaches make use of models such as fully convolutional networks (FCNs) for end-to-end pixel-wise segmentation and recurrent neural networks (RNNs) with long short-term memory (LSTM) units for sequence-to-sequence modeling. In this paper, we propose a new DL framework called FCSLSTM for 4D image segmentation with FCNs for the spatial model and LSTM for the temporal model. This is the first DL framework with deep integration of FCNs and LSTM for joint 4D segmentation that could be trained end-to-end. Our approach achieves promising results with the demonstrated application to longitudinal pediatric magnetic resonance imaging (MRI) segmentation.