MPVF: 4D Medical Image Inpainting by Multi-Pyramid Voxel Flows

Abstract

Generatinga detailed 4D medical image usually accompanies with prolonged examination time and increased radiation exposure risk. Modern deep learning solutions have exploited interpolation mechanisms to generate a complete 4D image with fewer 3D volumes. However, existing solutions focus more on 2D-slice information, thus missing the changes on the z-axis. This article tackles the 4D cardiac and lung image interpolation problem by synthesizing 3D volumes directly. Although heart and lung only account for a fraction of chest, they constantly undergo periodical motions of varying magnitudes in contrast to the rest of the chest volume, which is more stationary. This poses big challenges to existing models. In order to handle various magnitudes of motions, we propose a Multi-Pyramid Voxel Flows (MPVF) model that takes multiple multi-scale voxel flows into account. This renders our generation network rich information during interpolation, both globally and regionally. Focusing on periodic medical imaging, MPVF takes the maximal and the minimal phases of an organ motion cycle as inputs and can restore a 3D volume at any time point in between. MPVF is featured by a Bilateral Voxel Flow (BVF) module for generating multi-pyramid voxel flows in an unsupervised manner and a Pyramid Fusion (PyFu) module for fusing multiple pyramids of 3D volumes. The model is validated to outperform the state-of-the-art model in several indices with significantly less synthesis time.