NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video

TLDR

NeuralRecon is a novel framework for real‑time 3D scene reconstruction from monocular video. The method reconstructs local surfaces as sparse TSDF volumes per fragment using a neural network with a gated‑recurrent TSDF fusion module that sequentially integrates features from previous fragments. Experiments on ScanNet and 7‑Scenes demonstrate that NeuralRecon outperforms state‑of‑the‑art methods in accuracy and speed, being the first learning‑based system to produce dense, coherent 3D geometry in real time. Code is available at https://zju3dv.github.io/neuralrecon/.

Abstract

We present a novel framework named NeuralRecon for real-time 3D scene reconstruction from a monocular video. Unlike previous methods that estimate single-view depth maps separately on each key-frame and fuse them later, we propose to directly reconstruct local surfaces represented as sparse TSDF volumes for each video fragment sequentially by a neural network. A learning-based TSDF fusion module based on gated recurrent units is used to guide the network to fuse features from previous fragments. This de-sign allows the network to capture local smoothness prior and global shape prior of 3D surfaces when sequentially reconstructing the surfaces, resulting in accurate, coherent, and real-time surface reconstruction. The experiments on ScanNet and 7-Scenes datasets show that our system outperforms state-of-the-art methods in terms of both ac-curacy and speed. To the best of our knowledge, this is the first learning-based system that is able to reconstruct dense coherent 3D geometry in real-time. Code is available at the project page: https://zju3dv.github.io/neuralrecon/.