StereoScan: Dense 3d reconstruction in real-time

TLDR

Accurate 3‑D perception from video sequences is essential for computer vision and robotics, yet online constraints limit camera resolution and reconstruction accuracy, and heavy parallelism required for real‑time processing hampers deployment on mobile devices or driver‑assistance systems without FPGAs. This work introduces a novel real‑time method for constructing dense 3‑D maps from high‑resolution stereo sequences. The pipeline combines a sparse feature matcher with a robust visual odometry algorithm, efficient stereo matching, and a multi‑view linking scheme to generate consistent 3‑D point clouds. Experiments show the odometry achieves state‑of‑the‑art accuracy, runs at 25 fps, and produces depth maps at 3–4 fps, enabling online 3‑D reconstruction.

Abstract

Accurate 3d perception from video sequences is a core subject in computer vision and robotics, since it forms the basis of subsequent scene analysis. In practice however, online requirements often severely limit the utilizable camera resolution and hence also reconstruction accuracy. Furthermore, real-time systems often rely on heavy parallelism which can prevent applications in mobile devices or driver assistance systems, especially in cases where FPGAs cannot be employed. This paper proposes a novel approach to build 3d maps from high-resolution stereo sequences in real-time. Inspired by recent progress in stereo matching, we propose a sparse feature matcher in conjunction with an efficient and robust visual odometry algorithm. Our reconstruction pipeline combines both techniques with efficient stereo matching and a multi-view linking scheme for generating consistent 3d point clouds. In our experiments we show that the proposed odometry method achieves state-of-the-art accuracy. Including feature matching, the visual odometry part of our algorithm runs at 25 frames per second, while - at the same time - we obtain new depth maps at 3-4 fps, sufficient for online 3d reconstructions.