FlowNet3D: Learning Scene Flow in 3D Point Clouds

TLDR

Scene flow describes the 3D motion of points in dynamic environments; while most prior work estimates it from stereo or RGB‑D images, few methods learn it directly from point clouds. This paper introduces FlowNet3D, a deep neural network that learns scene flow from point clouds in an end‑to‑end fashion. The network jointly learns hierarchical point‑cloud features and flow embeddings using two novel learning layers for point sets, and is evaluated on synthetic FlyingThings3D data and real KITTI Lidar scans. When trained solely on synthetic data, FlowNet3D generalizes to real scans, surpassing baselines and matching prior state‑of‑the‑art performance, and its output supports scan registration and motion segmentation.

Abstract

Many applications in robotics and human-computer interaction can benefit from understanding 3D motion of points in a dynamic environment, widely noted as scene flow. While most previous methods focus on stereo and RGB-D images as input, few try to estimate scene flow directly from point clouds. In this work, we propose a novel deep neural network named FlowNet3D that learns scene flow from point clouds in an end-to-end fashion. Our network simultaneously learns deep hierarchical features of point clouds and flow embeddings that represent point motions, supported by two newly proposed learning layers for point sets. We evaluate the network on both challenging synthetic data from FlyingThings3D and real Lidar scans from KITTI. Trained on synthetic data only, our network successfully generalizes to real scans, outperforming various baselines and showing competitive results to the prior art. We also demonstrate two applications of our scene flow output (scan registration and motion segmentation) to show its potential wide use cases.