2023 · 71 citations · 32 references
ComputingComputer ScienceComputer VisionMachine VisionRobot Vision3D Imaging3D ReconstructionGeometry ProcessingPoint Cloud ProcessingScene ReconstructionScene Understanding3D Scene Representation3D Scene UnderstandingPoint Cloud RepresentationPoint Cloud Representation LearningPoint Cloud Scene Understanding
Contrastive Language-Image Pre-training, benefiting from large-scale unlabeled text-image pairs, has demonstrated great performance in open-world vision understanding tasks. However, due to the limited Text-3D data pairs, adapting the success of 2D Vision-Language Models (VLM) to the 3D space remains an open problem. Existing works that leverage VLM for 3D understanding generally resort to constructing intermediate 2D representations for the 3D data, but at the cost of losing 3D geometry information. To take a step toward open-world 3D vision understanding, we propose Contrastive Language-Image-Point Cloud Pretraining (CLIP <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) to directly learn the transferable 3D point cloud representation in realistic scenarios with a novel proxy alignment mechanism. Specifically, we exploit naturally-existed correspondences in 2D and 3D scenarios, and build well-aligned and instance-based text-image-point proxies from those complex scenarios. On top of that, we propose a cross-modal contrastive objective to learn semantic and instance-level aligned point cloud representation. Experimental results on both indoor and outdoor scenarios show that our learned 3D representation has great transfer ability in downstream tasks, including zero-shot and few-shot 3D recognition, which boosts the state-of-the-art methods by large margins. Furthermore, we provide analyses of the capability of different representations in real scenarios and present the optional ensemble scheme.
32
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Raffaelli Charles, Hao Su, Kaichun Mo et al. · 2017 · 9.8K citations
Carsten Rother, Vladimir Kolmogorov, Andrew Blake · ACM Transactions on Graphics · 2004 · 5.8K citations · Full text
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Angela Dai, Manolis Savva, Maciej Halber et al. · 2017 · 4K citations
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla et al. · 2020 · 3K citations