2020 · 524 citations · 31 references
ComputingAutonomous SystemsComputer ScienceComputer VisionMachine LearningRobot VisionImage FusionObject DetectionObject RecognitionPoint Cloud ProcessingRobotic VisionScene Understanding3D Object Detection FusionConvolutional Neural NetworksModal FusionMulti-modal PerceptionMulti-modal Sensor FusionNetwork FusionNon-maximum SuppressionSensor-level Fusion
Recent neural network advances have improved 3D LiDAR and 2D video object detection, yet effectively fusing both modalities remains challenging. The paper proposes the CLOCs fusion network to combine camera and LiDAR detections and will release the code. CLOCs fuses camera and LiDAR candidate outputs before NMS, using geometric and semantic consistency to enhance 3D and 2D detection with low complexity. On the KITTI benchmark, CLOCs achieves significant gains—particularly at long range—and tops the fusion-based leaderboard.
There have been significant advances in neural networks for both 3D object detection using LiDAR and 2D object detection using video. However, it has been surprisingly difficult to train networks to effectively use both modalities in a way that demonstrates gain over single-modality networks. In this paper, we propose a novel Camera-LiDAR Object Candidates (CLOCs) fusion network. CLOCs fusion provides a low-complexity multi-modal fusion framework that significantly improves the performance of single-modality detectors. CLOCs operates on the combined output candidates before Non-Maximum Suppression (NMS) of any 2D and any 3D detector, and is trained to leverage their geometric and semantic consistencies to produce more accurate final 3D and 2D detection results. Our experimental evaluation on the challenging KITTI object detection benchmark, including 3D and bird's eye view metrics, shows significant improvements, especially at long distance, over the state-of-the-art fusion based methods. At time of submission, CLOCs ranks the highest among all the fusion-based methods in the official KITTI leaderboard. We will release our code upon acceptance.
31
Feature Pyramid Networks for Object Detection
Tsung-Yi Lin, Piotr Dollár, Ross Girshick et al. · 2017 · 28.7K citations
Focal Loss for Dense Object Detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick et al. · 2017 · 25.7K citations
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick et al. · arXiv (Cornell University) · 2015 · 18.2K citations · Full text
Are we ready for autonomous driving? The KITTI vision benchmark suite
Andreas Geiger, P Lenz, R. Urtasun · 2012 · 14.4K citations