2023 · 282 citations · 31 references
ComputingComputer ScienceRobotics EngineeringAutonomous VehiclesComputer VisionSensor FusionDepth EstimationFeature-level FusionLate FusionLidar PerceptionVehicle PerceptionDecoder DesignFusion Of Heterogeneous SensorsMulti-modal PerceptionMulti-modal Sensor FusionMultimodal Sensor FusionObject Detection In Autonomous DrivingPerception SystemsPost-processing
Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) surpasses state-of-the-art 3D detection model [41] (56.6 mAP) using a 32-beam LiDAR. Our code is available on the ${\text{project page}}$.
31
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Feature Pyramid Networks for Object Detection
Tsung-Yi Lin, Piotr Dollár, Ross Girshick et al. · 2017 · 28.7K citations
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Raffaelli Charles, Hao Su, Kaichun Mo et al. · 2017 · 9.8K citations
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter · arXiv (Cornell University) · 2017 · 9.1K citations · Full text
Dynamic Graph CNN for Learning on Point Clouds
Yue Wang, Yongbin Sun, Ziwei Liu et al. · ACM Transactions on Graphics · 2019 · 6.7K citations · Full text