Multi-Modal 3D Object Detection in Autonomous Driving: A Survey and Taxonomy

TLDR

Autonomous vehicles rely on continuous perception, and 3D object detection—using cameras and LiDARs—is essential for identifying surrounding objects, but single‑modal approaches suffer performance gaps that motivate multi‑modal fusion, yet a comprehensive analysis of such methods is lacking. This survey reviews recent multi‑modal 3D object detection techniques, categorizes them, compares their strengths and weaknesses, and outlines current challenges and future research directions. The authors analyze datasets and metrics, classify fusion strategies by feature representation, alignment, and fusion stage, and benchmark their performance across mainstream datasets to reveal implementation insights.

Abstract

Autonomous vehicles require constant environmental perception to obtain the distribution of obstacles to achieve safe driving. Specifically, 3D object detection is a vital functional module as it can simultaneously predict surrounding objects' categories, locations, and sizes. Generally, autonomous vehicles are equipped with multiple sensors, including cameras and LiDARs. The fact that single-modal methods suffer from unsatisfactory detection performance motivates utilizing multiple modalities as inputs to compensate for single sensor faults. Although many multi-modal fusion detection algorithms exist, there is still a lack of comprehensive and in-depth analysis of these methods to clarify how to fuse multi-modal data effectively. Therefore, this paper surveys recent advancements in fusion detection methods. First, we present the broad background of multi-modal 3D object detection and identify the characteristics of widely used datasets along with their evaluation metrics. Second, instead of the traditional classification method of early, middle, and late fusion, we categorize and analyze all fusion methods from three aspects: feature representation, alignment, and fusion, which reveals how these fusion methods are implemented in an essential way. Third, we provide an in-depth comparison of their pros and cons and compare their performance in mainstream datasets. Finally, we further summarize current challenges and research trends for realizing the full potential of multi-modal 3D object detection.