DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection

TLDR

Lidars and cameras provide complementary information for 3D detection in autonomous driving, but aligning their augmented and aggregated features remains a key challenge. The paper proposes two novel techniques—InverseAug and LearnableAlign—to improve geometric alignment and feature correlation in multi‑modal 3D detection. The DeepFusion framework implements InverseAug and LearnableAlign, with code released at https://github.com/tensorflow/lingvo. By fusing camera features with deep lidar features, DeepFusion outperforms prior methods, improving baseline models on pedestrian detection by 6.7, 8.9, and 6.2 APH, achieving state‑of‑the‑art results on Waymo Open Dataset and demonstrating robustness to corruptions and out‑of‑distribution data.

Abstract

Lidars and cameras are critical sensors that provide complementary information for 3D detection in autonomous driving. While prevalent multi-modal methods [34], [36] simply decorate raw lidar point clouds with camera features and feed them directly to existing 3D detection models, our study shows that fusing camera features with deep lidar features instead of raw points, can lead to better performance. However, as those features are often augmented and aggregated, a key challenge in fusion is how to effectively align the transformed features from two modalities. In this paper, we propose two novel techniques: InverseAug that inverses geometric-related augmentations, e.g., rotation, to enable accurate geometric alignment between lidar points and image pixels, and LearnableAlign that leverages cross-attention to dynamically capture the correlations between image and lidar features during fusion. Based on InverseAug and LearnableAlign, we develop a family of generic multi-modal 3D detection models named DeepFusion, which is more accurate than previous methods. For example, DeepFusion improves Point-Pillars, CenterPoint, and 3D-MAN baselines on Pedestrian detection for 6.7,8.9, and 6.2 LEVEL_2 APH, respectively. Notably, our models achieve state-of-the-art performance on Waymo Open Dataset, and show strong model robustness against input corruptions and out-of-distribution data. Code will be publicly available at https://github.com/tensorflow/lingvo.