FS-Net: LiDAR-Camera Fusion With Matched Scale for 3D Object Detection in Autonomous Driving

Abstract

As a key task in autonomous driving, 3D object detection based on LiDAR-camera fusion is expected to achieve more robust results by the complementarity of the two sensors. However, LiDAR-camera fusion is non-trivial. An existing problem for this type of detector is that the scale and receptive field of LiDAR point features and image features are not matched, leading to information deficiency or redundancy in fusion. This paper proposes a Point-based Pyramid Attention Fusion (PPAF) module for LiDAR-camera fusion to solve the problem. The PPAF module learns corresponding image features of LiDAR points with a matched scale based on the image feature pyramid and attention mechanism for a better effect of fusion. Furthermore, based on the PPAF module, a new LiDAR-camera fusion-based 3D object detector named FS-Net is proposed, a two-stage detector with LiDAR voxel-based RPN and refinement network based on enriched LiDAR-camera features. Experiments on two public datasets demonstrate the effectiveness of our approach.