Deep Continuous Fusion for Multi-Sensor 3D Object Detection

Ming Liang, Bin Yang, Shenlong Wang, Raquel Urtasun

arXiv (Cornell University) · 2020 · 431 citations · 28 references

DOIFull text

Open access

Concepts

TL;DR

The paper proposes a novel end‑to‑end 3D object detector that fuses LIDAR and camera data using continuous convolutions for highly accurate localization. The method introduces a continuous fusion layer that encodes discrete image features and continuous geometric information, enabling an efficient end‑to‑end multi‑sensor 3D detector. Experiments on KITTI and a large‑scale benchmark demonstrate significant performance gains over the state of the art.

Abstract

In this paper, we propose a novel 3D object detector that can exploit both LIDAR as well as cameras to perform very accurate localization. Towards this goal, we design an end-to-end learnable architecture that exploits continuous convolutions to fuse image and LIDAR feature maps at different levels of resolution. Our proposed continuous fusion layer encode both discrete-state image features as well as continuous geometric information. This enables us to design a novel, reliable and efficient end-to-end learnable 3D object detector based on multiple sensors. Our experimental evaluation on both KITTI as well as a large scale 3D object detection benchmark shows significant improvements over the state of the art.

References

28