Real-time onboard visual-inertial state estimation and self-calibration of MAVs in unknown environments

TLDR

Visual‑inertial fusion is popular for MAV navigation because of its low weight, power, and cost, yet real‑time onboard processing is hindered by latency between inertial and visual data, prompting most systems to use off‑board computation. We present an onboard, real‑time navigation algorithm for a MAV equipped with a single camera and IMU. The algorithm introduces a speed‑estimation module that turns the camera into a metric body‑speed sensor using IMU data within an EKF, enabling full self‑calibration, initialization, and fallback for a keyframe‑based VSLAM that has been extended for scalable 6‑DoF pose estimation and relies on optical flow of at least two features and IMU readings for fast speed control. Nonlinear observability analysis and real experiments confirm that the method can control MAV speed, achieving 40 Hz operation on an onboard 1.6 GHz Atom computer.

Abstract

The combination of visual and inertial sensors has proved to be very popular in robot navigation and, in particular, Micro Aerial Vehicle (MAV) navigation due the flexibility in weight, power consumption and low cost it offers. At the same time, coping with the big latency between inertial and visual measurements and processing images in real-time impose great research challenges. Most modern MAV navigation systems avoid to explicitly tackle this by employing a ground station for off-board processing. In this paper, we propose a navigation algorithm for MAVs equipped with a single camera and an Inertial Measurement Unit (IMU) which is able to run onboard and in real-time. The main focus here is on the proposed speed-estimation module which converts the camera into a metric body-speed sensor using IMU data within an EKF framework. We show how this module can be used for full self-calibration of the sensor suite in real-time. The module is then used both during initialization and as a fall-back solution at tracking failures of a keyframe-based VSLAM module. The latter is based on an existing high-performance algorithm, extended such that it achieves scalable 6DoF pose estimation at constant complexity. Fast onboard speed control is ensured by sole reliance on the optical flow of at least two features in two consecutive camera frames and the corresponding IMU readings. Our nonlinear observability analysis and our real experiments demonstrate that this approach can be used to control a MAV in speed, while we also show results of operation at 40Hz on an onboard Atom computer 1.6 GHz.