Combined frame- and event-based detection and tracking

TLDR

The study presents an object‑tracking algorithm for moving platforms that leverages the DAVIS sensor. The algorithm fuses APS and DVS data, first generating ROIs from DVS events, then classifying them with a CNN on APS frames, and finally applying a particle filter to estimate the target location. The method achieves a 70‑fold speedup over full‑frame convolution, 90 % tracking accuracy on a predator‑prey robot dataset, and runs in under 20 ms per frame on a standard PC without a GPU.

Abstract

This paper reports an object tracking algorithm for a moving platform using the dynamic and active-pixel vision sensor (DAVIS). It takes advantage of both the active pixel sensor (APS) frame and dynamic vision sensor (DVS) event outputs from the DAVIS. The tracking is performed in a three step-manner: regions of interest (ROIs) are generated by a cluster-based tracking using the DVS output, likely target locations are detected by using a convolutional neural network (CNN) on the APS output to classify the ROIs as foreground and background, and finally a particle filter infers the target location from the ROIs. Doing convolution only in the ROIs boosts the speed by a factor of 70 compared with full-frame convolutions for the 240×180 frame input from the DAVIS. The tracking accuracy on a predator and prey robot database reaches 90% with a cost of less than 20ms/frame in Matlab on a normal PC without using a GPU.