Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

TLDR

Dynamic hand gesture detection and classification in real‑world human‑computer interaction systems is difficult due to high performer variability and the need for online, even pre‑gesture, classification to provide instantaneous feedback. The paper proposes a recurrent 3D convolutional neural network that simultaneously detects and classifies dynamic hand gestures from multimodal depth, color, and stereo‑IR data. The system employs a recurrent 3D CNN trained with connectionist temporal classification on unsegmented multimodal streams, and its performance is validated on a newly collected dataset featuring depth, color, and stereo‑IR sensors. The proposed system attains 83.8 % accuracy on the new dataset, surpassing existing state‑of‑the‑art methods and approaching human performance, and also sets new state‑of‑the‑art results on the SKIG and ChaLearn2014 benchmarks.

Abstract

Automatic detection and classification of dynamic hand gestures in real-world systems intended for human computer interaction is challenging as: 1) there is a large diversity in how people perform gestures, making detection and classification difficult, 2) the system must work online in order to avoid noticeable lag between performing a gesture and its classification, in fact, a negative lag (classification before the gesture is finished) is desirable, as feedback to the user can then be truly instantaneous. In this paper, we address these challenges with a recurrent three-dimensional convolutional neural network that performs simultaneous detection and classification of dynamic hand gestures from multi-modal data. We employ connectionist temporal classification to train the network to predict class labels from inprogress gestures in unsegmented input streams. In order to validate our method, we introduce a new challenging multimodal dynamic hand gesture dataset captured with depth, color and stereo-IR sensors. On this challenging dataset, our gesture recognition system achieves an accuracy of 83:8%, outperforms competing state-of-the-art algorithms, and approaches human accuracy of 88:4%. Moreover, our method achieves state-of-the-art performance on SKIG and ChaLearn2014 benchmarks.