Air-Writing Recognition using Deep Convolutional and Recurrent Neural Network Architectures

Abstract

In this paper, we explore deep learning architectures applied to the air-writing recognition problem where a person writes text freely in the three dimensional space. We focus on handwritten digits, namely from 0 to 9, which are structured as multidimensional time-series acquired from a Leap Motion Controller (LMC) sensor. We examine both dynamic and static approaches to model the motion trajectory. We train and compare several state-of-the-art convolutional and recurrent architectures. Specifically, we employed a Long Short-Term Memory (LSTM) network and also its bidirectional counterpart (BLSTM) in order to map the input sequence to a vector of fixed dimensionality, which is subsequently passed to a dense layer for classification among the targeted air-handwritten classes. In the second architecture we adopt 1D Convolutional Neural Networks (CNNs) to encode the input features before feeding them to an LSTM neural network (CNN-LSTM). The third architecture is a Temporal Convolutional Network (TCN) that uses dilated causal convolutions. Finally, a deep CNN architecture for automating the feature learning and classification from raw input data is presented. The performance evaluation has been carried out on a dataset of 10 participants, who wrote each digit at least 10 times, resulting in almost 1200 examples.