Deep Learning Based Hand Gesture Recognition and UAV Flight Controls

Abstract

Dynamic hand gesture recognition is desired as an alternative means for human-computer interactions. This paper presents a hand gesture recognition system that is designed for the control of flights of unmanned aerial vehicles (UAV). To train the system to recognize designed gestures, skeleton data collected from a Leap Motion Controller are converted to two different data models. As many as 9124 samples of training dataset, 1938 samples of testing dataset are created to train and test the proposed three deep learning neural networks, which are a 2-layer fully connected neural network, a 5-layer fully connected neural network and an 8-layer convolutional neural network. The static testing results show that the 2-layer fully connected neural network achieves an average accuracy of 98.2% on normalized datasets and 11% on raw datasets. The 5-layer fully connected neural network achieves an average accuracy of 95.2% on normalized datasets and 45% on raw datasets. The 8-layers convolutional neural network achieves an average accuracy of 96.2% on normalized datasets and raw datasets. Testing on a drone-kit simulator and a real drone shows that this system is feasible for drone flight controls.