Three-Dimensional Sign Language Recognition With Angular Velocity Maps and Connived Feature ResNet

Abstract

The objective of this letter is to design a unique spatio-temporal feature map characterization for three-dimensional (3-D) sign (or action) data. Current maps characterize geometric features, such as joint distances and angles or both, which could not accurately model the relative joint variations in a 3-D sign (or action) location data. Therefore, we propose a new color-coded feature map called joint angular velocity maps to accurately model the 3-D joint motions. Instead of using traditional convolutional neural networks (CNNs), we propose to develop a new ResNet architecture called connived feature ResNet, which has a CNN layer in the feedforward loop of the densely connected standard ResNet architecture. We show that this architecture avoids using dropout in the last layers and achieves the desired goal in less number of iterations compared to other ResNet and CNN based architectures used for sign (action) classification. To test our proposed model, we use our own motion captured 3-D sign language data (BVC3DSL) and other publicly available skeletal action data: CMU, HDM05, and NTU RGBD.