Training CNNs for 3-D Sign Language Recognition With Color Texture Coded Joint Angular Displacement Maps

Abstract

Convolutional neural networks (CNNs) can be remarkably effective for recognizing two-dimensional and three-dimensional (3-D) actions. To further explore the potential of CNNs, we applied them in the recognition of 3-D motion-captured sign language (SL). The sign's 3-D spatio-temporal information of each sign was interpreted using joint angular displacement maps (JADMs), which encode the sign as a color texture image; JADMs were calculated for all joint pairs. Multiple CNN layers then capitalized on the differences between these images and identify discriminative spatio-temporal features. We then compared the performance of our proposed model against those of the state-of-the-art baseline models by using our own 3-D SL dataset and two other benchmark action datasets, namely, HDM05 and CMU.