Proceedings of the AAAI Conference on Artificial Intelligence · 2018 · 425 citations · 37 references
ComputingComputer ScienceAudiovisual TranslationComputational LinguisticsNatural Language ProcessingAttention NetworksSpeech RecognitionAutomatic Speech TranslationContinuous Sign Language RecognitionLabelingLatent Space ExplorationLatent Space InterpretationSign Language RecognitionTemporal Segmentation
Millions of hearing‑impaired people rely on sign language, yet existing continuous sign‑language recognition systems depend on error‑prone temporal segmentation and labor‑intensive word‑level labeling. The authors propose a continuous sign‑language recognition framework that removes the need for temporal segmentation. The framework, called LS‑HAN, uses a two‑stream CNN to extract video features, maps them into a latent space to bridge semantic gaps, and applies a hierarchical attention network for recognition, evaluated on two large datasets. Experimental results show that LS‑HAN effectively improves continuous sign‑language recognition without temporal segmentation.
Millions of hearing impaired people around the world routinely use some variants of sign languages to communicate, thus the automatic translation of a sign language is meaningful and important. Currently, there are two sub-problems in Sign Language Recognition (SLR), i.e., isolated SLR that recognizes word by word and continuous SLR that translates entire sentences. Existing continuous SLR methods typically utilize isolated SLRs as building blocks, with an extra layer of preprocessing (temporal segmentation) and another layer of post-processing (sentence synthesis). Unfortunately, temporal segmentation itself is non-trivial and inevitably propagates errors into subsequent steps. Worse still, isolated SLR methods typically require strenuous labeling of each word separately in a sentence, severely limiting the amount of attainable training data. To address these challenges, we propose a novel continuous sign recognition framework, the Hierarchical Attention Network with Latent Space (LS-HAN), which eliminates the preprocessing of temporal segmentation. The proposed LS-HAN consists of three components: a two-stream Convolutional Neural Network (CNN) for video feature representation generation, a Latent Space (LS) for semantic gap bridging, and a Hierarchical Attention Network (HAN) for latent space based recognition. Experiments are carried out on two large scale datasets. Experimental results demonstrate the effectiveness of the proposed framework.
37
Learning Spatiotemporal Features with 3D Convolutional Networks
Du Tran, Lubomir Bourdev, Rob Fergus et al. · 2015 · 9.7K citations
Backpropagation through time: what it does and how to do it
Paul J. Werbos · Proceedings of the IEEE · 1990 · 4.9K citations
Hierarchical Attention Networks for Document Classification
Zichao Yang, Diyi Yang, Chris Dyer et al. · 2016 · 4.8K citations · Full text
Sequence to Sequence -- Video to Text
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue et al. · 2015 · 1.4K citations