Hierarchical LSTM for Sign Language Translation

TLDR

Continuous sign language translation is challenging because of sequential gesture variation and lack of word alignment, and existing hybrid HMM/CTC models struggle with mismatched word order relative to visual content. The paper proposes a hierarchical‑LSTM encoder‑decoder model that incorporates visual content and word embeddings to address these challenges. The HLSTM model extracts spatio‑temporal cues from video clips with a 3D CNN, adaptively mines key clips for viseme packing, processes frames, clips, and viseme units through a top‑layer HLSTM with temporal attention‑aware weighting, and then refines viseme vectors and generates semantic output with two additional LSTM layers while reducing computational complexity. The proposed model achieves promising performance on singer‑independent tests with seen sentences and outperforms comparison algorithms on unseen sentences.

Abstract

Continuous Sign Language Translation (SLT) is a challenging task due to its specific linguistics under sequential gesture variation without word alignment. Current hybrid HMM and CTC (Connectionist temporal classification) based models are proposed to solve frame or word level alignment. They may fail to tackle the cases with messing word order corresponding to visual content in sentences. To solve the issue, this paper proposes a hierarchical-LSTM (HLSTM) encoder-decoder model with visual content and word embedding for SLT. It tackles different granularities by conveying spatio-temporal transitions among frames, clips and viseme units. It firstly explores spatio-temporal cues of video clips by 3D CNN and packs appropriate visemes by online key clip mining with adaptive variable-length. After pooling on recurrent outputs of the top layer of HLSTM, a temporal attention-aware weighting mechanism is proposed to balance the intrinsic relationship among viseme source positions. At last, another two LSTM layers are used to separately recurse viseme vectors and translate semantic. After preserving original visual content by 3D CNN and the top layer of HLSTM, it shortens the encoding time step of the bottom two LSTM layers with less computational complexity while attaining more nonlinearity. Our proposed model exhibits promising performance on singer-independent test with seen sentences and also outperforms the comparison algorithms on unseen sentences.