IEEE Transactions on Multimedia · 2024 · 12 citations · 58 references
ComputingAudiologyComputer VisionRehabilitationSensor FusionCross-modal Sensory InteractionsMultimodal InteractionCross-modal FusionCrossmodal AttentionMultimodal Feature FusionMultimodal Feature RepresentationMultimodal Machine LearningMultimodal Recognition FusionMultisensor Feature FusionSign Language Recognition
Sign language recognition (SLR) can connect the hearing-impaired and able-bodied communities. The SLR works through multiple modalities of co-action, which has garnered attention. However, these methods are much less effective or even fail in recognition when confronted with missing modalities. Therefore, this paper proposes MMSLR, a multimodal SLR framework with cross-modal complementary information. The framework comprises three key components: the cross-modal information complementation (CMIC) module, the fusion and prediction module (FPM), and the sign language recognition module (SLRM). The CMIC module is designed with multilayer, multi-view spatial-temporal detectors to observe different modality features in both temporal and spatial dimensions. Additionally, it utilizes co-training to achieve complementary information among multi-modalities. The FPM integrates crossmodal attention with Canberra distance to eliminate inter-modal redundant information while fusing multimodal features. The SLRM constructed based on Transformer fuses partially obtained modalities from CMIC through bidirectional cross-channel attention. Teacher-Student pairs are constructed to transfer fullmodal features from FPM to the above fused modality features. Moreover, experimental results on the provided MM-Sentence and publicly available OH-Sentence, TH-Sentence and USTCCSL datasets demonstrate that MMSLR achieves state-of-the-art performance.
58
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Zhe Cao, Tomas Simon, Shih-En Wei et al. · 2017 · 7.3K citations
Connectionist temporal classification
Alex Graves, Santiago Fernández, Faustino Gomez et al. · 2006 · 5.4K citations