Multimodal Deep Learning

TLDR

Deep networks have been successfully applied to unsupervised feature learning for single modalities such as text, images, or audio. This study proposes a novel application of deep networks to learn features across multiple modalities. The authors design multimodal learning tasks and train deep networks to extract cross‑modality features, learn shared representations, and evaluate them on audio‑visual classification problems. Their models, tested on CUAVE and AVLetters, achieve state‑of‑the‑art visual speech classification on AVLetters and demonstrate effective shared representation learning between audio and video modalities.

Abstract

Deep networks have been successfully applied to unsupervised feature learning for single modalities (e.g., text, images or audio). In this work, we propose a novel application of deep networks to learn features over multiple modalities. We present a series of tasks for multimodal learning and show how to train deep networks that learn features to address these tasks. In particular, we demonstrate cross modality feature learning, where better features for one modality (e.g., video) can be learned if multiple modalities (e.g., audio and video) are present at feature learning time. Furthermore, we show how to learn a shared representation between modalities and evaluate it on a unique task, where the classifier is trained with audio-only data but tested with video-only data and vice-versa. Our models are validated on the CUAVE and AVLetters datasets on audio-visual speech classification, demonstrating best published visual speech classification on AVLetters and effective shared representation learning.