Deep Multimodal Representation Learning: A Survey

Wenzhong Guo, Jianwen Wang, Shiping Wang

IEEE Access · 2019 · 519 citations · 138 references

DOIFull text

Open access

Concepts

TL;DR

Multimodal representation learning seeks to bridge the heterogeneity gap among diverse data modalities, and deep learning approaches have recently gained prominence for their multi‑level abstraction capabilities. This survey comprehensively reviews deep multimodal representation learning, focusing on emerging technologies such as encoder‑decoder models, GANs, and attention mechanisms, and proposes future research directions. The authors classify methods into joint, coordinated, and encoder‑decoder frameworks, reviewing representative models and discussing their structures, objectives, applications, and trade‑offs.

Abstract

Multimodal representation learning, which aims to narrow the heterogeneity gap among different modalities, plays an indispensable role in the utilization of ubiquitous multimodal data. Due to the powerful representation ability with multiple levels of abstraction, deep learning-based multimodal representation learning has attracted much attention in recent years. In this paper, we provided a comprehensive survey on deep multimodal representation learning which has never been concentrated entirely. To facilitate the discussion on how the heterogeneity gap is narrowed, according to the underlying structures in which different modalities are integrated, we category deep multimodal representation learning methods into three frameworks: joint representation, coordinated representation, and encoder-decoder. Additionally, we review some typical models in this area ranging from conventional models to newly developed technologies. This paper highlights on the key issues of newly developed technologies, such as encoder-decoder model, generative adversarial networks, and attention mechanism in a multimodal representation learning perspective, which, to the best of our knowledge, have never been reviewed previously, even though they have become the major focuses of much contemporary research. For each framework or model, we discuss its basic structure, learning objective, application scenes, key issues, advantages, and disadvantages, such that both novel and experienced researchers can benefit from this survey. Finally, we suggest some important directions for future work.

References

138