IEEE Access · 2019 · 519 citations · 138 references
ComputingComputer ScienceComputer VisionMachine LearningVisual AttentionDeep LearningMultimodal IntegrationRepresentation LearningAttentionEncoder-decoder ArchitecturesGenerative Adversarial NetworksHeterogeneityJoint Representation LearningMultimodal FusionMultimodal Machine LearningMultimodal Representation Learning
Multimodal representation learning seeks to bridge the heterogeneity gap among diverse data modalities, and deep learning approaches have recently gained prominence for their multi‑level abstraction capabilities. This survey comprehensively reviews deep multimodal representation learning, focusing on emerging technologies such as encoder‑decoder models, GANs, and attention mechanisms, and proposes future research directions. The authors classify methods into joint, coordinated, and encoder‑decoder frameworks, reviewing representative models and discussing their structures, objectives, applications, and trade‑offs.
Multimodal representation learning, which aims to narrow the heterogeneity gap among different modalities, plays an indispensable role in the utilization of ubiquitous multimodal data. Due to the powerful representation ability with multiple levels of abstraction, deep learning-based multimodal representation learning has attracted much attention in recent years. In this paper, we provided a comprehensive survey on deep multimodal representation learning which has never been concentrated entirely. To facilitate the discussion on how the heterogeneity gap is narrowed, according to the underlying structures in which different modalities are integrated, we category deep multimodal representation learning methods into three frameworks: joint representation, coordinated representation, and encoder-decoder. Additionally, we review some typical models in this area ranging from conventional models to newly developed technologies. This paper highlights on the key issues of newly developed technologies, such as encoder-decoder model, generative adversarial networks, and attention mechanism in a multimodal representation learning perspective, which, to the best of our knowledge, have never been reviewed previously, even though they have become the major focuses of much contemporary research. For each framework or model, we discuss its basic structure, learning objective, application scenes, key issues, advantages, and disadvantages, such that both novel and experienced researchers can benefit from this survey. Finally, we suggest some important directions for future work.
138
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text