MSMO: Multimodal Summarization with Multimodal Output

TLDR

Multimodal summarization has attracted attention as multimedia data grows, yet current systems typically output only text. This paper introduces multimodal summarization with multimodal output (MSMO) as a new task. We collect a large‑scale MSMO dataset, propose a multimodal attention model that jointly generates text and selects the most relevant image, and develop a multimodal automatic evaluation (MMAE) method that assesses intra‑modality salience and inter‑modality relevance. Experiments show that multimodal output significantly enhances user satisfaction with summary informativeness, and that MMAE effectively evaluates multimodal outputs.

Abstract

Multimodal summarization has drawn much attention due to the rapid growth of multimedia data. The output of the current multimodal summarization systems is usually represented in texts. However, we have found through experiments that multimodal output can significantly improve user satisfaction for informativeness of summaries. In this paper, we propose a novel task, multimodal summarization with multimodal output (MSMO). To handle this task, we first collect a large-scale dataset for MSMO research. We then propose a multimodal attention model to jointly generate text and select the most relevant image from the multimodal input. Finally, to evaluate multimodal outputs, we construct a novel multimodal automatic evaluation (MMAE) method which considers both intra-modality salience and inter-modality relevance. The experimental results show the effectiveness of MMAE.