Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs

TLDR

Microblogs are widely used for news dissemination, but the proliferation of rumors and fake news—especially with added images and videos—threatens their credibility. The study introduces an attention‑based recurrent neural network (att‑RNN) that fuses multimodal features to detect rumors. The att‑RNN employs an LSTM to jointly encode text, social context, and image features, applying neural attention to weight the visual inputs during fusion in an end‑to‑end architecture. Experiments on Weibo and Twitter multimedia rumor datasets show that the att‑RNN outperforms baselines in detecting rumors.

Abstract

Microblogs have become popular media for news propagation in recent years. Meanwhile, numerous rumors and fake news also bloom and spread wildly on the open social media platforms. Without verification, they could seriously jeopardize the credibility of microblogs. We observe that an increasing number of users are using images and videos to post news in addition to texts. Tweets or microblogs are commonly composed of text, image and social context. In this paper, we propose a novel Recurrent Neural Network with an attention mechanism (att-RNN) to fuse multimodal features for effective rumor detection. In this end-to-end network, image features are incorporated into the joint features of text and social context, which are obtained with an LSTM (Long-Short Term Memory) network, to produce a reliable fused classification. The neural attention from the outputs of the LSTM is utilized when fusing with the visual features. Extensive experiments are conducted on two multimedia rumor datasets collected from Weibo and Twitter. The results demonstrate the effectiveness of the proposed end-to-end att-RNN in detecting rumors with multimodal contents.