A Context-Aware Late-Fusion Approach for Disaster Image Retrieval from Social Media

Abstract

Natural disasters, especially those related to flooding, are global issues that attract a lot of attention in many parts of the world. A series of research ideas focusing on combining heterogeneous data sources to monitor natural disasters have been proposed, including multi-modal image retrieval. Among these data sources, social media streams are considered of high importance due to the fast and localized updates on disaster situations. Unfortunately, the social media itself contains several factors that limit the accuracy of this process such as noisy data, unsynchronized content between image and collateral text, and untrusted information, to name a few. In this research work, we introduce a context-aware late-fusion approach for disaster image retrieval from social media. Several known techniques based on context-aware criteria are integrated, namely late fusion, tuning, ensemble learning, object detection and scene classification using deep learning. We have developed a method for image-text content synchronization and spatial-temporal-context event confirmation, and evaluated the role of using different types of features extracted from internal and external data sources. We evaluated our approach using the dataset and evaluation tool offered by MediaEval2017: Emergency Response for Flooding Events Task. We have also compared our approach with other methods introduced by MediaEval2017's participants. The experimental results show that our approach is the best one when taking the image-text content synchronization and spatial-temporal-context event confirmation into account.