A computationally efficient multi-modal classification approach of disaster-related Twitter images

Abstract

When natural disasters strike, annotated images and texts flood the Internet, and rescue teams become overwhelmed to prioritize often scarce resources, while relying heavily on human input. In this paper, a novel multi-modal approach is proposed to automate crisis data analysis using machine learning. Our multi-modal two-stage framework relies on computationally inexpensive visual and semantic features to analyze Twitter data. Level I classification consists of training classifiers separately on semantic descriptors and combinations of visual features. These classifiers' decisions are aggregated to form a new feature vector to train the second set of classifiers in Level II classification. A home-grown dataset is gathered from Twitter to train the classifiers. Low-level visual features achieved an accuracy of 91.10% which increased to 92.43% when semantic attributes were incorporated. Applying such data science techniques on social media seems to motivate an updated folk statement "an ANNOTATED image is worth a thousand words".