Fast near-duplicate detection from image streams on online social media during disaster events

Abstract

User-generated content on online social media (OSM) has several data mining applications, such as extracting useful information during disaster events. Since popular / important content is often re-posted by multiple people on OSM, identifying duplicate content is an important first step in many data mining applications. In this work, we develop a methodology to identify near-duplicate images posted on Twitter. The principal challenge in this task is to efficiently handle a continuous incoming stream of thousands of images, for which prior duplication detection methods are unsuitable. Experiments on real Twitter data collected during two recent disaster events demonstrate that the proposed methodology identifies duplicate images accurately as well as rapidly.