Large-scale visual sentiment ontology and detectors using adjective noun pairs

TLDR

The paper proposes a novel visual sentiment analysis framework that constructs a large‑scale Visual Sentiment Ontology and a detector library to enable automatic sentiment analysis of images. The authors build the ontology from psychological theory and web mining and develop SentiBank, a detector library that identifies 1,200 adjective‑noun pairs in images. Experiments on image tweets show that SentiBank‑based predictors significantly outperform text‑based methods, and the work provides a publicly available ontology, detector library, and benchmark.

Abstract

We address the challenge of sentiment analysis from visual content. In contrast to existing methods which infer sentiment or emotion directly from visual low-level features, we propose a novel approach based on understanding of the visual concepts that are strongly related to sentiments. Our key contribution is two-fold: first, we present a method built upon psychological theories and web mining to automatically construct a large-scale Visual Sentiment Ontology (VSO) consisting of more than 3,000 Adjective Noun Pairs (ANP). Second, we propose SentiBank, a novel visual concept detector library that can be used to detect the presence of 1,200 ANPs in an image. The VSO and SentiBank are distinct from existing work and will open a gate towards various applications enabled by automatic sentiment analysis. Experiments on detecting sentiment of image tweets demonstrate significant improvement in detection accuracy when comparing the proposed SentiBank based predictors with the text-based approaches. The effort also leads to a large publicly available resource consisting of a visual sentiment ontology, a large detector library, and the training/testing benchmark for visual sentiment analysis.