Weakly Supervised Coupled Networks for Visual Sentiment Analysis

TLDR

Automatic visual sentiment assessment has attracted attention as online expression grows, yet existing CNN‑based methods rely on holistic image appearance while ignoring that different regions may influence sentiment differently. This paper introduces a weakly supervised coupled convolutional network that leverages localized information to solve visual sentiment analysis. The method uses a two‑branch network: a fully convolutional branch that learns a sentiment‑specific soft map with cross‑spatial pooling from image‑level labels, and a classification branch that couples this map with deep features, all trained end‑to‑end in a unified framework. Extensive experiments on six benchmark datasets demonstrate that the proposed method performs favorably against state‑of‑the‑art visual sentiment analysis approaches.

Abstract

Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions on-line. In this paper, we solve the problem of visual sentiment analysis using the high-level abstraction in the recognition process. Existing methods based on convolutional neural networks learn sentiment representations from the holistic image appearance. However, different image regions can have a different influence on the intended expression. This paper presents a weakly supervised coupled convolutional network with two branches to leverage the localized information. The first branch detects a sentiment specific soft map by training a fully convolutional network with the cross spatial pooling strategy, which only requires image-level labels, thereby significantly reducing the annotation burden. The second branch utilizes both the holistic and localized information by coupling the sentiment map with deep features for robust classification. We integrate the sentiment detection and classification branches into a unified deep framework and optimize the network in an end-to-end manner. Extensive experiments on six benchmark datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods for visual sentiment analysis.