Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

TLDR

Semantic orientation distinguishes phrases with positive associations, such as “subtle nuances,” from those with negative associations, such as “very cavalier.” The study introduces an unsupervised algorithm that classifies product reviews as recommended (thumbs up) or not recommended (thumbs down) based on semantic orientation. The algorithm predicts a review’s label by computing the average semantic orientation of its adjective or adverb phrases, where each phrase’s orientation is the mutual information with the word “excellent” minus that with “poor,” and a review is deemed recommended if this average is positive. On 410 Epinions reviews across automobiles, banks, movies, and travel, the method achieved an overall 74% accuracy, ranging from 84% for automobile reviews to 66% for movie reviews.

Abstract

This paper presents a simple unsupervised learning algorithm for classifying reviews as recommended (thumbs up) or not recommended (thumbs down). The classification of a review is predicted by the average semantic orientation of the phrases in the review that contain adjectives or adverbs. A phrase has a positive semantic orientation when it has good associations (e.g., "subtle nuances") and a negative semantic orientation when it has bad associations (e.g., "very cavalier"). In this paper, the semantic orientation of a phrase is calculated as the mutual information between the given phrase and the word "excellent" minus the mutual information between the given phrase and the word "poor". A review is classified as recommended if the average semantic orientation of its phrases is positive. The algorithm achieves an average accuracy of 74% when evaluated on 410 reviews from Epinions, sampled from four different domains (reviews of automobiles, banks, movies, and travel destinations). The accuracy ranges from 84% for automobile reviews to 66% for movie reviews.