Overview of Opinion Analysis Pilot Task at NTCIR-6

Abstract

Abstract This paper describes an overview of the OpinionAnalysis Pilot Task from 2006 to 2007 at the Sixth NT-CIR Workshop. We created test collection for 32, 30,and 28 topics (11,907, 15,279, and 8,379 sentences)in Chinese, Japanese and English. Using this test col-lection, we conducted opinion extraction subtask. Thesubtask was defined from four perspectives: (a) opin-ionated sentence judgment, (b) opinion holder extrac-tion, (c) relevance sentence judgment, and (d) polarityjudgment. 21 run results were submitted by 14 partici-pants with five results submitted by the organizers. Weshow the evaluation results of the groups participatingin opinion extraction subtask. Keywords: Opinion Extraction, Opinion Holder, Rel-evance, Polarity, and NTCIR. 1 Introduction This paper describes an overview of the OpinionAnalysis Pilot Task [5] from 2006 to 2007 at the SixthNTCIR Workshop [4] (NTCIR-6 Opinion). This wasthe first effort to produce a multi-lingual test collectionfor evaluating opinion extraction at NTCIR.Opinion and sentiment analysis has been receivinga lot of attention in the natural language processing re-search community recently [2, 9, 7]. With the broadrange of information sources available on the web,and rapid increase in the uptake of social community-oriented websites that foster user-generated contentthere has been further interest by both commercial andgovernmental parties in trying to automatically ana-lyze and monitor the tide of prevalent attitudes on theweb. As a result, interest in automatically detectingsentences in which an opinion is expressed ([12] etc.),the polarity of the expression ([13] etc.), targets, andopinion holders ([1] etc.) has been receiving more at-tention in the research community. Applications in-clude tracking response to and opinions about com-mercial products, governmental policies, tracking blogentries for potential political scandals and so on.In the Sixth NTCIR Workshop, a new pilot task foropinion analysis has been introduced. The pilot taskhas tracks in three languages: Chinese, English, andJapanese. In this paper, we present an overview of thetest collection, task design, and evaluation results us-ing the test collection across the Chinese, Japanese,and English data.We believe that this pilot task presents a unique op-portunity to expand the study of opinionated text anal-ysis across languages due to the comparable nature ofthe corpus. The documents have been carefully se-lected based on the manual relevance judgments as-signed in a cross-lingual Information Retrieval task,ensuring a high quality corpus that is relevant in allthree languages.This paper is organized as follows. In Section 2,we explain the task design for the