2011 · 12 citations · 8 references
ComputingComputer ScienceInformation TechnologyComputational LinguisticsNatural Language ProcessingText AnalyticsAnnotationOpinion MiningSentiment AnalysisSyntactic AnalysisSyntactic AnnotationText MiningAutomatic AnnotationOpinion AnnotationOpinion ExtractionSemantic Relatedness
We present an end-to-end pipeline including a user interface for the production of word-level annotations for an opinion-mining task in the information technology (IT) domain. Our pre-annotation pipeline selects candidate sentences for annotation using results from a small amount of trained annotation to bias the random selection over a large corpus. Our user interface reduces the need for the user to understand the meaning of opinion in our domain context, which is related to community reaction. It acts as a preliminary buffer against low-quality annotators. Finally, our post-annotation pipeline aggregates responses and applies a more aggressive quality filter. We present positive results using two different evaluation philosophies and discuss how our design decisions enabled the collection of high-quality annotations under subjective and fine-grained conditions.
8
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
Data quality from crowdsourcing
Pei-Yun Hsueh, Prem Melville, Vikas Sindhwani · 2009 · 252 citations · Full text