Crowdsourcing syntactic relatedness judgements for opinion mining in the study of information technology adoption

Asad Sayeed, Bryan Rusk, Martin Petrov, Hieu Nguyen, Timothy J. Meyer, Amy Weinberg

2011 · 12 citations · 8 references

Concepts

Abstract

We present an end-to-end pipeline including a user interface for the production of word-level annotations for an opinion-mining task in the information technology (IT) domain. Our pre-annotation pipeline selects candidate sentences for annotation using results from a small amount of trained annotation to bias the random selection over a large corpus. Our user interface reduces the need for the user to understand the meaning of opinion in our domain context, which is related to community reaction. It acts as a preliminary buffer against low-quality annotators. Finally, our post-annotation pipeline aggregates responses and applies a more aggressive quality filter. We present positive results using two different evaluation philosophies and discuss how our design decisions enabled the collection of high-quality annotations under subjective and fine-grained conditions.

References

8