2010 · 32 citations · 26 references
ComputingComputer ScienceLinguisticsComputational LinguisticsMachine LearningNatural Language ProcessingAnnotationNatural Language UnderstandingWord Sense DisambiguationAnnotator AgreementAutomatic AnnotationLabelsSense AnnotationWord Sense Annotation
Manual annotation of natural language to capture linguistic information is essential for NLP tasks involving supervised machine learning of semantic knowledge. Judgements of meaning can be more or less subjective, in which case instead of a single correct label, the labels assigned might vary among annotators based on the annotators ’ knowledge, age, gender, intuitions, background, and so on. We introduce a framework ”Anveshan, ” where we investigate annotator behavior to find outliers, cluster annotators by behavior, and identify confusable labels. We also investigate the effectiveness of using trained annotators versus a larger number of untrained annotators on a word sense annotation task. The annotation data comes from a word sense disambiguation task for polysemous words, annotated by both trained annotators and untrained annotators from Amazon’s Mechanical turk. Our results show that Anveshan is effective in uncovering patterns in annotator behavior, and we also show that trained annotators are superior to a larger number of untrained annotators for this task.
26
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
Inter-Coder Agreement for Computational Linguistics
Ron Artstein, Massimo Poesio · Computational Linguistics · 2008 · 1.6K citations · Full text