Computational Linguistics · 2009 · 77 citations · 27 references
ComputingComputer ScienceData ScienceComputational LinguisticsNatural Language ProcessingUncertainty QuantificationAnnotationCorpus AnnotationData AnalysisData InterpretationData QualityKnowledge AnnotationAgreement MetricsAnnotation AgreementAnnotator AgreementBenchmark DatasetsData AnnotationGold Standard AnnotationInter-annotator AgreementLabel Noise Modeling
This article discusses the transition from annotated data to a gold standard, that is, a subset that is sufficiently noise-free with high confidence. Unless appropriately reinterpreted, agreement coefficients do not indicate the quality of the data set as a benchmarking resource: High overall agreement is neither sufficient nor necessary to distill some amount of highly reliable data from the annotated material. A mathematical framework is developed that allows estimation of the noise level of the agreed subset of annotated data, which helps promote cautious benchmarking.
27
Assessing agreement on classification tasks: the kappa statistic
Jean Carletta · ArXiv.org · 1996 · 2.1K citations · Full text
Automatic Labeling of Semantic Roles
Daniel Gildea, Daniel Jurafsky · Computational Linguistics · 2002 · 1.6K citations · Full text