Transactions of the Association for Computational Linguistics · 2019 · 194 citations · 50 references
ComputingComputer ScienceComputational LinguisticsCorpus LinguisticsNatural Language ProcessingAnnotationLanguage ModelingLanguage ModelsNatural Language UnderstandingOpinion MiningAnnotator AgreementHuman Agreement ModelingNatural Language InferenceObjective EvaluationText-based InferenceTextual EntailmentTextual Entailment RecognitionTextual Inference
We analyze human’s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation “noise”, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments.
50
Advances in Neural Information Processing Systems 28
Peter Flach, Meelis Kull · 2015 · 14.7K citations
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts et al. · 2015 · 3.5K citations · Full text
Automatic acquisition of hyponyms from large text corpora
Marti A. Hearst · 1992 · 3.3K citations · Full text
Fuzzy logic = computing with words
Lotfi A. Zadeh · IEEE Transactions on Fuzzy Systems · 1996 · 3.2K citations
Matthew Richardson, Pedro Domingos · Machine Learning · 2006 · 2.7K citations · Full text