Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021 · 21 citations · 24 references
ComputingComputer ScienceComputational LinguisticsNatural Language ProcessingPsycholinguisticsContext DependenceLanguage ModelingLanguage ModelsText ComprehensionContext Effects On Word MeaningContextual CuesIrrelevant FactsLanguage Model PretrainingModel Robustness TestingRelevant FactsRobustness TestingSemantic Similarity
Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this paper we tackle a component of this question by examining robustness of models' ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models' use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.
24
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
Know What You Don’t Know: Unanswerable Questions for SQuAD
Pranav Rajpurkar, Robin Jia, Percy Liang · 2018 · 2.2K citations · Full text
Language Models as Knowledge Bases?
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel et al. · 2019 · 1.8K citations · Full text
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Zhilin Yang, Peng Qi, Saizheng Zhang et al. · 2018 · 1.6K citations · Full text