arXiv (Cornell University) · 2019 · 179 citations · 0 references
ComputingComputer ScienceEnglishComputational LinguisticsNatural Language ProcessingReading ComprehensionLanguage ModelsNatural Language UnderstandingParsingQuestion AnsweringReadingText ComprehensionAccuracyBenchmark DatasetsExpert PerformanceNumerical ReasoningParsing Accuracy MetricsQuestion Answering SystemsQuestion Answering Systems EvaluationSemantic Parsing
Reading comprehension has recently seen rapid progress, with systems matching\nhumans on the most popular datasets for the task. However, a large body of work\nhas highlighted the brittleness of these systems, showing that there is much\nwork left to be done. We introduce a new English reading comprehension\nbenchmark, DROP, which requires Discrete Reasoning Over the content of\nParagraphs. In this crowdsourced, adversarially-created, 96k-question\nbenchmark, a system must resolve references in a question, perhaps to multiple\ninput positions, and perform discrete operations over them (such as addition,\ncounting, or sorting). These operations require a much more comprehensive\nunderstanding of the content of paragraphs than what was necessary for prior\ndatasets. We apply state-of-the-art methods from both the reading comprehension\nand semantic parsing literature on this dataset and show that the best systems\nonly achieve 32.7% F1 on our generalized accuracy metric, while expert human\nperformance is 96.0%. We additionally present a new model that combines reading\ncomprehension methods with simple numerical reasoning to achieve 47.0% F1.\n