Espacio Tiempo y Forma Serie I Prehistoria y Arqueología · 2011 · 83 citations · 7 references
Open access
Social SciencesEducational AssessmentPsychologyDecision TheoryPsychometricsReading ComprehensionQuestion AnsweringReading AccuracyTest ReliabilityMultiple-choice Question AnsweringSensitivity
There are several tasks where is preferable not responding than responding incorrectly. This idea is not new, but despite several previous attempts there isn’t a commonly accepted measure to assess non-response. We study here an extension of accuracy measure with this feature and a very easy to understand interpretation. The measure proposed (c@1) has a good balance of discrimination power, stability and sensitivity properties. We show also how this measure is able to reward systems that maintain the same number of correct answers and at the same time decrease the number of incorrect ones, by leaving some questions unanswered. This measure is well suited for tasks such as Reading Comprehension tests, where multiple choices per question are given, but only one is correct. 1
7
Building Watson: An Overview of the DeepQA Project
David Ferrucci, Eric W. Brown, Jennifer Chu‐Carroll et al. · AI Magazine · 2010 · 1.5K citations · Full text
Evaluating Evaluation Measure Stability
Chris Buckley, Ellen M. Voorhees · ACM SIGIR Forum · 2017 · 567 citations
Business, Educational Measurement, Performance Measurement +6
Overview of the TREC 2001 Question Answering Track
Ellen M. Voorhees · 2000 · 329 citations
The effect of topic set size on retrieval experiment error
Ellen M. Voorhees, Chris Buckley · 2002 · 248 citations
Evaluating evaluation metrics based on the bootstrap
Tetsuya Sakai · 2006 · 210 citations