arXiv (Cornell University) · 2020 · 13 citations · 42 references
BusinessEconomicsDecision TheoryMicroeconomicsPerformance MeasurementBenchmarkingEfficiency AnalysisModel-based Economic EvaluationPerformance EvaluationUtility FunctionsUtility TheoryBenchmark TransparencyEfficiencyElicitation Of Utility FunctionsStatistical BenchmarkingTransparencyUtility Benchmarking
Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -- in their current form -- can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).
42
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al. · 2016 · 6.3K citations · Full text
Justice as Fairness: A Restatement
Thaddeus Metz, John Rawls, Erin I. Kelly · The Philosophical Review · 2002 · 3.5K citations
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts et al. · 2015 · 3.5K citations · Full text
Know What You Don’t Know: Unanswerable Questions for SQuAD
Pranav Rajpurkar, Robin Jia, Percy Liang · 2018 · 2.2K citations · Full text