Publication | Open Access
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
405
Citations
0
References
2020
Year
Artificial IntelligenceStructured PredictionEngineeringMachine LearningSpurious BiasesLarge Language ModelLanguage ProcessingDataset Construction ConsistNatural Language ProcessingData ScienceComputational LinguisticsAdversarial Machine LearningLanguage StudiesMachine TranslationWinograd Schema ChallengeBenchmark DatasetsNlp TaskComputer ScienceAutomated ReasoningDomain Knowledge ModelingLinguistics
The Winograd Schema Challenge, a 273‑problem benchmark for commonsense pronoun resolution, was originally designed to stump statistical models, yet recent neural language models now achieve roughly 90% accuracy on its variants. This study introduces WinoGrande, a 44k‑problem dataset built to probe whether high WSC scores reflect genuine commonsense or dataset biases, and stresses the need for bias‑reduction in future benchmarks. WinoGrande was constructed via a carefully designed crowdsourcing pipeline followed by systematic bias mitigation using the AfLite algorithm, which extends human‑detectable word associations to machine‑detectable embedding associations. State‑of‑the‑art methods score 59.4–79.1% on WinoGrande—15–35% below human 94%—while also achieving new best results on WSC, DPR, COPA, KnowRef, and Winogender, demonstrating WinoGrande’s transfer‑learning value but highlighting overestimation of machine commonsense.
The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense.To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4 – 79.1%, which are ∼15-35% (absolute) below human performance of 94.0%, depending on the amount of the training data allowed (2% – 100% respectively).Furthermore, we establish new state-of-the-art results on five related benchmarks — WSC (→ 90.1%), DPR (→ 93.1%), COPA(→ 90.6%), KnowRef (→ 85.6%), and Winogender (→ 97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.