arXiv (Cornell University) · 2020 · 76 citations · 33 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingBenchmarkingQuestion AnsweringVisual Question AnsweringBenchmark DatasetsDistribution TestingOpen-ended Visual Question AnsweringOut-of-distribution Detection
Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on ``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the common training answer is 'no'. Second, the OOD test set is used for model selection. Third, a model's in-domain performance is assessed after retraining it on in-domain splits (VQA v2) that exhibit a more balanced distribution of labels. These three practices defeat the objective of evaluating generalization, and put into question the value of methods specifically designed for this dataset. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types. We provide short- and long-term solutions to avoid these pitfalls and realize the benefits of OOD evaluation.
33
VQA: Visual Question Answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu et al. · 2015 · 4.3K citations
Shortcut learning in deep neural networks
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis et al. · Nature Machine Intelligence · 2020 · 1.8K citations · Full text
Deeper, Broader and Artier Domain Generalization
Da Li, Yongxin Yang, Yi-Zhe Song et al. · 2017 · 1.3K citations · Full text
Visual question answering: A survey of methods and datasets
Qi Wu, Damien Teney, Peng Wang et al. · Computer Vision and Image Understanding · 2017 · 433 citations