GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

TLDR

GQA is a new dataset for real‑world visual reasoning and compositional question answering that aims to overcome limitations of prior VQA datasets and to serve as a resource for building more robust, consistent, and semantically aware vision‑language models. The dataset is constructed from Visual Genome scene graphs, generating 22 million reasoning questions paired with functional programs, applying a tunable smoothing technique to control answer distribution, and introducing new metrics for consistency, grounding, and plausibility, alongside a detailed baseline analysis. Baseline experiments show a blind LSTM scores 42.1 %, strong VQA models 54.1 %, while humans reach 89.3 %, underscoring a substantial performance gap and the effectiveness of the question engine and smoothing method.

Abstract

We introduce GQA, a new dataset for real-world visual reasoning and compositional question answering, seeking to address key shortcomings of previous VQA datasets. We have developed a strong and robust question engine that leverages Visual Genome scene graph structures to create 22M diverse reasoning questions, which all come with functional programs that represent their semantics. We use the programs to gain tight control over the answer distribution and present a new tunable smoothing technique to mitigate question biases. Accompanying the dataset is a suite of new metrics that evaluate essential qualities such as consistency, grounding and plausibility. A careful analysis is performed for baselines as well as state-of-the-art models, providing fine-grained results for different question types and topologies. Whereas a blind LSTM obtains a mere 42.1%, and strong VQA models achieve 54.1%, human performance tops at 89.3%, offering ample opportunity for new research to explore. We hope GQA will provide an enabling resource for the next generation of models with enhanced robustness, improved consistency, and deeper semantic understanding of vision and language.