Visual Abductive Reasoning

TLDR

Abductive reasoning seeks the most plausible explanation for partial observations, yet it is rarely studied in computer vision despite its frequent use in human reasoning. This work introduces the Visual Abductive Reasoning (VAR) task and dataset to evaluate machine intelligence’s ability to infer explanations from incomplete visual scenes. We present REASONER, a causal‑and‑cascaded reasoning Transformer that employs contextualized directional position embeddings and cascaded decoders to generate and refine premise and hypothesis sentences, guiding cross‑sentence flow with prediction scores. Benchmarking shows REASONER outperforms many video‑language models but remains far below human performance, highlighting the challenge and encouraging further research in reasoning beyond observation.

Abstract

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, REASONER (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative represen-tations for the premise and hypothesis. Then, multiple de-coders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR bench-marking results show that REASONER surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.