B<scp>reak</scp> It Down: A Question Understanding Benchmark

TLDR

Understanding natural language questions requires breaking them into the steps needed to compute an answer. The study introduces a Question Decomposition Meaning Representation (QDMR) for questions. QDMR is an ordered natural‑language list of steps, and the authors built a crowdsourcing pipeline to annotate over 83K question–QDMR pairs and trained a seq2seq model with copying to generate QDMR structures. QDMR improves open‑domain QA on HotpotQA, can be deterministically converted to pseudo‑SQL to ease semantic‑parsing annotation, and a seq2seq model trained on Break substantially outperforms several baselines.

Abstract

Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.