Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

Abstract

Coreference is a core nlp problem. How-ever, newswire data, the primary source of existing coreference data, lack the rich-ness necessary to truly solve coreference. We present a new domain with denser references—quiz bowl questions—that is challenging and enjoyable to humans, and we use the quiz bowl community to develop a new coreference dataset, together with an annotation framework that can tag any text data with coreferences and named entities. We also successfully integrate active learn-ing into this annotation pipeline to collect documents maximally useful to coreference models. State-of-the-art coreference sys-tems underperform a simple classifier on our new dataset, motivating non-newswire data for future coreference research. 1