2011 · 72 citations · 14 references
Language StudiesComputer ScienceGermanComputational LinguisticsCorpus LinguisticsNatural Language ProcessingAnnotationCorpus AnalysisCorpus AnnotationCorpus CompilationWritten CorporaAnnotated Text CorporaAnnotator AgreementDataset CurationTextual Entailment
We address the creation of cross-lingual textual entailment corpora by means of crowdsourcing. Our goal is to define a cheap and replicable data collection methodology that minimizes the manual work done by expert annotators, without resorting to preprocessing tools or already annotated monolingual datasets. In line with recent works emphasizing the need of large-scale annotation efforts for textual entailment, our work aims to: i) tackle the scarcity of data available to train and evaluate systems, and ii) promote the recourse to crowdsourcing as an effective way to reduce the costs of data collection without sacrificing quality. We show that a complex data creation task, for which even experts usually feature low agreement scores, can be effectively decomposed into simple subtasks assigned to non-expert annotators. The resulting dataset, obtained from a pipeline of different jobs routed to Amazon Mechanical Turk, contains more than 1,600 aligned pairs for each combination of texts-hypotheses in English, Italian and German. 1
14
Cheap and fast---but is it good?
Rion Snow, Brendan O’Connor, Daniel Jurafsky et al. · 2008 · 1.9K citations · Full text
The Seventh PASCAL Recognizing Textual Entailment Challenge.
Luisa Bentivogli, Peter E. Clark, Ido Dagan et al. · 2008 · 580 citations
Rada Mihalcea, Carlo Strapparava · 2009 · 341 citations · Full text