arXiv (Cornell University) · 2021 · 68 citations · 25 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningNatural Language ProcessingAnnotationGenerative ModelingLanguage ModelsNatural Language GenerationQuestion AnsweringText GenerationAdversarial ExamplesAdversarial Question AnsweringAdversarial RobustnessAnswer GenerationHuman-in-the-loop WorkflowsModel Robustness AnalysisModel Robustness AssessmentRobustness
Despite recent progress, state-of-the-art question answering models remain vulnerable to a variety of adversarial attacks. While dynamic adversarial data collection, in which a human annotator tries to write examples that fool a model-in-the-loop, can improve model robustness, this process is expensive which limits the scale of the collected data. In this work, we are the first to use synthetic adversarial data generation to make question answering models more robust to human adversaries. We develop a data generation pipeline that selects source passages, identifies candidate answers, generates questions, then finally filters or re-labels them to improve quality. Using this approach, we amplify a smaller human-written adversarial dataset to a much larger set of synthetic question-answer pairs. By incorporating our synthetic data, we improve the state-of-the-art on the AdversarialQA dataset by 3.7F1 and improve model generalisation on nine of the twelve MRQA datasets. We further conduct a novel human-in-the-loop evaluation to show that our models are considerably more robust to new human-written adversarial examples: crowdworkers can fool our model only 8.8% of the time on average, compared to 17.6% for a model trained without synthetic data.
25
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev et al. · 2016 · 6.3K citations · Full text
A large annotated corpus for learning natural language inference
Samuel R. Bowman, Gabor Angeli, Christopher Potts et al. · 2015 · 3.5K citations · Full text