Unsupervised Data Augmentation for Consistency Training

TLDR

Semi‑supervised learning improves deep‑learning models when labeled data is scarce, often using consistency training on large unlabeled datasets to enforce prediction invariance to input noise. The study proposes that high‑quality noising via advanced data augmentation is essential for effective semi‑supervised learning. The authors replace basic noising with advanced augmentation techniques such as RandAugment and back‑translation within a consistency‑training framework, yielding large gains on multiple language and vision tasks. The approach achieves state‑of‑the‑art performance on IMDb with only 20 labeled examples, outperforms prior methods on CIFAR‑10 with 250 examples, and improves results when fine‑tuned from BERT on ImageNet under both low‑ and high‑label regimes. Code is available at https://github.com/google-research/uda.

Abstract

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods such as RandAugment and back-translation, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 5.43 with only 250 examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. Code is available at https://github.com/google-research/uda.