arXiv (Cornell University) · 2016 · 201 citations · 22 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningOverfittingRepresentation LearningSemi-supervised LearningSupervised LearningWord EmbeddingsEmbedding RegularizationLearned Feature EmbeddingsSemi-supervised ClassificationSemi-supervised ModelsSemi-supervised Representation Learning
Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting. Code is available at https://github.com/tensorflow/models/tree/master/research/adversarial_text.
22
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky et al. · 2014 · 34.3K citations
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey E. Hinton, Simon Osindero, Yee‐Whye Teh · Neural Computation · 2006 · 16.4K citations
Convolutional Neural Networks for Sentence Classification
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Martı́n Abadi, Ashish Agarwal, Paul Barham et al. · arXiv (Cornell University) · 2016 · 9.8K citations · Full text
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget et al. · 2010 · 5.4K citations