2016 · 438 citations · 42 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingMachine TranslationParsingSequence LabelingText GenerationBeam TrainingNeural Language ModelsSequence GenerationSequence Representation LearningSequence-to-sequence GenerationSequence-to-sequence RecognitionTraining
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important generalpurpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work, we introduce a model and beamsearch training scheme, based on the work of This structured approach avoids classical biases associated with local training and unifies the training loss with the test-time usage, while preserving the proven model architecture of seq2seq and its efficient training approach. We show that our system outperforms a highlyoptimized attention-based seq2seq system and other baselines on three different sequence to sequence tasks: word ordering, parsing, and machine translation.
42
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky et al. · 2014 · 34.3K citations
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Distributed Representations of Words and Phrases and their Compositionality
Tomáš Mikolov, Ilya Sutskever, Kai Chen et al. · arXiv (Cornell University) · 2013 · 18.1K citations · Full text