Empirical Methods in Natural Language Processing · 2010 · 30 citations · 22 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingLanguage ModelingLanguage ModelsMachine TranslationNeural NetworksSpeech RecognitionWord EmbeddingsConvergenceInitializationNeural Language ModelsStatistical Machine TranslationTrainingTraining Time
Using multi-layer neural networks to estimate the probabilities of word sequences is a promising research area in statistical language modeling, with applications in speech recognition and statistical machine translation. However, training such models for large vocabulary tasks is computationally challenging which does not scale easily to the huge corpora that are nowadays available. In this work, we study the performance and behavior of two neural statistical language models so as to highlight some important caveats of the classical training algorithms. The induced word embeddings for extreme cases are also analysed, thus providing insight into the convergence issues. A new initialization scheme and new training techniques are then introduced. These methods are shown to greatly reduce the training time and to significantly improve performance, both in terms of perplexity and on a large-scale translation task.
22
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
A unified architecture for natural language processing
Ronan Collobert, Jason Weston · 2008 · 5.2K citations
Philipp Koehn, Richard Zens, Chris Dyer et al. · 2007 · 4.9K citations · Full text
Class-based n -gram models of natural language
Peter F. Brown, P.V. deSouza, Robert L. Mercer et al. · 1992 · 2.9K citations
Language Studies, Computer Science, Computational Linguistics +15