2014 · 17 citations · 13 references
ComputingComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsNeural NetworksPattern RecognitionSpeech RecognitionCovariance RegularizationLanguage Model RescoringNeural Language ModelsRecurrent Neural Language ModelsRecurrent Neural Network Language ModelsRegularizationTrainingWord Error Rate
Recurrent neural network language models (RNNLMs) have been proved superior to many other competitive language modeling techniques in terms of perplexity and word error rate. The remaining problem is the great computational complexity of RNNLMs in the output layer, resulting in long time for evaluation. Typically, a class-based RNNLM with the output layer factorized was proposed for speedup, which was still not fast enough for real-time systems. In this paper, a novel variance regularization algorithm is proposed for RNNLMs to address this problem. All the softmax-normalizing factors in the output layers are penalized to make them converge to one during the training phase, so that the output probability can be estimated efficiently via one dot-product of vectors in the output layer. The computational complexity of the output layer is reduced significantly from O(|V|H) to O(H). We further use this model for rescoring in an advanced CD-HMM-DNN system. Experimental results show that our proposed variance regularization algorithm works quite well, and the word prediction of the model is about 300 times faster than that of RNNLM without any obvious deteriorations in word error rate.
13
Class-based n -gram models of natural language
Peter F. Brown, P.V. deSouza, Robert L. Mercer et al. · 1992 · 2.9K citations
Language Studies, Computer Science, Computational Linguistics +15
Extensions of recurrent neural network language model
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget et al. · 2011 · 1.6K citations
A Scalable Hierarchical Distributed Language Model
Andriy Mnih, Geoffrey E. Hinton · 2008 · 852 citations