IEEE Transactions on Audio Speech and Language Processing · 2012 · 73 citations · 26 references
ComputingArabicComputer ScienceComputational LinguisticsMachine LearningNatural Language ProcessingDeep LearningLanguage ModelingLanguage ModelsNeural NetworksPattern RecognitionSpeech RecognitionNeural Language ModelsNeural Network ModelingSpeech-to-text SystemsWord Recognition AccuracyWord Representation Learning
This paper extends a novel neural network language model (NNLM) which relies on word clustering to structure the output vocabulary: Structured OUtput Layer (SOUL) NNLM. This model is able to handle arbitrarily-sized vocabularies, hence dispensing with the need for shortlists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which is based on the continuous word representation determined by the NNLM. Mandarin and Arabic data are used to evaluate the SOUL NNLM accuracy via speech-to-text experiments. Well tuned speech-to-text systems (with error rates around 10%) serve as the baselines. The SOUL model achieves consistent improvements over a classical shortlist NNLM both in terms of perplexity and recognition accuracy for these two languages that are quite different in terms of their internal structure and recognition vocabulary size. An enhanced training scheme is proposed that allows more data to be used at each training iteration of the neural network.
26
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget et al. · 2010 · 5.4K citations
Class-based n -gram models of natural language
Peter F. Brown, P.V. deSouza, Robert L. Mercer et al. · 1992 · 2.9K citations
Language Studies, Computer Science, Computational Linguistics +15
Applied Physics Letters · 2000 · 1.8K citations · Full text
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent · 2000 · 1.2K citations
A Scalable Hierarchical Distributed Language Model
Andriy Mnih, Geoffrey E. Hinton · 2008 · 852 citations