RNN language model with word clustering and class-based output layer

Yongzhe Shi, Wei-Qiang Zhang, Jia Liu, Michael T. Johnson

EURASIP Journal on Audio Speech and Music Processing · 2013 · 29 citations · 14 references

DOIFull text

Open access

Concepts

Abstract

The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

References

14