2018 · 58 citations · 29 references
ComputingComputer ScienceMachine LearningNatural Language ProcessingTime Series AnalysisDeep LearningHidden Markov ModelsNeural NetworksPattern RecognitionSpeech RecognitionConnectionist Temporal ClassificationKnowledge DistillationModel CompressionReal-time ProcessingRecurrent Neural NetworksSpeech Recognition AccuracyStreaming Speech RecognitionTemporal Classification
End-to-end acoustic models, such as connectionist temporal classification (CTC) and the attention model, have been studied, and their speech recognition accuracies come close to those of conventional deep neural network (DNN)-hidden Markov models. However, most high-performance end-to-end models are not suitable for real-time (streaming) speech recognition because they are based on bidirectional recurrent neural networks (RNNs). In this study, to improve the performance of unidirectional RNN-based CTC, which is suitable for real-time processing, we investigate the knowledge distillation (KD)-based model compression method for training a CTC acoustic model. we evaluate a frame-level KD method and a sequence-level KD method for CTC model. The speech recognition experiments on Wall Street Journal tasks demonstrate that, the frame-level KD worsens the WERs ofunidirectional CTC model, whereas sequence-level KD can improve the WERs of the model.
29
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
Distilling the Knowledge in a Neural Network
Geoffrey E. Hinton, Oriol Vinyals · arXiv (Cornell University) · 2015 · 14K citations · Full text