International Conference on Machine Learning · 2014 · 103 citations · 31 references
ComputingComputer ScienceMachine LearningNatural Language ProcessingTime Series AnalysisDeep LearningNeural NetworksPattern RecognitionShort-term MemorySpeech RecognitionTemporal Dependence ModelingAudio Signal ProcessingRecurrent Neural NetworksTemporal GranularityWord Classification
Sequence prediction and classification are ubiquitous and challenging problems in machine learning that can require identifying complex dependencies between temporally distant inputs. Recurrent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term memory implemented by their recurrent (feedback) connections. However, in practice they are difficult to train successfully when long-term memory is required. This paper introduces a simple, yet powerful modification to the simple RNN (SRN) architecture, the Clockwork RNN (CW-RNN), in which the hidden layer is partitioned into separate modules, each processing inputs at its own temporal granularity, making computations only at its prescribed clock rate. Rather than making the standard RNN models more complex, CW-RNN reduces the number of SRN parameters, improves the performance significantly in the tasks tested, and speeds up the network evaluation. The network is demonstrated in preliminary experiments involving three tasks: audio signal generation, TIMIT spoken word classification, where it outperforms both SRN and LSTM networks, and online handwriting recognition, where it outperforms SRNs.
31
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Jeffrey L. Elman · Cognitive Science · 1990 · 10.8K citations
Speech recognition with deep recurrent neural networks
Alex Graves, Abdelrahman Mohamed, Geoffrey E. Hinton · 2013 · 8.8K citations
Connectionist temporal classification
Alex Graves, Santiago Fernández, Faustino Gomez et al. · 2006 · 5.4K citations