2015 · 120 citations · 25 references
ComputingComputer ScienceMachine LearningNatural Language ProcessingTime Series AnalysisDeep LearningLanguage ModelingLong-term MemoryPattern RecognitionShort-term MemorySpeech RecognitionGated Recurrent Unit NetworksGatingGating StrategiesLong Short-term Memory NetworksRecurrenceRecurrent Neural NetworksSpectral RepresentationVanishing Gradient Problem
Long short-term memory (LSTM) recurrent neural networks (RNNs) have recently shown significant performance improvements over deep feed-forward neural networks (DNNs). A key aspect of these models is the use of time recurrence, combined with a gating architecture that ameliorates the vanishing gradient problem. Inspired by human spectrogram reading, in this paper we propose an extension to LSTMs that performs the recurrence in frequency as well as in time. This model first scans the frequency bands to generate a summary of the spectral information, and then uses the output layer activations as the input to a traditional time LSTM (T-LSTM). Evaluated on a Microsoft short message dictation task, the proposed model obtained a 3.6% relative word error rate reduction over the T-LSTM.
25
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Speech recognition with deep recurrent neural networks
Alex Graves, Abdelrahman Mohamed, Geoffrey E. Hinton · 2013 · 8.8K citations
Learning to Forget: Continual Prediction with LSTM
Felix A. Gers, Jürgen Schmidhuber, Fred Cummins · Neural Computation · 2000 · 5.4K citations