Publication | Closed Access
Learning to forget: continual prediction with LSTM
2.4K
Citations
0
References
1999
Year
Unknown Venue
Sequence ModellingLstm CellMachine LearningData ScienceEngineeringSparse Neural NetworkSequential LearningLstm NetworksMemoryStandard LstmEducationComputer ScienceContinual Learning (Lifelong Deep Learning)Deep LearningNeural Architecture SearchRecurrent Neural NetworkContinual Learning (Educational Psychology)Continual Prediction
LSTM networks excel at many tasks, but without explicit resets their internal state can grow unbounded and cause failure on continual input streams, a problem that all RNN algorithms share. The study aims to identify the weakness of LSTM networks when processing continual input streams without explicit sequence ends. The authors introduce an adaptive reset gate that allows an LSTM cell to learn when to reset its internal state, thereby preventing unbounded growth. With the adaptive reset gate, LSTM successfully solves the continual benchmark problem, outperforming other RNN algorithms.
Long short-term memory (LSTM) can solve many tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams without explicitly marked sequence ends. Without resets, the internal state values may grow indefinitely and eventually cause the network to break down. Our remedy is an adaptive gate that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review an illustrative benchmark problem on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve a continual version of that problem. LSTM with forget gates, however, easily solves it in an elegant way.