Learning to Forget: Continual Prediction with LSTM

TLDR

LSTM networks can solve many tasks that earlier RNNs could not, but without explicit resets their internal state can grow without bound and eventually break down. The study aims to identify the weakness of LSTM networks when processing continual input streams without explicit segment boundaries. The authors introduce an adaptive forget gate that allows the LSTM cell to learn when to reset its state, and evaluate it on benchmark problems where standard LSTM outperforms other RNNs. LSTM with the adaptive forget gate successfully solves the continual versions of the benchmark problems, whereas standard LSTM and other algorithms fail.

Abstract

Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive “forget gate” that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.