Publication | Closed Access
Understanding the Role of Training Regimes in Continual Learning
12
Citations
0
References
2020
Year
EngineeringMachine LearningSequential LearningEducationCognitionTraining RegimesRecurrent Neural NetworkPlasticity-stability DilemmaBatch SizeData ScienceContinual Learning (Lifelong Deep Learning)Just-in-time LearningCognitive ScienceLearning SciencesComputer ScienceLocal MinimaLearning TheoryLifelong LearningContinual Learning (Educational Psychology)
Catastrophic forgetting limits neural networks’ ability to learn sequential tasks, a problem rooted in the plasticity‑stability dilemma that has attracted much attention but has been rarely examined in terms of how training regimes affect forgetting. This study investigates whether the geometrical properties of local minima, shaped by training regimes, influence the degree of forgetting. The authors examine how dropout, learning‑rate decay, and batch size affect the width of task‑specific local minima to reduce catastrophic forgetting. Their results show that these simple training‑regime adjustments improve stability and outperform existing baselines.
Catastrophic forgetting affects the training of neural networks, limiting their ability to learn multiple tasks sequentially. From the perspective of the well established plasticity-stability dilemma, neural networks tend to be overly plastic, lacking the stability necessary to prevent the forgetting of previous knowledge, which means that as learning progresses, networks tend to forget previously seen tasks. This phenomenon coined in the continual learning literature, has attracted much attention lately, and several families of approaches have been proposed with different degrees of success. However, there has been limited prior work extensively analyzing the impact that different training regimes -- learning rate, batch size, regularization method-- can have on forgetting. In this work, we depart from the typical approach of altering the learning algorithm to improve stability. Instead, we hypothesize that the geometrical properties of the local minima found for each task play an important role in the overall degree of forgetting. In particular, we study the effect of dropout, learning rate decay, and batch size, on forming training regimes that widen the tasks' local minima and consequently, on helping it not to forget catastrophically. Our study provides practical insights to improve stability via simple yet effective techniques that outperform alternative baselines.