arXiv (Cornell University) · 2018 · 13 citations · 8 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningMemoryDeep LearningLong-term MemoryMemory ConsolidationMulti-task LearningNeural NetworksSequence LearningAccuracyAdversarial ExamplesCatastrophic ForgettingElastic Weight ConsolidationGradient DescentInitial Performance EvaluationKnowledge ConsolidationMulti-task Deep Learning
Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge. Here, we propose a new approach to sequential learning which leverages the recent discovery of adversarial examples. We use adversarial subspaces from previous tasks to enable learning of new tasks with less interference. We apply our method to sequentially learning to classify digits 0, 1, 2 (task 1), 4, 5, 6, (task 2), and 7, 8, 9 (task 3) in MNIST (disjoint MNIST task). We compare and combine our Adversarial Direction (AD) method with the recently proposed Elastic Weight Consolidation (EWC) method for sequential learning. We train each task for 20 epochs, which yields good initial performance (99.24% correct task 1 performance). After training task 2, and then task 3, both plain gradient descent (PGD) and EWC largely forget task 1 (task 1 accuracy 32.95% for PGD and 41.02% for EWC), while our combined approach (AD+EWC) still achieves 94.53% correct on task 1. We obtain similar results with a much more difficult disjoint CIFAR10 task, which to our knowledge had not been attempted before (70.10% initial task 1 performance, 67.73% after learning tasks 2 and 3 for AD+EWC, while PGD and EWC both fall to chance level). Our results suggest that AD+EWC can provide better sequential learning performance than either PGD or EWC.
8
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Stably maintained dendritic spines are associated with lifelong memories
Guang Yang, Feng Pan, Wen‐Biao Gan · Nature · 2009 · 1.2K citations · Full text
Labelling and optical erasure of synaptic memory traces in the motor cortex
Akiko Hayashi‐Takagi, Sho Yagishita, Mayumi Nakamura et al. · Nature · 2015 · 656 citations
Branch-specific dendritic Ca2+ spikes cause persistent synaptic plasticity
Joseph Cichon, Wen‐Biao Gan · Nature · 2015 · 536 citations
The Space of Transferable Adversarial Examples
Florian Tramèr, Nicolas Papernot, Ian Goodfellow et al. · arXiv (Cornell University) · 2017 · 438 citations · Full text