Learning where to learn: Gradient sparsity in meta and continual\n learning

Abstract

Finding neural network weights that generalize well from small datasets is\ndifficult. A promising approach is to learn a weight initialization such that a\nsmall number of weight changes results in low generalization error. We show\nthat this form of meta-learning can be improved by letting the learning\nalgorithm decide which weights to change, i.e., by learning where to learn. We\nfind that patterned sparsity emerges from this process, with the pattern of\nsparsity varying on a problem-by-problem basis. This selective sparsity results\nin better generalization and less interference in a range of few-shot and\ncontinual learning problems. Moreover, we find that sparse learning also\nemerges in a more expressive model where learning rates are meta-learned. Our\nresults shed light on an ongoing debate on whether meta-learning can discover\nadaptable features and suggest that learning by sparse gradient descent is a\npowerful inductive bias for meta-learning systems.\n