Recurrent neural networks learn robust representations by dynamically balancing compression and expansion

Abstract

Abstract Recordings of neural circuits in the brain reveal extraordinary dynamical richness and high variability. At the same time, dimensionality reduction techniques generally uncover low-dimensional structures underlying these dynamics. What determines the dimensionality of activity in neural circuits? What is the functional role of this dimensionality in behavior and task learning? In this work we address these questions using recurrent neural network (RNN) models, which have recently shown promise in predicting and explaining brain dynamics. Through simulations and mathematical analysis, we show how the dimensionality of RNN activity evolves over time and over stages of learning. We find that RNNs can learn to balance tendencies to expand and compress dimensionality in a way that matches task demands and further generalizes to new data. Strongly chaotic networks appear particularly adept in learning this balance in the case of classifying low-dimensional inputs, combining the natural tendency of chaos to expand dimensionality with opportunistic compression driven by stochastic gradient descent to form representations with good generalization properties. These findings shed light on fundamental dynamical mechanisms by which neural networks solve tasks with robust representations that generalize to new cases.