Spectral dynamics of learning in restricted Boltzmann machines

Abstract

The Restricted Boltzmann Machine (RBM), an important tool used in machine\nlearning in particular for unsupervized learning tasks, is investigated from\nthe perspective of its spectral properties. Starting from empirical\nobservations, we propose a generic statistical ensemble for the weight matrix\nof the RBM and characterize its mean evolution. This let us show how in the\nlinear regime, in which the RBM is found to operate at the beginning of the\ntraining, the statistical properties of the data drive the selection of the\nunstable modes of the weight matrix. A set of equations characterizing the\nnon-linear regime is then derived, unveiling in some way how the selected modes\ninteract in later stages of the learning procedure and defining a deterministic\nlearning curve for the RBM.\n