Weight Space Organization of Optimized Neural Networks

Abstract

We introduce techniques to study the weight space organization of neural networks optimizing different performance functions, by considering the total free energy in the joint weight space of two networks, and their correlation order parameter. The example of training noise performance functions shows that Hebbian-like and MSN-like networks occupy different regions in the «world map» projection of the weight space (MSN meaning the maximally stable network). The lines of maximum latitude, minimum susceptibility and band splitting separate the regions of Hebbian-like and MSN-like networks in similar ways. In the low storage limit, the differentiation of network behaviour is determined by the signal-to-noise ratio [Equation found] (m_tand α being the training overlap and the storage level, respectively). Possible applications of our technique are discussed.