Molecular Physics · 2018 · 59 citations · 24 references
ComputingComputer SciencePhysicsAlgorithmsArtificial IntelligenceMachine LearningDeep LearningEntropyLoss FunctionNeural NetworksOptimization For Machine LearningStochastic OptimizationEntropy MinimizationGradient DescentMinimax OptimizationOptimization AlgorithmsParameter InferenceStatistical Learning TheoryStochastic Gradient Descent
Finding parameters that minimise a loss function is central to machine learning, and although stochastic gradient descent (SGD) is widely used and achieves state‑of‑the‑art results, it typically cannot reach the global minimum, making its empirical success mysterious and suggesting that wide but shallow minima may be optimal under undersampling, analogous to energy–entropy competition in physics. The study derives a correspondence between parameter inference in machine learning and free‑energy minimisation in statistical physics. The authors model SGD as free‑energy minimisation with undersampling acting as temperature, and illustrate this framework on deep‑learning image classification and a linear neural network to analytically link entropy to out‑of‑sample error. The authors find that SGD’s stochasticity has a non‑trivial correlation structure that biases it toward wide minima, and demonstrate this effect in deep‑learning image classification and a linear neural network, where entropy is analytically linked to out‑of‑sample error.
Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent (SGD) algorithm is widely used and delivers state-of-the-art results for many problems. Nonetheless, SGD typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy–entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.
24
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
null · Philosophy study · 2016 · 28.3K citations · Full text
Mastering the game of Go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison et al. · Nature · 2016 · 15.8K citations
Advances in Neural Information Processing Systems 28
Peter Flach, Meelis Kull · 2015 · 14.7K citations
A Stochastic Approximation Method
Herbert Robbins, Sutton Monro · The Annals of Mathematical Statistics · 1951 · 9.6K citations · Full text