arXiv (Cornell University) · 2012 · 53 citations · 8 references
Open access
ComputingComputer ScienceMachine LearningOptimizationStatistical LearningAdaptive LearningConvex OptimizationGradient-based OptimizationOptimization For Machine LearningReinforcement LearningStochastic OptimizationAdaptive OptimizationExpected Prediction ErrorGradient DescentStochastic Gradient DescentTuning
The performance of stochastic gradient de-scent (SGD) depends critically on how learn-ing rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any one time. The method relies on local gradient variations across sam-ples. In our approach, learning rates can in-crease as well as decrease, making it suitable for non-stationary problems. Using a num-ber of convex and non-convex learning tasks, we show that the resulting algorithm matches the performance of SGD or other adaptive approaches with their best settings obtained through systematic search, and effectively re-moves the need for learning rate tuning. 1.
8
A Stochastic Approximation Method
Herbert Robbins, Sutton Monro · The Annals of Mathematical Statistics · 1951 · 9.6K citations · Full text
Léon Bottou, Y. Le Cun · 2003 · 337 citations