IEEE Transactions on Neural Networks and Learning Systems · 2019 · 51 citations · 36 references
ComputingComputer ScienceAlgorithmsMachine LearningOptimizationGradient-based OptimizationOptimization For Machine LearningStochastic OptimizationAccelerated OptimizationGradient DescentLinear ConvergenceSteepest DescentStochastic Gradient Descent
The limited memory version of the Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm is the most popular quasi-Newton algorithm in machine learning and optimization. Recently, it was shown that the stochastic L-BFGS (sL-BFGS) algorithm with the variance-reduced stochastic gradient converges linearly. In this paper, we propose a new sL-BFGS algorithm by importing a proper momentum. We prove an accelerated linear convergence rate under mild conditions. The experimental results on different data sets also verify this acceleration advantage.
36
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
A Stochastic Approximation Method
Herbert Robbins, Sutton Monro · The Annals of Mathematical Statistics · 1951 · 9.6K citations · Full text