2002 · 25 citations · 9 references
ComputingComputer ScienceMachine LearningNeural EngineeringOptimizationGradient-based OptimizationNeural NetworksOptimization For Machine LearningPattern RecognitionBackpropagationFeedforward Neural NetworksGradient DescentGradient Method OptimizationLearningLearning Rate EstimationNonlinear Function Approximation
In this work, an efficient training algorithm for feedforward neural networks is presented. It is based on a scaled version of the conjugate gradient method suggested by Perry, which employs the spectral steplength of Barzilai and Borwein that contains second order information without estimating the Hessian matrix. The learning rate is automatically adapted at each epoch, using the conjugate gradient values and the learning rate of the previous one. In addition, a new acceptability criterion for the learning rate is utilized based on non-monotone Wolfe conditions. The efficiency of the training algorithm is proved on the standard tests, including XOR, 3-bit parity, font learning and function approximation problems. Keywords: back propagation, supervised training, conjugate gradient methods, Perry’s method, spectral steplength, non-monotone Wolfe conditions. 1
9
Two-Point Step Size Gradient Methods
Jonathan Barzilai, Jonathan M. Borwein · IMA Journal of Numerical Analysis · 1988 · 2.9K citations
Increased rates of convergence through learning rate adaptation
Robert A. Jacobs · Neural Networks · 1988 · 1.8K citations
Accelerating the convergence of the back-propagation method
Thomas P. Vogl, J. K. Mangis, A. K. Rigler et al. · Biological Cybernetics · 1988 · 1.1K citations