2002 · 139 citations · 8 references
ComputingComputer ScienceAlgorithmsArtificial IntelligenceMachine LearningGradient-based OptimizationNeural NetworksOptimization For Machine LearningAccuracyBackpropagationConvergenceGradient DescentLearningOptimization AlgorithmsTrainingTraining Time
In gradient descent learning algorithms such as error backpropagation, the learning rate parameter can have a significant effect on generalization accuracy. In particular, decreasing the learning rate below that which yields the fastest convergence can significantly improve generalization accuracy, especially on large, complex problems. The learning rate also directly affects training speed, but not necessarily in the way that many people expect. Many neural network practitioners currently attempt to use the largest learning rate that still allows for convergence, in order to improve training speed. However, a learning rate that is too large can be as slow as a learning rate that is too small, and a learning rate that is too large or too small can require orders of magnitude more training time than one that is in an appropriate range. The paper illustrates how the learning rate affects training speed and generalization accuracy, and thus gives guidelines on how to efficiently select a learning rate that maximizes generalization accuracy.
8
Neural networks for pattern recognition
Choice Reviews Online · 1994 · 18.7K citations
Fundamentals of Artificial Neural Networks
M.H. Hassoun · Proceedings of the IEEE · 1996 · 1.9K citations
Increased rates of convergence through learning rate adaptation
Robert A. Jacobs · Neural Networks · 1988 · 1.8K citations
Accelerating the convergence of the back-propagation method
Thomas P. Vogl, J. K. Mangis, A. K. Rigler et al. · Biological Cybernetics · 1988 · 1.1K citations
Fundamentals of Artificial Neural Networks
M.H. Hassoun, Nathan Intrator, Susan R. McKay et al. · Computers in Physics · 1996 · 856 citations