IEEE Transactions on Neural Networks · 2006 · 168 citations · 19 references
ComputingComputer ScienceAlgorithmsArtificial IntelligenceMachine LearningAdaptive ControlKalman FilteringNeural NetworksAdaptive RateBackpropagationConvergenceFeedforward Neural NetworksLyapunov ControlLyapunov FunctionsLyapunov StabilityLyapunov Stability Theory
This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.
19
An introduction to computing with neural nets
Richard P. Lippmann · IEEE ASSP Magazine · 1987 · 7.9K citations
Linear system theory and design
Stephen M. Barnett · Automatica · 1986 · 3.7K citations
Increased rates of convergence through learning rate adaptation
Robert A. Jacobs · Neural Networks · 1988 · 1.8K citations
Accelerating the convergence of the back-propagation method
Thomas P. Vogl, J. K. Mangis, A. K. Rigler et al. · Biological Cybernetics · 1988 · 1.1K citations