Neural Computation · 1994 · 561 citations · 33 references
ComputingComputer ScienceAlgorithmsMachine LearningOptimizationDeep LearningGradient-based OptimizationLarge-scale OptimizationMatrix ComputationNeural NetworksNumerical AlgorithmsBackpropagationBoltzmann MachinesGradient DescentHessian ApproximationOptimization AlgorithmsStochastic Gradient Descent
Just storing the Hessian H (the matrix of second derivatives δ 2 E/δw i δw j of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly calculates Hv, where v is an arbitrary vector. To calculate Hv, we first define a differential operator R v {f(w)} = (δ/δr)f(w + rv)|r=0, note that R v {▽w} = Hv and R v {w} = v, and then apply R v {·} to the equations used to compute ▽ w . The result is an exact and numerically stable procedure for computing Hv, which takes about as much computation, and is about as local, as a gradient evaluation. We then apply the technique to a one pass gradient calculation algorithm (backpropagation), a relaxation gradient calculation algorithm (recurrent backpropagation), and two stochastic gradient calculation algorithms (Boltzmann machines and weight perturbation). Finally, we show that this technique can be used at the heart of many iterative techniques for computing various properties of H, obviating any need to calculate the full Hessian.
33
William H. Press, Saul A. Teukolsky, William T. Vetterling et al. · 1994 · 15.4K citations
A scaled conjugate gradient algorithm for fast supervised learning
Martin F. Møller · Neural Networks · 1993 · 4.1K citations
A Practical Bayesian Framework for Backpropagation Networks
David Mackay · Neural Computation · 1992 · 2.9K citations