Algorithms for the Solution of the Nonlinear Least-Squares Problem

TLDR

The paper proposes a modification to the Gauss–Newton method for solving nonlinear least‑squares problems. The method improves the Hessian approximation by selectively including neglected terms, computes the search direction without forming or factoring the Hessian, and offers three variants—one using second derivatives and two using only first derivatives. The approach yields benefits comparable to avoiding the formation of normal equations in the Gauss–Newton method.

Abstract

This paper describes a modification to the Gauss–Newton method for the solution of nonlinear least-squares problems. The new method seeks to avoid the deficiencies in the Gauss–Newton method by improving, when necessary, the Hessian approximation by specifically including or approximating some of the neglected terms. The method seeks to compute the search direction without the need to form explicitly either the Hessian approximation or a factorization of this matrix. The benefits of this are similar to that of avoiding the formation of the normal equations in the Gauss-Newton method. Three algorithms based on this method are described; one which assumes that second derivative information is available and two which only assume first derivatives can be computed.