Adaptive Levenberg–Marquardt Algorithm: A New Optimization Strategy for Levenberg–Marquardt Neural Networks

TLDR

Engineering data are often highly nonlinear and contain high‑frequency noise, causing the Levenberg–Marquardt algorithm to fail to converge when training neural networks. The study investigates why the LM algorithm frequently fails on such data and proposes an adaptive LM (AdaLM) algorithm to remedy this issue. AdaLM adjusts the descent direction and step size based on iteration number to avoid local minima and reduce parameter‑state effects, while the work also evaluates how different activation functions and parameter settings affect LM network performance. Experiments on common datasets and aero‑engine data demonstrate that AdaLM achieves higher accuracy and faster convergence than the traditional LM algorithm and its variants.

Abstract

Engineering data are often highly nonlinear and contain high-frequency noise, so the Levenberg–Marquardt (LM) algorithm may not converge when a neural network optimized by the algorithm is trained with engineering data. In this work, we analyzed the reasons for the LM neural network’s poor convergence commonly associated with the LM algorithm. Specifically, the effects of different activation functions such as Sigmoid, Tanh, Rectified Linear Unit (RELU) and Parametric Rectified Linear Unit (PRLU) were evaluated on the general performance of LM neural networks, and special values of LM neural network parameters were found that could make the LM algorithm converge poorly. We proposed an adaptive LM (AdaLM) algorithm to solve the problem of the LM algorithm. The algorithm coordinates the descent direction and the descent step by the iteration number, which can prevent falling into the local minimum value and avoid the influence of the parameter state of LM neural networks. We compared the AdaLM algorithm with the traditional LM algorithm and its variants in terms of accuracy and speed in the context of testing common datasets and aero-engine data, and the results verified the effectiveness of the AdaLM algorithm.