Anomalies in the Foundations of Ridge Regression

Abstract

Summary Errors persist in ridge regression, its foundations, and its usage, as set forth in Hoerl & Kennard (1970) and elsewhere. Ridge estimators need not be minimizing, nor a prospective ridge parameter be admissible. Conventional estimators are not LaGrange's solutions constrained to fixed lengths, as claimed, since such solutions are singular. Of a massive literature on estimation, prediction, cross–validation, choice of ridge parameter, and related issues, little emanates from constrained optimization to include inequality constraints. The problem traces to a misapplication of LaGrange's Principle, unrecognized singularities, and misplaced links between constraints and ridge parameters. Alternative principles, based on condition numbers, are seen to validate both conventional ridge and surrogate ridge regression to be defined. Numerical studies illustrate that ridge regression as practiced often exhibits pathologies it is intended to redress.