The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses

TLDR

Partial least squares (PLS) is used to address collinearity among predictors in multiple regression and is equivalent to the conjugate gradient method in numerical analysis. The authors estimate successive PLS components by iteratively regressing residuals, determine the optimal rank via cross‑validation, compute jackknife standard errors, and compare the method to ridge regression and principal components regression on a chemical activity dataset.

Abstract

The use of partial least squares (PLS) for handling collinearities among the independent variables X in multiple regression is discussed. Consecutive estimates $({\text{rank }}1,2,\cdots )$ are obtained using the residuals from previous rank as a new dependent variable y. The PLS method is equivalent to the conjugate gradient method used in Numerical Analysis for related problems. To estimate the “optimal” rank, cross validation is used. Jackknife estimates of the standard errors are thereby obtained with no extra computation. The PLS method is compared with ridge regression and principal components regression on a chemical example of modelling the relation between the measured biological activity and variables describing the chemical structure of a set of substituted phenethylamines.