LASSO: A feature selection technique in predictive modeling for machine learning

TLDR

Feature selection in machine learning removes redundant or irrelevant variables to improve interpretability and generalization, and while traditional methods such as OLS, stepwise, and partial least squares are sensitive to errors, alternatives like Ridge and LASSO have been developed over the past decades. The study investigates the characteristics of OLS, ridge, and LASSO regression methods. It applies these methods to real and simulated data using the R package to compare model fitting and prediction accuracy. The results show that the performance of OLS, ridge, and LASSO differs across datasets, with each method exhibiting distinct strengths in model fitting and predictive accuracy.

Abstract

Feature selection is one of the techniques in machine learning for selecting a subset of relevant features namely variables for the construction of models. The feature selection technique aims at removing the redundant or irrelevant features or features which are strongly correlated in the data without much loss of information. It is broadly used for making the model much easier to interpret and increase generalization by reducing the variance. Regression analysis plays a vital role in statistical modeling and in turn for performing machine learning tasks. The traditional procedures such as Ordinary Least Squares (OLS) regression, Stepwise regression and partial least squares regression are very sensitive to random errors. Many alternatives have been established in the literature during the past few decades such as Ridge regression and LASSO and its variants. This paper explores the features of the popular regression methods, OLS regression, ridge regression and the LASSO regression. The performance of these procedures has been studied in terms of model fitting and prediction accuracy using real data and simulated environment with the help of R package.