The Adaptive Lasso and Its Oracle Properties

TLDR

The lasso is a popular method for simultaneous estimation and variable selection, and its variable selection has been shown to be consistent under certain conditions. The authors derive a necessary condition for lasso consistency and introduce the adaptive lasso, which uses coefficient‑specific adaptive weights in the ℓ1 penalty. The adaptive lasso employs adaptive weights for each coefficient in the ℓ1 penalty and can be solved with the same efficient algorithm as the standard lasso. The adaptive lasso attains oracle properties and near‑minimax optimality, remains consistent in generalized linear models under mild conditions, whereas the standard lasso can be inconsistent in some scenarios, and the nonnegative garotte is also shown to be consistent.

Abstract

The lasso is a popular technique for simultaneous estimation and variable selection. Lasso variable selection has been shown to be consistent under certain conditions. In this work we derive a necessary condition for the lasso variable selection to be consistent. Consequently, there exist certain scenarios where the lasso is inconsistent for variable selection. We then propose a new version of the lasso, called the adaptive lasso, where adaptive weights are used for penalizing different coefficients in the ℓ1 penalty. We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance. Similar to the lasso, the adaptive lasso is shown to be near-minimax optimal. Furthermore, the adaptive lasso can be solved by the same efficient algorithm for solving the lasso. We also discuss the extension of the adaptive lasso in generalized linear models and show that the oracle properties still hold under mild regularity conditions. As a byproduct of our theory, the nonnegative garotte is shown to be consistent for variable selection.