Linear Model Selection by Cross-validation

TLDR

Leave‑one‑out cross‑validation, equivalent to AIC, Cp, and bootstrap, is asymptotically inconsistent, failing to select the best predictive model as sample size grows. The study aims to select the model with the best predictive ability among a class of linear models. The authors propose a leave‑nv‑out cross‑validation scheme with nv/n → 1 to correct the inconsistency and discuss its practical implementation. Simulation results show that this leave‑nv‑out approach successfully resolves the inconsistency and performs well in practice.

Abstract

Abstract We consider the problem of selecting a model having the best predictive ability among a class of linear models. The popular leave-one-out cross-validation method, which is asymptotically equivalent to many other model selection methods such as the Akaike information criterion (AIC), the C p , and the bootstrap, is asymptotically inconsistent in the sense that the probability of selecting the model with the best predictive ability does not converge to 1 as the total number of observations n → ∞. We show that the inconsistency of the leave-one-out cross-validation can be rectified by using a leave-n v -out cross-validation with n v , the number of observations reserved for validation, satisfying n v /n → 1 as n → ∞. This is a somewhat shocking discovery, because nv/n → 1 is totally opposite to the popular leave-one-out recipe in cross-validation. Motivations, justifications, and discussions of some practical aspects of the use of the leave-n v -out cross-validation method are provided, and results from a simulation study are presented.