The Annals of Statistics · 2005 · 1.1K citations · 27 references
ComputingBiostatisticsBayesian Data AnalysisStatistical ModelingUncertainty QuantificationBayesian InferenceBayesian StatisticsInferential StatisticsModel SelectionStatistical InferenceVariable SelectionForward SelectionModel UncertaintyPosterior Mean EstimationPriors For Variable SelectionRidge RegressionShrinkage PriorsSpike-and-slab PriorsSpike-and-slab Variable Selection
Variable selection in linear regression has both frequentist and Bayesian perspectives. The paper introduces a rescaled spike and slab model for variable selection. The authors develop a rescaled spike and slab framework that incorporates hierarchical priors, links to ridge regression, employs continuous bimodal priors for hypervariance, and uses a forward‑selection strategy to verify reduction of model uncertainty. Selective shrinkage via the rescaled spike and slab posterior improves variable selection accuracy and reduces model uncertainty.
Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure’s ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.
27
Bradley Efron, Trevor Hastie, Iain M. Johnstone et al. · The Annals of Statistics · 2004 · 9.5K citations · Full text
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Arthur E. Hoerl, Robert W. Kennard · Technometrics · 1970 · 8.9K citations
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Arthur E. Hoerl, Robert W. Kennard · Technometrics · 2000 · 6.6K citations
Linear Model Selection by Cross-validation
Jun Shao · Journal of the American Statistical Association · 1993 · 1.7K citations