Why We (Usually) Don't Have to Worry About Multiple Comparisons

TLDR

Applied researchers often face settings that appear to require multiple‑comparisons adjustments, yet classical procedures merely widen intervals or adjust p‑values, whereas multilevel models achieve partial pooling by shifting estimates toward each other. The authors challenge the traditional Type I error paradigm underlying multiple‑comparison corrections and argue that a hierarchical Bayesian perspective can eliminate the problem, proposing multilevel models for such settings and noting future improvements via time‑series data. They build multilevel models that address the multiple‑comparisons issue, yielding more efficient estimates especially when group‑level variation is.

Abstract

Abstract Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I error paradigm that underlies these corrections. Moreover we posit that the problem of multiple comparisons can disappear entirely when viewed from a hierarchical Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. Multilevel models perform partial pooling (shifting estimates toward each other), whereas classical procedures typically keep the centers of intervals stationary, adjusting for multiple comparisons by making the intervals wider (or, equivalently, adjusting the p values corresponding to intervals of fixed width). Thus, multilevel models address the multiple comparisons problem and also yield more efficient estimates, especially in settings with low group-level variation, which is where multiple comparisons are a particular concern. Keywords: Bayesian inferencehierarchical modelingmultiple comparisonsType S errorstatistical significance ACKNOWLEDGMENTS We thank the participants at the NCEE/IES multiple comparisons workshop for helpful comments and the National Science Foundation, National Institutes of Health, and Columbia University Applied Statistics Center for financial support. Notes The actual analysis also included birthweight group as a predictor in this model, but we ignore this in this description for simplicity of exposition. We recognize that this model could be improved, most naturally by embedding data from multiple years in a time series structure. The ability to include additional information in a reliable way is indeed a key advantage of multilevel models; however, here we chose a simple model because it uses no more information than was used in the published tables.