Permutation Multiple Tests of Binary Features Do Not Uniformly Control Error Rates

Abstract

Abstract Multiple testing for significant association between predictors and responses has a wide array of applications. One such application is pharmacogenomics, where testing for association between responses and many genetic markers is of interest. Permuting response group labels to generate a reference distribution is often thought of as a convenient thresholding technique that automatically captures dependence in the data. In reality, nontrivial model assumptions are required for permutation testing to control multiple testing error rates. When binary predictors (such as genetic markers) are individually tested by standard tests such as Fisher’s exact test, permutation multiple testing can give incorrect unconditional and, especially, conditional assessment of significances. Regardless of whether the sample sizes are equal, how misleading permutation assessment can be is primarily a function of the difference in the covariances among the genetic markers between the phenotype groups. Keywords: : Family-wise error rateFisher’s exact testPharmacogenomics