Guidelines for Improving the Power Values of Statistical Tests for Nonresponse Bias Assessment in OM Research

Abstract

ABSTRACT The assessment of nonresponse bias in survey‐based empirical studies plays an important role in establishing the credibility of research results. Statistical methods that involve the comparison of responses from two groups (e.g., early vs. late respondents) on multiple characteristics, which are relevant to the study, are frequently utilized in the assessment of nonresponse bias. We consider the concepts of individual and complete statistical power used for multiple testing and show their relevance for determining the number of statistical tests to perform when assessing nonresponse bias. Our analysis of factors that influence both individual and complete power levels, yielded recommendations that can be used by operations management (OM) empirical researchers to improve their assessment of nonresponse bias. A power analysis of 61 survey‐based research papers published in three prestigious academic operations management journals, over the last decade, showed the occurrence of very low (<0.4) power levels in some of the statistical tests used for assessing nonresponse bias. Such low power levels can lead to erroneous conclusions about nonresponse bias, and are indicators of the need for more rigor in the assessment of nonresponse bias in OM research.