An aberrant abundance of Cronbach’s alpha values at .70

Abstract

Cronbach’s alpha (α) is the most widely reported metric of the reliability of psychological measures. Decisions about an observed α’s adequacy are often made using rule-of-thumb thresholds, such as α of at least .70. Such thresholds can put pressure on researchers to make their measures meet these criteria, similar to the pressure to meet the significance threshold with p values. We examined whether α values reported in the psychology literature are inflated at the rule-of-thumb thresholds (α = .70, .80, .90) due to, for example, overfitting to in-sample data (α-hacking) or publication bias. We extracted reported α values from three very large datasets covering the general psychology literature (>30,000 α values taken from >74,000 published articles in APA journals), the Industrial and Organizational psychology literature (>89,000 α values taken from >14,000 published articles in I/O journals), and the APA’s PsycTests database which aims to cover all psychological measures published since 1894 (>67,000 α values taken from >60,000 measures). The distributions of these values show robust evidence of excesses at the α = .70 rule-of-thumb threshold which cannot be explained by justifiable measurement practices. We discuss the scope, causes, and consequences of α-hacking and how increased transparency, preregistration of measurement strategy, and standardized protocols could mitigate this problem. Code and data are available at osf.io/pe3t7. Supplementary materials at osf.io/5xzy4.