Personnel Psychology · 2006 · 144 citations · 30 references
Public HealthBehavioral SciencesPsychologyMethod ValidationPsychometricsTest DevelopmentBiasBias ControlPsychological TestingPublication BiasTest BiasTest ValidityBias In Psychological TestingFile Drawer EffectPublication Bias Detection
This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion‐related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate‐to‐severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.
30
The file drawer problem and tolerance for null results.
Robert Rosenthal · Psychological Bulletin · 1979 · 8.6K citations
Statistical Methods for Meta-Analysis.
P. R. Freeman, Larry V. Hedges, Ingram Olkin · Biometrics · 1986 · 4.8K citations
The Handbook of Research Synthesis.
D. Howel, H. Cooper, Larry V. Hedges · Biometrics · 1995 · 3.7K citations