Development of a general solution to the problem of validity generalization.

TLDR

Personnel psychologists have traditionally believed that employment test validities are situation specific, yet variability in raw validation results across studies suggests that differences in job performance factor structure and observer limitations may explain this phenomenon. The study aims to develop a Bayesian statistical model that tests whether observed variability in validity outcomes across similar jobs and tests is artifactual, enabling validity generalization without new validation studies. Using the Bayesian model, the authors analyze empirical validity coefficient distributions and, when generalization is not justified, provide an improved data‑analysis and decision‑making procedure for necessary situational validity studies. Application to four empirical validity coefficient distributions demonstrated the model’s power, revealing that required sample sizes for adequate power are much larger than assumed, leading to the conclusion that empirical validity studies are technically feasible far less frequently than believed, and that test validity is highly situation specific.

Abstract

U. S. Civil Service Commission andGeorge Washington UniversityJohn E. HunterMichigan State UniversityPersonnel psychologists have traditionally believed that employment test valid-ities are situation specific. This study presents a Bayesian statistical modelwhich allows one to explore the alternate hypothesis that variation in validityoutcomes from study to study for similar jobs and tests is artifactual in nature.Certain outcomes using this model permit validity generalization to new settingswithout carrying out a validation study of any kind. Where such generalizationis not justified, the procedure provides an improved method of data analysisand decision making for the necessary situational validity study. Application tofour distributions of empirical validity coefficients demonstrated the power ofthe model.A recent study (Schmidt, Hunter, & Urry,1976) addressed the belief, dominant in per-sonnel psychology, that meaningful empiricalvalidation studies are possible for most, if notall, jobs in most organizations. This studyshowed that, because of range restriction andless than perfect criterion reliability, thesample sizes necessary to provide adequatestatistical power are usually much larger thanhas typically been assumed, This finding leadsto the conclusion that empirical validitystudies are technically feasible much lessfrequently than the profession has assumed.The present study is addressed to another ofthe orthodoxies of personnel psychology: thebelief that test validity is generally highlysituation specific. Considerable variabilityfrom study to study is observed in raw valida-tion results even when jobs and tests appearto be similar or essentially identical (Ghiselli,1966). The explanation usually advanced forthis phenomenon is that the factor structureof job performance is different from job tojob and that the human observer or job ana-lyst is simply too poor an information receiver