Exploring upper limits of reliability and validity in job performance ratings.

Abstract

Explored possible upper bound limits in reliability and validity of performance ratings by creating a nearly ideal environment for obtaining performance evaluations and then examining closely the precision of these ratings. 16 scripts describing persons performing on 2 jobs–recruiting interviewer and manager—were prepared in such a way that the performers' effectiveness on various dimensions of performance approached a preset, realistic level. Five- to 7-min performances of these scripts were videotaped, and 14 raters evaluated the effectiveness of each performer on each performance dimension. Raters were very knowledgeable about the 2 jobs, had ample opportunity to observe the performers, and used a relatively advanced behavior scale type of rating form. Thus, the opportunity for obtaining highly reliable and valid ratings was excellent. Levels of convergent and discriminant validities were high compared to other studies, but interrater agreement was far from perfect. An analysis of the performance rating process reveals the possible reasons for error in ratings, and research approaches to learn more about this process are discussed.