Rating Scales As Measures of Clinical Judgment II: the Reliability of Example-Anchored Scales Under Conditions of Rater Heterogeneity and Divergent Behavior Sampling

Abstract

AN earlier communication to this journal (Taylor, 1968a) de- scribed a rationale and a method for constructing rating scales of clinical judgment.The method allows an assessment of unidimen- sionality for the variable being rated, as well as an estimate of expectable reliability between independent raters.It results in a thermometer-like rating scale of up to 100 points, with particular points being anchored by brief case-history vignettes.In the work reported earlier, a group of qualified clinicians used six such scales in rating individuals seen in jointly-conducted interviews.The same individuals were also rated on a series of traditional numerical rating scales, each end-anchored by polar adjectives.Under these conditions, with skilled clinicians rating the same observed behav- ior, it was shown that &dquo;example-anchored&dquo; scales constructed by the new technique had a markedly higher interjudge reliability, with the mean Pearson r being .76,and a reliability range from