Quantitative Interpretation of Two Qualitative Scales Used to Rate Military Intelligence

Abstract

Thirty-seven intelligence officers completed two replications of tasks designed to investigate their subjective, quantitative interpretations of the source reliability and information accuracy (plausibility) rating scales. In judging a report, subjects were influenced much more by the accuracy rating of the report's content than by the reliability rating of the report's source. The mean probabilities assigned to the truth likelihood of reports described a linear relationship between rating level and probability for each scale. Most subjects were unable to treat reliability and accuracy independently; for these subjects, the higher a report's reliability rating, the higher the accuracy rating expected, and vice versa. Subjects were relatively consistent in their interpretations, but marked differences between subjects were observed. Structural inadequacies of the scales are pointed out and the development of a single-dimensional, quantitative scale is recommended.