Rorschach interrater agreement estimates: An empirical evaluation

Abstract

A standardized estimation of Rorschach interrater agreement is needed. Percentage agreement, although widely used, is found to be unsuitable. Forty‐one protocols from adults in both a normal and a psychiatric sample were scored by two or three scorers, making 85 scoring pairs. Percentage agreement, correlations (phi and Pearson’s r ), and kappa were computed on single response, total score, and category level. Percentage agreement shows minimal variation. Even when exceeding 0.80, it can obscure major disagreements. Kappa and correlations both vary in a similar way with level of disagreement. Total score level does not give additional information compared to single score and category levels. Kappa proved to be conservative and reliable and is therefore suggested as a standard estimate.