Interjudge Agreement and the Maximum Value of Kappa

Abstract

The observed degree of agreement between judges is commonly summarized using Cohen's (1960) kappa. Previous research has related values of kappa to the marginal distributions of the agreement matrix. This manuscript provides an approach for calculating maximum values of kappa as a function of observed agreement proportions between judges. Solutions are provided separately for matrices of size 2 x 2, 3 x 3, 4 x 4, and k X k; plots are provided for the 2 x 2, 3 x 3, and 4 x 4 matrices.