Testing the Normal Approximation and Minimal Sample Size Requirements of Weighted Kappa When the Number of Categories is Large

Abstract

The results of this computer simulation study in dicate that the weighted kappa statistic, employing a standard error developed by Fleiss, Cohen, and Everitt (1969), holds for a large number of k cate gories of classification (e.g., 8 < k ≤ 10). These data are entirely consistent with an earlier study (Cicchetti & Fleiss, 1977), which showed the same results for 3 ≤ k ≤ 7. The two studies also indicate that the minimal N required for the valid ap plication of weighted kappa can be easily approxi mated by the simple formula 2 k 2 . This produces sample sizes that vary between a low of about 20 (when k = 3) to a high of about 200 (when k = 10). Finally, the range 3 ≤ k ≤ 10 should encompass most extant clinical scales of classification.