The American Statistician · 2007 · 69 citations · 19 references
EducationBiostatisticsItem Response TheoryPsychometricsStatisticsData AnalysisData InterpretationInterpretation Of ResultsMeasurement PrecisionScientific EvidenceAgreement AnalysisAgreement StatisticsCohen's Kappa InterpretationInter-observer AgreementInterrater AgreementKappa CoefficientKappa StatisticMeasures Of Agreement
Kappa-type indices use the concept of agreement to express the reproducibility of nominal measurements. This article grounds kappa-type indices in statistical modeling, making explicit the underlying premises and assumptions. We critically review whether the interpretation of the kappa index as a chance-corrected probability of agreement can be substantiated. Further, we show that the so-called paradoxical behavior of the kappa index is explained from the fact that it is a measure of predictive association, rather than a pure measure of reproducibility. We discuss a number of alternative forms, critically examining whether they can be translated in tangible real-life interpretations.
19
The Measurement of Observer Agreement for Categorical Data
J. Richard Landis, Gary G. Koch · Biometrics · 1977 · 78.9K citations · Full text