Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity

Abstract

Researchershavecriticizedchance-correctedagreementstatistics,particularly theKappastatistic,asbeingverysensitivetoraters'classiflcationprobabilities(marginal probabilities) and to trait prevalence in the subject population. Consequently, several authorshavesuggestedthatmarginalprobabilitiesbetestedforhomogeneityandthatany comparison between reliability studies be preceded by an assessment of trait prevalence amongsubjects.Theobjectiveofthispaperisthreefold:(i)todemonstratethatmarginal homogeneitytestingdoesnotpreventtheunpredictableresultsoftenobtainedwithsome ofthemostpopularagreementstatistics,(ii)topresentasimpleandreliableinter-rater agreement statistic, and (iii) to gain further insight into the dependency of agreement statistics upon trait prevalence.