Journalism & Mass Communication Quarterly · 1996 · 239 citations · 0 references
Social SciencesEducational MeasurementCategorical Data AnalysisMethod ValidationPsychometricsData CodingReliabilitySampling TechniquesSurvey SamplingTest ReliabilityInter-coder AgreementInter-coder ReliabilityInter-rater ReliabilityInter-rater Reliability AssessmentIntercoder Agreement In EvaluationInterrater ReliabilitySamplingSampling Error In Reliability
This study views intercoder reliability as a sampling problem. It develops a formula for generating sample sizes needed to have valid reliability estimates. It also suggests steps for reporting reliability. The resulting sample sizes will permit a known degree of confidence that the agreement in a sample of items is representative of the pattern that would occur if all content items were coded by all coders.