Communication Methods and Measures · 2016 · 25 citations · 15 references
EducationEducational AssessmentEducational MeasurementAssessment FairnessItem Response TheoryPsychometricsBiasBias ControlContent AnalysisResponse BiasRisk Of Bias AssessmentTest BiasCodingCohen's Kappa InterpretationInter-rater Reliability AssessmentKappa CoefficientKappa StatisticMeasurement Bias CorrectionRater Bias AnalysisStatistical Simulation
This study demonstrates that NVivo, a popular research tool for content analysis, produces biased Cohen’s kappa as it calculates the reliability statistic at the character level, which is unsuitable for most content-analytic studies adopting a higher unit of analysis (e.g., sentence, paragraph). Based on empirical data and a statistical simulation, this study proposes a method of bias correction applicable to situations where two independent raters conduct binary coding.
15
Case study research: design and methods
Simon Phelan · Evaluation & Research in Education · 2011 · 21.9K citations
Assessing agreement on classification tasks: the kappa statistic
Jean Carletta · ArXiv.org · 1996 · 2.1K citations · Full text
Inter-Coder Agreement for Computational Linguistics
Ron Artstein, Massimo Poesio · Computational Linguistics · 2008 · 1.6K citations · Full text