The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study

TLDR

Inter‑rater reliability is a standard quantitative method, but its relevance to qualitative coding—whether analysts should produce identical codes or distinct accounts—remains debated and has not been empirically tested. This study aimed to formally examine the expected level of inter‑rater reliability in qualitative research. Six researchers independently coded the same focus‑group transcript to identify themes. Analysts reached close agreement on core themes, yet each packaged the themes differently.

Abstract

Assessing inter-rater reliability, whereby data are independently coded and the codings compared for agreements, is a recognised process in quantitative research. However, its applicability to qualitative research is less clear: should researchers be expected to identify the same codes or themes in a transcript or should they be expected to produce different accounts? Some qualitative researchers argue that assessing inter-rater reliability is an important method for ensuring rigour, others that it is unimportant; and yet it has never been formally examined in an empirical qualitative study. Accordingly, to explore the degree of inter-rater reliability that might be expected, six researchers were asked to identify themes in the same focus group transcript. The results showed close agreement on the basic themes but each analyst `packaged' the themes differently.