Reliability and Inter-rater Reliability in Qualitative Research

TLDR

Reliability in qualitative research is a contested concept, especially in interdisciplinary fields such as CSCW and HCI where diverse epistemologies complicate its application. This article aims to clarify reliability in qualitative research by applying two complementary approaches—examining local norms and synthesizing methodological guidelines—to inform practice in CSCW and HCI. The authors first survey local norms in CSCW/HCI literature, then conduct a meta‑analysis of 2016‑2018 papers and combine these findings with methods literature to develop practical guidelines for reporting reliability, particularly inter‑rater reliability. The meta‑analysis shows that inter‑rater reliability is reported in only about one‑ninth of qualitative papers, and the resulting guidelines are intended to spark discussion and guide new scholars and reviewers.

Abstract

What does reliability mean for building a grounded theory? What about when writing an auto-ethnography? When is it appropriate to use measures like inter-rater reliability (IRR)? Reliability is a familiar concept in traditional scientific practice, but how, and even whether to establish reliability in qualitative research is an oft-debated question. For researchers in highly interdisciplinary fields like computer-supported cooperative work (CSCW) and human-computer interaction (HCI), the question is particularly complex as collaborators bring diverse epistemologies and training to their research. In this article, we use two approaches to understand reliability in qualitative research. We first investigate and describe local norms in the CSCW and HCI literature, then we combine examples from these findings with guidelines from methods literature to help researchers answer questions like: "should I calculate IRR?" Drawing on a meta-analysis of a representative sample of CSCW and HCI papers from 2016-2018, we find that authors use a variety of approaches to communicate reliability; notably, IRR is rare, occurring in around 1/9 of qualitative papers. We reflect on current practices and propose guidelines for reporting on reliability in qualitative research using IRR as a central example of a form of agreement. The guidelines are designed to generate discussion and orient new CSCW and HCI scholars and reviewers to reliability in qualitative research.