Answers to 20 Questions About Interrater Reliability and Interrater Agreement

TLDR

Interrater reliability and agreement indices have become increasingly common, partly driven by the rise of multilevel modeling in organizational research and their use to justify aggregating lower‑level data. This article aims to expose researchers to the issues surrounding the use of IRR and IRA indices with multilevel models. The authors present a question‑and‑answer tutorial, including appendices that demonstrate how to compute these indices in SPSS.

Abstract

The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.