The Intraclass Correlation Coefficient as a Measure of Reliability

TLDR

The intraclass correlation coefficient (ICC) is a variance‑ratio measure that can be interpreted as a correlation coefficient in one‑way classification, but its application to two‑way models requires caution because between‑rater variance is removed. The paper proposes a procedure for estimating the reliability of rating sets using the ICC. The procedure derives ICC estimates from analysis‑of‑variance based variance‑component estimation. The authors argue that applying the standard one‑way ICC definition to two‑way models is inappropriate for correlation interpretation and present alternative two‑way ICC definitions that legitimately serve as correlation coefficients.

Abstract

A procedure for estimating the reliability of sets of ratings in terms of the intraclass correlation coefficient is discussed. The procedure is based upon the analysis of variance and the estimation of variance components. For the one-way classification the intraclass correlation coefficient defined as the ratio of variances can be interpreted as a correlation coefficient. Caution, however, is urged in the application of the definition to a two-way model, i.e., one in which between-rater variance is removed. It is maintained that the frequent use of the standard definition of the one-way intraclass correlation coefficient applied to the two-way classification is not a proper procedure if in fact the coefficient is to be interpreted as a correlation coefficient. Definitions for reliability obtained from the two-way models are given which can legitimately be considered correlation coefficients.