Interobserver agreement, reliability, and generalizability of data collected in observational studies.

TLDR

Research in developmental and educational psychology has come to rely less on conventional psychometric tests and more on records of behavior made by human observers in natural and quasi‑natural settings. The study discusses three coefficients—interobserver agreement percentage, reliability coefficient, and generalizability coefficient—that reflect the quality of data collected in observational studies. The authors analyze these coefficients and examine the advantages of generalizability designs in observational research. The study concludes that high interobserver agreement alone is insufficient for data quality and that evidence of reliability or generalizability should also be reported.

Abstract

Research in developmental and educational psychology has come to rely less on conventional psychometric tests and more on records of behavior made by human observers in natural and quasi-natural settings. Three coefficients that purport to reflect the quality of data collected in these observational studies are discussed: the interobserver agreement percentage, the reliability coefficient, and the generalizability coefficient. It is concluded that although high interobserver agreement is desirable in observational studies, high agreement alone is not sufficient to insure the quality of the data that are collected. Evidence of the reliability or generalizability of the data should also be reported. Further advantages of generalizability designs are discussed.