Interobserver Agreement and Disagreement in Continuous Recording Exemplified by Measurement of Behavior State

Abstract

Continuous observational recording was used to measure behavior state of 3 adults with profound and multiple disabilities. Observations were made over 56.7 hours, including 21.9 hours with two observers recording independently. Overall percentage agreement was satisfactory (i.e., exceeded 80%). Agreement on occurrence was, however, generally unsatisfactory at a mean of 63.5%, with 5 of 16 Subject x State Agreement Indices over 80%. Percentage disagreement on occurrence, a previously unreported measure, was computed to quantify sources of disagreement. The agreement data were superimposed on participants' behavior state profiles to demonstrate how conclusions can be drawn from the data, despite their generally inadequate reliability. This approach to interobserver agreement and presentation of data is suggested for researchers investigating entities that are not directly amenable to observation.