Estimating the Reliability, Systematic Error and Random Error of Interval Data

Abstract

IN content analysis as in all situations in which unstructured observations play an important role, the reliability with which data are generated is of crucial importance. Low data reliability limits the confidence in the validity of subsequent inferences and the reliability of a population of data must be estimated from the agreement among many observers regarding a sample. The way reliability of data is assessed is not different in principle from the way the reliability of psychological tests is measured. However, in the process of developing recording instructions, de-fining units of analysis and operationalizing scales, the researcher requires more detailed information about the sources and kind of unreliability. Over-all measures of agreement do not provide such information readily. More specifically, the analyst of a recording instrument may wish to obtain: