Analyzing the Reliability of Multidimensional Measures: An Example from Intelligence Research

Abstract

Two aspects of the reliability of multidimensional measures can be distinguished: the amount of scale score variance that is accounted for by all underlying factors (composite reliability) and the degree to which the scale score reflects one particular factor (construct reliability). Confidence intervals for composite and construct reliabilities can be estimated by bootstrap methods. The authors demonstrate the application of these methods by analyzing the reliability of an eight-factor, nested-factor model that represents the structure of 45 tasks in an intelligence test ( N= 1,233). Composite reliabilities ranged between .78 and .93, whereas construct reliabilities ranged between .17 and .68 when the scale indicators were equally weighted to compute the scale scores and between .52 and .90 with weights based on pattern coefficients. The results indicate the importance of distinguishing diagnostic from research applications when judging whether the reliability values of multidimensional measures are substantial.