Assessing the Dimensionality of Item Response Matrices with Small Sample Sizes and Short Test Lengths

Abstract

Abstract The assumption of unidimensionality must be met to legitimately use common item response theory models. The validity of score-based inferences rests largely on the extent to which it can be shown that the dimensional structure underlying a test is consistent with the blueprint. Little research has been undertaken to examine the behavior of dimensionality assessment procedures in conditions similar to those encountered in small-volume administrations. The purpose of this study was to examine empirical. Type I error rates and rejections rates for 3-dimensionality assessment procedures with data sets simulated to reflect short tests and small samples. The TESTFACT (Wilson, Wood, & Gibbons, 1991) G2; difference test and the LISREL8 (Jöreskog & Sörbom, 1993a) chi-square statistic suffered from an inflated Type I error rate with unidimensional data sets, whereas the approximate chi-square statistic based on a NOHARM (Fraser & McDonald, 1988) analysis did not. Rejection rates with simulated 2-dimensional data sets were high for all procedures. The behavior of the G2; difference test was highly influenced by the independent variables manipulated, which was not the case for the approximate chi-square statistic. The implications of these results for small-volume administrations are discussed.