Using Multidimensional Item Response Theory to Understand What Items and Tests Are Measuring

TLDR

Item response theory models examinee-item interactions assuming a single underlying skill, but when data violate unidimensionality, multidimensional IRT is required to capture multiple abilities. The article aims to persuade testing professionals to adopt MIRT for statistically validating test specifications. Graphical MIRT analyses are introduced to illustrate what individual items and the overall test measure.

Abstract

Item response theory (IRT) describes the interaction between examinees and items using probabilistic models. One of the underlying assumptions of IRT is that examinees are all using the same skill or same composite of multiple skills to respond to each of the test items. When item response data do not satisfy the unidimensionality assumption, multidimensional item response theory (MIRT) should be used to model the item-examinee interaction. MIRT enables one to model the interaction of items that are capable of discrimi- nating between levels of several different abilities and examinees that vary in their proficiencies on these abilities. In this article graphical MIRT analyses designed to provide better insight into what individual items are measuring as well as what the test as a whole is assessing are presented and discussed. The goal of the article is to encourage testing practitioners to use MIRT as a means to statistically validate the test specifications.