Improving Person‐Fit Assessment by Correcting the Ability Estimate and Its Reference Distribution

Abstract

The standardized log‐likelihood of a response vector (l z ) is a popular IRT‐based person‐fit test statistic for identifying model‐misfitting response patterns. Traditional use of l z is overly conservative in detecting aberrance due to its incorrect assumption regarding its theoretical null distribution. This study proposes a method for improving the accuracy of person‐fit analysis using l z which takes into account test unreliability when estimating the ability and constructs the distribution for each l z through resampling methods. The Type I error and power (or detection rate) of the proposed method were examined at different test lengths, ability levels, and nominal α levels along with other methods, and power to detect three types of aberrance—cheating, lack of motivation, and speeding—was considered. Results indicate that the proposed method is a viable and promising approach. It has Type I error rates close to the nominal value for most ability levels and reasonably good power.