An Empirical Study of the Effects of Small Datasets and Varying Prior Variances on Item Parameter Estimation in BILOG

Abstract

Long-standing difficulties in estimating item parameters in item response theory (IRT) have been addressed recently with the application of Bayesian estimation models. The potential of these methods is enhanced by their availability in the BILOG com puter program. This study investigated the ability of BILOG to recover known item parameters under varying conditions. Data were simulated for a two- parameter logistic IRT model under conditions of small numbers of examinees and items, and different variances for the prior distributions of discrimina tion parameters. The results suggest that for samples of at least 250 examinees and 15 items, BILOG accurately recovers known parameters using the default variance. The quality of the estimation suffers for smaller numbers of examinees under the default variance, and for larger prior variances in general. This raises questions about how practi tioners select a prior variance for small numbers of examinees and items.