A Comparison of Exposure Control Procedures in CAT Systems Based on Different Measurement Models for Testlets

Abstract

This study compared several exposure control procedures for CAT systems based on the three-parameter logistic testlet response theory model (Wang, Bradlow, & Wainer, 2002 Wang, X., Bradlow, E. T. and Wainer, H. 2002. A general Bayesian model for testlets: Theory and application. Applied Psychological Measurement, 26(1): 109–128. [Crossref] , [Google Scholar]) and Masters’ (1982 Masters, G. N. 1982. A Rasch model for partial credit scoring. Psychometrika, 47: 149–174. [Crossref], [Web of Science ®] , [Google Scholar]) partial credit model when applied to a pool consisting entirely of testlets. The exposure control procedures studied were the modified within .10 logits procedure (Davis & Dodd, 2003 Davis, L. L. and Dodd, B. G. 2003. Item exposure constraints for testlets in the verbal reasoning section of the MCAT. Applied Psychological Measurement, 27: 335–356. [Crossref], [Web of Science ®] , [Google Scholar]), the Sympson-Hetter procedure (Sympson & Hetter, 1985 Sympson, J. B. and Hetter, R. D. October 1985. “Paper presented at the annual meeting of the Military Testing Association”. In Controlling item exposure rates in computerized adaptive testing, October, San Diego, CA: Navy Personnel Research and Development Center. [Google Scholar]) with a maximum exposure rate restricted to 0.19, the Sympson-Hetter procedure (Sympson & Hetter, 1985 Sympson, J. B. and Hetter, R. D. October 1985. “Paper presented at the annual meeting of the Military Testing Association”. In Controlling item exposure rates in computerized adaptive testing, October, San Diego, CA: Navy Personnel Research and Development Center. [Google Scholar]) with a maximum exposure rate restricted to 0.29, and maximum information, a no exposure control condition, used as a baseline. The exposure control procedures were evaluated for measurement precision, utilization of the testlet pool, and testlet overlap across test administrations. For both measurement models, the modified within .10 logits procedure provided better pool utilization with little decrement in precision of measurement than either of the Sympson-Hetter procedures.