Application of Three Cognitive Diagnosis Models to ESL Reading and Listening Assessments

Abstract

The present study investigated the functioning of three psychometric models for cognitive diagnosis—the general diagnostic model, the fusion model, and latent class analysis—when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983 Tatsuoka, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20: 345–354. [Crossref], [Web of Science ®] , [Google Scholar], 1990 Tatsuoka, K. K. 1990. “Toward an integration of item-response theory and cognitive error diagnosis”. In Diagnostic monitoring of skill and knowledge acquisition, Edited by: Fredericksen, N., Glaser, R., Lesgold, A. and Shafto, M. G. 453–388. Hillsdale, NJ: Erlbaum. [Google Scholar]), for the Listening and Reading sections of two field test forms of the TOEFL® iBT (Test of English as a Foreign Language™). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into “flat” profiles (i.e., “nonmasters of all skills” and “masters of all skills”). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.