The Impact of Q-Matrix Designs on Diagnostic Classification Accuracy in the Presence of Attribute Hierarchies

Abstract

There is an increasing demand for assessments that can provide more fine-grained information about examinees. In response to the demand, diagnostic measurement provides students with feedback on their strengths and weaknesses on specific skills by classifying them into mastery or nonmastery attribute categories. These attributes often form a hierarchical structure because student learning and development is a sequential process where many skills build on others. However, it remains to be seen if we can use information from the attribute structure and work that into the design of the diagnostic tests. The purpose of this study is to introduce three approaches of Q-matrix design and investigate their impact on classification results under different attribute structures. Results indicate that the adjacent approach provides higher accuracy in a shorter test length when compared with other Q-matrix design approaches. This study provides researchers and practitioners guidance on how to design the Q-matrix in diagnostic tests, which are in high demand from educators.