Application of the rule-space procedure to language testing: examining attributes of a free response listening test

Abstract

This study uses the rule-space methodology to explore the cognitive and linguistic attributes that underlie performance on an open-ended, short-answer, listening comprehension test. This is the first application of the methodology to language testing. The rule-space methodology is an adaptation of statistical pattern recognition techniques applied to the problem of diagnosing the cognitive attributes (knowledge, skills, abilities, strategies, etc.) underlying test performance. The methodology provides diagnostic information about the individual test-takers on each of these attributes. Based on a literature search, attribute candidates were identified which seemed likely to explain performance on the listening test. Two rule-space analyses were carried out, and the final attribute list had 15 attributes, with 14 interactions. With this, 96% of the test-takers were successfully classified into their respective latent knowledge states, and given scores on each attribute. This result suggests that the rule-space methodology can be used to identify attributes underlying performance on language tests. The study also provided useful information about the listening construct.