Computer-based vs. Paper-based Testing: Does the test administration mode matter?

Abstract

Evaluating the comparability of paper-based and computer-based tests is crucial before introducing computer aided assessment into any context. There have been several comparability studies that have examined the impact of transferring a test from paper to screen. Such studies have either focused on the comparability of the product of the tests i.e. scores, or on the processes used to achieve that product. However, Chapelle & Douglas (2006:46) argue that “To date not a single study of L2 testing has examined directly whether or not past experience with computers affects test performance on a computer-based L2 test”. Sawaki (2001) recommended that this type of empirical work should utilize different methodologies such as eye movement, verbal protocols, post hoc interviews and questionnaires in order to obtain useful results. Chalhoub-Deville and Deville (1999) pointed out that there is a scarcity of comparability research on localised language tests needed to detect any potential impact of the test delivery mode when converting conventional paper tests to computerised tests. Thus, our ongoing study explores the comparability of paper and computer-based testing in an L2 reading context and the impact of test takers' characteristics, i.e., computer familiarity, computer attitude, testing mode preference and test taking strategies on students' performance on computer-based tests, and in comparison with paper-based tests. 167 Saudi medical students participated in this study. The study used several quantitative and qualitative instruments to gather data. This paper reports on the results of the quantitative instruments of the study i.e. the tests and the questionnaires and the relevant interviews. We found no significant difference between each testing mode and none of the factors examined had an influence on students’ performance when doing the computer-based tests. The study is still in the process of analyzing the qualitative data and hopes to report on that soon.