QUALIFYING ESSAY READERS FOR AN ONLINE SCORING NETWORK (OSN)

Abstract

ABSTRACT Currently, personnel who evaluate examinee‐constructed responses for ETS‐administered testing programs must possess certain academic credentials. This study was designed to determine the extent to which current prerequisites for essay readers might be relaxed without sacrificing quality with regard to the accuracy with which essay scores are awarded. The specific interest was in a particular class of potential readers, namely, those who are not currently involved in postsecondary teaching and who therefore do not meet current standards for ETS essay readers. Secondary objectives of the study were to (1) provide information that would facilitate the determination of a passing score on the certification test that all ETS essay readers will be required to pass in the future and (2) evaluate, in a preliminary fashion, the effects of training on the performance of essay readers. To accomplish these objectives, both experienced and inexperienced readers were recruited for the study. All study participants evaluated a set of essays before any training and another set after they had undergone standard training for scoring essays. The results showed that training did affect the accuracy with which readers (especially those who were previously inexperienced) scored essays. Moreover, after training, a significant proportion of inexperienced readers exhibited a level of accuracy that was commensurate with that shown by experienced readers. A comparison of the performances of experienced and inexperienced readers suggested that a defensible passing score of about 90% accuracy could be set on the reader certification test when the standard of accuracy is agreement within one point of previously established essay scores.