Using Writing Process and Product Features to Assess Writing Quality and Explore How Those Features Relate to Other Literacy Tasks

Abstract

This paper explores automated methods for measuring features of student writing and determining their relationship to writing quality and other features of literacy, such as reading rest scores. In particular, it uses the e‐rater ® automatic essay scoring system to measure product features (measurable traits of the final written text) and features extracted from keystroke logs to measure process features (measurable features of the writing process). These techniques are applied to student essays written during large‐scale pilot administrations of writing assessments developed for ETS 's CBAL ™ research initiative. The design makes it possible to explore the factor structures of these product and process features and to examine how well they generalize beyond a single test session to predict underlying traits such as writing ability and reading level. Three product factors are identified, connected to fluency, accuracy, and content. The process factors are identified, corresponding to hesitancy behaviors, editing behaviors, and burst span (the extent to which text is produced in long bursts with only short internal pauses). The results suggest that writing process and product features have stable factor structures that generalize to predict writing quality, though there are some genre‐ or task‐specific differences.