Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement

TLDR

Interest in automated writing evaluation (AWE) in second‑language classrooms has risen, driven by the belief that AWE can free teachers to focus on higher‑level writing skills while handling lower‑level grammar, yet evidence supporting these claims remains limited. The study compared two second‑language writing classes, one receiving AWE plus teacher feedback and the other receiving teacher‑only feedback, to test claims about AWE’s impact and to situate findings within an argument‑based validation framework. The authors implemented a quasi‑experimental design with two classes, one assigned to an AWE + teacher feedback condition and the other to a teacher‑only condition, and examined resulting feedback patterns and student revisions. Using AWE alongside teacher feedback did not significantly increase higher‑level teacher feedback, teachers without AWE provided more lower‑level feedback, students revised teacher lower‑level feedback more than computer feedback, and those with AWE retained accuracy improvements long‑term while those without showed lower retention.

Abstract

Recent years have witnessed an increasing interest in the use of automated writing evaluation (AWE) in second language writing classrooms. This increase is partially due to the belief that AWE can assist teachers by allowing them to devote more feedback to higher-level (HL) writing skills, such as content and organization, while the technology addresses lower-level (LL) skills, such as grammar. As is speculated, student revisions will then be positively impacted. However, little evidence has supported these claims, calling into question the impact of AWE on teaching and learning. The current study explored these claims by comparing two second language writing classes that were assigned to either an AWE + teacher feedback condition or a teacher-only-feedback condition. Findings suggest that using AWE as a complement to teacher feedback did not have a significant impact on the amount of HL teacher feedback, but the teacher who did not use AWE tended to provide a greater amount of LL feedback than AWE alone. Furthermore, students seemed to revise the teacher's LL feedback more frequently than LL feedback from the computer. Interestingly, students retained their improvement in accuracy in the long-term when they had access to AWE, but students who did not have access appeared to have lower retention. We explain the relevance of our findings in relation to an argument-based validation framework to align our work with state-of-the-art research in the field and contribute to a broader discussion about how AWE can be best provided to support second language writing development.