Automated essay evaluation: the criterion online writing service

TLDR

The article presents the Criterion Online Essay Evaluation Service, a web‑based platform that automatically scores and evaluates student essays to provide teacher‑guided feedback for writing improvement. The service combines Critique Writing Analysis Tools, which detect grammatical, usage, mechanical, discourse, and stylistic errors, with e‑rater v2.0, an automated scoring engine, both powered by natural language processing and machine learning techniques. The tools outperform baseline algorithms and achieve inter‑rater agreement with human judges comparable to the agreement between two human judges.

Abstract

In this article, we describe a deployed educational technology application: the Criterion Online Essay Evaluation Service, a web-based system that provides automated scoring and evaluation of student essays. Criterion has two complementary applications: (1) Critique Writing Analysis Tools, a suite of programs that detect errors in grammar, usage, and mechanics, that identify discourse elements in the essay, and that recognize potentially undesirable elements of style, and (2) e-rater version 2.0, an automated essay scoring system. Critique and e-rater provide students with feedback that is specific to their writing in order to help them improve their writing skills and is intended to be used under the instruction of a classroom teacher. Both applications employ natural language processing and machine learning techniques. All of these capabilities outperform baseline algorithms, and some of the tools agree with human judges in their evaluations as often as two judges agree with each other.