AN EXAMINATION OF RATER ORIENTATIONS AND TEST-TAKER PERFORMANCE ON ENGLISH-FOR-ACADEMIC-PURPOSES SPEAKING TASKS

TLDR

The study explores oral English‑for‑academic‑purposes proficiency through two coordinated exploratory investigations. The authors conducted a verbal‑report study of expert rating orientations and a discourse‑quality analysis of TOEFL independent and integrated tasks across proficiency levels. Experts applied criteria similar to ETS draft scales, and empirical evidence confirmed that raters consider multiple performance features, with holistic scores driven by all assessment categories rather than grammar alone.

Abstract

This report documents two coordinated exploratory studies into the nature of oral English-for-academic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine field experts' rating orientations, and Study II investigated the quality of test-taker discourse on two different Test of English as a Foreign Language™ (TOEFL®) task types (independent and integrated) at different levels of proficiency. Study I showed that, with no guidance, domain experts distinguished and described qualitatively different performances using a common set of criteria very similar to those included in draft rating scales developed for the tasks at ETS. Study II provided empirical support for the criteria applied by the judges. The findings indicate that raters take a range of performance features into account within each conceptual category and that holistic ratings are driven by all of the assessment categories rather than, as has been suggested in earlier studies, predominantly by grammar.