Two Experiments on Automatic Scoring of Spoken Language Proficiency

Abstract

New scoring methods used in the SET-10 spoken English test subsume pronunciation scoring into a more extensive analysis that estimates several skills underlying spoken language performance. The SET-10 test is described and its performance is reviewed. Two experiments explore the relation of SET-10 scoring to other conventional ways of reporting spoken language performance. Experiment 1 concerns the Council of Europe’s Framework for describing second language proficiency. Samples of non-native English speech were: scored in SET-10, an automatic speaking test, and rated independently by three raters. Rater reliability in using the Council of Europe's scale and the comparability of the human and automatic measures are reported. Experiment 2 concerns the prediction of the intelligibility of non-native speech using SET-10 scoring and modifications to that scoring. A novel method for estimating intelligibility is described and preliminary results are reported. Both experiments support the notion that fully automatic methods of spoken language performance measurement can be used to predict more traditional assessments. Background Since the 1960’s, there have been several major (and many minor) efforts to establish curriculum-independent models of language learning and language proficiency (see North,