Overview of the 2017 Spoken CALL Shared Task

Abstract

We present an overview of the third edition of the Spoken CALL Shared Task. Groups competed on a prompt-response task using English-language data collected, through an online CALL game, from Swiss German teens in their second and third years of learning English. Each item consists of a written German prompt and an audio file containing a spoken response. The task is to accept linguistically correct responses and reject linguistically incorrect ones, with “linguistically correct” defined by a gold standard derived from human annotations. Scoring was performed using a metric based on the idea of maximising the ratios correct-accept-rate/false-accept-rate and correct-reject-rate/false-reject-rate. The third edition received sixteen entries, with the best score substantially improving on last year's edition of the task. We analyse factors which make it difficult to label items correctly, concluding that, as in the previous edition, good speech recognition is most important. Finally, we suggest a strategy for continuing the task.