A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors

Abstract

The study reported here examined the evaluation criteria used to assess the proficiency and effectiveness of the language produced in an oral performance test of English conducted in an American university context. Empirical methods were used to analyze qualitatively and quantitatively transcriptions of the Oral English Tests (OET) of 44 prospective Graduate Student Instructors (GSI). The language required to complete the tasks on the test was conceptualized from the functional perspective of transactional and interactional language use as defined by Brown and Yule (1989). Listening comprehension and pronunciation were also analyzed and scored. Stepwise logistic regression was used to determine the extent to which these linguistic features contributed to final ratings. These quantitative findings were then compared to ‘real-time’ written comments made by evaluators during the tests. Intuitive methods were then used to further explore those features of candidate performance attended to by evaluators: interviews were conducted with experienced evaluators to determine the features they judged necessary for communicating effectively in instructional settings. Results indicate that the three data sources converge on two main features — pronunciation and listening comprehension — that are important in describing and evaluating the proficiency of prospective GSIs.