Impact of Rater Characteristics and Prosodic Features of Speaker Accentedness on Ratings of International Teaching Assistants' Oral Performance

Abstract

Abstract Few prior studies have examined degree of fidelity between raters' assessments of oral performances and objectively observable prosodic indices of accentedness. Prosodic indices of accentedness quantify trait-relevant variance, whereas rater background variables represent trait-irrelevant variance. The present study, therefore, investigated the extent to which raters' background characteristics and instrumentally measured prosodic indices of speakers' accentedness jointly influenced the rating of oral performances. Seventy U.S. undergraduate students rated the speaking and teaching proficiency of 11 international teaching assistants (ITAs). Using the PRAAT computer program, 5 min of continuous speech from each of the ITAs were instrumentally analyzed for a number of indices of speech rate, pausing, stress, and intonation. Dependent variables were undergraduates' ratings of ITA oral proficiency and instuctional competence. Rater background variables such as the listener's native speaker status and experience as a language tutor explained 7–9% of the variance in oral performance ratings, whereas 18–19% was attributable to the prosody variables. These findings suggest that U.S. undergraduates are sensitive to trait-relevant indicators of ITA oral proficiency. At the same time, their speech evaluations are subject to substantial bias based on their own backgrounds. Notes 1Space limitations prohibit printing of the dendrogram. A copy is available upon request from the author. 2The author expresses appreciation to Ramon Littell and Judith Singer for their advice about model parameters. Any errors of execution or interpretation are solely the author's. 3The R 2 statistic in MRCM cannot be interpreted in quite the same way as in a traditional regression. That is, it does not translate directly into percentage of variance in the dependent variable that is explained by the independent variables (CitationSinger, 1998; CitationSnijders & Bosker, 1994). However, one way of gauging how much the variance in outcome variables is explained by predictors is to compute how much the covariance component has diminished between models. The variance computation presented here is based on CitationBryk and Raudenbush (1992, p. 65) and CitationSinger (1998, p. 332).