A many-facet Rasch analysis of the second language group oral discussion task

Abstract

FACETS many-facet Rasch analysis software (Linacre, 1998a) was utilized to look at two consecutive administrations of a large-scale (more than 1000 examinees) second language oral assessment in the form of a peer group discussion task with Japanese English-major university students. Facets modeled in the analysis were examinee, prompt, rater, and five rating category ‘items.’ Unidimensionality was shown to be strong in both datasets, and approaches to interpreting fit values for the facets modeled in the analysis were discussed. Examinee ability was the most substantial facet, followed by rater severity, and item. The prompt facet was negligible in magnitude. Rater differences in terms of severity were generally large, but this characteristic was not stable over time for individuals; returning raters tended to move toward greater severity and consistency, while new raters showed much more inconsistency. Analysis of the scales showed general validity in gradations of scale steps, though raters had some difficulty discerning between categories at the ends of the scales for pronunciation and communicative skills.