An examination of rater orientations on a paired candidate discussion task through stimulated verbal recall

Abstract

This paper reports on the findings from an exploratory study in which rater orientations were examined through the use of stimulated verbal recall. Two trained and experienced raters of paired candidate discussion tasks produced retrospective verbal reports on six paired candidate discussion performances, producing a total of twelve retrospective verbal reports. Through an analysis of these verbal reports, it was possible to explore the raters' adherence to criterion and non-criterion features of the performances. It was found that trained and experienced raters attended to many non-criterion features of the paired candidate discussions, and as reflected in the focus of their comments, the raters differed in the extent to which they viewed the performance as co-constructed. These findings have implications for both the development of rating scales and the training of raters for paired candidate discussion tasks.