Ranking or rating? Some data and their implications for the measurement of evaluative response

Abstract

Two groups of 80 subjects rated 15 paintings on a five‐point pleasingness scale in accordance with a ‘relative’ instruction, requiring them to compare the paintings with one another, or an ‘absolute’ instruction, requiring them to compare the paintings with other, familiar ‘anchor’ paintings. Each subject also ranked the paintings on pleasingness, either before or after doing the ratings. Within‐subject correlations between ranks and ratings revealed that, for all but a small minority of subjects, there was a moderate to high level of correspondence between the two measures. There was a tendency for those subjects who had shown good correspondence between their ranks and their ratings to show a greater spread in their ratings. The relative instruction also produced a greater spread in subjects' ratings compared with the ratings obtained under the absolute instruction. The means of the ranks and the ratings that the subjects assigned to the individual paintings were highly correlated in both groups. The results are discussed in terms of the respective merits of ranking and rating procedures, and it is concluded that, where practical and statistical considerations permit, and where the items are highly discriminable, a ranking procedure has advantages. Where ranking is precluded, however, a rating procedure with ‘relative’ instructions offers some of the same desirable characteristics.