Ties in Paired-Comparison Experiments Using a Modified Thurstone-Mosteller Model

Abstract

Whlen making paired comparisons a judge frequently is unable to express any real preference in a number of the pairs he judges. Nevertheless, some of the methods in current use do not permit the judge to declare a tie. In other cases ties are permitted, but are ignored in performing the analysis. Alternatively, the ties are sometimes divided, equally or randomly, between the tied members of a pair. In assessing the merits of these various procedures it is helpful to distinguish between hypothesis testing and estimation. Several authors have shown that tied observations should be omitted in tests for the equality of treatment means. For the sign test, which may be regarded as a non-subjective form of paired comparisons, Hemelrijk [6] has proved that ignoring ties makes for a more powerful test of the null hypothesis than that obtained by dividing them equally among the positive and negative observations. Putter [9] has shown that random allocation of the tied observations reduces both the exact power and the asymptotic efficiency of the sign test. In a more general context Tocher [11] has essentially shown that ties are better ignored in tests involving discontinuous variates. A related question is whether ties should be permitted at all. This has been considered by Gridgeman [4] who has proposed a probabilistic model allowing for ties. He concludes that, in discrimination tests, power is theoretically increased by permitting the judge to declare a tie when unable to express a real preference, if the recorded ties are left out of consideration in analyzing the data. In practice, however, the increase in power may be offset by a decrease in the subject's efficiency of decision. In such circumstances Gridgeman recommends the prohibition of ties.