Testing the Reliability of Weight Elicitation Methods: Direct Rating versus Point Allocation

Abstract

Two commonly used methods of assigning numerical judgments (i.e., importance weights) to attributes in order to signify their relative importance are point allocation (PA) and direct rating (DR). These methods may seem to be minor variants of each other, yet they produce very different profiles of attribute weights when rank ordered from most to least important. The weights elicited by DR were more reliable than those elicited by PA in a test–retest situation. An important practical implication of this is for multicriteria decision making. Using people's test–retest data as attribute weights on simulated alternative values revealed that the same alternative would be chosen on 88% of occasions with DR, but only 74% of occasions with PA. Moreover, subjects reacted more favorably to DR than to PA.