Reliability of Nominal Data Based on Qualitative Judgments

TLDR

Most research on marketing measurement reliability focuses on multi‑item quantitative scales, while nominal scale data derived from qualitative judgments—used in consumer response, cognitive response, meta‑analysis, and content analysis—has received little attention. The authors aim to evaluate and improve the quality of judgment‑based nominal scale data, especially when multiple judges are used. They review existing approaches from other disciplines and introduce a new reliability index tailored to inter‑judge data, illustrated with a cognitive response experiment that compares it to common measures. They conclude with recommendations for better study design when using judgment‑coded data.

Abstract

Most research related to the reliability and validity of marketing measures has focused on multi-item quantitative scales. In contrast, little attention has been given to the quality of nominal scale data developed from qualitative judgments. Judgment-based (“coded”) nominal scale data are important and frequently used in marketing research-for example, in analysis of consumer responses to open-ended survey questions, in cognitive response research, in meta-analysis, and in content analysis. The authors address opportunities and challenges involved in evaluating and improving the quality of judgment-based nominal scale data, with specific emphasis on the use of multiple judges. They review approaches commonly used in other disciplines, then develop a new index of reliability that is more appropriate for the type of interjudge data typically found in marketing studies. Data from a cognitive response experiment are used to illustrate the new index and compare it with other common measures. The authors conclude with suggestions on how to improve the design of studies that rely on judgment-coded data.