Measuring Subgroup Preferences in Conjoint Experiments

TLDR

Conjoint analysis, widely used in political science, estimates average marginal component effects (AMCEs) to assess how profile features influence support, yet these effects are frequently compared across subgroups. The authors aim to demonstrate that conditional AMCEs can mislead subgroup inference and to propose improved reporting using marginal means and omnibus tests. They illustrate the issue with published examples and recommend using marginal means and an omnibus F‑test for clearer subgroup analysis. They find that such misuse yields subgroup preference inferences with arbitrary sign, magnitude, and statistical significance.

Abstract

Conjoint analysis is a common tool for studying political preferences. The method disentangles patterns in respondents’ favorability toward complex, multidimensional objects, such as candidates or policies. Most conjoints rely upon a fully randomized design to generate average marginal component effects (AMCEs). They measure the degree to which a given value of a conjoint profile feature increases, or decreases, respondents’ support for the overall profile relative to a baseline, averaging across all respondents and other features. While the AMCE has a clear causal interpretation (about the effect of features), most published conjoint analyses also use AMCEs to describe levels of favorability. This often means comparing AMCEs among respondent subgroups. We show that using conditional AMCEs to describe the degree of subgroup agreement can be misleading as regression interactions are sensitive to the reference category used in the analysis. This leads to inferences about subgroup differences in preferences that have arbitrary sign, size, and significance. We demonstrate the problem using examples drawn from published articles and provide suggestions for improved reporting and interpretation using marginal means and an omnibus F-test. Given the accelerating use of these designs in political science, we offer advice for best practice in analysis and presentation of results.