Measuring and Bounding Experimenter Demand

TLDR

The study proposes a technique for assessing the robustness of experimental and survey findings to demand effects. The method deliberately induces demand to bound its influence, models participants’ beliefs about the researcher’s objectives, and uses demand treatments to estimate bounds and compute demand‑robust treatment effects, applied to 11 classic tasks. The bounds average 0.13 standard deviations, suggesting that typical demand effects are probably modest. JEL codes: C83, C90, D83, D91.

Abstract

We propose a technique for assessing robustness to demand effects of findings from experiments and surveys. The core idea is that by deliberately inducing demand in a structured way we can bound its influence. We present a model in which participants respond to their beliefs about the researcher’s objectives. Bounds are obtained by manipulating those beliefs with “demand treatments.” We apply the method to 11 classic tasks, and estimate bounds averaging 0.13 standard deviations, suggesting that typical demand effects are probably modest. We also show how to compute demand-robust treatment effects and how to structurally estimate the model. (JEL C83, C90, D83, D91)