American Journal of Political Science · 2013 · 972 citations · 24 references
Social SciencesBehavioral SciencesPsychologyExperimental DesignExperimental PsychologyPsychometricsAttention AssessmentBehavioral Data CollectionBiasData AnalysisSampling TechniquesSurvey ResearchExperimental MethodologyMultiple-choice ItemsScreenersSubjectsSurvey QuestionsSurvey Research DesignSurvey Response BiasSurvey-based Measurement
Survey research depends on attentive respondents, yet many participants fail to pay attention. The article proposes using Screeners to mitigate inattentiveness in self‑administered surveys. The authors demonstrate that Screeners can detect inattentive respondents and reduce noise, then explore understudied aspects of their implementation. Multiple Screener items improve data quality, whereas a single Screener is insufficient; however, Screener performance correlates with political characteristics, so studies should report results conditional on attention levels to balance internal and external validity.
Good survey and experimental research requires subjects to pay attention to questions and treatments, but many subjects do not. In this article, we discuss “Screeners” as a potential solution to this problem. We first demonstrate Screeners’ power to reveal inattentive respondents and reduce noise. We then examine important but understudied questions about Screeners. We show that using a single Screener is not the most effective way to improve data quality. Instead, we recommend using multiple items to measure attention. We also show that Screener passage correlates with politically relevant characteristics, which limits the generalizability of studies that exclude failers. We conclude that attention is best measured using multiple Screener questions and that studies using Screeners can balance the goals of internal and external validity by presenting results conditional on different levels of attention.
24
The Framing of Decisions and the Psychology of Choice
Amos Tversky, Daniel Kahneman · Science · 1981 · 17.3K citations
Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn · Psychological Science · 2011 · 6.7K citations
Identification and Estimation of Local Average Treatment Effects
Guido W. Imbens, Joshua D. Angrist · Econometrica · 1994 · 4.1K citations