The Reliability of Survey Attitude Measurement

TLDR

The study examines theoretical hypotheses linking question and respondent characteristics to the reliability of survey attitude measurement and discusses the findings within a general framework for survey error sources. To test these hypotheses, reliability was estimated for 96 survey attitude measures using data from five, 3‑wave national reinterview surveys—the Michigan Election Panel Surveys and two General Social Survey reinterview studies. Reliability varied with question attributes: more response options and extensive verbal labeling increased reliability, while a “don’t know” option did not; topic differences confounded these effects, and respondent factors such as older age and lower schooling were associated with lower reliability.

Abstract

Several theoretical hypotheses are developed concerning the relation of question and respondent characteristics to the reliability of survey attitude measurement. To test these hypotheses, reliability is estimated for 96 survey attitude measures using data from five, 3-wave national reinterview surveys-three Michigan Election Panel Surveys and two reinterview studies conducted by the General Social Survey. As hypothesized, a number of attributes of questions are linked to estimated reliability. Attitude questions with more response options tended to have higher reliabilities, although there are some important exceptions. More extensive verbal labeling of numbered response options was found to be associated with higher reliability, but questions explicitly offering a “don't know” alternative were not found to be more reliable. Question characteristics were confounded to an unknown degree with topic differences of questions, which were significantly linked to reliability, leaving the influence of question characteristics on reliability somewhat ambiguous. Characteristics of respondents were also found to be related to levels of reliability. Older respondents and those with less schooling provided the least reliable attitude reports. These results are discussed within a general framework for the consideration of survey errors and their sources.