Recommended standard final outcome categories and standard definitions of response rate for social surveys

Abstract

Response rates are one of the most important indicators of survey quality and one of the indicators most likely to be reported. In order to be able to make valid comparisons between response rates obtained on different surveys and by different organisations, response rates must be defined and calculated in a standard way. At present in the UK there are no standards in this area. In consequence, practice varies considerably between surveys and between organisations. The authors of this paper became aware of minor differences between Social Survey Division, ONS and the National Centre for Social Research in definitions and calculations of response on major government surveys that they carry out and between surveys within each organisation. We were also aware that international comparisons, for example of response on Labour Force Surveys and Time Use Surveys in different countries, are affected by the fact that each country has a different definition of what counts as response and often differ in how response rates are calculated. Progress has been made in the USA by the American Association of Public Opinion Research (AAPOR) to draw up standard definitions applicable to random digit dial telephone surveys, face-toface surveys where one person is selected per household and mail surveys of named persons. The AAPOR standards (AAPOR, 2000) provide models which might be adapted and extended for UK purposes. This paper proposes standards which are applicable to the UK, in particular to major government, academic and public sector surveys. We hope that adoption of these standards will enable meaningful comparisons to be made between surveys and will aid understanding of trends and patterns in response rates. We hope that eventually commissioners of surveys will specify and assess response rates on the basis of these standards, thus improving the utility, validity and fairness of comparisons and judgements. This is particularly important when surveys are commissioned via competitive tendering. This paper deals with face-to-face interviewer surveys of households and of individuals. We aim to gain acceptance of standards for these surveys and to gain experience of implementing the standards before extending the recommendations to encompass other types of surveys. In due course we hope to make recommendations with respect to panel surveys, mail surveys, telephone surveys and surveys of establishments. In the meanwhile, we hope that this paper will serve as a useful reference document for anyone planning a survey, commissioning a survey or producing a survey technical report or quality profile.