Understanding Evidence-Based Research Methods: Pilot Testing Surveys

Abstract

This column focuses on pretesting surveys, which represent an important series of steps in the survey process that must be undertaken prior to administering an actual (final) survey to participants. The authors welcome comments from all readers who have suggestions about the way information is presented or questions about the content of this column. It is not comprehensive and does not replace information found in textbooks or peer-reviewed articles. Beyond their experience, the authors have used textbooks and articles as sources and recommend that you also refer to them for more detailed explanations of what is discussed here.Scenario: When Allan and Gwen last talked, Gwen mentioned the need to pretest the survey items. Since then, she has e-mailed the survey items to Allan and is now calling to discuss the steps involved in pretesting. Their conversation follows.Gwen: Hello, Allan. I hope you received the survey items I sent you and have had a chance to review them-have you?Allan: Yes, I have. They look great and I'm excited to get this project moving forward so we can start collecting data with this survey. Can we do that now?Gwen: Not quite yet. First, as I mentioned last time, we need to pretest the survey items. Let's discuss pretesting and develop a plan to carry it out. Then, in a few weeks, we can discuss administering the actual survey to participants.Allan: Ok, but is pretesting really necessary? After all, you and I are satisfied, and I've never heard of others doing this.Gwen: Because everything rests on the quality of the data we collect, this is a very important step in our study. Given its importance, researchers are using this approach more frequently and more is being published about different ways to accomplish this step (Willis, 2005). Although the specifics of pretesting vary among researchers, the primary goal is the same: to make sure that the survey items can measure the domains of interest reliably and validly when used in real situations.For our purposes, I am using Dillman, Smyth, and Christian's (2009) four-stage pretesting approach: (1) obtaining expert feedback about the initial version of the survey; (2) conducting cognitive interviews; (3) collecting preliminary data on the survey; and (4) reviewing the final survey.By sending you the initial version of the survey and asking you to review it, I have taken one step in the direction of obtaining expert feedback. To minimize bias from your responses, I will obtain input from other experts. For our purposes, an expert can be a professional with detailed knowledge about the domain we want to measure with the survey. So other experts might be your hospital colleagues or employees, especially those with experience in facility design. Similarly, I can also ask some of my colleagues for input to review the survey.Allan: What exactly happens when a survey is reviewed? What type of feedback do you expect to receive from me and other experts?Gwen: One approach is to send the survey to all of the experts and ask them to read the questions to see whether the questions appear to be measuring what we are trying to measure. In addition to reading the questions for content, we can ask the experts to look for anything that is confusing or does not make sense. For example, sometimes the response scale (1 = Strongly Disagree to 5 = Strongly Agree) may not correspond very well with the survey item, or the survey item is too complicated. After receiving all of the expert responses back, changes to the survey items are made and a revised version of the survey can be resent to the experts. This iterative process can stop once the changes suggested by the experts are not substantive or are unlikely to result in significant changes to the survey items.Allan: Ok, that makes sense, but it sounds like it could take a long time. Remember, this project is on a tight time line. How is the approach you just described different from Dillman et al. …