1. Steps to Design a Better SurveyJean E. Fox Scott S. Fricker Office of Survey Methods Research Bureau of Labor Statistics October 19, 2012
2. Introduction Our backgrounds Usability Survey Methodology Goal of the presentation Combine what we know from our fields to improve usability surveys
3. Types of Usability Surveys Usability Tests Post-task Post-test (e.g., SUS) Ethnographic work Learn how people do their work Solicit input from users Administered Self-administered (online, paper) By interviewer (oral)
4. Introduction Three steps we‟ll discuss 1. Decide what you really need to know 2. Write the questions following best practices 3. Test the survey
5. Step 1Decide what you really need to know
6. Decide What You Really Need to Know Are you asking for data you really need? Will you really use it? Can you get the data somewhere else?
7. Decide What You Really Need to Know Are you asking questions respondents can answer? Can you include “screeners”? – Questions to allow respondents skip irrelevant questions Do you need separate surveys?
8. Decide What You Really Need to Know Are you asking for data in a format you can analyze? Open-ended vs multiple choice Are you really going to analyze it?
9. Step 2Write the questions following best practices
11. Types of Scales Likert-type item Semantic Differential
12. Types of Scales Bi-polar Previous examples Uni-polar
13. Rating Scales How many response options do you usually use in a rating scale? 3…5…7…10… or something else? Number of options Generally, scales with 5-7 options are the most reliable. The optimum size depends on the issue being rated (Alwin, 1997; Garner, 1960) – More options for bi-polar scales
14. Scales Do you usually have a neutral midpoint? Odd or Even number of options Without a midpoint, respondents tend to choose randomly between two middle options. For usability, generally include a mid-point.
15. Rating Scales Do you label the endpoints, a few options, or all of them? Labels Use text labels for each option Avoid numbers, unless they are meaningful – Especially avoid using negative numbers. Respondents do not like to select negative options.
16. Rating Scales Be sure the scale is balanced. This scale has 3 “satisfied” options, but only one “dissatisfied” option.
17. Ranking Definitions Rating: Select a value for individual items from a scale Ranking: Select an order for the items, comparing each against all the others.
18. Ranking Consider other options before using ranking Ranking is difficult and less enjoyable than other evaluation methods (Elig and Frieze, 1979). You don‟t get any interval level data
19. Ranking Recommendations Use ratings instead if you can. – Determine ranks from average ratings. Use rankings if you need respondents to prioritize options.
20. Question Wording
21. Double-Barreled Questions Avoid double-barreled questions They force respondents to make a single response to multiple questions They assume that respondents logically group the topics together, which may or may not be true Recommendations – Watch for the use of “and” in questions. – Eliminate all double-barreled questions. – Divide them into multiple questions.
22. Agree / Disagree Items Who uses agree / disagree items? Why? They are fairly easy to write You can cover lots of topics with one scale It‟s a fairly standard scale It‟s familiar to respondents
23. Agree / Disagree Items Unfortunately, they can be problematic They are prone to acquiescence bias – The tendency to agree with a statement They require an additional level of processing for the respondent – Respondents need to translate their response to the agree/disagree scale.
24. Agree / Disagree Items Recommendation Avoid agree / disagree items if possible Use “construct specific” responses
25. Other Issues Be sure the responses match the question. Speak the respondent‟s language Avoid jargon unless appropriate Remember that responses can be impacted by Question order The size of the text field Graphics, even seemingly innocuous ones
26. Broader Issue - Satisficing Responding to surveys often requires considerable effort Rather than finding the „optimal‟ answer, people may take shortcuts, choose the first minimally acceptable answer “Satisficing” (Krosnick, 1991) – depends on: Task difficulty, respondent ability and motivation
27. Satisficing – Remedies Minimize task difficulty Minimize number of words in questions Avoid double-barreled questions Decompose questions when needed – Instead of asking how much someone spent on clothing, ask about different types of clothing separately Use ratings not rankings Label response options
28. Satisficing – Remedies, cont. Maximize motivation Describe purpose and value of study Provide instructions to think carefully Include random probes (“why do you say that?”) Keep surveys short Put important questions early
29. Satisficing – Remedies, cont. Minimize “response effects” Avoid blocks of ratings on the same scale (prevents „straight-lining‟) Do not offer „no opinion‟ response options Avoid agree/disagree, yes/no, true/false questions
30. Step 3Test the survey
31. Testing Surveys Be sure your questions work Consider an expert review Need an expert For usability testing, be sure to include the survey in your pilot test. A common technique for evaluating surveys is Cognitive Interviewing (see Willis, 2005)
32. Cognitive Interviewing Cognitive interviewing basics Have participant complete the survey Afterwards, ask participants questions, such as – In your own words, what was the question asking? – What did you consider in determining your response? – Was there anything difficult about this question?
33. Cognitive Interviewing Cognitive interviewing basics (con‟t) Review the qualitative data you get to identify potential problems and solutions Like usability testing, there are different approaches (e.g., think aloud)
34. Summary Decide what you really need to know Write the questions following best practices Test the survey
35. Contact Information Jean E. Fox Scott S. FrickerFox.Jean@bls.gov Fricker.Scott@bls.gov 202-691-7370 202-691-7390
36. ReferencesAlwin, D.F. (1997). Feeling Thermometers Versus 7-Point Scales: Which Are Better? Sociological Methods and Research, 25(3), pp 318 – 340Elig, T. W., & Frieze, I.H. (1979). Measuring causal attributions for success and failure. Journal of Personality and Social Psychology, 37(4), 621- 634.Garner, W.R. (1960). Rating scales, discriminability, and information transmission. The Psychological Review, 67 (6), 343-352.Krosnick, J.A. (1991). Response strategies for coping with the cognitive demands of attitude strength in surveys. In J.M. Tanur (ed.) Questions About Questions: Inquiries into the Cognitive Bases of Surveys. New York: Russell Sage Foundation, pp. 177 – 203.Krosnick, J.A. and Presser, S. (2010). Question and questionnaire design. In Handbook of Survey Research, 2nd Edition, Peter V. Marsden and James D. Wright (Eds). Bingley, UK: Emerald Group Publishing Ltd.Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design, Thousand Oaks, CA: Sage Publications, Inc.