Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Usability Testing Online Surveys (and forms)

93 views

Published on

Learn about Tourangeau's Four Stage Questionnaire Response model and about cognitive interviewing of surveys.

Published in: Design
  • Be the first to comment

  • Be the first to like this

Usability Testing Online Surveys (and forms)

  1. 1. @uxordie Text fields and dropdowns and radios oh my! USABILITY TESTING ONLINE SURVEYS (AND FORMS) Eva Kaniasty UXPA International Conference @uxordie
  2. 2. @uxordie@uxordie 1. Why surveys? 
 Isn’t user research enough? 2. Why pre-test surveys?
 Let me count the ways. 3. Four stage questionnaire model
 The Cognitive Psych (Tourangeau) 4. The cognitive interview
 Usability testing… just for surveys. (Willis) 5. Survey Pre-Testing Process
 3 scenarios. AGENDA? AGENDA. 2
  3. 3. @uxordie WHY SURVEYS?
  4. 4. @uxordie@uxordie • Validate qualitative research results • Get feedback from a larger sample • Build a participant database • Bridge gap between quant and qual • Build a strong case for change • Kill features WHY SURVEYS? 4 “Doesn’t everybody love a survey?” • Execs • Business People • Marketing People • Sales People
  5. 5. @uxordie@uxordie WHY SURVEYS? 5 • Surveys are easier to do • Surveys are faster to do • Surveys are less biased • Surveys are more objective • Survey data is more valid • Survey data is statistically significant Why are surveys better than user research? THE TRUTH ABOUT SURVEYS? • Survey methodology is well-researched • Best practices are well- established THE TRUTH ABOUT SURVEYS
  6. 6. @uxordie@uxordie MIXED METHODS RESEARCH 6 • Full picture of user behavior • Mitigate methodological weaknesses • Understand the what and the why • Make better decisions aka Triangulation
  7. 7. @uxordie@uxordie • Complex onboarding • Complex customization • Data-entry driven applications • Multi-step data collection workflows • Skills testing WHAT ABOUT FORMS? 7 What kinds of forms benefit from testing? • B2B eCommerce • Healthcare • Education • Government
  8. 8. @uxordie WHY TEST SURVEYS?
  9. 9. @uxordie@uxordie WHY TEST SURVEYS? 9 ERROR TYPES • Sampling •Measurement • Coverage • Non-Response “There are no stupid questions… or are there?”
  10. 10. @uxordie@uxordie WHY TEST SURVEYS? 10 “Stop doing survey research.” 
 - Jared Spool https://articles.uie.com/on_surveys DO SURVEY SUCK? • Bias is incredibly easy to introduce. • Bad questions can yield misleading answers. • You can’t adjust a survey mid-stream. • Surveys are easy to manipulate. • Surveys are not a substitute for qualitative research. • Surveys are not a shortcut. • Surveys lack context. • Surveys are subjective. “How many researchers does it take to write a bad survey question?” “Surveys are the most dangerous research tool—misunderstood and misused. They frequently straddle the qualita>ve and quan>ta>ve, and at their worst represent the worst of both..” 
 - Erika Hall”
  11. 11. @uxordie FOUR STAGE QUESTIONNAIRE MODEL
  12. 12. @uxordie@uxordie 1. Comprehension 
 Do you understand the question? 2. Retrieval
 Do you know or remember the answer? 3. Judgement
 Can you formulate a truthful, accurate response? 4. Response
 Can you map your answer to the response options provided? FOUR STAGE QUESTIONNAIRE MODEL 12 Roger Tourangeau
  13. 13. @uxordie@uxordie “How many times have you participated in an online discussion about photography in the last year?” 0 | Once or twice | 3-5 times | 6-10 times | more than 10 times FOUR STAGES 13 COGNITIVE PROCESS “Well, I’ve commented on things on Facebook, but I don’t think that’s what they mean. I also took an online class, which definitely included discussion, so that counts.” (Comprehension) “It was still cold when I took the class, so that’s within the last year.” (Retrieval) “There were 12 classes, but I know I missed a couple of them.” (Judgement) “Overall, I was pretty active, so let’s just say that it’s more than 10.” (Response)
  14. 14. @uxordie EXTREME(!) EXAMPLE
  15. 15. @uxordie@uxordie 1. COMPREHENSION 15 “How many times have you seen a provider in the last year?” What’s a provider? Does it mean a doctor? If so, does this include chiropractors …therapists… pharmacists? Is this calendar year or last 12 months?
  16. 16. @uxordie@uxordie 2. RETRIEVAL 16 “How many times have you seen a primary care doctor in the last 12 months?” I had a checkup a while ago, but it may be more than a year at this point.
  17. 17. @uxordie@uxordie 3.JUDGEMENT 17 “How many times have you seen a primary care doctor in the last 6 months?” I guess it’s 0. I probably should be taking better care of my health though. 
 I did talk to her on the phone about a refill, so I’ll just say 1.
  18. 18. @uxordie@uxordie 4. RESPONSE 18 “How many times have you spoken to your primary care doctor in the last 6 months?” Through email On the phone At the office Wait a second… my doctor has this new telemedicine thingy, so I did have a live chat as well. Should I list that as part of email? 1 0 0
  19. 19. @uxordie@uxordie Are we done with this question yet? 19 “How many times have you spoken to your primary care doctor in the last 6 months?” Through email On the phone At the office Wait researcher, you also forgot to randomize the answer options, but you won’t notice this until you watch a few people take the survey. 1 0 0 Other 0 Enter all that apply: time(s) time(s) time(s) time(s)
  20. 20. @uxordie@uxordie COGNITIVE BURDEN 20 Are 30 clear questions better than 20 difficult ones? Perhaps Yes Maybe Don’t know
  21. 21. @uxordie TESTING SURVEYS
  22. 22. @uxordie@uxordie “The administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends.” COGNITIVE INTERVIEWING 22 Beatty & Willis (2007) What is cognitive interviewing?
  23. 23. @uxordie@uxordie Think-Aloud + Cognitive probes = Cognitive Interview USABILITY TESTING SURVEYS 23 How is cognitive interviewing different from usability testing?
  24. 24. @uxordie@uxordie 1. Comprehension/Interpretation 
 “What does the term ‘leaderboard’ mean to you?” 2. Paraphrasing
 “Tell me in your own words what this is asking you.” 3. Recall
 “How do you know that you used it 10 times in the past 3 months?” 4. Confidence Judgement
 “How sure are you that you used it at least 10 times?” 5. Specific
 “What information do you review when you use the leaderboard?” 6. General
 “I noticed you changed your answer. Tell me what you were thinking.” COGNITIVE PROBES 24 Gordon B. Willis “How frequently do you use the leaderboard feature?”
  25. 25. @uxordie COGNITIVE PROBES
  26. 26. @uxordie@uxordie Q. “When was the last time you visited your mother to help her with her computer?” COGNITIVE INTERVIEW 26 A. “I live with my mother actually, so I wouldn’t be visiting her.” P. “OK, so do you remember the last time you helped her with her computer at home?” (General) A. “Yes… it was about a month ago.” P. “How sure are you it was a month ago? (Confidence Judgement) A. “I’m pretty sure, because it was the weekend of the big snow storm at the end of March.” P. “Does anyone else help your mom with her computer?” (Specific) A. “Not really. I don’t have any siblings, and my dad is just as bad at technology as she is.”
  27. 27. @uxordie@uxordie Q. “How concerned are you about lead contamination in your community garden?” COGNITIVE INTERVIEW 27 A. “… I don’t really know. This is the first time I’ve thought about it honestly. Now that you brought it up though, I feel like I should be very concerned.” P. “Are you familiar with the term ‘lead contamination’?” (Comprehension) A. “I am… I know that lead paint was used in old buildings, so sometimes it can get in the soil, but I’m not sure why this would affect a community garden if it’s not near a building.” P. “You said that ‘you should be very concerned’… How sure are you that you’re very concerned?” (Confidence Judgement) A. I guess I just assumed that this wouldn’t be a problem in my community garden. So maybe I’m not that concerned. Honestly, I don’t know.
  28. 28. @uxordie@uxordie Q. “Did you go to the beach last weekend?” COGNITIVE INTERVIEW 28 A. “I did actually. I went snorkeling.” P. “Tell me about it.” (General) A. “I was at a conference in Puerto Rico, and I got there a few days before.” P. “Was this the weekend that just passed, or the weekend before?” (Interpretation) A. “It was Saturday, so two days ago.” P. “Do you remember the last time you went to beach back home?” (Recall) A. …
  29. 29. @uxordie@uxordie Q. “How often do you use our website to book a dog sitter?” COGNITIVE INTERVIEW 29 A. “OK… In the past, I’ve traveled a few times per year, but if it’s just a weekend my friend would watch him, so maybe… 6 times per year… so that’s. ‘once every 2-3 months’… P. “I noticed you hesitated before choosing your answer….” (General) A. “Yeah… I was trying to figure out what it would be on average.” P. “So… how sure are you that it’s ‘once every 2-3 months’?” (Confidence Judgement) A. “Well, to be honest, my dog died a few months ago… so I haven’t used it since then, 
 obviously.” Once a month or more / Once every 2-3 months / Every six months / Once a year / Less than once a year
  30. 30. @uxordie SURVEY PRE-TESTING
  31. 31. @uxordie@uxordie • Exploratory interview incorporating cognitive probes. MODERATED INTERVIEW 31 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement
 • Asking the right questions • Understanding respondent background knowledge • Exposing assumptions • Setting the right response intervals PRO TIP • Doubles as a round of qualitative research.
  32. 32. @uxordie@uxordie • Observation and cognitive interview of draft survey. MODERATED DRY RUN 32 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement | Response
 • Terminology and interpretation issues • Fixing confusing response categories • Layout issues PRO TIP • Probe on response categories as well as the questions.
  33. 33. @uxordie@uxordie • Self-administration of survey to hand-picked, sample before official launch. PILOT 33 HOW USE IT TO EXPLORE Comprehension | Response • Some terminology issues • Missing response categories • Layout issues PRO TIP • Include open-ended responses for reporting problems in each survey section.
  34. 34. @uxordie@uxordie ANY QUESTIONS? 34

×