Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS

Speaker: Caroline Jarrett

To help us get the best out of this tricky research method, Caroline will describe the Survey Octopus, a friendly creature that helps her to tackle all the issues that may lie between 'What we want to ask, and who we want to ask', and a solid, reliable number that can be used to make decisions.

Along the way, we'll encounter the key concept in survey methodology, Total Survey Error, and the various types of error that can affect your survey.

  • Be the first to comment

  • Be the first to like this

THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS

  1. 1. Getting valid results from surveys Caroline Jarrett @cjforms #UXNZ2015
  2. 2. 2 What would you do for a dollar?
  3. 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 3 $1 in the envelope beats $10 guaranteed later 3Caroline Jarrett @cjforms (CC) BY SA-4.0
  4. 4. 4
  5. 5. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5 Response
  6. 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6 I’m a forms specialist 6Image credit: Flickr, taxrebate.org.uk
  7. 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7 Why do people answer questions? 7Caroline Jarrett @cjforms (CC) BY SA-4.0
  8. 8. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 8
  9. 9. Caroline Jarrett @cjforms (CC) BY SA-4.0 9 “Please have a look at this survey?” 9 Kill survey! Kill! Kill!
  10. 10. Caroline Jarrett @cjforms (CC) BY SA-4.0 10 “Please have a look at this survey?” 10 What number do you need to make your decision?
  11. 11. 11 The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
  12. 12. Caroline Jarrett @cjforms (CC) BY SA-4.0 12 The aim of a survey is to get a number that helps you to make a decision 12
  13. 13. Caroline Jarrett @cjforms (CC) BY SA-4.0 13
  14. 14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14 “Yes, to make <decision> we need <number>” 14 Great! Let’s work through a few key topics
  15. 15. 15 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  16. 16. 16
  17. 17. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 17
  18. 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 18 To work that out, let’s visit the Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0 18
  19. 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19 Start with how many will answer 19 Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork: Who answers?
  20. 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20 Whether they’ll answer depends on effort 20 Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0
  21. 21. Caroline Jarrett @cjforms (CC) BY SA-4.0 21 And on the reward you’re offering Goals and resources: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 21 Caroline Jarrett @cjforms (CC) BY SA-4.0
  22. 22. Then there’s the ‘Justin Bieber North Korea’ problem 22http://www.bbc.com/news/10506482
  23. 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23 If we ask ‘anyone’, we’ll have extra work here Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Whose answers can we use?
  24. 24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24 So it matters where we get our sample 24 Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the list you sample from
  25. 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 2525 And now it’s easy to work out how many to ask Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0
  26. 26. Caroline Jarrett @cjforms (CC) BY SA-4.0 26 We thought about a lot of topics to work that out Goals Sample Questions Fieldwork 26 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response
  27. 27. 27Image credit: https://www.flickr.com/photos/deathtogutenberg/5708175370
  28. 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28 What about this bit? Caroline Jarrett @cjforms (CC) BY SA-4.0
  29. 29. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 29
  30. 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30 A good question gets good answers Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Is the question easy to answer?
  31. 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 31 Good answers help you to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0 Insight: Is the answer useful?
  32. 32. 32 In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0
  33. 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 33 All the topics are connected Goals Sample Questions Fieldwork 33 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
  34. 34. 34 Let’s think a bit more about response
  35. 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35 People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” Response depends on effort, reward, and trust 35
  36. 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36 A good question works in three ways 36 Appropriate Obvious Interesting
  37. 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 3737 Why did you visit our website today? Appropriate Obvious Interesting
  38. 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38 Would you recommend us to a friend or family member? In a shop, buying a baby carriage In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 38
  39. 39. Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 39
  40. 40. 40 Let’s think a bit more about fieldwork
  41. 41. 41Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/
  42. 42. 1950s mindset: “Ask Everything” 42 Survey = Big Honkin’ Survey
  43. 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43 Big Honkin’ Survey = think hard about everything Goals Sample Questions Fieldwork 43 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
  44. 44. 2015 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 44
  45. 45. 45http://www.disambiguity.com/death-to-it-depends/
  46. 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 46 Light Touch Survey = easier choices + iteration Goals Sample Questions Fieldwork 46 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
  47. 47. From goals to insight - quickly • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 47
  48. 48. Time for new question 48 One way to iterate, improve, increase Caroline Jarrett @cjforms (CC) BY SA-4.0
  49. 49. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 49
  50. 50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50 Likert had several different types of question in his response formats Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 50
  51. 51. You can find an academic paper to support almost any number of points Krosnick and Presser refer to over 80 papers 51 Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf
  52. 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52 Respondents focus on the actual question Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 52
  53. 53. Tip Don’t stress too much about the number of points in your rating scale 53Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
  54. 54. Caroline Jarrett @cjforms (CC) BY SA-4.0 54 Well, OK, stress a little bit. 54 This scale is downright peculiar. Avoid.
  55. 55. 55 What about the ‘valid results’ bit?
  56. 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 5656 The aim is to get the best number you can, within the resources you have
  57. 57. Caroline Jarrett @cjforms (CC) BY SA-4.0 57 What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 57 The aim is to get the best number you can, within the resources you have
  58. 58. Caroline Jarrett @cjforms (CC) BY SA-4.0 58 The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number
  59. 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 5959 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have Resources What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use
  60. 60. Caroline Jarrett @cjforms (CC) BY SA-4.0 6060 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct By making good choices, you minimise the error Resources
  61. 61. 61 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  62. 62. What number do you need to make the decision? Use your UX skills (especially technical communication) Go for Light Touch surveys Use the Survey Octopus to make good choices Caroline Jarrett @cjforms (CC) BY SA-4.0 62
  63. 63. Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 63

×