Download to read offline
Uploaded byUX New Zealand 2015
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
Caroline Jarrett discusses the importance of trust, perceived effort, and rewards in obtaining valid results from surveys. She emphasizes that a good survey aims to obtain numbers for informed decision-making, requires careful sampling, and can benefit from a light touch approach to simplify the process. Effective question design and understanding the reasons behind survey participation are crucial for maximizing response rates and obtaining useful data.
More Related Content
Similar to THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
- 1. Getting valid results fromsurveys Caroline Jarrett @cjforms #UXNZ2015
- 2. 2 What would youdo for a dollar?
- 3. Caroline Jarrett @cjforms(CC) BY SA-4.0 3 $1 in the envelope beats $10 guaranteed later 3Caroline Jarrett @cjforms (CC) BY SA-4.0
- 4.
- 5. People will onlyrespond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5 Response
- 6. Caroline Jarrett @cjforms(CC) BY SA-4.0 6 I’m a forms specialist 6Image credit: Flickr, taxrebate.org.uk
- 7. Caroline Jarrett @cjforms(CC) BY SA-4.0 7 Why do people answer questions? 7Caroline Jarrett @cjforms (CC) BY SA-4.0
- 8. People ask meabout surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 8
- 9. Caroline Jarrett @cjforms(CC) BY SA-4.0 9 “Please have a look at this survey?” 9 Kill survey! Kill! Kill!
- 10. Caroline Jarrett @cjforms(CC) BY SA-4.0 10 “Please have a look at this survey?” 10 What number do you need to make your decision?
- 11. 11 The survey isa systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
- 12. Caroline Jarrett @cjforms(CC) BY SA-4.0 12 The aim of a survey is to get a number that helps you to make a decision 12
- 13. Caroline Jarrett @cjforms(CC) BY SA-4.0 13
- 14. Caroline Jarrett @cjforms(CC) BY SA-4.0 14 “Yes, to make <decision> we need <number>” 14 Great! Let’s work through a few key topics
- 15. 15 Total Survey Errordiagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
- 16.
- 17. People ask meabout surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 17
- 18. Caroline Jarrett @cjforms(CC) BY SA-4.0 18 To work that out, let’s visit the Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0 18
- 19. Caroline Jarrett @cjforms(CC) BY SA-4.0 19 Start with how many will answer 19 Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork: Who answers?
- 20. Caroline Jarrett @cjforms(CC) BY SA-4.0 20 Whether they’ll answer depends on effort 20 Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0
- 21. Caroline Jarrett @cjforms(CC) BY SA-4.0 21 And on the reward you’re offering Goals and resources: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 21 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 22. Then there’s the‘Justin Bieber North Korea’ problem 22http://www.bbc.com/news/10506482
- 23. Caroline Jarrett @cjforms(CC) BY SA-4.0 23 If we ask ‘anyone’, we’ll have extra work here Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Whose answers can we use?
- 24. Caroline Jarrett @cjforms(CC) BY SA-4.0 24 So it matters where we get our sample 24 Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the list you sample from
- 25. Caroline Jarrett @cjforms(CC) BY SA-4.0 2525 And now it’s easy to work out how many to ask Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0
- 26. Caroline Jarrett @cjforms(CC) BY SA-4.0 26 We thought about a lot of topics to work that out Goals Sample Questions Fieldwork 26 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response
- 27.
- 28. Caroline Jarrett @cjforms(CC) BY SA-4.0 28 What about this bit? Caroline Jarrett @cjforms (CC) BY SA-4.0
- 29. People ask meabout surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 29
- 30. Caroline Jarrett @cjforms(CC) BY SA-4.0 30 A good question gets good answers Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Is the question easy to answer?
- 31. Caroline Jarrett @cjforms(CC) BY SA-4.0 31 Good answers help you to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0 Insight: Is the answer useful?
- 32. 32 In your lastfive days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0
- 33. Caroline Jarrett @cjforms(CC) BY SA-4.0 33 All the topics are connected Goals Sample Questions Fieldwork 33 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
- 34. 34 Let’s think abit more about response
- 35. Caroline Jarrett @cjforms(CC) BY SA-4.0 35 People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” Response depends on effort, reward, and trust 35
- 36. Caroline Jarrett @cjforms(CC) BY SA-4.0 36 A good question works in three ways 36 Appropriate Obvious Interesting
- 37. Caroline Jarrett @cjforms(CC) BY SA-4.0 3737 Why did you visit our website today? Appropriate Obvious Interesting
- 38. Caroline Jarrett @cjforms(CC) BY SA-4.0 38 Would you recommend us to a friend or family member? In a shop, buying a baby carriage In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 38
- 39. Tip Test your questionsby interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 39
- 40. 40 Let’s think abit more about fieldwork
- 41.
- 42. 1950s mindset: “AskEverything” 42 Survey = Big Honkin’ Survey
- 43. Caroline Jarrett @cjforms(CC) BY SA-4.0 43 Big Honkin’ Survey = think hard about everything Goals Sample Questions Fieldwork 43 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
- 44. 2015 mindset: theLight Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 44
- 45.
- 46. Caroline Jarrett @cjforms(CC) BY SA-4.0 46 Light Touch Survey = easier choices + iteration Goals Sample Questions Fieldwork 46 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
- 47. From goals toinsight - quickly • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 47
- 48. Time for newquestion 48 One way to iterate, improve, increase Caroline Jarrett @cjforms (CC) BY SA-4.0
- 49. People ask meabout surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” “I prefer 5 points in a rating scale, but my boss likes 7. Who’s is right?” 49
- 50. Caroline Jarrett @cjforms(CC) BY SA-4.0 50 Likert had several different types of question in his response formats Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 50
- 51. You can findan academic paper to support almost any number of points Krosnick and Presser refer to over 80 papers 51 Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf
- 52. Caroline Jarrett @cjforms(CC) BY SA-4.0 52 Respondents focus on the actual question Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 52
- 53. Tip Don’t stress toomuch about the number of points in your rating scale 53Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
- 54. Caroline Jarrett @cjforms(CC) BY SA-4.0 54 Well, OK, stress a little bit. 54 This scale is downright peculiar. Avoid.
- 55. 55 What about the‘valid results’ bit?
- 56. Caroline Jarrett @cjforms(CC) BY SA-4.0 5656 The aim is to get the best number you can, within the resources you have
- 57. Caroline Jarrett @cjforms(CC) BY SA-4.0 57 What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 57 The aim is to get the best number you can, within the resources you have
- 58. Caroline Jarrett @cjforms(CC) BY SA-4.0 58 The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number
- 59. Caroline Jarrett @cjforms(CC) BY SA-4.0 5959 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have Resources What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use
- 60. Caroline Jarrett @cjforms(CC) BY SA-4.0 6060 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct By making good choices, you minimise the error Resources
- 61. 61 Total Survey Errordiagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
- 62. What number doyou need to make the decision? Use your UX skills (especially technical communication) Go for Light Touch surveys Use the Survey Octopus to make good choices Caroline Jarrett @cjforms (CC) BY SA-4.0 62
- 63.
Editor's Notes
- #5 This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
- #6 People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- #13 The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
- #19 The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
- #20 The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
- #21 The octopus again. This time we're looking at 'the questions we ask'.
- #22 The resources you have will help you to decide on the reward you’re offering
- #23 Prank leaves Justin Bieber facing tour of North Korea By Daniel Emery Technology reporter, BBC News 5 July 2010 Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea. A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on. There are now almost half a million votes to send the singer to the secretive communist nation. The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days. Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
- #25 The octopus, with focus on 'The list you sample from'
- #27 The octopus again; we've looked at 6 of the 8 tentacles.
- #34 The octopus again; we've looked at 6 of the 8 tentacles.
- #36 People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- #37 Obvious questions require no effort to answer. Interesting questions are rewarding to answer. Appropriate questions are ones that inspire trust in the respondent.
- #38 People come to the web with their own questions, so they’re likely to know why they are on your website. It’s a relatively interesting question to answer, and it’s appropriate to ask visitors why they’re visiting.
- #39 If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
- #44 The octopus again; we've looked at 6 of the 8 tentacles.
- #46 Excerpt from Death to ‘it depends’ by Leisa Reichelt (@leisa) Death to ‘it depends’ Lately I find myself on a mission for mass simplification. Possibly over simplification, but I’m not sure it matters. It’s one of the things I care most about at the moment – how can we simplify what we are asking people to do so that there is nothing else they can do but start doing it, instead of following their natural inclination to make a list, hire a consultant, write a white paper, do anything but doing the thing. It requires that I stop saying (or even thinking) one of the things I have probably said most in my entire working life – ‘it depends’. That’s hard, but I think it’s the right thing to do. It depends is paralysing. Read more at:http://www.disambiguity.com/death-to-it-depends/
- #47 The octopus again; we've looked at 6 of the 8 tentacles.
- #49 A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up. It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
- #58 This is a more conventional way of looking at the octopus tentacles
- #59 If we just look at the issues (no tentacles) we get this slide
- #60 This slide translates the issues into the technical terms used by survey methodologists
- #61 This slide translates the issues into the technical terms used by survey methodologists