Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Survey Octopus - getting valid data from surveys

1,276 views

Published on

Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016

Published in: Design
  • Could you use an extra $1750 a week? I'm guessing you could right? If you would like to see how you could make this type of money, right from the comfort of your own home, you absolutely need to check out this short free video. ■■■ http://t.cn/AisJWzdm
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I earned $5000 ultimate month by using operating online only for 5 to 8 hours on my computer and this was so smooth that i personally couldn't accept as true with before working on this website. if you too need to earn this sort of huge cash then come and be part of us. do this internet-website online... ●●● http://t.cn/AisJWYf4
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Easy and hassle free way to make money online! I have just registered with this site and straight away I was making money! It doesn't get any better than this. Thank you for taking out all the hassle and making money answering surveys as easy as possible even for non-techie guys like me! ★★★ http://t.cn/AieXAuZz
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Visit Here to Read PDF Format === http://bestadaododadj.justdied.com/235451090X-octopus-one-la-conspiration-desbanksters.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • How we discovered the real reason nice guys don't get laid, and a simple "fix" that allows you to gain the upper hand with a girl... without changing your personality or pretending to be someone you're not. learn more... ▲▲▲ http://ishbv.com/unlockher/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

The Survey Octopus - getting valid data from surveys

  1. 1. The Survey Octopus Getting valid data from surveys Caroline Jarrett @cjforms 2016 #uxinthecity
  2. 2. Caroline Jarrett @cjforms What would you do for a dollar? 2
  3. 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 $1 in the envelope beats $10 (guaranteed) later Caroline Jarrett @cjforms (CC) BY SA-4.0 3
  4. 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 4
  5. 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
  6. 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 I’m a forms specialist Image credit: Flickr, taxrebate.org.uk 6
  7. 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why do people answer questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 7
  8. 8. Caroline Jarrett @cjforms “Please have a look at this survey” 8
  9. 9. Caroline Jarrett @cjforms The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
  10. 10. Caroline Jarrett @cjforms The survey is a process for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  11. 11. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  12. 12. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  13. 13. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers of the attributes of the larger population of which the entities are members.
  14. 14. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions
  15. 15. Caroline Jarrett @cjforms To make decisions People The survey is a process for getting answers to questions getting numbers
  16. 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 16
  17. 17. Caroline Jarrett @cjforms “The numbers” or “The number”? 17
  18. 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork used to be expensive so a survey was a rare event. Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/ 18
  19. 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 1950s mindset: “Ask Everything” Survey = Big Honkin’ Survey 19
  20. 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 2016 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 20
  21. 21. Tip Interview first Caroline Jarrett @cjforms (CC) BY SA-4.0 21
  22. 22. Caroline Jarrett @cjforms One way to iterate, improve, increase Time for new question Caroline Jarrett @cjforms (CC) BY SA4.0 22
  23. 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 23
  24. 24. Caroline Jarrett @cjforms 24
  25. 25. Caroline Jarrett @cjforms Asking the right question Asking the right people 25
  26. 26. Caroline Jarrett @cjforms 26
  27. 27. Caroline Jarrett @cjforms http://www.bbc.com/news/10506482 27
  28. 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 Asking the right people is better than asking lots of people Sample: the list you sample from Caroline Jarrett @cjforms (CC) BY SA-4.0 28
  29. 29. Caroline Jarrett @cjforms 29
  30. 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 There’s often a ‘zone of indifference’ Hate it Love it 30
  31. 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Feeling special is rewarding People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 31
  32. 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 Ask fewer people to get better response Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 32
  33. 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try mail, phone or face-to-face for better response rates Fieldwork: Who answers? Caroline Jarrett @cjforms (CC) BY SA-4.0 33
  34. 34. Did we get answers from the right people? Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 34
  35. 35. Population of assorted birds Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 35
  36. 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t just want answers, we want representative answers Response Caroline Jarrett @cjforms (CC) BY SA-4.0 36
  37. 37. Caroline Jarrett @cjforms Asking the right question Asking the right people 37
  38. 38. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 38
  39. 39. Caroline Jarrett @cjforms 39
  40. 40. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 40
  41. 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 Helps a lot if you ask good questions Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 41
  42. 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 42
  43. 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 A good question works in three ways Obvious Interesting Appropriate 43
  44. 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why did you visit our website today? Obvious Interesting Appropriate 44
  45. 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 45
  46. 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 46
  47. 47. Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 47
  48. 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t want any old answers. We want useful answers. Response Caroline Jarrett @cjforms (CC) BY SA-4.0 48
  49. 49. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 49
  50. 50. Caroline Jarrett @cjforms On your most recent working day, what percentage of time did you spend using the Internet? % Caroline Jarrett @cjforms (CC) BY SA-4.0 50
  51. 51. Caroline Jarrett @cjforms (CC) BY SA-4.0 The approximate curve of forgetting Major life event Noticeable, occasional Unremarkable, repetitive Perfect recall Day Gone Week Month YearHour 51
  52. 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Can’t remember = can’t answer Response Caroline Jarrett @cjforms (CC) BY SA-4.0 52
  53. 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights are the answers that you use to make the decisions Insights Caroline Jarrett @cjforms (CC) BY SA-4.0 53
  54. 54. Caroline Jarrett @cjformsCaroline Jarrett @cjforms (CC) BY SA-4.0 54
  55. 55. Caroline Jarrett @cjforms "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett 55
  56. 56. Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 56
  57. 57. Caroline Jarrett @cjforms Can we make the decision yet? Can we make the decision yet? Can we make the decision yet? Caroline Jarrett @cjforms (CC) BY SA4.0 57
  58. 58. Tip Try writing your presentation first 58
  59. 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 We’ve met the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0 59
  60. 60. What could possibly go wrong? Caroline Jarrett @cjforms (CC) BY SA-4.0 60
  61. 61. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 61

×