Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Survey that work LibDesign2016 @cjforms

930 views

Published on

Introduction to survey methods at LibDesign2016. A workshop led by Caroline Jarrett for people working in the library service and public sector in the Czech Republic. Caroline Jarrett led this workshop in Prague in September 2016 as part of the LibDesign 2016 conference.

Published in: Design
  • Real people just like you are kissing the idea of punching the clock for someone else goodbye, and embracing a new way of living. The internet economy is exploding, and there are literally THOUSANDS of great earnings opportunities available right now, all just one click away. ●●● http://t.cn/AisJWCv6
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Get Paid For Your Opinions! Earn $5-$10 cash on your first survey. ●●● http://t.cn/AieXAuZz
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Survey that work LibDesign2016 @cjforms

  1. 1. Surveys that work Caroline Jarrett @cjforms 2016 #surveysthatwork An introduction to using survey methods
  2. 2. Caroline Jarrett @cjforms 2 Let’s find out about our experience
  3. 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fill in this questionnaire 1. How many surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better survey, based on experience of writing or answering? __________________________________ __________________________________ Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys, 49th Society for Technical Communication Conference, Nashville TN USA
  4. 4. Caroline Jarrett @cjforms 4 Now work in pairs
  5. 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try this as an interview 1. How many surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better survey, based on experience of writing or answering? __________________________________ __________________________________ Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys, 49th Society for Technical Communication Conference, Nashville TN USA
  6. 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 6 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  7. 7. Agenda Introductions Goals Sample Break QuestionnaireQuestions Lunch Fieldwork Break Responses Insights Finish 7
  8. 8. Caroline Jarrett @cjforms The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
  9. 9. Caroline Jarrett @cjforms The survey is a process for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  10. 10. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  11. 11. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  12. 12. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers of the attributes of the larger population of which the entities are members.
  13. 13. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
  14. 14. Caroline Jarrett @cjforms The survey is a process for getting answers to questions To make decisions People getting numbers Caroline Jarrett @cjforms (CC) BY SA-4.0
  15. 15. Caroline Jarrett @cjforms (CC) BY SA-4.0 15 The aim of a survey is to get a number that helps you to make a decision
  16. 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 16 The aim of a survey is to get a number that helps you to make a decision Goals Sample Fieldwork Responses Insights Questionnaire Questions
  17. 17. Caroline Jarrett @cjforms 17
  18. 18. Caroline Jarrett @cjforms Asking the right question Asking the right people 18
  19. 19. Caroline Jarrett @cjforms 19
  20. 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 20 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  21. 21. Agenda Introductions Goals and sample Questions and questionnaire 12:30 Lunch Fieldwork Responses and Insights 16:30 Finish 21
  22. 22. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Goals Establish your goals for the survey Questions you need answers to Goals 22
  23. 23. Caroline Jarrett @cjforms 23 Goals
  24. 24. Caroline Jarrett @cjforms (CC) BY SA-4.0 Establish your goals for the research Goals 24 What do you want to know?
  25. 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 Establish your goals for the research Goals What do you want to know? Why do you want to know? 25
  26. 26. Caroline Jarrett @cjforms (CC) BY SA-4.0 Establish your goals for the research What do you want to know? Why do you want to know? What decisions will you make based on these answers? Goals 26
  27. 27. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals What are Apple’s goals for the survey? What do they want to know? Why do they want to know it? What decisions will they make as a result of the survey?
  28. 28. Caroline Jarrett @cjforms 28 Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/ Goals
  29. 29. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals 1950s mindset: “Ask Everything” Survey = Big Honkin’ Survey
  30. 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals 2016 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 30
  31. 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals This one-box survey asks one open question 31
  32. 32. Caroline Jarrett @cjforms Goals One way to iterate, improve, increase Time for new question Caroline Jarrett @cjforms (CC) BY SA-4.0 32
  33. 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals What’s the Most Crucial Question? • We want to ask the fewest questions that will help us to make the decision so we need to know which are the most useful questions • Even better: know the specific Most Crucial Question • A Most Crucial Question has a numeric answer 33
  34. 34. Caroline Jarrett @cjforms Goals Another way: narrow downLots of questions Useful questions MCQ Caroline Jarrett @cjforms (CC) BY SA-4.0 34
  35. 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample 35
  36. 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Asking the right people is better than asking lots of people Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the list you sample from 36
  37. 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Choose a good list Coverage error: Mismatch between the people you want to ask and the list you choose to sample from Caroline Jarrett @cjforms (CC) BY SA-4.0
  38. 38. Caroline Jarrett @cjforms Sample http://www.bbc.com/news/10506482 38
  39. 39. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Difference between response, response rate and representativeness Concept Definition Example Response Number of answers 5,000 Response rate Response divided by the number of invitations 10% Representativeness Whether respondents you get are typical of the users you want Image credit: North Korean flag, http://commons.wikimedia.org/wiki/File:Flag_of_North_Korea.svg 39
  40. 40. Sample Did we get answers from the right people? Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 40
  41. 41. Sample Population of assorted birds Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 41
  42. 42. Caroline Jarrett @cjforms Goals Iterate, improve, increase to understand the people you want to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 42
  43. 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Decide how to target the correct people • Go where they are • Use a list • Buy a sample • Send and hope • Try a ‘snowball’ Image credit: Flickr sunchild57 43
  44. 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Non-response error is the one that hurts Non-response error: The ones who answer differ from the ones you ask in a way that affects the survey statistic Caroline Jarrett @cjforms (CC) BY SA-4.0
  45. 45. Fieldwork Let’s try the toothpaste Image credit: Caroline Jarrett 45
  46. 46. Caroline Jarrett @cjforms Sample 46
  47. 47. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample I wanted to tell them something • Perceived effort: three questions is OK • Perceived reward: the opportunity to tell them something • Trust: Probably enough 47
  48. 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Response depends on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 48
  49. 49. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Response relies on effort, reward, and trust 49
  50. 50. Sample An example survey • Jane went shopping for a iPhone cover for her daughter’s phone. There was very little choice, but they found one that was OK. Then there weren’t any tills. “The experience was surprisingly poor for such a big brand” • Will Jane decide to answer the survey? 50
  51. 51. Caroline Jarrett @cjforms Sample 51
  52. 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Your views What is the perceived effort? What is the perceived reward? What about trust?
  53. 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample There’s often a ‘zone of indifference’ Hate it Love it 53
  54. 54. Caroline Jarrett @cjforms (CC) BY SA-4.0 Jane had a question for Apple Burning Issue Why don’t you have tills? 54
  55. 55. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample What do people want to tell you? Burning Issues 55
  56. 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Overcome the ‘Zone of Indifference’ by asking about the Burning Issues 56
  57. 57. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Think about the people you want to ask • Who are they? • How will you find them? • Do they want to answer your questions? • What do they want to tell you? 57
  58. 58. Caroline Jarrett @cjforms (CC) BY SA-4.0 There is always sampling error Sampling error: Ask a sample instead of asking everyone Caroline Jarrett @cjforms (CC) BY SA-4.0
  59. 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 If you get all the other decisions right, then you can calculate a margin of error 59
  60. 60. Caroline Jarrett @cjforms (CC) BY SA-4.0 Ask fewer people to get better response Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 60
  61. 61. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try mail, phone or face-to-face for better response rates Fieldwork: Who answers? Caroline Jarrett @cjforms (CC) BY SA-4.0 61
  62. 62. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t just want answers, we want representative answers Response Caroline Jarrett @cjforms (CC) BY SA-4.0 62
  63. 63. Caroline Jarrett @cjforms Asking the right question Asking the right people 63
  64. 64. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 64
  65. 65. Caroline Jarrett @cjforms (CC) BY SA-4.0 A survey is only valid if the questions match the reason you’re doing it Lack of validity: mismatch between what you ask and what you need to know Caroline Jarrett @cjforms (CC) BY SA-4.0 65
  66. 66. Caroline Jarrett @cjforms (CC) BY SA-4.0 Here is a question from that survey Are you aware that you can upgrade your contract or activate an iPhone on a rate plan with a mobile carrier directly at the Apple Store? Apple Most Crucial Question? 66
  67. 67. Caroline Jarrett @cjforms (CC) BY SA-4.0 Most Crucial Question meets Burning Issue Apple Are you aware that you can upgrade your contract or activate an iPhone on a rate plan with a mobile carrier directly at the Apple Store? Why don’t they have tills? 67
  68. 68. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process People you will invite to answer Establish your goals for the survey Goals Questions you need answers to Sample Decide who to ask and how many Test the questions Questions Questions people can answer 68
  69. 69. Caroline Jarrett @cjforms Questions 69
  70. 70. Caroline Jarrett @cjforms Questions In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 70
  71. 71. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Helps a lot if you ask good questions Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions: What are you asking about? How many questions? 71
  72. 72. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions There are four steps to answer a question Understand Find Judge Place Adapted from Tourangeau, R., Rips, L. J. and Rasinski, K. A. (2000) “The psychology of survey response”
  73. 73. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions There are four steps to answer a question Step A good question … 1. Read and understand is legible and makes sense 2. Find an answer asks for answers that we know 3. Judge the answer asks for answers we’re happy to reveal 4. Place the answer offers appropriate spaces for the answers Adapted from Tourangeau, R., Rips, L. J. and Rasinski, K. A. (2000) “The psychology of survey response”
  74. 74. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 1: read and understand 74
  75. 75. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 1: read and understand Hermann grid illusion 75
  76. 76. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 2: find the answer In your last five days at work, what percentage of your work time do you estimate that you spend using publicly- available online services (not including email, instant messaging and search) to do your work using a work computer or other device? 76
  77. 77. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions The approximate curve of forgetting
  78. 78. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 3: judge the answer 78
  79. 79. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 4: place the answer 79
  80. 80. Caroline Jarrett @cjforms Questions Understand Find Judge Place 80
  81. 81. Write a better version of this question In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 81
  82. 82. Caroline Jarrett @cjforms On your most recent working day, what percentage of time did you spend using the Internet? % Caroline Jarrett @cjforms (CC) BY SA-4.0 82
  83. 83. Caroline Jarrett @cjforms Do you use the Internet for your work? Yes No Other Caroline Jarrett @cjforms (CC) BY SA-4.0 83
  84. 84. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Write good questions • Mix question types: choice and open • Avoid leading questions • Present one question at a time • Keep positive; negatives are harder to understand • Ask questions that users can answer Image credit: shutterstock.com 84
  85. 85. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Test the questions Questions Questions people can answer Build the questionnaire Questionnaire Questions people can interact with 85
  86. 86. Caroline Jarrett @cjforms (CC) BY SA-4.0 A good question gets good answers Measurement error: Mismatches between the questions you ask and the answers that people give you Caroline Jarrett @cjforms (CC) BY SA-4.0
  87. 87. Caroline Jarrett @cjforms Questionnaire Caroline Jarrett @cjforms (CC) BY SA-4.0 87
  88. 88. Caroline Jarrett @cjforms Questionnaire "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett 88
  89. 89. Questionnaire Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 89
  90. 90. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire “Place the answer” is also about using the right widget to collect the answer Use For Radio buttons A single known answer Check boxes Multiple known answers Text boxes Unknown answers Allen Miller, S. J. and Jarrett, C. (2001) “Should I use a drop-down?” http://www.formsthatwork.com/files/Articles/dropdown.pdf 90
  91. 91. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Likert had several types of response format in his scales Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 91
  92. 92. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire You can find an academic paper to support almost any number of response points Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. http://bit.ly/KNWlio 92
  93. 93. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Grids are often full of problems at all four steps 93
  94. 94. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Grids are a major cause of survey drop-out 35% 20% 20% 15% 5% 5% Total incompletes across the 'main' section of the questionnaire (after the introduction stage) Subject Matter Media Downloads Survey Length Large Grids Open Questions Other Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar From Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne 94
  95. 95. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire But it’s the topic that matters most 35% 20% 20% 15% 5% 5% Total incompletes across the 'main' section of the questionnaire (after the introduction stage) Subject Matter Media Downloads Survey Length Large Grids Open Questions Other Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar From Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne 95
  96. 96. Questionnaire Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 96
  97. 97. Caroline Jarrett @cjforms Your answers to this survey are important for our work But what’s in it for me? And I’m really ready for a break. 97
  98. 98. Agenda Lunch Fieldwork Break Responses Insights Finish QuestionnaireQuestions Goals Sample Introductions Break 98
  99. 99. Caroline Jarrett @cjforms 99
  100. 100. Goals Sample Goals and sample for the survey • We’ve had this request for help with a survey • We’ll be having a meeting to discuss the survey • Decide on the topics you’ll want to discuss at the meeting • Also, prepare a suggestion for the Most Crucial Question 100
  101. 101. Questions Write a good question We have discussed some possible questions Decide on the question you will ask Check that users can: - Read and understand it - Find the answer - Judge the answer 101
  102. 102. Questionnaire Make a questionnaire Make a paper version of your questionnaire (We’ll be testing the questionnaires a bit later) 102
  103. 103. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 103 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  104. 104. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Fieldwork Run the survey from invitation to follow-up People who actually answer 104
  105. 105. Caroline Jarrett @cjforms Fieldwork 105
  106. 106. Caroline Jarrett @cjforms Fieldwork 106
  107. 107. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Recap: Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 107
  108. 108. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork The elements of a good invitation • Trust: – Say who you are – Say why you’ve contacted this person specifically • Perceived reward: – Explain the purpose of the survey – Explain why this person’s responses will help that purpose – If there is an incentive, offer it • Perceived effort: – Outline the topic of the survey – Say when the survey will close – Do NOT say how long it will take • (unless you have tested the heck out of it and are extremely sure that you know the answer) 108
  109. 109. Fieldwork Write the invitation and thank-you • Hints: – the invitation can be part of the questionnaire – thank-you is on a separate page 109
  110. 110. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Test it: pilot study • Run the survey from invitation to the follow-up • Look for mechanical problems like wrong link in the invitation, no thank-you page • Find out what your response rate is so that you can work out your sample size “If you don’t have time to do a pilot study, you don’t have time to do the survey” 110
  111. 111. Fieldwork Test it 111
  112. 112. Fieldwork Report back on your questionnaire 112
  113. 113. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Think about the test and iterate • Are the people you tested with representative? • Did you test the whole survey – From invitation to follow up? – Including the analysis of responses? – Including finding out whether you can make the decision? • What do you need to change for the next version? 113
  114. 114. Agenda Responses Insights Finish Break Goals Sample Fieldwork QuestionnaireQuestions Introductions Break Lunch 114
  115. 115. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 115 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  116. 116. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Responses Clean the data Answers 116
  117. 117. Caroline Jarrett @cjforms (CC) BY SA-4.0 The answers that you get will tell you whether you had good questions Caroline Jarrett @cjforms (CC) BY SA-4.0 Measurement error: Mismatches between the questions you ask and the answers people actually give you
  118. 118. Caroline Jarrett @cjforms Responses 118
  119. 119. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Clean your data • Look for gaps and missing entries • Remove any (unintended) duplicate responses • Read the answers to make sure that they make sense compared to the questions Adapted from Boslaugh, S. and P. A. Watters (2008) Statistics in a nutshell O’Reilly119
  120. 120. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Decide whose answers to include Adjustment error: Problems when deciding whether to include or exclude someone’s answers Caroline Jarrett @cjforms (CC) BY SA-4.0
  121. 121. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Look after your data • Data analysis can take a long time; you won’t want to repeat it – Make copies of your data, especially before any drastic change – ‘Undo’ doesn’t always work on large files • Make notes of what you did – It helps if you have to defend your conclusions – It’s hard to remember the details a year later Image credit: Shutterstock 121
  122. 122. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Decide what to do when people have skipped questions or dropped out 1. Remove the whole of that person’s response 2. Use the partial responses, and accept that your number of responses is lower for some questions 3. Calculate an “imputed value” – Include a flag showing that the value is calculated – Estimate the most likely value using the other data 122
  123. 123. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses If you’re losing people, have you still got representativeness? 123
  124. 124. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses You can interpret data well – or poorly Processing error: Bad choices about how to interpret the answers Caroline Jarrett @cjforms (CC) BY SA-4.0
  125. 125. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Typing in the answers = coding Image credit: https://www.census.gov/history/www/census_then_now/notable_alumni/herman_hollerith.html 125
  126. 126. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses If you ask for answers, you have to read and think about them 126
  127. 127. Responses Coding example 1 • A survey asked ‘where do you live?’ It had a free text box for countries outside the UK. • These are example answers. How would you code them for processing? CZ Czech Czech Republic DEMOCRATIC REPUBLIC OF CONGO Denmark Dhaka, Bangladesh 127
  128. 128. Responses Coding example 1 • My coding answers CZ Czech Republic Czech Czech Republic Czech Republic Czech Republic DEMOCRATIC REPUBLIC OF CONGO Democratic Republic of Congo Denmark Denmark Dhaka, Bangladesh Bangladesh 128
  129. 129. Responses Coding example 2 • Some more answers from the same survey. How would you code them for processing? Dubai, UAE Dubai, United Arab Emirates E Africa Ecuador Egypt Egypt and we have an office in London el salvador 129
  130. 130. Responses Coding example 2 • My coding answers Dubai, UAE United Arab Emirates Dubai, United Arab Emirates United Arab Emirates E Africa Africa Ecuador Ecuador Egypt Egypt Egypt and we have an office in London Egypt el salvador El Salvador 130
  131. 131. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Coding example 3 • Some more examples Home is in Thailand currently working in Vietnam on Temporary Residence Card I am consultant/engineer and I work in Spain and Portugal. KYRGYZSTAN NOW IS IN KAZAKHSTAN Live in North East England but live in Makeni, Sierra Leone for about 4 months of every year No fixed abode at present but mainly in Turkey, Malaysia and Thailand 131
  132. 132. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses CAQDAS tools are available (but are a big challenge) Before buying one, read this site: http://www.surrey.ac.uk/sociology/research/ researchcentres/caqdas/support/choosing/index.htm http://bit.ly/Surrey1234 Image credit: http://www.surrey.ac.uk/sociology/research/researchcentres/caqdas/support/choosing/index.htm 132
  133. 133. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Wordle from a survey on usability certification 133
  134. 134. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Wordle.net example: in favour of Facebook 134
  135. 135. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Another: against Facebook 135
  136. 136. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 136 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  137. 137. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Insights Analyse and present the results Decisions 137
  138. 138. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Explore your data and ask questions 138
  139. 139. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Explore your data and ask questions 139
  140. 140. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Use graphs and charts to understand relationships in the data Anscombe, F. J.. (1973). Graphs in Statistical Analysis. The American Statistician, 27(1), 17–21. http://doi.org/10.2307/2682899 140
  141. 141. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Two datasets, same summaries • X Mean: 54.26 • Y Mean: 47.83 • X SD: 16.76 • Y SD: 26.93 • Corr.: -0.06 https://twitter.com/JustinMatejka/status/770682771656368128 141
  142. 142. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Justin Matejka’s dataset 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 y https://twitter.com/JustinMatejka/status/770682771656368128 142
  143. 143. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Alberto Cairo’s dataset 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 120 y-dino https://twitter.com/JustinMatejka/status/770682771656368128 143
  144. 144. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Use descriptive statistics to explore numerical data • Most seen for statistics – Mean (arithmetic average) – Standard deviation (spread of answers) • Useful for thinking about the data – Range (lowest to highest) – Mode (most common answer) 144
  145. 145. Insights A ‘Like / Dislike’ question got these responses Strongly dislike 2 Dislike 6 Neither dislike nor like 14 Like 31 Strongly like 13 Total responses 66 Please work out: the percentage of respondents who ‘like’ 145
  146. 146. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights There are many ways to combine ratings into means and percentages • 47% 31 ticked ‘like’ so 31/66 = 47% • 67% ‘Top box’ / ‘top 2 box’ uses the positive responses • 68% ‘0 to 4’ weights responses: 0%, 25%, 50%, 75%, 100% • 74% ‘1 to 5’ weights responses: 1, 2, 3, 4, 5 (then divide by 5) • 36% ‘-1 to 1’ weights responses: -100%, -50%, 0, 50%, 100% 67% 68% 74% 36% 146
  147. 147. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Net Promoter Score™ has a special analysis method Image credit: https://www.netpromoter.com/know/147
  148. 148. Caroline Jarrett @cjforms Asking the right question Asking the right people Choose whichever method you like, but you must make the choice when you decide on the goals of the survey 148
  149. 149. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 149 Establish your goals for the survey Goals Questions you need answers to Decide who to ask and how many People you will invite to answer Sample Clean the data Responses Answers Analyse and present the results Insights Decisions Test the questions Questions Questions people can answer Run the survey from invitation to follow-up Fieldwork People who actually answer Build the questionnaire Questionnaire Questions people can interact with
  150. 150. Caroline Jarrett @cjforms (CC) BY SA-4.0 All the topics are connected Goals Sample Questionnaire Fieldwork Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response Questions 150
  151. 151. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use Caroline Jarrett @cjforms (CC) BY SA-4.0 151
  152. 152. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use Caroline Jarrett @cjforms (CC) BY SA-4.0 152
  153. 153. Caroline Jarrett @cjforms 153 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  154. 154. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 154
  155. 155. Caroline Jarrett @cjforms (CC) BY SA-4.0 Should I do this survey? Do I know how I’m going to use the answers? Is a survey the right way to get the answers? Do I have time to test and to iterate? Do people have answers to these questions? Do people want to respond to my request?Yes Yes Yes Yes GoYes 155
  156. 156. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 156
  157. 157. Caroline Jarrett @cjforms (CC) BY SA-4.0 Some tests to make sure it’s a good one Establish your goals for the survey Know the decisions that you’ll make Find out what people want to tell you by interviewing Goals Decide who to ask and how many Check that the list of people to ask is representative Find out your response rate with a pilot study Sample Test the questions and build the questionnaire Test the questions in cognitive interviewing Usability test of the questionnaire Questions Run the survey from invitation to follow-up Do a pilot study (again) to check everything from invitation to insights Fieldwork Clean the data Check the responses as they arrive Check that the answers are representative Responses Analyse and present the results Ask: ‘so what’? Iterate and improve Insights

×