Survey Says! Measuring for Change
•
3 likes•1,377 views
This document provides guidance on best practices for designing and conducting surveys. It emphasizes keeping surveys short, using clear and concise questions, providing exhaustive answer options, and avoiding biases. It also recommends testing surveys beforehand through user testing and pilot studies. Proper sampling, sample sizes, incentives, and expressing gratitude to respondents are discussed as well. The overall message is that surveys should be designed with a focus on the respondent experience in order to obtain high quality, useful data.
Recommended
Recommended
More Related Content
Similar to Survey Says! Measuring for Change
Similar to Survey Says! Measuring for Change (20)
Recently uploaded
Recently uploaded (20)
Survey Says! Measuring for Change
- 1. @Curio_Research SURVEY SAYS! MEASURING WITH SURVEYS Lauren Isaacson Market and UX Research Consultant Curio Research 1
- 2. @Curio_Research THANK YOU FOR COMING!
- 5. @Curio_Research SURVEYS! WHAT ARE THEY GOOD FOR? Absolutely something… 5 https://upload.wikimedia.org/wikipedia/commons/thumb/3/3b/Love_Hate_Love_%2815139163553%29.jpg/1024px-Love_Hate_Love_%2815139163553%29.jpg
- 6. @Curio_Research WE LOVE TO SEND SURVEYS
- 7. “ @Curio_Research If you cannot measure it, you cannot improve it. -Lord Kelvin 7
- 10. @Curio_Research WE HATE TO GET SURVEYS
- 13. @Curio_Research
- 15. @Curio_Research WHAT MAKES A GOOD SURVEY? 15 http://www.thebluediamondgallery.com/pictures/survey.jpg
- 17. @Curio_Research THE SHORTER THE BETTER ➤ Have a singular focus - Keep the number of research objectives to a minimum ➤ Use quantitative questions - Open ended text response questions make the survey longer and more taxing on the respondent ➤ No longer than 10 minutes to take - Fewer Questions = More Responses + More Completes https://pixabay.com/p-161539/
- 18. @Curio_Research MOBILE FIRST 44% of SurveyMonkey surveys were taken on a mobile device last March* *SurveyMonkey European Survey Research Association 2017 Presentation
- 19. @Curio_Research DO YOU REALLY NEED TO ASK THAT? REALLY? ARE YOU SURE? If you knew this information, what would you do?
- 20. @Curio_Research MAKE ALL QUESTIONS TWEET LENGTH QUESTIONS Simply stated. Direct. Clear. Jargon free.
- 22. @Curio_Research HELP RESPONDENTS GIVE YOU CORRECT ANSWERS
- 23. @Curio_Research HELP RESPONDENTS BE ACCURATE ➤ Make answer options exhaustive - Think about and do some research to give respondents every possible option ➤ Consider the ranges you offer carefully - Respondents will interpret the ranges you give as typical of the population - Below average, average, above average ➤ Don’t expect respondents to remember anything - Give specifics https://c1.staticflickr.com/8/7299/12033015254_7c20d3425c.jpghttp://maxpixel.freegreatpicture.com/static/photo/1x/Accuracy-Dartboard-Sport-Competition-Target-Darts-857092.jpg
- 24. @Curio_Research HELP RESPONDENTS BE ACCURATE ➤ Offer an opt-out - Other, None, Not Applicable ➤ Let people enter in their own answer - Other _________ • Only use this if you actually plan to use the data https://c1.staticflickr.com/8/7299/12033015254_7c20d3425c.jpg https://pixabay.com/p-1955257/
- 25. @Curio_Research DO NOT MAKE QUESTIONS REQUIRED
- 27. @Curio_Research ➤ Do you have a non-human companion? Dog Cat
- 28. @Curio_Research ➤ What kind of pet do you have? Dog Cat
- 29. @Curio_Research ➤ What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other I don’t have a pet
- 30. @Curio_Research ➤ What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other I don’t have a pet
- 31. @Curio_Research ➤ What kind of pet do you have? RANDOMIZE ANCHOR Dog Cat Amphibian Reptile Rodent Fish Bird Other I don’t have a pet
- 32. @Curio_Research ➤ What kind of pet do you have? Short question Plain language Exhaustive answer options Allows multiple answers Randomized and Anchored It’s not mandatory Dog Cat Amphibian Reptile Rodent Fish Bird Other I don’t have a pet
- 33. @Curio_Research ➤ What kind of pet do you have? https://c1.staticflickr.com/8/7073/6878033654_cf98354007_b.jpg Dog Cat Amphibian Reptile Rodent Fish Bird Other I don’t have a pet
- 35. @Curio_Research GOOD SCALES ARE: ➤ Likert Scales - Odd numbers, 5 or 7 - Neutral is a perfectly valid answer ➤ Balanced - The same number of positive and negative options with a neutral option in the middle ➤ Vertical - Mobile first https://upload.wikimedia.org/wikipedia/commons/9/9e/Scales_of_Justice_%28PSF%29.png
- 37. @Curio_Research ➤ Please rate your level of satisfaction with this conference: Excellent Good Decent Poor
- 38. @Curio_Research ➤ How is the conference going so far: Excellent Good Decent Poor
- 39. @Curio_Research ➤ How is the conference going so far: ExcellentGoodDecentPoor
- 40. @Curio_Research ➤ How is the conference going so far: ExcellentGoodDecentPoor Neutral
- 41. @Curio_Research ➤ How is the conference going so far: I’m having a great time! It’s pretty good It kind of bad It’s really bad Neutral
- 42. @Curio_Research ➤ How is the conference going so far: I’m having a great time! It’s pretty good It kind of bad It’s really bad Neutral
- 43. @Curio_Research ➤ How is the conference going so far: I’m having a great time! It’s pretty good It kind of bad It’s really bad Neutral Short question Plain language Neutral option Likert Scale Balanced scale Vertical
- 44. @Curio_Research ➤ How is the conference going so far: I’m having a great time! It’s pretty good It kind of bad It’s really bad Neutral https://www.flickr.com/photos/47823583@N03/8508102188/
- 45. @Curio_Research MINIMIZE GRID USAGE Relying on grids is cheating
- 47. @Curio_Research BIAS IS REAL Agree or Disagree?
- 48. @Curio_Research ACQUIESCENCE & DESIRABILITY ➤ People naturally want to please and fit in with each other, even people they’ve never met ➤ Avoid: - Agree/Disagree - Yes/No - True/False ➤ Rely on actual usage or purchase data instead of doing a survey. https://www.flickr.com/photos/47652574@N06/35966892356/
- 49. @Curio_Research SAMPLING & RESEARCH BIAS ➤ Incorporate randomness into how you select respondents ➤ Dice, random number generators, etc. ➤ Be very careful about how your questions are worded ➤ Try to avoid revealing the sponsor of the survey if you can ➤ It’s easier than you think to communicate what you want people to answer. - Try deliberately using carefully crafted red herrings - Red herrings are also good for insuring data quality as well https://www.flickr.com/photos/9345620@N04/2651935525/
- 51. @Curio_Research ➤ Do you use Gmail? Yes No
- 52. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email
- 53. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email
- 54. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email FantasticMail
- 55. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email FantasticMail RANDOMIZE ANCHOR
- 56. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email FantasticMail Doesn’t lead Plain language Data quality red herring Allows multiple answers Randomized and Anchored It’s not mandatory
- 57. @Curio_Research ➤ Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email FantasticMail https://www.flickr.com/photos/meme_tn/6878030444
- 59. @Curio_Research INCENTIVES ARE NOT NECESSARILY DIFFICULT OR EXPENSIVE ➤ Methods: - Online gift cards - Account credit - Random prize drawing ➤ Incentives can greatly increase your response rate for better data analysis ➤ Offering an incentive demonstrates respect for your users. Their time is just as valuable as yours
- 60. @Curio_Research OPEN WITH GRATITUDE CLOSE WITH GRATITUDE And a final open end
- 61. @Curio_Research BE GRACIOUS ➤ Thank respondents for their participation at the opening and the closing ➤ Tell them how the data will be used - Will it improve a service they use? - Help a non-profit apply for a grant? ➤ Offer to share the results when the study is over ➤ Present a final open end so they can share something you might have missed http://maxpixel.freegreatpicture.com/static/photo/1x/Fig-Mourning-Angel-Sculpture-Gratitude-Consolation-1990440.jpg
- 64. @Curio_Research USER TESTING ➤ Similar as user testing a prototype, but less formal ➤ Ideally, do a cognitive interview with a few people from the target population - Grab a few friends or coworkers who didn’t work on the survey ➤ Have them talk through taking the survey - Did they understand the intent of your questions? - Did they suggest other options? - How much time did it take? Less that 10 minutes? https://www.flickr.com/photos/notionscapital/10089497055
- 66. @Curio_Research PILOT STUDY ➤ A pilot study is like a dress rehearsal for surveys ➤ Recruit from the same population and look for pitfalls, mistakes, and misunderstandings in the recruiting, invitation, the survey itself, and the incentives ➤ When analyzing the results for: - Odd/unexpected answer patterns - Too many “other” entries - Completion rate and time to complete https://sd.keepcalm-o-matic.co.uk/i/keep-calm-its-only-dress-rehearsal.png
- 68. @Curio_Research WHERE TO GET RESPONDENTS? ➤ Your own customer database - Avoid giving your customer base Survey Fatigue ➤ Panel or sample provider - Vet the quality of the sample - Ask the provider about their quality controls: • CAPTCHAs? • Proxy server detection? • Geo-IP validation? • Other authentication measures? https://www.flickr.com/photos/guiguibu91/2889883615
- 69. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/
- 70. @Curio_Research POPULATION, CL, AND ME DEFINITIONS ➤ Population = The total number of people in the group you are trying to study - The number of users of your product - The number of women in the US between the ages of 25 - 34 with college degrees ➤ Confidence Level = The probability your sample accurately reflects the population you are studying - 95% is the industry standard ➤ Margin of Error = The range, as a percentage, that your population’s responses will deviate from the answers of our sample - 5% is the industry standard
- 71. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My user base is 425. How many survey responses do I need for a 95% confidence level and a 5% margin of error?
- 72. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My user base is 425. How many survey responses do I need for a 95% confidence level and a 5% margin of error? 203
- 73. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My user base is 5,000. How many survey responses do I need for a 95% confidence level and a 5% margin of error?
- 74. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My user base is 500,000. How many survey responses do I need for a 95% confidence level and a 5% margin of error? 384
- 75. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My population size is about 5,000,000. How many survey responses do I need for a 95% confidence level and a 5% margin of error?
- 76. @Curio_Research HOW MANY RESPONDENTS DO YOU NEED? https://qualitypursuit.wordpress.com/statistical-tools/ My population size is about 5,000,000. How many survey responses do I need for a 95% confidence level and a 5% margin of error? 385
- 77. @Curio_Research THE MAGIC SAMPLE SIZE IS: 400
- 79. @Curio_Research INVITATION TIPS ➤ Personalize your messages ➤ Keep email messages short and to the point ➤ Try using different forms of contact: ➤ Email, snail mail, text message ➤ Keep changing the message ➤ Strategically time the delivery of your invite ➤ Provide clear instructions ➤ Tell them how their input will be mutually beneficial https://commons.wikimedia.org/wiki/File:Invitation_from_the_National_Guard_of_Washington_to_the_opening_ball_for_the_officers_of_the_United_States_-_Front.jpg
- 80. @Curio_Research POINT OF CONTACT Who ya gonna call?
- 81. @Curio_Research MONITORING PROGRESS ➤ Response rate ➤ Completion rate ➤ Bounced emails ➤ Comments about the survey in the open ends? ➤ When and how often do you send a reminder? http://keytokorean.com/wp-content/uploads/2013/09/keep-calm-and-check-progress-5.png
- 82. @Curio_Research ANALYZING THE DATA 82 https://c1.staticflickr.com/3/2827/9423385629_171671f9c2_b.jpg
- 83. @Curio_Research HAVE A DATA ANALYSIS PLAN BEFORE YOU EVEN WRITE THE SURVEY 1. What are you objectives for the data? 2. What variables do you want to use in your cross tabs? 3. Additional data analysis? Regression? Conjoint? Text analytics? 4. Will you need specialized software to run the analysis? https://www.flickr.com/photos/76074333@N00/318034222
- 84. @Curio_Research CLEANING YOUR DATA ➤ Remove: - Extreme outliers - Respondents who didn’t answer all of the questions - Respondents who used red herring answers - Respondents who straight-lined ➤ Do you still have the sample size you need to continue? - No? Only remove the worst 20% of the iffy responses or get more respondents. https://static.pexels.com/photos/40862/lego-legomaennchen-males-workers-40862.jpeg
- 85. @Curio_Research WHAT DOES MY STANDARD DEVIATION MEAN? Low Number Homogeneous Data High Number Heterogeneous Data
- 86. @Curio_Research TOP 2 BOXES/PERCENT POSITIVE ➤ When analyzing scale data calculate the percentage of respondents calculate the percentage that chose the top 2 boxes - 5 point scale = top 2 boxes - 7 point scale = top 3 boxes It’s really bad It’s kind of bad Neutral It’s pretty good I’m having a great time Responses 1 24 19 59 78 % of Total 0% 13% 10% 33% 43% Top 2 Boxes 76%
- 87. @Curio_Research BUT WAIT, THERE’S MORE! 87 https://www.flickr.com/photos/tetsumo/3802078924
- 88. @Curio_Research FURTHER READING ➤Internet, Mail, and Mixed-Mode Surveys - The Tailored Design Method - Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian ➤Questionnaire Design - Ian Brace ➤People Are Not Robots - Annie Pettit
- 89. @Curio_Research ONLINE SOURCES ➤ Twitter - #NewMR and @NewMR_News - Annie Pettit @LoveStats - Jeffrey Henning @JHenning - Ray Poynter @RayPoynter ➤ Websites - Pew Research Methods pewresearch.org/methods/ - Me curioresearch.net/portfolio/