Angels and Demons: Is your survey penalizing the right people?

328 views

Published on

Most surveys use data quality techniques such as red herrings, speeding, straightlining, and jibberish monitoring to determine which completes represent good responders or poor responders. This paper, presented at the 2013 NewMR online festival, demonstrates which techniques properly identify good and poor responders.

Published in: Marketing, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
328
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Angels and Demons: Is your survey penalizing the right people?

  1. 1. Angels and Demons: Are you penalizing the right people? Annie Pettit, Chief Research Officer Peanut Labs Canada Sponsor logos will go here Festival of NewMR 2013 – Main Stage
  2. 2. Raise your hand if right now you are… 2 1/14/2014 Festival of NewMR 2013 – Main Stage
  3. 3. Lazy Bum Larry Festival of NewMR 2013 – Main Stage 3
  4. 4. Good Faith Gail Festival of NewMR 2013 – Main Stage
  5. 5. Method to our madness • 8000 people answered a survey • Every survey measured against each of 30 data quality measures • Calculated number of errors for each person (from 0 to 30) • Grouped people into Angels and Demons based on number of “fails” 5 Festival of NewMR 2013 – Main Stage
  6. 6. Are responders angels or demons? One or more errors 23 % 77 % Zero errors Festival of NewMR 2013 – Main Stage
  7. 7. 1 1 Red Herring • 5.8% of angels chose 1 red herring • 14.5% of demons chose 1 red herring ( Why? • Sounds like a brand they know • It IS a local brand they know • Babies biting ankles Festival of NewMR 2013 – Main Stage
  8. 8. 2 Red Herrings • 1.4% of angels chose 2 red herrings • 40.2% of demons chose 2 red herrings Why? • They didn’t notice it. Twice. • They aren’t reading it. At all. • They need to finish it. NOW! Festival of NewMR 2013 – Main Stage
  9. 9. 2 Tier 1 Speeding (<50% of the mean) • 8.4% of angels speed • 40.7% of demons speed ( Why? • They’re computer literate. • They read well. Really well. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  10. 10. Tier 2 Speeding (Fastest 1.5%) • 0% of angels speed • 28% of demons speed Why? • They’re good at clicking. • They aren’t reading. At all. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  11. 11. 3 Straightlining • 14.2% of angels straightline • 60.7% of demons straightline Festival of NewMR 2013 – Main Stage ( Why? • Every item was phrased the same way • It’s their true opinion
  12. 12. Good vs. Bad Grids What is your level of agreement with each of the following statement? This product: Strongly Agree • • • • • • Somewhat Agree Neutral Tastes great Looks appetizing Smells nice Well packaged Interesting logo Pretty colours Somewhat Disagree • • • • • • Strongly Disagree Tastes terrible Looks appetizing Smells strange Well packaged Odd logo Pretty colors Festival of NewMR 2013 – Main Stage
  13. 13. Straightlining on A Good Grid • 2.8% of angels straightline • 58.4% of demons straightline Why? • They aren’t reading. At all. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  14. 14. 4 Rank From 1 through 6 • 7.9% of angels rank incorrectly • 50% of demons rank incorrectly – Choose same number twice – Choose numbers outside the limits – Straightline Why? • They aren’t reading. At all. • They click really well. Festival of NewMR 2013 – Main Stage
  15. 15. 5 Contradictions • 28.4% of angels contradict themselves only once • 39.5% of angels contradict themselves twice • 7.9% of demons contradict themselves only once • 73.8% of demons contradict themselves twice ( Why? • Because it’s a valid answer • Babies biting ankles Festival of NewMR 2013 – Main Stage
  16. 16. 6 Verbatim Length • 0.9% of angels share less than 10 characters • 56.5% of demons share less than 10 characters Why? • asdf • aaaa • don’t know Festival of NewMR 2013 – Main Stage
  17. 17. 7 Non-traditional behaviours • 12.5% of angels buy non-grocery items at grocery stores • 37.4% of demons buy non-grocery items at grocery stores ( Why? • Because it’s a valid answer • Babies biting ankles Festival of NewMR 2013 – Main Stage
  18. 18. 8 High Incidence Awareness • 3.5% of angels knew <3 of 8 big name brands • 48.1% of demons knew <3 of 8 big name brands Why? • Immigrants • Non consumers • Non shoppers 18 Festival of NewMR 2013 – Main Stage
  19. 19. Conclusions Angels don’t exist Perfection doesn’t exist It’s not fair to expect perfection from responders acting in good faith Festival of NewMR 2013 – Main Stage
  20. 20. Conclusions To Penalize Angels: • Use 1 red herring • Use only positive grids • Measure only one or two contradictions • Don’t permit 1 or 2 mistakes To Penalize Demons: • Use 2 red herrings • Use mixed grids • Use a rank question • Measure short verbatims • Focus on multiple mistakes Festival of NewMR 2013 – Main Stage
  21. 21. Thank you! Contact me with any questions. Annie Pettit, PhD Chief Research Officer Peanut Labs annie.pettit@peanutlabs.com www.peanutlabsmedia.com 416.273.9395 twitter.com/LoveStats facebook.com/AnniePettit ca.linkedin.com/in/AnniePettit Festival of NewMR 2013 – Main Stage
  22. 22. From sampling to surveys to listening, you need to get your research done quickly and accurately. Peanut Labs tools help you build your own surveys and select targeted sample from our profiled panel. And, our Evolisten social media listening tool helps you gain a broader understanding of your consumers. www.peanutlabs.com Use our DIY, permission based, profiled sampling tool to select and launch targeted sample Use our DIY survey design tool to craft your surveys Obtain quick poll results with our DIY targeted polling tool. Answers in mere minutes! Listen to millions of social media opinions about people, places, brands, and more. Festival of NewMR 2013 – Main Stage

×