0
Angels and Demons:
Are you penalizing the right people?
Annie Pettit, Chief Research Officer
Peanut Labs
Canada

Sponsor l...
Raise your hand if right now you
are…

2

1/14/2014

Festival of NewMR 2013 – Main Stage
Lazy Bum Larry

Festival of NewMR 2013 – Main Stage

3
Good Faith Gail

Festival of NewMR 2013 – Main Stage
Method to our madness
• 8000 people answered a survey
• Every survey measured against each of 30 data quality
measures
• C...
Are responders angels or
demons?
One or
more
errors

23
%
77
%

Zero errors

Festival of NewMR 2013 – Main Stage
1

1 Red Herring

• 5.8% of angels chose 1
red herring

• 14.5% of demons
chose 1 red herring

(

Why?
• Sounds like a bra...
2 Red Herrings
• 1.4% of angels chose 2
red herrings

• 40.2% of demons
chose 2 red herrings

Why?
• They didn’t notice it...
2

Tier 1 Speeding
(<50% of the mean)

• 8.4% of angels speed

• 40.7% of demons
speed

(

Why?
• They’re computer literat...
Tier 2 Speeding
(Fastest 1.5%)
• 0% of angels speed

• 28% of demons speed

Why?
• They’re good at clicking.
• They aren’t...
3

Straightlining

• 14.2% of angels
straightline

• 60.7% of demons
straightline

Festival of NewMR 2013 – Main Stage

(
...
Good vs. Bad Grids
What is your level of agreement with each of the following
statement? This product:
Strongly
Agree

•
•...
Straightlining on A Good Grid
• 2.8% of angels
straightline

• 58.4% of demons
straightline

Why?
• They aren’t reading. A...
4

Rank From 1 through 6

• 7.9% of angels rank
incorrectly

• 50% of demons rank
incorrectly
– Choose same number twice
–...
5

Contradictions

• 28.4% of angels
contradict themselves
only once
• 39.5% of angels
contradict themselves
twice

• 7.9%...
6

Verbatim Length

• 0.9% of angels share
less than 10 characters

• 56.5% of demons
share less than 10
characters

Why?
...
7

Non-traditional behaviours

• 12.5% of angels buy
non-grocery items at
grocery stores

• 37.4% of demons buy
non-grocer...
8

High Incidence Awareness

• 3.5% of angels knew
<3 of 8 big name
brands

• 48.1% of demons knew
<3 of 8 big name
brands...
Conclusions
Angels don’t exist
Perfection doesn’t exist
It’s not fair to expect
perfection from responders
acting in good ...
Conclusions
To Penalize Angels:
• Use 1 red herring
• Use only positive grids
• Measure only one or two
contradictions
• D...
Thank you!
Contact me with any questions.

Annie Pettit, PhD
Chief Research Officer
Peanut Labs
annie.pettit@peanutlabs.co...
From sampling to surveys to listening, you need to get your research
done quickly and accurately. Peanut Labs tools help y...
Upcoming SlideShare
Loading in...5
×

Angels and Demons: Is your survey penalizing the right people?

119

Published on

Most surveys use data quality techniques such as red herrings, speeding, straightlining, and jibberish monitoring to determine which completes represent good responders or poor responders. This paper, presented at the 2013 NewMR online festival, demonstrates which techniques properly identify good and poor responders.

Published in: Marketing, Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
119
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Angels and Demons: Is your survey penalizing the right people?"

  1. 1. Angels and Demons: Are you penalizing the right people? Annie Pettit, Chief Research Officer Peanut Labs Canada Sponsor logos will go here Festival of NewMR 2013 – Main Stage
  2. 2. Raise your hand if right now you are… 2 1/14/2014 Festival of NewMR 2013 – Main Stage
  3. 3. Lazy Bum Larry Festival of NewMR 2013 – Main Stage 3
  4. 4. Good Faith Gail Festival of NewMR 2013 – Main Stage
  5. 5. Method to our madness • 8000 people answered a survey • Every survey measured against each of 30 data quality measures • Calculated number of errors for each person (from 0 to 30) • Grouped people into Angels and Demons based on number of “fails” 5 Festival of NewMR 2013 – Main Stage
  6. 6. Are responders angels or demons? One or more errors 23 % 77 % Zero errors Festival of NewMR 2013 – Main Stage
  7. 7. 1 1 Red Herring • 5.8% of angels chose 1 red herring • 14.5% of demons chose 1 red herring ( Why? • Sounds like a brand they know • It IS a local brand they know • Babies biting ankles Festival of NewMR 2013 – Main Stage
  8. 8. 2 Red Herrings • 1.4% of angels chose 2 red herrings • 40.2% of demons chose 2 red herrings Why? • They didn’t notice it. Twice. • They aren’t reading it. At all. • They need to finish it. NOW! Festival of NewMR 2013 – Main Stage
  9. 9. 2 Tier 1 Speeding (<50% of the mean) • 8.4% of angels speed • 40.7% of demons speed ( Why? • They’re computer literate. • They read well. Really well. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  10. 10. Tier 2 Speeding (Fastest 1.5%) • 0% of angels speed • 28% of demons speed Why? • They’re good at clicking. • They aren’t reading. At all. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  11. 11. 3 Straightlining • 14.2% of angels straightline • 60.7% of demons straightline Festival of NewMR 2013 – Main Stage ( Why? • Every item was phrased the same way • It’s their true opinion
  12. 12. Good vs. Bad Grids What is your level of agreement with each of the following statement? This product: Strongly Agree • • • • • • Somewhat Agree Neutral Tastes great Looks appetizing Smells nice Well packaged Interesting logo Pretty colours Somewhat Disagree • • • • • • Strongly Disagree Tastes terrible Looks appetizing Smells strange Well packaged Odd logo Pretty colors Festival of NewMR 2013 – Main Stage
  13. 13. Straightlining on A Good Grid • 2.8% of angels straightline • 58.4% of demons straightline Why? • They aren’t reading. At all. • They know surveys. Really well. Festival of NewMR 2013 – Main Stage
  14. 14. 4 Rank From 1 through 6 • 7.9% of angels rank incorrectly • 50% of demons rank incorrectly – Choose same number twice – Choose numbers outside the limits – Straightline Why? • They aren’t reading. At all. • They click really well. Festival of NewMR 2013 – Main Stage
  15. 15. 5 Contradictions • 28.4% of angels contradict themselves only once • 39.5% of angels contradict themselves twice • 7.9% of demons contradict themselves only once • 73.8% of demons contradict themselves twice ( Why? • Because it’s a valid answer • Babies biting ankles Festival of NewMR 2013 – Main Stage
  16. 16. 6 Verbatim Length • 0.9% of angels share less than 10 characters • 56.5% of demons share less than 10 characters Why? • asdf • aaaa • don’t know Festival of NewMR 2013 – Main Stage
  17. 17. 7 Non-traditional behaviours • 12.5% of angels buy non-grocery items at grocery stores • 37.4% of demons buy non-grocery items at grocery stores ( Why? • Because it’s a valid answer • Babies biting ankles Festival of NewMR 2013 – Main Stage
  18. 18. 8 High Incidence Awareness • 3.5% of angels knew <3 of 8 big name brands • 48.1% of demons knew <3 of 8 big name brands Why? • Immigrants • Non consumers • Non shoppers 18 Festival of NewMR 2013 – Main Stage
  19. 19. Conclusions Angels don’t exist Perfection doesn’t exist It’s not fair to expect perfection from responders acting in good faith Festival of NewMR 2013 – Main Stage
  20. 20. Conclusions To Penalize Angels: • Use 1 red herring • Use only positive grids • Measure only one or two contradictions • Don’t permit 1 or 2 mistakes To Penalize Demons: • Use 2 red herrings • Use mixed grids • Use a rank question • Measure short verbatims • Focus on multiple mistakes Festival of NewMR 2013 – Main Stage
  21. 21. Thank you! Contact me with any questions. Annie Pettit, PhD Chief Research Officer Peanut Labs annie.pettit@peanutlabs.com www.peanutlabsmedia.com 416.273.9395 twitter.com/LoveStats facebook.com/AnniePettit ca.linkedin.com/in/AnniePettit Festival of NewMR 2013 – Main Stage
  22. 22. From sampling to surveys to listening, you need to get your research done quickly and accurately. Peanut Labs tools help you build your own surveys and select targeted sample from our profiled panel. And, our Evolisten social media listening tool helps you gain a broader understanding of your consumers. www.peanutlabs.com Use our DIY, permission based, profiled sampling tool to select and launch targeted sample Use our DIY survey design tool to craft your surveys Obtain quick poll results with our DIY targeted polling tool. Answers in mere minutes! Listen to millions of social media opinions about people, places, brands, and more. Festival of NewMR 2013 – Main Stage
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×