Most surveys use data quality techniques such as red herrings, speeding, straightlining, and jibberish monitoring to determine which completes represent good responders or poor responders. This paper, presented at the 2013 NewMR online festival, demonstrates which techniques properly identify good and poor responders.

1. Angels and Demons:
Are you penalizing the right people?
Annie Pettit, Chief Research Officer
Peanut Labs
Canada
Sponsor logos will go here
Festival of NewMR 2013 – Main Stage

2. Raise your hand if right now you
are…
2
1/14/2014
Festival of NewMR 2013 – Main Stage

3. Lazy Bum Larry
Festival of NewMR 2013 – Main Stage
3

4. Good Faith Gail
Festival of NewMR 2013 – Main Stage

5. Method to our madness
• 8000 people answered a survey
• Every survey measured against each of 30 data quality
measures
• Calculated number of errors for each person (from 0 to 30)
• Grouped people into Angels and Demons based on number of
“fails”
5
Festival of NewMR 2013 – Main Stage

6. Are responders angels or
demons?
One or
more
errors
23
%
77
%
Zero errors
Festival of NewMR 2013 – Main Stage

7. 1
1 Red Herring
• 5.8% of angels chose 1
red herring
• 14.5% of demons
chose 1 red herring
(
Why?
• Sounds like a brand they know
• It IS a local brand they know
• Babies biting ankles
Festival of NewMR 2013 – Main Stage

8. 2 Red Herrings
• 1.4% of angels chose 2
red herrings
• 40.2% of demons
chose 2 red herrings
Why?
• They didn’t notice it. Twice.
• They aren’t reading it. At all.
• They need to finish it. NOW!
Festival of NewMR 2013 – Main Stage

9. 2
Tier 1 Speeding
(<50% of the mean)
• 8.4% of angels speed
• 40.7% of demons
speed
(
Why?
• They’re computer literate.
• They read well. Really well.
• They know surveys. Really well.
Festival of NewMR 2013 – Main Stage

10. Tier 2 Speeding
(Fastest 1.5%)
• 0% of angels speed
• 28% of demons speed
Why?
• They’re good at clicking.
• They aren’t reading. At all.
• They know surveys. Really well.
Festival of NewMR 2013 – Main Stage

11. 3
Straightlining
• 14.2% of angels
straightline
• 60.7% of demons
straightline
Festival of NewMR 2013 – Main Stage
(
Why?
• Every item was phrased the same way
• It’s their true opinion

12. Good vs. Bad Grids
What is your level of agreement with each of the following
statement? This product:
Strongly
Agree
•
•
•
•
•
•
Somewhat
Agree
Neutral
Tastes great
Looks appetizing
Smells nice
Well packaged
Interesting logo
Pretty colours
Somewhat
Disagree
•
•
•
•
•
•
Strongly
Disagree
Tastes terrible
Looks appetizing
Smells strange
Well packaged
Odd logo
Pretty colors
Festival of NewMR 2013 – Main Stage

13. Straightlining on A Good Grid
• 2.8% of angels
straightline
• 58.4% of demons
straightline
Why?
• They aren’t reading. At all.
• They know surveys. Really well.
Festival of NewMR 2013 – Main Stage

14. 4
Rank From 1 through 6
• 7.9% of angels rank
incorrectly
• 50% of demons rank
incorrectly
– Choose same number twice
– Choose numbers outside the
limits
– Straightline
Why?
• They aren’t reading. At all.
• They click really well.
Festival of NewMR 2013 – Main Stage

15. 5
Contradictions
• 28.4% of angels
contradict themselves
only once
• 39.5% of angels
contradict themselves
twice
• 7.9% of demons
contradict themselves
only once
• 73.8% of demons
contradict themselves
twice
(
Why?
• Because it’s a valid answer
• Babies biting ankles
Festival of NewMR 2013 – Main Stage

16. 6
Verbatim Length
• 0.9% of angels share
less than 10 characters
• 56.5% of demons
share less than 10
characters
Why?
• asdf
• aaaa
• don’t know
Festival of NewMR 2013 – Main Stage

17. 7
Non-traditional behaviours
• 12.5% of angels buy
non-grocery items at
grocery stores
• 37.4% of demons buy
non-grocery items at
grocery stores
(
Why?
• Because it’s a valid answer
• Babies biting ankles
Festival of NewMR 2013 – Main Stage

18. 8
High Incidence Awareness
• 3.5% of angels knew
<3 of 8 big name
brands
• 48.1% of demons knew
<3 of 8 big name
brands
Why?
• Immigrants
• Non consumers
• Non shoppers
18
Festival of NewMR 2013 – Main Stage

19. Conclusions
Angels don’t exist
Perfection doesn’t exist
It’s not fair to expect
perfection from responders
acting in good faith
Festival of NewMR 2013 – Main Stage

20. Conclusions
To Penalize Angels:
• Use 1 red herring
• Use only positive grids
• Measure only one or two
contradictions
• Don’t permit 1 or 2 mistakes
To Penalize Demons:
• Use 2 red herrings
• Use mixed grids
• Use a rank question
• Measure short verbatims
• Focus on multiple mistakes
Festival of NewMR 2013 – Main Stage

21. Thank you!
Contact me with any questions.
Annie Pettit, PhD
Chief Research Officer
Peanut Labs
annie.pettit@peanutlabs.com
www.peanutlabsmedia.com
416.273.9395
twitter.com/LoveStats
facebook.com/AnniePettit
ca.linkedin.com/in/AnniePettit
Festival of NewMR 2013 – Main Stage

22. From sampling to surveys to listening, you need to get your research
done quickly and accurately. Peanut Labs tools help you build your
own surveys and select targeted sample from our profiled panel.
And, our Evolisten social media listening tool helps you gain a
broader understanding of your consumers.
www.peanutlabs.com
Use our DIY, permission based, profiled sampling
tool to select and launch targeted sample
Use our DIY survey design tool to craft your
surveys
Obtain quick poll results with our DIY targeted
polling tool. Answers in mere minutes!
Listen to millions of social media opinions about
people, places, brands, and more.
Festival of NewMR 2013 – Main Stage