For the most part, people who answer marketing research surveys want and try to do a good job. However, sometimes respondents want to get through a survey as quick as possible in order to earn the incentive and move to the next task.
- Learn the various types of data quality questions you can use, beyond speeding and straightlining.
- How to fit them into your questionnaire with minimal impact on responders.
- And most importantly, how to use the data quality questions effectively so that you don't accidentally exclude data from honest respondents.
Best VIP Call Girls Noida Sector 39 Call Me: 8448380779
How to seamlessly integrate data quality measures into your questionnaire
1. Please tweet! @PeanutLabsMedia @LoveStats
How to seamlessly
integrate data quality
measures into your
questionnaire
Presented by Annie Pettit
Chief Research Officer at
Peanut Labs
Please tweet! @LoveStats @PeanutLabsMedia
2. Please tweet! @PeanutLabsMedia @LoveStats
How to distinguish between these people?
But,
incentive getting,
satisficing,
cheaters make lots
of mistakes
When answering
surveys, good
honest people will
naturally make a
mistake or two
3. Please tweet! @PeanutLabsMedia @LoveStats
Multi-Select Questions
Red Herrings
High/Low Incidence
Over/Under Clicking
Following Instructions
Sed turpis elit, venenatis ut
porttitor sed, venenatis in arcu.
Nam consequat leo ut lorem
viverra, ac commodo ex
ullamcorper?
Praesent
Maximus
Ullamcorper
Faucibus
Lacinia
Tincidunt
Efficitur
4. Please tweet! @PeanutLabsMedia @LoveStats
Red Herrings
• Question Type: Multi-selects, Single-selects,
Rating scales
• Concern: Respondents may have misread or
confused a name. It may be a real name!
• Cheaters: Incentive Getting. Responder is
providing the minimum amount of information
required to proceed.
• Application: Choose at least TWO fake names
and Google them to ensure they are extremely
low incidence
• Scoring: Flag answers in sets of two
Which of the following stores
have you visited in the past 3
months? (Please select all that
apply)
Abercrombie & Fitch
Aéropostale
American Apparel
American Eagle Outfitters
Anthropologie
Bellamis New York
Bloomingdale's
Brooks Brothers
Club Forenzo
Coldwater Creek
Dillard's
DKNY
Other
5. Please tweet! @PeanutLabsMedia @LoveStats
High Incidence
• Question Type: Multi-selects
• Concern: Respondents are tired or bored.
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Works best on longer lists. Include
at least a few answers that ought to be selected
by everyone
• Scoring: Flag people with the fewest clicks for
very common responses.
Which of the following activities
have you participated in during
the last 3 months? (Please select
all that apply)
Attended a sporting event
Attended a music event
Exercised
Listened to music
Used the internet
Visited a community center
Visited a library
Watched TV
Went to a grocery store
Went to the movies
Went to school
Went to work
6. Please tweet! @PeanutLabsMedia @LoveStats
Low Incidence
• Question Type: Multi-selects
• Concern: Respondents are thinking of people
other than themselves
• Cheaters: Incentive getting. They may be trying
to avoid screening out so they will qualify for an
incentive.
• Application: Incorporate at least 3 extremely low
incidence options. Rare doesn't happen in pairs.
• Scoring: Flag people who select two or more rare
answers.
Which of the following ailments
do you have? (Please select all
that apply)
Acid Reflux
Acidosis
Adrenal Disorders
AIDS
Alzheimer's Disease
Amyloidosis
Anemia
Anorexia Nervosa
Arteriosclerosis
Autism
Blood Pressure (High)
Bronchitis
Cancer
7. Please tweet! @PeanutLabsMedia @LoveStats
Underclicking
• Question Type: Multi-selects
• Concern: Respondents are tired or bored.
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Works best on longer lists. Include
at least a few answers that ought to be selected
by everyone.
• Scoring: Flag people with the fewest clicks across
the whole question.
Which of the following stores
have you visited in the past 3
months? (Please select all that
apply)
Albertsons
Big Lots
Costco
CVS
Dollar General
Family Dollar
Kroger
Publix
Safeway
Target
Walgreen
Walmart
8. Please tweet! @PeanutLabsMedia @LoveStats
Overclicking
• Question Type: Multi-selects
• Concern: They may be thinking of their
household or family members.
• Cheaters: Incentive getting. They may be trying
to qualify for the incentive.
• Application: Works best on longer questions.
Ensure there are options that don’t really go
together.
• Scoring: Flag people with the most clicks across
the whole question.
Which of the following stores
have you visited in the past 3
months? (Please select all that
apply)
Abercrombie & Fitch
Aéropostale
American Apparel
American Eagle Outfitters
Anthropologie
Barneys New York
Bloomingdale's
Brooks Brothers
Club Monaco
Coldwater Creek
Dillard's
DKNY
Eddie Bauer
9. Please tweet! @PeanutLabsMedia @LoveStats
Following Instructions
• Question Type: Unvalidated multi-selects
• Concern: Respondents may have misunderstood
the task, or want to provide a ‘more accurate’
answer
• Cheaters: Satisficing. The responder wants to
finish and get their incentive.
• Application: Word questions so it makes sense to
ask for exactly 2 or 3 choices. Don’t validate!
• Scoring: Flag anyone who chooses more or fewer
responses than requested.
Which three of the following
stores did you visit most often?
(Please select only 3)
Abercrombie & Fitch
Aéropostale
American Apparel
American Eagle Outfitters
Anthropologie
Bellamis New York
Bloomingdale's
Brooks Brothers
Club Forenzo
Coldwater Creek
Dillard's
DKNY
Other
10. Please tweet! @PeanutLabsMedia @LoveStats
Don’t Know
• Question Type: Multi-selects, Single-selects
• Concern: They probably do know if they would
think about it a little bit
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Follow good survey design practice
of including DK wherever it is possible
• Scoring: Count how many times they select
‘Don’t Know’ across the survey
Which of the following stores
have you visited in the past 3
months? (Please select all that
apply)
Abercrombie & Fitch
Aéropostale
American Apparel
American Eagle Outfitters
Anthropologie
Barneys New York
Bloomingdale's
Brooks Brothers
Club Monaco
Coldwater Creek
Dillard's
DKNY
Don’t know
11. Please tweet! @PeanutLabsMedia @LoveStats
Grid Rating Questions
Straightlining
Sed turpis elit, venenatis ut
porttitor sed, venenatis in
arcu?
o o o o o
o o o o o
o o o o o
o o o o o
Praesent maximus
Ullamcorper faucibus
Lacinia tincidunt
Efficitur aliquam
12. Please tweet! @PeanutLabsMedia @LoveStats
Straightlining
• Question Type: Multi-select ratings
• Concern: Respondents may have only glanced at
the items, not noticed reverse keyed items
• Cheaters: Satisficing. Get in and get out!
• Application: Be sure to include positively and
negatively keyed items.
• Scoring: Measure every grid for patterns -
vertical, diagonal, repetitive. Each straightline
generates a flag.
What is your opinion about
each of these statements?
o o o o o
o o o o o
o o o o o
o o o o o
Type A
X
X
X
X
X
Type B
X
X
X
X
X
Type C
X
X
X
Tastes good
Smells tempting
Feels nice
Looks pretty
13. Please tweet! @PeanutLabsMedia @LoveStats
“Select the third answer”
• Question Type: Multi-Select Rating
• Concern: It’s confusing.
It creates suspicion.
SQUIRREL!
• Cheaters: Satisficing. Answer quickly and get
your incentive.
• Application: Ask responders to choose a specific
answer, “Select Somewhat Disagree in this row”
• Scoring: Don’t! This is a terrible measure!
What is your opinion about each of
these statements?
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
Tastes bland
Would recommend
Select Disagree Somewhat
Box is ugly
Smells delicious
Agree
Strongly
Agree
Somewhat
Neutral
Disagree
Somewhat
Disagree
Strongly
Squirrel!
14. Please tweet! @PeanutLabsMedia @LoveStats
Open-End Verbatims
Pellentesque quis aliquet felis,
sit amet ultricies erat. Etiam vel
metus augue.
15. Please tweet! @PeanutLabsMedia @LoveStats
dunno NA
I lik that their
crunchy
? none
1) If Congress cannot do its job - pass a
Budget EVERY year (not a CR), they should
NOT get paid AND should NOT get ANY
benefits; 2) An ex government official should
be prohibited form becoming a lobbyist within
2 years of leaving office; 3) Contracting out
should NOT be done if NO REAL costs savings
can be realized; and 4) People making over
$250, 000 should pay a higher percentage of
income tax - minimum of 30%
Verbatim Length
• Question Type: Long verbatims
• Concern: It’s difficult to introspect and self-
evaluate
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Create at least one question that
requires a long answer. “Describe three reasons
why…” (And benefit from the tidbits of gold!)
• Scoring: Flag any response under 10 characters.
16. Please tweet! @PeanutLabsMedia @LoveStats
asdf
You’re stupid
dumbdumbdumb
ass
hjsrthetutjytnse5hyyjn,jiuuyj
ydtpysrgawe.btzergrsebearh
uyllkjhhgcfddwad
Verbatim Gibberish
• Question Type: Short verbatims, Long verbatims
• Concern: It’s difficult to introspect and self-evaluate
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Ensure there is at least one open-end
question for a text response.
• Scoring: Look for no spaces, improbable letter
combinations (hh, kk, yy, hj, bt, js). Assign points
based on the severity (e.g., crude words=3, ‘ass’=2,
‘asdf’=1
18. Please tweet! @PeanutLabsMedia @LoveStats
Contradictions
• Question Type: Any two questions
• Concern: Respondents may misread one of the
questions. The question was poorly worded.
• Cheaters: Satisficing.
• Application: Ensure the questions are far apart.
Focus on related, contextual questions, not
identical questions. Allow improbable answers.
• Scoring: Flag cases where the answers don’t
match
Q3. What pets do you have in
your household?
Dog
Cat
Fish
Bird
Small mammal
Other
None
Q19. How often does your household
buy pet food?
o At least once per week
o About once per month
o Several times per year
o About once per year
o Less often
o Never
19. Please tweet! @PeanutLabsMedia @LoveStats
Rank Orders
• Question Type: Unvalidated fill-in-the-blank rank
orders
• Concern: They didn’t read carefully, They weren’t
thinking carefully. They didn’t understand what they
were being asked for.
• Cheaters: Satisficing. Responder is incentive getting.
• Application: Create a question with 5 to 8 options.
Don’t validate!
• Scoring: Flag cases with any numbers less than or
greater than the minimum/maximum, or if there
are any duplicate numbers
Please rank the importance of
these store features from 1 to 5.
1 = Most Important
5 = Least Important
(Please use each number only once.)
Price
Location
Style
Service
Selection
20. Please tweet! @PeanutLabsMedia @LoveStats
Sums
• Question Type: Unvalidated fill-in-the-blank
sums
• Concern: They didn’t understand the question.
Their math doesn’t reflect reality. They aren’t
good with math.
• Cheaters: Satisficing.
• Application: Create a question with 4 to 6
options so the math is reasonable. Don’t
validate!
• Scoring: Flag any response that doesn’t add to
100.
What percentage of your
monthly income is spent in these
areas? (Please make sure the
numbers add up to 100%.)
Housing
Food
Utilities (e.g., heat, electricity)
Transportation
Entertainment
21. Please tweet! @PeanutLabsMedia @LoveStats
Speeding
• Question Type: The entire survey
• Concern: Some people are fast readers. Some
have a lot of survey experience. Some got all the
skips and short paths.
• Cheaters: Satisficing. Incentive getting.
• Application: Wait until all the completes are in.
Fastest 5% is ideal. Fastest 2% is impossible!
• Scoring: Give one point to the fastest 5%. Give
two points to the fastest 2%.
Seconds Cumulative
Frequency
0 - 40 0
41 - 50 0
51 - 60 0.2
61 - 70 0.5
71 - 80 0.9
81 - 90 1.5
91 - 100 2.4
101 - 110 3.5
111 - 120 4.7
121 - 130 6.4
131 - 140 8.3
141 - 150 10.5
151 - 160 13.6
161 - 170 16.6
171 - 180 19.7
181 - 190 23.6
191 - 200 27.6
23. Please tweet! @PeanutLabsMedia @LoveStats
Question Level Cut-Scores
Red
Herring
Percent of
people failing
0 78.9
1 15.4
2 2.9
3 2.8
Over
Clicking
Percent of
people failing
0 76.3
1 13.5
2 9.4
3 0.8
Rank
Order
Percent of
people failing
0 77.7
1 9.6
2 2.9
3 9.8
Ideal Scenario Too Few Fail Too Many Fail
•Check and improve
the scoring
•Use it anyways
•Don’t use it at all
•Check and improve
the scoring
•Use it anyways
• Anything
between 3
and 7 is ideal
24. Please tweet! @PeanutLabsMedia @LoveStats
Question Level Scores of Individual People
Person 1
(Regular Person)
Person 2
(Satisficer)
Person 3
(Incentive Getter)
Overclick 0 0 1
Underclick 0 1 0
Speeding 0 2 0
Red Herring 1 1 2
Low Incidence 0 0 1
High Incidence 0 1 0
Rank Order 0 0 0
Sums 0 0 0
TOTAL 1 5 4
This is not a problem THESE are problems!
26. Please tweet! @PeanutLabsMedia @LoveStats
Tips
• Try to incorporate at least 4 different measures into every
survey
• Don’t cram too many tests into one question
– You COULD put overclicking, underclicking, high incidence, low
incidence, and red herrings in the same question. But if that’s
the only question a good respondent had trouble with, you’ve
excluded good data.
– Use each question a maximum of two times
• Spread the tests throughout the survey - beginning, middle,
and end.
• This doesn’t have to be a lot of work. Create SPSS/Excel syntax
that you can use repeatedly.
• Remember that human beings make mistakes. You makes lots
of mistakes everyday. Don’t expect other people to be better
than you.
27. Please tweet! @PeanutLabsMedia @LoveStats
Thank you!
Annie Pettit
Chief Research Officer
Peanut Labs
annie@peanutlabs.com
twitter.com/LoveStats
ca.linkedin.com/in/anniepettit/
Questions about our Sample Services?
Jonathan Cheriff
Director of Sales & Marketing
jonathan.cheriff@peanutlabs.com
twitter.com/paperbackdad
www.linkedin.com/in/jonathancheriff