#CRC2014 @LoveStats 
A Tale of Two Surveys 
Presented by 
Annie Pettit, Chief Research Officer 
Peanut Labs is a 
company
#CRC2014 @LoveStats 
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to heaven, we were all going direct the other way—in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.
#CRC2014 @LoveStats 
Based on any advertising, such as TV commercials, roadside billboards, banners on internet websites, or other similar forms of advertising, please list in the textbox below the products or types of products that have you yourself have seen publicly advertised between January 1, 2013 and December 31, 2013. (Please type your answer in the box below.)
#CRC2014 @LoveStats 
Based on any advertising, such as TV commercials, roadside billboards, banners on internet websites, or other similar forms of advertising, please list in the textbox below the products or types of products that have you yourself have seen publicly advertised between January 1, 2013 and December 31, 2013. (Please type your answer in the box below.) 
What products did you see advertised last year? 
What products did you see advertised last week?
#CRC2014 @LoveStats 
Survey method 
•Test n=530 
•Control n=530 
•US Census Rep 
•Non-branded computer and video gaming survey 
•20 minute survey
#CRC2014 @LoveStats 
Traditional 
Humanized 
What is your favorite video/computer game? 
If you could only play one video game for the rest of your life, what game would you choose? 
Have you ever lined up at a store so you could be among the first to purchase a new video/computer game? 
Are you one of those people who lines up at stores to be the first to buy a new game? 
How much do you agree or disagree with these statements about gaming? 
Love 'em or hate 'em, what do you think about video games? 
What is your opinion about each of the following television shows? 
Awesome or terrible, how do you rate these TV shows? 
Is there any other information you would like to share with us regarding your opinions and use of video/computer games? 
Sharing is caring! Is there anything else you'd like to share about gaming?
#CRC2014 @LoveStats 
Data Quality
#CRC2014 @LoveStats 
Speeding 
•Failed if the survey was completed in the slowest 5% of completion times 
0 
1 
2 
3 
4 
5 
Traditional Survey (%) 
Humanized Survey (%) 
Speeding
#CRC2014 @LoveStats 
Straightlining 
•Failed if data demonstrated straightlining to a half positive, half negative 8 question grid 
0 
5 
10 
15 
20 
Traditional Survey (%) 
Humanized Survey (%) 
Straightlining
#CRC2014 @LoveStats 
Open Ends 
•Failed if less than 10 characters were provided in response to a request for “3 reasons why…” 
0 
5 
10 
Traditional Survey (%) 
Humanized Survey (%) 
Open Ends
#CRC2014 @LoveStats 
Red Herring 
•Failed if two fake computer games were selected 
0 
1 
2 
3 
4 
5 
Traditional Survey (%) 
Humanized Survey (%) 
Red Herring
#CRC2014 @LoveStats 
Survey Evaluation 
1.0 
2.0 
3.0 
4.0 
5.0 
Too long 
Too complex 
Fun to answer 
Would 
recommend to 
my friends 
Traditional Wording 
Fun Wording 
Agree 
Disagree
#CRC2014 @LoveStats 
Responder Opinions 
Traditional Survey 
Humanized Survey 
It was short and nice:) 
nice short survey not enough points though 
I found this to be an interesting survey. Thank you. 
I love surveys and quizzes, and this one was very interesting, because i like sharing things about myself. 
the survey should be worth more points considering how long the survey was 
very original method of conducting a survey! 
NA 
This was one of the Best surveys I have done!!! 
No 
loved it.. funny and quick 
Good 
"May the survey FORCE be with you." Was a nice hilariously awesome thing to see. I laughed a lot. o3o 
This survey was enjoyable. 
this is an awesome survey. I thoroughly enjoyed myself while taking it. 
i dont know. 
good job, sir! If i had a fedora, i'd tip it in your direction. 
It was good survey 
I laughed out loud the way some questions were phrased, I wanted to say this is the most refreshing survey ive taken I had a lot of fun
#CRC2014 @LoveStats 
Survey Results
#CRC2014 @LoveStats 
Purchase Intent 
0% 
10% 
20% 
30% 
40% 
Not at all 
Somewhat 
Very 
Extremely 
Raw - Traditional 
Raw - Humanized
#CRC2014 @LoveStats 
What is a fair price? 
0% 
10% 
20% 
30% 
$0 
$1 to $14 
$15 to $19 
$20 to $29 
$30 to $39 
$40 to $49 
$50 or more 
Raw - Traditional 
Raw - Humanized
#CRC2014 @LoveStats 
Rating Ads 
0.0% 
10.0% 
20.0% 
30.0% 
40.0% 
50.0% 
V 
Unlikely 
S 
Unlikely 
Neutral 
S Likely 
V Likely 
Traditional 
Humanized 
0.0% 
10.0% 
20.0% 
30.0% 
40.0% 
50.0% 
V 
Unlikely 
S 
Unlikely 
Neutral 
S Likely 
V Likely 
Traditional 
Humanized 
0.0% 
10.0% 
20.0% 
30.0% 
40.0% 
V 
Unlikely 
S 
Unlikely 
Neutral 
S Likely 
V Likely 
Traditional 
Humanized 
0.0% 
10.0% 
20.0% 
30.0% 
40.0% 
50.0% 
V 
Unlikely 
S 
Unlikely 
Neutral 
S Likely 
V Likely 
Traditional 
Humanized 
On the package 
On the products 
Before the game 
On the clothes
#CRC2014 @LoveStats 
Implementing the results
#CRC2014 @LoveStats 
There is no perfect question 
Are you the primary grocery shopper in your household? 
Which of the following statements best describes your role in buying groceries for your household? 
Are you responsible for most of the grocery shopping for your household? 
Do you normally do most of the grocery shopping for your household?
#CRC2014 @LoveStats 
•Agree Strongly 
•Agree Somewhat 
•Neutral 
•Disagree Somewhat 
•Disagree Strongly 
•Love 
•Like 
•Neutral 
•Dislike 
•Hate 
Ask for simpler wording
#CRC2014 @LoveStats 
Which of the following do you enjoy? 
•Biking 
•Cooking 
•Hiking 
•Reading 
Which of the following do you enjoy? 
•Bike 
•Cook 
•Hiking 
•Read 
Don’t compromise on grammar 
Let’s eat Grandma. Let’s eat, Grandma. Punctuation saves lives.
#CRC2014 @LoveStats 
For each of the following descriptions of shopping behaviors, please indicate whether the description is highly characteristic, somewhat characteristic, slightly characteristic, or not characteristic at all of you when you visit a membership-only warehouse club store. 
How descriptive are these characteristics when you visit a membership-only warehouse club store. 
Ask for shorter questions
#CRC2014 @LoveStats 
Ask for a bit of humour 
Religion Politics Sex Violence Race 
Pets Movies Neutral jokes Common slang
#CRC2014 @LoveStats 
Answer options can be fun too 
What we usually say: 
What you can say: 
Every day 
Every day. I think I'm addicted! 
More than 10 hours 
More than 10 hours. You can't make me stop! 
No one, you normally play by yourself 
Me, myself, and I 
Never 
Never. I have better things to do! 
Zero 
Zero, zilch, nada!
#CRC2014 @LoveStats 
Thank you for your participation 
You’ve been a great help. Thanks so much! 
Say please and thank you
#CRC2014 @LoveStats 
Let the people speak 
What we usually say: 
What you can say: 
Please describe 3 reasons why that is your favorite video/computer game. 
Give me 3 good reasons why! 
Is there any other information you would like to share with us regarding your opinions and use of video/computer games? (Optional) 
Sharing is caring! Is there anything else you'd like to share about gaming? (Answer if you wish) 
Compared to other surveys you have taken, what is your opinion about this survey? (Please select one in each row) 
What did you think of this survey? (Please select one in each row) 
This is the last page of questions. Is there any other information you would like to share about this survey? (Optional) 
This is the end, my friend. Is there anything about this survey you'd care to share? (Answer if you wish)
#CRC2014 @LoveStats 
9/23/2014 
27 
Thank you! 
Annie Pettit 
Chief Research Officer 
Peanut Labs 
annie@peanutlabs.com 
@LoveStats on twitter 
facebook.com/anniepettit 
ca.linkedin.com/in/anniepettit/ 
jonathan.cheriff@peanutlabs.com 
: ) 
180 Montgomery Street, Suite 1700 - San Francisco, CA 94104

A Tale of Two Surveys: How using real words instead of mumbo jumbo affects survey data quality

  • 1.
    #CRC2014 @LoveStats ATale of Two Surveys Presented by Annie Pettit, Chief Research Officer Peanut Labs is a company
  • 2.
    #CRC2014 @LoveStats Itwas the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to heaven, we were all going direct the other way—in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.
  • 3.
    #CRC2014 @LoveStats Basedon any advertising, such as TV commercials, roadside billboards, banners on internet websites, or other similar forms of advertising, please list in the textbox below the products or types of products that have you yourself have seen publicly advertised between January 1, 2013 and December 31, 2013. (Please type your answer in the box below.)
  • 4.
    #CRC2014 @LoveStats Basedon any advertising, such as TV commercials, roadside billboards, banners on internet websites, or other similar forms of advertising, please list in the textbox below the products or types of products that have you yourself have seen publicly advertised between January 1, 2013 and December 31, 2013. (Please type your answer in the box below.) What products did you see advertised last year? What products did you see advertised last week?
  • 5.
    #CRC2014 @LoveStats Surveymethod •Test n=530 •Control n=530 •US Census Rep •Non-branded computer and video gaming survey •20 minute survey
  • 6.
    #CRC2014 @LoveStats Traditional Humanized What is your favorite video/computer game? If you could only play one video game for the rest of your life, what game would you choose? Have you ever lined up at a store so you could be among the first to purchase a new video/computer game? Are you one of those people who lines up at stores to be the first to buy a new game? How much do you agree or disagree with these statements about gaming? Love 'em or hate 'em, what do you think about video games? What is your opinion about each of the following television shows? Awesome or terrible, how do you rate these TV shows? Is there any other information you would like to share with us regarding your opinions and use of video/computer games? Sharing is caring! Is there anything else you'd like to share about gaming?
  • 7.
  • 8.
    #CRC2014 @LoveStats Speeding •Failed if the survey was completed in the slowest 5% of completion times 0 1 2 3 4 5 Traditional Survey (%) Humanized Survey (%) Speeding
  • 9.
    #CRC2014 @LoveStats Straightlining •Failed if data demonstrated straightlining to a half positive, half negative 8 question grid 0 5 10 15 20 Traditional Survey (%) Humanized Survey (%) Straightlining
  • 10.
    #CRC2014 @LoveStats OpenEnds •Failed if less than 10 characters were provided in response to a request for “3 reasons why…” 0 5 10 Traditional Survey (%) Humanized Survey (%) Open Ends
  • 11.
    #CRC2014 @LoveStats RedHerring •Failed if two fake computer games were selected 0 1 2 3 4 5 Traditional Survey (%) Humanized Survey (%) Red Herring
  • 12.
    #CRC2014 @LoveStats SurveyEvaluation 1.0 2.0 3.0 4.0 5.0 Too long Too complex Fun to answer Would recommend to my friends Traditional Wording Fun Wording Agree Disagree
  • 13.
    #CRC2014 @LoveStats ResponderOpinions Traditional Survey Humanized Survey It was short and nice:) nice short survey not enough points though I found this to be an interesting survey. Thank you. I love surveys and quizzes, and this one was very interesting, because i like sharing things about myself. the survey should be worth more points considering how long the survey was very original method of conducting a survey! NA This was one of the Best surveys I have done!!! No loved it.. funny and quick Good "May the survey FORCE be with you." Was a nice hilariously awesome thing to see. I laughed a lot. o3o This survey was enjoyable. this is an awesome survey. I thoroughly enjoyed myself while taking it. i dont know. good job, sir! If i had a fedora, i'd tip it in your direction. It was good survey I laughed out loud the way some questions were phrased, I wanted to say this is the most refreshing survey ive taken I had a lot of fun
  • 14.
  • 15.
    #CRC2014 @LoveStats PurchaseIntent 0% 10% 20% 30% 40% Not at all Somewhat Very Extremely Raw - Traditional Raw - Humanized
  • 16.
    #CRC2014 @LoveStats Whatis a fair price? 0% 10% 20% 30% $0 $1 to $14 $15 to $19 $20 to $29 $30 to $39 $40 to $49 $50 or more Raw - Traditional Raw - Humanized
  • 17.
    #CRC2014 @LoveStats RatingAds 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% V Unlikely S Unlikely Neutral S Likely V Likely Traditional Humanized 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% V Unlikely S Unlikely Neutral S Likely V Likely Traditional Humanized 0.0% 10.0% 20.0% 30.0% 40.0% V Unlikely S Unlikely Neutral S Likely V Likely Traditional Humanized 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% V Unlikely S Unlikely Neutral S Likely V Likely Traditional Humanized On the package On the products Before the game On the clothes
  • 18.
  • 19.
    #CRC2014 @LoveStats Thereis no perfect question Are you the primary grocery shopper in your household? Which of the following statements best describes your role in buying groceries for your household? Are you responsible for most of the grocery shopping for your household? Do you normally do most of the grocery shopping for your household?
  • 20.
    #CRC2014 @LoveStats •AgreeStrongly •Agree Somewhat •Neutral •Disagree Somewhat •Disagree Strongly •Love •Like •Neutral •Dislike •Hate Ask for simpler wording
  • 21.
    #CRC2014 @LoveStats Whichof the following do you enjoy? •Biking •Cooking •Hiking •Reading Which of the following do you enjoy? •Bike •Cook •Hiking •Read Don’t compromise on grammar Let’s eat Grandma. Let’s eat, Grandma. Punctuation saves lives.
  • 22.
    #CRC2014 @LoveStats Foreach of the following descriptions of shopping behaviors, please indicate whether the description is highly characteristic, somewhat characteristic, slightly characteristic, or not characteristic at all of you when you visit a membership-only warehouse club store. How descriptive are these characteristics when you visit a membership-only warehouse club store. Ask for shorter questions
  • 23.
    #CRC2014 @LoveStats Askfor a bit of humour Religion Politics Sex Violence Race Pets Movies Neutral jokes Common slang
  • 24.
    #CRC2014 @LoveStats Answeroptions can be fun too What we usually say: What you can say: Every day Every day. I think I'm addicted! More than 10 hours More than 10 hours. You can't make me stop! No one, you normally play by yourself Me, myself, and I Never Never. I have better things to do! Zero Zero, zilch, nada!
  • 25.
    #CRC2014 @LoveStats Thankyou for your participation You’ve been a great help. Thanks so much! Say please and thank you
  • 26.
    #CRC2014 @LoveStats Letthe people speak What we usually say: What you can say: Please describe 3 reasons why that is your favorite video/computer game. Give me 3 good reasons why! Is there any other information you would like to share with us regarding your opinions and use of video/computer games? (Optional) Sharing is caring! Is there anything else you'd like to share about gaming? (Answer if you wish) Compared to other surveys you have taken, what is your opinion about this survey? (Please select one in each row) What did you think of this survey? (Please select one in each row) This is the last page of questions. Is there any other information you would like to share about this survey? (Optional) This is the end, my friend. Is there anything about this survey you'd care to share? (Answer if you wish)
  • 27.
    #CRC2014 @LoveStats 9/23/2014 27 Thank you! Annie Pettit Chief Research Officer Peanut Labs annie@peanutlabs.com @LoveStats on twitter facebook.com/anniepettit ca.linkedin.com/in/anniepettit/ jonathan.cheriff@peanutlabs.com : ) 180 Montgomery Street, Suite 1700 - San Francisco, CA 94104