We like to think that everyone answering our surveys is perfectly fluent in English but let's be realistic. About 10% of Americans have difficulty reading/writing in English because it is not their native language. And when we apply standard techniques to identify which survey takers provide good data and which are simply giving random answers, we are often making the mistake of applying measurements that all require high level language skills.

1. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and PollingPlease tweet! @LoveStats @PeanutLabsMediaPlease tweet! @LoveStats @PeanutLabsMedia
I. Can’t.
Understand. You:
How non-native
English speakers
respond to English
surveys
Presented by Annie Pettit
Chief Research Officer at Peanut Labs
DIY Sample and Polling

2. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
20% of US
residents don’t
speak English at
home
9% of US
residents don’t
speak English
well

3. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Survey
•12 minutes, US, Census Rep
•Driving, shopping, electronics, eating out,
working

4. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
N=1212
N=228
N=204
Method
Simple
Language

5. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
DATA QUALITY

6. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Native Speakers Are Click Happy
% of respondents choosing many answer options
0%
5%
10%
15%
20%
Q8 Q11 Q16 Q24 Q26 Q30
Somewhat Good Very Good Extremely Good

7. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Speakers Are Click Averse
% of respondents choosing minimal answer options
0%
20%
40%
60%
Q8 Q11 Q16 Q24 Q26 Q30
Somewhat Good Very Good Extremely Good

8. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Speakers Choose Red Herrings
Sounds familiar? Looks familiar?
0%
2%
4%
6%
8%
10%
Somewhat Good Very Good Extremely Good

9. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Speakers Don’t Follow Instructions
Failure rate on “Choose 3 items”
0%
20%
40%
60%
80%
Choose 3 Choose 3
Somewhat Good Very Good Extremely Good

10. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Speakers Choose None of Above
% choosing many None of Above answers
0%
4%
8%
12%
1 2 3
Somewhat Good Very Good Extremely Good

11. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Provide Shorter Verbatims
% providing shortest responses
0%
5%
10%
15%
20%
25%
30%
1 2 3 4 5
Series1 Series3 Series5
Blank % in Q1 % in Q2 % in Q3 % in Q4
Answer Shortest Answers Longest Answers

12. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL are really fast or really slow
% in Slowest to Fastest Deciles of Responses
0%
5%
10%
15%
20%
25%
1 2 3 4 5 6 7 8 9 10
Poor or Somewhat Good
Very Good
Extremely Good
Slowest Decile Fastest Decile

13. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
ESL Responders Make More Errors
0%
5%
10%
15%
20%
25%
30%
1 2 3 4 5 6 7 8 9 10 11 12
Total Number of Errors
Somewhat Good
Very Good
Extremely Good
0 1 2 3 4 5 6 7 8 9 10 11
Number of errors made

14. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
RESULT OF THESE
DATA QUALITY MEASURES

15. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Higher % of ESL People Cut
Are their
opinions less
important?
Is it fair to cut
them? 0%
2%
4%
6%
8%
10%
12%
14%
1 2 3
Somewhat good
English
Good English Excellent
English

16. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Honest ESL People Get Cut
Any final comments on this survey?
• great and interesting survey
• Have nothing for u guy, sorry im
dumb lol lmao XD
• i 'm very active
• I enjoyed this survey
• it was a good survey
• it was very great to fill all ur surveys
• very nice survey, not enough
points
• Well nothing to share now but their
are something friendly way of
living now.
What 3 things do you do with your phone?
• 1..To call someone 2..To know what is
going on around the world. 3..to text my
friends
• coupons, texts, facebook
• i phone, awesome, can download
games and I used iPhone and mobile
• internet and check emails and talk and
text
• Looking information on the internet. Call
the doctors.
• Play games, Watch Youtube, and Watch
read, Talk Text Listen to music
• text. chatting with friends and talking on
the phone

17. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Most Language Discriminatory
Variables Beta Sig
Under Click V1 -.283 .000
Contradict V1 -.150 .002
Can’t Rank -.137 .006
Lies -.151 .011
Under Click V2 -.132 .030
Eliminate
based on
language

18. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Eliminate
based on
quality
Least Language Discriminatory
Variables Beta Sig.
Over Click V1 .004 .944
Over Click V2 -.005 .927
Straightlining V1 -.005 .926
Over Click V3 .005 .923
Don’t Follow Instructions -.008 .903
Verbatim V1 .011 .838
Invalid Phone Number .015 .756
Contradict V2 -.019 .704

19. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
CONCLUSIONS

20. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Recommendations
•Choose less language discriminative data
quality questions
• Overclicking
• Straightlining
• Follow instructions
• Verbatim quality
• Contradictions
• Red herrings
• None of the above

21. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Recommendations
• Avoid cultural references, slang, norms
• Consider forcing minimum responses
• Keep questions short
• Create white space
Have you seen any advertising for this brand?
Please consider TV ads, billboards, internet ads,
and other ads.
“Costs an
arm and
a leg”

22. Please tweet! @PeanutLabsMedia @LoveStats
DIY Sample and Polling
Thank you!
Annie Pettit
Chief Research Officer
annie@peanutlabs.com
ca.linkedin.com/in/AnniePettit/
facebook.com/AnniePettit
twitter.com/LoveStats
Jonathan Cheriff
Director of Sales & Marketing
jonathan.cheriff@peanutlabs.com
Find PeanutLabs on
LinkedIn Facebook Twitter YouTube
DIY SURVEY PROGRAMMING DIY POLLINGDIY SAMPLE