The document provides tips for avoiding common pitfalls in survey design. It discusses 11 types of mistakes to avoid such as using double-barreled or leading questions, not providing all answer options, asking ambiguous or open-ended questions without parameters, and not pre-planning how the data will be analyzed. The document encourages survey designers to consider the respondent's point of view, use clear and concise language, avoid biases, and test the survey design.

1. 1
AMA Iowa Experience Event
February 5, 2016
SURVEY DESIGN
Avoiding the Common Pitfalls
that Can Sabotage Your Data
Presented by:

2. 2

3. 3
Types of Research

4. 4
Types of Research
Qualitative Research Quantitative Research
• In-person Focus Groups • Phone Surveys
• Online Chat Focus Groups • Mail Surveys
• Bulletin Board Focus Groups • Online Surveys
• IDI’s – individual interviews • Omnibus Surveys
• Dyads, Triads
• Online Communities
• Journals, Diaries
• Usability Testing
• Mock Juries
• Ethnography (observational)

5. 5
Consider the respondent’s point of view
Survey Design
Will they be able to
answer your
questions?
Will they want to
answer your
questions?
Will they be
motivated to answer
all your questions?

6. 6
Survey Design

7. 7
Do not have
mutually exclusive responses
Mistake #1
Good:
What is your age?
____ Under 20
____ 20 – 29
____ 30 – 39
____ 40 – 49
____ 50 +
Bad:
What is your age?
____ Under 20
____ 20 – 30
____ 30 – 40
____ 40 – 50
____ Over 50

8. 8
Asking double-barreled questions
Bad:
How satisfied are you with the pay and work benefits of
your current job?
Good:
• How satisfied are you with the pay of your current job?
• How satisfied are you with the work benefits of your
current job?
Mistake #2

9. 9
Not providing all answer options
What type of vehicle do you own?
____ Van
____ SUV
____ Sedan
____ Truck
____ Convertible
____ I do not own a vehicle
Other _________________
Mistake #3

10. 10
Not providing all answer options
What is the main reason why you attended AMA
Iowa’s Q&AMA event?
____ The event sounded interesting
____ I wanted to see what a trivia night was like
____ Networking opportunities
Other (please
specify)________________________
Mistake #3

11. 11
Questions are not clear and precise,
have ambiguity
In a typical month, how often do you shop at a mall?
Ambiguous:
____ Never
____ Occasionally
____ Sometimes
____ Often
____ Regularly
Mistake #4
Good:
____ Not at all
____ 1 or 2 times
____ 3 or 4 times
____ More than 4 times

12. 12
Using absolutes in questions
Mistake #5
Bad:
Do you always eat breakfast?
____ Yes
____ No
Good:
How many days a week do you usually eat breakfast?
____ Every day
____ 5-6 days
____ 3-4 days
____ 1-2 days
____ I usually don’t eat breakfast

13. 13
Questions that are leading or biased
Bad: Is Colgate your favorite toothpaste? Yes or No
Good: What is your favorite toothpaste brand? __________
Bad:
The Mac operating system rarely gets infected by viruses and
therefore the Department of Education should only purchase
Mac computers. Yes, No, Maybe
Good:
The Department of Education should only purchase Mac computers.
Yes, No, Maybe
Why do you say this? _______________________________
Mistake #6

14. 14
Using Loaded Questions
Bad: What kind of wine do you like to drink?
Good:
Do you like to drink wine?
____ Yes
____ No
If Yes, what kind of wine do you like to drink?
(could be open-ended or drop-down menu)
If No, why do you not like to drink wine?
Mistake #7

15. 15
Confusing rating scale questions;
Reversing rating scales
Q1: Please rank your overall satisfaction with Company ABC
on a scale of 1 to 5.
1 2 3 4 5
Q2: Please rank your satisfaction with Company ABC’s
customer service staff on a scale of 1 to 5.
5 4 3 2 1
Mistake #8

16. 16
Confusing rating scale questions;
Reversing rating scales
Q1: Please rank your overall satisfaction with Company ABC on a
scale of 1 to 5, where 1 is Not At All Satisfied and 5 is Very
Satisfied.
1 2 3 4 5
Q2: Please rank your satisfaction with Company ABC’s customer
service staff.
Not At Not Very
All Satisfied Satisfied Neutral Satisfied Satisfied
Mistake #8

17. 17
Not using language
your respondent understands
Using too much jargon, slang, acronyms
Bad: What marketing and research organizations are you
a member of? (Please select all that apply.)
____ AMA
____ DMA
____ MRA
____ NAMA
____ PRSA
Other _____________
Mistake #9
Good:
Need to spell out acronyms

18. 18
Open-ended questions that
do not have parameters
Bad:
According to you, what are the factors that contribute to the success of a
company in the poor economic conditions?
Good:
In your opinion, what are the key factors that contribute to the success of a
manufacturing company in the present recessionary economy?
Please briefly describe up to three.
1. ________________________________________________
2. ________________________________________________
3. ________________________________________________
Mistake #10

19. 19
Asking too many open-ended questions
Creates fatigue and
disinterest in completing survey
Adds unnecessary length to survey
Mistake #11

20. 20
Pre-survey design, identify your plan for analysis
Don’t let them off the hook too easily
from answering a question
Proof, PROOF, Proof
Test, TEST, Test
Organize survey to minimize bias
Design Tips

21. 21
Q & A