This document discusses the importance of question design in surveys and presents strategies for minimizing researcher bias and obtaining objective data. It notes that surveys are inherently shaped by the researcher's perspectives and that response options can introduce ambiguity and "noise" that obscures the desired information. Specific problematic examples are provided. The document advocates for clear, logically structured questions with unambiguous response anchors that are symmetrically distanced and intuitively ordered. While good questions don't guarantee valid results, they can help ensure the researcher is hearing the intended "signal" rather than unwanted "noise" in the data.

1. Will Evans
Yana Kuchirko

2. "They've done studies, you
know. 60% of the time, it
works every time. ”
- Brian Fantana
06/16/12 2

3. Surveys Aren’t Objective
They are created by people who are biased and analyze their
results based on their own (mis)perceptions.
06/16/12 3

4. Epistemological Dead End?
No. Awareness that the researcher plays an integral
role in the process of measuring any given
phenomena by deciding how to measure is key.
There are ways to minimize researcher bias by
creating better questions.
06/16/12 4

5. Methods are not “just methods”
How you measure what you are studying shapes what you find.
06/16/12 5

6. Let’s say you are studying…
How often teenagers use your website
06/16/12 6

7. You might ask teenagers…
“How often do you my website?”
c.Very rarely
d.Rarely
e.Occasionally
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
06/16/12 7

8. You might ask teenagers…
“How often do you my website?”
c.Very rarely
d.Rarely And what’s wrong
e.Occasionally with these????
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
06/16/12 8

9. You might ask teenagers…
“How often do you my website?” ! !!
VI TY
EC TI
c.Very rarely
BJ
d.Rarely SU And what’s wrong
e.Occasionally with these????
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
06/16/12 9

10. “Hmmm, for me “frequently”
really means….”
A few times a week A few times a day
So what are we really measuring? NOISE.
06/16/12 10

11. What is “noise”?
What we observe…
06/16/12 11

12. What is “noise”?
We WANT this!
We DON’T WANT this!
06/16/12 12

13. An imaginary scenario
Let’s really stretch our thinking a bit here to
provide a more concrete understanding of
“noise” in your data.
Imagine you have AT&T phone service and
you’re trying to make a call. The signal isn’t
clearly going through and you end up hearing
everything but the other person’s voice.
Frustrating, right? Thank God this is only a
pretend scenario.
06/16/12 13

14. How is AT&T related to “Noise”?
When you develop bad questions, you don’t
“hear” the message of your data clearly.
06/16/12 14

15. So…why is “noise” bad???
• Badly structured and poorly worded
questions that obfuscate meaning for
participants provide bad data.
• Bad data
– Doesn’t answer your research question
– Makes it difficult to interpret results
– Is pretty much useless.
06/16/12 15

16. More bad examples
of frequently used response options
a. Completely satisfied a. Totally like
b. Very satisfied b. Very much like
c. Fairly well satisfied c. Moderately like
d. Somewhat dissatisfied d. Somewhat like
e. Very dissatisfied e. Not like
What’s the difference between What’s the difference between
“fairly well” and “somewhat”? “moderately like” and “somewhat like”?
06/16/12 16

17. More bad examples
of frequently used response options
a. Completely satisfied a. Totally like
b. Very satisfied b. Very much like
c. Fairly well satisfied c. Moderately like
d. Somewhat dissatisfied d. Somewhat like
e. Very dissatisfied e. Not like
What’s the difference between What’s the difference between
“fairly well” and “somewhat”? “moderately like” and “somewhat like”?
Semantics
06/16/12 17

18. Words like….
• Sometimes
• Often
• Moderately
• Very
• Not very much
Are subjective
= they mean different things for different people
06/16/12 18

19. A better alternative?
Actually asking people how often they do certain
activities.
06/16/12 19

20. In order to gather objective data…
…your questions must mean the same thing
for everyone.
How often do you use my website?
d.Never
e.A few times a year
f.Once a month
g.2-3 times a month
h.Once a week
i.A few times a week
j.Every day
“Never” means never for everyone!
06/16/12 20

21. Importance of “Anchors”
Responses options serve as “anchors” for each question,
determining the “location” of each responses as qualitatively
distinct from the other.
06/16/12 21

22. Importance of “Anchors”
Responses options serve as “anchors” for each
question, determining the “location” of each
responses as qualitatively distinct from the other.
Strongly Strongly
disagree Disagree Neutral Agree Agree
06/16/12 22

23. (Ideal) Response Symmetry
• Good questions aim to have symmetrical
quantitative/qualitative distance between
anchors
a. Never
b. A few times a year
The distance between “never” and
c. Once a month
“a few times a year” is proportionate to
d. 2-3 times a month “every day” and “a few times a week”.
e. Once a week
f. A few times a week
g. Every day
06/16/12 23

24. Adherence to Logic & Linearity
People are accustomed to paradigms that are
intuitive, and often “linear”.
Anchors should not be the exception.
06/16/12 24

25. Intuitive Anchor Directions
Never Always
Disagree Agree
Very poor Very good
Not very Very
important Important
06/16/12 25

26. Value of Clarity
If questions are clear and concise, participants would
spend less time analyzing the questions themselves and
more time on answering them. Mystic Arts, LLC
06/16/12 26

27. So do good questions guarantee valid
results?
• No. But good questions offer more
assurance that you are listening to signal
and not the noise.
• But answering your research question can be
done in other ways
• Behavioral Observations
• Open ended qualitative questions
• Many more…
06/16/12 27

28. The End.
So was this presentation simply awesome?
a. Hell yeah!
b. Definitely!
c. Totally rocked!
06/16/12 28

29. Thanks!