How to create surveys that don't suck

1,217
-1

Published on

Published in: Technology, Business

How to create surveys that don't suck

  1. 1. Yana Kuchirko@mslogophiliacWill Evans@semanticwill
  2. 2. "Theyve done studies, you know. 60% of the time, it works every time. ” - Brian Fantana03/18/12 Will Evans & Yana Kuchirko 2
  3. 3. Surveys Aren’t ObjectiveThey are created by people who are biased and analyze their results based on their own (mis)perceptions.03/18/12 Will Evans & Yana Kuchirko 3
  4. 4. Epistemological Dead End? No. Awareness that the researcher plays an integral role in the process of measuring any given phenomena by deciding how to measure is key. There are ways to minimize researcher bias by creating better questions.03/18/12 Will Evans & Yana Kuchirko 4
  5. 5. Methods are not “just methods”How you measure what you are studying shapes what you find. 03/18/12 Will Evans & Yana Kuchirko 5
  6. 6. Let’s say you are studying… How often teenagers use your website03/18/12 Will Evans & Yana Kuchirko 6
  7. 7. You might ask teenagers…“How often do you my website?”c.Very rarelyd.Rarelye.Occasionallyf.Frequentlyg.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 7
  8. 8. You might ask teenagers…“How often do you my website?”c.Very rarelyd.Rarely And what’s wronge.Occasionally with these????f.Frequentlyg.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 8
  9. 9. You might ask teenagers…“How often do you my website?” ! !! VI TY EC TIc.Very rarely BJd.Rarely SU And what’s wronge.Occasionally with these????f.Frequentlyg.Very frequently Response options source: http://www.dataguru.org 03/18/12 Will Evans & Yana Kuchirko 9
  10. 10. “Hmmm, for me “frequently” really means….” A few times a week A few times a daySo what are we really measuring? NOISE. 03/18/12 Will Evans & Yana Kuchirko 10
  11. 11. What is “noise”?What we observe… 03/18/12 Will Evans & Yana Kuchirko 11
  12. 12. What is “noise”? We WANT this! We DON’T WANT this!03/18/12 Will Evans & Yana Kuchirko 12
  13. 13. An imaginary scenarioLet’s really stretch our thinking a bit here toprovide a more concrete understanding of“noise” in your data.Imagine you have AT&T phone service andyou’re trying to make a call. The signal isn’tclearly going through and you end up hearingeverything but the other person’s voice.Frustrating, right? Thank God this is only apretend scenario.03/18/12 Will Evans & Yana Kuchirko 13
  14. 14. How is AT&T related to “Noise”?When you develop bad questions, you don’t“hear” the message of your data clearly. 03/18/12 Will Evans & Yana Kuchirko 14
  15. 15. So…why is “noise” bad???• Badly structured and poorly worded questions that obfuscate meaning for participants provide bad data.• Bad data – Doesn’t answer your research question – Makes it difficult to interpret results – Is pretty much useless.03/18/12 Will Evans & Yana Kuchirko 15
  16. 16. More bad examplesof frequently used response options a. Completely satisfied a. Totally like b. Very satisfied b. Very much like c. Fairly well satisfied c. Moderately like d. Somewhat dissatisfied d. Somewhat like e. Very dissatisfied e. Not like What’s the difference between What’s the difference between “fairly well” and “somewhat”? “moderately like” and “somewhat like”?03/18/12 Will Evans & Yana Kuchirko 16
  17. 17. More bad examplesof frequently used response options a. Completely satisfied a. Totally like b. Very satisfied b. Very much like c. Fairly well satisfied c. Moderately like d. Somewhat dissatisfied d. Somewhat like e. Very dissatisfied e. Not like What’s the difference between What’s the difference between “fairly well” and “somewhat”? “moderately like” and “somewhat like”? Semantics03/18/12 Will Evans & Yana Kuchirko 17
  18. 18. Words like…. • Sometimes • Often • Moderately • Very • Not very much Are subjective = they mean different things for different people 03/18/12 Will Evans & Yana Kuchirko 18
  19. 19. A better alternative?Actually asking people how often they do certain activities. 03/18/12 Will Evans & Yana Kuchirko 19
  20. 20. In order to gather objective data……your questions must mean the same thingfor everyone.How often do you use my website?d.Nevere.A few times a yearf.Once a monthg.2-3 times a monthh.Once a weeki.A few times a weekj.Every day “Never” means never for everyone!03/18/12 Will Evans & Yana Kuchirko 20
  21. 21. Importance of “Anchors”Responses options serve as “anchors” for each question,determining the “location” of each responses as qualitativelydistinct from the other. 03/18/12 Will Evans & Yana Kuchirko 21
  22. 22. Importance of “Anchors”Responses options serve as “anchors” for eachquestion, determining the “location” of eachresponses as qualitatively distinct from the other. Strongly Strongly disagree Disagree Neutral Agree Agree03/18/12 Will Evans & Yana Kuchirko 22
  23. 23. (Ideal) Response Symmetry• Good questions aim to have symmetrical quantitative/qualitative distance between anchors a. Never b. A few times a year The distance between “never” and c. Once a month “a few times a year” is proportionate to d. 2-3 times a month “every day” and “a few times a week”. e. Once a week f. A few times a week g. Every day03/18/12 Will Evans & Yana Kuchirko 23
  24. 24. Adherence to Logic & LinearityPeople are accustomed to paradigms that are intuitive, and often “linear”. Anchors should not be the exception.03/18/12 Will Evans & Yana Kuchirko 24
  25. 25. Intuitive Anchor Directions Never AlwaysDisagree AgreeVery poor Very goodNot very Veryimportant Important 03/18/12 Will Evans & Yana Kuchirko 25
  26. 26. Value of ClarityIf questions are clear and concise, participants wouldspend less time analyzing the questions themselves andmore time on answering them. Mystic Arts, LLC 03/18/12 Will Evans & Yana Kuchirko 26
  27. 27. So do good questions guarantee valid results?• No. But good questions offer more assurance that you are listening to signal and not the noise.• But answering your research question can be done in other ways • Behavioral Observations • Open ended qualitative questions • Many more…03/18/12 Will Evans & Yana Kuchirko 27
  28. 28. The End. So was this presentation simply oh-my-fucking-god-awesome? a. Hell yeah! b. Definitely! c. Totally rocked!03/18/12 Will Evans & Yana Kuchirko 28
  29. 29. Thanks!

×