Ask Questions on a
5-point or 7-point Scale
Are you happy with your job?
Sample Survey Question
No
21%
Yes
79%
People Happy at Their Jobs
I am happy with my job.
Sample Survey Question
People Happy at Their Jobs
Strongly Disagree Disagree Neutral Agree Strongly Agree
Average = 3.05
Break Down Big Concepts
into Tangible Pieces
Are you satisfied?
Sample Survey Question
Sample Survey Question
Please Rate Your Satisfaction with Each Feature
Very
unsatisfied
Somewhat
Unsatisfied
Neural
Somewh...
Satisfaction by Feature
Feature 1 Feature 2 Feature 3
Compare Answers
Across Questions
Sample Survey Question
Please Rate Your Experience with Our Business
Very
unsatisfied
Somewhat
Unsatisfied
Neural
Somewhat...
Business Ratings
Product Quality Price Account Manager
Business Ratings by Company Size
Product Quality Price Acount Manager Product Quality Price Acount Manager
Small Business ...
Comparing Answers with SurveyMonkey
Analyze + Compare
Beware of Bias
Priming
Bargh, Chen, and Burrows (1996)
Rude aggressively, bold, rude, bother, disturb, intrude, annoyingly, interrupt, audaciousl...
Bargh, Chen, and Burrows (1996)
People Who Interrupted After Sentence Task
Polite Neutral Rude
Order Bias
How important is link building?
Question Order Bias
How much link building do you
plan to do this year?
Randomization in SurveyMonkey and Google Forms
Randomize Order
Stereotype Threat
“sports intelligence”
Stone, Lynch, Sjomeling, Darley (1999)
“natural athletic ability”
Inzlicht (2000)
2 Women 2 Men
Female Math Test Performance when Others Present
Double-Barreled Questions
Do you play games on your
smart phone or tablet?
Double-Barreled Question
Leading Questions
Do you support helping the poor?
Leading Questions
Do you support welfare?
Understand How
and Why People Lie
Sensitive Information
Use ranges for demographic info.
Sensitive Information
Boredom
Set expectations & show progress.
Boredom
Guessing is Hard
Be reasonable about estimations.
Guessing is Hard
Cool Things You
Can Do With Data
How often do you use OSE?
Sample Survey Question
Sample Survey Question 1
How often do you use Open Site Explorer for the following tasks?
Never Rarely Sometimes Often Alw...
Sample Survey Question 2
How would you rate Open Site Explorer for the following tasks?
Never Rarely Sometimes Often Alway...
Sample Survey Question 3
How important are the following Open Site Explorer features?
Never Rarely Sometimes Often Always
...
Feature Prioritization Chart
Frequency
0
7.5
15
22.5
30
Rating
0 4 8 12 16
Categorical
Do you drink coffee?
Categorical Response
Yes | No
Coffee Drinkers
Categorical Response
No
Yes
No Yes
Ordinal
Ordinal Response
How much coffee do you drink?
0 cups per day
1-3 cups per day
4-5 cups per day
6+ cups per day
Ordinal Response
Coffee Drinkers
6+ Cups
4-5 Cups
1-3 Cups
0 Cups
0 Cups 1-3 Cups 4-5 Cups 6+ Cups
Interval & Ratio
How often do you drink coffee?
Interval & Ratio Response
Never Rarely Sometimes Often Always
How many cups of coffee do you
drink each day?
Interval & Ratio Response
______________
(fill in the blank)
Source: FIVETHIRTYEIGHT
Mean, Median, Mode
Source: Pew Global
Correlation
Source: Wired
Cluster Analysis
Interval & Ratio Response
Regression
Outcome
Variable 1
Variable 2
Variable 3
(unknown)
relative influence on outcome
Interval & Ratio Response
Excel Add-In: Analysis Toolpak
Free Add-In
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Asking Better Survey Questions - MozCon 2014
Upcoming SlideShare
Loading in …5
×

Asking Better Survey Questions - MozCon 2014

352 views

Published on

Bad Data, Bad Decisions: The Art of Asking Better Questions, Stephanie Beadell's presentation from MozCon 2014

Published in: Marketing
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
352
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Asking Better Survey Questions - MozCon 2014

  1. 1. Ask Questions on a 5-point or 7-point Scale
  2. 2. Are you happy with your job? Sample Survey Question
  3. 3. No 21% Yes 79% People Happy at Their Jobs
  4. 4. I am happy with my job. Sample Survey Question
  5. 5. People Happy at Their Jobs Strongly Disagree Disagree Neutral Agree Strongly Agree Average = 3.05
  6. 6. Break Down Big Concepts into Tangible Pieces
  7. 7. Are you satisfied? Sample Survey Question
  8. 8. Sample Survey Question Please Rate Your Satisfaction with Each Feature Very unsatisfied Somewhat Unsatisfied Neural Somewhat Satisfied Very Satisfied Feature 1 Feature 2 Feature 3
  9. 9. Satisfaction by Feature Feature 1 Feature 2 Feature 3
  10. 10. Compare Answers Across Questions
  11. 11. Sample Survey Question Please Rate Your Experience with Our Business Very unsatisfied Somewhat Unsatisfied Neural Somewhat Satisfied Very Satisfied Product Quality Price Account Manager
  12. 12. Business Ratings Product Quality Price Account Manager
  13. 13. Business Ratings by Company Size Product Quality Price Acount Manager Product Quality Price Acount Manager Small Business Enterprise
  14. 14. Comparing Answers with SurveyMonkey Analyze + Compare
  15. 15. Beware of Bias
  16. 16. Priming
  17. 17. Bargh, Chen, and Burrows (1996) Rude aggressively, bold, rude, bother, disturb, intrude, annoyingly, interrupt, audaciously, brazen, impolitely, infringe, obnoxious, aggravating, bluntly Polite honor, considerate, appreciate, patiently, cordially, yield, polite, cautiously, courteous, graciously, sensitively; discreetly, behaved, unobtrusively Neutral exercising, flawlessly; occasionally rapidly, gleefully, practiced, optimistically, successfully, normally, send, watches, encourages. gives, clears, prepares Sentence Scramble Task
  18. 18. Bargh, Chen, and Burrows (1996) People Who Interrupted After Sentence Task Polite Neutral Rude
  19. 19. Order Bias
  20. 20. How important is link building? Question Order Bias How much link building do you plan to do this year?
  21. 21. Randomization in SurveyMonkey and Google Forms Randomize Order
  22. 22. Stereotype Threat
  23. 23. “sports intelligence” Stone, Lynch, Sjomeling, Darley (1999) “natural athletic ability”
  24. 24. Inzlicht (2000) 2 Women 2 Men Female Math Test Performance when Others Present
  25. 25. Double-Barreled Questions
  26. 26. Do you play games on your smart phone or tablet? Double-Barreled Question
  27. 27. Leading Questions
  28. 28. Do you support helping the poor? Leading Questions Do you support welfare?
  29. 29. Understand How and Why People Lie
  30. 30. Sensitive Information
  31. 31. Use ranges for demographic info. Sensitive Information
  32. 32. Boredom
  33. 33. Set expectations & show progress. Boredom
  34. 34. Guessing is Hard
  35. 35. Be reasonable about estimations. Guessing is Hard
  36. 36. Cool Things You Can Do With Data
  37. 37. How often do you use OSE? Sample Survey Question
  38. 38. Sample Survey Question 1 How often do you use Open Site Explorer for the following tasks? Never Rarely Sometimes Often Always Find Backlinks Anchor Text Analysis Competitor Analysis
  39. 39. Sample Survey Question 2 How would you rate Open Site Explorer for the following tasks? Never Rarely Sometimes Often Always Find Backlinks Anchor Text Analysis Competitor Analysis
  40. 40. Sample Survey Question 3 How important are the following Open Site Explorer features? Never Rarely Sometimes Often Always Find Backlinks Anchor Text Analysis Competitor Analysis
  41. 41. Feature Prioritization Chart Frequency 0 7.5 15 22.5 30 Rating 0 4 8 12 16
  42. 42. Categorical
  43. 43. Do you drink coffee? Categorical Response Yes | No
  44. 44. Coffee Drinkers Categorical Response No Yes No Yes
  45. 45. Ordinal
  46. 46. Ordinal Response How much coffee do you drink? 0 cups per day 1-3 cups per day 4-5 cups per day 6+ cups per day
  47. 47. Ordinal Response Coffee Drinkers 6+ Cups 4-5 Cups 1-3 Cups 0 Cups 0 Cups 1-3 Cups 4-5 Cups 6+ Cups
  48. 48. Interval & Ratio
  49. 49. How often do you drink coffee? Interval & Ratio Response Never Rarely Sometimes Often Always
  50. 50. How many cups of coffee do you drink each day? Interval & Ratio Response ______________ (fill in the blank)
  51. 51. Source: FIVETHIRTYEIGHT Mean, Median, Mode
  52. 52. Source: Pew Global Correlation
  53. 53. Source: Wired Cluster Analysis
  54. 54. Interval & Ratio Response Regression Outcome Variable 1 Variable 2 Variable 3 (unknown) relative influence on outcome
  55. 55. Interval & Ratio Response Excel Add-In: Analysis Toolpak Free Add-In

×