Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How to get better
results from a survey
Caroline Jarrett
@cjforms
Content Strategy Summit 2015 #CSSummit
Caroline Jarrett @cjforms (CC) BY SA-4.0
I’m a forms specialist
2
Image credit: Flickr, taxrebate.org.uk
Caroline Jarrett @cjforms (CC) BY SA-4.0
Why do people answer questions?
3Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms (CC) BY SA-4.0
A dollar bill with a mail survey works
better than $10 (guaranteed) later
4
Carol...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward, and trust
People will only respond if they tru...
6
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms (CC) BY SA-4.0
7
The aim of a survey is to get a number
that helps you to make a decision
Caroline Jarrett @cjforms (CC) BY SA-4.0
8
To get better results from your survey,
think about the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
People ask me about surveys
• “How many people do I need to ask?”
• “How many que...
Agenda “How many people do I need to ask?”
“How many questions can I have?”
“What makes a good question?”
“How many points...
Caroline Jarrett @cjforms (CC) BY SA-4.0
To find out how many people to ask,
start at how many we need to answer
Caroline ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Whether they’ll answer depends
on effort
Questions:
What are you
asking about?
Ho...
Caroline Jarrett @cjforms (CC) BY SA-4.0
And on the reward you’re offering
Goals:
Why are you asking?
Is helping you a rew...
Then there’s the
‘Justin Bieber North Korea’ problem
14
http://www.bbc.com/news/10506482
Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t just want answers,
we want answers from the right people
Response
Caroli...
Caroline Jarrett @cjforms (CC) BY SA-4.0
So it matters where
we get our sample from
Sample:
the list you
sample from
Carol...
Caroline Jarrett @cjforms (CC) BY SA-4.0
17
And now it’s easy to work out
how many to ask
Caroline Jarrett @cjforms (CC) B...
Caroline Jarrett @cjforms (CC) BY SA-4.0
We thought about a lot of topics
to work that out
Goals
Sample
Questions
Fieldwor...
Caroline Jarrett @cjforms (CC) BY SA-4.0
To get really good results, we want
useful answers from the right people
Response...
Caroline Jarrett @cjforms (CC) BY SA-4.0
“You're using this site from outside the
UK. Where are you answering from?
We’d l...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Insights are the numbers
that you use for decisions
Insights 21
Caroline Jarrett ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
We’ve thought about a lot of issues
Goals
Questions
Sample
Fieldwork
ResponseResp...
Caroline Jarrett @cjforms (CC) BY SA-4.0
To get better results from your survey,
think about the Survey Octopus
23
Agenda “How many people do I need to ask?”
“How many questions can I have?”
“What sorts of questions are best?”
“How many ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
“How many questions can I have?”
Let’s work backwards from insights
25
Which answ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
You need accurate answers
to the questions
26
Will your respondents give you
good...
27
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
availa...
28
Caroline Jarrett @cjforms (CC) BY SA-4.0
29
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
https://commons.wikimedia....
Caroline Jarrett @cjforms (CC) BY SA-4.0
You need questions that your
respondents can answer accurately
Which
questions
wi...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Now we get to fieldwork
31
How many
of those
questions
are people
willing to
answ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Fieldwork used to be expensive
so a survey was a rare event.
32
Image credit: htt...
Caroline Jarrett @cjforms (CC) BY SA-4.0
1950s mindset: “Ask Everything”
33
Survey =
Big Honkin’ Survey
Caroline Jarrett @cjforms (CC) BY SA-4.0
2015 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• As...
Tip Interview first
Caroline Jarrett @cjforms (CC) BY SA-4.0
35
Time for new question
Caroline Jarrett @cjforms (CC) BY SA-4.0
36
One way to
iterate,
improve,
increase
Big Honkin’ Survey
• Negotiate to get survey
down to 20 questions
• Ask 10,000 people
• Get 1,000 responses
• Take a week ...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Do both!
Big Honkin’ Survey
• Big numbers are
impressive
• Can compare answers
fr...
Agenda “How many people do I need to ask?”
“How many questions can I have?”
“What makes a good question?”
“How many points...
Caroline Jarrett @cjforms
40
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
...
Tip
Always allow for ‘other’
Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk
41
42
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
availa...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward, and trust
People will only respond if they tru...
Caroline Jarrett @cjforms (CC) BY SA-4.0
44
A good question works in three ways
Appropriate
Obvious Interesting
Caroline Jarrett @cjforms (CC) BY SA-4.0
45
Why did you visit our website today?
Appropriate
Obvious Interesting
Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to
a friend or family member?
46
In a shop,
buying a
baby ...
Tip
Test your questions by
interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0
47
Agenda “How many people do I need to ask?”
“How many questions can I have?”
“What makes a good question?”
“How many points...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Likert had several different types
of question in his response formats
Likert, Re...
Caroline Jarrett @cjforms (CC) BY SA-4.0
You can find an academic paper to
support almost any number of points
• Krosnick ...
Tip
Don’t stress too much about
the number of points in your
rating scale
51
Picture credit: Flickr - Bill Soderman (Bills...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Well, OK, stress a little bit.
52
This scale is
downright
peculiar. Avoid.
Summary
53
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett @cjforms (CC) BY SA-4.0
54
The aim is to get the best number you
can, within the resources you have
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim is to get the best number you
can, within the resources you have
What you...
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim is to get the best number you
can, within the resources you have
What you...
Caroline Jarrett @cjforms (CC) BY SA-4.0
57
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Rep...
58
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R....
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett
Twitter @cjforms
http://www.slideshare.net/cjforms
carolinej@eff...
Upcoming SlideShare
Loading in …5
×

How to get better results from a survey: Meet the Survey Octopus

2,332 views

Published on

The Survey Octopus is a friendly creature who will help you to think about all the crucial issues in crafting a survey.

Presentation by Caroline Jarrett @cjforms for the 2014 Content Strategy Summit #CSSummit

Published in: Design
  • Be the first to comment

How to get better results from a survey: Meet the Survey Octopus

  1. 1. How to get better results from a survey Caroline Jarrett @cjforms Content Strategy Summit 2015 #CSSummit
  2. 2. Caroline Jarrett @cjforms (CC) BY SA-4.0 I’m a forms specialist 2 Image credit: Flickr, taxrebate.org.uk
  3. 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why do people answer questions? 3Caroline Jarrett @cjforms (CC) BY SA-4.0
  4. 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 A dollar bill with a mail survey works better than $10 (guaranteed) later 4 Caroline Jarrett @cjforms (CC) BY SA-4.0
  5. 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
  6. 6. 6 Caroline Jarrett @cjforms (CC) BY SA-4.0
  7. 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7 The aim of a survey is to get a number that helps you to make a decision
  8. 8. Caroline Jarrett @cjforms (CC) BY SA-4.0 8 To get better results from your survey, think about the Survey Octopus
  9. 9. Caroline Jarrett @cjforms (CC) BY SA-4.0 People ask me about surveys • “How many people do I need to ask?” • “How many questions can I have?” • “Please tell me whether this is a good question” • “I think it’s best to have 5 points in my rating scale, but my boss wants to have 7. Who is right?” 9
  10. 10. Agenda “How many people do I need to ask?” “How many questions can I have?” “What makes a good question?” “How many points in a rating scale?” 10 Caroline Jarrett @cjforms (CC) BY SA-4.0
  11. 11. Caroline Jarrett @cjforms (CC) BY SA-4.0 To find out how many people to ask, start at how many we need to answer Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork: Who answers? 11
  12. 12. Caroline Jarrett @cjforms (CC) BY SA-4.0 Whether they’ll answer depends on effort Questions: What are you asking about? How many questions? 12 Caroline Jarrett @cjforms (CC) BY SA-4.0
  13. 13. Caroline Jarrett @cjforms (CC) BY SA-4.0 And on the reward you’re offering Goals: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 13 Caroline Jarrett @cjforms (CC) BY SA-4.0
  14. 14. Then there’s the ‘Justin Bieber North Korea’ problem 14 http://www.bbc.com/news/10506482
  15. 15. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t just want answers, we want answers from the right people Response Caroline Jarrett @cjforms (CC) BY SA-4.0 15
  16. 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 So it matters where we get our sample from Sample: the list you sample from Caroline Jarrett @cjforms (CC) BY SA-4.0 16
  17. 17. Caroline Jarrett @cjforms (CC) BY SA-4.0 17 And now it’s easy to work out how many to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the number of people to ask
  18. 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 We thought about a lot of topics to work that out Goals Sample Questions Fieldwork Response 18 Caroline Jarrett @cjforms (CC) BY SA-4.0
  19. 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 To get really good results, we want useful answers from the right people Response Response 19 Caroline Jarrett @cjforms (CC) BY SA-4.0
  20. 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 “You're using this site from outside the UK. Where are you answering from? We’d like answers like these – Anchorage Alaska U.S.A. – Cameroon – Dubai, United Arab Emirates Even better, extra-interesting answers like these: – Cote d'Ivoire / France / UK. – Canada, originally from the UK – Philippines, would like to work in UK with my husband if possible But definitely not like these: – ä¸å›½ – its not important – nnnnnn – none of your business 20
  21. 21. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights are the numbers that you use for decisions Insights 21 Caroline Jarrett @cjforms (CC) BY SA-4.0
  22. 22. Caroline Jarrett @cjforms (CC) BY SA-4.0 We’ve thought about a lot of issues Goals Questions Sample Fieldwork ResponseResponse Insights 22 Caroline Jarrett @cjforms (CC) BY SA-4.0
  23. 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 To get better results from your survey, think about the Survey Octopus 23
  24. 24. Agenda “How many people do I need to ask?” “How many questions can I have?” “What sorts of questions are best?” “How many points in a rating scale?” 24 Caroline Jarrett @cjforms (CC) BY SA-4.0
  25. 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 “How many questions can I have?” Let’s work backwards from insights 25 Which answers do you need to make the decisions?
  26. 26. Caroline Jarrett @cjforms (CC) BY SA-4.0 You need accurate answers to the questions 26 Will your respondents give you good answers to all of them?
  27. 27. 27 In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0
  28. 28. 28 Caroline Jarrett @cjforms (CC) BY SA-4.0
  29. 29. 29 "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett
  30. 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 You need questions that your respondents can answer accurately Which questions will get you the answers that you can use? 30
  31. 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Now we get to fieldwork 31 How many of those questions are people willing to answer?
  32. 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork used to be expensive so a survey was a rare event. 32 Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/
  33. 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 1950s mindset: “Ask Everything” 33 Survey = Big Honkin’ Survey
  34. 34. Caroline Jarrett @cjforms (CC) BY SA-4.0 2015 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 34
  35. 35. Tip Interview first Caroline Jarrett @cjforms (CC) BY SA-4.0 35
  36. 36. Time for new question Caroline Jarrett @cjforms (CC) BY SA-4.0 36 One way to iterate, improve, increase
  37. 37. Big Honkin’ Survey • Negotiate to get survey down to 20 questions • Ask 10,000 people • Get 1,000 responses • Take a week (or more) to analyze the responses • Have a big presentation a month later Light touch survey • ‘Question of the day’ (one question) • Ask 100 people • Get 50 responses • Analyze them same day • Present same day • Repeat 4 x 5 = 20 times 37 Caroline Jarrett @cjforms (CC) BY SA-4.0
  38. 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 Do both! Big Honkin’ Survey • Big numbers are impressive • Can compare answers from different segments • Easier ‘sell’ to stakeholders Light touch survey • Quick, useful results • Rapidly get better at doing surveys • Wonderful way to test the questions for the Big Honkin’ Survey 38
  39. 39. Agenda “How many people do I need to ask?” “How many questions can I have?” “What makes a good question?” “How many points in a rating scale?” 39 Caroline Jarrett @cjforms (CC) BY SA-4.0
  40. 40. Caroline Jarrett @cjforms 40 "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett
  41. 41. Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 41
  42. 42. 42 In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0
  43. 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 43
  44. 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 44 A good question works in three ways Appropriate Obvious Interesting
  45. 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 45 Why did you visit our website today? Appropriate Obvious Interesting
  46. 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? 46 In a shop, buying a baby carriage In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate
  47. 47. Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 47
  48. 48. Agenda “How many people do I need to ask?” “How many questions can I have?” “What makes a good question?” “How many points in a rating scale?” 48 Caroline Jarrett @cjforms (CC) BY SA-4.0
  49. 49. Caroline Jarrett @cjforms (CC) BY SA-4.0 Likert had several different types of question in his response formats Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 49
  50. 50. Caroline Jarrett @cjforms (CC) BY SA-4.0 You can find an academic paper to support almost any number of points • Krosnick and Presser refer to over 80 papers 50 Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf
  51. 51. Tip Don’t stress too much about the number of points in your rating scale 51 Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
  52. 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Well, OK, stress a little bit. 52 This scale is downright peculiar. Avoid.
  53. 53. Summary 53 Caroline Jarrett @cjforms (CC) BY SA-4.0
  54. 54. Caroline Jarrett @cjforms (CC) BY SA-4.0 54 The aim is to get the best number you can, within the resources you have
  55. 55. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 55
  56. 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number 56
  57. 57. Caroline Jarrett @cjforms (CC) BY SA-4.0 57 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have Resources What you want to ask about The resources you have The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use
  58. 58. 58 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  59. 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 59

×