Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Survey Octopus
Getting valid data from surveys
Caroline Jarrett
@cjforms
2016 #uxinthecity
Caroline Jarrett @cjforms
What would you do for a dollar?
2
Caroline Jarrett @cjforms (CC) BY SA-4.0
$1 in the envelope beats $10 (guaranteed) later
Caroline Jarrett @cjforms (CC) BY...
Caroline Jarrett @cjforms (CC) BY SA-4.0
4
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward, and trust
People will only respond if they tru...
Caroline Jarrett @cjforms (CC) BY SA-4.0
I’m a forms specialist
Image credit: Flickr, taxrebate.org.uk
6
Caroline Jarrett @cjforms (CC) BY SA-4.0
Why do people answer questions?
Caroline Jarrett @cjforms (CC) BY SA-4.0
7
Caroline Jarrett @cjforms
“Please have a look at this survey”
8
Caroline Jarrett @cjforms
The survey is a
systematic method
for gathering information from
(a sample of) entities
for the ...
Caroline Jarrett @cjforms
The survey is a
process
for gathering information from
(a sample of) entities
for the purpose of...
Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) entities
for the pur...
Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpo...
Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpo...
Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpo...
Caroline Jarrett @cjforms
To make decisions People
The survey is a
process for getting
answers to questions
getting numbers
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim of a survey is to get a number
that helps you to make a decision
16
Caroline Jarrett @cjforms
“The numbers” or “The number”?
17
Caroline Jarrett @cjforms (CC) BY SA-4.0
Fieldwork used to be expensive
so a survey was a rare event.
Image credit: http:/...
Caroline Jarrett @cjforms (CC) BY SA-4.0
1950s mindset: “Ask Everything”
Survey =
Big Honkin’ Survey
19
Caroline Jarrett @cjforms (CC) BY SA-4.0
2016 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• As...
Tip Interview first
Caroline Jarrett @cjforms (CC) BY SA-4.0
21
Caroline Jarrett @cjforms
One way to
iterate,
improve,
increase
Time for new question
Caroline Jarrett @cjforms (CC) BY SA...
Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim of a survey is to get a number
that helps you to make a decision
23
Caroline Jarrett @cjforms
24
Caroline Jarrett @cjforms
Asking
the right
question
Asking the
right people
25
Caroline Jarrett @cjforms
26
Caroline Jarrett @cjforms
http://www.bbc.com/news/10506482 27
Caroline Jarrett @cjforms (CC) BY SA-4.0
Asking the right people is better
than asking lots of people Sample:
the list you...
Caroline Jarrett @cjforms
29
Caroline Jarrett @cjforms (CC) BY SA-4.0
There’s often a ‘zone of indifference’
Hate it Love it
30
Caroline Jarrett @cjforms (CC) BY SA-4.0
Feeling special is rewarding
People will only respond if they trust
you. After th...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Ask fewer people to get better response
Sample:
the number of
people to ask
Carol...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Try mail, phone or face-to-face for
better response rates
Fieldwork:
Who answers?...
Did we get answers from
the right people?
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
34
Population of assorted birds
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
35
Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t just want answers,
we want representative answers
Response
Caroline Jarr...
Caroline Jarrett @cjforms
Asking
the right
question
Asking the
right people
37
Caroline Jarrett @cjforms
Asking one
person the
right question
is better than
Asking 10,000
people the
wrong question
38
Caroline Jarrett @cjforms
39
Caroline Jarrett @cjforms
In your last five days at work, what
percentage of your work time do you
estimate that you spent...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Helps a lot if you ask good questions
Questions:
What are you
asking about?
How m...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward and trust
People will only respond if they trus...
Caroline Jarrett @cjforms (CC) BY SA-4.0
A good question works in three ways
Obvious Interesting
Appropriate
43
Caroline Jarrett @cjforms (CC) BY SA-4.0
Why did you visit our website today?
Obvious Interesting
Appropriate
44
Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to
a friend or family member?
In a shop,
buying a
baby bug...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to
a friend or family member?
In a shop,
buying a
baby bug...
Tip
Test your questions by
interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0
47
Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t want any old answers.
We want useful answers.
Response
Caroline Jarrett ...
Caroline Jarrett @cjforms
In your last five days at work, what
percentage of your work time do you
estimate that you spent...
Caroline Jarrett @cjforms
On your most recent working day,
what percentage of time did you spend
using the Internet?
%
Car...
Caroline Jarrett @cjforms (CC) BY SA-4.0
The approximate curve of forgetting
Major life event
Noticeable, occasional
Unrem...
Caroline Jarrett @cjforms (CC) BY SA-4.0
Can’t remember = can’t answer
Response
Caroline Jarrett @cjforms (CC) BY SA-4.0
52
Caroline Jarrett @cjforms (CC) BY SA-4.0
Insights are the answers that you use
to make the decisions
Insights Caroline Jar...
Caroline Jarrett @cjformsCaroline Jarrett @cjforms (CC) BY SA-4.0
54
Caroline Jarrett @cjforms
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
htt...
Tip
Always allow for ‘other’
Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk
56
Caroline Jarrett @cjforms
Can we make the decision yet?
Can we make the decision yet?
Can we make the decision yet?
Caroli...
Tip
Try writing your
presentation first
58
Caroline Jarrett @cjforms (CC) BY SA-4.0
We’ve met the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
59
What could possibly go wrong?
Caroline Jarrett @cjforms (CC) BY SA-4.0
60
Caroline Jarrett @cjforms (CC) BY SA-4.0
Caroline Jarrett
Twitter @cjforms
http://www.slideshare.net/cjforms
carolinej@eff...
Upcoming SlideShare
Loading in …5
×

The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016

Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016

  • Be the first to comment

The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016

  1. 1. The Survey Octopus Getting valid data from surveys Caroline Jarrett @cjforms 2016 #uxinthecity
  2. 2. Caroline Jarrett @cjforms What would you do for a dollar? 2
  3. 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 $1 in the envelope beats $10 (guaranteed) later Caroline Jarrett @cjforms (CC) BY SA-4.0 3
  4. 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 4
  5. 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
  6. 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 I’m a forms specialist Image credit: Flickr, taxrebate.org.uk 6
  7. 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why do people answer questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 7
  8. 8. Caroline Jarrett @cjforms “Please have a look at this survey” 8
  9. 9. Caroline Jarrett @cjforms The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
  10. 10. Caroline Jarrett @cjforms The survey is a process for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  11. 11. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  12. 12. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  13. 13. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers of the attributes of the larger population of which the entities are members.
  14. 14. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions
  15. 15. Caroline Jarrett @cjforms To make decisions People The survey is a process for getting answers to questions getting numbers
  16. 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 16
  17. 17. Caroline Jarrett @cjforms “The numbers” or “The number”? 17
  18. 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork used to be expensive so a survey was a rare event. Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/ 18
  19. 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 1950s mindset: “Ask Everything” Survey = Big Honkin’ Survey 19
  20. 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 2016 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 20
  21. 21. Tip Interview first Caroline Jarrett @cjforms (CC) BY SA-4.0 21
  22. 22. Caroline Jarrett @cjforms One way to iterate, improve, increase Time for new question Caroline Jarrett @cjforms (CC) BY SA4.0 22
  23. 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 23
  24. 24. Caroline Jarrett @cjforms 24
  25. 25. Caroline Jarrett @cjforms Asking the right question Asking the right people 25
  26. 26. Caroline Jarrett @cjforms 26
  27. 27. Caroline Jarrett @cjforms http://www.bbc.com/news/10506482 27
  28. 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 Asking the right people is better than asking lots of people Sample: the list you sample from Caroline Jarrett @cjforms (CC) BY SA-4.0 28
  29. 29. Caroline Jarrett @cjforms 29
  30. 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 There’s often a ‘zone of indifference’ Hate it Love it 30
  31. 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Feeling special is rewarding People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 31
  32. 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 Ask fewer people to get better response Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 32
  33. 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try mail, phone or face-to-face for better response rates Fieldwork: Who answers? Caroline Jarrett @cjforms (CC) BY SA-4.0 33
  34. 34. Did we get answers from the right people? Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 34
  35. 35. Population of assorted birds Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 35
  36. 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t just want answers, we want representative answers Response Caroline Jarrett @cjforms (CC) BY SA-4.0 36
  37. 37. Caroline Jarrett @cjforms Asking the right question Asking the right people 37
  38. 38. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 38
  39. 39. Caroline Jarrett @cjforms 39
  40. 40. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 40
  41. 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 Helps a lot if you ask good questions Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 41
  42. 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 42
  43. 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 A good question works in three ways Obvious Interesting Appropriate 43
  44. 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why did you visit our website today? Obvious Interesting Appropriate 44
  45. 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 45
  46. 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 46
  47. 47. Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 47
  48. 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t want any old answers. We want useful answers. Response Caroline Jarrett @cjforms (CC) BY SA-4.0 48
  49. 49. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 49
  50. 50. Caroline Jarrett @cjforms On your most recent working day, what percentage of time did you spend using the Internet? % Caroline Jarrett @cjforms (CC) BY SA-4.0 50
  51. 51. Caroline Jarrett @cjforms (CC) BY SA-4.0 The approximate curve of forgetting Major life event Noticeable, occasional Unremarkable, repetitive Perfect recall Day Gone Week Month YearHour 51
  52. 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Can’t remember = can’t answer Response Caroline Jarrett @cjforms (CC) BY SA-4.0 52
  53. 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights are the answers that you use to make the decisions Insights Caroline Jarrett @cjforms (CC) BY SA-4.0 53
  54. 54. Caroline Jarrett @cjformsCaroline Jarrett @cjforms (CC) BY SA-4.0 54
  55. 55. Caroline Jarrett @cjforms "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett 55
  56. 56. Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 56
  57. 57. Caroline Jarrett @cjforms Can we make the decision yet? Can we make the decision yet? Can we make the decision yet? Caroline Jarrett @cjforms (CC) BY SA4.0 57
  58. 58. Tip Try writing your presentation first 58
  59. 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 We’ve met the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0 59
  60. 60. What could possibly go wrong? Caroline Jarrett @cjforms (CC) BY SA-4.0 60
  61. 61. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 61

×