SlideShare a Scribd company logo
1 of 69
The Survey
Octopus
Caroline Jarrett
@cjforms
Seminar, University of Auckland
An approach to teaching Total Survey Error
Is there anyone here who…
• Is a survey methodologist?
• Teaches classes in survey methodology?
• Uses surveys in research?
• Responds to surveys?
• Eats octopus?
2
I’m a forms specialist
3
Why do people answer questions?
4Caroline Jarrett @cjforms (CC) BY SA-4.0
People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
5
“Please have a look at this survey?”
6
Kill survey!
Kill! Kill!
7
8
Bad surveys are training us
to ignore all surveys
9
“Please have a look at this survey?”
10
Let’s chat about a
few of the issues
Some questions about TSE
• Do you teach it?
• Do you use it?
11
Survey Methodology Courses and
TSE/Big Data Issues: Classroom Experiences
Among University Instructors
Wojciech Jablonski
University of Lodz | Poland
TSE15 | Baltimore, MD, USA | 22 September 2015
TSE CONCEPT
1%
36%
44%
19%
In what way have you used the Total Survey Error
concept during you survey methods classes?
TSE is a framework of my
survey methods classes
I have discussed the TSE
concept but it is not a
framework of my survey
methods classes
I have not dicsussed the TSE
concept during my survey
methods classes
Unable to say/Other answers
Jablonksi, W. (2015) Survey Methodology Courses and TSE/Big Data Issues, TSE15 Baltimore
14
Survey Methodology. Robert M Groves (ed)
15
Caroline Jarrett @cjforms (CC) BY SA-4.0
16
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
A conversation about measurement
17
18
GDS19
March 2014 version of https://www.gov.uk/service-manual/measurement/user-satisfaction.html
21
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
22
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
First I tried removing all the errors
Caroline Jarrett @cjforms (CC) BY SA-4.0
Then I tried to approximate the
technical terms with plainer ones
ConstructWhat you want
to ask about
Measurement
The questions
you ask
ResponseThe answers
you get
Edited response
The answers
you use
Representation Who you want
to ask
Sampling frame
The list you use
to sample from
Sample The ones you
ask
Respondents The ones who
answer
Post-survey
adjustments
The ones whose
answers you can use
Survey statisticThe number
Caroline Jarrett @cjforms (CC) BY SA-4.0
23
I used this diagram to discuss
satisfaction with colleagues
The number
The sample you use
The ones who answer
The sample you ask
The list you use to
sample from
Who you want
to find out from
Who
The answers you use
The answers you get
The questions
you ask
What you want
to find out about
What
Caroline Jarrett @cjforms (CC) BY SA-4.0
24
25
Services so good that
people prefer to use
them
Satisfaction
Overall, how satisfied
were you with renewing
your car tax today?
Scale of 5 points from
very satisfied to very
dissatisfied
Ratings weighted from
100% for 'very satisfied' to
0% for 'very dissatisfied'
UK population
People who renew car tax
online
Everyone who finishes the
renewal
People who answer the
question
94% The number
(Any exclusions for repeated
answers or test answers?)
The sample you use
The ones who answer
The sample you ask
Everyone who finishes the
renewal
The list you use to
sample from
Who you want
to find out from
Who
The answers you use
The answers you get
The questions
you ask
What you want
to find out about
What
Satisfaction with “renew tax disc”
Caroline Jarrett @cjforms (CC) BY SA-4.0
26
27
We were only asking successful users if they were
satisfied
We were asking users to give feedback from the Done page – a
page you could only reach if you completed using the service
successfully.
This was a good way to make our services look really good, but a
terrible way to get the information needed to improve them.
We were missing out on the most important feedback
– from the users who failed to complete the
transaction or otherwise got stuck.
We’ve changed the guidance in the Service Manual. Now we’re
asking for feedback in many more places.
https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
28
29
Measure satisfaction across the whole service
More often than not, the end of the transaction isn’t the end of the
service. For example, if you claim Carer’s Allowance, the end of the
transaction means you’ve finished filling in your claim. You’re still
waiting for a decision.
You must prompt users to give feedback at service endpoints. In the
example above, this means prompting the user to give feedback
when they get their benefit decision. This could be some time after
they finish their transaction – the prompt could be in an email or a
letter.
You must also allow users to give feedback from anywhere in the
service, in case anything goes wrong.
And you must be able to show that you’re collecting user satisfaction
data appropriately at your Service Assessment.
https://www.gov.uk/service-manual/measurement/user-satisfaction.html
Some other chats about surveys
30
My ‘plain language’ diagram was OK,
but not that different from many others
31
32
“You need a memory hook”
Christine Elgood
@elgoodgames
33
To get better results from your survey,
think about the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
34
“How many people do I need in my
sample?”
To work that out, let’s visit the Octopus
35
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Start with how many will answer
36
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Fieldwork:
Who answers?
Whether they’ll answer depends on effort
37
Questions:
What are you asking about?
How many questions?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
And on the reward you’re offering
Goals and resources:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other incentive?
38
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Then there’s the ‘Justin Bieber North Korea’ problem
39
http://www.bbc.com/news/10506482
If we ask ‘anyone’,
we’ll have extra work here
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Whose answers
can we use?
So it matters where we get our sample
41
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Sample:
the list you
sample from
42
And now it’s easy to work out how many to ask
Sample:
the number of
people to ask
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
We thought about a lot of topics to work that out
Goals
Sample
Questions
Fieldwork
43
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
What about this bit?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
45
“Is this a good question?
46
Caroline Jarrett @cjforms (CC) BY SA-4.0
47
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
https://commons.wikimedia.org/wiki/File:Pho
ne_photography.jpg#/media/File:Phone_phot
ography.jpg
Modified by Caroline Jarrett
A good question gets good answers
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Is the question
easy to answer?
Good answers help you to make
decisions
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Insight:
Is the answer
useful?
All the topics are connected
Goals
Sample
Questions
Fieldwork
50
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
Insight
Response
I tried the Survey Octopus at the
Content Strategy Summit
51
52
People want a process, too
53
54
Questions
Response
Fieldwork
Sample
Goals
Response
Insight
Questionnaires
Caroline Jarrett @cjforms (CC) BY SA-4.0
My survey process has 6 steps
55
Caroline Jarrett @cjforms (CC) BY SA-4.0
A survey is a quantitative method
56
57
To get better results from your survey,
think about the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
58
The aim of a survey is to get a number
that helps you to make a decision
Caroline Jarrett @cjforms (CC) BY SA-4.0
Some options for talking about
Total Survey Error
59
1. As a Survey Octopus.
60
61
To get better results from your survey,
think about the Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
2. As a more conventional diagram
62
The aim is to get the best number you
can, within the resources you have
What you want to ask about
The resources you have
The questions you ask
The answers you get
The answers you use
The number
Who you want to ask
The list that you sample from
The sample you ask
The ones who answer
The ones whose answers
you can use
63
Caroline Jarrett @cjforms (CC) BY SA-4.0
3. As a link to the terms used by
survey methodologists (searchable)
64
65
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
The aim is to get the best number you
can, within the resources you have
What you want
to ask about
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
The number
Caroline Jarrett @cjforms (CC) BY SA-4.0
ResourcesThe resources
you have
4. With all the errors in it
66
67
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
The aim is to get the best statistic you
can, within the resources you have
Measurement
error
Processing
error
Coverage error
Sampling error
Nonresponse
error
Adjustment error
Caroline Jarrett @cjforms (CC) BY SA-4.0
(Lack of)
Validity
68
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
Caroline Jarrett
Twitter @cjforms
http://www.slideshare.net/cjforms
carolinej@effortmark.co.uk
69
Caroline Jarrett @cjforms (CC) BY SA-4.0

More Related Content

Viewers also liked

Octopus Microfinance Suite Functionalities Overview
Octopus Microfinance Suite Functionalities OverviewOctopus Microfinance Suite Functionalities Overview
Octopus Microfinance Suite Functionalities Overviewvincent.biot
 
Installation Guide - Octopus
Installation Guide - OctopusInstallation Guide - Octopus
Installation Guide - Octopusvincent.biot
 
6305 Group2 Presentation 20080924
6305 Group2 Presentation 200809246305 Group2 Presentation 20080924
6305 Group2 Presentation 20080924infocycle
 
Loyalty Programme Marketing
Loyalty Programme MarketingLoyalty Programme Marketing
Loyalty Programme MarketingHuw Hopkin
 

Viewers also liked (6)

Octopus
OctopusOctopus
Octopus
 
Octopus Microfinance Suite Functionalities Overview
Octopus Microfinance Suite Functionalities OverviewOctopus Microfinance Suite Functionalities Overview
Octopus Microfinance Suite Functionalities Overview
 
Installation Guide - Octopus
Installation Guide - OctopusInstallation Guide - Octopus
Installation Guide - Octopus
 
6305 Group2 Presentation 20080924
6305 Group2 Presentation 200809246305 Group2 Presentation 20080924
6305 Group2 Presentation 20080924
 
Loyalty Programme Marketing
Loyalty Programme MarketingLoyalty Programme Marketing
Loyalty Programme Marketing
 
01
0101
01
 

More from Caroline Jarrett

Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024Caroline Jarrett
 
Using surveys to uncover audience insights
Using surveys to uncover audience insightsUsing surveys to uncover audience insights
Using surveys to uncover audience insightsCaroline Jarrett
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptxCaroline Jarrett
 
The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...Caroline Jarrett
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Caroline Jarrett
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptxCaroline Jarrett
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023Caroline Jarrett
 
Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Caroline Jarrett
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxCaroline Jarrett
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptxCaroline Jarrett
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxCaroline Jarrett
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxCaroline Jarrett
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxCaroline Jarrett
 
Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022Caroline Jarrett
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
 
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorSurveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
 

More from Caroline Jarrett (20)

Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024
 
Using surveys to uncover audience insights
Using surveys to uncover audience insightsUsing surveys to uncover audience insights
Using surveys to uncover audience insights
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx
 
The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptx
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptx
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023
 
Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptx
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptx
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptx
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptx
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference call
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptx
 
Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3
 
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorSurveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
 

Recently uploaded

社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeralNABLAS株式会社
 
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam DunksNOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam Dunksgmuir1066
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...ssuserf63bd7
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp onlinebalibahu1313
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Jon Hansen
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxDilipVasan
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group MeetingAlison Pitt
 
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一0uyfyq0q4
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyRafigAliyev2
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfRobertoOcampo24
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理pyhepag
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Calllward7
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonPayment Village
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfEmmanuel Dauda
 
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一hwhqz6r1y
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...Amil baba
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理pyhepag
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理cyebo
 

Recently uploaded (20)

社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeral
 
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam DunksNOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
如何办理滑铁卢大学毕业证(Waterloo毕业证)成绩单本科学位证原版一比一
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdf
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一
如何办理澳洲悉尼大学毕业证(USYD毕业证书)学位证成绩单原版一比一
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
Machine Learning for Accident Severity Prediction
Machine Learning for Accident Severity PredictionMachine Learning for Accident Severity Prediction
Machine Learning for Accident Severity Prediction
 
123.docx. .
123.docx.                                 .123.docx.                                 .
123.docx. .
 
一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
 

Survey Octopus Uni of Auckland cjforms

  • 1. The Survey Octopus Caroline Jarrett @cjforms Seminar, University of Auckland An approach to teaching Total Survey Error
  • 2. Is there anyone here who… • Is a survey methodologist? • Teaches classes in survey methodology? • Uses surveys in research? • Responds to surveys? • Eats octopus? 2
  • 3. I’m a forms specialist 3
  • 4. Why do people answer questions? 4Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 5. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” 5
  • 6. “Please have a look at this survey?” 6 Kill survey! Kill! Kill!
  • 7. 7
  • 8. 8
  • 9. Bad surveys are training us to ignore all surveys 9
  • 10. “Please have a look at this survey?” 10 Let’s chat about a few of the issues
  • 11. Some questions about TSE • Do you teach it? • Do you use it? 11
  • 12. Survey Methodology Courses and TSE/Big Data Issues: Classroom Experiences Among University Instructors Wojciech Jablonski University of Lodz | Poland TSE15 | Baltimore, MD, USA | 22 September 2015
  • 13. TSE CONCEPT 1% 36% 44% 19% In what way have you used the Total Survey Error concept during you survey methods classes? TSE is a framework of my survey methods classes I have discussed the TSE concept but it is not a framework of my survey methods classes I have not dicsussed the TSE concept during my survey methods classes Unable to say/Other answers Jablonksi, W. (2015) Survey Methodology Courses and TSE/Big Data Issues, TSE15 Baltimore
  • 14. 14
  • 15. Survey Methodology. Robert M Groves (ed) 15 Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 16. 16 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  • 17. A conversation about measurement 17
  • 18. 18
  • 19. GDS19
  • 20. March 2014 version of https://www.gov.uk/service-manual/measurement/user-satisfaction.html
  • 21. 21 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  • 22. 22 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct First I tried removing all the errors Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 23. Then I tried to approximate the technical terms with plainer ones ConstructWhat you want to ask about Measurement The questions you ask ResponseThe answers you get Edited response The answers you use Representation Who you want to ask Sampling frame The list you use to sample from Sample The ones you ask Respondents The ones who answer Post-survey adjustments The ones whose answers you can use Survey statisticThe number Caroline Jarrett @cjforms (CC) BY SA-4.0 23
  • 24. I used this diagram to discuss satisfaction with colleagues The number The sample you use The ones who answer The sample you ask The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Caroline Jarrett @cjforms (CC) BY SA-4.0 24
  • 25. 25 Services so good that people prefer to use them Satisfaction Overall, how satisfied were you with renewing your car tax today? Scale of 5 points from very satisfied to very dissatisfied Ratings weighted from 100% for 'very satisfied' to 0% for 'very dissatisfied' UK population People who renew car tax online Everyone who finishes the renewal People who answer the question 94% The number (Any exclusions for repeated answers or test answers?) The sample you use The ones who answer The sample you ask Everyone who finishes the renewal The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Satisfaction with “renew tax disc” Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 26. 26
  • 27. 27 We were only asking successful users if they were satisfied We were asking users to give feedback from the Done page – a page you could only reach if you completed using the service successfully. This was a good way to make our services look really good, but a terrible way to get the information needed to improve them. We were missing out on the most important feedback – from the users who failed to complete the transaction or otherwise got stuck. We’ve changed the guidance in the Service Manual. Now we’re asking for feedback in many more places. https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
  • 28. 28
  • 29. 29 Measure satisfaction across the whole service More often than not, the end of the transaction isn’t the end of the service. For example, if you claim Carer’s Allowance, the end of the transaction means you’ve finished filling in your claim. You’re still waiting for a decision. You must prompt users to give feedback at service endpoints. In the example above, this means prompting the user to give feedback when they get their benefit decision. This could be some time after they finish their transaction – the prompt could be in an email or a letter. You must also allow users to give feedback from anywhere in the service, in case anything goes wrong. And you must be able to show that you’re collecting user satisfaction data appropriately at your Service Assessment. https://www.gov.uk/service-manual/measurement/user-satisfaction.html
  • 30. Some other chats about surveys 30
  • 31. My ‘plain language’ diagram was OK, but not that different from many others 31
  • 32. 32 “You need a memory hook” Christine Elgood @elgoodgames
  • 33. 33 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 34. 34 “How many people do I need in my sample?”
  • 35. To work that out, let’s visit the Octopus 35 Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 36. Start with how many will answer 36 Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork: Who answers?
  • 37. Whether they’ll answer depends on effort 37 Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 38. And on the reward you’re offering Goals and resources: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 38 Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 39. Then there’s the ‘Justin Bieber North Korea’ problem 39 http://www.bbc.com/news/10506482
  • 40. If we ask ‘anyone’, we’ll have extra work here Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Whose answers can we use?
  • 41. So it matters where we get our sample 41 Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the list you sample from
  • 42. 42 And now it’s easy to work out how many to ask Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 43. We thought about a lot of topics to work that out Goals Sample Questions Fieldwork 43 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response
  • 44. What about this bit? Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 45. 45 “Is this a good question?
  • 47. 47 "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett
  • 48. A good question gets good answers Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Is the question easy to answer?
  • 49. Good answers help you to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0 Insight: Is the answer useful?
  • 50. All the topics are connected Goals Sample Questions Fieldwork 50 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
  • 51. I tried the Survey Octopus at the Content Strategy Summit 51
  • 52. 52
  • 53. People want a process, too 53
  • 55. My survey process has 6 steps 55 Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 56. A survey is a quantitative method 56
  • 57. 57 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 58. 58 The aim of a survey is to get a number that helps you to make a decision Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 59. Some options for talking about Total Survey Error 59
  • 60. 1. As a Survey Octopus. 60
  • 61. 61 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 62. 2. As a more conventional diagram 62
  • 63. The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 63 Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 64. 3. As a link to the terms used by survey methodologists (searchable) 64
  • 65. 65 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have What you want to ask about The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number Caroline Jarrett @cjforms (CC) BY SA-4.0 ResourcesThe resources you have
  • 66. 4. With all the errors in it 66
  • 67. 67 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best statistic you can, within the resources you have Measurement error Processing error Coverage error Sampling error Nonresponse error Adjustment error Caroline Jarrett @cjforms (CC) BY SA-4.0 (Lack of) Validity
  • 68. 68 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.

Editor's Notes

  1. Start of a blog post for Government Digital Service: "My new favourite form. Really" https://gds.blog.gov.uk/2015/09/22/my-new-favourite-form-really/
  2. An email from Virgin Atlantic: Tell us all about it Hello Thank you for choosing Virgin Atlantic for your recent flight. We're always looking to improve our service. So it would be great if you could share your experience by completing our survey. It'll take about 20 minutes and you can complete it on PCs, tablets and smartphones. Don't worry if you get interrupted, you can come back later and continue where you left off. If you've already completed a survey about a specific part of your journey, we're really grateful. Now that you've completed your trip, we'd love to hear your feedback about your experience as a whole, so please let us know your thoughts using the button below. Thanks for your time. Best wishes. The Virgin Atlantic team
  3. virgin atlantic Survey Offline Thank you for clicking through to this survey. The survey is now closed and has been taken offline.
  4. In what way have you used the Total Survey Error concept during you survey methods classes? 1% TSE is a framework of my survey methods classes 36% I have discussed the TSE concept but it is not a framework of my survey methods classes 44% I have not discussed the TSE concept during my survey methods classes 19% unable to sa/other answers
  5. STATS 740 Sample Surveys (15 points) Taught: Semester Two, City Campus. Prerequisites: STATS 340 or 341, or equivalent. Assessment: Final exam 60%; coursework (four assignments) 40%. Textbooks: Recommended: Survey Methodology. Robert M Groves (ed), Wiley, 2004; T. Lumley, Complex Surveys: a guide to analysis using R, Wiley Interscience, 2009; Lu Ann Aday & L.J. Cornelius, Designing and Conducting Health Surveys. 3rd. Edition, Jossey-Bass, 2006 For advice: Chris Wild (ext 88797), Nichola Shackleton (ext 89894) STATS 740 is a course in the design, management and analysis of sample surveys, with Particular emphasis on surveys in the health sector. It is useful for students who are interested in either undertaking surveys, or analysing data collected from surveys. Topics studied include: Revision of statistical aspects of sampling. Preparing surveys. Research design: methodology and data collection; issues of sample design and sample selection. Conducting surveys: questionnaires and questions; non-sampling issues; project management; maintaining data quality. Analysis: computer software, weighting, special features of survey data. Dissemination. Other types of surveys.
  6. The full survey lifecycle from a Total Survey Error perspective, showing the seven types of error
  7. A page from gov.uk showing the performance of its online tax renewal service - including a figure of 94 percent for user satisfaction. We need to understand total survey error to assess whether this figure is accurate or useful in assessing true performance.
  8. Excerpt from the GDS Service Design Manual page on User Satisfaction https://www.gov.uk/service-manual/measurement/user-satisfaction.html As it was in March 2014. This excerpt reads: A good service makes it possible for users to successfully complete tasks. User satisfaction helps you to measure the overall quality of the service experience. Many government transactions are mandatory for users, so they’re not naturally enjoyable, and are sometimes referred to as grudge transactions. Still, try to make them as pleasant as possible for users, who may be nervous or stressed when interacting with the government. Asking users how satisfied they are with a service can provide you with a measure of all the parts that contribute to the overall user experience, like ease of use, navigation and design. What you will be measuring Define user satisfaction as the percentage of people who answered either “very satisfied” or “satisfied” on a 5-point scale in response to this question: Q: Overall, how satisfied were you with this [eg car tax] service today? very satisfied satisfied neither satisfied or dissatisfied dissatisfied very dissatisfied
  9. Further excerpt from https://www.gov.uk/service-manual/measurement/user-satisfaction.html As it was in March 2014 Asking users how satisfied they are with a service can provide you with a measure of all the parts that contribute to the overall user experience, like ease of use, navigation and design. What you will be measuring Define user satisfaction as the percentage of people who answered either “very satisfied” or “satisfied” on a 5-point scale in response to this question: Q: Overall, how satisfied were you with this [eg car tax] service today? very satisfied satisfied neither satisfied or dissatisfied dissatisfied very dissatisfied How to measure user satisfaction GOV.UK will provide a user satisfaction survey at the end of your transactional service and make this data available. What happens when a user exits the transaction midway through Measure all user journeys through your transaction to better understand dropout points. If a user drops out you should try and collect some data around the quality of your service to explain why they didn’t complete the transaction. For example, ‘please tell us why are you unable to complete this transaction.’ Although this survey will not be contribute to the overall measure of user satisfaction it will help you understand service dropout points and find ways to improve transactions.
  10. The full survey lifecycle from a Total Survey Error perspective, showing the seven types of error
  11. This survey lifecycle has the technical terms used by the survey methodologists On the left: Construct Measurement Response Edited response Ending with survey statistics. On the right: Representation Sampling frame Sample Respondents Post-survey adjustments And then also ending at Survey statistics
  12. This slide translates the technical terms into plain language What you want to ask about = Construct The questions you ask = Measurement The answers you get = Response The answers you use = Edited response The number = Survey statistic Who you want to ask = Representation The list you use to sample from = Sampling frame The ones you ask = Sample The ones who answer = Respondents The ones whose answers you can use = Post
  13. https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
  14. Measure satisfaction across the whole service
  15. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  16. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  17. The octopus again. This time we're looking at 'the questions we ask'.
  18. The resources you have will help you to decide on the reward you’re offering
  19. Prank leaves Justin Bieber facing tour of North Korea By Daniel Emery Technology reporter, BBC News 5 July 2010 Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea. A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on. There are now almost half a million votes to send the singer to the secretive communist nation. The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days. Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
  20. The octopus, with focus on 'The list you sample from'
  21. The octopus again; we've looked at 6 of the 8 tentacles.
  22. Two questions from a survey: '24: Do you use a Windows or Mac computer'? '25. What is your gender'?
  23. Photo of a Samsung (Android) mobile with the same questions as previous slide. If you only have an Android mobile, how do you answer ‘do you use a Windows or Mac computer’ when the answer options are ‘Windows’, ‘Mac’ and ‘Both’?
  24. The octopus again; we've looked at all the tentacles
  25. Tweet from @jberzowska: “I'm entirely charmed by with the survey Octopus from @cjforms, a charming checklist for survey planning. #cssummit” https://twitter.com/jberzowska/status/645995337522372608
  26. Goals: Establish your goals for the survey, leading to questions that you need answers to Sample: Decide who to ask and how many, leading to people who you will invite to answer Questions: Test the questions and build the questionnaire, leading to a questionnaire Fieldwork: Run the survey from invitation to follow-up, leading to people who actually answer Response: Clean the data, leading to answers Insight: Analyse and present the results, leading to decisions
  27. The Survey sits between what you want to ask about, who you want to ask, and the number
  28. This is a more conventional way of looking at the octopus tentacles
  29. This slide translates the technical terms into plain language What you want to ask about = Construct The questions you ask = Measurement The answers you get = Response The answers you use = Edited response The number = Survey statistic Who you want to ask = Representation The list you use to sample from = Sampling frame The ones you ask = Sample The ones who answer = Respondents The ones whose answers you can use = Post
  30. This version adds in all the types of error Between construct and measurement: (lack of) validity Between measurement and response: measurement error Between response and edited response: processing error Between representation and sampling frame: coverage error Between sampling frame and sampling: sampling error Between sample and respondents: nonresponse error Between respondents and post-survey adjustments: adjustment error