SlideShare a Scribd company logo

Surveys that work: a webinar for UXPA 2021

Many of us receive multiple requests to complete surveys every day. Some of us find that colleagues or clients think of ‘doing a survey’ as the same as ‘doing some research’ – which may explain why organizations send out so many survey requests. In this webinar, you’ll meet the Survey Octopus, Caroline Jarrett’s friendly way of talking about the many issues that make surveys one of the most challenging research methods. The Survey Octopus will help you to: Explain to colleagues that a survey may not be the first research method to try Help to justify a choice to work with a “non significant” number of responses Think about the steps that go into delivering a survey that works As a bonus, Caroline will also explain how her Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.

1 of 67
Download to read offline
Surveys that work
An introduction to
the Survey Octopus
and Total Survey Error
Caroline Jarrett
@cjforms
#UXPA2021
What would you do for a dollar?
Caroline Jarrett @cjforms (CC) BY SA-4.0 2
Caroline Jarrett @cjforms (CC) BY SA-4.0 3
$1 in the envelope beats $10 guaranteed later
Caroline Jarrett @cjforms (CC) BY SA-4.0 3
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Response
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 4
Would you
answer this
survey?
Caroline Jarrett @cjforms (CC) BY SA-4.0 5
Caroline Jarrett @cjforms (CC) BY SA-4.0 6
I’m the forms specialist
Image credit: Flickr, taxrebate.org.uk 6
Ad

Recommended

Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021Caroline Jarrett
 
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorSurveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
 
Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Caroline Jarrett
 
Survey that work LibDesign2016 @cjforms
Survey that work LibDesign2016 @cjformsSurvey that work LibDesign2016 @cjforms
Survey that work LibDesign2016 @cjformsCaroline Jarrett
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Caroline Jarrett
 
Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020Caroline Jarrett
 

More Related Content

Similar to Surveys that work: a webinar for UXPA 2021

How to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusHow to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusCaroline Jarrett
 
The inevitable survey 2021
The inevitable survey 2021The inevitable survey 2021
The inevitable survey 2021Caroline Jarrett
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptxCaroline Jarrett
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxCaroline Jarrett
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
 
Surveys that work EBI_2017
Surveys that work EBI_2017Surveys that work EBI_2017
Surveys that work EBI_2017Caroline Jarrett
 
Survey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsSurvey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsCaroline Jarrett
 
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSTHE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSUX New Zealand 2015
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
 
The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...Caroline Jarrett
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
 
Survey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjformsSurvey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjformsCaroline Jarrett
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023Caroline Jarrett
 
Social Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterSocial Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterJonathas Magalhães
 
Forms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsForms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsCaroline Jarrett
 
Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Caroline Jarrett
 
A deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxA deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxCaroline Jarrett
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxCaroline Jarrett
 
Getting supporters active
Getting supporters activeGetting supporters active
Getting supporters activeFairSay
 

Similar to Surveys that work: a webinar for UXPA 2021 (20)

How to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusHow to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey Octopus
 
The inevitable survey 2021
The inevitable survey 2021The inevitable survey 2021
The inevitable survey 2021
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptx
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1
 
Surveys that work EBI_2017
Surveys that work EBI_2017Surveys that work EBI_2017
Surveys that work EBI_2017
 
Survey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsSurvey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjforms
 
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSTHE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2
 
The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptx
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3
 
Survey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjformsSurvey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjforms
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023
 
Social Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterSocial Query: A Query Routing System for Twitter
Social Query: A Query Routing System for Twitter
 
Forms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsForms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjforms
 
Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Design tips for surveys UIE 2012
Design tips for surveys UIE 2012
 
A deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxA deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLx
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptx
 
Getting supporters active
Getting supporters activeGetting supporters active
Getting supporters active
 

More from Caroline Jarrett

The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...Caroline Jarrett
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptxCaroline Jarrett
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptxCaroline Jarrett
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxCaroline Jarrett
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxCaroline Jarrett
 
A chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsA chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsCaroline Jarrett
 
Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Caroline Jarrett
 
Usability evaluation of forms 1998
Usability evaluation of forms 1998Usability evaluation of forms 1998
Usability evaluation of forms 1998Caroline Jarrett
 
Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Caroline Jarrett
 
Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Caroline Jarrett
 
Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
 
Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Caroline Jarrett
 
Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Caroline Jarrett
 
Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Caroline Jarrett
 
Discussing a Topic Map for How to Design a Better Form 2019 March
Discussing a Topic Map for How to Design a Better Form 2019 MarchDiscussing a Topic Map for How to Design a Better Form 2019 March
Discussing a Topic Map for How to Design a Better Form 2019 MarchCaroline Jarrett
 

More from Caroline Jarrett (15)

The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptx
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptx
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptx
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptx
 
A chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsA chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjforms
 
Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...
 
Usability evaluation of forms 1998
Usability evaluation of forms 1998Usability evaluation of forms 1998
Usability evaluation of forms 1998
 
Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016
 
Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...
 
Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010
 
Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000
 
Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009
 
Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007
 
Discussing a Topic Map for How to Design a Better Form 2019 March
Discussing a Topic Map for How to Design a Better Form 2019 MarchDiscussing a Topic Map for How to Design a Better Form 2019 March
Discussing a Topic Map for How to Design a Better Form 2019 March
 

Recently uploaded

Communicative Grammar English .pnnnnkkptx
Communicative Grammar English .pnnnnkkptxCommunicative Grammar English .pnnnnkkptx
Communicative Grammar English .pnnnnkkptxRawalRafiqLeghari
 
Heathers Symbols for Gerald Freedman Theatre.pdf
Heathers Symbols for Gerald Freedman Theatre.pdfHeathers Symbols for Gerald Freedman Theatre.pdf
Heathers Symbols for Gerald Freedman Theatre.pdfLizShek
 
SECMOL School In Leh- A Role Model of Vernacular , Passive and Sustainable H...
SECMOL School In Leh- A  Role Model of Vernacular , Passive and Sustainable H...SECMOL School In Leh- A  Role Model of Vernacular , Passive and Sustainable H...
SECMOL School In Leh- A Role Model of Vernacular , Passive and Sustainable H...JIT KUMAR GUPTA
 
Land Management and sourcing land for sustainable Urban Development.
Land Management and sourcing land for sustainable Urban Development.Land Management and sourcing land for sustainable Urban Development.
Land Management and sourcing land for sustainable Urban Development.JIT KUMAR GUPTA
 
Bashundhara Residential _ Analysis and Case Study-MaHi.pptx
Bashundhara Residential _ Analysis and Case Study-MaHi.pptxBashundhara Residential _ Analysis and Case Study-MaHi.pptx
Bashundhara Residential _ Analysis and Case Study-MaHi.pptxTafshirul Alam
 
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.ppt
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.pptsustainable-interior-design-strategy-with-case-study_TANIA-BERA.ppt
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.pptAr. TANIA BERA
 
Hill Area development- Issues and Options
Hill Area development- Issues and OptionsHill Area development- Issues and Options
Hill Area development- Issues and OptionsJIT KUMAR GUPTA
 
Mother Courage and Her Children Shop Order.pdf
Mother Courage and Her Children Shop Order.pdfMother Courage and Her Children Shop Order.pdf
Mother Courage and Her Children Shop Order.pdfLizShek
 
Friendship Hospital _ Case Study-MaHi.pptx
Friendship Hospital _ Case Study-MaHi.pptxFriendship Hospital _ Case Study-MaHi.pptx
Friendship Hospital _ Case Study-MaHi.pptxTafshirul Alam
 
Ordering Principles in Architecture-MaHi.pptx
Ordering Principles in Architecture-MaHi.pptxOrdering Principles in Architecture-MaHi.pptx
Ordering Principles in Architecture-MaHi.pptxTafshirul Alam
 
Amadeus Hang and Cable Layouts UNCSA.pdf
Amadeus Hang and Cable Layouts UNCSA.pdfAmadeus Hang and Cable Layouts UNCSA.pdf
Amadeus Hang and Cable Layouts UNCSA.pdfLizShek
 
HTM Example Layouts Freedman Theatre.pdf
HTM Example Layouts Freedman Theatre.pdfHTM Example Layouts Freedman Theatre.pdf
HTM Example Layouts Freedman Theatre.pdfLizShek
 
Task Pack 1- Nature - Ula Academics KIDS.pdf
Task Pack 1- Nature - Ula Academics KIDS.pdfTask Pack 1- Nature - Ula Academics KIDS.pdf
Task Pack 1- Nature - Ula Academics KIDS.pdfUla Academics
 
Copy of VSST102 Digital Portfolio.pdffff
Copy of VSST102 Digital Portfolio.pdffffCopy of VSST102 Digital Portfolio.pdffff
Copy of VSST102 Digital Portfolio.pdffffschell
 
Studio Surfaced - Textile Design Studio based out of Delhi NCR
Studio Surfaced - Textile Design Studio based out of Delhi NCRStudio Surfaced - Textile Design Studio based out of Delhi NCR
Studio Surfaced - Textile Design Studio based out of Delhi NCRSurfaced - Designs By Meera Narula
 
VSST103 - Digital Portfolio.pdf character
VSST103 - Digital Portfolio.pdf characterVSST103 - Digital Portfolio.pdf character
VSST103 - Digital Portfolio.pdf characterschell
 
urban design case study of old city Jodhpur
urban design case study of  old city Jodhpururban design case study of  old city Jodhpur
urban design case study of old city JodhpurSreyaseeNath1
 
Interior Landscape and Landscape Features Extraction
Interior Landscape and Landscape Features ExtractionInterior Landscape and Landscape Features Extraction
Interior Landscape and Landscape Features ExtractionNityaSharma77
 
Mother Courage and Her Children Layouts.pdf
Mother Courage and Her Children Layouts.pdfMother Courage and Her Children Layouts.pdf
Mother Courage and Her Children Layouts.pdfLizShek
 
Passing Strange Lighting Plot AS BUILT).pdf
Passing Strange Lighting Plot AS BUILT).pdfPassing Strange Lighting Plot AS BUILT).pdf
Passing Strange Lighting Plot AS BUILT).pdfLizShek
 

Recently uploaded (20)

Communicative Grammar English .pnnnnkkptx
Communicative Grammar English .pnnnnkkptxCommunicative Grammar English .pnnnnkkptx
Communicative Grammar English .pnnnnkkptx
 
Heathers Symbols for Gerald Freedman Theatre.pdf
Heathers Symbols for Gerald Freedman Theatre.pdfHeathers Symbols for Gerald Freedman Theatre.pdf
Heathers Symbols for Gerald Freedman Theatre.pdf
 
SECMOL School In Leh- A Role Model of Vernacular , Passive and Sustainable H...
SECMOL School In Leh- A  Role Model of Vernacular , Passive and Sustainable H...SECMOL School In Leh- A  Role Model of Vernacular , Passive and Sustainable H...
SECMOL School In Leh- A Role Model of Vernacular , Passive and Sustainable H...
 
Land Management and sourcing land for sustainable Urban Development.
Land Management and sourcing land for sustainable Urban Development.Land Management and sourcing land for sustainable Urban Development.
Land Management and sourcing land for sustainable Urban Development.
 
Bashundhara Residential _ Analysis and Case Study-MaHi.pptx
Bashundhara Residential _ Analysis and Case Study-MaHi.pptxBashundhara Residential _ Analysis and Case Study-MaHi.pptx
Bashundhara Residential _ Analysis and Case Study-MaHi.pptx
 
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.ppt
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.pptsustainable-interior-design-strategy-with-case-study_TANIA-BERA.ppt
sustainable-interior-design-strategy-with-case-study_TANIA-BERA.ppt
 
Hill Area development- Issues and Options
Hill Area development- Issues and OptionsHill Area development- Issues and Options
Hill Area development- Issues and Options
 
Mother Courage and Her Children Shop Order.pdf
Mother Courage and Her Children Shop Order.pdfMother Courage and Her Children Shop Order.pdf
Mother Courage and Her Children Shop Order.pdf
 
Friendship Hospital _ Case Study-MaHi.pptx
Friendship Hospital _ Case Study-MaHi.pptxFriendship Hospital _ Case Study-MaHi.pptx
Friendship Hospital _ Case Study-MaHi.pptx
 
Ordering Principles in Architecture-MaHi.pptx
Ordering Principles in Architecture-MaHi.pptxOrdering Principles in Architecture-MaHi.pptx
Ordering Principles in Architecture-MaHi.pptx
 
Amadeus Hang and Cable Layouts UNCSA.pdf
Amadeus Hang and Cable Layouts UNCSA.pdfAmadeus Hang and Cable Layouts UNCSA.pdf
Amadeus Hang and Cable Layouts UNCSA.pdf
 
HTM Example Layouts Freedman Theatre.pdf
HTM Example Layouts Freedman Theatre.pdfHTM Example Layouts Freedman Theatre.pdf
HTM Example Layouts Freedman Theatre.pdf
 
Task Pack 1- Nature - Ula Academics KIDS.pdf
Task Pack 1- Nature - Ula Academics KIDS.pdfTask Pack 1- Nature - Ula Academics KIDS.pdf
Task Pack 1- Nature - Ula Academics KIDS.pdf
 
Copy of VSST102 Digital Portfolio.pdffff
Copy of VSST102 Digital Portfolio.pdffffCopy of VSST102 Digital Portfolio.pdffff
Copy of VSST102 Digital Portfolio.pdffff
 
Studio Surfaced - Textile Design Studio based out of Delhi NCR
Studio Surfaced - Textile Design Studio based out of Delhi NCRStudio Surfaced - Textile Design Studio based out of Delhi NCR
Studio Surfaced - Textile Design Studio based out of Delhi NCR
 
VSST103 - Digital Portfolio.pdf character
VSST103 - Digital Portfolio.pdf characterVSST103 - Digital Portfolio.pdf character
VSST103 - Digital Portfolio.pdf character
 
urban design case study of old city Jodhpur
urban design case study of  old city Jodhpururban design case study of  old city Jodhpur
urban design case study of old city Jodhpur
 
Interior Landscape and Landscape Features Extraction
Interior Landscape and Landscape Features ExtractionInterior Landscape and Landscape Features Extraction
Interior Landscape and Landscape Features Extraction
 
Mother Courage and Her Children Layouts.pdf
Mother Courage and Her Children Layouts.pdfMother Courage and Her Children Layouts.pdf
Mother Courage and Her Children Layouts.pdf
 
Passing Strange Lighting Plot AS BUILT).pdf
Passing Strange Lighting Plot AS BUILT).pdfPassing Strange Lighting Plot AS BUILT).pdf
Passing Strange Lighting Plot AS BUILT).pdf
 

Surveys that work: a webinar for UXPA 2021

  • 1. Surveys that work An introduction to the Survey Octopus and Total Survey Error Caroline Jarrett @cjforms #UXPA2021
  • 2. What would you do for a dollar? Caroline Jarrett @cjforms (CC) BY SA-4.0 2
  • 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 3 $1 in the envelope beats $10 guaranteed later Caroline Jarrett @cjforms (CC) BY SA-4.0 3
  • 4. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Response Diagram inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 4
  • 5. Would you answer this survey? Caroline Jarrett @cjforms (CC) BY SA-4.0 5
  • 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6 I’m the forms specialist Image credit: Flickr, taxrebate.org.uk 6
  • 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7 Why do people answer questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 7
  • 8. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “How many people do I need in my sample?” “Will that be statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 8
  • 9. I wrote a book It seemed easier than continuing to answer lots of survey questions https://rosenfeldmedia.com/books/surveys-that-work/ Caroline Jarrett @cjforms (CC) BY SA-4.0 9
  • 10. Let’s think about these topics today “Please have a look at this survey” “Tell me whether this is a good question” “How many people do I need in my sample?” “Will that be statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 10
  • 11. The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons. Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 12. I change the definition a bit systematic method becomes process gathering information becomes ask questions entities become people quantitative descriptors become numbers attributes of the larger population become make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 13. The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14 Let’s rearrange that somewhat The survey is a process for getting answers to questions To make decisions From people getting numbers
  • 15. Caroline Jarrett @cjforms (CC) BY SA-4.0 15 Start with why and who; end with the number The survey is a process for getting answers to questions Why you want to ask Who you want to ask The number
  • 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 16 A survey is a quantitative method
  • 17. Key Point 1 A survey is a quantitative method The result of a survey is a number Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 18. Christian Rohrer mapped many more methods https://www.nngroup.com/articles/which-ux-research-methods/
  • 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19 Survey methodologists interview first
  • 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20 Establish your goals for the survey What do you want to know? Why do you want to know? What decision will you make based on these answers? What number do you need for the decision?
  • 21. Key Point 2 The aim of a survey is to get a number that will help with a decision Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 22. Caroline Jarrett @cjforms (CC) BY SA-4.0 22 Let’s have a good look at that process The survey is a process for getting answers to questions Why you want to ask Who you want to ask The number
  • 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23 The Survey Octopus has things to think about Why you want to ask Who you want to ask The number
  • 24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24 The topics are all somewhat connected 24
  • 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 25 I made a process from the things to think about Goals Questions Questionnaire Response Sample Fieldwork Response Reports 25
  • 26. Here are the 7 steps as a linear process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean and analyse the data Present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Reports Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 27. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “How many people do I need in my sample?” “Will that be statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 27
  • 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28 Is this a good question? Would you recommend us to a friend or family member? 28
  • 29. Caroline Jarrett @cjforms (CC) BY SA-4.0 29 “Recommend to friend/family” can be OK A shop selling clothes What do you want to know? Whether they will recommend Why do you want to know? To compare numbers over time What number do you need to make a decision? Some good reasons (investigate and fix problems) Some bad ones (punish staff) 29
  • 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30 “Recommend to friend/family” can be very weird A shop selling clothes A hospital ward What do you want to know? Whether they will recommend ???? Why do you want to know? To compare numbers over time Because someone said they had to ask What number do you need to make a decision? Some good reasons (investigate and fix problems) Some bad ones (punish staff) ???? What decision?
  • 31. Questions can go wrong in other ways Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 32 Make sure that you test your questionnaire
  • 33. Key Point 3 To find out whether a question is a good one, test it with people who will answer it Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 34. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “How many people do I need in my sample?” “Will that be statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 34
  • 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35 To work that out, let’s visit the Octopus 35
  • 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36 Start with the number of answers you need Fieldwork: Who answers? 36
  • 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 37 Start with the number of answers you need 10,000 Fieldwork: Who answers? 37
  • 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38 Whether they’ll answer depends on effort Questions: What are you asking about? How many questions? 38
  • 39. Caroline Jarrett @cjforms (CC) BY SA-4.0 39 Whether they’ll answer depends on effort 20 Questions: What are you asking about? How many questions? 39
  • 40. Caroline Jarrett @cjforms (CC) BY SA-4.0 40 And on the reward you’re offering Goals: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 40
  • 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 41 And on the reward you’re offering Goals: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? Make improvements 41
  • 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 42 How do our users feel about this? Sample: the list you sample from 42
  • 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43 How do our users feel about this? Sample: the list you sample from Ask anyone? 43
  • 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 44 How are you delivering the questionnaire? Fieldwork: How are you asking? 44
  • 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 45 How are you delivering the questionnaire? As a banner on the website Fieldwork: How are you asking? 45
  • 46. Response rates vary by method of asking Method of asking Possible response rate Possible replies for 1000 invitations Mailed paper survey Up to 70% 700 Email invitation to selected group, specific one-off survey Up to 20% 200 Routine email after every transaction Maybe 5% 50 Banner invitation on a website Less than 0.1% 1 Caroline Jarrett @cjforms (CC) BY SA-4.0 46
  • 47. There are many influences on response rate It depends on • How you’re asking • Why you’re asking • What you’re asking and the type of questionnaire you make • Who you’re asking Fieldwork Goals Questions Questionnaire Sample Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 48 And now it’s easy to work out how many to ask Sample: the number of people to ask 48
  • 49. Work through from response rate Topic Answer Effect on response rate How you’re asking Website banner Maybe 1 in 1000 (0.1%) Why you’re asking Make improvements OK, no change What you’re asking 20 questions Quite long, reduces rate The type of questionnaire Pops up from banner OK, no change Who you’re asking Any website visitor Not personal, reduces rate Possible eventual rate Maybe 1 in 2500 (0.04%) Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50 But wait, there’s more! Response: Dealing with the answers 50
  • 51. Caroline Jarrett @cjforms (CC) BY SA-4.0 51 Who will process those 200,000 data items? Response: Dealing with the answers 10,000 20 51
  • 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52 And do you really need all of them? Report: Creating the decision 52
  • 53. Key Point 5 A sample size calculation needs the entire Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 54. People ask me about surveys “Please have a look at this survey” “Tell me whether this is a good question” “How many people do I need in my sample?” “Will that be statistically significant?” Caroline Jarrett @cjforms (CC) BY SA-4.0 54
  • 55. What type of significance do you need? • A result that is statistically significant is one that is mathematically unlikely to be the result of chance • A result that is significant in practice is one that is meaningful in the real world Caroline Jarrett @cjforms (CC) BY SA-4.0 55
  • 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 56 If you ask the wrong questions, you’ll fail at validity (Lack of) validity Sampling error 56
  • 57. Key Point 6 Asking one person the right question gets better results than asking 10,000 people the wrong question Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 58. Caroline Jarrett @cjforms (CC) BY SA-4.0 Many statisticians aren’t keen, either Scientists rise up against statistical significance https://www.nature.com/articles/d41586-019-00857-9 58
  • 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 59 Let’s look at another crucial error Non-response error 59
  • 60. Non-response error happens when the people who do not respond are different to the people who do respond in a way that affects your decision Caroline Jarrett @cjforms (CC) BY SA-4.0 60
  • 61. Jane Matthews told me a story • 20 people attend a workshop; they all seem to enjoy it • Only get 3 or 4 back from a web survey “If we rely on those responses, we might be at risk of making bad decisions” • Now changing to phoning half the people Credit: https://janematthews.com/
  • 62. Caroline Jarrett @cjforms (CC) BY SA-4.0 62 There are errors all around the Survey Octopus (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error 62
  • 63. Key Point 7 Significance in practice comes from making good decisions throughout the survey Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 64. Caroline Jarrett @cjforms (CC) BY SA-4.0 64 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use Who you want to ask The list that you sample from The sample you ask The ones who respond The ones whose responses you can use The number (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error
  • 65. Caroline Jarrett @cjforms (CC) BY SA-4.0 65 This version uses terms from survey methodology Construct Measurement Response Edited response Representation Sampling frame Sample Respondents Post-survey adjustments Survey statistic (Lack of) validity Measurement error Processing error Coverage error Sampling error Non-response error Adjustment error 65
  • 66. Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley. 66

Editor's Notes

  1. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
  2. This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
  3. The octopus again; we've looked at 6 of the 8 tentacles.
  4. The octopus again; we've looked at 6 of the 8 tentacles.
  5. If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
  6. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  7. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  8. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  9. The octopus again. This time we're looking at 'the questions we ask'.
  10. The octopus again. This time we're looking at 'the questions we ask'.
  11. The resources you have will help you to decide on the reward you’re offering
  12. The resources you have will help you to decide on the reward you’re offering
  13. The octopus, with focus on 'The list you sample from'
  14. The octopus, with focus on 'The list you sample from'
  15. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  16. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  17. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  18. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  19. The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
  20. The octopus again; we've looked at 6 of the 8 tentacles.
  21. The octopus again; we've looked at 6 of the 8 tentacles.
  22. The octopus again; we've looked at 6 of the 8 tentacles.
  23. This is a more conventional way of looking at the octopus tentacles
  24. This slide translates the issues into the technical terms used by survey methodologists