SlideShare a Scribd company logo

Surveys that work EBI_2017

Slides from a workshop introduction to survey methods. The workshop was prepared for staff of the European Bioinformatics Institute in Cambridge, February 2017

1 of 143
Download to read offline
Surveys that work
An introduction to
using survey methods
Caroline Jarrett
@cjforms
2017 #surveysthatwork
Caroline Jarrett @cjforms (CC) BY SA-4.0
Introductions
(We’re Caroline Jarrett and Jane Matthews)
• Your name and role
• A random thing about yourself
Image credit: Caroline Jarrett
2
Caroline Jarrett @cjforms
Let’s find out about our experience
3
Caroline Jarrett @cjforms (CC) BY SA-4.0
Fill in this questionnaire
1. How many surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better survey, based on
experience of writing or answering?
__________________________________
__________________________________
Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys,
49th Society for Technical Communication Conference, Nashville TN USA
Caroline Jarrett @cjforms
Now work in pairs
5
Caroline Jarrett @cjforms (CC) BY SA-4.0
Try this as an interview
1. How many surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better survey, based on
experience of writing or answering?
__________________________________
__________________________________
Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys,
49th Society for Technical Communication Conference, Nashville TN USA

Recommended

Survey that work LibDesign2016 @cjforms
Survey that work LibDesign2016 @cjformsSurvey that work LibDesign2016 @cjforms
Survey that work LibDesign2016 @cjformsCaroline Jarrett
 
Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020Caroline Jarrett
 
Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Caroline Jarrett
 
Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021Caroline Jarrett
 
Survey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjformsSurvey Octopus TSE 2015 @cjforms
Survey Octopus TSE 2015 @cjformsCaroline Jarrett
 
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorSurveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
 
Survey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsSurvey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsCaroline Jarrett
 
The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...The Survey Octopus - getting valid data from surveys, presentation for UX in ...
The Survey Octopus - getting valid data from surveys, presentation for UX in ...Caroline Jarrett
 

More Related Content

What's hot

How to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusHow to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusCaroline Jarrett
 
Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021Caroline Jarrett
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
 
How to look at a form - Industry Conf 2016
How to look at a form -  Industry Conf 2016How to look at a form -  Industry Conf 2016
How to look at a form - Industry Conf 2016Caroline Jarrett
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
 
Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxCaroline Jarrett
 
Five Steps to Better Surveys: part of the Surveys that Work training course f...
Five Steps to Better Surveys: part of the Surveys that Work training course f...Five Steps to Better Surveys: part of the Surveys that Work training course f...
Five Steps to Better Surveys: part of the Surveys that Work training course f...Caroline Jarrett
 
How to look at the content in a form
How to look at the content in a formHow to look at the content in a form
How to look at the content in a formCaroline Jarrett
 
Six crucial survey concepts that UX professionals need to know
Six crucial survey concepts that UX professionals need to knowSix crucial survey concepts that UX professionals need to know
Six crucial survey concepts that UX professionals need to knowCaroline Jarrett
 
Effective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA WorkshopEffective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA WorkshopAmanda Stockwell
 
10 tips for a better survey at STC2011
10 tips for a better survey at STC201110 tips for a better survey at STC2011
10 tips for a better survey at STC2011Caroline Jarrett
 
Forms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsForms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsCaroline Jarrett
 
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSTHE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSUX New Zealand 2015
 
Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Caroline Jarrett
 
Design tips for complex forms by @cjforms 2013
Design tips for complex forms by @cjforms 2013Design tips for complex forms by @cjforms 2013
Design tips for complex forms by @cjforms 2013Caroline Jarrett
 
Fitness Faction
Fitness FactionFitness Faction
Fitness Factionkamathash
 

What's hot (19)

How to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey OctopusHow to get better results from a survey: Meet the Survey Octopus
How to get better results from a survey: Meet the Survey Octopus
 
Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3
 
How to look at a form - Industry Conf 2016
How to look at a form -  Industry Conf 2016How to look at a form -  Industry Conf 2016
How to look at a form - Industry Conf 2016
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2
 
Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010Surveys that work: using questionnaires to gather useful data, November 2010
Surveys that work: using questionnaires to gather useful data, November 2010
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptx
 
Five Steps to Better Surveys: part of the Surveys that Work training course f...
Five Steps to Better Surveys: part of the Surveys that Work training course f...Five Steps to Better Surveys: part of the Surveys that Work training course f...
Five Steps to Better Surveys: part of the Surveys that Work training course f...
 
How to look at the content in a form
How to look at the content in a formHow to look at the content in a form
How to look at the content in a form
 
Six crucial survey concepts that UX professionals need to know
Six crucial survey concepts that UX professionals need to knowSix crucial survey concepts that UX professionals need to know
Six crucial survey concepts that UX professionals need to know
 
Effective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA WorkshopEffective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA Workshop
 
10 tips for a better survey at STC2011
10 tips for a better survey at STC201110 tips for a better survey at STC2011
10 tips for a better survey at STC2011
 
Forms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjformsForms workshop for ConCon Manchester 2016 by @cjforms
Forms workshop for ConCon Manchester 2016 by @cjforms
 
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSTHE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
 
Design tips for surveys UIE 2012
Design tips for surveys UIE 2012Design tips for surveys UIE 2012
Design tips for surveys UIE 2012
 
Design tips for complex forms by @cjforms 2013
Design tips for complex forms by @cjforms 2013Design tips for complex forms by @cjforms 2013
Design tips for complex forms by @cjforms 2013
 
Fitness Faction
Fitness FactionFitness Faction
Fitness Faction
 
Kristens
KristensKristens
Kristens
 

Similar to Surveys that work EBI_2017

The inevitable survey 2021
The inevitable survey 2021The inevitable survey 2021
The inevitable survey 2021Caroline Jarrett
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Caroline Jarrett
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptxCaroline Jarrett
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
 
Forms that work: Understanding forms to improve their design by @cjforms
Forms that work: Understanding forms to improve their design by @cjformsForms that work: Understanding forms to improve their design by @cjforms
Forms that work: Understanding forms to improve their design by @cjformsCaroline Jarrett
 
A deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxA deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxCaroline Jarrett
 
Social Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterSocial Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterJonathas Magalhães
 
Lesson 6 - Primary Research Methods 2
Lesson 6  - Primary Research Methods 2Lesson 6  - Primary Research Methods 2
Lesson 6 - Primary Research Methods 2Kavita Parwani
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023Caroline Jarrett
 
Questionnairre desisgn-Advance Research Methodology
Questionnairre desisgn-Advance Research MethodologyQuestionnairre desisgn-Advance Research Methodology
Questionnairre desisgn-Advance Research MethodologyRehan Ehsan
 
Advocacy 101 (UCL Digifest 2014)
Advocacy 101 (UCL Digifest 2014)Advocacy 101 (UCL Digifest 2014)
Advocacy 101 (UCL Digifest 2014)Right to Research
 
Survey (Primer on Questions, Sampling + Case Study)
Survey (Primer on Questions, Sampling + Case Study)Survey (Primer on Questions, Sampling + Case Study)
Survey (Primer on Questions, Sampling + Case Study)Dada Veloso-Beltran
 
Career and College Readiness
Career and College Readiness Career and College Readiness
Career and College Readiness msweducational
 
Survey Methodology for Security and Privacy Researchers
Survey Methodology for Security and Privacy ResearchersSurvey Methodology for Security and Privacy Researchers
Survey Methodology for Security and Privacy ResearchersElissa Redmiles
 
Advanced Social Media Techniques CBMI 2013
Advanced Social Media Techniques CBMI 2013Advanced Social Media Techniques CBMI 2013
Advanced Social Media Techniques CBMI 2013Christopher Rice
 

Similar to Surveys that work EBI_2017 (17)

The inevitable survey 2021
The inevitable survey 2021The inevitable survey 2021
The inevitable survey 2021
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference call
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptx
 
Forms that work: Understanding forms to improve their design by @cjforms
Forms that work: Understanding forms to improve their design by @cjformsForms that work: Understanding forms to improve their design by @cjforms
Forms that work: Understanding forms to improve their design by @cjforms
 
Better UX Surveys part 1
Better UX Surveys part 1Better UX Surveys part 1
Better UX Surveys part 1
 
A deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxA deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLx
 
Social Query: A Query Routing System for Twitter
Social Query: A Query Routing System for TwitterSocial Query: A Query Routing System for Twitter
Social Query: A Query Routing System for Twitter
 
Lesson 6 - Primary Research Methods 2
Lesson 6  - Primary Research Methods 2Lesson 6  - Primary Research Methods 2
Lesson 6 - Primary Research Methods 2
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023
 
Questionnairre desisgn-Advance Research Methodology
Questionnairre desisgn-Advance Research MethodologyQuestionnairre desisgn-Advance Research Methodology
Questionnairre desisgn-Advance Research Methodology
 
Advocacy 101 (UCL Digifest 2014)
Advocacy 101 (UCL Digifest 2014)Advocacy 101 (UCL Digifest 2014)
Advocacy 101 (UCL Digifest 2014)
 
Survey (Primer on Questions, Sampling + Case Study)
Survey (Primer on Questions, Sampling + Case Study)Survey (Primer on Questions, Sampling + Case Study)
Survey (Primer on Questions, Sampling + Case Study)
 
Career and College Readiness
Career and College Readiness Career and College Readiness
Career and College Readiness
 
Survey Methodology for Security and Privacy Researchers
Survey Methodology for Security and Privacy ResearchersSurvey Methodology for Security and Privacy Researchers
Survey Methodology for Security and Privacy Researchers
 
Advanced Social Media Techniques CBMI 2013
Advanced Social Media Techniques CBMI 2013Advanced Social Media Techniques CBMI 2013
Advanced Social Media Techniques CBMI 2013
 

More from Caroline Jarrett

The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...Caroline Jarrett
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptxCaroline Jarrett
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxCaroline Jarrett
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptxCaroline Jarrett
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxCaroline Jarrett
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxCaroline Jarrett
 
A chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsA chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsCaroline Jarrett
 
Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Caroline Jarrett
 
Usability evaluation of forms 1998
Usability evaluation of forms 1998Usability evaluation of forms 1998
Usability evaluation of forms 1998Caroline Jarrett
 
Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Caroline Jarrett
 
Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Caroline Jarrett
 
Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Caroline Jarrett
 
Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Caroline Jarrett
 
Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Caroline Jarrett
 

More from Caroline Jarrett (14)

The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptx
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptx
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptx
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptx
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptx
 
A chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsA chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjforms
 
Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...
 
Usability evaluation of forms 1998
Usability evaluation of forms 1998Usability evaluation of forms 1998
Usability evaluation of forms 1998
 
Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016Write Clearly: take your web writing to the next level, May 2016
Write Clearly: take your web writing to the next level, May 2016
 
Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...Understanding the costs of data capture: paper, automatic and with the intern...
Understanding the costs of data capture: paper, automatic and with the intern...
 
Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000Understand and investigate the costs of dealing with forms, October 2000
Understand and investigate the costs of dealing with forms, October 2000
 
Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009Search is now normal behaviour: what do we do about that? November 2009
Search is now normal behaviour: what do we do about that? November 2009
 
Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007Building a site for diverse audiences, May 2007
Building a site for diverse audiences, May 2007
 

Recently uploaded

Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............mahetamanav24
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...ThinkInnovation
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)UNCResearchHub
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsDataArchiva
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensKondapi V Siva Rama Brahmam
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxHizkiaJastis
 
What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?Denodo
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxPoonamRijal
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 

Recently uploaded (13)

Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
Electricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptxElectricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptx
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data Goals
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample Screens
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptx
 
What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptx
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 

Surveys that work EBI_2017

  • 1. Surveys that work An introduction to using survey methods Caroline Jarrett @cjforms 2017 #surveysthatwork
  • 2. Caroline Jarrett @cjforms (CC) BY SA-4.0 Introductions (We’re Caroline Jarrett and Jane Matthews) • Your name and role • A random thing about yourself Image credit: Caroline Jarrett 2
  • 3. Caroline Jarrett @cjforms Let’s find out about our experience 3
  • 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fill in this questionnaire 1. How many surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better survey, based on experience of writing or answering? __________________________________ __________________________________ Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys, 49th Society for Technical Communication Conference, Nashville TN USA
  • 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try this as an interview 1. How many surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better survey, based on experience of writing or answering? __________________________________ __________________________________ Jarrett, C. and Bachmann, K (2002) Creating Effective User Surveys, 49th Society for Technical Communication Conference, Nashville TN USA
  • 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 7
  • 9. Caroline Jarrett @cjforms The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
  • 10. Caroline Jarrett @cjforms The survey is a process for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  • 11. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  • 12. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
  • 13. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers of the attributes of the larger population of which the entities are members.
  • 14. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 15. Caroline Jarrett @cjforms The survey is a process for getting answers to questions To make decisions People getting numbers Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 16
  • 17. Is this a survey or something else? • Review these questions • Decide whether they are a survey or something else 17
  • 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 18
  • 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision Goals Sample Fieldwork Responses Insights Questionnaire Questions 19
  • 21. Caroline Jarrett @cjforms Asking the right question Asking the right people 21
  • 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 23
  • 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals The survey process Questions you need answers to Establish your goals for the survey Goals 25
  • 26. Caroline Jarrett @cjforms (CC) BY SA-4.0 Establish your goals for the survey Goals What do you want to know? Why do you want to know? What decisions will you make based on these answers? 26
  • 27. Goals An example • Here’s one of our examples • What do you think the goals are? • What do you think the decisions are likely to be? 27
  • 28. Goals What are your goals for your survey? • What do you want to know? • Why do you want to know it? • What decision(s) will you make as a result of the survey?
  • 29. Caroline Jarrett @cjforms Goals Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/ 29
  • 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals 1950s mindset: “Ask Everything” Survey = Big Honkin’ Survey
  • 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals 2016 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 31
  • 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals This one-box survey asks one open question 32
  • 33. Caroline Jarrett @cjforms Goals One way to iterate, improve, increase Time for new question Caroline Jarrett @cjforms (CC) BY SA-4.0 33
  • 34. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals What’s the Most Crucial Question? • We want to ask the fewest questions that will help us to make the decision so we need to know which is are the most useful questions • Even better: know the specific Most Crucial Question • A Most Crucial Question has a numeric answer 34
  • 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals What’s the Most Crucial Question? Look through the questions in this survey What is the Most Crucial Question?
  • 36. Caroline Jarrett @cjforms Goals Another way: narrow downLots of questions Useful questions MCQ Caroline Jarrett @cjforms (CC) BY SA-4.0 36
  • 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 Goals Talk to users about the topics in your survey • Who are they? • How will you find them? • Do they want to answer your questions? • Do they understand your questions? 37
  • 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Questions you need answers to Goals Decide who to ask and how many People you will invite to answer Sample 38
  • 39. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Asking the right people is better than asking lots of people Sample: the list you sample from Caroline Jarrett @cjforms (CC) BY SA-4.0 39
  • 40. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Choose a good list Coverage error: Mismatch between the people you want to ask and the list you choose to sample from Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Difference between response, response rate and representativeness Concept Definition Example Response Number of answers 5,000 Response rate Response divided by the number of invitations 10% Representativeness Whether respondents you get are typical of the users you want Image credit: North Korean flag, http://commons.wikimedia.org/wiki/File:Flag_of_North_Korea.svg 42
  • 43. Sample Did we get answers from the right people? Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 43
  • 44. Sample Check the representativeness of your sample Population of assorted birds Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 44
  • 45. Caroline Jarrett @cjforms Sample Iterate, improve, increase to understand the people you want to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 45
  • 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Decide how to target the correct people • Go where they are • Use a list • Send and hope • Try a ‘snowball’ • Buy a sample Image credit: Flickr sunchild57 46
  • 47. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Non-response error is the one that hurts Non-response error: The ones who answer differ from the ones who don’t answer in a way that affects the survey statistic Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 48. Sample Let’s try the toothpaste Image credit: Caroline Jarrett 48
  • 49. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Response depends on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 49
  • 50. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Response relies on effort, reward, and trust 50
  • 51. Sample An example invitation What is the perceived effort? What is the perceived reward? What about trust?
  • 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample There’s often a ‘zone of indifference’ Hate it Love it 52
  • 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample What do people want to tell you? Burning Issues 53
  • 54. Sample What are the Burning Issues? • Think about a training course (other than today!) that you’ve attended • Make a note of any Burning Issue that you had
  • 55. Sample What are the Burning Issues? • Now see if there’s somewhere on this survey to share your Burning Issue
  • 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample Overcome the ‘Zone of Indifference’ by asking about the Burning Issues 56
  • 57. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample There is always sampling error Sampling error: Ask a sample instead of asking everyone Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 58. Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample If you get the other decisions right, then you can calculate a margin of error 58
  • 59. Caroline Jarrett @cjforms Asking the right question Asking the right people 59
  • 60. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 60
  • 61. Caroline Jarrett @cjforms (CC) BY SA-4.0 A survey is only valid if the questions match the reason you’re doing it Lack of validity: mismatch between what you ask and what you need to know Caroline Jarrett @cjforms (CC) BY SA-4.0 61
  • 62. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Questions you need answers to People you will invite to answer Goals Sample Test the questions Questions Questions people can answer 62
  • 64. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Helps a lot if you ask good questions Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 64
  • 65. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions There are four steps to answer a question Understand Find Judge Place Adapted from Tourangeau, R., Rips, L. J. and Rasinski, K. A. (2000) “The psychology of survey response”
  • 66. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions There are four steps to answer a question Step A good question … 1. Read and understand is legible and makes sense 2. Find an answer asks for answers that we know 3. Judge the answer asks for answers we’re happy to reveal 4. Place the answer offers appropriate spaces for the answers Adapted from Tourangeau, R., Rips, L. J. and Rasinski, K. A. (2000) “The psychology of survey response”
  • 67. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 1: read and understand 67
  • 68. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 1: read and understand Hermann grid illusion 68
  • 69. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 2: find the answer In your last five days at work, what percentage of your work time do you estimate that you spend using publicly- available online services (not including email, instant messaging and search) to do your work using a work computer or other device? 69
  • 70. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions The approximate curve of forgetting
  • 71. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 3: judge the answer 71
  • 72. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questions Four step examples: 4: place the answer 72
  • 74. Questions Any problems with the 4 steps? • Think about the four steps of answering a question: – Read and understand the question – Find the answer – Judge whether the answer fits – Place the answer • Any problems with any of the questions? • If so, which step(s) are problematic? 74
  • 75. Improve a question • We’ve chosen a question from a longer survey. • Can you improve it? 75
  • 76. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process 76 Establish your goals for the survey Decide who to ask and how many Build the questionnaire Questions you need answers to People you will invite to answer Goals Sample Questionnaire Test the questions Questions Questions people can answer Questions people can interact with
  • 77. Caroline Jarrett @cjforms (CC) BY SA-4.0 A good question gets good answers Measurement error: Mismatches between the questions you ask and the answers that people give you Caroline Jarrett @cjforms (CC) BY SA-4.0
  • 78. Caroline Jarrett @cjforms Questionnaire Caroline Jarrett @cjforms (CC) BY SA-4.0 78
  • 79. Caroline Jarrett @cjforms Questionnaire "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett 79
  • 80. Questionnaire Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 80
  • 81. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire “Place the answer” is also about using the right widget to collect the answer Use For Radio buttons A single known answer Check boxes Multiple known answers Text boxes Unknown answers 81
  • 82. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Likert had several types of response format in his scales Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 82
  • 83. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire You can find an academic paper to support almost any number of response points Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier. http://bit.ly/KNWlio 83
  • 84. Questionnaire An example Here’s an example of a Likert response format • Any problems you can see? • Any particularly good practice? 84
  • 85. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Grids are often full of problems at all four steps 85
  • 86. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Grids are a major cause of survey drop-out 35% 20% 20% 15% 5% 5% Total incompletes across the 'main' section of the questionnaire (after the introduction stage) Subject Matter Media Downloads Survey Length Large Grids Open Questions Other Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar From Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne 86
  • 87. Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire But it’s the topic that matters most 35% 20% 20% 15% 5% 5% Total incompletes across the 'main' section of the questionnaire (after the introduction stage) Subject Matter Media Downloads Survey Length Large Grids Open Questions Other Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar From Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne 87
  • 88. Questionnaire Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 88
  • 89. Caroline Jarrett @cjforms Your answers to this survey are important for our work But what’s in it for me? And I’m really ready for a break. 89
  • 91. Goals Sample Goals and sample for the survey • We’ve had a request for help with a survey • We’ll be having a meeting to discuss the survey • Decide on the topics you’ll want to discuss at the meeting • Also, prepare a suggestion for the Most Crucial Question 91
  • 92. Questions Write questions • We have discussed some possible questions • Decide on the MCQ that you will ask – Check that users can: • Read and understand it • Find the answer • Judge the answer • Decide if you need any extra questions to frame the MCQ • Is there a Burning Issue? 92
  • 93. Questionnaire Make a questionnaire Make a paper version of your questionnaire (We’ll be testing the questionnaires a bit later) 93
  • 94. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 94
  • 95. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Run the survey from invitation to follow-up Fieldwork People who actually answer 95
  • 98. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Recap: Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 98
  • 99. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork The elements of a good invitation • Trust: – Say who you are – Say why you’ve contacted this person specifically • Perceived reward: – Explain the purpose of the survey – Explain why this person’s responses will help that purpose – If there is an incentive, offer it • Perceived effort: – Outline the topic of the survey – Say when the survey will close – Do NOT say how long it will take • (unless you have tested the heck out of it and are extremely sure that you know the answer) 99
  • 100. Fieldwork Write the invitation and thank-you • Hints: – the invitation can be part of the questionnaire – thank-you is on a separate page 100
  • 101. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Test it: pilot study • Run the survey from invitation to the follow-up • Look for mechanical problems like wrong link in the invitation, no thank-you page • Find out what your response rate is so that you can work out your sample size “If you don’t have time to do a pilot study, you don’t have time to do the survey” 101
  • 102. Fieldwork Test it and report back 102
  • 103. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork Think about the test and iterate • Are the people you tested with representative? • Did you test the whole survey – From invitation to follow up? – Including the analysis of responses? – Including finding out whether you can make the decision? • What do you need to change for the next version? 103
  • 105. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 105
  • 106. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Clean the data Responses Answers 106
  • 107. Caroline Jarrett @cjforms (CC) BY SA-4.0 The answers that you get will tell you whether you had good questions Caroline Jarrett @cjforms (CC) BY SA-4.0 Measurement error: Mismatches between the questions you ask and the answers people actually give you
  • 109. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Clean your data • Look for gaps and missing entries • Remove any (unintended) duplicate responses • Read the answers to make sure that they make sense compared to the questions Image credit: Shutterstock Adapted from Boslaugh, S. and P. A. Watters (2008) Statistics in a nutshell O’Reilly109
  • 110. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Decide whose answers to include Caroline Jarrett @cjforms (CC) BY SA-4.0 Adjustment error: Problems when deciding whether to include or exclude someone’s answers
  • 111. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Look after your data • Data analysis can take a long time; you won’t want to repeat it – Make copies of your data, especially before any drastic change – ‘Undo’ doesn’t always work on large files • Make notes of what you did – It helps if you have to defend your conclusions – It’s hard to remember the details a year later Image credit: Shutterstock 111
  • 112. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Decide what to do when people have skipped questions or dropped out 1. Remove the whole of that person’s response 2. Use the partial responses, and accept that your number of responses is lower for some questions 3. Calculate an “imputed value” – Include a flag showing that the value is calculated – Estimate the most likely value using the other data 112
  • 113. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses If you’re losing people, have you still got representativeness? Image credit: Caroline Jarrett / CorelDraw113
  • 114. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses You can interpret data well – or poorly Caroline Jarrett @cjforms (CC) BY SA-4.0 Processing error: Bad choices about how to interpret the answers
  • 115. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Typing in the answers = coding Image credit: https://www.census.gov/history/www/census_then_now/notable_alumni/herman_hollerith.html 115
  • 116. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses If you ask for answers, you have to read and think about them 116
  • 117. Responses Have a go at coding Here are some answers from a survey • Are there any themes? • How would you code them? 117
  • 118. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses CAQDAS tools are available (but are a big challenge) Before buying one, read this site: http://www.surrey.ac.uk/sociology/research/ researchcentres/caqdas/support/choosing/index.htm http://bit.ly/Surrey1234 Image credit: http://www.surrey.ac.uk/sociology/research/researchcentres/caqdas/support/choosing/index.htm 118
  • 119. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Wordle from a survey on usability certification 119
  • 120. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Wordle.net example: in favour of Facebook 120
  • 121. Caroline Jarrett @cjforms (CC) BY SA-4.0 Responses Another: against Facebook 121
  • 122. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 122
  • 123. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Analyse and present the results Insights Decisions 123
  • 124. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Explore your data and ask questions 124
  • 125. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Explore your data and ask questions 125
  • 126. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Use graphs and charts to understand relationships in the data Anscombe, F. J.. (1973). Graphs in Statistical Analysis. The American Statistician, 27(1), 17–21. http://doi.org/10.2307/2682899 126
  • 127. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Two datasets, same summaries • X Mean: 54.26 • Y Mean: 47.83 • X SD: 16.76 • Y SD: 26.93 • Corr.: -0.06 https://twitter.com/JustinMatejka/status/770682771656368128 127
  • 128. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Justin Matejka’s dataset 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 y https://twitter.com/JustinMatejka/status/770682771656368128 128
  • 129. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Alberto Cairo’s dataset 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 120 y-dino https://twitter.com/JustinMatejka/status/770682771656368128 129
  • 130. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Use descriptive statistics to explore numerical data • Most seen for statistics – Mean (arithmetic average) – Standard deviation (spread of answers) • Useful for thinking about the data – Range (lowest to highest) – Mode (most common answer) 130
  • 131. Insights A ‘Like / Dislike’ question got these responses Strongly dislike 2 Dislike 6 Neither dislike nor like 14 Like 31 Strongly like 13 Total responses 66 Please work out: the percentage of respondents who ‘like’ 131
  • 132. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights There are many ways to combine ratings into means and percentages • 47% 31 ticked ‘like’ so 31/66 = 47% • 67% ‘Top box’ / ‘top 2 box’ uses the positive responses • 68% ‘0 to 4’ weights responses: 0%, 25%, 50%, 75%, 100% • 74% ‘1 to 5’ weights responses: 1, 2, 3, 4, 5 (then divide by 5) • 36% ‘-1 to 1’ weights responses: -100%, -50%, 0, 50%, 100% 67% 68% 74% 36% 132
  • 133. This example has a graph • This example uses the calculation: Poor = 1 Reasonable = 2 Good = 3 Excellent = 4 • Is the graph an appropriate illustration of the data? 133
  • 134. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights Net Promoter Score™ has a special analysis method Image credit: https://www.netpromoter.com/know/134
  • 135. Caroline Jarrett @cjforms Asking the right people Asking the right question Choose whichever method you like, but you must make the choice when you decide on the goals of the survey 135
  • 136. Caroline Jarrett @cjforms (CC) BY SA-4.0 The survey process Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean the data Analyse and present the results Questions you need answers to People you will invite to answer Goals Sample Questionnaire Fieldwork People who actually answer Responses Insights Answers Decisions Test the questions Questions Questions people can answer Questions people can interact with 136
  • 137. Caroline Jarrett @cjforms (CC) BY SA-4.0 All the topics are connected Goals Sample Fieldwork Response Insight Response Caroline Jarrett @cjforms (CC) BY SA-4.0 Questionnaire Questions 137
  • 138. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use The number Caroline Jarrett @cjforms (CC) BY SA-4.0 138
  • 139. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim is to get the best number you can, within the resources you have What you want to ask about The reason you’re doing it The questions you ask The answers you get The answers you use Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use The number Caroline Jarrett @cjforms (CC) BY SA-4.0 139
  • 140. Caroline Jarrett @cjforms Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley. 140
  • 141. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 141
  • 142. Caroline Jarrett @cjforms (CC) BY SA-4.0 Should I do this survey? Yes Yes Yes Yes GoYes Do I know how I’m going to use the answers? Do people want to respond to my request? Do people have answers to these questions? Do I have time to test and to iterate? Is a survey the right way to get the answers? 142
  • 143. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 143

Editor's Notes

  1. The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
  2. The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
  3. The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
  4. Screenshot of the Suttons Seeds website with a pop-up box: "Help us improve. We value your opinion. What do you like about our site and what can we improve on?"
  5. A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up. It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
  6. Another useful iteration: start with whatever questions your stakeholders have, then narrow down to useful questions, then narrow again to your MCQ
  7. The octopus, with focus on 'The list you sample from'
  8. Prank leaves Justin Bieber facing tour of North Korea By Daniel Emery Technology reporter, BBC News 5 July 2010 Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea. A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on. There are now almost half a million votes to send the singer to the secretive communist nation. The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days. Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
  9. A classic example of the difference between response and representativeness: a Justin Bieber fan site organised a poll to see where the teen star should have his next concert. The poll got a big response but the winning location was North Korea. It seems unlikely that the respondents were representative of true Bieber fans.
  10. The picture reflects the mistakes we can make if we do our sampling based solely on the judgement of an interviewer
  11. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
  12. This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
  13. The octopus again. This time we're looking at 'the questions we ask'.
  14. The octopus again. This time we're looking at 'the questions we ask'.
  15. The four steps are: Read and understand the question Find the answer Judge whether the answer you’ve found aligns with what you say and the way you want to portray yourself Place the answer on the form (or give it to the interviewer)
  16. Two questions from a survey: '24: Do you use a Windows or Mac computer'? '25. What is your gender'?
  17. Photo of a Samsung (android) mobile with the same questions as previous slide. If you only have an Android mobile, how do you answer ‘do you use a Windows or Mac computer’ when the answer options are ‘Windows’, ‘Mac’ and ‘Both’?
  18. A model wears a t-shirt with Gender: 'Male' (crossed out), 'Female' (crossed out) and 'Other' (added and ticked'.
  19. Question from Likert’s 1932 paper. 13. How much military training should we have? We need universal compulsory military training (1) We need Citizens Military Training Camps and Reserve Officers Training Corps, but not universal military training (2) We need some facilities for training reserve officers but not as much as at present (3) We need only such military training as is required to maintain our regular army (4) All military training should be abolished (5) Another question, this time using what is often called a ‘Likert Scale’ but should really be called a ‘Likert response format’ 17. The United States, whether a member or not, should co-operate fully in the humanitarian and economic programs of the League of Nations. Strongly approve (5) Approve (4) Undecided (3) Disapprove (2) Strongly disapprove (1)
  20. Krosnick and Presser refer to ~87 papers on response points. This selection of questions from different surveys has: One with seven response points in the range One with two response points (yes/no) One with five response points plus ‘not applicable’ One with three response points One with four response points plus a comment box One with four response points on their own One with 10 response points plus ‘Don’t Know’
  21. Ratings screens examples
  22. People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
  23. A bar chart, answers to question "Are you a parent or guardian of a child in any of the following age bands (please tick all that apply)? There are five bands of two years each starting with 'Aged 2 or under' and going evenly up to 'Aged 13-15'. These all have roughly similar numbers of respondents - around 15%. There's one massive bar for the category 'Aged 16+', with around 45% response.
  24. The octopus again; we've looked at 6 of the 8 tentacles.
  25. This is a more conventional way of looking at the octopus tentacles
  26. This is a more conventional way of looking at the octopus tentacles
  27. Is this the only way I can get this data? If yes: Do users want to talk to us about these topics? If yes: Do users have answers to these questions? If yes: Do I have time to do a pilot? If yes: Do I know how I’m going to use the answers? GO