Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX New Zealand conference 2015 #uxnz2015
The Survey Octopus - getting valid data from surveys, presentation for UX in ...Caroline Jarrett
Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016
Surveys that work: a webinar for FocusVision 2021Caroline Jarrett
Creating surveys that work for participants and deliver high quality insight is no mean feat. This is because the survey process is complex, with multiple considerations at every step in the journey.
In this webinar for FocusVision, I introduce the Survey Octopus, my friendly way of talking about the many issues that make surveys one of the most challenging research methods. I also explain how the Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.
The Survey Octopus will help you design better surveys by thoughtfully considering:
• What you want to ask about
• Who you want to ask
• The number of people you need to ask
How to get better results from a survey: Meet the Survey OctopusCaroline Jarrett
The Survey Octopus is a friendly creature who will help you to think about all the crucial issues in crafting a survey.
Presentation by Caroline Jarrett @cjforms for the 2014 Content Strategy Summit #CSSummit
Surveys that Work 2020: training course for HMRC user researchers 2020Caroline Jarrett
Slides from a training course on effective surveys, delivered to usability researchers at HMRC. The course took place at HMRC's Longbenton, Newcastle, offices, on January 30, 2020. Survey examples submitted by participants for review have been removed from this presentation.
Slides from a workshop introduction to survey methods. The workshop was prepared for staff of the European Bioinformatics Institute in Cambridge, February 2017
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
A presentation for Harvard University's User Research Community on some of the key issues in creating effective surveys, including: why run a survey, writing good questions, statistical significance and how to avoid errors.
Total Survey Error for non-specialists: creating better conversations. A presentation of the Survey Octopus at the TSE2015 conference in Baltimore, September 2015.
Introduction to survey methods at LibDesign2016. A workshop led by Caroline Jarrett for people working in the library service and public sector in the Czech Republic. Caroline Jarrett led this workshop in Prague in September 2016 as part of the LibDesign 2016 conference.
The Survey Octopus - getting valid data from surveys, presentation for UX in ...Caroline Jarrett
Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016
Surveys that work: a webinar for FocusVision 2021Caroline Jarrett
Creating surveys that work for participants and deliver high quality insight is no mean feat. This is because the survey process is complex, with multiple considerations at every step in the journey.
In this webinar for FocusVision, I introduce the Survey Octopus, my friendly way of talking about the many issues that make surveys one of the most challenging research methods. I also explain how the Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.
The Survey Octopus will help you design better surveys by thoughtfully considering:
• What you want to ask about
• Who you want to ask
• The number of people you need to ask
How to get better results from a survey: Meet the Survey OctopusCaroline Jarrett
The Survey Octopus is a friendly creature who will help you to think about all the crucial issues in crafting a survey.
Presentation by Caroline Jarrett @cjforms for the 2014 Content Strategy Summit #CSSummit
Surveys that Work 2020: training course for HMRC user researchers 2020Caroline Jarrett
Slides from a training course on effective surveys, delivered to usability researchers at HMRC. The course took place at HMRC's Longbenton, Newcastle, offices, on January 30, 2020. Survey examples submitted by participants for review have been removed from this presentation.
Slides from a workshop introduction to survey methods. The workshop was prepared for staff of the European Bioinformatics Institute in Cambridge, February 2017
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorCaroline Jarrett
A presentation for Harvard University's User Research Community on some of the key issues in creating effective surveys, including: why run a survey, writing good questions, statistical significance and how to avoid errors.
Total Survey Error for non-specialists: creating better conversations. A presentation of the Survey Octopus at the TSE2015 conference in Baltimore, September 2015.
Introduction to survey methods at LibDesign2016. A workshop led by Caroline Jarrett for people working in the library service and public sector in the Czech Republic. Caroline Jarrett led this workshop in Prague in September 2016 as part of the LibDesign 2016 conference.
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Caroline Jarrett
Surveys are easy to do – but harder to do well. In this interactive workshop - delivered to the UX Insight Festival 2020 - I take you through using Total Survey Error as a way of balancing the issues and good practice in survey design to get the best results from your survey.
The session also covered my 7-step survey process, starting with Goals and thinking about Sampling, Questions, Questionnaires, Fieldwork, Responses and Reports. Plus we tackle some of the questions I'm most often asked about creating surveys that work.
Many of us receive multiple requests to complete surveys every day. Some of us find that colleagues or clients think of ‘doing a survey’ as the same as ‘doing some research’ – which may explain why organizations send out so many survey requests.
In this webinar, you’ll meet the Survey Octopus, Caroline Jarrett’s friendly way of talking about the many issues that make surveys one of the most challenging research methods.
The Survey Octopus will help you to:
Explain to colleagues that a survey may not be the first research method to try
Help to justify a choice to work with a “non significant” number of responses
Think about the steps that go into delivering a survey that works
As a bonus, Caroline will also explain how her Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.
Forms – the only non-optional part of most user experiences, but often the part that gets the least attention. This session at the 2016 Industry Conf in Newcastle was an opportunity to lead the audience through the design of typical forms and look at the problems and potential ways to improve them.
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 3 of the course: responses and reports.
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 2 of the course: questions, questionnaire and fieldwork
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
This presentation to the 22nd Australasian Computer-Human Interaction Conference, OZCHI 2010, compares survey processes and looks at some of the detail of designing surveys – including how to avoid survey error.
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 1 of the course: goals and sample.
Forms workshop for ConCon Manchester 2016 by @cjformsCaroline Jarrett
How to think about and write for forms, starting with 'what is a form' and then working through how people read forms and how that affects how we write for them.
Working with complex forms such as insurance applications, medical claims, government transactions? This workshop at UXPA2013 has tips for improving them.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
What Elements of an Online Profile Predict Target-Rater Agreement in Personal...docdave
Study showing people really can read your personality from your online profile. Presented April 2008 at the Int'l Conference on Weblogs and Social Media (ICWSM) in Seattle, Washington, USA.
Plain language to improve your survey houston 2022Caroline Jarrett
Plain language skills are vital for surveys - and especially to writing good questions and creating them for your survey audience. This presentation was prepared for the University of Houston's 8th Biannual Forum on Plain English, 24 February 2022.
Surveys are still really popular as a research method with colleagues (if not with service designers).
These slides are from a workshop at the 2021 Service Design in Government conference (@sdingov21) on 'how to improve the survey that is going to happen whether you like it or not'.
In the workshop we looked at a 7-step process for a survey and considered ways of encouraging colleagues to combine surveys with other research methods.
We also practiced techniques for looking at – and improving - a questionnaire.
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
Slides from a short presentation on creating effective surveys. The event was a conference call for members of a community network organised by Janus Boye of Boye & Company.
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Caroline Jarrett
In this webinar for product managers, Caroline introduces two key concepts from her book on surveys: identifying the most crucial question as part of getting clear on your goals, and allowing respondents to tell you the things that they want to - their burning issue. The webinar was organised by Productboard and held on March 30, 2023.
In this half day workshop for ~WebExpo2023 Caroline Jarrett shares four ways to improve your survey so that you get plenty of useful responses.
Goals: Ruthlessly focus your survey on an immediate decision.
Sample: Write an invitation that makes people want to answer.
Questions: Ditch the rating scales.
Responses: Lose your fear of open answers.
Some thoughts on good survey design delivered to students at Olin College of Engineering. Caroline's talk covers her survey process, survey goals and focusing on a specific decision, sample and sampling error, ditching rating scales, and losing fear of open answers.
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
In this webinar for Delib, Caroline shows you how to get better results from shorter, more frequent surveys - with a special emphasis on local government and the requirement to run statutory consultations. Understanding and identifying the Most Crucial Question and making space for the Burning Issue are both helpful techniques for creating shorter more focused surveys.
A presentation for the the Content Wrangler's coffee and content session on how to design and run surveys and gain actionable insights from the survey data.
Ideas for extracting the maximum value from a survey that is going to happen anyway.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Caroline Jarrett
Surveys are easy to do – but harder to do well. In this interactive workshop - delivered to the UX Insight Festival 2020 - I take you through using Total Survey Error as a way of balancing the issues and good practice in survey design to get the best results from your survey.
The session also covered my 7-step survey process, starting with Goals and thinking about Sampling, Questions, Questionnaires, Fieldwork, Responses and Reports. Plus we tackle some of the questions I'm most often asked about creating surveys that work.
Many of us receive multiple requests to complete surveys every day. Some of us find that colleagues or clients think of ‘doing a survey’ as the same as ‘doing some research’ – which may explain why organizations send out so many survey requests.
In this webinar, you’ll meet the Survey Octopus, Caroline Jarrett’s friendly way of talking about the many issues that make surveys one of the most challenging research methods.
The Survey Octopus will help you to:
Explain to colleagues that a survey may not be the first research method to try
Help to justify a choice to work with a “non significant” number of responses
Think about the steps that go into delivering a survey that works
As a bonus, Caroline will also explain how her Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.
Forms – the only non-optional part of most user experiences, but often the part that gets the least attention. This session at the 2016 Industry Conf in Newcastle was an opportunity to lead the audience through the design of typical forms and look at the problems and potential ways to improve them.
Surveys that work: training course for Rosenfeld Media, day 3 Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 3 of the course: responses and reports.
Surveys that work: training course for Rosenfeld media, day 2Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 2 of the course: questions, questionnaire and fieldwork
Surveys that work: using questionnaires to gather useful data, November 2010Caroline Jarrett
This presentation to the 22nd Australasian Computer-Human Interaction Conference, OZCHI 2010, compares survey processes and looks at some of the detail of designing surveys – including how to avoid survey error.
Surveys that work:training course for Rosenfeld Media, day 1Caroline Jarrett
Surveys seem easy: anyone can throw together a few questions, send them out, and hope that they are rewarded with a decent response. But we’ve all seen examples of poorly conceived surveys that couldn’t possibly deliver real insights for the organisation that sponsored them.
This highly participative three-session training - arranged by Rosenfeld Media as part of its Virtual Training with UX Industry Leaders programme - takes you through the whole process of creating an effective survey, from defining a goal through analysis of data and creating a presentation.
These slides come from day 1 of the course: goals and sample.
Forms workshop for ConCon Manchester 2016 by @cjformsCaroline Jarrett
How to think about and write for forms, starting with 'what is a form' and then working through how people read forms and how that affects how we write for them.
Working with complex forms such as insurance applications, medical claims, government transactions? This workshop at UXPA2013 has tips for improving them.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
What Elements of an Online Profile Predict Target-Rater Agreement in Personal...docdave
Study showing people really can read your personality from your online profile. Presented April 2008 at the Int'l Conference on Weblogs and Social Media (ICWSM) in Seattle, Washington, USA.
Plain language to improve your survey houston 2022Caroline Jarrett
Plain language skills are vital for surveys - and especially to writing good questions and creating them for your survey audience. This presentation was prepared for the University of Houston's 8th Biannual Forum on Plain English, 24 February 2022.
Surveys are still really popular as a research method with colleagues (if not with service designers).
These slides are from a workshop at the 2021 Service Design in Government conference (@sdingov21) on 'how to improve the survey that is going to happen whether you like it or not'.
In the workshop we looked at a 7-step process for a survey and considered ways of encouraging colleagues to combine surveys with other research methods.
We also practiced techniques for looking at – and improving - a questionnaire.
Some thoughts on surveys: Boye and Company member conference callCaroline Jarrett
Slides from a short presentation on creating effective surveys. The event was a conference call for members of a community network organised by Janus Boye of Boye & Company.
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Caroline Jarrett
In this webinar for product managers, Caroline introduces two key concepts from her book on surveys: identifying the most crucial question as part of getting clear on your goals, and allowing respondents to tell you the things that they want to - their burning issue. The webinar was organised by Productboard and held on March 30, 2023.
In this half day workshop for ~WebExpo2023 Caroline Jarrett shares four ways to improve your survey so that you get plenty of useful responses.
Goals: Ruthlessly focus your survey on an immediate decision.
Sample: Write an invitation that makes people want to answer.
Questions: Ditch the rating scales.
Responses: Lose your fear of open answers.
Some thoughts on good survey design delivered to students at Olin College of Engineering. Caroline's talk covers her survey process, survey goals and focusing on a specific decision, sample and sampling error, ditching rating scales, and losing fear of open answers.
Two ways to improve your survey, webinar for Delib 2023.pptxCaroline Jarrett
In this webinar for Delib, Caroline shows you how to get better results from shorter, more frequent surveys - with a special emphasis on local government and the requirement to run statutory consultations. Understanding and identifying the Most Crucial Question and making space for the Burning Issue are both helpful techniques for creating shorter more focused surveys.
A presentation for the the Content Wrangler's coffee and content session on how to design and run surveys and gain actionable insights from the survey data.
Ideas for extracting the maximum value from a survey that is going to happen anyway.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Did you love the form that you filled in most recently? Or did you hit some problems? Most of us find all sorts of small or major problems with lots of the forms we are forced to use.
In this talk for #WebExpo2023, Caroline turns that around. She points out the ways in which not fixing your forms is costing your organisation a lot of money. She then goes on to share plenty of practical tips for making improvements that will enable people to successfully complete your forms.
Social Query is a new and efficient way to get
answers on the social networks. However, the popular method of sharing public questions could be optimized by directing the question to an expert, a process called query routing. In this work, we propose a Social Query System for query routing on Twitter, currently, one of the most popular social networks. The Social Query Systems analyzes the information about the questioner’s followers and recommends the most suitable users to answer the questions. The use of the system changes the usual process, working apart of Twitter and allowing questioner and responder exceed the limit of 140 characters. Through a qualitative evaluation, we showed promising results and ideas for improving the system and the recommendation algorithm.
Forms that work: Understanding forms to improve their design by @cjformsCaroline Jarrett
A day-long workshop on forms design, focusing on why businesses need forms and how people interact with them.
Accessibility note: I've tried to make this version of the presentation accessible. If you find that it's not working for you, please let me know and I'll try my best to solve the problems.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
A presentation made to charities to introduce the idea of sharing their followers' social cognitive surplus for the benefit of their charities - provided that we could insure that their personal identity information never sold or compromised.
In this workshop for the Virtual SDinGov 2024 , Caroline takes participants through two sets of guidelines in search of advice on how to make a single forms question accessible. She then introduces her own question protocol as a method of scrutinising and improving any question.
The Phylogenetic Tree in forms design - making forms work for complex academ...Caroline Jarrett
How can we guide busy academics in specialist fields through application processes that are complex, vary greatly depending on the funder, and always seem to be extra urgent? Especially when the stakes are high: awards can be in the millions, and research income is important to fund work that we can all benefit from.
For this year's HE Connect conference, Cambridge University Senior Product Manager Karen Fernandes and forms expert Caroline Jarrett reflected on how current work at Cambridge, and government forms patterns, can help (or hinder) this sort of multi-person, multi-challenge process.
In this member call for Boye & Co Caroline takes participants through her process for expert reviews of forms. She also shares some of her top tips for making them easier to use and more effective.
What is a service designer SDinGOV 22 with all stickies.pptxCaroline Jarrett
In this case study for the 2022 Service Design in Government conference Caroline challenges people to think about their own definitions and shares her own - which is based on her three-layer model for creating good forms.
Helping teenage boys to become responsible adults.pptxCaroline Jarrett
Teenage boys use our services but many of us know little about them. In this session, Bukola (Kiki) Jolugbo and Caroline Jarrett shared some facts about teenage boys and some principles for helping them to become responsible adults.
Overview of how to make good forms that explains that a form builder can help, but it's essential to understand why you're asking the questions - and to write good questions.
Inwards and outwards research: choosing your research methods according to th...Caroline Jarrett
Is your user research looking inwards, at how your service works, or outwards, at the lives of those it affects?
The right research in the right direction at the right time can truly add value - but there’s usually no point in running a survey of 10,000 people in discovery or waiting until beta to look for high-level user needs.
This session, run with Clara Greo at the 2020 Service Design in Government conference, was a chance for colleagues to share their research questions, and think about how to map them to the right methods.
Write Clearly: take your web writing to the next level, May 2016Caroline Jarrett
These slides, setting out a series of rules for producing clear and effective web writing, come from a workshop delivered to staff of EBI/EMBL in May 2016
Understanding the costs of data capture: paper, automatic and with the intern...Caroline Jarrett
Organisations have sometimes been surprised and disappointed when they re-engineer a forms-based data capture process but fail to achieve their anticipated savings.
This paper, delivered to the CIMTECH conference, University of Hertfordshire, in 2000 explains:
how capture costs are built up from data entry plus dealing with problems
an example of costs for an automated process, and for dealing with the paper forms that are left after you bring in an internet process
four techniques for investigating the costs of your current process.
Book Formatting: Quality Control Checks for DesignersConfidence Ago
This presentation was made to help designers who work in publishing houses or format books for printing ensure quality.
Quality control is vital to every industry. This is why every department in a company need create a method they use in ensuring quality. This, perhaps, will not only improve the quality of products and bring errors to the barest minimum, but take it to a near perfect finish.
It is beyond a moot point that a good book will somewhat be judged by its cover, but the content of the book remains king. No matter how beautiful the cover, if the quality of writing or presentation is off, that will be a reason for readers not to come back to the book or recommend it.
So, this presentation points designers to some important things that may be missed by an editor that they could eventually discover and call the attention of the editor.
Between Filth and Fortune- Urban Cattle Foraging Realities by Devi S Nair, An...Mansi Shah
This study examines cattle rearing in urban and rural settings, focusing on milk production and consumption. By exploring a case in Ahmedabad, it highlights the challenges and processes in dairy farming across different environments, emphasising the need for sustainable practices and the essential role of milk in daily consumption.
ARENA - Young adults in the workplace (Knight Moves).pdfKnight Moves
Presentations of Bavo Raeymaekers (Project lead youth unemployment at the City of Antwerp), Suzan Martens (Service designer at Knight Moves) and Adriaan De Keersmaeker (Community manager at Talk to C)
during the 'Arena • Young adults in the workplace' conference hosted by Knight Moves.
Connect Conference 2022: Passive House - Economic and Environmental Solution...TE Studio
Passive House: The Economic and Environmental Solution for Sustainable Real Estate. Lecture by Tim Eian of TE Studio Passive House Design in November 2022 in Minneapolis.
- The Built Environment
- Let's imagine the perfect building
- The Passive House standard
- Why Passive House targets
- Clean Energy Plans?!
- How does Passive House compare and fit in?
- The business case for Passive House real estate
- Tools to quantify the value of Passive House
- What can I do?
- Resources
Can AI do good? at 'offtheCanvas' India HCI preludeAlan Dix
Invited talk at 'offtheCanvas' IndiaHCI prelude, 29th June 2024.
https://www.alandix.com/academic/talks/offtheCanvas-IndiaHCI2024/
The world is being changed fundamentally by AI and we are constantly faced with newspaper headlines about its harmful effects. However, there is also the potential to both ameliorate theses harms and use the new abilities of AI to transform society for the good. Can you make the difference?
Storytelling For The Web: Integrate Storytelling in your Design ProcessChiara Aliotta
In this slides I explain how I have used storytelling techniques to elevate websites and brands and create memorable user experiences. You can discover practical tips as I showcase the elements of good storytelling and its applied to some examples of diverse brands/projects..
5. People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
Response
6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6
I’m a forms specialist
6Image credit: Flickr, taxrebate.org.uk
7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7
Why do people answer questions?
7Caroline Jarrett @cjforms (CC) BY SA-4.0
8. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
8
9. Caroline Jarrett @cjforms (CC) BY SA-4.0 9
“Please have a look at this survey?”
9
Kill survey!
Kill! Kill!
10. Caroline Jarrett @cjforms (CC) BY SA-4.0 10
“Please have a look at this survey?”
10
What number do
you need to make
your decision?
11. 11
The survey is a systematic method
for gathering information from
(a sample of) entities for the purpose of
constructing quantitative descriptors of
the attributes of the larger population of
which the entities are members.
Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor &
Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
12. Caroline Jarrett @cjforms (CC) BY SA-4.0 12
The aim of a survey is to get a number
that helps you to make a decision
12
14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14
“Yes, to make <decision> we need <number>”
14
Great!
Let’s work
through a few
key topics
15. 15
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
17. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
17
18. Caroline Jarrett @cjforms (CC) BY SA-4.0 18
To work that out, let’s visit the Octopus
Caroline Jarrett
@cjforms
(CC) BY SA-4.0 18
19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19
Start with how many will answer
19
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Fieldwork:
Who answers?
20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20
Whether they’ll answer depends on effort
20
Questions:
What are you asking about?
How many questions?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
21. Caroline Jarrett @cjforms (CC) BY SA-4.0 21
And on the reward you’re offering
Goals and resources:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other incentive?
21
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
22. Then there’s the ‘Justin Bieber North Korea’ problem
22http://www.bbc.com/news/10506482
23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23
If we ask ‘anyone’, we’ll have extra work here
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Whose answers
can we use?
24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24
So it matters where we get our sample
24
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Sample:
the list you
sample from
25. Caroline Jarrett @cjforms (CC) BY SA-4.0 2525
And now it’s easy to work out how many to ask
Sample:
the number of
people to ask
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
26. Caroline Jarrett @cjforms (CC) BY SA-4.0 26
We thought about a lot of topics to work that out
Goals
Sample
Questions
Fieldwork
26
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28
What about this bit?
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
29. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
29
30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30
A good question gets good answers
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response:
Is the question
easy to answer?
31. Caroline Jarrett @cjforms (CC) BY SA-4.0 31
Good answers help you to make decisions
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Insight:
Is the answer
useful?
32. 32
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.0
33. Caroline Jarrett @cjforms (CC) BY SA-4.0 33
All the topics are connected
Goals
Sample
Questions
Fieldwork
33
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
Insight
Response
35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
Response depends on effort, reward, and trust
35
36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36
A good question works in three ways
36
Appropriate
Obvious Interesting
37. Caroline Jarrett @cjforms (CC) BY SA-4.0 3737
Why did you visit our website today?
Appropriate
Obvious Interesting
38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38
Would you recommend us to a friend or family member?
In a shop,
buying a baby carriage
In a hospital,
having a miscarriage
Obvious Yes
Interesting Yes
Appropriate Yes Cruelly inappropriate
38
39. Tip
Test your questions by
interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0 39
43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43
Big Honkin’ Survey = think hard about everything
Goals
Sample
Questions
Fieldwork
43
Caroline Jarrett
@cjforms
(CC) BY SA-4.0
Response
Insight
Response
44. 2015 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
44
47. From goals to insight - quickly
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
47
48. Time for new question
48
One way to
iterate,
improve,
increase
Caroline Jarrett @cjforms (CC) BY SA-4.0
49. People ask me about surveys
“Please have a look at this survey”
“How many people do I need in my sample?”
“Tell me whether this is a good question”
“I prefer 5 points in a rating scale, but my boss likes 7.
Who’s is right?”
49
50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50
Likert had several different types
of question in his response formats
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 50
51. You can find an academic paper to support
almost any number of points
Krosnick and Presser refer to over 80 papers
51
Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design.
Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier.
http://comm.stanford.edu/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf
52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52
Respondents focus on the actual question
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 52
53. Tip
Don’t stress too much about
the number of points in your
rating scale
53Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
54. Caroline Jarrett @cjforms (CC) BY SA-4.0 54
Well, OK, stress a little bit.
54
This scale is
downright peculiar.
Avoid.
56. Caroline Jarrett @cjforms (CC) BY SA-4.0 5656
The aim is to get the best number you can,
within the resources you have
57. Caroline Jarrett @cjforms (CC) BY SA-4.0 57
What you want to ask about
The resources you have
The questions you ask
The answers you get
The answers you use
The number
Who you want to ask
The list that you sample from
The sample you ask
The ones who answer
The ones whose answers
you can use
57
The aim is to get the best number you can,
within the resources you have
58. Caroline Jarrett @cjforms (CC) BY SA-4.0 58
The aim is to get the best number you can,
within the resources you have
What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
The number
59. Caroline Jarrett @cjforms (CC) BY SA-4.0 5959
Survey statistic
Post-survey
adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
The aim is to get the best number you can,
within the resources you have
Resources
What you want
to ask about
The resources
you have
The questions
you ask
The answers
you get
The answers
you use
Who you want
to ask
The list you use
to sample from
The ones you
ask
The ones who
answer
The ones whose
answers you can use
60. Caroline Jarrett @cjforms (CC) BY SA-4.0 6060
Survey statistic
Post-survey adjustments
Respondents
Sample
Sampling frame
Representation
Edited response
Response
Measurement
Construct
By making good choices, you minimise the error
Resources
61. 61
Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
62. What number do you need to make the decision?
Use your UX skills
(especially technical communication)
Go for Light Touch surveys
Use the Survey Octopus to make good choices
Caroline Jarrett @cjforms (CC) BY SA-4.0 62
This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The octopus again. This time we're looking at 'the questions we ask'.
The resources you have will help you to decide on the reward you’re offering
Prank leaves Justin Bieber facing tour of North Korea
By Daniel Emery Technology reporter, BBC News
5 July 2010
Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea.
A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on.
There are now almost half a million votes to send the singer to the secretive communist nation.
The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days.
Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
The octopus, with focus on 'The list you sample from'
The octopus again; we've looked at 6 of the 8 tentacles.
The octopus again; we've looked at 6 of the 8 tentacles.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Obvious questions require no effort to answer. Interesting questions are rewarding to answer. Appropriate questions are ones that inspire trust in the respondent.
People come to the web with their own questions, so they’re likely to know why they are on your website. It’s a relatively interesting question to answer, and it’s appropriate to ask visitors why they’re visiting.
If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
The octopus again; we've looked at 6 of the 8 tentacles.
Excerpt from Death to ‘it depends’ by Leisa Reichelt (@leisa)
Death to ‘it depends’
Lately I find myself on a mission for mass simplification. Possibly over simplification, but I’m not sure it matters.
It’s one of the things I care most about at the moment – how can we simplify what we are asking people to do so that there is nothing else they can do but start doing it, instead of following their natural inclination to make a list, hire a consultant, write a white paper, do anything but doing the thing.
It requires that I stop saying (or even thinking) one of the things I have probably said most in my entire working life – ‘it depends’. That’s hard, but I think it’s the right thing to do.
It depends is paralysing.
Read more at:http://www.disambiguity.com/death-to-it-depends/
The octopus again; we've looked at 6 of the 8 tentacles.
A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up.
It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
This is a more conventional way of looking at the octopus tentacles
If we just look at the issues (no tentacles) we get this slide
This slide translates the issues into the technical terms used by survey methodologists
This slide translates the issues into the technical terms used by survey methodologists