SlideShare a Scribd company logo
1 of 75
Design tips
for surveys
2012
Caroline Jarrett
User Interface Engineering / Rosenfeld Media Seminar
We need lots of data.
Your samples are too small.
We’re going to do a survey.
But, but… er, er, …
OK, you’re the boss
Please make it one that
I’ll enjoy answering
2
We need surveys in our toolbox
1. You get lots of lovely numbers
2. It’s much easier to manage what you can measure
3. They’re going to do a survey anyway,
so let’s make it a good one
And the more of them I
do, the more I like them
3
Survey = Questionnaire + Process
4
Agenda Entice people to take a survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
5
Your answers to this survey are
important for my business.
But what’s in it for me?
And I’m really busy today.
6
Response relies on
effort, reward, and trust
Trust
Perceived
effort
Perceived
reward
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
7
Reward, trust and effort for this invitation
Intrinsic reward
- but is it enough?
Short is good,
but do I trust
you enough to
believe you?
Who are you?
Will you spam me?
8
We’ll offer a chance to win $100.
That will get lots of responses.
But I don’t believe
I’ll be the winner
9
A small immediate payment works best for mail
0%
50%
100%
No incentive $300 lottery $2 bill
Response rates
Carlson, 1996 quoted in Dillman, D.A., Smyth, J. D. and Christian, L. M. (2009)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 10
‘Small immediate payment’ is difficult online
0%
50%
100%
No incentive Prize draw (2x
$50,2x $25 prize)
$2 via Pay Pal
before survey
$2 via PayPal on
completion
Accessed first page
Michael Bosnjak and Tracy L. Tute (2003)
“Prepaid and Promised Incentives in Web Surveys : An Experiment”
Social Science Computer Review 2003 21: 20
11
Tip 1
Offer a meaningful,
immediate reward
Picture credit:
Flickr -
Jessica Wilson (jek in the box)
12
Let’s look again at trust
Who are you?
Will you spam me?
13
A well-known brand on a well-designed site
I definitely know you
And you won’t spam me 14
Tip 2
Be trustworthy and
look trustworthy
15
You know that survey? I want
to know where they live
We have to ask them about
navigation, obviously
What about the new
graphics?
I really need the
‘recommend to a
friend’ question
We have to ask about
the content!
And finding us … we
have to know about
that Don’t forget about
the new feature
16
I respond to a lot of surveys
89 invitations
67 completed surveys
1387questions 17
They wanted my opinions but it took 6 screens
to get to the first opinion question
18
Could the BBC trim some of these?
• Gender
• Region you live in
• Ethnic origin
• Type of Internet
connection
• Rate how it looks
• What do you like about it
• What do you dislike about it
• Your main reason for visiting
the site today
• And many more
Image credit: Flickr - Enokson
19
Tip 3
Focus on the questions
that deliver the insights
that you need the most
Image credit: Fraser Smith
20
Remember 1387 questions?
This survey contributed 30 of them
21
On and on
22
On and on and on and on and on
I love John Lewis!
I shop there online and offline!
Don’t make me feel bad because
I didn’t read your buying guide.
Just let me tell you I love you…
23
Your users may not be as interested in your
design decisions as you are.
“Wood for the trees” Image credit: Flickr - Paul & Aline Burland (P – A – S – T)
24
Tip 4
To get a good survey, talk
to lots of users
25
Agenda Entice people to take a survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
26
Back to the BBC. After I gave my initial
opinions, they asked me some ratings
27
Most surveys I see have ratings
from 1 to 5
Wouldn’t it be easier to have just two ?
7 points are more accurate!
What about 1 to 10?
28
Likert had several different types
of question in his scales
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 29
You can find an academic paper to support
almost any number of points in a range
• Krosnick and Presser refer to ~87 papers on ranges
• Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design.
Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier
• If you’d like to track down the research yourself:
http://comm.stanford.edu/faculty/krosnick/
Handbook%20of%20Survey%20Research.pdf
30
Users don’t care much about the number of
points. They care more about the questions.
What’s the difference
between those questions?
This survey is too repetitive.
31
Tip 5
Don’t stress about the
number of points in your
rating scale
Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
32
The BBC didn’t stress about the number of
points in the rating scale. This one has 10
33
This one has 5. And I never noticed at the time.
34
Let’s have another look
at one of Likert’s questions
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55.
Complex clause A rather big commitment
And what are the
programs anyway?
Double-barrelled
35
Before- and after- for Likert’s question?
17a. The United States should join in the humanitarian work of the League of Nations
17b. The United States should join in the economic work of the League of Nations
36
The BBC also asked me a complex question
37
Let’s split it and simplify it
It is easy to select products on this service
It is easy to find suppliers on this service
38
Tip 6
Refine questions so
they’re crystal clear
Present one question at a time
Use plain language
http://www.plainlanguage.gov/howto/index.cfm
39
How much do you care about the economic
programs of the League of Nations?
Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.
Archives of Psychology, 140, 1–55. 40
OK, so how much do you care about…
41
Two ways of asking the same question:
One of them calls for imagination
What is your favourite meal?
• Steak and chips
Imagine you are on death row and
have to choose your last meal
• Scallops with black pudding and
cream, rib eye steak with chips
and a dolce latte cream sauce;
stinking bishop (cheese) with
1960 port (year of my birth).
Wine would have to be Chateau
Lafite 1st Cru Pauillac 2000.
Puleston, J. and Sleep, D (2011) “The Game Experiments”
ESOMAR Congress
42
Could we make the question more interesting?
Please rate the overall
information content
on this web site
Boring
43
Could we make the question more interesting?
Please rate the overall
information content
on this web site
Story:
If you could wave a
magic wand and
make a new page for this web site,
what would it do or say?
Game:
You’ve asked us to improve the
information content on this web site.
Here are some different changes we
thought of and what they would cost. If
you had $10,000 to spend, which
would you choose?
44
Cambridgeshire police tried the game idea
45
Tip 7
Write interesting questions
Picture credit: Shutterstock.com
46
Agenda Entice people to take your survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
47
Does your household have access
to running water?
Picture credits: Flickr - James Mooney; Caroline Jarrett 48
“Shared reference”:
both sides interpret in the same way
Reiss, E. (2000) “Practical Information Architecture:
A Hands-On Approach to Structuring Successful Web Sites”
Picture credits: Flickr - James Mooney; Caroline Jarrett 49
“Shared reference”:
both sides interpret in the same way
Photo credit: Flickr - chaptstickaddict
Service area?
50
Tip 8
Make sure that your users
understand your questions
in the same way that you do
51
Let’s look at putting the answer on the page.
52
Here, the ‘agree’ and ‘disagree’ have floated
53
Not hard to fix.
(And let’s delete excess instructions while we’re about it.)
54
Tip 9
Design the page so people
can answer easily
http://www.slideshare.net/cjforms/labels-and-buttons-on-forms
55
Agenda Entice people to take your survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
56
We got so much great data
that I put every detail in
this report
!!!!
57
Surveys can get you lots of lovely data
58
Cutting out some detail can help a lot
59
This graph could be good news or bad news
depending on the question we ask of it
60
This graph could be good news or bad news
depending on the question we ask of it
How are we
doing in the 35-
44 age group? Strong response from key target
61
This graph could be good news or bad news
depending on the question we ask of it
How are we
doing with the
seniors?
Disappointing results from seniors
62
Tip 10
Think about the story the
data can tell
Perceptual Edge has a Graph Design IQ test:
http://www.perceptualedge.com/files/GraphDesignIQ.html
63
My best ever survey was very simple:
needed data, got it, made decision
64
“Quality doesn’t matter”
We are focusing our time on technicalities of research,
whereas the expectation from clients is actually
in a totally different space.
The clients are saying, ‘Inspire me,
help me to take some transformational action’
Stan Sthanunathan
Vice President
Marketing Strategy & Insights
The Coca-Cola Company
“Research”, Oct 2009 http://www.research-live.com/4001230.article
Image credit: http://www.eltiempo.com/gente/ARTICULO-WEB-
NEW_NOTA_INTERIOR-9736789.html
65
Insight = information that provokes action
If information doesn’t change our,
or our consumers’, behavior
then we shouldn’t be doing it.
Danny Russell,
Marketing Strategy Director,
BSkyB
“Research”, January 2012
http://www.research-live.com/features/who-are-you-talking-to?/4006763.article
66
Tip 11
Know what decision you’ll
make based on this survey
Image credit: Fraser Smith
67
We need to know our
satisfaction score
Hmm… satisfaction has many aspects.
Let’s ask about aesthetics, desirability,
navigation….
68
But what about
satisfaction?
Good news! We’re rating 5 for
aesthetics, 5 for desirability
and 5 for navigation!
69
Tip 12
Know “The Most
Crucial Question”
and don’t
forget to ask it
70Image credit:
Shutterstock
Agenda Entice people to take your survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
71
Remember my definition?
Survey = Questionnaire
+ Process
72
A successful survey is a questionnaire
plus a process that involves lots of
testing
Bonus
Tip
73
Agenda Entice people to take your survey
Get users to engage with questions
Break: seminar Q & A
Get users to answer accurately
Find insight for your stakeholders
Bonus tip
Wrap up: seminar Q & A
74
Caroline Jarrett
twitter @cjforms
caroline.jarrett@effortmark.co.uk
Slides: http://www.slideshare.net/cjforms
Blog posts and more resources:
http://www.rosenfeldmedia.com/books/survey-design/
75

More Related Content

What's hot

Writing surveysthatwork
Writing surveysthatworkWriting surveysthatwork
Writing surveysthatwork
rebeccaweiss
 
Making sense of the data
Making sense of the dataMaking sense of the data
Making sense of the data
Dana Chisnell
 
Desirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a DesignDesirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a Design
Megan Grocki
 
Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants
Dana Chisnell
 
Worker to hard worker
Worker to hard workerWorker to hard worker
Worker to hard worker
burton96
 

What's hot (20)

Effective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA WorkshopEffective Use of Surveys in UX | Triangle UXPA Workshop
Effective Use of Surveys in UX | Triangle UXPA Workshop
 
Better surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptxBetter surveys - presentation to Loyal Health March 2022.pptx
Better surveys - presentation to Loyal Health March 2022.pptx
 
The inevitable survey 2021
The inevitable survey 2021The inevitable survey 2021
The inevitable survey 2021
 
Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1Surveys that work:training course for Rosenfeld Media, day 1
Surveys that work:training course for Rosenfeld Media, day 1
 
Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2Surveys that work: training course for Rosenfeld media, day 2
Surveys that work: training course for Rosenfeld media, day 2
 
Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3 Surveys that work: training course for Rosenfeld Media, day 3
Surveys that work: training course for Rosenfeld Media, day 3
 
Writing surveysthatwork
Writing surveysthatworkWriting surveysthatwork
Writing surveysthatwork
 
Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021Surveys that work: a webinar for UXPA 2021
Surveys that work: a webinar for UXPA 2021
 
Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...Inwards and outwards research: choosing your research methods according to th...
Inwards and outwards research: choosing your research methods according to th...
 
Making sense of the data
Making sense of the dataMaking sense of the data
Making sense of the data
 
Counter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methodsCounter to intent: Voters' mental models of alternative counting methods
Counter to intent: Voters' mental models of alternative counting methods
 
Desirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a DesignDesirability Testing: Analyzing Emotional Response to a Design
Desirability Testing: Analyzing Emotional Response to a Design
 
UX Lesson 2: User Research
UX Lesson 2: User ResearchUX Lesson 2: User Research
UX Lesson 2: User Research
 
Design Research Synthesis
Design Research SynthesisDesign Research Synthesis
Design Research Synthesis
 
Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants Gaining design insight through recruiting research participants
Gaining design insight through recruiting research participants
 
Audience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of InfluenceAudience Research on a Dime - Nonprofit of Influence
Audience Research on a Dime - Nonprofit of Influence
 
U Penn Wharton design challenge '17
U Penn Wharton design challenge '17U Penn Wharton design challenge '17
U Penn Wharton design challenge '17
 
Worker to hard worker
Worker to hard workerWorker to hard worker
Worker to hard worker
 
How to find out about the usability of your web site using a survey by @cjforms
How to find out about the usability of your web site using a survey by @cjformsHow to find out about the usability of your web site using a survey by @cjforms
How to find out about the usability of your web site using a survey by @cjforms
 
Coaching teams in creative problem solving
Coaching teams in creative problem solvingCoaching teams in creative problem solving
Coaching teams in creative problem solving
 

Similar to Design tips for surveys UIE 2012

AMA Reseach & Strategy Summit: Community Workshop
AMA Reseach & Strategy Summit: Community WorkshopAMA Reseach & Strategy Summit: Community Workshop
AMA Reseach & Strategy Summit: Community Workshop
Tom De Ruyck
 

Similar to Design tips for surveys UIE 2012 (20)

Taking Quantitative Research to a New Level
Taking Quantitative Research to a New LevelTaking Quantitative Research to a New Level
Taking Quantitative Research to a New Level
 
Social Media Week London -- Quizzes: The Secret Weapon of the World's Best Co...
Social Media Week London -- Quizzes: The Secret Weapon of the World's Best Co...Social Media Week London -- Quizzes: The Secret Weapon of the World's Best Co...
Social Media Week London -- Quizzes: The Secret Weapon of the World's Best Co...
 
AMA Reseach & Strategy Summit: Community Workshop
AMA Reseach & Strategy Summit: Community WorkshopAMA Reseach & Strategy Summit: Community Workshop
AMA Reseach & Strategy Summit: Community Workshop
 
Keeping research respondents engaged and interested in research
Keeping research respondents engaged and interested in researchKeeping research respondents engaged and interested in research
Keeping research respondents engaged and interested in research
 
Online Survey Research: Harnessing the Power of "Big Data"
Online Survey Research: Harnessing the Power of "Big Data"Online Survey Research: Harnessing the Power of "Big Data"
Online Survey Research: Harnessing the Power of "Big Data"
 
How to Effectively Incorporate Quizzes in Your Content Strategy
How to Effectively Incorporate Quizzes in Your Content StrategyHow to Effectively Incorporate Quizzes in Your Content Strategy
How to Effectively Incorporate Quizzes in Your Content Strategy
 
How to Make One of Those Insanely Viral Quizzes You've Been Seeing Everywhere
How to Make One of Those Insanely Viral Quizzes You've Been Seeing EverywhereHow to Make One of Those Insanely Viral Quizzes You've Been Seeing Everywhere
How to Make One of Those Insanely Viral Quizzes You've Been Seeing Everywhere
 
The secret to creating viral content out of thin air
The secret to creating viral content out of thin airThe secret to creating viral content out of thin air
The secret to creating viral content out of thin air
 
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
Feedback & Surveys - How to use the Constant Contact Toolkit Part 2
 
Latest updates in social media
Latest updates in social mediaLatest updates in social media
Latest updates in social media
 
Survey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjformsSurvey octopus uxnz2015 @cjforms
Survey octopus uxnz2015 @cjforms
 
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYSTHE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
THE SURVEY OCTOPUS: GETTING VALID DATA FROM SURVEYS
 
Why it's time for a new kind of quantitative research
Why it's time for a new kind of quantitative researchWhy it's time for a new kind of quantitative research
Why it's time for a new kind of quantitative research
 
How to Create the Quizzes Your Audience Will Love – Qzzr/NUVI Webinar
How to Create the Quizzes Your Audience Will Love – Qzzr/NUVI WebinarHow to Create the Quizzes Your Audience Will Love – Qzzr/NUVI Webinar
How to Create the Quizzes Your Audience Will Love – Qzzr/NUVI Webinar
 
A deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLxA deep dive into questions by @cjforms at UxLx
A deep dive into questions by @cjforms at UxLx
 
50 Shades of Respondent Grey
50 Shades of Respondent Grey50 Shades of Respondent Grey
50 Shades of Respondent Grey
 
Clean Interviewing_ A Hands-On Approach to Customer-Centric Product Developme...
Clean Interviewing_ A Hands-On Approach to Customer-Centric Product Developme...Clean Interviewing_ A Hands-On Approach to Customer-Centric Product Developme...
Clean Interviewing_ A Hands-On Approach to Customer-Centric Product Developme...
 
The Insane Viral Power of a Quality Quiz
The Insane Viral Power of a Quality QuizThe Insane Viral Power of a Quality Quiz
The Insane Viral Power of a Quality Quiz
 
Cardiff university 7 pillars of campaign analytics
Cardiff university 7 pillars of campaign analyticsCardiff university 7 pillars of campaign analytics
Cardiff university 7 pillars of campaign analytics
 
How to Design a Brilliant Product Without Killing Your Cofounder
How to Design a Brilliant Product Without Killing Your CofounderHow to Design a Brilliant Product Without Killing Your Cofounder
How to Design a Brilliant Product Without Killing Your Cofounder
 

More from Caroline Jarrett

More from Caroline Jarrett (20)

Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024Creating truly accessible forms, SDinGov 2024
Creating truly accessible forms, SDinGov 2024
 
Using surveys to uncover audience insights
Using surveys to uncover audience insightsUsing surveys to uncover audience insights
Using surveys to uncover audience insights
 
2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx2023 Surveys that work Olin.pptx
2023 Surveys that work Olin.pptx
 
The Phylogenetic Tree in forms design - making forms work for complex academ...
The Phylogenetic Tree in forms design - making forms work for complex  academ...The Phylogenetic Tree in forms design - making forms work for complex  academ...
The Phylogenetic Tree in forms design - making forms work for complex academ...
 
Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023Four ways to make a better survey WebExpo2023
Four ways to make a better survey WebExpo2023
 
Content design in practice May 2023.pptx
Content design in practice May 2023.pptxContent design in practice May 2023.pptx
Content design in practice May 2023.pptx
 
Two ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptxTwo ways to improve your survey, webinar for Delib 2023.pptx
Two ways to improve your survey, webinar for Delib 2023.pptx
 
How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023How to save money by fixing forms, WebExpo2023
How to save money by fixing forms, WebExpo2023
 
Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...Two ways to improve your surveys: the Most Crucial Question and the Burning I...
Two ways to improve your surveys: the Most Crucial Question and the Burning I...
 
How to look at a form Boye 22.pptx
How to look at a form Boye 22.pptxHow to look at a form Boye 22.pptx
How to look at a form Boye 22.pptx
 
Prune tune postpone explain.pptx
Prune tune postpone explain.pptxPrune tune postpone explain.pptx
Prune tune postpone explain.pptx
 
What is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptxWhat is a service designer SDinGOV 22 with all stickies.pptx
What is a service designer SDinGOV 22 with all stickies.pptx
 
Helping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptxHelping teenage boys to become responsible adults.pptx
Helping teenage boys to become responsible adults.pptx
 
Some thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference callSome thoughts on surveys: Boye and Company member conference call
Some thoughts on surveys: Boye and Company member conference call
 
Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022Plain language to improve your survey houston 2022
Plain language to improve your survey houston 2022
 
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey ErrorSurveys that work: an introduction to the Survey Octopus and Total Survey Error
Surveys that work: an introduction to the Survey Octopus and Total Survey Error
 
Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021Surveys that work: a webinar for FocusVision 2021
Surveys that work: a webinar for FocusVision 2021
 
A chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjformsA chat about forms xgov 2020 cjforms
A chat about forms xgov 2020 cjforms
 
Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...Surveys that work: an introduction to using Total Survey Error for the UX Ins...
Surveys that work: an introduction to using Total Survey Error for the UX Ins...
 
Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020Surveys that Work 2020: training course for HMRC user researchers 2020
Surveys that Work 2020: training course for HMRC user researchers 2020
 

Recently uploaded

一比一原版谢菲尔德大学毕业证成绩单如何办理
一比一原版谢菲尔德大学毕业证成绩单如何办理一比一原版谢菲尔德大学毕业证成绩单如何办理
一比一原版谢菲尔德大学毕业证成绩单如何办理
cyebo
 
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
JIT KUMAR GUPTA
 
Presentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentationPresentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentation
ajroy0196
 
Heuristic Evaluation of System & Application
Heuristic Evaluation of System & ApplicationHeuristic Evaluation of System & Application
Heuristic Evaluation of System & Application
Jaime Brown
 

Recently uploaded (20)

Heidi Livengood's Professional CADD Portfolio
Heidi Livengood's Professional CADD PortfolioHeidi Livengood's Professional CADD Portfolio
Heidi Livengood's Professional CADD Portfolio
 
Bit Dhrumi shah Graphic Designer portfolio
Bit Dhrumi shah Graphic Designer portfolioBit Dhrumi shah Graphic Designer portfolio
Bit Dhrumi shah Graphic Designer portfolio
 
Week of Action 2022_EIT Climate-KIC_Headers
Week of Action 2022_EIT Climate-KIC_HeadersWeek of Action 2022_EIT Climate-KIC_Headers
Week of Action 2022_EIT Climate-KIC_Headers
 
Abdulaziz Tariq Abdulaziz Mustafa CV 2024
Abdulaziz Tariq Abdulaziz Mustafa CV 2024Abdulaziz Tariq Abdulaziz Mustafa CV 2024
Abdulaziz Tariq Abdulaziz Mustafa CV 2024
 
Design lessons from Singapore | Volume 3
Design lessons from Singapore | Volume 3Design lessons from Singapore | Volume 3
Design lessons from Singapore | Volume 3
 
BIT Khushi gandhi project.pdf graphic design
BIT Khushi gandhi project.pdf graphic designBIT Khushi gandhi project.pdf graphic design
BIT Khushi gandhi project.pdf graphic design
 
一比一原版谢菲尔德大学毕业证成绩单如何办理
一比一原版谢菲尔德大学毕业证成绩单如何办理一比一原版谢菲尔德大学毕业证成绩单如何办理
一比一原版谢菲尔德大学毕业证成绩单如何办理
 
Redefining Globalization, urbanisation and Localisation
Redefining Globalization, urbanisation and LocalisationRedefining Globalization, urbanisation and Localisation
Redefining Globalization, urbanisation and Localisation
 
Dos And Dont's Of Logo Design For 2024..
Dos And Dont's Of Logo Design For 2024..Dos And Dont's Of Logo Design For 2024..
Dos And Dont's Of Logo Design For 2024..
 
spColumn-Manual design column by spcolumn software.pdf
spColumn-Manual design column by spcolumn software.pdfspColumn-Manual design column by spcolumn software.pdf
spColumn-Manual design column by spcolumn software.pdf
 
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...Knowing, Understanding and Planning Cities- Role and Relevance  Physical Plan...
Knowing, Understanding and Planning Cities- Role and Relevance Physical Plan...
 
iF_Design_Trend_Report_twentytwenrythree
iF_Design_Trend_Report_twentytwenrythreeiF_Design_Trend_Report_twentytwenrythree
iF_Design_Trend_Report_twentytwenrythree
 
Recycled Modular Low Cost Construction .pdf
Recycled Modular Low Cost Construction .pdfRecycled Modular Low Cost Construction .pdf
Recycled Modular Low Cost Construction .pdf
 
Presentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentationPresentation on 3D Printing.pptx presentation
Presentation on 3D Printing.pptx presentation
 
Claire's designing portfolio presentation
Claire's designing portfolio presentationClaire's designing portfolio presentation
Claire's designing portfolio presentation
 
Heuristic Evaluation of System & Application
Heuristic Evaluation of System & ApplicationHeuristic Evaluation of System & Application
Heuristic Evaluation of System & Application
 
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
NO1 Popular kala jadu karne wale ka contact number kala jadu karne wale baba ...
 
Spring 2024 wkrm_Enhancing Campus Mobility.pdf
Spring 2024 wkrm_Enhancing Campus Mobility.pdfSpring 2024 wkrm_Enhancing Campus Mobility.pdf
Spring 2024 wkrm_Enhancing Campus Mobility.pdf
 
FW25-26 Fashion Key Items Trend Book Peclers Paris
FW25-26 Fashion Key Items Trend Book Peclers ParisFW25-26 Fashion Key Items Trend Book Peclers Paris
FW25-26 Fashion Key Items Trend Book Peclers Paris
 
Design Portofolios - Licensed Architect / BIM Specialist
Design Portofolios - Licensed Architect / BIM SpecialistDesign Portofolios - Licensed Architect / BIM Specialist
Design Portofolios - Licensed Architect / BIM Specialist
 

Design tips for surveys UIE 2012

  • 1. Design tips for surveys 2012 Caroline Jarrett User Interface Engineering / Rosenfeld Media Seminar
  • 2. We need lots of data. Your samples are too small. We’re going to do a survey. But, but… er, er, … OK, you’re the boss Please make it one that I’ll enjoy answering 2
  • 3. We need surveys in our toolbox 1. You get lots of lovely numbers 2. It’s much easier to manage what you can measure 3. They’re going to do a survey anyway, so let’s make it a good one And the more of them I do, the more I like them 3
  • 5. Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 5
  • 6. Your answers to this survey are important for my business. But what’s in it for me? And I’m really busy today. 6
  • 7. Response relies on effort, reward, and trust Trust Perceived effort Perceived reward Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 7
  • 8. Reward, trust and effort for this invitation Intrinsic reward - but is it enough? Short is good, but do I trust you enough to believe you? Who are you? Will you spam me? 8
  • 9. We’ll offer a chance to win $100. That will get lots of responses. But I don’t believe I’ll be the winner 9
  • 10. A small immediate payment works best for mail 0% 50% 100% No incentive $300 lottery $2 bill Response rates Carlson, 1996 quoted in Dillman, D.A., Smyth, J. D. and Christian, L. M. (2009) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 10
  • 11. ‘Small immediate payment’ is difficult online 0% 50% 100% No incentive Prize draw (2x $50,2x $25 prize) $2 via Pay Pal before survey $2 via PayPal on completion Accessed first page Michael Bosnjak and Tracy L. Tute (2003) “Prepaid and Promised Incentives in Web Surveys : An Experiment” Social Science Computer Review 2003 21: 20 11
  • 12. Tip 1 Offer a meaningful, immediate reward Picture credit: Flickr - Jessica Wilson (jek in the box) 12
  • 13. Let’s look again at trust Who are you? Will you spam me? 13
  • 14. A well-known brand on a well-designed site I definitely know you And you won’t spam me 14
  • 15. Tip 2 Be trustworthy and look trustworthy 15
  • 16. You know that survey? I want to know where they live We have to ask them about navigation, obviously What about the new graphics? I really need the ‘recommend to a friend’ question We have to ask about the content! And finding us … we have to know about that Don’t forget about the new feature 16
  • 17. I respond to a lot of surveys 89 invitations 67 completed surveys 1387questions 17
  • 18. They wanted my opinions but it took 6 screens to get to the first opinion question 18
  • 19. Could the BBC trim some of these? • Gender • Region you live in • Ethnic origin • Type of Internet connection • Rate how it looks • What do you like about it • What do you dislike about it • Your main reason for visiting the site today • And many more Image credit: Flickr - Enokson 19
  • 20. Tip 3 Focus on the questions that deliver the insights that you need the most Image credit: Fraser Smith 20
  • 21. Remember 1387 questions? This survey contributed 30 of them 21
  • 23. On and on and on and on and on I love John Lewis! I shop there online and offline! Don’t make me feel bad because I didn’t read your buying guide. Just let me tell you I love you… 23
  • 24. Your users may not be as interested in your design decisions as you are. “Wood for the trees” Image credit: Flickr - Paul & Aline Burland (P – A – S – T) 24
  • 25. Tip 4 To get a good survey, talk to lots of users 25
  • 26. Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 26
  • 27. Back to the BBC. After I gave my initial opinions, they asked me some ratings 27
  • 28. Most surveys I see have ratings from 1 to 5 Wouldn’t it be easier to have just two ? 7 points are more accurate! What about 1 to 10? 28
  • 29. Likert had several different types of question in his scales Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 29
  • 30. You can find an academic paper to support almost any number of points in a range • Krosnick and Presser refer to ~87 papers on ranges • Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier • If you’d like to track down the research yourself: http://comm.stanford.edu/faculty/krosnick/ Handbook%20of%20Survey%20Research.pdf 30
  • 31. Users don’t care much about the number of points. They care more about the questions. What’s the difference between those questions? This survey is too repetitive. 31
  • 32. Tip 5 Don’t stress about the number of points in your rating scale Picture credit: Flickr - Bill Soderman (BillsoPHOTO) 32
  • 33. The BBC didn’t stress about the number of points in the rating scale. This one has 10 33
  • 34. This one has 5. And I never noticed at the time. 34
  • 35. Let’s have another look at one of Likert’s questions Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. Complex clause A rather big commitment And what are the programs anyway? Double-barrelled 35
  • 36. Before- and after- for Likert’s question? 17a. The United States should join in the humanitarian work of the League of Nations 17b. The United States should join in the economic work of the League of Nations 36
  • 37. The BBC also asked me a complex question 37
  • 38. Let’s split it and simplify it It is easy to select products on this service It is easy to find suppliers on this service 38
  • 39. Tip 6 Refine questions so they’re crystal clear Present one question at a time Use plain language http://www.plainlanguage.gov/howto/index.cfm 39
  • 40. How much do you care about the economic programs of the League of Nations? Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. Archives of Psychology, 140, 1–55. 40
  • 41. OK, so how much do you care about… 41
  • 42. Two ways of asking the same question: One of them calls for imagination What is your favourite meal? • Steak and chips Imagine you are on death row and have to choose your last meal • Scallops with black pudding and cream, rib eye steak with chips and a dolce latte cream sauce; stinking bishop (cheese) with 1960 port (year of my birth). Wine would have to be Chateau Lafite 1st Cru Pauillac 2000. Puleston, J. and Sleep, D (2011) “The Game Experiments” ESOMAR Congress 42
  • 43. Could we make the question more interesting? Please rate the overall information content on this web site Boring 43
  • 44. Could we make the question more interesting? Please rate the overall information content on this web site Story: If you could wave a magic wand and make a new page for this web site, what would it do or say? Game: You’ve asked us to improve the information content on this web site. Here are some different changes we thought of and what they would cost. If you had $10,000 to spend, which would you choose? 44
  • 45. Cambridgeshire police tried the game idea 45
  • 46. Tip 7 Write interesting questions Picture credit: Shutterstock.com 46
  • 47. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 47
  • 48. Does your household have access to running water? Picture credits: Flickr - James Mooney; Caroline Jarrett 48
  • 49. “Shared reference”: both sides interpret in the same way Reiss, E. (2000) “Practical Information Architecture: A Hands-On Approach to Structuring Successful Web Sites” Picture credits: Flickr - James Mooney; Caroline Jarrett 49
  • 50. “Shared reference”: both sides interpret in the same way Photo credit: Flickr - chaptstickaddict Service area? 50
  • 51. Tip 8 Make sure that your users understand your questions in the same way that you do 51
  • 52. Let’s look at putting the answer on the page. 52
  • 53. Here, the ‘agree’ and ‘disagree’ have floated 53
  • 54. Not hard to fix. (And let’s delete excess instructions while we’re about it.) 54
  • 55. Tip 9 Design the page so people can answer easily http://www.slideshare.net/cjforms/labels-and-buttons-on-forms 55
  • 56. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 56
  • 57. We got so much great data that I put every detail in this report !!!! 57
  • 58. Surveys can get you lots of lovely data 58
  • 59. Cutting out some detail can help a lot 59
  • 60. This graph could be good news or bad news depending on the question we ask of it 60
  • 61. This graph could be good news or bad news depending on the question we ask of it How are we doing in the 35- 44 age group? Strong response from key target 61
  • 62. This graph could be good news or bad news depending on the question we ask of it How are we doing with the seniors? Disappointing results from seniors 62
  • 63. Tip 10 Think about the story the data can tell Perceptual Edge has a Graph Design IQ test: http://www.perceptualedge.com/files/GraphDesignIQ.html 63
  • 64. My best ever survey was very simple: needed data, got it, made decision 64
  • 65. “Quality doesn’t matter” We are focusing our time on technicalities of research, whereas the expectation from clients is actually in a totally different space. The clients are saying, ‘Inspire me, help me to take some transformational action’ Stan Sthanunathan Vice President Marketing Strategy & Insights The Coca-Cola Company “Research”, Oct 2009 http://www.research-live.com/4001230.article Image credit: http://www.eltiempo.com/gente/ARTICULO-WEB- NEW_NOTA_INTERIOR-9736789.html 65
  • 66. Insight = information that provokes action If information doesn’t change our, or our consumers’, behavior then we shouldn’t be doing it. Danny Russell, Marketing Strategy Director, BSkyB “Research”, January 2012 http://www.research-live.com/features/who-are-you-talking-to?/4006763.article 66
  • 67. Tip 11 Know what decision you’ll make based on this survey Image credit: Fraser Smith 67
  • 68. We need to know our satisfaction score Hmm… satisfaction has many aspects. Let’s ask about aesthetics, desirability, navigation…. 68
  • 69. But what about satisfaction? Good news! We’re rating 5 for aesthetics, 5 for desirability and 5 for navigation! 69
  • 70. Tip 12 Know “The Most Crucial Question” and don’t forget to ask it 70Image credit: Shutterstock
  • 71. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 71
  • 72. Remember my definition? Survey = Questionnaire + Process 72
  • 73. A successful survey is a questionnaire plus a process that involves lots of testing Bonus Tip 73
  • 74. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 74
  • 75. Caroline Jarrett twitter @cjforms caroline.jarrett@effortmark.co.uk Slides: http://www.slideshare.net/cjforms Blog posts and more resources: http://www.rosenfeldmedia.com/books/survey-design/ 75