SlideShare a Scribd company logo
1 of 34
Download to read offline
@uxordie
Text fields and dropdowns and radios oh my!
USABILITY TESTING ONLINE SURVEYS (AND FORMS)
Eva Kaniasty
UXPA International Conference
@uxordie
@uxordie@uxordie
1. Why surveys? 

Isn’t user research enough?
2. Why pre-test surveys?

Let me count the ways.
3. Four stage questionnaire model

The Cognitive Psych (Tourangeau)
4. The cognitive interview

Usability testing… just for surveys. (Willis)
5. Survey Pre-Testing Process

3 scenarios.
AGENDA? AGENDA.
2
@uxordie
WHY SURVEYS?
@uxordie@uxordie
• Validate qualitative research results
• Get feedback from a larger sample
• Build a participant database
• Bridge gap between quant and qual
• Build a strong case for change
• Kill features
WHY SURVEYS?
4
“Doesn’t everybody love a survey?”
• Execs
• Business People
• Marketing People
• Sales People
@uxordie@uxordie
WHY SURVEYS?
5
• Surveys are easier to do
• Surveys are faster to do
• Surveys are less biased
• Surveys are more objective
• Survey data is more valid
• Survey data is statistically
significant
Why are surveys better than user research?
THE TRUTH ABOUT SURVEYS?
• Survey methodology is
well-researched
• Best practices are well-
established
THE TRUTH ABOUT SURVEYS
@uxordie@uxordie
MIXED METHODS RESEARCH
6
• Full picture of user behavior
• Mitigate methodological weaknesses
• Understand the what and the why
• Make better decisions
aka Triangulation
@uxordie@uxordie
• Complex onboarding
• Complex customization
• Data-entry driven applications
• Multi-step data collection workflows
• Skills testing
WHAT ABOUT FORMS?
7
What kinds of forms benefit from testing?
• B2B eCommerce
• Healthcare
• Education
• Government
@uxordie
WHY TEST SURVEYS?
@uxordie@uxordie
WHY TEST SURVEYS?
9
ERROR TYPES
• Sampling
•Measurement
• Coverage
• Non-Response
“There are no stupid questions… or are there?”
@uxordie@uxordie
WHY TEST SURVEYS?
10
“Stop doing survey research.” 

- Jared Spool
https://articles.uie.com/on_surveys
DO SURVEY SUCK?
• Bias is incredibly easy to introduce.
• Bad questions can yield misleading
answers.
• You can’t adjust a survey mid-stream.
• Surveys are easy to manipulate.
• Surveys are not a substitute for qualitative
research.
• Surveys are not a shortcut.
• Surveys lack context.
• Surveys are subjective.
“How many researchers does it take to write a bad survey question?”
“Surveys are the most dangerous research
tool—misunderstood and misused. They
frequently straddle the qualita>ve and
quan>ta>ve, and at their worst represent
the worst of both..” 

- Erika Hall”
@uxordie
FOUR STAGE
QUESTIONNAIRE
MODEL
@uxordie@uxordie
1. Comprehension 

Do you understand the question?
2. Retrieval

Do you know or remember the answer?
3. Judgement

Can you formulate a truthful, accurate response?
4. Response

Can you map your answer to the response options
provided?
FOUR STAGE QUESTIONNAIRE MODEL
12
Roger Tourangeau
@uxordie@uxordie
“How many times have you participated in an online discussion about photography in the
last year?”
0 | Once or twice | 3-5 times | 6-10 times | more than 10 times
FOUR STAGES
13
COGNITIVE PROCESS
“Well, I’ve commented on things on Facebook, but I don’t think that’s what they mean. I
also took an online class, which definitely included discussion, so that
counts.” (Comprehension)
“It was still cold when I took the class, so that’s within the last year.” (Retrieval)
“There were 12 classes, but I know I missed a couple of them.” (Judgement)
“Overall, I was pretty active, so let’s just say that it’s more than 10.” (Response)
@uxordie
EXTREME(!) EXAMPLE
@uxordie@uxordie
1. COMPREHENSION
15
“How many times have you seen a provider in the last year?”
What’s a
provider?
Does it mean
a doctor?
If so, does this
include
chiropractors
…therapists…
pharmacists?
Is this
calendar
year or last
12 months?
@uxordie@uxordie
2. RETRIEVAL
16
“How many times have you seen a primary care doctor in the last 12 months?”
I had a checkup
a while ago, but
it may be more
than a year at
this point.
@uxordie@uxordie
3.JUDGEMENT
17
“How many times have you seen a primary care doctor in the last 6 months?”
I guess it’s 0. I
probably should
be taking better
care of my health
though.


I did talk to her on
the phone about
a refill, so I’ll just
say 1.
@uxordie@uxordie
4. RESPONSE
18
“How many times have you spoken to your primary care doctor in the last 6 months?”
Through email
On the phone
At the office
Wait a second… my
doctor has this new
telemedicine thingy, so
I did have a live chat
as well. Should I list that
as part of email?
1
0
0
@uxordie@uxordie
Are we done with this question yet?
19
“How many times have you spoken to your primary care doctor in the last 6 months?”
Through email
On the phone
At the office
Wait researcher, you also
forgot to randomize the
answer options, but you
won’t notice this until you
watch a few people take
the survey.
1
0
0
Other 0
Enter all that apply:
time(s)
time(s)
time(s)
time(s)
@uxordie@uxordie
COGNITIVE BURDEN
20
Are 30 clear questions better than 20 difficult ones?
Perhaps
Yes
Maybe
Don’t know
@uxordie
TESTING SURVEYS
@uxordie@uxordie
“The administration of draft survey questions
while collecting additional verbal information about the survey
responses,
which is used to evaluate the quality of the response
or to help determine whether the question is generating the
information that its author intends.”
COGNITIVE INTERVIEWING
22
Beatty & Willis (2007)
What is cognitive interviewing?
@uxordie@uxordie
Think-Aloud
+
Cognitive probes
=
Cognitive Interview
USABILITY TESTING SURVEYS
23
How is cognitive interviewing different from usability testing?
@uxordie@uxordie
1. Comprehension/Interpretation 

“What does the term ‘leaderboard’ mean to you?”
2. Paraphrasing

“Tell me in your own words what this is asking you.”
3. Recall

“How do you know that you used it 10 times in the past 3 months?”
4. Confidence Judgement

“How sure are you that you used it at least 10 times?”
5. Specific

“What information do you review when you use the leaderboard?”
6. General

“I noticed you changed your answer. Tell me what you were thinking.”
COGNITIVE PROBES
24
Gordon B. Willis
“How frequently do you use the leaderboard feature?”
@uxordie
COGNITIVE PROBES
@uxordie@uxordie
Q. “When was the last time you visited your mother to help her with her computer?”
COGNITIVE INTERVIEW
26
A. “I live with my mother actually, so I wouldn’t be visiting her.”
P. “OK, so do you remember the last time you helped her with her computer at
home?” (General)
A. “Yes… it was about a month ago.”
P. “How sure are you it was a month ago? (Confidence Judgement)
A. “I’m pretty sure, because it was the weekend of the big snow storm at the end of March.”
P. “Does anyone else help your mom with her computer?” (Specific)
A. “Not really. I don’t have any siblings, and my dad is just as bad at technology as she is.”
@uxordie@uxordie
Q. “How concerned are you about lead contamination in your community garden?”
COGNITIVE INTERVIEW
27
A. “… I don’t really know. This is the first time I’ve thought about it honestly. Now that you
brought it up though, I feel like I should be very concerned.”
P. “Are you familiar with the term ‘lead contamination’?” (Comprehension)
A. “I am… I know that lead paint was used in old buildings, so sometimes it can get in the
soil, but I’m not sure why this would affect a community garden if it’s not near a building.”
P. “You said that ‘you should be very concerned’… How sure are you that you’re very
concerned?” (Confidence Judgement)
A. I guess I just assumed that this wouldn’t be a problem in my community garden. So
maybe I’m not that concerned. Honestly, I don’t know.
@uxordie@uxordie
Q. “Did you go to the beach last weekend?”
COGNITIVE INTERVIEW
28
A. “I did actually. I went snorkeling.”
P. “Tell me about it.” (General)
A. “I was at a conference in Puerto Rico, and I got there a few days before.”
P. “Was this the weekend that just passed, or the weekend
before?” (Interpretation)
A. “It was Saturday, so two days ago.”
P. “Do you remember the last time you went to beach back home?” (Recall)
A. …
@uxordie@uxordie
Q. “How often do you use our website to book a dog sitter?”
COGNITIVE INTERVIEW
29
A. “OK… In the past, I’ve traveled a few times per year, but if it’s just a weekend my friend
would watch him, so maybe… 6 times per year… so that’s. ‘once every 2-3 months’…
P. “I noticed you hesitated before choosing your answer….” (General)
A. “Yeah… I was trying to figure out what it would be on average.”
P. “So… how sure are you that it’s ‘once every 2-3 months’?” (Confidence Judgement)
A. “Well, to be honest, my dog died a few months ago… so I haven’t used it since then, 

obviously.”
Once a month or more / Once every 2-3 months / Every six months / Once a year / Less than
once a year
@uxordie
SURVEY PRE-TESTING
@uxordie@uxordie
• Exploratory interview
incorporating cognitive
probes.
MODERATED INTERVIEW
31
HOW USE IT TO EXPLORE
Comprehension | Retrieval |
Judgement

• Asking the right questions
• Understanding respondent
background knowledge
• Exposing assumptions
• Setting the right response
intervals
PRO TIP
• Doubles as a round of
qualitative research.
@uxordie@uxordie
• Observation and cognitive
interview of draft survey.
MODERATED DRY RUN
32
HOW USE IT TO EXPLORE
Comprehension | Retrieval | Judgement
| Response

• Terminology and interpretation
issues
• Fixing confusing response
categories
• Layout issues
PRO TIP
• Probe on response categories
as well as the questions.
@uxordie@uxordie
• Self-administration of survey
to hand-picked, sample
before official launch.
PILOT
33
HOW USE IT TO EXPLORE
Comprehension | Response
• Some terminology issues
• Missing response categories
• Layout issues
PRO TIP
• Include open-ended
responses for reporting
problems in each survey
section.
@uxordie@uxordie
ANY QUESTIONS?
34

More Related Content

Similar to Usability Testing Online Surveys (and forms)

Similar to Usability Testing Online Surveys (and forms) (19)

How to grow your perfect audience online fast with this little strategy ana...
How to grow your perfect audience online fast with this little strategy   ana...How to grow your perfect audience online fast with this little strategy   ana...
How to grow your perfect audience online fast with this little strategy ana...
 
How To Write A Research Paper Quickly. How To Write A Researc
How To Write A Research Paper Quickly. How To Write A ResearcHow To Write A Research Paper Quickly. How To Write A Researc
How To Write A Research Paper Quickly. How To Write A Researc
 
Why it's time for a new kind of quantitative research
Why it's time for a new kind of quantitative researchWhy it's time for a new kind of quantitative research
Why it's time for a new kind of quantitative research
 
Keep it Simple Stupid - Simplicity gets you the best results
Keep it Simple Stupid - Simplicity gets you the best resultsKeep it Simple Stupid - Simplicity gets you the best results
Keep it Simple Stupid - Simplicity gets you the best results
 
“High Life”: Helping High Risk Urban Teens Create a Weed Prevention Game App
“High Life”: Helping High Risk Urban Teens Create a Weed Prevention Game App“High Life”: Helping High Risk Urban Teens Create a Weed Prevention Game App
“High Life”: Helping High Risk Urban Teens Create a Weed Prevention Game App
 
QualitativeLecture
QualitativeLectureQualitativeLecture
QualitativeLecture
 
2. proposal(1)
2. proposal(1)2. proposal(1)
2. proposal(1)
 
Can PowerPoint be Saved
Can PowerPoint be SavedCan PowerPoint be Saved
Can PowerPoint be Saved
 
Designing Online Email Based Storytelling Group for Older Adults
Designing Online Email Based Storytelling Group for Older AdultsDesigning Online Email Based Storytelling Group for Older Adults
Designing Online Email Based Storytelling Group for Older Adults
 
Cs faculty newsletter jan 20
Cs faculty newsletter jan 20Cs faculty newsletter jan 20
Cs faculty newsletter jan 20
 
Great Voice Experiences Start with Listening: Best Practices in Research and ...
Great Voice Experiences Start with Listening: Best Practices in Research and ...Great Voice Experiences Start with Listening: Best Practices in Research and ...
Great Voice Experiences Start with Listening: Best Practices in Research and ...
 
Critical Lens Essay - Critical Lens Quote
Critical Lens Essay - Critical Lens QuoteCritical Lens Essay - Critical Lens Quote
Critical Lens Essay - Critical Lens Quote
 
survey after kamishibai and results.pptx
survey after kamishibai and results.pptxsurvey after kamishibai and results.pptx
survey after kamishibai and results.pptx
 
Quizzes secret weapon - 5 march2015
Quizzes   secret weapon - 5 march2015Quizzes   secret weapon - 5 march2015
Quizzes secret weapon - 5 march2015
 
SE 3 Questions
SE 3 QuestionsSE 3 Questions
SE 3 Questions
 
A Story Rich World - UPA NYC - Sept 14
A Story Rich World - UPA NYC - Sept 14A Story Rich World - UPA NYC - Sept 14
A Story Rich World - UPA NYC - Sept 14
 
Development pro forma(3) (1)
Development pro forma(3) (1)Development pro forma(3) (1)
Development pro forma(3) (1)
 
How to Design a Survey that Responders Want to Answer
How to Design a Survey that Responders Want to AnswerHow to Design a Survey that Responders Want to Answer
How to Design a Survey that Responders Want to Answer
 
QA Analysis.pptx
QA Analysis.pptxQA Analysis.pptx
QA Analysis.pptx
 

More from Eva Kaniasty

Strategic UX Triangle
Strategic UX TriangleStrategic UX Triangle
Strategic UX Triangle
Eva Kaniasty
 
Mobile App Design for Behavior Change
Mobile App Design for Behavior ChangeMobile App Design for Behavior Change
Mobile App Design for Behavior Change
Eva Kaniasty
 
How to be a Usability Star
How to be a Usability StarHow to be a Usability Star
How to be a Usability Star
Eva Kaniasty
 

More from Eva Kaniasty (14)

UX For Evil (and for good)
UX For Evil (and for good)UX For Evil (and for good)
UX For Evil (and for good)
 
Fast and Unbiased Usability Research
Fast and Unbiased Usability ResearchFast and Unbiased Usability Research
Fast and Unbiased Usability Research
 
Visualizing User Feedback
Visualizing User FeedbackVisualizing User Feedback
Visualizing User Feedback
 
Putting Lipstick on UX Feedback
Putting Lipstick on UX FeedbackPutting Lipstick on UX Feedback
Putting Lipstick on UX Feedback
 
UX and the Product Manager
UX and the Product ManagerUX and the Product Manager
UX and the Product Manager
 
Bathroom Microinteractions
Bathroom MicrointeractionsBathroom Microinteractions
Bathroom Microinteractions
 
Don't Fear the Research!
Don't Fear the Research!Don't Fear the Research!
Don't Fear the Research!
 
Strategic UX Triangle
Strategic UX TriangleStrategic UX Triangle
Strategic UX Triangle
 
Zen of Consulting
Zen of ConsultingZen of Consulting
Zen of Consulting
 
Mobile App Design for Behavior Change
Mobile App Design for Behavior ChangeMobile App Design for Behavior Change
Mobile App Design for Behavior Change
 
How to be a Usability Star
How to be a Usability StarHow to be a Usability Star
How to be a Usability Star
 
Web Analytics and Usability Testing
Web Analytics and Usability TestingWeb Analytics and Usability Testing
Web Analytics and Usability Testing
 
Usability Elevator Pitch
Usability Elevator PitchUsability Elevator Pitch
Usability Elevator Pitch
 
Innovative Interfaces: Transforming Data Into Insight
Innovative Interfaces:  Transforming Data Into InsightInnovative Interfaces:  Transforming Data Into Insight
Innovative Interfaces: Transforming Data Into Insight
 

Recently uploaded

如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
Design-System - FinTech - Isadora Agency
Design-System - FinTech - Isadora AgencyDesign-System - FinTech - Isadora Agency
Design-System - FinTech - Isadora Agency
Isadora Agency
 
挂科办理天主教大学毕业证成绩单一模一样品质
挂科办理天主教大学毕业证成绩单一模一样品质挂科办理天主教大学毕业证成绩单一模一样品质
挂科办理天主教大学毕业证成绩单一模一样品质
yzeoq
 
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
Amil baba
 
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
ugzga
 
Evaluating natural frequencies and mode shapes.pptx
Evaluating natural frequencies and mode shapes.pptxEvaluating natural frequencies and mode shapes.pptx
Evaluating natural frequencies and mode shapes.pptx
joshuaclack73
 

Recently uploaded (20)

如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UMN毕业证书)明尼苏达大学毕业证成绩单本科硕士学位证留信学历认证
 
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Bath毕业证书)巴斯大学毕业证成绩单本科硕士学位证留信学历认证
 
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(ArtEZ毕业证书)ArtEZ艺术学院毕业证成绩单本科硕士学位证留信学历认证
 
Design-System - FinTech - Isadora Agency
Design-System - FinTech - Isadora AgencyDesign-System - FinTech - Isadora Agency
Design-System - FinTech - Isadora Agency
 
挂科办理天主教大学毕业证成绩单一模一样品质
挂科办理天主教大学毕业证成绩单一模一样品质挂科办理天主教大学毕业证成绩单一模一样品质
挂科办理天主教大学毕业证成绩单一模一样品质
 
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
NO1 Best Vashikaran Specialist in Uk Black Magic Specialist in Uk Black Magic...
 
And that's about to change! (Service Design Drinks Berlin May 2024)
And that's about to change! (Service Design Drinks Berlin May 2024)And that's about to change! (Service Design Drinks Berlin May 2024)
And that's about to change! (Service Design Drinks Berlin May 2024)
 
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
如何办理(Birmingham毕业证书)伯明翰大学学院毕业证成绩单本科硕士学位证留信学历认证
 
GBU INDOOR STADIUM CASE STUDY DESCRIBING ITS STRUCTURE
GBU INDOOR STADIUM CASE STUDY DESCRIBING ITS STRUCTUREGBU INDOOR STADIUM CASE STUDY DESCRIBING ITS STRUCTURE
GBU INDOOR STADIUM CASE STUDY DESCRIBING ITS STRUCTURE
 
Branding in the Psychedelic Landscape Report.pdf
Branding in the Psychedelic Landscape Report.pdfBranding in the Psychedelic Landscape Report.pdf
Branding in the Psychedelic Landscape Report.pdf
 
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(UAL毕业证书)伦敦艺术大学毕业证成绩单本科硕士学位证留信学历认证
 
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(Columbia College毕业证书)纽约市哥伦比亚大学毕业证成绩单本科硕士学位证留信学历认证
 
Levi's Advertisement and camapign design
Levi's Advertisement and camapign designLevi's Advertisement and camapign design
Levi's Advertisement and camapign design
 
Top 10 Website Designing Hacks for Beginners.pptx.pptx
Top 10 Website Designing Hacks for Beginners.pptx.pptxTop 10 Website Designing Hacks for Beginners.pptx.pptx
Top 10 Website Designing Hacks for Beginners.pptx.pptx
 
Spring Summer 2026 Inspirations trend book Peclers Paris
Spring Summer 2026 Inspirations trend book Peclers ParisSpring Summer 2026 Inspirations trend book Peclers Paris
Spring Summer 2026 Inspirations trend book Peclers Paris
 
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
如何办理(UCI毕业证书)加利福尼亚大学尔湾分校毕业证成绩单本科硕士学位证留信学历认证
 
CADD 141 - BIRD Scooter - Cup Holder Photos.pdf
CADD 141 - BIRD Scooter - Cup Holder Photos.pdfCADD 141 - BIRD Scooter - Cup Holder Photos.pdf
CADD 141 - BIRD Scooter - Cup Holder Photos.pdf
 
Webhost NVME Cloud VPS Hosting1234455678
Webhost NVME Cloud VPS Hosting1234455678Webhost NVME Cloud VPS Hosting1234455678
Webhost NVME Cloud VPS Hosting1234455678
 
Evaluating natural frequencies and mode shapes.pptx
Evaluating natural frequencies and mode shapes.pptxEvaluating natural frequencies and mode shapes.pptx
Evaluating natural frequencies and mode shapes.pptx
 
Naer VR: Advanced Research and Usability Testing Project
Naer VR: Advanced Research and Usability Testing ProjectNaer VR: Advanced Research and Usability Testing Project
Naer VR: Advanced Research and Usability Testing Project
 

Usability Testing Online Surveys (and forms)

  • 1. @uxordie Text fields and dropdowns and radios oh my! USABILITY TESTING ONLINE SURVEYS (AND FORMS) Eva Kaniasty UXPA International Conference @uxordie
  • 2. @uxordie@uxordie 1. Why surveys? 
 Isn’t user research enough? 2. Why pre-test surveys?
 Let me count the ways. 3. Four stage questionnaire model
 The Cognitive Psych (Tourangeau) 4. The cognitive interview
 Usability testing… just for surveys. (Willis) 5. Survey Pre-Testing Process
 3 scenarios. AGENDA? AGENDA. 2
  • 4. @uxordie@uxordie • Validate qualitative research results • Get feedback from a larger sample • Build a participant database • Bridge gap between quant and qual • Build a strong case for change • Kill features WHY SURVEYS? 4 “Doesn’t everybody love a survey?” • Execs • Business People • Marketing People • Sales People
  • 5. @uxordie@uxordie WHY SURVEYS? 5 • Surveys are easier to do • Surveys are faster to do • Surveys are less biased • Surveys are more objective • Survey data is more valid • Survey data is statistically significant Why are surveys better than user research? THE TRUTH ABOUT SURVEYS? • Survey methodology is well-researched • Best practices are well- established THE TRUTH ABOUT SURVEYS
  • 6. @uxordie@uxordie MIXED METHODS RESEARCH 6 • Full picture of user behavior • Mitigate methodological weaknesses • Understand the what and the why • Make better decisions aka Triangulation
  • 7. @uxordie@uxordie • Complex onboarding • Complex customization • Data-entry driven applications • Multi-step data collection workflows • Skills testing WHAT ABOUT FORMS? 7 What kinds of forms benefit from testing? • B2B eCommerce • Healthcare • Education • Government
  • 9. @uxordie@uxordie WHY TEST SURVEYS? 9 ERROR TYPES • Sampling •Measurement • Coverage • Non-Response “There are no stupid questions… or are there?”
  • 10. @uxordie@uxordie WHY TEST SURVEYS? 10 “Stop doing survey research.” 
 - Jared Spool https://articles.uie.com/on_surveys DO SURVEY SUCK? • Bias is incredibly easy to introduce. • Bad questions can yield misleading answers. • You can’t adjust a survey mid-stream. • Surveys are easy to manipulate. • Surveys are not a substitute for qualitative research. • Surveys are not a shortcut. • Surveys lack context. • Surveys are subjective. “How many researchers does it take to write a bad survey question?” “Surveys are the most dangerous research tool—misunderstood and misused. They frequently straddle the qualita>ve and quan>ta>ve, and at their worst represent the worst of both..” 
 - Erika Hall”
  • 12. @uxordie@uxordie 1. Comprehension 
 Do you understand the question? 2. Retrieval
 Do you know or remember the answer? 3. Judgement
 Can you formulate a truthful, accurate response? 4. Response
 Can you map your answer to the response options provided? FOUR STAGE QUESTIONNAIRE MODEL 12 Roger Tourangeau
  • 13. @uxordie@uxordie “How many times have you participated in an online discussion about photography in the last year?” 0 | Once or twice | 3-5 times | 6-10 times | more than 10 times FOUR STAGES 13 COGNITIVE PROCESS “Well, I’ve commented on things on Facebook, but I don’t think that’s what they mean. I also took an online class, which definitely included discussion, so that counts.” (Comprehension) “It was still cold when I took the class, so that’s within the last year.” (Retrieval) “There were 12 classes, but I know I missed a couple of them.” (Judgement) “Overall, I was pretty active, so let’s just say that it’s more than 10.” (Response)
  • 15. @uxordie@uxordie 1. COMPREHENSION 15 “How many times have you seen a provider in the last year?” What’s a provider? Does it mean a doctor? If so, does this include chiropractors …therapists… pharmacists? Is this calendar year or last 12 months?
  • 16. @uxordie@uxordie 2. RETRIEVAL 16 “How many times have you seen a primary care doctor in the last 12 months?” I had a checkup a while ago, but it may be more than a year at this point.
  • 17. @uxordie@uxordie 3.JUDGEMENT 17 “How many times have you seen a primary care doctor in the last 6 months?” I guess it’s 0. I probably should be taking better care of my health though. 
 I did talk to her on the phone about a refill, so I’ll just say 1.
  • 18. @uxordie@uxordie 4. RESPONSE 18 “How many times have you spoken to your primary care doctor in the last 6 months?” Through email On the phone At the office Wait a second… my doctor has this new telemedicine thingy, so I did have a live chat as well. Should I list that as part of email? 1 0 0
  • 19. @uxordie@uxordie Are we done with this question yet? 19 “How many times have you spoken to your primary care doctor in the last 6 months?” Through email On the phone At the office Wait researcher, you also forgot to randomize the answer options, but you won’t notice this until you watch a few people take the survey. 1 0 0 Other 0 Enter all that apply: time(s) time(s) time(s) time(s)
  • 20. @uxordie@uxordie COGNITIVE BURDEN 20 Are 30 clear questions better than 20 difficult ones? Perhaps Yes Maybe Don’t know
  • 22. @uxordie@uxordie “The administration of draft survey questions while collecting additional verbal information about the survey responses, which is used to evaluate the quality of the response or to help determine whether the question is generating the information that its author intends.” COGNITIVE INTERVIEWING 22 Beatty & Willis (2007) What is cognitive interviewing?
  • 23. @uxordie@uxordie Think-Aloud + Cognitive probes = Cognitive Interview USABILITY TESTING SURVEYS 23 How is cognitive interviewing different from usability testing?
  • 24. @uxordie@uxordie 1. Comprehension/Interpretation 
 “What does the term ‘leaderboard’ mean to you?” 2. Paraphrasing
 “Tell me in your own words what this is asking you.” 3. Recall
 “How do you know that you used it 10 times in the past 3 months?” 4. Confidence Judgement
 “How sure are you that you used it at least 10 times?” 5. Specific
 “What information do you review when you use the leaderboard?” 6. General
 “I noticed you changed your answer. Tell me what you were thinking.” COGNITIVE PROBES 24 Gordon B. Willis “How frequently do you use the leaderboard feature?”
  • 26. @uxordie@uxordie Q. “When was the last time you visited your mother to help her with her computer?” COGNITIVE INTERVIEW 26 A. “I live with my mother actually, so I wouldn’t be visiting her.” P. “OK, so do you remember the last time you helped her with her computer at home?” (General) A. “Yes… it was about a month ago.” P. “How sure are you it was a month ago? (Confidence Judgement) A. “I’m pretty sure, because it was the weekend of the big snow storm at the end of March.” P. “Does anyone else help your mom with her computer?” (Specific) A. “Not really. I don’t have any siblings, and my dad is just as bad at technology as she is.”
  • 27. @uxordie@uxordie Q. “How concerned are you about lead contamination in your community garden?” COGNITIVE INTERVIEW 27 A. “… I don’t really know. This is the first time I’ve thought about it honestly. Now that you brought it up though, I feel like I should be very concerned.” P. “Are you familiar with the term ‘lead contamination’?” (Comprehension) A. “I am… I know that lead paint was used in old buildings, so sometimes it can get in the soil, but I’m not sure why this would affect a community garden if it’s not near a building.” P. “You said that ‘you should be very concerned’… How sure are you that you’re very concerned?” (Confidence Judgement) A. I guess I just assumed that this wouldn’t be a problem in my community garden. So maybe I’m not that concerned. Honestly, I don’t know.
  • 28. @uxordie@uxordie Q. “Did you go to the beach last weekend?” COGNITIVE INTERVIEW 28 A. “I did actually. I went snorkeling.” P. “Tell me about it.” (General) A. “I was at a conference in Puerto Rico, and I got there a few days before.” P. “Was this the weekend that just passed, or the weekend before?” (Interpretation) A. “It was Saturday, so two days ago.” P. “Do you remember the last time you went to beach back home?” (Recall) A. …
  • 29. @uxordie@uxordie Q. “How often do you use our website to book a dog sitter?” COGNITIVE INTERVIEW 29 A. “OK… In the past, I’ve traveled a few times per year, but if it’s just a weekend my friend would watch him, so maybe… 6 times per year… so that’s. ‘once every 2-3 months’… P. “I noticed you hesitated before choosing your answer….” (General) A. “Yeah… I was trying to figure out what it would be on average.” P. “So… how sure are you that it’s ‘once every 2-3 months’?” (Confidence Judgement) A. “Well, to be honest, my dog died a few months ago… so I haven’t used it since then, 
 obviously.” Once a month or more / Once every 2-3 months / Every six months / Once a year / Less than once a year
  • 31. @uxordie@uxordie • Exploratory interview incorporating cognitive probes. MODERATED INTERVIEW 31 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement
 • Asking the right questions • Understanding respondent background knowledge • Exposing assumptions • Setting the right response intervals PRO TIP • Doubles as a round of qualitative research.
  • 32. @uxordie@uxordie • Observation and cognitive interview of draft survey. MODERATED DRY RUN 32 HOW USE IT TO EXPLORE Comprehension | Retrieval | Judgement | Response
 • Terminology and interpretation issues • Fixing confusing response categories • Layout issues PRO TIP • Probe on response categories as well as the questions.
  • 33. @uxordie@uxordie • Self-administration of survey to hand-picked, sample before official launch. PILOT 33 HOW USE IT TO EXPLORE Comprehension | Response • Some terminology issues • Missing response categories • Layout issues PRO TIP • Include open-ended responses for reporting problems in each survey section.