1.
When Qual Met UX:
How to Combine Traditional Qualitative
Methods and User Research Approaches
to Satisfy Your Clients and Add Value to
Your Findings
Kinsey Gimbel & Jon Strohl
2018 QRCA Annual Conference

2.
2
• Focus Groups
• Driven by Marketing Department
• Conducted by Market Researchers
• Terminology often centers around
“participants,” “themes,” and “discussions.”
• Usability Testing
• Driven by Engineering or Product Development
• Conducted by UX Researchers
• Terminology often centers around “users,”
“tasks,” “success/failure metrics,” and
“severity ratings.”
How It Can Appear
Qualitative
Research
User Experience
Research

3.
3
How We See It

4.
4
Kinsey Jon
Image from: https://bazeuniversity.edu.ng/main/faculty-of-management-and-social-sciences/public-administration.php
Image from: http://historytothepublic.org/what-is-oral-history/
Introductions

5.
5
Bringing Different Kinds of Research Together!

6.
6
1. What is user experience research?
2. How do you implement a cross-cutting Qual/UX research program?
3. What can you take away to start weaving UX into your work?
4. In summary...
Our Road Map for Today

7.
What is User Experience
Research?

8.
8
References: https://www.invisionapp.com/blog/a-brief-history-of-user-experience/
http://www.uxbooth.com/articles/where-ux-comes-from/
The History of UX

9.
9 Image from: https://www.interaction-design.org/literature/article/the-7-factors-that-influence-user-experience
Usability and Much More...

10.
10
Qualitative Quantitative
BehavioralAttitudinal
UX Research Spectrum

11.
11
User Interviews
Focus Groups
Observation
Diary Writing
Surveys
Personas
Website Analytics
Competitive Analysis
Stakeholder Interviews
Journey Mapping
Heuristic Evaluation
UX Testing
Benchmarking
Card Sorting
Wire-Framing
Prototyping
Eye Tracking
Tree Testing
Task Analysis
Surveys
Experiments
A/B Testing
Website Analytics
Benchmarking
Discover
Understand Users and
Environment
Design and Develop
Early and Late
Stage Testing
Evaluate
Monitor
Performance
PHASE 1 PHASE 2 PHASE 3
UX Research Phases

12.
12 Image from: https://webdesign.tutsplus.com/articles/a-designers-introduction-to-agile-methodology--cms-23349
Product Development Process

13.
UX Methods

14.
14
A method for detecting usability issues in an interface in which a small set of
evaluators examine the interface and judge its compliance with usability best
practices and principles (heuristics).
Visibility of system status
Match between system and
the real world
User control and freedom
Consistency and standards
Error prevention
Affordance of the system
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and
recover from errors
Help and documentation
Minimize the cognitive load/burden
Heuristic Evaluations
*Learn more about heuristic evaluations at https://www.nngroup.com/articles/ten-usability-heuristics/

15.
15
A Typical UX Test Session

16.
16
UX Testing Logistics

17.
17
Screen-based eye tracker
Eye-tracking glasses
Eye-tracking VR integration
Mobile device testing
UX Testing Tools – Eye Tracking

18.
18
Metrics Tells us…
Performance
+ First click accuracy
+ Task accuracy
+ Time to complete tasks
+ Click patterns
+ Conversion rate
+ How easily users can get started
+ How easily users complete tasks
+ The actions users take during use
+ How efficiently the product can be navigated
+ Likelihood of converting to desired outcome
Self-Report
+ Difficulty ratings
+ Satisfaction ratings
+ Think-aloud protocol
+ Debriefing interview
+ Perceived usability of the product
+ How satisfying users find the experience
+ What users are thinking during use
+ Users’ feedback on completing tasks
Implicit
+ Eye tracking
+ Behavioral analysis
+ Verbalization analysis
+ Pupil dilation
+ Visibility of design elements
+ How attention is allocated during use
+ Behavioral and verbalization patterns
+ Mental workload while completing tasks
UX Metrics

19.
19
Card Sorting

20.
20 Image from: http://www.steptwo.com.au/wp-content/uploads/KMC_TreeTesting_Online.jpg
Tree Testing

21.
21
Focus Groups
+ Past participant experiences,
beliefs, opinions
+ In-group activities
UX Test
+ Difficulty and satisfaction ratings
+Task accuracy and clicks
+ Think-aloud comments
+ Debriefing interview
+ SUS, NPS
Heuristic
Evaluation
+ Expert reviews
+ Severity ratings
UX as a Mixed-Methods Approach

22.
Integrating Qual and UX

23.
23
• Challenge: Improve a federal
agency’s website designed for job
seekers and provide strategic
direction for future development.
• Method: Conducted focus groups
with multiple audience segments
across the United States.
• Result: Insights about what job
seekers think about applying for
federal employment and their
past experiences with the
process.
Focus Groups for a Career Building and
Development Website

24.
24
• Difficult to receive useful feedback on
how participants would behave
• Lacked depth on feedback of interface
Focus Groups for a Career Building and
Development Website: Method Limitations

25.
25
Privacy
Security
• Challenge: Understand the perception of
a social media organization’s security,
and identify ways the organization could
improve it’s privacy functions.
• Method: Conducted focus groups with
multiple audience segments across the
United States to learn how people
comprehend privacy and security in the
online space.
• Result: Insights about how consumers
conceptualize security and privacy.
Focus Groups on Social Media
Privacy and Security

26.
26
• Difficult to get interactive and truly
responsive feedback.
• Types of tasks that can be provided
to a group are more limited.
Focus Groups on Social Media
Privacy and Security: Method Limitations

27.
27
• Challenge: Provide guidance on the optimal design of a student loan
form.
• Phase 1: Focus groups on consumers’ experiences with student
loans. IDIs on students’ own paperwork.
• Phase 2: Multiple
rounds of testing with
draft forms.
• Result: An optimized
form that was ready for
dissemination.
Mixed-Methods Multiphase Form Design

28.
28
Mixed-Methods Multiphase Interactive Voice
Response (IVR) System Design
• Challenge: Redesign an agency’s IVR
system’s structure and scripts.
• Phase 1: Focus groups to assess how
people feel about calling the line, their
past experiences, and initial
impressions.
• Phase 2: Iterative rounds of UX testing
with revised selection options.
• Result: Recommendations and scripts
for development and implementation.
Image from: http://money.cnn.com/gallery/smallbusiness/2013/12/19/worst-customer-service/index.html

29.
What Can You Take Away
to Start Integrating UX Into
Your Work?

30.
30
Adding a UX Twist to What You Already Do Well
1. Consider using different methods across disciplines to
answer your client’s research questions—both the ones they
ask and the ones they don’t!
2. Consider including specialized tasks and exercises in your
data collection.
3. Have participants “think aloud” when completing a task,
and follow up later with discussion questions and
standardized scales/survey questions.
4. Report observations of participants’ behavior/metrics in
addition to what they say.
5. Ensure recommendations will create a positive user
experience!