1. @kaniasty | #uxpabos17
You can have it both ways
Fast and Unbiased Usability Research
Eva Kaniasty, Red Pill UX

2. @kaniasty | #uxpabos17
2
What do we want? Everything.
When do we want it? Now

3. @kaniasty | #uxpabos17
3
What is good research?
“You can’t always get what
you want.”

4. @kaniasty | #uxpabos17
4
Bias is everywhere!
• Inter-evaluator bias
• Loaded or leading questions
• Cognitive biases (e.g. primacy, recency)
• Group dynamics (i.e. groupthink)
• Prejudgements
• Selection bias
• Respondent bias (e.g. positive, social)
“Soylent Green…
Bias…
it’s made of
people!”

5. @kaniasty | #uxpabos17
5
• Plan
• Structure
• Focus
• Automate
• Don’t rely on memory
Reduce Your Cognitive Load
“An ounce of prevention is worth a
pound of cure.” - Ben Franklin

6. @kaniasty | #uxpabos17
6
“Efficiency is intelligent
laziness.”
Objectivity through Optimization

7. @kaniasty | #uxpabos17
Good + Efficient
Participant Recruiting

8. @kaniasty | #uxpabos17
8
Recruiting
Recruiting Brief
Screening Survey
Scheduling
- Research Recruiter
- UserInterviews.com
- Stakeholders
Participant Data
- Contact Info
- Time Zones
- Screening Data
“Keep it simple…”

9. @kaniasty | #uxpabos17
9
Customer Super Screener (i.e. survey)
Better screening
- Larger pool of potential
participants
- Foot in the door
- Comparison sample
- Bias fighter
Measure twice, cut once
Collect data as you screen
- Demographics
- Feature usage
- Card sorts
- Desirability (product reaction)
- Technical
- Quotes (collected judiciously)
- Opt-in for future research
(online & moderated)*Don’t forget legal review

10. @kaniasty | #uxpabos17
10
Recruiting Logistics
Written Consent
•Context
•Logistics
•Recording
•Compensation
•(Optional) Nondisclosure
*Don’t forget legal review
“80 percent of success is just showing up.”
Reducing No-Shows
•Written consent
•Email confirmation
•Remote tool testing/setup
•Phone/Texting
Session Status Tracking
Compensation

11. @kaniasty | #uxpabos17
Good + Efficient
Study Design

12. @kaniasty | #uxpabos17
12
“Get your priorities straight.”

13. @kaniasty | #uxpabos17
13
Moderator Guide
Pre-Test Interview
- Background / Profile Fit
- Past experience/usage
- Top 3 tasks/needs
First Impressions
- (if website) HP overview
- (if navigation) Navigation
expectations
- (if new user) Free exploration
- (if current user) Recent task
Tasks or Process
- Probing questions
- Ease of use ratings
- Task Success
Wrap-Up
- Salient Likes/Dislikes
- “Change one thing”
- System Usability Scale/Global
Measures
“How the sausage gets made.”

14. @kaniasty | #uxpabos17
Good & Efficient
Data Collection

15. @kaniasty | #uxpabos17
15
Session Housekeeping
Participant Briefing
- Context-setting
- Recording consent
- Think-aloud instruction
- Honest Feedback
- Neutral Observer
Logistics
- Apps / Links
- Notes templates
- Equipment
- Logins
- Recording
- Reset
“Don’t forget to eat.” - Mom

16. @kaniasty | #uxpabos17
16
Recording & Photography
• Personal room
• Test meeting OR
Tech call
• Mac & Windows
compatible
Mobile Screen
Mobile Device
• Sleds

17. @kaniasty | #uxpabos17
17
Super Notes!
Mod Guide + Notes + Audio
PearNote Mac
OneNote Windows
Livescribe

18. @kaniasty | #uxpabos17
18
The Session Summary
This participant had major issues with X and Y features
because she expected [X] and [Y] to happen. She followed an
interesting workflow that was different from what we
expected. She also did [this interesting or surprising thing].
She was hesitant to explore the UI and fit the ‘Ms. X’ persona
pretty well…
“Take 5.”

19. @kaniasty | #uxpabos17
19
Avoiding the Slow TV Syndrome
• Re-watching videos
• Relying on notes alone
• Human transcription
• Automatic transcription
VLC player

20. @kaniasty | #uxpabos17
20
Task Data

21. @kaniasty | #uxpabos17
21
Usability Measures
Custom Tools
Question Banks

22. @kaniasty | #uxpabos17
22
Question Banks
Product Reaction CardsSemantic Differentials
Ease of Use

23. @kaniasty | #uxpabos17
23
Spreadsheets

24. @kaniasty | #uxpabos17
Good & Efficient
Reporting

25. @kaniasty | #uxpabos17
25
Analysis & Reporting Strategy
Findings Report
- Narrative
- Includes detailed study
methodology & background
- Includes raw data
- Includes participant detail
- Includes session summaries
- Focused on detailed findings
Strategic Brief Presentation
- Focused on the big picture
- Focused on priorities
- Focused on critical findings
- Includes recommendations
- Visually impactful
- Includes quotes
- Synthesizes data from multiple
sources
- Summary deliverables:
(e.g. profiles, personas, user journeys)
“Use the right tool
for the job.”

26. @kaniasty | #uxpabos17
26
NOT Survey Reports
No Custom Color Coding
Always reports %

27. @kaniasty | #uxpabos17
Participant Data
27
♀ 5 Female ♂ 5 Male
4 on Laptops 4 on Smartphones
(4 iPhone, 3 Android, 1 Windows Phone)
Customers
Company 1, Company 2,
Company 3
Customers RepresentedProfile 1 | Profile 2
Session Summaries
as part of detailed
findings / or with
recordings

28. @kaniasty | #uxpabos17
28
Task Data

29. @kaniasty | #uxpabos17
29
Structured Data

30. @kaniasty | #uxpabos17
30
Usability Measures
Tullis & Albert

31. @kaniasty | #uxpabos17
31
Data Analysis

32. @kaniasty | #uxpabos17
32
Infogr.am
PROS:
Range of Graphs
Fun viz options
Full control over colors
Option to show values (not %)
CONS:
Limited font control
No option to show confidence intervals

33. @kaniasty | #uxpabos17
33
plot.ly

34. @kaniasty | #uxpabos17
34
“Last but not least… qualitative data analysis.”

35. @kaniasty | #uxpabos17
35
Final Presentation
Strategic Brief
- Focused on the big picture
- Focused on priorities
- Focused on critical findings
- Includes recommendations
- Visually impactful
- Includes quotes
- Synthesizes data from multiple
sources
- Summary deliverables:
(e.g. profiles, personas, user journeys)
“Looks matter”

36. @kaniasty | #uxpabos17
Questions?