Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
@kaniasty | #uxpabos17
You can have it both ways
Fast and Unbiased Usability Research
Eva Kaniasty, Red Pill UX
@kaniasty | #uxpabos17
2
What do we want? Everything.
When do we want it? Now
@kaniasty | #uxpabos17
3
What is good research?
“You can’t always get what
you want.”
@kaniasty | #uxpabos17
4
Bias is everywhere!
• Inter-evaluator bias
• Loaded or leading questions
• Cognitive biases (e.g....
@kaniasty | #uxpabos17
5
• Plan
• Structure
• Focus
• Automate
• Don’t rely on memory
Reduce Your Cognitive Load
“An ounce...
@kaniasty | #uxpabos17
6
“Efficiency is intelligent
laziness.”
Objectivity through Optimization
@kaniasty | #uxpabos17
Good + Efficient
Participant Recruiting
@kaniasty | #uxpabos17
8
Recruiting
Recruiting Brief
Screening Survey
Scheduling
- Research Recruiter
- UserInterviews.com...
@kaniasty | #uxpabos17
9
Customer Super Screener (i.e. survey)
Better screening
- Larger pool of potential
participants
- ...
@kaniasty | #uxpabos17
10
Recruiting Logistics
Written Consent
•Context
•Logistics
•Recording
•Compensation
•(Optional) No...
@kaniasty | #uxpabos17
Good + Efficient
Study Design
@kaniasty | #uxpabos17
12
“Get your priorities straight.”
@kaniasty | #uxpabos17
13
Moderator Guide
Pre-Test Interview
- Background / Profile Fit
- Past experience/usage
- Top 3 tas...
@kaniasty | #uxpabos17
Good & Efficient
Data Collection
@kaniasty | #uxpabos17
15
Session Housekeeping
Participant Briefing
- Context-setting
- Recording consent
- Think-aloud ins...
@kaniasty | #uxpabos17
16
Recording & Photography
• Personal room
• Test meeting OR
Tech call
• Mac & Windows
compatible
M...
@kaniasty | #uxpabos17
17
Super Notes!
Mod Guide + Notes + Audio
PearNote Mac
OneNote Windows
Livescribe
@kaniasty | #uxpabos17
18
The Session Summary
This participant had major issues with X and Y features
because she expected...
@kaniasty | #uxpabos17
19
Avoiding the Slow TV Syndrome
• Re-watching videos
• Relying on notes alone
• Human transcriptio...
@kaniasty | #uxpabos17
20
Task Data
@kaniasty | #uxpabos17
21
Usability Measures
Custom Tools
Question Banks
@kaniasty | #uxpabos17
22
Question Banks
Product Reaction CardsSemantic Differentials
Ease of Use
@kaniasty | #uxpabos17
23
Spreadsheets
@kaniasty | #uxpabos17
Good & Efficient
Reporting
@kaniasty | #uxpabos17
25
Analysis & Reporting Strategy
Findings Report
- Narrative
- Includes detailed study
methodology ...
@kaniasty | #uxpabos17
26
NOT Survey Reports
No Custom Color Coding
Always reports %
@kaniasty | #uxpabos17
Participant Data
27
♀ 5 Female ♂ 5 Male
4 on Laptops 4 on Smartphones 

(4 iPhone, 3 Android, 1 Win...
@kaniasty | #uxpabos17
28
Task Data
@kaniasty | #uxpabos17
29
Structured Data
@kaniasty | #uxpabos17
30
Usability Measures
Tullis & Albert
@kaniasty | #uxpabos17
31
Data Analysis
@kaniasty | #uxpabos17
32
Infogr.am
PROS:
Range of Graphs
Fun viz options
Full control over colors
Option to show values (...
@kaniasty | #uxpabos17
33
plot.ly
@kaniasty | #uxpabos17
34
“Last but not least… qualitative data analysis.”
@kaniasty | #uxpabos17
35
Final Presentation
Strategic Brief
- Focused on the big picture
- Focused on priorities
- Focuse...
@kaniasty | #uxpabos17
Questions?
Upcoming SlideShare
Loading in …5
×

Fast and Unbiased Usability Research

368 views

Published on

Tips and tricks for optimizing your research process.

Published in: Design
  • Be the first to comment

Fast and Unbiased Usability Research

  1. 1. @kaniasty | #uxpabos17 You can have it both ways Fast and Unbiased Usability Research Eva Kaniasty, Red Pill UX
  2. 2. @kaniasty | #uxpabos17 2 What do we want? Everything. When do we want it? Now
  3. 3. @kaniasty | #uxpabos17 3 What is good research? “You can’t always get what you want.”
  4. 4. @kaniasty | #uxpabos17 4 Bias is everywhere! • Inter-evaluator bias • Loaded or leading questions • Cognitive biases (e.g. primacy, recency) • Group dynamics (i.e. groupthink) • Prejudgements • Selection bias • Respondent bias (e.g. positive, social) “Soylent Green… Bias… 
 it’s made of people!”
  5. 5. @kaniasty | #uxpabos17 5 • Plan • Structure • Focus • Automate • Don’t rely on memory Reduce Your Cognitive Load “An ounce of prevention is worth a pound of cure.” - Ben Franklin
  6. 6. @kaniasty | #uxpabos17 6 “Efficiency is intelligent laziness.” Objectivity through Optimization
  7. 7. @kaniasty | #uxpabos17 Good + Efficient Participant Recruiting
  8. 8. @kaniasty | #uxpabos17 8 Recruiting Recruiting Brief Screening Survey Scheduling - Research Recruiter - UserInterviews.com - Stakeholders Participant Data - Contact Info - Time Zones - Screening Data “Keep it simple…”
  9. 9. @kaniasty | #uxpabos17 9 Customer Super Screener (i.e. survey) Better screening - Larger pool of potential participants - Foot in the door - Comparison sample - Bias fighter Measure twice, cut once Collect data as you screen - Demographics - Feature usage - Card sorts - Desirability (product reaction) - Technical - Quotes (collected judiciously) - Opt-in for future research 
 (online & moderated)*Don’t forget legal review
  10. 10. @kaniasty | #uxpabos17 10 Recruiting Logistics Written Consent •Context •Logistics •Recording •Compensation •(Optional) Nondisclosure *Don’t forget legal review “80 percent of success is just showing up.” Reducing No-Shows •Written consent •Email confirmation •Remote tool testing/setup •Phone/Texting Session Status Tracking Compensation
  11. 11. @kaniasty | #uxpabos17 Good + Efficient Study Design
  12. 12. @kaniasty | #uxpabos17 12 “Get your priorities straight.”
  13. 13. @kaniasty | #uxpabos17 13 Moderator Guide Pre-Test Interview - Background / Profile Fit - Past experience/usage - Top 3 tasks/needs First Impressions - (if website) HP overview - (if navigation) Navigation expectations - (if new user) Free exploration - (if current user) Recent task Tasks or Process - Probing questions - Ease of use ratings - Task Success Wrap-Up - Salient Likes/Dislikes - “Change one thing” - System Usability Scale/Global Measures “How the sausage gets made.”
  14. 14. @kaniasty | #uxpabos17 Good & Efficient Data Collection
  15. 15. @kaniasty | #uxpabos17 15 Session Housekeeping Participant Briefing - Context-setting - Recording consent - Think-aloud instruction - Honest Feedback - Neutral Observer Logistics - Apps / Links - Notes templates - Equipment - Logins - Recording - Reset “Don’t forget to eat.” - Mom
  16. 16. @kaniasty | #uxpabos17 16 Recording & Photography • Personal room • Test meeting OR Tech call • Mac & Windows compatible Mobile Screen Mobile Device • Sleds
  17. 17. @kaniasty | #uxpabos17 17 Super Notes! Mod Guide + Notes + Audio PearNote Mac OneNote Windows Livescribe
  18. 18. @kaniasty | #uxpabos17 18 The Session Summary This participant had major issues with X and Y features because she expected [X] and [Y] to happen. She followed an interesting workflow that was different from what we expected. She also did [this interesting or surprising thing].
 
 She was hesitant to explore the UI and fit the ‘Ms. X’ persona pretty well… “Take 5.”
  19. 19. @kaniasty | #uxpabos17 19 Avoiding the Slow TV Syndrome • Re-watching videos • Relying on notes alone • Human transcription • Automatic transcription VLC player
  20. 20. @kaniasty | #uxpabos17 20 Task Data
  21. 21. @kaniasty | #uxpabos17 21 Usability Measures Custom Tools Question Banks
  22. 22. @kaniasty | #uxpabos17 22 Question Banks Product Reaction CardsSemantic Differentials Ease of Use
  23. 23. @kaniasty | #uxpabos17 23 Spreadsheets
  24. 24. @kaniasty | #uxpabos17 Good & Efficient Reporting
  25. 25. @kaniasty | #uxpabos17 25 Analysis & Reporting Strategy Findings Report - Narrative - Includes detailed study methodology & background - Includes raw data - Includes participant detail - Includes session summaries - Focused on detailed findings Strategic Brief Presentation - Focused on the big picture - Focused on priorities - Focused on critical findings - Includes recommendations - Visually impactful - Includes quotes - Synthesizes data from multiple sources - Summary deliverables: 
 (e.g. profiles, personas, user journeys) “Use the right tool 
 for the job.”
  26. 26. @kaniasty | #uxpabos17 26 NOT Survey Reports No Custom Color Coding Always reports %
  27. 27. @kaniasty | #uxpabos17 Participant Data 27 ♀ 5 Female ♂ 5 Male 4 on Laptops 4 on Smartphones 
 (4 iPhone, 3 Android, 1 Windows Phone) Customers Company 1, Company 2, Company 3 Customers RepresentedProfile 1 | Profile 2 Session Summaries as part of detailed findings / or with recordings
  28. 28. @kaniasty | #uxpabos17 28 Task Data
  29. 29. @kaniasty | #uxpabos17 29 Structured Data
  30. 30. @kaniasty | #uxpabos17 30 Usability Measures Tullis & Albert
  31. 31. @kaniasty | #uxpabos17 31 Data Analysis
  32. 32. @kaniasty | #uxpabos17 32 Infogr.am PROS: Range of Graphs Fun viz options Full control over colors Option to show values (not %) CONS: Limited font control No option to show confidence intervals
  33. 33. @kaniasty | #uxpabos17 33 plot.ly
  34. 34. @kaniasty | #uxpabos17 34 “Last but not least… qualitative data analysis.”
  35. 35. @kaniasty | #uxpabos17 35 Final Presentation Strategic Brief - Focused on the big picture - Focused on priorities - Focused on critical findings - Includes recommendations - Visually impactful - Includes quotes - Synthesizes data from multiple sources - Summary deliverables: 
 (e.g. profiles, personas, user journeys) “Looks matter”
  36. 36. @kaniasty | #uxpabos17 Questions?

×