Bill Albert from Bentley University led a webinar on remote testing methods, focusing on various tools and techniques for usability testing. The session covered the definitions and categories of remote testing tools, their advantages such as cost-effectiveness and rapid feedback, and the importance of understanding user experience metrics. Special guests included leaders from usertesting.com, optimal workshop, and userzoom, sharing insights on tool selection based on research goals.

1. Bill Albert, Director of Design & Usability
Center at Bentley University presents a
complimentary webinar on Remote Testing
Methods and Tools.
With special guests:
Webinar:
Remote Testing Methods & Tools
December 13th, 2011

2. Webinar Overview
1. Brief introductions & goal of the webinar
2. Updated overview of competitive landscape, updated figures, etc
3. The different methods / categories / tools. When to use what?
4. Types of research goals
1. Categories of tools available
2. Overview of some of those tools
5. Examples:
– Usertesting.com
– Optimal Workshop
– UserZoom
6. Takeaways, Q&A
2

3. Introduction
• Speakers today:
o Bill Albert, Director of Design & Usability Center, Bentley University
o Dave Garr, Co-Founder, UserTesting.com
o Andrew Mayfield, CEO, Optimal Workshop
o Matt Paulus, Director of US Sales, UserZoom
• Goals of today’s session
3

4. Remote Testing Methods and Tools
Bill Albert
December 13, 2011
walbert@bentley.edu
781.891.2500 | www.bentley.edu/usability

5. Agenda
• Introduction to remote usability testing
• Quantitative-based tools
• Qualitative-based tools
5

6. About me
• Director, Design and Usability Center, Bentley University
• Experience with remote testing tools
• I am data driven!
6

7. I am a big fan of usability testing in the lab!
7

8. Introduction to remote usability testing

9. Defining Remote Testing Tools
What do we mean by “remote testing tools”?
• A remote testing tool is any technology that allows a researcher to collect
data from users about their experience in using an interface, without
direct, face-to-face contact.
There are two ways of organizing remote testing tools
• Moderated vs. Unmoderated
• Qualitative vs. Quantitative-based
Moderated Unmoderated
Qualitative-based Qualitative-based
Quantitative-based Quantitative-based
9

10. Unmoderated Testing Tools
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
10

11. Our UX Toolkit
11

12. Why Should You Care?
• Saves time
o Lab study takes 2-4 weeks from start to finish, unmoderated typically takes
hours to a few days*
• Saves money
o Participants compensation typically a lot less ($10 vs. $100)
o Tools are becoming very inexpensive
• Reliable metrics
o Only (reasonable) way to collect UX data from large sample sizes
• Geography is not a limitation
o Collect feedback from customers all over the world
• Greater Customer insight
o Richest dataset about the customer experience
12

13. Why is this important NOW?
• Questions from senior management are becoming more
complex and time sensitive
• Traditional usability is no longer enough
• Push to measure the UX
• Convergence with market research and web analytics to paint a
more complete picture of the UX
• Budgets/timeline constraints
13

14. Which one goes first?
Lab first, then Unmoderated Unmoderated first, then Lab
Identify/fix “low hanging fruit”, then Identify the most significant issues
focus on remaining tasks with large online through metrics, then use lab
sample size study to gather deeper qualitative
understanding of those issues
Generate new concepts, ideas, Collect video clips or more quotes of
questions through lab testing, then users to help bring metrics to life
test/validate online
Validate attitudes/preferences Gather all the metrics to validate
observed in lab testing design – if it tests well, then no
need to bring users into the lab
14

15. Quantitative-based tools

16. Unmoderated Testing Tools - Quantitative
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
16

17. Overview
Common Research Questions:
• What are the usability issues, and how big?
• Which design is better, and by how much?
• How do customer segments differ?
• What are user design preferences?
• Is the new design better than the old design?
• Where are users most likely to abandon a transaction?
Types of Studies: Typical Metrics:
• Comprehensive evaluation • Task success
• UX benchmark • Task time
• Competitive evaluation • Self-report ratings such as ease of
• Live site vs. prototype comparison use, confidence, satisfaction
• Feature/function test • SUS
• Discovery • Click paths
• Abandonment
17

18. Strengths / Limitations / Myths
Strengths:
• Comparing products
• Measuring user experience
• Finding the right participants
• Focusing on design improvements
• Insight into the user’s real
experience
Limitations: Myths:
• Not well suited to rapid, iterative • Only test with websites
design • Very expensive
• Need a deep understanding of • Only gather quantitative data
issues • A lot of noise in the data
• Studies that require long sessions • Same as any market research study
• Lose control over prototypes • Comparing products
• Internet access
18

19. Full Service Tools – Online Usability Testing
19

20. Self Service Tools – Online Usability Testing
20

21. Card Sorting / IA Tools
21

22. Online Surveys
22

23. Click and Mouse Tools
23

24. Qualitative-based tools

25. Unmoderated Testing Tools - Qualitative
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
25

26. Overview
Common Research Questions:
• What are the big pain usability pain points, and
wins?
• How do users think, and feel about the design?
• Are we on the right (design) track?
• What is the overall experience like?
• Why might users abandon a transaction?
Types of Studies: Common Metrics:
• Low fidelity prototype • Frequency of issues
• High fidelity prototype • Verbatim comments
• Competitive evaluation • Task success (via video)
• Comprehensive evaluation • Task time (via video)
• Feature/function test
• Discovery
26

27. Video Tools
27

28. Report Tools
28

29. Expert Review
29

30. Additional Resources
Online:
www.measuringUX.com
www.measuringusability.com
http://usability.bentley.edu
http://www.usefulusability.com/24-usability-testing-tools/
http://remoteresear.ch/tools/
Follow me:
@UXMetrics
@BentleyDUC
LinkedIn Group:
Bentley Design and
Usability Center
30

31. Dave Garr
Co-Founder
UserTesting.com

32. What’s your visitor thinking?
In Cyberspace, no one can hear your visitor scream.

33. Solution?
33

34. How It Works

35. “I don’t know why it keeps doing this.
This is definitely very confusing and frustrating.”

36. Write an Annotation

37. “I can’t figure out which cartridge goes with my printer”

38. How to share clips

39. View Tester’s Answers

40. Follow up with users

41. Export to Excel

42. Mobile Testing
• Test mobile apps
• Test mobile-friendly sites
42

43. “I’d like to look at these three star reviews, … it doesn’t let me
do that… I’d have to go through all of the reviews.”
What do you want to test?
43

44. Presenting Your Findings
44

45. Andrew Mayfield
CEO
Optimal Workshop

46. Quantitative analysis
to assess the
effectiveness of an
Information
Architecture is
hard work.

47. Peter Morville’s
UX Honeycomb

48. Optimal Workshop

49. The Suite
by Optimal Workshop

50. Qualitative
vs
Quantitative
See latest blog post on this topic
www.optimalworkshop.com/blog

53. Participant Centric Analysis

55. Remote studies enable near unlimited
iteration and participation
BENCHMARK
Radical
Safe or
option
existing
options
?

56. Matt Paulus
Director of US Sales
UserZoom

57. UserZoom’s Software Solutions
• On-demand, web-based SaaS
• 100% self-service
• All-in-one quantitative unmoderated
online research
• Enterprise level software
• Feature rich & intuitive

58. Unmoderated Remote Testing Methodology
• Quantitative remote testing
• Natural context
• International/domestic testing
• No moderation needed
• Behavior (analytics) + Feedback (online
Surveys)

59. UserZoom Products

60. UserZoom Products
Live Websites Wireframes & Prototypes
Compatible Platforms
Multimedia Mobile Interfaces
content & Apps
60

61. UserZoom Task-Based Usability Testing

62. UserZoom Results – Open Ended/Text Clouds

63. UserZoom Results – Net Promotor Score

64. UserZoom Results – Filters/Segmentation

65. UserZoom Results – Effectiveness & Efficiency

66. UserZoom Results – Clickstreams

67. UserZoom Results- Click Heatmaps

68. Key Takeaways
• Remote testing tools are becoming more advanced every day, and are a
critical part of the UX toolkit
• Find the right type of tool based on your research question
• Demo a select number of remote testing tools to find the one that will
deliver the results you need, within budget and timeframe
68

69. Quick Poll!
69

70. Thank You!
Bill Albert Dave Garr Andrew Mayfield Matt Paulus
Director of Design Co-Founder CEO Director of US
& Usability Center UserTesting.com Optimal Sales
Bentley University Workshop UserZoom
Floor open to Q&A