This document discusses conducting user experience (UX) benchmarking studies. It begins by outlining the agenda and explaining why companies should run UX benchmarking studies to establish baselines, identify areas for improvement, and track changes over time both internally and against competitors. It provides tips on setting goals and objectives, recruiting appropriate participants, sample sizes needed, and key performance indicators to measure. Finally, it discusses analyzing the data, identifying best practices, and how benchmarking can help build a mature UX organization.

1. Moving the Needle with UX Benchmarking
Dana Bishop, Sr. Director UX Research

2. Quick Housekeeping
• Control panel on the side of your screen if you
have any comments during the presentation
• Time at the end for Q&A
• Today’s webinar will be recorded for future
viewing
• All attendees will receive a copy of the
slides/recording
• Continue the discussion using #uzwebinar
Let’s make sure you’re all set up for the webinar!

3. Agenda
Why run UX benchmarking studies?
Dana Bishop
dbishop@userzoom.com
+1 (866) 599-1550
Expert tips on conducting UX benchmarks
Setting your benchmarking goals & objectives
Analyzing your data - work smarter, not harder!
Why benchmarking is integral for UX maturity

4. UX Benchmarking
LONGITUDINAL
BENCHMARKING
COMPETITIVE
TAKE
ACTION
COLLECT
DATA
ANALYSIS

5. Benchmarking creates a baseline for
understanding the current user
experience on your website or app

6. Tip: Include survey questions to help keep abreast of changes in usage, needs, and expectations of your users.
ü The first run provides a baseline to measure against in the future.
ü Helps deliver insights and establish goals to guide you where to focus efforts to improve and
optimize the user experience.
ü However, once is not enough! Keep testing to track how the experience changes over time.
o At regular intervals (e.g. annually, bi-annually, or quarterly).
o Retest after rolling out major updates or redesigns.
ü Compare how design iterations impact the user experience.
ü Track and measure changes in KPIs and other metrics over time, and what’s driving those changes.
Internal UX Benchmarking
The benefits of a longitudinal benchmark program for your digital properties

7. Example Longitudinal Scorecard: Task Success
Task success rate improves significantly month-over-month
81.1
77.3
82.3
76.6
77.2
81.2
40
50
60
70
80
90
100
Oct-16 Nov Dec Jan Feb Mar Apr May June July Aug Sept Oct-17
Task 1 Task 2 Task 3
Differences of approximately +/- 12% is significant at the 90% confidence
level
Significant
improvement

8. Example Month-over-Month Improvement: Details
How many steps are involved and where you are in process was much easier to understand in the redesign,
compared to the previous “tabbed” design
Tabs did not make steps/ process clear
Redesigned booking process makes it easier
to navigate site and complete booking
successfully; steps clearly laid out
September 2017
= Statistically
meaningful difference
Benchmark Metrics 9/17 10/17
Booking Ease 64% 83%
Clarity of Navigation/ Steps 60% 75%
Task Success 63% 77%
October 2017

9. Example Pre-to-Post Measures
Positive Brand Perception drops 35% after using site: BCBS Massachusetts did not meet user’s expectations
68%
64% 62%
75%
70%
46% 46%
40%
Texas New Jersey California Massachusetts
Selected 6-7 on 7-point scale where 1 = Negative and 7 = Positive
What is your perception of [SITE]?

10. Don’t miss out on the many
benefits of expanding your testing
universe to include your
competitors

11. Competitive UX Benchmarking
ü Competitive benchmarking provides valuable measurements of how you perform relative to your
competitors.
ü Competitive analysis offers a way to interpret your usability standing, feature set, and more, in your
industry.
ü Benchmarking your competitors gives you the added benefit of learning from their successes and
failures; gaining insight into what they’re doing that works, what doesn’t work, and why.
ü Identify best-in-class examples to emulate.
Why benchmark your competitors too?
Tip: Which competitors should you include? The ones you look at and discuss the most, the ones you
feel you’re falling behind, and the ones that you think you can learn the most from.

12. Learning From Your Competitors’ Successes and Failures
Task Ease: Wells Fargo fared significantly worse for ease of finding a branch
82%
78%
66%
60%
Bank of America Chase Citibank Wells Fargo
* Selected 6-7 on a 7-point scale where 1 = Very difficult and 7 = Very easy

13. Learning From Your Competitors
Find a Branch: Top Problems and Frustrations – Wells Fargo
18%
16%
14%
14%
12%
8%
6%
6%
2%
4%
56%
Unclear if location had a physical branch
Difficult to know where to ’start’ looking for this
information
Text was too small / difficult to read
Information on the site was confusing / unclear
Too many clicks required to find information
Information was not where I expected it to be
Site was slow
Too much scrolling to find information
Encountered errors / technical issues
Other, please specify
I did not encounter any problems or frustrations
n=50

14. Learning From Your Competitors
Wells Fargo had the highest Non-Success rate of all sites for this task at 62%
Of those who were not successful:
• 42% used incomplete address information in the locator tool
• 26% chose a location without a physical branch
Sample Video (uses correct address, but confused by results)
Participant ID 130:
“…It says Union Trust. Is that the name of the bank? I’m confused.”

15. Take the time up front to carefully
identify the purpose of the
research

16. Setting Your Research Goals & Objectives
STAY FOCUSED
Don’t set too many goals or
ask questions that are not
directly tied back to helping
answer your top questions.
ROADMAP
Collect the data you need to
assist in making sound
decisions.
KNOWN PROBELMS
Address problems you or your
customers have already
identified.
YOUR HYPOTHESIS
Design your tasks and questions to get the
answer you need to confirm or refute your
hypothesis.
INDUSTRY BEST
PRACTICE
DEFINE
Be clear and specific when you
establish your benchmarking goals.
Identify the questions, concerns, areas of interest, and purpose of the research.
Understand business concerns; and review previous research and knowledge.

17. The greatest insights are derived from
collecting feedback from real users

18. Your Participants
ü Recruit participants who match your demographics (or better yet personas, if you have them).
ü Create a screener survey that will qualify participants based on your pre-defined criteria (e.g., age,
previous experience, or other demographics, traits, or attitudes).
ü Use the same screener each time you run your benchmarking study. Consistency is imperative to
yield reliable longitudinal data.
Existing customers and/or intended target audience.
Tip: Keep the proportions even across all competitors. Otherwise, your competitive data will be skewed.

19. How Many Participants?
ü Need samples large enough to make projections to the population.
ü A “significant” difference is one that we are confident we didn’t find by chance alone.
ü Confidence increases with sample size and the accuracy of our measures.
ü For quantitative benchmarking studies, as a rule of thumb, sample sizes of 50n or more are
recommended.
ü An additional consideration is the inclusion of multiple segments or groups. If you are going to run
any segmentation analysis (comparing two or more different groups) you will need to increase your
sample size accordingly.
Benchmarking studies require a larger sample size to get statistically significant and sound
results.
Tip: Because of the large sample size, these studies require more time in the field.

20. We know that finding the “right”
participants matters

21. UserZoom’s Intelligent Sourcing Engine
Tip: Leverage our Intelligent Sourcing Engine, IntelliZoom, for access to over 120 million
users worldwide. Now with instant, automated access to over 28 million of them for User
Videos in hours.
Our Participants Your Participants
Intelligent Sourcing Engine
Invitation Links
120
MILLION+
UNLIMI
TED
and FREE
UserZoom
Panel
Unique URLs

22. Or Invite Your Own
Our Participants Your Participants
Intelligent Sourcing Engine
Invitation Links
120
MILLION+
NO LIMITS
FREE
Email QR Code Mobile Intercepts Web Intercepts Private Panel Social Sites
Unique URLs

23. Designing a benchmarking study
TAKE YOUR TIME.
DESIGN ONCE, RUN MULTIPLE TIMES.
Tasks should be consistent across all
competitors. Review each site to
confirm all tasks can be completed
across competitors.
CONSISTENCY
Carefully consider the usability
and business metrics you want
to track.
KEY PERFORMANCE INDICATORS (KPIs)
Makes it quick and easy to
design and build your
benchmark, then repeat/ re-
run as needed in the future.
USERZOOM PLATFORM
Include the primary tasks users
do on your site or app.
A Retail benchmark might include:
1. Browse for Product
2. Search for Product
3. Purchase Process
TASKS

24. User Experience Key Performance Indicators (KPIs)
BEHAVIORAL (WHAT THEY DID)
|__| Task Success
|__| Task Time
|__| Page Views, Clicks
|__| Problems & Frustrations
ATTITUDINAL (WHAT THEY SAY)
|__| Usability (Ease Of Use)
|__| Trust & Credibility
|__| Appearance
|__| Loyalty (NPS)

25. Example Competitive Benchmarking KPIs
Out of the sites studied, Citibank had the highest success rate, but the lowest scores on all of the other KPIs. Although
Chase participants had the lowest overall task success rate, users reported that they encountered the fewest
problems or frustrations.
Key Performance Indicators
Bank of America
n=50
Chase
n=50
Citibank
n=50
Wells Fargo
n=50
Task Success 64% 60% 72% 70%
Average Time on Task 2.59 min 2.75 min 4.31 min 2.37 min
Average # of Page Views 4.2 pages 4.3 pages 7.6 pages 3.8 pages
Did not encounter problems or frustrations 44% 62% 34% 38%
Provided just the right amount of information 64% 84% 50% 64%
= highest proportion / fastest time / lowest # of pages = lowest proportion / slowest time / highest # of pages

26. If you took the time up front to
design your study with the data
analysis in mind, then it will be a
breeze

27. 01
Pre-planned analyses.
Know what you will do with
each piece of data you
collect.
02
Comparative analysis of KPIs.
Analyze longitudinal changes and
trends.
03
Include qualitative feedback and
behavioral data to support the
“why” behind the results.
Synthesize findings into insights.
04
Identify best-in-class examples.
Think through implications,
recommendations, and next steps.
Data
Analysis
Analysis Rule #1: Work smarter, not harder!

28. Single UX Score
Score = Avg of top 2 box % (all 4 measures) & Task Success % (all tasks). Then converted to
Index scale.
Behavior Attitudes
Usability
Trust
Appearance
NPS (Loyalty)
Task Success

29. Example Single UX Score: Healthcare Insurance
48%
56%
63%
76%
Massachusetts
New Jersey
California
Texas
GreatGoodAveragePoorVery Poor

30. Get the complete story behind your users’ experiences
UX METRICS
Automate the collection of data across large,
statistically significant representative
populations to measure the user experience.
USER VIDEOS
Capture video and screen recordings of users as
they walk you through their experience with your
website or app.

31. Get the complete story behind your users’ experiences
HEATMAPS & CLICKSTREAMS
Capture video and screen recordings of users
as they walk you through their experience
with your website or app.
NPS & RATINGS
Automate the collection of data across large,
statistically significant representative populations
to measure the user experience.

32. Building a Mature UX Organization
UX Resources and
Techniques
Timing of UX
Involvement
Leadership and
Company Culture

33. Building a Mature UX Organization
ü Leadership understands the value and necessity of UX from a business
perspective.
ü Valued and respected in the company culture.
ü UX processes are connected and integrated with other corporate processes
that enable individuals to work together to create the user experience of the
product(s).
ü UX is integrated into every stage.
EXCEPTIONAL
ADOPTING &
GROWTH
ü In house and/or ability to bring in UX expertise quickly as
needed.
ü Use of appropriate techniques and deliverables.
ü UX is involved earlier in the process.
BEGINNING
ü Limited UX resources.
ü UX testing is not involved early in the design cycle.
ü Not an integral part of the company culture,
business objectives, or design and development
processes.
Low
High
UX Benchmarking is directly
actionable and helpful for
organizations on both a strategic
and tactical level

34. Walkthrough of UserZoom’s
Research Platform
Upcoming Webinar Part 2:
Designing Impactful UX Benchmark Studies with UserZoom
February 20 @ 8am PST |11am EST
1
2 Single UX Score
Please join us for a deep-dive into how to utilize UserZoom for all your benchmarking needs
Professional Services Team
Expertise
3
4 DIY Benchmarking

35. Q&A
marketing@userzoom.com +1 (866) 599-1550