Join Dr. Andrea Peer and learn: -How Universal Measures makes tangible the abstract concept of experience for your organization -How practitioners can make experience a critical KPI for their organization -Ways to establish experience score goals for all lines of business -The benefits Universal Measures brings to executives and stakeholders

1. MAKE USER EXPERIENCE PART OF
THE KPI CONVERSATION WITH
UNIVERSAL MEASURES
Dr. Andrea Peer | UserZoom

2. Agenda
Where Did KPIs Come From?
WEBINAR SPEAKER
Dr. Andrea Peer
Onboarding Customer Success
Manager at UserZoom
Contact Info
Connecting Experience to ROI
Consider an eXperience Score
as a KPI
Universal Measures to grow
experience practice
Q&A
1
2
3
4
5

3. Quick Housekeeping
• Control panel on the side of your screen if you
have any comments during the presentation
• Time at the end for Q&A
• Today’s webinar will be recorded for future viewing
• All attendees will receive a copy of the
slides/recording
• Continue the discussion using #uzwebinar
Let’s make sure you’re all set up for the webinar!

4. Where did KPIs come from?

5. Where did KPIs come from?
INDICATORS
20th Century - Fredrick Taylor gave us
Scientific Management
• Operational performance
• Efficiency and effectiveness
1920 – DuPont Analysis, ROI
• Financial Indicators
1930’s Tableau de Board…..Dashboard
1992 - Balanced Score Card
• Strategic Performance

6. Cue the world of
dashboards and
information vomit

7. So Many KPIs
Financial Indicators
System Produced indicators
Experience indicators
Return On
Investment Transactions
Net Profit/LossCash Flow
Capital Reserves / Debt
Site Traffic Bounce Rate
Click Analytics
CTA Engagement CSat
NPS
Task
completion
Time on
Task
Satisfaction
Ease of Use SUS
SUPRQ
SUM
Journey
Metrics

8. One Score Obsession
Snapstreaks quantify a
friendship in Snapchat
Financial Score communicates
your financial health
The number you need to
increase customer loyalty
Helps to focus and ground
Simplifies abstract concepts

9. Poll Results

10. Poll Results

11. Connecting experience
measures to ROI

12. The KPI Conversation
Financial Indicators
System Produced indicators
Experience indicators
Return On
Investment Transactions
Net Profit/LossCash Flow
Capital Reserves / Debt
Site Traffic Bounce Rate
Click Analytics
CTA Engagement CSat
NPS
Task
completion
Time on
Task
Satisfaction
Ease of Use SUS
SUPRQ
SUM
Journey
Metrics

13. Tying UX to ROI
1
CHAPTER
Justifying Cost-Justifying Usability
Randolph G. Bias School of Information, The University of Texas at Austin
Clare-Marie Karat IBM TJ Watson Research Center
1.1 INTRODUCTION
Sir Walter Scott wrote, in Marmion,
"Oh! what a tangled web we weave
When first we practise to deceive!" (Scott, 1805)
Sometimes it is quoted as "... what a wicked Web we weave."
This year is the bicentennial anniversary of the publication of Scott's book.
Did he anticipate the World Wide Web, 200 years ago? Scott's words are relevant
to today's networked world and the development activities that support it, includ-
ing designing for ease of use.
Usability is not the end-all-be-all; rather, it must be considered alongside (and
in equal measure with!) functionality and schedule. The problem is not that Web
site developers are wicked; rather, they are too often in a hurry and are not oper-
ating in a reward structure that motivates attention to usability.
So the Web is not wicked. But tangled?. Oh yeah.
Most of us are wowed by the Internet. It is amazing what we can accomplish
while sitting at home in our pajamas. But when we say "us" or "we," we are refer-
ring to that small subset of the population who have some expectation of what
will happen if we "right click," or those of us who have heard of, for example,
Doug Engelbart. When one of "us" sits down with one of "them" (the great
unwashed masses), the madcap hilarity begins. "They" are so stupid! Can you
believe it?, some of them don't even know that an underline, on a Web page,
means the text is a link to another page (except when it isn't). "They" don't even
v1, Feb 2005. Includes material adapted from the chapter: Cost benefits framework and case studies, in Bias, R.G. & Mayhew,
D.J. (2005) Cost-Justifying Usability: An Update for the Internet Age.
1
Cost benefits evidence and case studies
Nigel Bevan
Serco Usability Services
nbevan@usability.serco.com
Contents
1 – Potential benefits of usability 2
1.1 Development savings 4
1.2 Increase e-commerce sales 5
1.3 Product sales 7
1.4 Usage benefits for employers 8
1.5 Reduce support and maintenance costs 10
2 – Estimating costs 11
2.1 User centered design methods 11
2.2 Costs of user centered design 13
3 – Making the cost-benefit case 14
4 – Case studies 16
4.1 Israel Aircraft Industries 16
4.2 Inland Revenue/EDS 23
4.3 Comparison 26
4.4 Taking up the methods 27
Acknowledgements 28
References 28
This paper summarises the benefits that can be obtained from use of user centered design, and
discusses how to select appropriate methods and justify their cost benefits. It includes two case
studies of the cost benefits of using a usability maturity model to improve the usability capability of
an organisation.
Usability Return on Investment (ROI)
By Jakob Nielsen, J.M. Berger, and Shuli Gilutz
3rd
edition
48105 WARM SPRINGS BLVD. FREMONT, CA 94539–7498 USA WWW.NNGROUP.COM
Copyright © Nielsen Norman Group; All Rights Reserved.
To buy a copy, download from http://www.nngroup.com/reports/roi
NN/g,
2008
Bevan,
2005
Bias, 2005

14. Tying UX to ROI
Scribecreatedbywww.PopulationDesign.com
© 2010 Human Factors International, Inc. All rights reserved. www.HumanFactors.com | 800.242.4480 USA & Canada | +1.641.472.4480 International

15. Tying UX to ROI
Is ROI an Effective Approach for Persuading
Decision-Makers of the Value of User-Centered Design?
Susan Dray,
Moderator
Dray & Associates, Inc.
Minneapolis,
MN USA
dray@acm.org
Clare-Marie Karat
IBM T. J. Watson
Research Center
Hawthorne, NY USA
ckarat@us.ibm.com
Daniel Rosenberg
Oracle
Redwood Shores,
CA USA
daniel.Rosenberg
@oracle.com
David Siegel
Dray & Associates, Inc.
Minneapolis,
MN USA
david.siegel@
acm.org
Dennis Wixon
Microsoft
Redmond, WA USA
denniswi@
microsoft.com
ABSTRACT
This panel examines the utility and effectiveness of various
ways of making the business case for user-centered design
(UCD). Most of the discussion in our field has assumed
that measuring and demonstrating ROI for usability is the
key to this effort. However, experience shows that the most
brilliant ROI analysis may not win the day in the real world
of business. Our panelists range from people who claim
that ROI is an important persuasive tool as long as the
communication about ROI is happening within a healthy
business relationship, to people who claim that a focus on
ROI can actually be destructive. We also explore the idea
that there are important business contexts where ROI
simply does not fit. Through the presentations by the
panelists and through discussion of a business case
scenario, we explore some alternatives to ROI in making
the business case for user-centered design.
Author Keywords
Return on Investment, ROI, cost-justification, UCD
ACM Classification Keywords
H5.3. Group and Organization Interfaces (eval/methods)
INTRODUCTION
Over the years, there has been much discussion in the field
of usability about how to objectively and convincingly
demonstrate the benefits that usability produces. This is not
simply an academic exercise, but rather is part of our
profession’s effort to prove its value and thus promote
adoption of UCD methods. Most of the discussion in the
field has focused on measuring and demonstrating return on
investment (ROI) for usability, on the assumption that a
well-constructed ROI analysis is an effective persuasive
tool. Although we certainly hope that accuracy has
something to do with effectiveness of an ROI analysis, it is
also possible that the most brilliant ROI analysis will not
win support for UCD in the real world of business.
If we are to understand the strengths and weakness of the
ROI paradigm as an approach to persuasion, we must
examine questions such as:
• What are the inherent strengths and weaknesses of ROI
arguments?
• What makes for an effective ROI argument, beyond the
quality of the calculations?
• What types of factors, besides ROI, make compelling
cases for UCD?
• Can we identify situations where ROI arguments, or other
approaches are inherently more or less compelling?
• What are the most challenging UCD benefits to create a
compelling argument for? How can ROI and other
approaches address them?
The following sections give an overview of contrasting
perspectives on answers to questions like these.
HUMAN DYNAMICS, GOALS, AND BUSINESS CONTEXT
Beyond the quality of the data on which the ROI calculation
is based, a number of factors relating to the dynamics of the
business are critical in determining the effectiveness of ROI
arguments. First and foremost are the people involved. The
person who hears the argument must have a strategic
business vision or at least be a pragmatist who will
experiment with new methods. If the usability professional
is able to make a credible presentation of the ROI argument
for usability and build trust with the visionary person, then
the stage is set for the next critical factor.
A second factor is how well the ROI arguments tie to the
business goals and strategies of the organization. If the
usability goals can help to achieve organizational goals or
even take them a step further, this approach can
demonstrate to the organization a new type of partnership
they find very valuable.
A final critical factor involves world events that are
affecting the business world. One is the global marketplace
and the movement of jobs and work to areas of the world
where people can work at lower cost. Other significant
events are the impending retirement of the "Baby Boom"
generation with a resulting skills shortage, and increasing
life expectancy rates in many parts of the world. If an ROI
argument can be heard by visionaries, and will help them to
achieve business goals or make strategies successful, and if
Copyright is held by the author/owner(s).
CHI 2005, April 2–7, 2004, Portland, Oregon, USA.
ACM 1-59593-002-7/05/0004.
CHI 2005 | Panels April 2-7 | Portland, Oregon, USA
1168
Dray, 2005

16. Tying UX to ROI
Accomplishment
Joy
Freedom
Delight
Connection

17. Poll Question

18. Poll Question

19. Consider an eXperience Score
as a Key Performance Indicator

20. What is your
eXperience
Score?
Your experience score is the indicator
that your product and service is having
the desired experience outcome.

21. EXPERIENCE SCORES
• Reduce the fuzziness of the abstract concept of experience
• Define what experience means for your product(s) and service(s)
• Provide 1 score for experience that your organization can track
• Make experience research actionable
eXperience Score As A KPI

22. But wait…Experience
CANNOT be
accurately conveyed
by 1 score!
eXperience Score As A KPI

23. eXperience Score As A KPI

24. eXperience Score is part of a
bigger journey to grow your
experience practice.
Cue…Universal Measures

25. 1. Create your eXperience score
2. Determine the story you want to tell with the score to grow
experience practice for your organization
3. Create your executive dashboards and product scorecards to
ensure the experience score is front and center and it supports
the story you are driving
4. Analyze the relationship between your eXperience score and
other business KPIs.
Universal Measures

26. 1. Create your eXperience score
2. Determine the story you want to tell with the score to grow
experience practice for your organization
3. Create your executive dashboards and product scorecards to
ensure the experience score is front and center and it supports
the story you are driving
4. Analyze the relationship between your eXperience score and
other business KPIs.
Universal Measures

27. How does your organization
define & measure the ideal
experience with your product(s)
or service(s)?

28. What is your
eXperience
Score?
Your experience score is the indicator
that your product and service is having
the desired experience outcome.

29. Define Your Experience Goal
What is the intended experience
with your product and service?
What can people do? See? Know?
What and how can people achieve?
How can people be? Feel? Understand?

30. Create your experience score
Pick at least 1
measure in
each cell

31. Create your experience score
Pick at least 1
measure in
each cell Task Completion/
Success
Ease of Use
SUSJourney Score

32. What is a eXperience Score?
Final eXperience
Score - 61%
SUS
40%
Journey
85%
Avg Task
Completion
60%
Avg Ease of
Use
60%
System Scores Task Scores
EXAMPLE

33. What is a eXperience Score?
System Scores
Final eXperience
Score - 63%
Joy Factor
1
45%
Joy Factor
2
80%
Joy Factor
3
65%EXAMPLE
Avg Task
Completion
60%
Task Scores

34. What is a eXperience Score?
System Scores
Final eXperience
Score - 53%
SUPRQ
40%
Avg Task
Completion
60%
Confidence
60%
Task Scores
EXAMPLE

35. Immediate Considerations
• Can I weight variables in my algorithm that are more important?
• How many variables are too many?
• What if a variable isn’t relevant to my research method?
Tactical Questions
• Do I have to change my scales?
• How do you do the calculation?
eXperience Score Questions

36. Long Term Considerations
• Can I change my variables over time?
• Do I report one score across all of my products or per product?
• Do all my products have the same variables?
• What is a good growth goal for my eXperience Score?
eXperience Score Questions

37. 1. Create your eXperience score
2. Determine the story you want to tell with the score to grow
experience practice for your organization
3. Create your executive dashboards and product scorecards to
ensure the experience score is front and center and it supports
the story you are driving
4. Analyze the relationship between your eXperience score and
other business KPIs.
Universal Measures

38. Common Stories for Executive Dashboards
- The black box of dev story
- The pre prod investment story
- The post prod baseline and gap story
- The portfolio opportunity story
- The full spectrum story
- The user data powerhouse story
- The experience investment vs value story
What story do you want to tell?

39. 1. Create your eXperience score
2. Determine the story you want to tell with the score to grow
experience practice for your organization
3. Create your executive dashboards and product scorecards to
ensure the experience score is front and center and it supports
the story you are driving
4. Analyze the relationship between your eXperience score and
other business KPIs.
Universal Measures

40. Creating your Executive Dashboard
& Product Scorecards
2 Types of Dashboards
Executive UX Dashboard Product UX Scorecard
Purpose: Show impact of UX on
org and highlight UX win/work
Target Audience: Senior
Leadership
Frequency: Monthly or quarterly
Purpose: Show result of UX effort
for a specific product line
Target Audience: Development
team leaders/PMs/POs/etc.
Frequency: Bi-Monthly, Monthly

41. Executive Dashboard
• Let’s deep dive on an executive dashboard that works for your organization. Let’s craft
the story you want to tell to your executives.
• What connection can we make to your analytics dataset?
• What connection is important to make to your key business measures?
• What other reports in the business do we need to connect to?
• Determine the process/channel to serve the dashboards to your internal stakeholders
• Determine frequency
Product Scorecard
• Let’s deep dive on a product scorecard that works for your organization. Let’s craft the
story you need to tell for development and product teams.
• Evaluate PDLC process and systems you can integrate with (ie: Jira, Rally)
• Determine the process/channel to serve the dashboards to your internal stakeholders
• Determine frequency based on PDLC integration strategy
Creating your Executive Dashboard & Product Scorecards

42. 42
Experience Scorecard - Quarterly Report (Q1 2016)
ROI Impact
20 product lines, 5 personas impacted
Prior to Development
53 Severe impact discoveries found and 60% of the
discoveries addressed
20 Moderate impact discoveries found and added to the UX
backlog
Post Development (Beta)
3 Severe impact discoveries found and addressed
2 Moderate impact discoveries found and added to the UX
backlog
Post Development (Full Prod)
5 Severe impact discoveries found and added to backlog for
next sprint
7 Moderate impact discoveries found and added to the UX
backlog
UX Performance
Cadence: 1 research engagement per design sprint
Avg Field Time: 5 hours
Avg Participants per Study: 32
Persona Representation: 10% Bob, 50% Julie, 40% Chip
Research Conversation: 40%
Avg
SUS
+15%
Avg Sat
+10%
UX Score
75%

43. 43
Product Selector
Still at the will of consumer knowledge.
Improvement over No/Building/Good selector.
In the Product Selector Modal...
84% prefer 'Average' (over ‘Typical’)
What Product Selector Does Well…
39% said “Average + ‘9/10’ normalizes the experience”
Device Selection
Ease of Adding Device: Desktop 6.5, Mobile 6.1
Satisfaction. w/ Device Presentation:
Desktop 5.7, Mobile 5.5
Cost Awareness
How clear is payment 1? Avg: 5.8
How clear is payment 2? Avg: 6.2
Experience Scorecard - Project
Product: Page communicates value
proposition but is a bit top heavy.
Cart Concept: The productX performs well
with minimum usability issues.
Client Call: Agreeable at first; comfort
reduces when paired with Product.
Page Ratings
(x/7)
85 Non Customer prospects: 60 Desktop | 25 Mobile
What’s Next
Suggestion: The main difference between product X
and Y are abc. Make those two points bold or bring
them out more by shortening sentences into smaller
fragments.
→ A/B Test results indicate design A recommended
over design B as more value measures are favorable
with design A. Pull xxx from design B as a secondary
recommendation.
45
%
20
%
20
%
15
%
Completely OK
Warranted
Makes me
uncomfortable
Wouldn’t do this at all
How do you feel about being
asked for a XYZ?
5.7 5.5 5.1 4.9
Info Satisfaction
Clarity of info
Trust
Likelihood to Click
“Start Shopping”
1
2
3
4
5
6
7
UX Score
60%

44. 1. Create your eXperience score
2. Determine the story you want to tell with the score to grow
experience practice for your organization
3. Create your executive dashboards and product scorecards to
ensure the experience score is front and center and it supports
the story you are driving
4. Analyze the relationship between your eXperience score and
other business KPIs.
Universal Measures

45. Connecting your eXperience
Score to Business KPIs
Financial Indicators
System Produced indicators
Experience indicators
Return On
Investment Transactions
Net Profit/LossCash Flow
Capital Reserves / Debt
Site Traffic Bounce Rate
Click Analytics
CTA Engagement CSat
NPS
Task
completion
Time on
Task
Satisfaction
Ease of Use SUS
SUPRQ
SUM
Journey
Metrics

46. Connecting your eXperience
Score to Business KPIs
System Produced indicators
Experience indicators
NPS
eXperience Score
“R = 0.6”
“eXperience Score of 70% will increase NPS by 1 point”

47. Connecting your eXperience
Score to Business KPIs
System Produced indicators
Experience indicators
eXperience Score
“R = 0.8”
“eXperience Score of 74% will increase CTA Engagement by 5 points”
CTA Engagement

48. Connecting your eXperience
Score to Business KPIs
Financial Indicators
Experience indicators
Return On
Investment
Task
completion
Time on
Task
Satisfaction
Ease of Use SUS
SUPRQ
SUM
Journey
Metrics
“R = 0.65”
“eXperience Score of 82% will increase ROI by x amount”

49. Universal Measurement Investment

50. • Executive leadership
• Small amount of time at meetings/forums where executives reports on
product investment/impact are evaluated
• Long term – some minor infrastructure investments to make the
measures more easily available
• Product Managers
• Incorporate the universal measures metric into your current product
success tracking
• Bring UX into the PDLC – process modifications
• UX Strategist / Leaders
• Champion the effort – education, demonstrate, negotiate
Investment to Create your Universal Measures

51. • UX Researchers
• Consistently bring the experience score measures into your practice
• Establish task libraries for quick access
• Collect and analyze the experience scores to serve up to the forums
where the experience score is shared
• UX Designers
• Collect and analyze the experience scores to serve up to the forums
where the experience score is shared
• Incorporate experience score language when evaluating designs
• Development Teams
• Digital Analytics Teams
Investment to Create your Universal Measures

52. • Development Teams
• Establish checkpoints for experience score investment
• Long term - Minor modifications in ticket tracking system
• UserZoom tracking code into code base
• Digital Analytics Teams
• URL alignment
• Segment alignment
• Sharing of analytics data for correlation studies
Investment to Create your Universal Measures

53. Business executives
• 1 experience measure to track for all lines of business
• Establish experience score goals for all lines of business
• 1 experience measure that is related to your ROI measures
• Quickly see impact of UX investment with the executive dashboard
Product managers
• 1 experience measure to track with each major product iteration
• Establish experience score goals for your product
• Helps determine product experience investment decisions with a consistent
view of investment and value
• 1 experience measure that is related to the success measures of the product
What do Universal Measures mean for…

54. UX Strategist / Leaders
• Provides a consistent way to measure UX across lines of business
• 1 experience measure to track for all lines of business
• Establish experience score goals for all lines of business
• 1 experience measure that is related to your ROI measures
• Quickly show impact of UX investment and tell your UX maturity story
UX Research and Designer
• Provides a consistent way to show research results across lines of business
• 1 experience measure to track for all lines of business
• Helps determine investment decisions based on research findings
• Support UX debt tracking technique to increase adoption of research
What do Universal Measures mean for…

55. Development Teams
• Quickly determine what UX research is a must-act-upon find
• 1 experience measure to track and try to achieve
• Consistent way to approach UX
• Supports efforts to establish a consistent way to incorporate UX into
products
Digital Analytics Teams
• 1 experience measure to track for all lines of business
• Establish experience score goals for all lines of business
• Correlate your experience measure to your analytics measures
• Identify experience measure as a leading indicator for analytics measures
• Experience cost savings prior to large scale A/B testing
What do Universal Measures mean for…

56. eXperience Score
• Reduces the fuzziness out of the abstract concept of experience
• Defines what experience means for your product(s) and service(s)
• Provides 1 score for experience that your organization can track
The Story
• Strategy to grow experience practice within your organization
Executive Dashboard & Product Scorecard(s)
• Makes experience research actionable
Universal Measures Summary

57. Q&A Session
Type your questions into the chat panel

58. THANK YOU
FOR JOINING TODAY’S WEBINAR!
Discover our upcoming webinars: userzoom.com/webinar
Connect with Dr. Andrea Peer: https://www.linkedin.com/in/andreapeer/
marketing@userzoom.com US HQ: 866-599-1550