How to define design metrics to measure its impact in a meaningful way

1. Design
Metrics

2. Define
a problem
Create
a solution
Measure
the impact
Collaborate with the brands & business functions
Design process

3. Finding the balance — Reconciling user and business goals

4. Finding design metrics
1st step
What does success look like for
our users?
— Start by identifying the outcome that users want to achieve
with our product.
— Identify the main value proposition of our product or service.

5. Finding design metrics
2nd step
What is the main action that a
user has to take to extract value
from our product?
— Identify key product interactions

6. Finding business metrics
1st step
What does success look like for
our business?
Two basic levers:
— Increasing revenue (disproportionate to costs)
— Decreasing cost (disproportionate to revenue)
5 categories of non-
fi
nancial business bene
fi
ts:
— Organizational bene
fi
ts (working more effectively
and ef
fi
ciently)
— Cultural bene
fi
ts (improving internal culture)
— Talent bene
fi
ts (attracting or retaining more talent)
— Data bene
fi
ts (having a better quality of data)
— Brand bene
fi
ts (being more attractive to customers,
partners, and talent)

7. Criteria for good metrics (as de
fi
ned by Croll & Joskowitz in Lean Analytics)
Understandable
You should be able to remember, understand and discuss it.
Comparable
You can compare it to other time periods, groups of users, or
competitors (e.g., "1% increase in sales conversion over the last week"
communicates more than saying "we have 5% sales conversion")
Actionable
You know how you need to change your design based on it.
Normalized (i.e., ratio or rate)
— Is actionable (e.g., "2% conversion rate (20 sign-ups from 1000
visitors)" gives your more information to act on than saying that you
have "20 new sign-ups”)
— Compares factors that are opposed (e.g., ratio of paying vs.
freemium user helps understand if we have a good balance of paying
customers).
Focused on behavior (speci
fi
c for design metrics)
It's clear what interactions are tied to desired user behaviors and
results
Good
metric
Understandable
Comparable
Actionable
Normalised
Behavioural

8. Awesome design metrics
OK metric:
"new events created per week" (this allows us to compare
our progress over time)
Good metric:
"new events created per week per user" (starts measuring
user behavior and value created)
Great metric:
"% of users who create a new task, per week" (compares
our total user base with those who are taking action)
Awesome metric:
"% of users who create 3+ daily tasks, per week" (we have a
speci
fi
c amount of action user needs to take to extract
value from a product)
Example:
Google calendar
“new events created”

9. 9
NPS formula
0 1 2 3 4 5 6 7 8 9 10
Not at all
likely
Extremely
likely
Sample distribution: Average = 6 NPS = -100

10. 10
NPS formula
0 1 2 3 4 5 6 7 8 9 10
Not at all
likely
Extremely
likely
Sample distribution: Average = 8 NPS = 0

11. 11
NPS formula
0 1 2 3 4 5 6 7 8 9 10
Not at all
likely
Extremely
likely
Sample distribution: Average = 9 NPS = 100

12. 12
An eleven-point scale pretends noise is science
Are you
fi
nding this session interesting?
Yes Not sure No
3 2 1

13. 13
An eleven-point scale pretends noise is science
How are you
fi
nding this session?
Very
interesting
Mildly
interesting
Not sure
Mildly not
interesting
Very not
interesting
4 3 2
5 1

14. 14
An eleven-point scale pretends noise is science
How are you
fi
nding this session?
Very
interesting
6 5 Not sure 3 2
Very not
interesting
4 3 2
5 1
6
7

15. 15
An eleven-point scale pretends noise is science
How likely are you to recommend [COMPANY] to a friend or colleague?
0 1 2 3 4 5 6 7 8 9 10
Not at all
likely
Extremely
likely

16. 16
The best questions are about past behaviour, not future behaviour.
Will you try to live a healthy lifestyle?
Are you going to give up sugar?
Will you purchase this product?
Actual behaviour vs. Prediction of behaviour

17. 17
Is NPS really about loyalty and growth?
In the last 6 weeks, have you referred us to a
friend or colleague?
We could ask a different question:

18. 18
Is NPS really about loyalty and growth?
In the last 6 weeks, have you referred us to a
friend or colleague?
We could ask a different question:
Were you referred to us by a friend or
colleague?
+

19. 19
Netflix asked questions about past behaviour, not prediction of future
behaviour.
In the last 6 weeks, have you referred us to a
friend or colleague?
Were you referred to us by a friend or
colleague?
Yes No
Yes No

20. 20
The real value is in the follow up question
What did we do well?
To promoters
What could we improve?
To detractors
Why answers

21. 21
We can’t reduce user
experience to a single number
Customer experience is the sum total of all the interactions
our customers have with our products, sites, employees, and
the brand. Every sequence of interactions will differ for
every customer.

22. 22
The combination of what is
going on and why it’s happening
is the most powerful