Slides Joni Hoadley recently used in his discussion w/ mentees of The Product Mentor. Synopsis: Every good product manager should listen to the voice of their customers. The key is using the right data to inform and influence your product decisions. The Product Mentor is a program designed to pair Product Mentors and Mentees from around the World, across all industries, from start-up to enterprise, guided by the fundamental goals…Better Decisions. Better Products. Better Product People. Throughout the program, each mentor leads a conversation in an area of their expertise that is live streamed and available to both mentee and the broader product community. http://TheProductMentor.com

1. Metrics that matter
Forget the vanity metrics and measure key outcomes

2. Joni Hoadley
@JoniHoadley
www.jonihoadley.com
• 20+ years product management experience
• Coach and consultant for companies looking to bring
customer-focused products to market more quickly
• 12 years as product management leader at Sonos

3. Good
Bad
Ugly
The
The
AndThe

4. Bad
The
vanity
metrics
=

5. What are vanity metrics?
Vanity Metrics definition - A standard for measuring something without
context, vanity metrics are easily manipulated, and do not lead to actions you
can take to grow your business.
Examples - app downloads, page views, time on site, twitter followers,
facebook likes, etc.

6. How to identify vanity metrics
Source - https://www.crazyegg.com/blog/mirror-mirror/

7. Yes, this entire page is useless
Source - https://www.crazyegg.com/blog/mirror-mirror/

8. Good
Bad
Ugly
The
The
AndThe

9. Ugly
AndThe
Flawed
GRAPHS
=

10. The ugly truth about presenting data

11. Where to learn more
• Find out which type of
chart works best at From
Data to Viz
• Select which type of data
you have (chronological,
numeric, categorical) and
this site will guide you
through the best options.

12. Recommended
Reading

13. Good
Bad
Ugly
The
The
AndThe

14. Good
The
measuring
foroutcomes
=

15. Outcome-oriented Metrics
Vanity Metrics Outcome-oriented Metrics
• Raw numbers such as page
views without context
• Don’t represent customer
value
• Don’t provide guidance for
future direction
• Link actions to results
• Focus on delivering
customer value
• Provide insight into the
health of your product and
company
vs.

16. Where to start?
KPIs

17. Start with strategy

18. WOW

19. Measure what matters
Quality, not quantity - Zappos
measures call time instead of call
volume.
They also invented their
Happiness Experience Form
which is a 100-point scale based
on how well reps fulfilled on
various goals.
http://callcriteria.com/kpi-lessons-masters-customer-
service-zappos-case-study/

20. Real world example from Sonos

21. Pirate Metric Framework
A
A
R
R
R
Acquisition - how well are you getting
customers to your site or app?
Activation - are your customers having
a great ‘first run’ experience?
Retention - how often are your
customers coming back?
Referral - are they telling others about
your product?
Revenue - are they paying for your
service?

22. Pirate Metric Framework
How many people are signing up
for a free trial?
What % of people are watching a
video from start to finish?
What % of people are returning to
watch a video again in 30 days?
What % of people are sending
links of videos to others?
What % of people are paying for
monthly subscription?
A
A
R
R
R
Acquisition - how well are you getting
customers to your site or app?
Activation - are your customers having
a great ‘first run’ experience?
Retention - how often are your
customers coming back?
Referral - are they telling others about
your product?
Revenue - are they paying for your
service?

23. 4 steps to measuring for outcomes
1. Identify the key outcomes.
2. Form a hypothesis.
3. Build your experiment.
4. Measure and analyze

24. Experimentation: Building a
culture of continuous learning
Source: Eric Ries, The Lean Startup Methodology

25. Measuring Net Promoter Score
Would you recommend <my product> to a friend?
https://www.netpromoter.com/know/

26. The Problem with NPS
Would you recommend <my product> to a friend?
https://www.netpromoter.com/know/

27. How to derive outcomes
from NPS
If answer = 0-8, follow up by asking:
“What would it take for you to recommend <our product> to someone you know?”
https://www.netpromoter.com/know/

28. Key Takeaways
1. Measure outcomes, not vanity metrics.
2. Make sure your data is correct.
3. Choose the right type of visualization for the
data.
4. Define KPIs based on your company’s strategy
and top-line metric.
5. Experiment with small changes and foster a
culture of continuous learning.

29. Joni Hoadley
@JoniHoadley
www.jonihoadley.com
Thank you!

30. References
• From Data to Viz - https://www.data-to-viz.com/
• Edward Tufte - https://www.edwardtufte.com/tufte/books_vdqi
• Feature Factory by John Cutler - https://hackernoon.com/12-
signs-youre-working-in-a-feature-factory-44a5b938d6a2
• Zappos case study - http://callcriteria.com/kpi-lessons-masters-
customer-service-zappos-case-study/
• Pirate Metrics/Dave McClure - https://www.slideshare.net/
dmc500hats/startup-metrics-for-pirates-long-version
• NPS - https://www.netpromoter.com/know/