As a product manager, it's your job to realize your product’s vision by executing your chosen strategy. It’s also your job to deliver value to the business. Ultimately, these two outcomes are aligned so the temptation is to focus primarily on business metrics. Doing this can cause you to lose focus on the real value you are trying to achieve, in favor of moving the vanity metrics such as launches and time spent. Join Sari Harrison, Product Management Instructor at Product School, as she explains how to use immeasurable success criteria along with your more standard KPIs to deliver products that don't just get used a lot, but deliver real value.

1. Measure the Immeasurable: Beyond Vanity
Metrics
Sari Harrison Hannah Flynn
With: Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio using
your computer's microphone and speakers (VoIP). A headset is
recommended.
Webinar will begin:
11:00 am, PST
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use Telephone"
after joining the webinar and call in using the numbers below.
United States: +1 (914) 614-3221
Access Code: 530-037-997
Audio PIN: Shown after joining the webinar
--OR--
Empowering Product Teams
Webinar Series

2. Revulytics gives any software producers deep and actionable insight into
who is using their software products and how they are being used, and the
out-of-box analytics that enable them to grow incremental revenue, convert
and retain customers, and make decisions about licensing and cloud
transformation strategies.

3. Empowering Product Teams
Webinar Series
Click on the Questions panel to
interact with the presenters
www.productmanagementtoday.com/webinar-series/empowering-product-teams

4. About Sari Harrison
Sari spent more than 20 years in high-tech R&D at Apple and Microsoft. As a software engineer, engineering
manager, product manager, and product management leader, she has delivered half a dozen successful “1.0”
products, along with countless iterations of existing products, in B2B2C and B2C. She is now focused on teaching
product management and product management consulting.
About Hannah Flynn
Hannah went to The University of Chicago, where she majored in Environmental Studies with a concentration in
Economics and Policy. She now works with Aggregage on social media strategy and webinar production for Product
Management Today and B2B Marketing Zone.
Empowering Product Teams
Webinar Series

5. MEASURE THE
IMMEASURABLE
BEYOND VANITY METRICS
SARI HARRISON
HTTPS://MEDIUM.COM/@SARIHARRISON
WWW.SECONDNATUREINNOVATION.COM

6. WE ARE RESPONSIBLE FOR
THE IMPACT OF OUR
PRODUCTS

7. GOAL: SHIFT THE METRICS CONVERSATIONS
“Impact”
Happier
More productive
More free time
Amazing experiences
Revenue
Usage
NPS
“Vanity”
Time spent
AARRR

8. VANITY METRICS ARE
TRAILING INDICATORS!
Adding value? Or extracting value?
Adding value? Or monthly active users?
Adding value? Or referrals?

9. MIND THE GAP
Vision/value
proposition,
strategy
Vanity-based
Metrics, KPIs,
OKRs

10. CLOSE THE GAP
Vision/value proposition,
strategy, goals
Things we measure and
strive forImpact Metrics

11. ABOUT ME
Apple
• Engineer
• Engineering
manager
• DRI
• 92-00
Microsoft
• PM
• Lead PM
• GPM
• 01-14
Apple
• Head of
Apple maps
PM
• 14-18
SecondNatureInnovation
• PM
Consultant
• Innovation
Coach
• PM Trainer
Data-Driven becomes a “thing”

12. APPLE MAPS VISION
Apple ecosystem customers are able to
seamlessly explore and navigate the
world

13. APPLE MAPS
STRATEGY
Beautiful
Safe and relaxing
Supports the Apple ecosystem
Unmatched attention to detail
Fresh and accurate data
Consistent with the Apple brand

14. APPLE MAPS
STRATEGY
Beautiful
Safe and relaxing
Supports the Apple ecosystem
Unmatched attention to detail
Fresh and accurate data
Consistent with the Apple brand

15. APPLE MAPS MEASURABLE METRICS
Usage
• Launches
• Time spent
• Feature usage
• MAU, DAU, WAU
• …
Financial
• Cost
• ~0 revenue
Quality
• Major navigational errors
• Time from report to fix
• Search “wins”
• Autocomplete performance
• …

16. APPLE MAPS
IMMEASURABLES
Most beautiful
Easiest to read and understand
Safest, most relaxing experience
Glanceable
Community friendly (brand)
Most trusted
Proactive

17. Question: should
we aggressively
reroute around
traffic ala Waze?
Answer (typical metrics): yes
Answer: no
Not community friendly
Not safest and most relaxing
experience
Engagement != valuable

18. Answer (typical metrics): no
Answer: yes
Apple ecosystem integration
Least effort
Answer (typical metrics): yes
Answer: no, users did not
understand what was going
to happen

19. POLL QUESTION: DO YOU HAVE A STRATEGY THAT
GOES BEYOND VANITY?
Yes/No/We don’t have a strategy

20. COUNTER EXAMPLE – BING ADS
Prioritization exercise was to estimate revenue…
1. Number of customers that adopt the new format
2. CPC/CPM … average? min? max?
3. How often ad would be displayed
= Total guesswork 🔮
= Gamed
= Wrong 100% of the time

21. POLL: DOES YOUR CURRENT PRIORITIZATION
FRAMEWORK GO BEYOND VANITY?
Yes/no/we don’t prioritize

22. BEYOND VANITY FRAMEWORK
• Types of success metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

23. BEYOND VANITY FRAMEWORK
• Types of metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

24. USAGE
Monthly active users, time spent, launches, page
views, churn, usage, abandonment…
Trailing indicator
Measurable
Product is only one factor

25. FINANCIAL
Revenue, subscriptions, advertising dollars, cost
Trailing indicators of value
Measurable
Product is only one factor

26. QUALITY
Performance, bug counts, reliability
Leading indicator
Measurable
Product is primary factor

27. IMPACT:
The “metrics”
that fill the gap
Unique to your
[business/product/feature] vision
Set the standard. Not leading or
trailing.
Possibly measurable
Product is the only factor

28. BEYOND VANITY FRAMEWORK
• Types of metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

29. ANY METRIC APPLIES TO SOME UNIT OF THE
BUSINESS
Business
• Is our business thriving?
• Are we extracting value
from our customers in
return for the value we
provide?
• Is our business model
working?
• Examples: revenue, sales,
cost per user, pricing
Product
• Is our product adding
value?
• Is it high quality?
• Examples: usage,
engagement, retention,
referral
Feature/Feature Area
• Did we solve the problem
we intended to solve?
• Did we do it well?
• Is it discoverable?
• Examples: usage,
abandonment,
performance

30. BEYOND VANITY FRAMEWORK
• Types of metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

31. CONVERSATIONS
THAT INVOLVE
METRICS
understanding
inspiring ideas
prioritization
hypothesis generation and validation
readiness
context and alignment

32. BEYOND VANITY FRAMEWORK
• Types of success metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

33. CALL TO ACTION
1. CREATE IMPACT METRICS TO COMPLEMENT YOUR EXISTING
USAGE, FINANCIAL, AND QUALITY METRICS
2. BE THOUGHTFUL ABOUT WHICH METRICS YOU USE FOR WHAT
CONVERSATION

34. BEYOND VANITY FRAMEWORK
• Types of success metrics
• Things that metrics apply to
• What we are using them for
• Call to action
• How exactly do we do this?

35. IT ALL STARTS FROM YOUR USER GOALS
Vision and Strategy Goals
Metrics,
Features (or
sub-features)…

36. APPLE MAPS VISION
Apple ecosystem customers are able to
seamlessly explore and navigate the
world

37. APPLE MAPS
STRATEGY
Beautiful
Safe and relaxing
Apple ecosystem integration
Unmatched attention to detail
Fresh and accurate data
Consistent with the Apple brand

38. APPLE MAPS
GOALS
Help the user plan how to get somewhere
Help the user navigate to their chosen destination
Help the user decide where to go
Help the user discover new things
Help the user understand the world around them
User has to do as little work as possible
User feels safe and relaxed while navigating
Developer efficiency
Operational efficiency
Fresh and accurate data

39. APPLE MAPS
USER GOALS
Help the user plan how to get somewhere
Help the user navigate to their chosen destination
Help the user decide where to go
Help the user discover new things
Help the user understand the world around them
User has to do as little work as possible
User feels safe and relaxed while navigating
User trusts our answer
Developer efficiency
Operational efficiency
Fresh and accurate data

40. BING ADS
GOALS
Reduce friction for
advertisers
Make Bing profitable
Help the user discover new
products and services
Ads should not negatively
impact Bing usage metrics

41. BING ADS USER
GOALS
Reduce friction for
advertisers
Make Bing profitable
Help the user discover new
products and services
Ads should not negatively
impact Bing usage metrics

42. BING ADS USER
GOALS
Reduce friction for advertisers
Make Bing profitable
Help the user discover new
products and services
Ads should not slow the user down
in finding relevant search results

43. TURNING
GOALS INTO
METRICS
FB: Meaningful interactions
Apple Maps: Right about proactive
suggestions
Apple Maps: Stress while navigating
Bing Ads: Google advertiser friction
Bing Ads: Time to relevant results
Office for Mac: Feels Mac-like and Office-like
to the users
Mediaroom: Time user spends in interface

44. QUANTITATIVE
MEASURABILITY
FB: Meaningful interactions
Apple Maps: Right about proactive
suggestions
Apple Maps: Stress while navigating
Bing Ads: Google advertiser friction
Bing Ads: Time to relevant results
Office for Mac: Feels Mac-like and Office-like
to users
Mediaroom: Time user spends in interface

45. HOW TO
MEASURE THE
IMMEASURABLE
Survey
Usability interviews
In-app feedback
Focus groups
Heuristic evaluation

46. STOP AND THINK – WHAT’S THE RIGHT
METRIC?
Metrics
• Usage
• Financial
• Quality
• Impact
Things
• Business
• Product
• Feature
Conversations
• Understanding
• Inspiring Ideas
• Prioritization
• Validation
• Readiness
• Context

47. USING IMPACT METRICS DAY TO DAY
All hands Brainstorming
Hypothesis
generation
and validation
Feature
validation
Prioritization

48. USING IMPACT METRICS DAY TO DAY
All hands Brainstorming
Hypothesis
generation
and validation
Feature
validation
Prioritization

49. INNOVATE EVERYWHERE, INCLUDING YOUR METRICS
Are we using the
right metric in this
conversation?
Is there a better
one?
Is there one we
should retire?
How can we
measure it?
How do they impact
each other?
Can we avoid the
negative in addition
to increasing the
positive?
Define
Measure
Analyze
Hypothesize
Adjust

50. Empowering Product Teams
Webinar Series
Q&A
Hannah Flynn
With: Moderated by:
Product Management Instructor, Product School
Linkedin page: /in/sariharrison/
Twitter ID: @sariharrison
Website: www.secondnatureinnovation.com
Sari Harrison
Site editor, Product Management Today
Linkedin page: /in/hannahmflynn
Twitter ID: @prodmgmttoday
Email: hannah@aggregage.com
Website: productmanagementtoday.com
www.productmanagementtoday.com/webinar-series/empowering-product-teams

51. THANK YOU!
WWW.SECONDNATUREINNOVATION.COM