Tractionconf.io 2017.

1. How to Optimize Your Product
and Business Using Analytics
Dan Olsen May 31, 2017

2. My Background
Educa/on
n Engineering background
n Stanford MBA
n UX design and coding
Experience
n Managed submarine design
n Product Management leader at Intuit
n CEO & Cofounder, TechCrunch award winner YourVersion
n PM consultant: Box, Facebook, MicrosoK, Medallia
n Founder: Lean Product & Lean UX Meetup Silicon Valley
TwiMer: @danolsen
My slides: hMp://dan-olsen.com Copyright © 2017 @danolsen

3. The 3 Phases of Your Product
Copyright © 2017 @danolsen
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
BEFORE your
product is live
Phase 1
AFTER your
product is live
Phase 2
# of Customers
Time
0
lots
Phase 3

4. The 3 Phases of Your Product
BEFORE
Product-Market Fit
AFTER
Product-Market Fit
Growth
Mainly
Quan/ta/ve
BEFORE your
product is live
Phase 1
Test hypotheses,
gain confidence
before building
Mainly
Qualita/ve
AFTER your
product is live
Phase 2
Improve product-
market fit
Qualita/ve &
Quan/ta/ve
Goal:
Methods:
Phase 3
What phase are you in?

5. n Book giveaway on TwiMer
n Tweet: include @danolsen
n Hashtags
#leanstartup
#prodmgmt
#growth
n hMp:///ny.cc/LPP
That’s Why I Wrote
Copyright © 2017 @danolsen

6. What is
Product-Market
Fit?

7. The Product-Market Fit Pyramid

8. The Product-Market Fit Pyramid

9. The Product-Market Fit Pyramid

10. The Product-Market Fit Pyramid

11. The Lean Product Process

12. The Lean Product Process

13. The Lean Product Process

14. The Lean Product Process

15. The Lean Product Process

16. The Lean Product Process

17. The Lean Product Process

18. The Lean Product Process

19. Valida/ng New vs. Exis/ng Products
New Product
Qualita/ve interviews
Exis0ng Product
Quan/ta/ve data
Oprah Spock

20. Qualita/ve Complements Quan/ta/ve
Copyright © 2017 Olsen Solu/ons
Qual
Why?
Quant
What?
How many?

21. A Playbook
For Achieving
Product-
Market Fit

23. Iden/fy highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2017 @danolsen
Lean Product Analy/cs Process
Iden/fy What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Poten/al
Global
Level
Metric
Level
Select
Top Metric
Learn
& Iterate

24. Valuable to Have a Holis/c Analy/cs
Framework: Dave McClure’s AARRR!

25. Focus on right metric at right /me
Iden/fy your
“Metric that MaMers Most”
(MTMM)
Copyright © 2017 @danolsen

26. What is your
“Metric that MaMers Most”?

27. Copyright © 2017 @danolsen
What’s the right order to op/mize?
acquisi/on
conversion
reten/on

28. Conversion:
What % of the water (prospec/ve
customers) do you get into the bucket?
prospective
customers
customers

29. What great Conversion looks like

30. Acquisi0on:
How much water (prospec/ve customers)
are you spraying?
prospective
customers
customers

31. What great Acquisi0on looks like

32. Reten0on:
What % of the water (customers) do you
keep in the bucket over /me?

33. What great Reten0on looks like

34. If you could only track 1 metric to
measure your Product-Market Fit,
which metric would it be?
Copyright © 2017 @danolsen

35. Copyright © 2017 @danolsen
Net Promoter Score (NPS)
Key follow-up ques/ons:
• Why did you give the score you did?
• What do we need to do to improve?

36. Copyright © 2017 @danolsen
n “How would you feel if you could no longer use
Product X?”
n Very disappointed
n Somewhat disappointed
n Not disappointed
n General guideline: 40% or more
“very disappointed” = product-market fit
Sean Ellis’ Product-Market Fit Ques/on

37. Research Methods
Courtesy of Christian Rohrer, http://xdstrategy.com

38. Reten/on Rate
n Reten/on rate tracks what % of your
customers are s/ll ac/ve over /me
~80%
never use
app again
Curve
eventually
flattens out

39. Cohort Analysis
Copyright © 2017 @danolsen

40. Cohort Analysis: Data
Copyright © 2017 @danolsen

41. Improving Reten/on Rate Over Time=
Increasing Product-Market Fit

42. Improving Reten/on Rate Over Time=
Increasing Product-Market Fit

43. Improving Reten/on Rate Over Time=
Increasing Product-Market Fit

45. Optimizing Revenue
Copyright © 2017 @danolsen

46. Profit = Revenue - Cost
Unique Visitors x Ad Revenue per Visitor
Impressions/Visitor x Effec/ve CPM / 1000
Visits/Visitor x Pageviews/Visit x Impressions/PV
New Visitors + Returning Visitors
Invited Visitors + Uninvited Visitors
# of Users Sending Invites x Invites Sent/User x Invite Conversion Rate
Define the Equa/on of your Business
“Peeling the Onion”
Adver/sing Business Model:
Copyright © 2017 @danolsen

47. Copyright © 2017 @danolsen
( SEO Visitors + SEM Visitors + Viral Visitors ) x Trial Conversion Rate
Paying Users x Revenue per Paying User
New Paying Users + Repeat Paying Users
Previous Paying Users x ( 1 – Cancella/on Rate )
Trial Users x Conv Rate
Profit = Revenue - Cost
Equa/on of your Business:
Subscrip/on Business Model

48. How to Track Your Metrics
n Track each metric as daily /me series
n Create ra/os from primary metrics: X / Y
n Example: How good is your registra/on page?
n Okay: # of registered users per day
n BeMer: registra/on conversion rate =
# registered users / # uniques to reg page
Date
Unique
Visitors
Page
views
Ad
Revenue
New User
Sign-ups
…
4/24/08 10,100 29,600 25 490
4/25/08 10,500 27,100 24 480
…
Copyright © 2017 @danolsen

49. Registra/on Page Conversion Rate
Daily Signup Page Yield vs. Time
New Registered Users divided by Unique Visitors to Signup Page
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0
DailySignupPageYield
Changed
messaging
Added questions
to signup page
Started requiring
registration
Copyright © 2017 @danolsen
Registration Page Conversion Rate vs. Time
RegistrationPageConversionRate

51. View Each Metric as a Gauge
Copyright © 2017 @danolsen
Minimum
Possible
Value
Maximum
Possible
Value
Current
Value

52. Copyright © 2017 @danolsen
Priori/zing Product Ideas by ROI
Investment (developer-weeks)
Return (Value Created)
Idea C
Idea B
Idea D
Idea A
Idea F
1
1
2 3 4
2
3
4
?

53. Iden/fying the “Cri/cal Few” Metrics
n What is the upside poten/al of each metric?
n How many resources will it take to “move the needle”?
n Developer-days, /me, money
n How much will the needle move? Revenue impact?
n Which metrics have highest ROI opportuni/es?
Return
Investment
Return
Investment
Return
Investment
Metric A
Good ROI
Metric B
Bad ROI
Metric C
Great ROI
Copyright © 2017 @danolsen

54. Case Study
Copyright © 2017 @danolsen

55. Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7/02
10/14/02
10/21/02
10/28/02
11/4/02
11/11/02
11/18/02
11/25/02
12/2/02
12/9/02
12/16/02
12/23/02
12/30/02
1/6/03
1/13/03
1/20/03
AbandonmentRate(7DayMovingAverage)
Steps 1-2
Copyright © 2017 @danolsen
Case Study: Account Signup Process Redesign

56. Conversion Rate
Copyright © 2017 @danolsen
Users
exposed to
ability to
take ac/on
Users that
took ac/on
100%
x%
n Can track many different conversion rates for
many different ac/ons

57. Copyright © 2017 @danolsen
Analyzed Drop-Off at Each Major Sec/on
100%
62.3%
58.8%
50.9%
34.4%
32.7%
0%
20%
40%
60%
80%
100%
% of Users
Sign in /
Registra/on
Account
Type
Cash vs.
Margin
5 Partner
Pages
3 Partner
Pages
Focus on
biggest
drop

58. Copyright © 2017 @danolsen
Open
Account
Sign in
Account
Selec/on
Register
56%
44%
Forget
Password
Registra/on
Process
45% drop off
(20% of total)
36% overall
drop off for
this step
70%
(32% of Total)
17% drop off
(10% of total)
20% drop off
(6% of total)
30%
(14% of Total)
80%
(26% of Total)
55%
(24% of Total)
64%
of Total
Analysis of Sign In/Registra/on Flow
Change
Password
83%
(46% of Total)

59. Abandonment Rate (7 Day Moving Average)
0%
10%
20%
30%
40%
50%
60%
70%
80%
10/7/02
10/14/02
10/21/02
10/28/02
11/4/02
11/11/02
11/18/02
11/25/02
12/2/02
12/9/02
12/16/02
12/23/02
12/30/02
1/6/03
1/13/03
1/20/03
AbandonmentRate(7DayMovingAverage)
Steps 1-2
Copyright © 2017 @danolsen
Redesigned User Flow Improved
Registra/on Conversion Rate
37% improvement
in conversion rate
Released
New Design

60. Google Analy/cs
Goal Funnel Visualiza/on
Copyright © 2017 @danolsen

61. Case Study
Copyright © 2017 @danolsen

62. • Which metric has highest ROI opportunity?
Case Study:
Op/mizing Friendster’s Viral Loop
Active
Users
Prospective
Users
Invite Click
Succeed
Invite
click-through rate
Conversion
rate
Don’t
Click
Fail
Invites per
sender
% of users
sending
invites
• Mul/plied together, these metrics determine your viral ra/o
Users
% of users
who are
active
= 15%
= 2.3
= 85%
Registration
Process
Copyright © 2017 @danolsen

63. The Upside Poten/al of a Metric
0
100%
0
100%
0
?
Registra/on
Process Yield
% of users sending
invita/ons
Avg # of invites
sent per sender
2.3
85%
15%
Max possible
improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2017 @danolsen

64. Is anyone feeling a
sense of déjà vu
right now?

65. The Upside Poten/al of a Metric
0
100%
0
100%
0
?
Registra/on
Process Yield
% of users sending
invita/ons
Avg # of invites
sent per sender
2.3
85%
15%
Max possible
improvement
0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?%
Copyright © 2017 @danolsen
Metric B
Bad ROI
Metric A
Good ROI
Metric C
Great ROI

66. Okay, so how can we improve the metric?
n How do we increase the average number of
invites being sent out per sender?
n For each idea:
n What’s the expected benefit? (how much will it
improve the metric?)
n What’s the expected cost? (how many engineer-
days will it take?)
n You want to iden/fy highest ROI idea
Copyright © 2017 @danolsen

67. AKer Launching Address Book Importer…
Copyright © 2017 @danolsen

68. Copyright © 2017 @danolsen
A/B Tes/ng

69. Profitability, anyone?

70. Profitability, anyone?
To be profitable, you want:
LTV – CAC > 0
or
LTV / CAC > 3
Two key metrics:
• Customer Life/me Value (LTV)
• Customer Acquisi/on Cost (CAC)

71. Life/me Value (LTV)
n Life/me value of a customer = how much value
your average customer will generate
n LTV = ARPU x Avg Customer Life/me x Gross Margin
n ARPU (Avg Revenue / User) = Total Revenue / # of Users
n Average Customer Life/me
n How long your average customer generates revenue
n Equals 1 / churn rate (5% monthly churn = avg life 20 months)
n Gross Margin: the % of revenues leK over aKer
subtrac/ng the cost of providing the product/service
Copyright © 2017 @danolsen
Note: for simplicity, this LTV equa/on ignores the “cost of capital”

72. Customer Acquisi/on Cost (CAC)
n CAC is the average cost for you to obtain a
revenue-genera/ng customer
n Takes into account:
n Your cost of acquiring a prospect (CPA)
n Your conversion rate for conver/ng prospects
to revenue-genera/ng customers
n CPA varies by channel; diminishing returns
n CAC=Cost per Acquisi/on / Conversion Rate
Copyright © 2017 @danolsen

73. What You Want to See Over Time
Copyright © 2017 @danolsen
n LTV increasing as you improve your value
proposi/on, customer reten/on, & pricing
n CAC decreasing as you op/mize your marke/ng:
segments, channels, messaging

74. Ra/o of LTV to CAC:
Real data from HubSpot
Copyright © 2017 @danolsen

75. Iden/fy highest
ROI idea
Design and
Implement
Analyze How
the Metric
Changes
Brainstorm
Ideas to
Improve Metric
Copyright © 2017 @danolsen
Lean Product Analy/cs Process
Iden/fy What
Your Metrics Are
Measure Metrics
Baseline Values
Evaluate Metrics
Upside Poten/al
Metric
Level
Select
Top Metric
Learn
& Iterate
Global
Level

76. In general, best to op/mize in this order
1st: Product-market fit = reten/on rate
2nd: Conversion rate
Then, depending on which is most important:
Acquisi/on: CAC* and # of prospects
*Referral: virality (if free users are cri/cal)
Revenue: LTV
Eventually: profitability
Copyright © 2017 @danolsen

77. Questions?
@danolsen
http://dan-olsen.com
dan@olsensolutions.com
Book: http://tiny.cc/LPP
Copies available to sign
I’ll contact book
winners on Twitter