The Art of Metrics

George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
The Art of Metrics
notes about commonly missed aspects of
designing an analytics strategy

george@
useful.io
we build software products using Meteor.js

“Analytics” does not refer to a single business function or purpose.
It’s a multi-purpose toolkit for decision making and monitoring.
operations
ensure your business operates smoothly
troubleshooting / optimizing
why things happen? how can I improve?
predicting the future / investor pitch
will my start-up grow to…?

• Agenda:
• Part A: things people forget over and over
• Part B & C: things people avoid over and over

Part A: Tips on picking
good metrics

Good metrics depend on
your growth stage
Stage
EMPATHY
I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS
I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY
I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE
The users and features fuel growth organically
and artificially.
SCALE
I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
Gate
Thefivestages
Source: Lean Analytics (by A.Croll & B. Yoskovitz) highly recommended reading

Good metrics depend on
what you do
Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce
SaaS Media
Mobile 
app
User-gen 
content
2-sided 
market
Interviews; qualitative results; quantitative scoring; surveys
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)
A more information-rich version is available at: bit.ly/BigLeanTable
Source: Lean Analytics (by A.Croll & B. Yoskovitz) highly recommended reading

Good metrics are actionable,
auditable, accessible
• Actionable: need to result to an action
• Auditable: can be replicated, accuracy can be
veriﬁed
• Accessible: can be measured, everyone
understands what they represent, everyone has
access
source: Eric Rees’ (The Lean Startup)

Good metrics are “mindful”
• Qualitative versus quantitative metrics discover a story (*) vs
prove/conﬁrm a story
• Vanity versus actionable metrics feel good vs changing your
behavior by helping you pick a course of action.
• Exploratory versus reporting metrics speculative insights vs
day-to-day operations health status.
• Leading versus lagging metrics predict the future vs explain
the past.
• Correlated versus causal metrics two metrics change
together(**) vs one metric causes another metric to change.

Good metrics: are rates
0
50
100
150
200
April May June July August
Active users (x100)
0
50
100
150
200
0
500
1000
1500
2000

Good metrics: are not only
about growth
Eric Rees’ (The Lean Startup)
“Sustainable growth is a requirement for any business. […]
New customers come from the actions of past customers”
The three growth engines:
Virality
Rate of growing
Stickiness
Rate of churn (compound)
Price
Rate of expenditure

Part B: Churn rate

Churn
• understand how good you are at retaining
customers
• identify what actions result in a higher retention rate

Churn
• users who abandoned (over a speciﬁc time frame)
• there’s more than one types of churn

–source: Localitics: http://info.localytics.com/blog/how-to-keep-your-app-users-with-the-3x3-rule
“58% of app users churn in the ﬁrst 30 days
after they download your app, on average. And,
over the ﬁrst three months, 75% of app users
will churn.”

Churn: an oversimplified
formula
https://www.shopify.com/technology/4018382-defining-churn-rate-no-really-this-actually-requires-an-entire-blog-post
have you spotted the flaw?
(number of churns during period)
(number of customers at beginning of period)

Churn in practice…
https://www.shopify.com/technology/4018382-deﬁning-churn-rate-no-really-this-actually-requires-an-entire-blog-post
http://christophjanz.blogspot.de/2012/05/know-your-user-cohorts.html

Predicting Churn
compounding 
depends on user replacement  
5% a month could be between…  
46% - 80%… per year!

Measuring Churn
• New customers will have a higher churn rate than
mature customers
• Measuring Churn will very likely force you to do
more ﬁne-grained segmentation.
• Second reminder: make sure you factor-in user
acquisition… by modeling / calculating separately
http://sixteenventures.com/saas-churn-rate

“Churn will grow because your product market fit is
just now improving”
[as a result]
“Churn plagues early stage companies with volatile
product market fit, so assume the worst!”
source: Nikos Moraitakis: https://medium.com/@moraitakis/early-stage-saas-financial-planning-1f9e848f63ef#.7084b188y

–source: Bessemer Venture Partners via http://sixteenventures.com/saas-churn-rate
“[…] an acceptable [SaaS] churn rate is in the 5
– 7% range ANNUALLY, ”

–source: Fred Wilson (http://avc.com/2011/07/301010/)
“[Typically] 30% of the registered users/
downloads will use the service each month
10% of the registered users/downloads will use
the service each day
the max number of concurrent users of a real-
time service will be 10% of the number of daily
users[…]”
There is no single truth

Part C:
Customer Acquisition Costs
(CAC)

–Anonymous
“Keeping a customer is [generally] cheaper
than getting a new one.”

Customer Acquisition Costs
(CAC)
• The actual cost of growth (almost…)
• Crash course (*):
• Average Revenue Per User (ARPU): Revenues / Users
• Lifetime Value (LTV): ARPU * Lifetime
• Customer Acquisition Cost (CAC):  
Total monthly acq costs / new monthly customers

–Bill Clark, Micro Ventures (http://microventures.com/education/metrics-every-startup-should-track)
“you have to take every dollar you spend on
reaching potential customers—from marketing
to public relations and beyond—and divide it
by the number of customers you actually
acquire in a given period”

CAC: making or breaking?
Are We Bleeding? CAC > LTV
Are We Happy(*)? 3 x CAC < LTV

• CAC(s) are not uniform; put effort into creating a
detailed model
• CAC will most likely grow* (but beware channel
capacity and growth rate ceilings)

Breaking down CAC
source: Rob Moffat https://medium.com/venture-capital-growth-hacking/understanding-customer-acquisition-costs-74aec7538b4d

CAC considerations
• Focus on CPA (acquisition) not CPV (visit);
conversion rate varies dramatically per channel
• Use it as a driver to ask questions like:
“What can I do to grow volume of acquisitions through my free
channels?”
“What is likely to happen to CPA in my paid channels as I grow
spend? What limits will I hit?”
“Are there other marketing channels I should be looking at?”
“Start layering in paid acquisition, starting with the cheapest channels”
source: https://medium.com/venture-capital-growth-hacking/understanding-customer-acquisition-costs-74aec7538b4d

Recommended further reading
(topics not covered in this presentation)
• On ﬁnancial planning: “https://medium.com/@moraitakis/early-stage-saas-
ﬁnancial-planning-1f9e848f63ef#.7084b188y"  
— by Nikos Moraitakis
• On market sizing: “http://www.bothsidesofthetable.com/2009/06/16/sorry-guys-its-
the-size-of-the-wave-not-the-motion-of-the-ocean/“  
— by Mark Suster
• On growth: “http://www.paulgraham.com/growth.html”  
— by Paul Graham
• On the tech implementation of analytics: “https://segment.com/academy/“ &
“http://info.localytics.com/blog"

Front-load your analytics strategy.
It will pay off.

Thank you
george@useful.io
george@voulgaris.net
@gevou

The Art of Metrics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to The Art of Metrics

Similar to The Art of Metrics (20)

Recently uploaded

Recently uploaded (20)

The Art of Metrics