A Primer in Startup Analytics

A Primer in Startup Analytics
Concepts and best practices for data-driven decision-making
@gevou
George Voulgaris

George Voulgaris, Ph.D. - A Primer in Startup Analytics
Podgorica, Nov 7th, 2015
Instead of an
introduction

Analytics is about…

You may try this:
• Build the product.
• Throw in analytics.
• Pull up your analytics. Get overwhelmed. Go back to building
product.
Validated Learning
1

or maybe this:
• Build an experimental product.
• Measure how people respond to your experiment.
• Learn whether your experiment worked or not.
Validated Learning
1

Validated Learning
• (*) especially in circumstances of extreme uncertainty, e.g.
startups
• Business plans used to be a guess-game based on market
research (*), a large number of assumptions (including the
existence of interest for my product), and a wild guess of the
proﬁts of my company in 18 months (!!!!)
1

Decisions
pop quiz:
what is the most common practice when people are faced
with dilemmas and decisions in startups?
2

Decisions
• As a product owner / entrepreneur you have to face many
decisions roadmap, features, marketing, focus… everything.
• Decisions are draining and exhausting
• It is often better to make a decision than to make the right
decision
2

Decisions
21.Evaluate your business assumptions 
Gather data and adapt your early-stage roadmap while trying to achieve
product-market ﬁt.
2.Continuous validation; Optimize all areas of your business 
In a dynamic business environment the need for monitoring and re-
evaluation is essential even after achieving product-market.
3.Build you business growth case 
Essentially a by-product of all the above; build a data-backed case that you
have a business with high-growth potential. (read: investor pitch and relations)

Decisions
2We will talk about
1.Evaluate your business assumptions 
Gather data and adapt your early-stage roadmap while trying to achieve
product-market ﬁt.
2.Continuous validation; Optimize all areas of your business 
In a dynamic business environment the need for monitoring and re-
evaluation is essential even after achieving product-market.
3.Build you business growth case 
Essentially a by-product of all the above; build a data-backed case that you
have a business with high-growth potential. (read: investor pitch and relations)

• It’s easy to start measuring stuff and design pretty charts.
• Decide what matters the most
1Decide what is it want
to learn

What matters most to you depends on the product / business
maturity stage and the user engagement stage.
There are few frameworks that can help.
Decide what is it want
to learn
1

How do your users become aware of you?
SEO, SEM, widgets, email, PR, campaigns, blogs ...
Do drive-by visitors subscribe, use, etc?  
Features, design, tone, compensation, affirmation ...
Does a one-time user become engaged?  
Notifications, alerts, reminders, emails, updates...
Do you make money from user activity?  
Transactions, clicks, subscriptions, DLC, analytics...
Do users promote your product? 
Email, widgets, campaigns, likes, RTs, affiliates...
Acquisition
Activation
Retention
Revenue
Referral
Dave McClure’s Pirate Metrics
Source: http://www.slideshare.net/dmc500hats/startup-metrics-for-pirates-long-version
to learn
1

Lean Analytics  
(by A.Croll & B. Yoskovitz recommended reading)
Stage
EMPATHY
I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS
I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY
I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE
The users and features fuel growth organically
and artificially.
SCALE
I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
Gate
Thefivestages
to learn
1

Lean Analytics  
(by A.Croll & B. Yoskovitz recommended reading)
Stage
EMPATHY
I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS
I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY
I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE
The users and features fuel growth organically
and artificially.
SCALE
I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
Gate
Thefivestages
to learn
1
Be Honest!

Eric Rees’ (The Lean Startup)
Sustainable growth is a requirement for any business.
“New customers come from the actions of past customers”
The three growth engines:
Virality
Make people invite others as a necessary side effect
Rate of growing
Stickiness
Keep people coming back / addictive / network effects / lock-in
Rate of churn (compound)
Price
Reinvest revenue of past to get new customers.
Rate of expenditure
Both resonate with…
to learn
1

decide what you need to
learn first
2

Lean Analytics  
(Adapted version from the book by A.Croll & B.
Yoskovitz recommended reading)
Pick a KPI
Draw a line in
the sand
Identify ways
to improve this
Make a guess Find relevant
patterns in
Design
experiments
Implement
what seems to
Find
correlations
data
available
no data
Analyze the
behavior of the
Line-in-the-sand
was crossed
Line-in-the-sand
was NOT crossed
Give up or
pivot
Go back to
Discovery.  
Try again
Identify business
objectives &
decision processes
learn first
2

learn first
2Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce
SaaS Media
Mobile 
app
User-gen 
content
2-sided 
market
Interviews; qualitative results; quantitative scoring; surveys
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)
A considerably more information-rich version is available at: bit.ly/BigLeanTable

hail
web site
CTAs signup
new free
account
create a new
App
upgrade
Inbound
web, useful.io, universe…
price lists, features,
examples, CTAs
learn first
2

Empathy
hail
web site
CTAs signup
new free
account
create a new
App
upgrade
Inbound
web, useful.io, universe…
price lists, features,
examples, CTAs
Acquisition Activation
Stickiness*
Revenue
learn first
2

learn first
2• Deciding what to measure has no value… unless you also decide
on some success/failure criteria & threshold.
• Deciding on the actions to be followed based on the outcome
and iterating/ﬁne-tuning is more important than picking the
correct threshold upfront
• You will NOT get it right from the beginning
• Guess, iterate, use other people wisdom/experience (*with a
generous pinch of salt)

figure out the appropriate
primitive metrics
3

Pick metrics that hurt
• Metrics need to change a behavior
4

Draw lines in the sand
• Deciding what to measure has no value… unless you also
decide on some success/failure criteria & threshold.
• You will NOT get it right from the beginning
• Guess, iterate, use other people wisdom/experience (*with a
generous pinch of salt)
5

NOTE!!!
Being able to monitor many things will tempt you to try to  
ﬁx everything at once.
TRY NOT TO
You should focus in optimizing one parameter (KPI) in each
experiment cycle to reduce complexity and to support
causality*.

Few things to keep in
mind

Good KPIs are:
• Actionable
• Accessible
• Auditable
Good KPIs are…
1

Good KPIs are:
• rates
Good KPIs are…
2

Eric Rees’ (The Lean Startup)
Sustainable growth is a requirement for any business.
“New customers come from the actions of past customers”
The three growth engines:
Virality
Make people invite others as a necessary side effect
Rate of growing
Stickiness
Keep people coming back / addictive / network effects / lock-in
Rate of churn (compound)
Price
Reinvest revenue of past to get new customers.
Rate of expenditure
Both resonate with…
Good KPIs are…
2

Some terms to familiarise yourself with:
• Qualitative versus quantitative metrics Qualitative metrics are unstructured, anecdotal, revealing, and
hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less
insight.
• Vanity versus actionable metrics Vanity metrics might make you feel good, but they don’t change how
you act. Actionable metrics change your behavior by helping you pick a course of action.
• Exploratory versus reporting metrics Exploratory metrics are speculative and try to ﬁnd unknown
insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-
day operations.
• Leading versus lagging metrics Leading metrics give you a predictive understanding of the future; lag-
ging metrics explain the past. Leading metrics are better because you still have time to act on them.
• Correlated versus causal metrics If two metrics change together, they’re correlated, but if one metric
causes another metric to change, they’re causal. If you ﬁnd a causal relationship between something you
want (like revenue) and something you can control (like which ad you show), then you can change the future.
Terms
2

Some Revenue related metrics to familiarise with:
• Average Revenue Per User (ARPU): Revenues / Users
• Churn: users abandoned (month)
• Lifetime: 1 / churn
• Lifetime Value (LTV): ARPU * Lifetime
• Customer Acquisition Cost (CAC):  
Total monthly acq costs / new monthly customers  
(e.g. 3 x CAC < LTV)
Terms
2

Churn
• understand how good you are at retaining customers
• identify what actions result in a higher retention rate
3

Churn
• There are more than “churns” you can measure
• The right one depends on your business
• Keeping a customer is generally cheaper than getting a new
one
3

Cost of Customer
• var areWeBleeding = (LTV > CAC) ? true : false;
• Customer Acquisition Costs are not uniform.
• Put effort into creating a breakdown of different channels
4

Cost of Customer
4
source: https://medium.com/venture-capital-growth-hacking/understanding-customer-acquisition-costs-74aec7538b4d

Take-aways
• Don’t be afraid to make mistakes…
• …as long as you plan ahead, document and…
• …Iterate
• Pick KPIs that hurt (not ones that will make you look pretty)
• Discipline pays-off:  
Respect the time-limits, decision processes, and the agreed actions. Use iterations to change
the parameters. Prefer to improve the process over tweaking a single experiment.
• Don’t try to optimize many things at once; look-out for repercussions/dependencies if you do.
• Focus on the OMTT; use complementary intelligence to support corrective actions.

Perfect!

George @ Voulgaris .net 
 
@gevou

A Primer in Startup Analytics

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Viewers also liked

Viewers also liked (20)

Similar to A Primer in Startup Analytics

Similar to A Primer in Startup Analytics (20)

Recently uploaded

Recently uploaded (20)

A Primer in Startup Analytics