5. Most startups don’t know what they’ll
be when they grow up.
Hotmail
was a
database
company
Flickr
was going to
be an MMO
Twitter
was a
podcasting
company
Autodesk
made
desktop
automation
Paypal
first built for
Palmpilots
Freshbooks
was invoicing
for a web
design firm
Wikipedia
was to be
written by
experts only
Mitel
was a
lawnmower
company
7. Everyone’s idea is
the best right?
People love
this part!
(but that’s not always
a good thing)
This is where
things fall apart.
No data, no
learning.
16. A good metric is:
Understandable
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
Comparative
Comparison is
context.
A ratio or rate
The only way to
measure
change and roll
up the tension
between two
metrics (MPH)
Behavior
changing
If you’re busy
explaining the
data, you won’t
be busy acting
on it.
18. Metrics help you know yourself.
Acquisition
Hybrid
Loyalty
70%
of retailers
20%
of retailers
10%
of retailers
You are
just like
Customers that
buy >1x in 90d
Once
2-2.5
per year
>2.5
per year
Your customers
will buy from you
Then you are
in this mode
1-15%
15-30%
>30%
Low acquisition
cost, high checkout
Increasing return
rates, market share
Loyalty, selection,
inventory size
Focus on
(Thanks to Kevin Hillstrom for this.)
19. Qualitative
Unstructured, anecdotal,
revealing, hard to
aggregate, often too
positive & reassuring.
Warm and fuzzy.
Quantitative
Numbers and stats.
Hard facts, less insight,
easier to analyze; often
sour and disappointing.
Cold and hard.
20. Exploratory
Speculative. Tries to find
unexpected or
interesting insights.
Source of unfair
advantages.
Cool.
Reporting
Predictable. Keeps you
abreast of the normal,
day-to-day operations.
Can be managed by
exception.
Necessary.
21. Rumsfeld on Analytics
(Or rather, Avinash Kaushik channeling Rumsfeld)
Things we
know
don’t
know
we know Are facts which may be wrong and
should be checked against data.
we don’t
know
Are questions we can answer by
reporting, which we should baseline
& automate.
we know
Are intuition which we should
quantify and teach to improve
effectiveness, efficiency.
we don’t
know
Are exploration which is where
unfair advantage and interesting
epiphanies live.
22. MayAprMarFeb
Slicing and dicing data
Jan
0
5,000
Activeusers
Cohort:
Comparison of
similar groups
along a timeline.
(this is the April cohort)
A/B test:
Changing one thing
(i.e. color) and
measuring the
result (i.e. revenue.)
Multivariate
analysis
Changing several
things at once to
see which correlates
with a result.
☀
☁
☀
☁
Segment:
Cross-sectional
comparison of all
people divided by
some attribute (age,
gender, etc.)
☀
☁
24. January February March April May
Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50
Is this company
growing or stagnating?
Cohort 1 2 3 4 5
January
February
March
April
May
$5 $3 $2 $1 $0.5
$6 $4 $2 $1
$7 $6 $5
$8 $7
$9
How about
this one?
25. Cohort 1 2 3 4 5
January
February
March
April
May
Averages
$5 $3 $2 $1 $0.5
$6 $4 $2 $1
$7 $6 $5
$8 $7
$9
$7 $5 $3 $1 $0.5
Look at the
same data
in cohorts
26. Lagging
Historical. Shows you
how you’re doing;
reports the news.
Example: sales.
Explaining the
past.
Leading
Forward-looking.
Number today that
predicts tomorrow;
reports the news.
Example: pipeline.
Predicting the
future.
27. A Facebook user reaching 7 friends within 10 days of signing up
(Chamath Palihapitiya)
If someone comes back to Zynga a day after signing up for a game,
they’ll probably become an engaged, paying user (Nabeel Hyatt)
A Dropbox user who puts at least one file in one folder on one device
(ChenLi Wang)
Twitter user following a certain number of people, and a certain
percentage of those people following the user back (Josh Elman)
A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
Some examples
(From the 2012 Growth Hacking conference. http://growthhackersconference.com/)
30. Correlated
Two variables that are
related (but may be
dependent on
something else.)
Ice cream &
drowning.
Causal
An independent variable
that directly impacts a
dependent one.
Summertime &
drowning.
31. A leading, causal metric
is a superpower.
h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
36. Aunshul Rege of Rutgers University, USA in 2009
Experienced scammers expect a “strike rate” of 1 or 2 replies per 1,000 messages
emailed; they expect to land 2 or 3 “Mugu” (fools) each week.
One scammer boasted “When you get a reply it’s 70% sure you’ll get the money”
“By sending an email that repels all but the most gullible,” says [Microsoft Researcher
Corman] Herley, “the scammer gets the most promising marks to self-select, and tilts
the true to false positive ratio in his favor.”
1000 emails
1-2 responses
1 fool and their money, parted.
Bad language (0.1% conversion)
Gullible (70% conversion)
1000 emails
100 responses
1 fool and their money, parted.
Good language (10% conversion)
Not-gullible (.07% conversion)
This would be horribly
inefficient since
humans are involved.
37. Turns out the word “Nigeria” is the best
way to identify promising prospects.
40. Eric’s three engines of growth
Virality
Make people
invite friends.
How many they
tell, how fast they
tell them.
Price
Spend money to
get customers.
Customers are
worth more than
they cost.
Stickiness
Keep people
coming back.
Approach
Get customers
faster than you
lose them.
Math that
matters
41. Dave’s Pirate Metrics
AARRR
Acquisition
How do your users become aware of you?
SEO, SEM, widgets, email, PR, campaigns, blogs ...
Activation
Do drive-by visitors subscribe, use, etc?
Features, design, tone, compensation, affirmation ...
Retention
Does a one-time user become engaged?
Notifications, alerts, reminders, emails, updates...
Revenue
Do you make money from user activity?
Transactions, clicks, subscriptions, DLC, analytics...
Referral
Do users promote your product?
Email, widgets, campaigns, likes, RTs, affiliates...
42. Stage
EMPATHY
I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS
I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY
I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE
The users and features fuel growth organically
and artificially.
SCALE
I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
Gate
Thefivestages
43. Empathy stage:
Localmind hacks Twitter
Needed to find out if a core assumption—strangers answering
questions—was valid.
Ran Twitter experiment instead of writing code
Asked senders of geolocated Tweets from Times Square random
questions; counted response rate
Conclusion: high enough to proceed
44. Stickiness stage:
qidiq streamlines invites
Survey owner adds recipient to group
Survey owner asks question
Recipient reads survey question
Recipient responds to question
Recipient sees survey results
(Later, if needed…)
Recipient visits site; no password!
Recipient does password recovery
One-time link sent to email
Recipient creates password
Recipient can edit profile, etc.
Survey owner adds recipient to group
Survey owner asks question
Recipient gets invite
Recipient reads survey question
Recipient responds to question
Recipient installs mobile app
Recipient creates account, profile
Recipient sees survey results
Recipient can edit profile, etc.
10-25%RESPONSERATE
70-90%RESPONSERATE
47. (Which means eye
charts like these.)
Customer Acquisition Cost
paid direct search wom
inherent
virality
VISITOR
Freemium/trial offer
Enrollment
User
Disengaged User
Cancel
Freemium
churn
Engaged User
Free user
disengagement
Reactivate
Cancel
Trial abandonment
rate
Invite Others
Paying Customer
Reactivation
rate
Paid
conversion
FORMER USERS
User Lifetime Value
Reactivate
FORMER CUSTOMERS
Customer Lifetime Value
Viral coefficient
Viral rate
Resolution
Support data
Account Cancelled Billing Info Exp.
Paid Churn Rate
Tiering
Capacity Limit
Upselling
rate Upselling
Disengaged DissatisfiedTrial Over
48. Model + Stage = One Metric That Matters.
One Metric
That Matters.
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Thestageyou’reat
54. Moz cuts down on metrics
SaaS-based SEO toolkit in the scale stage. Focused on net adds.
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds up:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds flat:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somehow?
Is customer support falling apart?
Net adds down:
55. Metrics are like squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/
56. Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce
SaaS Media
Mobile
app
User-gen
content
2-sided
market
Interviews; qualitative results; quantitative scoring; surveys
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)
62. Baseline:
5-7% growth a week
“A good growth rate during YC
is 5-7% a week,” he says. “If
you can hit 10% a week you're
doing exceptionally well. If you
can only manage 1%, it's a sign
you haven't yet figured out
what you're doing.” At revenue
stage, measure growth in
revenue. Before that, measure
growth in active users.
Paul Graham, Y Combinator
• Are there enough people who really care
enough to sustain a 5% growth rate?
• Don’t strive for a 5% growth at the expense
of really understanding your customers
and building a meaningful solution
• Once you’re a pre-revenue startup at or
near product/market fit, you should have
5% growth of active users each week
• Once you’re generating revenues, they
should grow at 5% a week
63. Baseline:
10% visitor engagement/day
Fred Wilson’s social ratios
30% of users/month use web or mobile app
10% of users/day use web or mobile app
1% of users/day use it concurrently
64. Baseline:
2-5% monthly churn
• The best SaaS get 1.5% - 3% a month. They have multiple Ph.D’s
on the job.
• Get below a 5% monthly churn rate before you know you’ve got a
business that’s ready to grow (Mark MacLeod) and around 2%
before you really step on the gas (David Skok)
• Last-ditch appeals and reactivation can have a big impact.
Facebook’s “don’t leave” reduces attrition by 7%.
65. Who is worth more?
Today
A
Lifetime:
$200
Roberto Medri, Etsy
B
Lifetime:
$200
Visits
66. Baseline:
Calculating customer lifetime
25%
monthly churn
100/25=4
The average
customer lasts
4 months
5%
monthly churn
100/5=20
The average
customer lasts
20 months
2%
monthly churn
100/2=50
The average
customer lasts
50 months
67. Baseline:
CAC under 1/3 of CLV
• CLV is wrong. CAC Is probably wrong, too.
• Time kills all plans: It’ll take a long time to find
out whether your churn and revenue projections
are right
• Cashflow: You’re basically “loaning” the
customer money between acquisition and CLV.
• It keeps you honest: Limiting yourself to a
CAC of only a third of your CLV will forces you
to verify costs sooner.
Lifetime of 20 mo.
$30/mo. per
customer
$600 CLV
$200 CAC
Now segment
those users!
1/3 spend
69. Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
70. Do AirBnB hosts
get more business
if their property is
professionally
photographed?
71. Gut instinct (hypothesis)
Professional photography helps AirBnB’s business
Candidate solution (MVP)
20 field photographers posing as employees
Measure the results
Compare photographed listings to a control group
Make a decision
Launch photography as a new feature for all hosts
75. Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure
the results
Make changes
in production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
76. “Gee, those
houses that do
well look really
nice.”
Maybe it’s the
camera.
“Computer: What
do all the
highly rented
houses have in
common?”
Camera model.
With data:
find a commonality
Without data: make a
good guess
77. Landing page design A/B testing
Cohort analysis General analytics
URL shortening
Funnel analytics
Influencer Marketing
Publisher analytics
SaaS analytics
Gaming analytics
User interaction Customer satisfaction KPI dashboardsUser segmentation
User analytics Spying on users
92. A Facebook user reaching 7 friends within 10 days of signing up (Chamath
Palihapitiya)
If someone comes back to Zynga a day after signing up for a game, they’ll probably
become an engaged, paying user (Nabeel Hyatt)
A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)
Twitter user following a certain number of people, and a certain percentage of those
people following the user back (Josh Elman)
A LinkedIn user getting to X connections in Y days (Elliot Schmukler)
(from the 2012 Growth Hacking conference) (These are also great segments to analyze.)
The leading indicator
93. • Growth hacking is simply what marketing should have been
doing, but it fell in love with Don Draper and opinions along the
way
• Optimize a factor you think is correlated with growth
The growth hack
99. The Zero Overhead principle
A central theme to this new wave of innovation is the
application of core product tenets from the consumer space
to the enterprise.
In particular, a universal lesson that I keep sharing with all
entrepreneurs building for the enterprise is the Zero
Overhead Principle: no feature may add training costs to
the user.
DJ Patil
101. The Lean Analytics lifecycle of an Intrapreneur
Empathy Consulting to test ideas and
bootstrap the business
Lock-in, IP control,
overfitting
Stickiness Standardization and integration;
shift from custom to generic
Ability to integrate;
support
Virality Word of mouth, references, case
studies
Bad vibes; exclusivity
Revenue Growing direct sales, professional
services, support
Pipeline, revenue
recognition, comp
Scale Channels, analysts, ecosystems,
APIs, vertically targeted products
Crossing the chasm;
Gorillas
Stage Do this Fear this
102. Enterprise-focused startups:
Metrics that matter
• Ease of customer engagement
and feedback
• Alpha/beta pipeline
• Stickiness and usability
• Integration costs
• User engagement with the app
• Disentanglement after a sale
• Support costs
• User groups and feedback
• Pitch success for channel tools
• Barriers to exit
104. If a startup is an organization designed
to search for a sustainable, repeatable
business model, then an established
company is an organization designed
to perpetuate one.
105. The Lean Analytics lifecycle of an Intrapreneur
Empathy
Find problems; don’t test demand.
Skip the business case, do analytics
Entitled, aggrieved
customers
Stickiness
Know your real minimum based on
expectations, regulations
Hidden “must haves”,
feature creep
Virality
Build inherent virality in from the
start; attention is the new currency
Luddites who don’t
understand sharing
Revenue
Consider the ecosystem, channels,
and established agreements
Channel conflict,
resistance, contracts
Scale
Hand the baton to others gracefully Hating what happens
to your baby
Beforehand
Get buy-in Political fallout
107. Frame it like a study
Product creation is almost
accidental.
Unlike a VC or startup, when
the initiative fails the
organization still learns.
http://www.flickr.com/photos/creative_tools/8544475139
109. Use outliers and missed searches to
hunt for good ideas & adjacencies
(Multi-billion-dollar hygiene product company)
1/8 men have an incontinence issue. 1/3
women do.
When search results show a significant
number of men searching, this suggests the
adjacent (male) market is underserved.
110. Use data to create a taste for
data
Sitting on Billions of rows of
transactional data
David Boyle ran 1M online surveys
Once the value was obvious to
management, got license to dig.
111. Focus on the desired behavior, not just
the information.
http://www.psychologytoday.com/blog/yes/
200808/changing-minds-and-changing-towels
26% increase in towel
re-use with an appeal
to social norms; 33%
increase when tied to
the specific room.
Energy Conservation “Nudges” and Environmentalist
Ideology: Evidence from a Randomized Residential Electricity
Field Experiment - Costa & Kahn 2011
The effectiveness of energy
conservation “nudges” depends on
an individual’s political ideology ...
Conservatives who learn that their
consumption is less than their
neighbors’ “boomerang” whereas
liberals reduce their consumption.
113. Know what has to
be built in-house
SAP integration
Employment law
114. Run it as a consulting business first.
(Just don’t get addicted to it. Your goal is to
learn and overcome integration challenges and
find the 20% of features that 80% of the market
will pay for.)
115. When in doubt, collect data
From tackling the FTA rate to
visualizing the criminal justice
supply chain.
120. Traction graphs
Your business model
The stage you’re at
Your one metric
... change often if
you’re doing it right.
So how do you track
that over time?
121. Traction graphs
Jan Feb Mar Apr May Jun
Signups
per day
Conversion
rate
Churn
rate
Viral
coefficient
This axis changes for
each metric
122. Traction graphs
Jan Feb Mar Apr May Jun
Signups
per day
Conversion
rate
Churn
rate
Viral
coefficient
0%
123. Use vanity to get to
meaningful metrics
Your goal is to produce
outcomes
If the outcomes require
action, and vanity motivates
actors, use it
But show how the vanity
metric is a leading indicator of
the real one
x
Web traffic
Revenue
Activation
Cart
Size
Conversion
rate
124. The three threes
Three
assumptions
What big bets are you making?
•“People will answer questions”
•“Organizers are frustrated with how to run conferences”
•“We'll make money from parents”
•“Amazon is reliable enough for our users.”
Three actions
to take
What are you doing to make these assumptions happen (or
identify they’re wrong and change course?)
•Product enhancements
•Marketing strategies
Three experiments
to run
•Feature tests
•Continuous deployment
•A/B testing
•Customer survey
125. The three threes
Three
assumptions
Three actions
to take
Three experiments
to run
Monthly
Weekly
Daily
Board, investors,
founders
Executive team
Employees
Strategy
Tactics
Execution
126. The three threes
Three
assumptions
Three actions
to take
Three experiments
to run
Get more
people
Increase
answer %
Test better
questions
Change
the UI
Test
timings
Questions
from peers
Many people will
answer questions
127. The problem-solution canvas
CURRENT STATUS
• List key metrics you’re
tracking, where they’re at, and
compare with last few weeks
• How are things trending?
LAST WEEK’S LESSONS LEARNED
AND ACCOMPLISHMENTS)
• What did you learn last week?
• What was accomplished?
• On track: YES / NO?
The Goal is to Learn
128. The problem-solution canvas
HYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start
working on next week. Rank them.
• Why do you believe each solution will
help you solve or complete solve the
problem?
METRICS / PROOF + GOALS
Problem #1 (put name here)
• Metrics you’ll use to measure whether
or not the solutions are doing what you
hoped (solving the problem)
• List proof (qualitative) you’ll use as well
• Define goals for the metric
HYPOTHESIZED SOLUTIONS
• List possible solutions that you’ll start
working on next week. Rank them.
METRICS / PROOF + GOALS
• Metrics you’ll use to measure whether
or not the solutions are doing what you
hoped (solving the problem)
Problem #2 (put name here)
129. “The most important figures that one
needs for management are unknown
or unknowable, but successful
management must nevertheless take
account of them.”
Lloyd S. Nelson
130. Pic by Twodolla on Flickr. http://www.flickr.com/photos/twodolla/3168857844