Quick presentation at GROWTalk Montreal on February 19, 2013. The book's out in early March.

1. Lean Analytics
Use data to build a
better business faster.
www.leananalyticsbook.com
@byosko | @acroll
@leananalytics

2. Kevin Costner is a lousy entrepreneur.
Don’t sell what you can make.
Make what you can sell.

3. Analytics is the measurement
of movement towards your
business goals.
http://www.flickr.com/photos/itsgreg/446061432/

4. Small business example:
Solare watches the
numbers
• Stage: Revenue
• Model: Retailer
• Solare is an Italian fine-dining restaurant under new management. The new team
is trying to identify the key metrics and leading indicators

5. Solare watches the numbers
• A line in the sand: Gross Revenue to Labor Cost
• Under 30% is good
• Below 24% is great
• Lower than 20% and you may be under-staffing, leading to dissatisfied
customers
• A leading indicator: Total covers is 5x reservations at 5PM
• If you have 50 reservations at 5, you’ll have 250 covers that night.
• This ratio varies by restaurant.

6. In a startup, the purpose of
analytics is to iterate to a
product/market fit before
the money runs out.

7. What I’ll cover
•What makes a good metric
•Understanding cohorts and segments
•The Lean Analytics cycle
•The Stages of Lean Analytics
•Picking One Metric That Matters

8. Qualitative or Quantitative
5 things you Exploratory or Reporting
need to know Vanity or Actionable
about metrics Correlated or Causal
Leading or Lagging

9. Qualitative Quantitative
Unstructured, Numbers and stats;
anecdotal, hard facts but less
revealing, hard to insight.
aggregate.
Warm and fuzzy. Cold and hard.
http://www.flickr.com/photos/zooboing/8388257248/ http://www.flickr.com/photos/x1brett/4665645157/

10. Simply: you can’t count smiles.
Discover qualitatively, prove quantitatively.
Qualitative is inspiration, quantitative is verification.

11. Exploratory Reporting
Speculative, trying Predictable, keeping
to find unexpected you abreast of
or interesting normal, managerial
insights. operations.
http://www.flickr.com/photos/50755773@N06/5415295449/ http://www.flickr.com/photos/elwillo/4737933662/

12. Donald Rumsfeld on analytics
Are facts which may be wrong and
we know should be checked against data.
know
we don’t Are questions we can answer by
reporting, which we should baseline
know & automate.
Things we
Are intuition which we should
we know quantify and teach to improve
don’t effectiveness, efficiency.
know
we don’t Are exploration which is where
unfair advantage and interesting
know epiphanies live.
(Or rather, Avinash Kaushik channeling Rumsfeld)

13. Vanity Actionable
Picks a
direction.
Makes you feel
good, but doesn’t
change how you’ll
act.
http://www.flickr.com/photos/lostseouls/807253220/ http://www.flickr.com/photos/aussiegall/6382775153/

14. A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Time on site, or Poor version of engagement. Lots of time spent on
pages/visit support pages is actually a bad sign.
How many recipients will act on what’s in them?
Emails collected
Number of Outside app stores, downloads alone don’t lead to
downloads lifetime value. Measure activations/active accounts.

15. http://www.flickr.com/photos/circasassy/7858155676/
it’s a
how you behave,
If it won’t change
bad
metric.

16. 2-sided market model:
AirBnB and photography
• Stage: Revenue
• Model: 2-sided marketplace
• Rental-by-owner marketplace that allows property owners to list and market
their houses. Offers a variety of related services as well.

17. AirBnB tests a hypothesis
• The hypothesis: “Hosts with professional photography will get more business.
And hosts will sign up for professional photography as a service.”
• Built a concierge MVP
• Found that professionally photographed listings got 2-3x more bookings than the
market average.
• In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for
hosts.

18. NIGHTS BOOKED
10 million
8 million
6 million
20 photographers
4 million
2 million
2008 2009 2010 2011 2012

19. A few words on causality.
http://www.flickr.com/photos/roryfinneren/65729247

20. 50
37.5
25
12.5
0
1 2 3 4 5 6 7 8 9 10
Seat rentals
http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg

21. http://www.imdb.com/media/rm3768753408/tt0073195

22. http://www.flickr.com/photos/kapungo/2287237966

23. 10000
1000
100
10
1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings

24. http://www.flickr.com/photos/25159787@N07/3766111564

25. http://www.flickr.com/photos/wheressteve/3284532080

26. http://www.flickr.com/photos/wtlphotos/1086968783

27. 10000
1000
100
10
1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings Temperature

28. http://www.flickr.com/photos/stuttermonkey/57096884

29. http://www.flickr.com/photos/germanuncut77/3785152581

30. http://www.flickr.com/photos/fasteddie42/2421039207

31. Correlated Causal
Two variables that An independent
change in similar factor that directly
ways , perhaps impacts a
because they’re dependent one.
linked to somethingCausal
else.
Summer
al
Ca
us
us
Ca
Correlated al Drowning
Ice cream
consumption

32. http://www.flickr.com/photos/bootbearwdc/1243690099/

33. Causality is a superpower, because it lets you
change the future.
Correlation lets you Causality lets you
predict the future change the future
“I will have 420 “If I can make more
engaged users and first-time visitors stay
75 paying customers on for 17 minutes I
next month.” will increase sales in
90 days.”
Optimize the
Find correlation Test causality
causal factor

34. Leading Lagging
Number today that Historical metric that
shows metric shows how you’re
tomorrow—makes doing—reports the
the news. news.

35. What mode of e-commerce are you?
How many of
your customers Then you are in Your customers You are just
Focus on
buy a second this mode will buy from you like
time in 90 days?
Low CAC,
1-15% Acquisition Once 70% high
of retailers checkout
15-30% Hybrid 2-2.5 20% Increasing
per year of retailers returns
Loyalty,
>30% Loyalty >2.5 10% inventory
per year of retailers expansion
(Thanks to Kevin Hilstrom for this.)

36. • A Facebook user reaching 7 friends within 10 days of
signing up (Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up
for a game, they’ll probably become an engaged, paying
user (Nabeel Hyatt)
• A Dropbox user who puts at least one file in one folder
on one device (ChenLi Wang)
• Twitter user following a certain number of people, and a
certain percentage of those people following the user
back (Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot
Schmukler)
(These are also great segments to analyze.)
(from the 2012 Growth Hacking conference)

37. So how do you
test things?
Segmentation.
http://www.flickr.com/photos/zlakfoto/5294803278/

38. Segments, cohorts, A/B, and multivariates
Cohort:
Comparison of
similar groups
along a timeline.
Segment: A/B test: ☀ Multivariate
Cross-sectional ☀ Changing one analysis
comparison of all thing (i.e. color)
☁ Changing several
people divided by and measuring ☀ things at once to
some attribute
☁ the result (i.e. see which correlates
☁
(age, gender, etc.) revenue.) with a result.

39. Why use cohorts? Here’s an example.
Is this
January February March April May
company
growing or
Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50
stagnating?
Cohort 1 2 3 4 5
January $5 $3 $2 $1 $0.5
How about February $6 $4 $2 $1
now?
March $7 $6 $5
April $8 $7
May $9

40. Why use cohorts? Here’s an example.
Cohort 1 2 3 4 5
January $5 $3 $2 $1 $0.5
Look at the February $6 $4 $2 $1
same data
in cohorts March $7 $6 $5
April $8 $7
May $9
Averages $7 $5 $3 $1 $0.5

41. The Lean Analytics Framework.

42. Eric Ries’
Three engines
Stickiness Virality Price
Approach Keep people Make people Spend revenue
coming back. invite friends. getting customers.
Math that Get customers How many they Customers are
matters faster than you tell, how fast worth more than
lose them. they tell them. they cost to get.

43. The five Stages of Lean Analytics
The business you’re in
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy
The stage you’re at
One Metric
Stickiness
Virality
Revenue That Matters.
Scale

44. Mobile app model:
Localmind hacks Twitter
• Stage: Empathy
• Model: UGC/mobile
• Real-time question and answer platform tied to locations.
• Needed to find out if a core behavior—answering questions about a place—
happened enough to make the business real

45. Localmind hacks Twitter
• Before writing a line of code, Localmind was concerned that people would never
answer questions.
• This was their biggest risk: if questions went unanswered users would have a
terrible experience and stop using Localmind.
• Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about
the area: how busy is it; is the subway running on time; is something open;
etc.
• The response rate to their tweeted questions was very high.
• Good enough proxy to de-risk the solution, and convince the team and
investors that it was worth building Localmind.

46. Stickiness stage:
WP Engine discovers the
2% cancellation rate
• Stage: Stickiness
• Model: SaaS
• Wordpress hosting company founded in July 2010, it raised $1.2M in November
2011

47. WP-Engine discovers the 2%
cancellation rate
• All companies have cancellations, but founder Jason Cohen was alarmed that he
was losing a quarter of customers every year.
• Jason called customers himself. “Not everyone wanted to speak with me, but
enough people were willing to talk, even after they had left, that I learned a lot
about why they were leaving.”
• Asked around. Turns out 2% is best case for most hosting companies.
• Without this, the company would have been getting diminishing returns over-
optimizing churn; instead, they could focus on maximizing revenues or lowering
acquisition costs.

48. Virality stage:
qidiq streamlines invites
• Stage: Virality
• Model: SaaS
• Tool to poll small groups, built in the Year One Labs accelerator

49. Initial design Redesigned workflow
Survey owner adds recipient to group Survey owner adds recipient to group
70-90% RESPONSE RATE
Survey owner asks question Survey owner asks question
Recipient gets invite Recipient reads survey question
10-25% RESPONSE RATE
Recipient installs mobile app Recipient responds to question
Recipient sees survey results
Recipient creates account, profile
Recipient can edit profile, view past (Later, if needed…)
questions, etc.
Recipient visits website
Recipient reads survey question
Recipient has no password!
Recipient responds to question
Recipient does password recovery
Recipient sees survey results
One-time link sent to email
Recipient creates password
Recipient can edit profile, view past
questions, etc.

50. Revenue stage:
Backupify’s customer
lifecycle
• Stage: Scale
• Model: SaaS
• Leading backup provider for cloud based data.
• The company was founded in 2008 by Robert May and Vik Chadha
• Has gone on to raise $19.5M in several rounds of financing.

51. Shifting to Customer Acquisition
Payback as a key metric
• Initially focused on site visitors
• Then focused on trials
• Then switched to signups
• Today, MRR
• In early 2010, CAC was $243 and ARPU was only $39
• Pivoted to target business users
• CLV-to-CAC today is 5-6x
• Now they track Customer Acquisition Payback
• Target is less than 12 months

52. What these have in common:
The Lean Analytics Cycle
Success! Pick a KPI Draw a line
in the sand
Pivot or
give up Draw a new line Find a
potential
Try again improvement
Did we move
the needle? Without With data:
data: make a find a
good guess commonality
Design a test
Measure
the results Hypothesis
Make changes
in production

53. What’s your OMTM?
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy Interviews; qualitative results; quantitative scoring; surveys
Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,
Stickiness conversion listings churn churn, virality spam returns
CAC, shares, Inherent WoM, app Invites, Content
Virality reactivation
SEM, sharing
virality, CAC ratings, CAC sharing virality, SEM
(Money from transactions) (Money from active users) (Money from ad clicks)
Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate
Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs
Affiliates, Other API, magic Spinoffs, Analytics, Syndication,
Scale white-label verticals #, mktplace publishers user data licenses

54. Choose only one metric.

55. Yes, one metric.

56. It will soon change.

57. In a startup,
focus is hard to achieve.

58. Having only one metric
addresses this problem.

59. Metrics are like
squeeze toys.
http://www.flickr.com/photos/connortarter/4791605202/

60. ARCHIMEDES
HAD TAKEN
BATHS BEFORE.

61. Once, a leader convinced others in
the absence of data.

62. Now, a leader knows what
questions to ask.

63. Ben Yoskovitz
byosko@gmail.com
@byosko
Alistair Croll
acroll@gmail.com
@acroll