Presentation on Lean Analytics at MicroConf 2013. Understanding what metrics are the most value, when, for your type of business. * What makes a good metric? * Types of metrics (qualitative vs. quantitative, vanity vs. actionable, etc.) * Lean Analytics framework Shared a number of case studies: Airbnb, Buffer, ClearFit, OffceDrop and others.

1. (or flail around blindly...it’s up to you)
Measure What Matters
Ben Yoskovitz
@byosko

2. I am a
product guy
entrepreneur
blogger
angel investor
http://instigatorblog.com

3. What I’ve learned
It’s easy to get zombified.
Recruitment sucks.
Startup accelerators are fun.
Eventually you get it right.
TBD.

4. I’m not an analytics expert.

5. Or a data scientist.

6. http://leananalyticsbook.com
But I wrote a book on
analytics anyway.

7. Why did I write it?
h"p://www.flickr.com/photos/horiavarlan/4290549806/sizes/l/in/photostream/

8. We’re all liars.

9. The basics of Lean Startup
Everyone’s idea is
the best right?
People love
this part!
(but that’s not always a
good thing)
This is where things
fall apart.
No data, no
learning.

10. What I Hope You Get From It
h"p://www.flickr.com/photos/romtomtom/4382603005/sizes/o/in/photostream/

11. Follow the Lean model, and it
becomes increasingly hard to lie,
especially to yourself.
The importance of intellectual honesty

12. Using your gut properly
Instincts are experiments.
Data is proof.

13. Better decision making abilities
Everyone has data, the key is
figuring out what pieces will
improve your learning and
decision making.

14. Focus
Don’t chase shiny objects. You
might succeed without focus, but
it’ll be by accident.

15. Measure what matters
1 What makes a good metric?
2 Types of metrics
3 Analytical superpowers
4 Lean Analytics framework
5 The One Metric That Matters
6 Lean Analytics Cycle

16. What Makes a Good Metric?
h"p://www.flickr.com/photos/artnoose/2263480871/sizes/l/in/photostream/

17. Analytics is the measurement of
movement towards your
business goals.
What is analytics?

18. comparative
www.naturalhealth365.com
&
www.orangeclaire.com

19. understandable
www.speakingpracEcally.com

20. ratio or rate
h"p://www.flickr.com/photos/pyth0ns/4816846174/

21. changes your behavior
h"p://www.flickr.com/photos/68001867@N00/426536440/sizes/z/in/photostream/

22. If it won’t change
how you behave,
it’s a
bad
metric.
If a metric won’t change how
you behave, it’s a
h"p://www.flickr.com/photos/circasassy/7858155676/

23. h"p://www.flickr.com/photos/maImaIla/3822631755/
Types of Metrics

24. Warm and fuzzy. Cold and hard.
Unstructured,
anecdotal,
revealing, hard to
aggregate.
Numbers and stats;
hard facts but less
insight.
Qualitative
vs.
Quantitative

25. Discover qualitatively.
Prove quantitatively.
Quantitative vs. qualitative data

26. Do hosts with
professional
photography get
more business?
Airbnb experiments...

27. Professional photography helps Airbnb’s business
Gut instinct
Concierge MVP
20 photographers in the field
Test results
Two to three times more bookings!
Back to the beginning
Use additional data to keep experimenting

28. 5,000 shoots / month in Feb. 2012

29. Makes you feel
good but doesn’t
change how you’ll
act.
Helps you pick a
direction and
change your
behavior.
“Up and to the right.” These are good.
Vanity Actionable
vs.

30. Hits
A metric from the early, foolish days of the Web.
Count people instead.
Page views
Marginally better than hits. Unless you’re displaying
ad inventory, count people.
Visits
Is this one person visiting a hundred times, or are a
hundred people visiting once? Fail.
Users
This tells you nothing about what they did, why they
stuck around, or if they left.
Followers/
friends/likes
Count actions instead. Find out how many followers
will do your bidding.
Logins
But what are they actually doing when they login?
Logins don’t tell you about actions and value.
Vanity metrics are bad!

31. Speculative, tries to
find unexpected or
interesting insights.
Predictable, keeps
you abreast of
normal, day-to-day
operations.
The cool stuff. The necessary stuff.
Exploratory Reporting
vs.

32. Pivoting from friends to moms
•Started as Circle of Friends
•Leveraged Facebook early
•Grew to 10M users
But engagement sucked!

33. Moms are crazy!
(in a good way)
Engagement solved!
• Messages to one another were on average 50% longer.
• 115% more likely to attach a picture to a post they wrote.
• 110% more likely to engage in a threaded (i.e. deep) conversation.
• Friends, once invited, were 50% more likely to become engaged users.
• 180% more likely to click on Facebook news feed items.
• 60% more likely to accept invitations to the app.

34. Historical metric
that shows you
how you’re doing:
reports the news.
Number today that
shows a metric
tomorrow: makes
the news.
Try and get here.Start here.
Lagging Leading
vs.

35. h"p://www.flickr.com/photos/bloke_with_camera/401812833/sizes/o/in/photostream/
Analytical Superpowers
(or what the heck is growth hacking?)

36. 1
10
100
1000
10000
Ice cream consumption Drownings
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

37. Summer
Ice cream
consumption
Drowning
Correlated
Causal
Causal
Two variables that
change in similar ways,
perhaps because
they’re linked to
something else.
Correlated
An independent factor
that directly impacts a
dependent one.
Causal
vs.

38. Correlation lets you
predict the future
Causality lets you
change the future
“I will have 420 engaged users
and 75 paying customers next
month.”
“If I can make more first-time
visitors stay on for 17 minutes I
will increase sales in 90 days.”
Find correlation Test causality
Optimize the
causal factor
Causality is a superpower, because it lets
you change the future.

39. Lean Analytics Framework
h"p://www.flickr.com/photos/ikhlasulamal/2331176652/

40. Your Business + Stage
What business
are you in?
What stage
are you at?
•E-Commerce
•SaaS
•Free Mobile App
•2-Sided Marketplace
•Media
•User-Generated Content
•Empathy
•Stickiness
•Virality
•Revenue
•Scale

41. business models

42. The SaaS Customer
Lifecycle
Customer Acquisition Cost
paid direct search wom
inherent
virality
VISITOR
Freemium/trial offer
Enrollment
User
Disengaged User
Cancel
Freemium
churn
Engaged User
Free user
disengagement
Reactivate
Cancel
Trial abandonment rate
Invite Others
Paying Customer
Reactivation rate
Paid
conversion
FORMER USERS
User Lifetime Value
Reactivate
FORMER CUSTOMERS
Customer Lifetime Value
Viral coefficient
Viral rate
Resolution
Support data
Account Cancelled Billing Info Exp.
Paid Churn Rate
Tiering
Capacity Limit
Upselling
rate Upselling
Disengaged DissatisfiedTrial Over

43. •Stage: Revenue / Scale
•Model: SaaS (Paid)
•Recruitment marketing and
assessment software
•Switched business models from
monthly subscription to pay per
job posting
Does recurring revenue
work for everyone?

44. 10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers,
and hiring. You have to understand your customer, who
they are, how and why they buy, and how they value
your product or service.” - Ben Baldwin, co-founder
Lots of money!

45. h"p://www.flickr.com/photos/86791111@N00/3126955746/
don’t just follow
the leader

46. lean analytics stages

47. EMPATHY
STICKINESS
GROWTHRATE
VIRALITY
REVENUE
SCALE
Lean Analytics
Stages
I’ve found a real, poorly-met need that
a reachable market faces.
I’ve figured out how to solve the problem in
a way they will adopt and pay for.
I’ve built the right product/features/
functionality that keeps users around.
The users and features fuel growth
organically and artificially.
I’ve found a sustainable, scalable business
with the right margins in a healthy
ecosystem.
“Gates” needed to
move forward

48. •Stage: Scale
•Model: SaaS
•Popular social sharing application
•Focused primarily on customer
acquisition
•Charged from day one
From Stickiness to Scale
(through Revenue)

49. 20%
60%
20%
2%
of visitors created an account
(acquisition / Empathy)
of sign-ups returned in the 1st month
(engagement / Stickiness)
of sign-ups were active after 6 months
(engagement / Stickiness)
convert from free to paid
(Virality & Revenue)
Buffer charges early to prove
people want the problem solved

50. skip steps at your
own risk

51. One Metric
That Matters.
How It All Comes Together
The business you’re in
E-Com SaaS Mobile 2-Sided Media UCG
Empathy
Stickiness
Virality
Revenue
Scale
Thestageyou’reat

52. Choose only one metric and
draw a line in the sand.
Putting the pieces together...

53. •Stage: Revenue
•Model: SaaS (Freemium)
•Paper and digital collaboration
•180,000 users
•Paid churn is their One Metric
That Matters (OMTM)
Building a revenue engine

54. • Target < 4% paid churn (hitting 2% lately
on a monthly basis)
•Anything over 5% means they don’t have
a business that will generate positive
margin returns: the bucket is too leaky
The OMTM: Paid Churn

55. • Can we acquire more valuable customers?
•What product features can increase engagement?
• Can we improve customer support?
•Was a marketing campaign successful?
•Were customer complaints lowered?
•Was a product upgrade valuable?
If Paid Churn: Why & Next Steps:
Paid Churn = “business health” indicator
•Are the new customers not the right segment?
• Did a marketing campaign fail?
• Did a product upgrade fail somehow?
• Is customer support falling apart?

56. Some interesting benchmarks
Growth
5% / week (revenue or active
users)
Churn
2% / month
Engaged visitors
30% monthly users
10% daily users
Time on site
17 minutes
Page load time
< 5 seconds
CLV:CAC
3:1
Mobile file size
< 50MB
Free to paid
2% of free users

57. Lean Analytics cycle
h"p://www.flickr.com/photos/jrodmanjr/4728457415/

58. Identify a key business problem,
pick the OMTM, draw a line in the
sand, and get started.
Time to experiment

59. Draw a new line
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a KPI
The Lean Analytics Cycle

60. Thank you.
byosko@gmail.com
@byosko
ORDER!
follow me.
instigatorblog.com
leananalyticsbook.com
subscribe.
email me.