0
Vanity Dies Hard
Back	
  to	
  Basics	
  
@ElliotSusel	
  
Vanity Metrics
Actionable Metrics
Innovation Accounting
Systems
Vanity Metric
Low	
  correla/on	
  to	
  what	
  makes	
  your	
  
business	
  successful	
  
These are important…
But	
  let’s	
  not	
  overstate	
  their	
  
significance	
  
Success Theater
Look	
  at	
  our	
  [insert	
  vanity	
  metric]!	
  
Our	
  business	
  is	
  amazing!	
  
What then DO
we measure?
Actionable Metric
High	
  correla/on	
  to	
  what	
  makes	
  your	
  
business	
  successful	
  
Actionable Metric
But there’s a problem
The	
  ac/onable	
  metric	
  is	
  a	
  lie	
  
What impacts
ride volume?
•  Outreach	
  
•  Promo/ons	
  
•  Support	
  
•  Technology	
  
Stuff we control
•  Weather	
  
•  Seasonality	
  
•  Holidays	
  
•  Events	
  (Sports,	
  Religious,	
  Concerts,	
  Others…)	
  
•  Syst...
THIS IS SCARY
We	
  think	
  we	
  have	
  meaningful	
  data	
  
We	
  do	
  not	
  have	
  meaningful	
  data	
  
Meaningful
Conclusions?
Innovation Accounting
Understand	
  the	
  factors	
  impac/ng	
  
your	
  metrics	
  
Understanding Ride Volume
•  New	
  vs	
  Return	
  Customers	
  
•  Distribu/on	
  by	
  plaMorm	
  
•  Distribu/on	
  by...
Why?
Validated	
  learning	
  is	
  a	
  startup’s	
  
most	
  valuable	
  commodity	
  
Nice concepts
How	
  do	
  we	
  execute?	
  
A SYSTEM,
YOU SAY?
Losers	
  have	
  goals	
  
Winners	
  create	
  systems	
  
Components of Our System	
  
•  Every	
  development	
  request	
  has	
  a	
  goal	
  
•  Every	
  goal	
  has	
  quan/fic...
Stuff we’ve learned
Or	
  at	
  least,	
  closing	
  thoughts	
  
Culture Matters
If	
  leadership	
  doesn’t	
  value	
  analy/cs,	
  
no	
  one	
  will	
  value	
  analy/cs	
  
Over-reporting
Don’t	
  do	
  it	
  
Accidental Learning
Informa/on	
  radiators	
  are	
  your	
  friend	
  
Create a System
A	
  project	
  accounts	
  for	
  metrics	
  from	
  
incep/on	
  to	
  delivery	
  
Then tune it
Because	
  you’re	
  never	
  done	
  learning	
  
Questions?
@ElliotSusel	
  
Upcoming SlideShare
Loading in...5
×

Lean Startup Analytics/Metrics

897

Published on

Presented at Lean Startup Circle on 4/15/14 for their back to basics night. The theme for the evening was analytics.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
897
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Today we’re going to talk about vanity… but probably not in the way that you expected
  • Downloads, app opens, app reviews, # of registered users
  • But if you were to overstate their significanceDanger… you run the risk of losing sight of what really matters.
  • Rides.
  • The understanding of the metric is incomplete if we don’t understand why it’s changing
  • Ridership metrics subject to the following factors which are completely out of our hands and may completely invalidate all would-be meaningful conclusions
  • Ridership metrics subject to the following factors which are completely out of our hands and may completely invalidate all would-be meaningful conclusions.
  • You don’t even know if all the “great changes” you’re making are actually improving the product, or if it was getting better in spite of your efforts.
  • How do we get to meaningful conclusions?
  • Examples of how we could do this. Callback to things that do/don’t impact metrics. Such as rain. We know it increases ridership, but we can’t control whether or not it rains, at least, not yet.
  • Meaningfulanalytics and metrics empower you to draw intelligent conclusions that you might not have been able to determine otherwise. It lets you validate and challenge your assumptions.
  • Targeting this conversation at small companies (>30 and <100). Speak to my experience. Talk about systems.
  • NOTE: THIS IS NOT PERFECT
  • If you send me 30 pages of reports I will read 0 pages of reports
  • On that note, I’d like to close out and leave some time for questions
  • Transcript of "Lean Startup Analytics/Metrics"

    1. 1. Vanity Dies Hard Back  to  Basics  
    2. 2. @ElliotSusel  
    3. 3. Vanity Metrics Actionable Metrics Innovation Accounting Systems
    4. 4. Vanity Metric Low  correla/on  to  what  makes  your   business  successful  
    5. 5. These are important… But  let’s  not  overstate  their   significance  
    6. 6. Success Theater Look  at  our  [insert  vanity  metric]!   Our  business  is  amazing!  
    7. 7. What then DO we measure?
    8. 8. Actionable Metric High  correla/on  to  what  makes  your   business  successful  
    9. 9. Actionable Metric
    10. 10. But there’s a problem The  ac/onable  metric  is  a  lie  
    11. 11. What impacts ride volume?
    12. 12. •  Outreach   •  Promo/ons   •  Support   •  Technology   Stuff we control
    13. 13. •  Weather   •  Seasonality   •  Holidays   •  Events  (Sports,  Religious,  Concerts,  Others…)   •  System  Outages  (Fleet,  Dispatch,  Internal)     Stuff we don’t control
    14. 14. THIS IS SCARY We  think  we  have  meaningful  data   We  do  not  have  meaningful  data  
    15. 15. Meaningful Conclusions?
    16. 16. Innovation Accounting Understand  the  factors  impac/ng   your  metrics  
    17. 17. Understanding Ride Volume •  New  vs  Return  Customers   •  Distribu/on  by  plaMorm   •  Distribu/on  by  fleet/geographic  region   •  Distribu/on  by  day  of  week  and  /me  of  day  
    18. 18. Why? Validated  learning  is  a  startup’s   most  valuable  commodity  
    19. 19. Nice concepts How  do  we  execute?  
    20. 20. A SYSTEM, YOU SAY? Losers  have  goals   Winners  create  systems  
    21. 21. Components of Our System   •  Every  development  request  has  a  goal   •  Every  goal  has  quan/fica/on   •  Architecture  review  includes  analy/cs   •  Code  review  includes  analy/cs   •  Every  PROD  deployment  requires  hooks  to   monitoring/analy/cs  tools  
    22. 22. Stuff we’ve learned Or  at  least,  closing  thoughts  
    23. 23. Culture Matters If  leadership  doesn’t  value  analy/cs,   no  one  will  value  analy/cs  
    24. 24. Over-reporting Don’t  do  it  
    25. 25. Accidental Learning Informa/on  radiators  are  your  friend  
    26. 26. Create a System A  project  accounts  for  metrics  from   incep/on  to  delivery  
    27. 27. Then tune it Because  you’re  never  done  learning  
    28. 28. Questions? @ElliotSusel  
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×