Successfully reported this slideshow.
Your SlideShare is downloading. ×

Lean Analytics overview from GROWtalk Montreal

Loading in …3

Check these out next

1 of 63 Ad

More Related Content

Viewers also liked (16)


Recently uploaded (20)

Lean Analytics overview from GROWtalk Montreal

  1. 1. Lean Analytics Use data to build a better business faster. @byosko | @acroll @leananalytics
  2. 2. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
  3. 3. Analytics is the measurement of movement towards your business goals.
  4. 4. Small business example: Solare watches the numbers • Stage: Revenue • Model: Retailer • Solare is an Italian fine-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicators
  5. 5. Solare watches the numbers • A line in the sand: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-staffing, leading to dissatisfied customers • A leading indicator: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant.
  6. 6. In a startup, the purpose of analytics is to iterate to a product/market fit before the money runs out.
  7. 7. What I’ll cover •What makes a good metric •Understanding cohorts and segments •The Lean Analytics cycle •The Stages of Lean Analytics •Picking One Metric That Matters
  8. 8. Qualitative or Quantitative 5 things you Exploratory or Reporting need to know Vanity or Actionable about metrics Correlated or Causal Leading or Lagging
  9. 9. Qualitative Quantitative Unstructured, Numbers and stats; anecdotal, hard facts but less revealing, hard to insight. aggregate. Warm and fuzzy. Cold and hard.
  10. 10. Simply: you can’t count smiles. Discover qualitatively, prove quantitatively. Qualitative is inspiration, quantitative is verification.
  11. 11. Exploratory Reporting Speculative, trying Predictable, keeping to find unexpected you abreast of or interesting normal, managerial insights. operations.
  12. 12. Donald Rumsfeld on analytics Are facts which may be wrong and we know should be checked against data. know we don’t Are questions we can answer by reporting, which we should baseline know & automate. Things we Are intuition which we should we know quantify and teach to improve don’t effectiveness, efficiency. know we don’t Are exploration which is where unfair advantage and interesting know epiphanies live. (Or rather, Avinash Kaushik channeling Rumsfeld)
  13. 13. Vanity Actionable Picks a direction. Makes you feel good, but doesn’t change how you’ll act.
  14. 14. A metric from the early, foolish days of the Web. Hits Count people instead. Marginally better than hits. Unless you’re displaying Page views ad inventory, count people. Is this one person visiting a hundred times, or are a Visits hundred people visiting once? Fail. This tells you nothing about what they did, why they Unique visitors stuck around, or if they left. Followers/ Count actions instead. Find out how many followers friends/likes will do your bidding. Time on site, or Poor version of engagement. Lots of time spent on pages/visit support pages is actually a bad sign. How many recipients will act on what’s in them? Emails collected Number of Outside app stores, downloads alone don’t lead to downloads lifetime value. Measure activations/active accounts.
  15. 15. it’s a how you behave, If it won’t change bad metric.
  16. 16. 2-sided market model: AirBnB and photography • Stage: Revenue • Model: 2-sided marketplace • Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.
  17. 17. AirBnB tests a hypothesis • The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” • Built a concierge MVP • Found that professionally photographed listings got 2-3x more bookings than the market average. • In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for hosts.
  18. 18. NIGHTS BOOKED 10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012
  19. 19. A few words on causality.
  20. 20. 50 37.5 25 12.5 0 1 2 3 4 5 6 7 8 9 10 Seat rentals
  21. 21.
  22. 22.
  23. 23. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings
  24. 24.
  25. 25.
  26. 26.
  27. 27. 10000 1000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings Temperature
  28. 28.
  29. 29.
  30. 30.
  31. 31. Correlated Causal Two variables that An independent change in similar factor that directly ways , perhaps impacts a because they’re dependent one. linked to somethingCausal else. Summer al Ca us us Ca Correlated al Drowning Ice cream consumption
  32. 32.
  33. 33. Causality is a superpower, because it lets you change the future. Correlation lets you Causality lets you predict the future change the future “I will have 420 “If I can make more engaged users and first-time visitors stay 75 paying customers on for 17 minutes I next month.” will increase sales in 90 days.” Optimize the Find correlation Test causality causal factor
  34. 34. Leading Lagging Number today that Historical metric that shows metric shows how you’re tomorrow—makes doing—reports the the news. news.
  35. 35. What mode of e-commerce are you? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you like time in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)
  36. 36. • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya) • If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt) • A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang) • Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman) • A LinkedIn user getting to X connections in Y days (Elliot Schmukler) (These are also great segments to analyze.) (from the 2012 Growth Hacking conference)
  37. 37. So how do you test things? Segmentation.
  38. 38. Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁ (age, gender, etc.) revenue.) with a result.
  39. 39. Why use cohorts? Here’s an example. Is this   January February March April May company growing or Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50 stagnating? Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 How about February $6 $4 $2 $1 now? March $7 $6 $5 April   $8 $7 May       $9
  40. 40. Why use cohorts? Here’s an example. Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 Look at the February $6 $4 $2 $1   same data in cohorts March $7 $6 $5     April $8 $7       May $9         Averages $7 $5 $3 $1 $0.5
  41. 41. The Lean Analytics Framework.
  42. 42. Eric Ries’ Three engines Stickiness Virality Price Approach Keep people Make people Spend revenue coming back. invite friends. getting customers. Math that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them. they tell them. they cost to get.
  43. 43. The five Stages of Lean Analytics The business you’re in E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy The stage you’re at One Metric Stickiness Virality Revenue That Matters. Scale
  44. 44. Mobile app model: Localmind hacks Twitter • Stage: Empathy • Model: UGC/mobile • Real-time question and answer platform tied to locations. • Needed to find out if a core behavior—answering questions about a place— happened enough to make the business real
  45. 45. Localmind hacks Twitter • Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind. • Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc. • The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.
  46. 46. Stickiness stage: WP Engine discovers the 2% cancellation rate • Stage: Stickiness • Model: SaaS • Wordpress hosting company founded in July 2010, it raised $1.2M in November 2011
  47. 47. WP-Engine discovers the 2% cancellation rate • All companies have cancellations, but founder Jason Cohen was alarmed that he was losing a quarter of customers every year. • Jason called customers himself. “Not everyone wanted to speak with me, but enough people were willing to talk, even after they had left, that I learned a lot about why they were leaving.” • Asked around. Turns out 2% is best case for most hosting companies. • Without this, the company would have been getting diminishing returns over- optimizing churn; instead, they could focus on maximizing revenues or lowering acquisition costs.
  48. 48. Virality stage: qidiq streamlines invites • Stage: Virality • Model: SaaS • Tool to poll small groups, built in the Year One Labs accelerator
  49. 49. Initial design Redesigned workflow Survey owner adds recipient to group Survey owner adds recipient to group 70-90% RESPONSE RATE Survey owner asks question Survey owner asks question Recipient gets invite Recipient reads survey question 10-25% RESPONSE RATE Recipient installs mobile app Recipient responds to question Recipient sees survey results Recipient creates account, profile Recipient can edit profile, view past (Later, if needed…) questions, etc. Recipient visits website Recipient reads survey question Recipient has no password! Recipient responds to question Recipient does password recovery Recipient sees survey results One-time link sent to email Recipient creates password Recipient can edit profile, view past questions, etc.
  50. 50. Revenue stage: Backupify’s customer lifecycle • Stage: Scale • Model: SaaS • Leading backup provider for cloud based data. • The company was founded in 2008 by Robert May and Vik Chadha • Has gone on to raise $19.5M in several rounds of financing.
  51. 51. Shifting to Customer Acquisition Payback as a key metric • Initially focused on site visitors • Then focused on trials • Then switched to signups • Today, MRR • In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x • Now they track Customer Acquisition Payback • Target is less than 12 months
  52. 52. What these have in common: The Lean Analytics Cycle Success! Pick a KPI Draw a line in the sand Pivot or give up Draw a new line Find a potential Try again improvement Did we move the needle? Without With data: data: make a find a good guess commonality Design a test Measure the results Hypothesis Make changes in production
  53. 53. What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits, Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Affiliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licenses
  54. 54. Choose only one metric.
  55. 55. Yes, one metric.
  56. 56. It will soon change.
  57. 57. In a startup, focus is hard to achieve.
  58. 58. Having only one metric addresses this problem.
  59. 59. Metrics are like squeeze toys.
  61. 61. Once, a leader convinced others in the absence of data.
  62. 62. Now, a leader knows what questions to ask.
  63. 63. Ben Yoskovitz @byosko Alistair Croll @acroll