Lean Analytics overview from GROWtalk Montreal

9,125 views
8,931 views

Published on

Quick presentation at GROWTalk Montreal on February 19, 2013. The book's out in early March.

Published in: Business

Lean Analytics overview from GROWtalk Montreal

  1. 1. Lean Analytics Use data to build a better business faster. www.leananalyticsbook.com @byosko | @acroll @leananalytics
  2. 2. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
  3. 3. Analytics is the measurementof movement towards yourbusiness goals. http://www.flickr.com/photos/itsgreg/446061432/
  4. 4. Small business example:Solare watches thenumbers• Stage: Revenue• Model: Retailer• Solare is an Italian fine-dining restaurant under new management. The new team is trying to identify the key metrics and leading indicators
  5. 5. Solare watches the numbers• A line in the sand: Gross Revenue to Labor Cost • Under 30% is good • Below 24% is great • Lower than 20% and you may be under-staffing, leading to dissatisfied customers• A leading indicator: Total covers is 5x reservations at 5PM • If you have 50 reservations at 5, you’ll have 250 covers that night. • This ratio varies by restaurant.
  6. 6. In a startup, the purpose ofanalytics is to iterate to aproduct/market fit beforethe money runs out.
  7. 7. What I’ll cover•What makes a good metric•Understanding cohorts and segments•The Lean Analytics cycle•The Stages of Lean Analytics•Picking One Metric That Matters
  8. 8. Qualitative or Quantitative5 things you Exploratory or Reportingneed to know Vanity or Actionableabout metrics Correlated or Causal Leading or Lagging
  9. 9. Qualitative Quantitative Unstructured, Numbers and stats; anecdotal, hard facts but less revealing, hard to insight. aggregate. Warm and fuzzy. Cold and hard.http://www.flickr.com/photos/zooboing/8388257248/ http://www.flickr.com/photos/x1brett/4665645157/
  10. 10. Simply: you can’t count smiles.Discover qualitatively, prove quantitatively.Qualitative is inspiration, quantitative is verification.
  11. 11. Exploratory Reporting Speculative, trying Predictable, keeping to find unexpected you abreast of or interesting normal, managerial insights. operations.http://www.flickr.com/photos/50755773@N06/5415295449/ http://www.flickr.com/photos/elwillo/4737933662/
  12. 12. Donald Rumsfeld on analytics Are facts which may be wrong and we know should be checked against data. know we don’t Are questions we can answer by reporting, which we should baseline know & automate.Things we Are intuition which we should we know quantify and teach to improve don’t effectiveness, efficiency. know we don’t Are exploration which is where unfair advantage and interesting know epiphanies live. (Or rather, Avinash Kaushik channeling Rumsfeld)
  13. 13. Vanity Actionable Picks a direction.Makes you feelgood, but doesn’tchange how you’llact.http://www.flickr.com/photos/lostseouls/807253220/ http://www.flickr.com/photos/aussiegall/6382775153/
  14. 14. A metric from the early, foolish days of the Web. Hits Count people instead. Marginally better than hits. Unless you’re displaying Page views ad inventory, count people. Is this one person visiting a hundred times, or are a Visits hundred people visiting once? Fail. This tells you nothing about what they did, why theyUnique visitors stuck around, or if they left. Followers/ Count actions instead. Find out how many followers friends/likes will do your bidding.Time on site, or Poor version of engagement. Lots of time spent on pages/visit support pages is actually a bad sign. How many recipients will act on what’s in them?Emails collected Number of Outside app stores, downloads alone don’t lead to downloads lifetime value. Measure activations/active accounts.
  15. 15. http://www.flickr.com/photos/circasassy/7858155676/ it’s a how you behave, If it won’t changebadmetric.
  16. 16. 2-sided market model:AirBnB and photography• Stage: Revenue• Model: 2-sided marketplace• Rental-by-owner marketplace that allows property owners to list and market their houses. Offers a variety of related services as well.
  17. 17. AirBnB tests a hypothesis• The hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.”• Built a concierge MVP• Found that professionally photographed listings got 2-3x more bookings than the market average.• In mid-to-late 2011, AirBnB had 20 photographers in the field taking pictures for hosts.
  18. 18. NIGHTS BOOKED10 million 8 million 6 million 20 photographers 4 million 2 million 2008 2009 2010 2011 2012
  19. 19. A few words on causality. http://www.flickr.com/photos/roryfinneren/65729247
  20. 20. 5037.5 2512.5 0 1 2 3 4 5 6 7 8 9 10 Seat rentals http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg
  21. 21. http://www.imdb.com/media/rm3768753408/tt0073195
  22. 22. http://www.flickr.com/photos/kapungo/2287237966
  23. 23. 100001000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings
  24. 24. http://www.flickr.com/photos/25159787@N07/3766111564
  25. 25. http://www.flickr.com/photos/wheressteve/3284532080
  26. 26. http://www.flickr.com/photos/wtlphotos/1086968783
  27. 27. 100001000 100 10 1 Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Ice cream consumption Drownings Temperature
  28. 28. http://www.flickr.com/photos/stuttermonkey/57096884
  29. 29. http://www.flickr.com/photos/germanuncut77/3785152581
  30. 30. http://www.flickr.com/photos/fasteddie42/2421039207
  31. 31. Correlated CausalTwo variables that An independentchange in similar factor that directlyways , perhaps impacts abecause they’re dependent one.linked to somethingCausalelse. Summer al Ca us us Ca Correlated al Drowning Ice creamconsumption
  32. 32. http://www.flickr.com/photos/bootbearwdc/1243690099/
  33. 33. Causality is a superpower, because it lets youchange the future. Correlation lets you Causality lets you predict the future change the future“I will have 420 “If I can make moreengaged users and first-time visitors stay75 paying customers on for 17 minutes Inext month.” will increase sales in 90 days.” Optimize theFind correlation Test causality causal factor
  34. 34. Leading LaggingNumber today that Historical metric thatshows metric shows how you’retomorrow—makes doing—reports thethe news. news.
  35. 35. What mode of e-commerce are you? How many of your customers Then you are in Your customers You are just Focus on buy a second this mode will buy from you liketime in 90 days? Low CAC, 1-15% Acquisition Once 70% high of retailers checkout 15-30% Hybrid 2-2.5 20% Increasing per year of retailers returns Loyalty, >30% Loyalty >2.5 10% inventory per year of retailers expansion (Thanks to Kevin Hilstrom for this.)
  36. 36. • A Facebook user reaching 7 friends within 10 days of signing up (Chamath Palihapitiya)• If someone comes back to Zynga a day after signing up for a game, they’ll probably become an engaged, paying user (Nabeel Hyatt)• A Dropbox user who puts at least one file in one folder on one device (ChenLi Wang)• Twitter user following a certain number of people, and a certain percentage of those people following the user back (Josh Elman)• A LinkedIn user getting to X connections in Y days (Elliot Schmukler) (These are also great segments to analyze.) (from the 2012 Growth Hacking conference)
  37. 37. So how do you test things? Segmentation.http://www.flickr.com/photos/zlakfoto/5294803278/
  38. 38. Segments, cohorts, A/B, and multivariates Cohort: Comparison of similar groups along a timeline. Segment: A/B test: ☀ Multivariate Cross-sectional ☀ Changing one analysis comparison of all thing (i.e. color) ☁ Changing several people divided by and measuring ☀ things at once to some attribute ☁ the result (i.e. see which correlates ☁(age, gender, etc.) revenue.) with a result.
  39. 39. Why use cohorts? Here’s an example. Is this   January February March April May companygrowing or Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50stagnating? Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5How about February $6 $4 $2 $1 now? March $7 $6 $5 April   $8 $7 May       $9
  40. 40. Why use cohorts? Here’s an example. Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5Look at the February $6 $4 $2 $1  same data in cohorts March $7 $6 $5     April $8 $7       May $9         Averages $7 $5 $3 $1 $0.5
  41. 41. The Lean Analytics Framework.
  42. 42. Eric Ries’Three engines Stickiness Virality PriceApproach Keep people Make people Spend revenue coming back. invite friends. getting customers.Math that Get customers How many they Customers are matters faster than you tell, how fast worth more than lose them. they tell them. they cost to get.
  43. 43. The five Stages of Lean Analytics The business you’re in E- 2-sided Mobile User-gen SaaS Media commerce market app content EmpathyThe stage you’re at One Metric Stickiness Virality Revenue That Matters. Scale
  44. 44. Mobile app model:Localmind hacks Twitter• Stage: Empathy• Model: UGC/mobile• Real-time question and answer platform tied to locations.• Needed to find out if a core behavior—answering questions about a place— happened enough to make the business real
  45. 45. Localmind hacks Twitter• Before writing a line of code, Localmind was concerned that people would never answer questions. • This was their biggest risk: if questions went unanswered users would have a terrible experience and stop using Localmind.• Ran an experiment on Twitter • Tracked geolocated tweets in Times Square • Sent @ messages to people who had just tweeted, asking questions about the area: how busy is it; is the subway running on time; is something open; etc.• The response rate to their tweeted questions was very high. • Good enough proxy to de-risk the solution, and convince the team and investors that it was worth building Localmind.
  46. 46. Stickiness stage:WP Engine discovers the2% cancellation rate• Stage: Stickiness• Model: SaaS• Wordpress hosting company founded in July 2010, it raised $1.2M in November 2011
  47. 47. WP-Engine discovers the 2%cancellation rate• All companies have cancellations, but founder Jason Cohen was alarmed that he was losing a quarter of customers every year.• Jason called customers himself. “Not everyone wanted to speak with me, but enough people were willing to talk, even after they had left, that I learned a lot about why they were leaving.”• Asked around. Turns out 2% is best case for most hosting companies.• Without this, the company would have been getting diminishing returns over- optimizing churn; instead, they could focus on maximizing revenues or lowering acquisition costs.
  48. 48. Virality stage:qidiq streamlines invites• Stage: Virality• Model: SaaS• Tool to poll small groups, built in the Year One Labs accelerator
  49. 49. Initial design Redesigned workflow Survey owner adds recipient to group Survey owner adds recipient to group 70-90% RESPONSE RATE Survey owner asks question Survey owner asks question Recipient gets invite Recipient reads survey question10-25% RESPONSE RATE Recipient installs mobile app Recipient responds to question Recipient sees survey results Recipient creates account, profile Recipient can edit profile, view past (Later, if needed…) questions, etc. Recipient visits website Recipient reads survey question Recipient has no password! Recipient responds to question Recipient does password recovery Recipient sees survey results One-time link sent to email Recipient creates password Recipient can edit profile, view past questions, etc.
  50. 50. Revenue stage:Backupify’s customerlifecycle• Stage: Scale• Model: SaaS• Leading backup provider for cloud based data.• The company was founded in 2008 by Robert May and Vik Chadha• Has gone on to raise $19.5M in several rounds of financing.
  51. 51. Shifting to Customer AcquisitionPayback as a key metric• Initially focused on site visitors• Then focused on trials• Then switched to signups• Today, MRR• In early 2010, CAC was $243 and ARPU was only $39 • Pivoted to target business users • CLV-to-CAC today is 5-6x• Now they track Customer Acquisition Payback • Target is less than 12 months
  52. 52. What these have in common:The Lean Analytics CycleSuccess! Pick a KPI Draw a line in the sand Pivot or give up Draw a new line Find a potential Try again improvementDid we movethe needle? Without With data: data: make a find a good guess commonality Design a test Measure the results Hypothesis Make changes in production
  53. 53. What’s your OMTM? E- 2-sided Mobile User-gen SaaS Media commerce market app content Empathy Interviews; qualitative results; quantitative scoring; surveys Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,Stickiness conversion listings churn churn, virality spam returns CAC, shares, Inherent WoM, app Invites, Content Virality reactivation SEM, sharing virality, CAC ratings, CAC sharing virality, SEM (Money from transactions) (Money from active users) (Money from ad clicks) Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs Affiliates, Other API, magic Spinoffs, Analytics, Syndication, Scale white-label verticals #, mktplace publishers user data licenses
  54. 54. Choose only one metric.
  55. 55. Yes, one metric.
  56. 56. It will soon change.
  57. 57. In a startup,focus is hard to achieve.
  58. 58. Having only one metricaddresses this problem.
  59. 59. Metrics are like squeeze toys.http://www.flickr.com/photos/connortarter/4791605202/
  60. 60. ARCHIMEDES HAD TAKENBATHS BEFORE.
  61. 61. Once, a leader convinced others inthe absence of data.
  62. 62. Now, a leader knows whatquestions to ask.
  63. 63. Ben Yoskovitzbyosko@gmail.com@byoskoAlistair Crollacroll@gmail.com@acroll

×