Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lean analytics: Five lessons beyond the basics

13,929 views

Published on

Slides from an August 6, 2013 Webinar with Eric Ries.

Published in: Business, Technology
  • Be the first to comment

Lean analytics: Five lessons beyond the basics

  1. 1. www.leananalyticsbook.com @leananalytics @byosko | @acroll Lean Analytics Use data to build a better business faster.
  2. 2. A (really) quick intro to Lean principles
  3. 3. The basic Lean message is learn and adapt, fast.
  4. 4. Kevin Costner is a lousy entrepreneur. Don’t sell what you can make. Make what you can sell.
  5. 5. In a startup, the purpose of analytics is to iterate to a product/market fit before the money runs out.
  6. 6. Everyone’s idea is the best right? People love this part! (but that’s not always a good thing) This is where things fall apart. No data, no learning.
  7. 7. The five Stages of Lean Analytics Empathy Stickiness Virality Revenue Scale Thestageyou’reat E-commerce SaaS Media Mobile app User-gen content 2-sided market The business you’re in One Metric That Matters.
  8. 8. If it won’t change how you behave, it’s a bad metric. http://www.flickr.com/photos/circasassy/7858155676/
  9. 9. Five lessons beyond the basics • Mining your existing data • A new product every day • Are you at your local maximum? • It’s different when you’re big • Build analytics into how you operate
  10. 10. Mining your existing data Moving beyond guesswork.
  11. 11. Identify a key business problem, pick the OMTM, draw a line in the sand, and get started.
  12. 12. Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: find a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI The Lean Analytics Cycle
  13. 13. Do hosts withDo hosts with professionalprofessional photography getphotography get more business?more business? Do hosts withDo hosts with professionalprofessional photography getphotography get more business?more business? Airbnb experiments...
  14. 14. Professional photography helps Airbnb’s business Gut instinct Concierge MVP 20 photographers in the field Test results Two to three times more bookings! Back to the beginning Use additional data to keep experimenting
  15. 15. 5,000 shoots/month in Feb. 2012
  16. 16. Hang on a second...
  17. 17. Really? Professional photography helps Airbnb’s business Gut instinct
  18. 18. Draw a new line Pivot or give up Try again Success! Did we move the needle? Measure the results Make changes in production Design a test Hypothesis With data: find a commonality Without data: make a good guess Find a potential improvement Draw a linePick a KPI Remember this?
  19. 19. “Gee, those houses that do well look really nice.” Maybe it’s the camera. “Computer: What do all the highly rented houses have in common?” Camera model. With data: find a commonality Without data: make a good guess
  20. 20. The only thing worse than bad feedback is no feedback at all. - Dave McClure, Startupfest 2012
  21. 21. Because it gives you nothing to analyze.
  22. 22. Virality stage: Circle of Moms finds an engaged market • Stage: Stickiness • Model: UGC • Launched as Circle of Friends in 2007, it was a way for small groups to interact atop Facebook’s platform; but when engagement wasn’t good enough, the founders decided to dig deeper.
  23. 23. The problem: Not enough engagement • Too few people were actually using the product • Less than 20% of any circles had any activity after their initial creation • A few million monthly uniques from 10M registered users, but no sustained traction
  24. 24. A new product every day Why cohorts matter more than you think.
  25. 25. Segments, cohorts, A/B, and multivariates Segment: Cross-sectional comparison of all people divided by some attribute (age, gender, etc.) ☀ ☁ Cohort: Comparison of similar groups along a timeline. A/B test: Changing one thing (i.e. color) and measuring the result (i.e. revenue.) Multivariate analysis: Changing several things at once to see which correlates with a result. ☀ ☁ ☀ ☁ JanJan FebFeb MarMar AprApr MayMay
  26. 26. Why use cohorts? Here’s an example. January February March April May Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50 Is this company growing or stagnating? Cohort 1 2 3 4 5 How about this one? January $5 $3 $2 $1 $0.5 February $6 $4 $2 $1 March $7 $6 $5 April $8 $7 May $9
  27. 27. Two views of the same company Cohort 1 2 3 4 5 January $5 $3 $2 $1 $0.5 February $6 $4 $2 $1 March $7 $6 $5 April $8 $7 May $9 Averages $7 $5 $3 $1 $0.5 Look at the same data in cohorts
  28. 28. Are we at a local maximum? You need optimization and inspiration.
  29. 29. The problem with local maxima (http://www.slideshare.net/bokardo/metricsdriven-design-4317168)
  30. 30. The vertebrate eye is “backwards and upside down”. Light travels through cornea, lens, aqueous fluid, blood vessels, ganglion cells, amacrine cells, horizontal cells, and bipolar cells. The cephalopod eye is constructed the “right way out”, with the nerves attached to the rear of the retina. No blind spot. Extension of the brain Invagination of the head
  31. 31. It’s different when you’re big Tilting at the corporate windmill.
  32. 32. “A startup is an organization designed to search for a sustainable, repeatable business model.” - Steve Blank Remember when I said,
  33. 33. “A big company is an organization designed to perpetuate a business model.” - Me
  34. 34. In other words, if your job is change you have your work cut out for you.
  35. 35. Span of control and the railroads The story of Daniel C. McCallum
  36. 36. http://www.flickr.com/photos/art_es_anna/288880795/
  37. 37. Barnes and Noble tried pretty hard. • $1B invested in Nook • $475M operating loss in April • CEO gone
  38. 38. The Skunk Works
  39. 39. Since large organizations make data that perpetuates things, you have to willfully ignore common wisdom at the outset. http://www.flickr.com/photos/jb912/8940173789
  40. 40. Disruption may not fit neatly into an existing part of the organization, so you need to incubate it internally for a while.
  41. 41. Unlike a VC or startup, when the initiative fails the organization still learns.
  42. 42. Big organizations might not innovate at speed—but they innovate at scale.
  43. 43. The Lean Analytics lifecycle for an Intrapreneur Empathy Find problems; don’t test demand. Skip the business case, do analytics Entitled, aggrieved customers Stickiness Know your real minimum based on expectations, regulations Hidden “must haves”, feature creep Virality Build inherent virality in from the start; attention is the new currency Luddites who don’t understand sharing Revenue Consider the ecosystem, channels, and established agreements Channel conflict, resistance, contracts Scale Hand the baton to others gracefully Hating what happens to your baby Stage Do this Fear this Beforehand Get buy-in Political fallout
  44. 44. A large organization needs a portfolio of investments, judged by different criteria. Treasury bonds Venture capital
  45. 45. A large organization needs a portfolio of investments, judged by different criteria. Safe Predictable Evolutionary Achieve the business plan Risky Uncertain Revolutionary Discover the business model Times of peace Times of war
  46. 46. Acquisition or disruption? Innovate fast: nothing to lose Innovate slowly: conform to brand, legislation, approvals Found P/M fit Found P/M fit Integration complexity, founder churn, indigestion Biz case Scale Scale Biz model VC pitch
  47. 47. Moving from box to box • How businesses think about products or companies • Lean is about moving up and to the right
  48. 48. Frame the project as a learning exercise; product creation is almost accidental. Learn it’s a failure Learn it’s a success Industry knowledge Industry knowledge Possible product Business plan metrics here Market Business model metrics here Lean product Lines in the sand here
  49. 49. Build analytics into how you operate the business Three threes and the Problem-Solution Canvas
  50. 50. Three threes Three assumptions What big bets are you making? •“People will answer questions” •“Organizers are frustrated with how to run conferences” •“We’ll make money from parents” •“Amazon is reliable enough for our users.” Three actions to take What are you doing to make these assumptions happen (or identify they’re wrong and change course?) •Product enhancements •Marketing strategies Three experiments to run What are you going to learn today? •Feature tests •Continuous deployment •A/B testing •Customer survey
  51. 51. Three threes Monthly Weekly Daily Board, investors, founders Executive team Employees Strategy Tactics Execution Three assumptions Three actions to take Three experiments to run
  52. 52. Three threes Get more people Increase answer % Test better questions Change the UI Test timings Questions from peers Many people will answer questions Three assumptions Three actions to take Three experiments to run
  53. 53. In the end, be subversive.
  54. 54. http://www.flickr.com/photos/bootbearwdc/1243690099/
  55. 55. Alistair Croll acroll@gmail.com @acroll Ben Yoskovitz byosko@gmail.com @byosko

×