Download as PDF, PPTX
Uploaded byLean Startup Co.
Dan mc kinley_final
This document summarizes Dan McKinley's talk at the Lean Startup Conference about using data-driven decision making at Etsy. Some key points: 1) In the early days of Etsy, product decisions were based on "coolest ideas" but later shifted to using A/B testing and metrics to evaluate ideas. 2) Examples show how estimating metrics like conversion rates and order values can help determine if ideas are worth pursuing. 3) This approach of using basic math and estimates led to systemic improvements in Etsy's sales and product decisions over time.
More Related Content
Dan mc kinley_final
- 1. Data Driven DoneRight Dan McKinley @mcfunley mcfunley.com Lean Startup Conference -‐ San Francisco, CA December 10th, 2014
- 2.
- 4. 2007 2003 2014 1998
- 5. 2007 2003 2014 1998
- 6. Handmade & vintagemarketplace. $1.3 Billion in Gross Sales in 2013. 600ish Employees.
- 7. Less than 20people in 2007.
- 8. photo: Ian Normanon Flickr.
- 9. “What do webuild?” photo: Wilhelm Joys Andersen on Flickr.
- 10. “We should bedata-‐driven.”
- 11. We should bedata driven. This is data. ∴ we should be driven by this.
- 12. Data-‐Driven PrioriHes Thisis a thing that’s possible. ! I fell ass-‐backwards into learning a thing or two about it.
- 13.
- 14.
- 15.
- 18. Ideas that sound awesome are awesome. -‐ Etsy product philosophy in the early days. photo: srslyguys on Flickr.
- 19. photo: JusDn Brockieon Flickr.
- 20. Products in theFree-‐Growth Universe (They RULE) Released a thing Released a thing Released a thing Oh man that thing we released, I tell you what
- 21.
- 22. Product Release Plan 1. Build feature. 2. Rent warehouse for launch party. 3. Release feature. 4. Launch party. 5.Wait 20 months. 6.Delete feature for lack of usage. photo: James Willamor on flickr.
- 23.
- 24.
- 25. A/B tesHng: asoluHon?
- 27. Product selecHon: sHll “coolest idea.” photo: nanpalmero on Flickr.
- 28. nanpalmero on Flickr. Product selecHon: sHll “coolest idea.” But the measurement had changed! photo: masterpunk.etsy.com
- 29. “Only Hope stayed under the rim of the jar.” photo: ketrin1407 on Flickr.
- 30. Something that Actually Started to Work
- 31. Time 2005 Idea Code Release
- 32. 2005 2010 Idea Code Release Idea Code A/B test Release?
- 33. “Oh, crap” 20052010 Idea Code Release Idea Code A/B test Release?
- 34. Wasted effort (maybe). THE HELL WITH IT! 2005 2010 Idea Code Release Idea Code A/B test Release?
- 35. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release
- 36. Use data here. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release
- 37. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Possibly quite crappy.
- 38. Made less crappyhere. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Possibly quite crappy.
- 39. 2005 2010 2012 Idea Code Release Idea Code A/B test Release? Idea Validate Prototype A/B Test Refinement A/B test Release Maybe give up. Maybe give up. Maybe give up (hopefully not).
- 40.
- 41. Here are tworeal ideas. Neither should sound completely nuts. photo: machineisorganic on Flickr.
- 42. Idea #1 Let’sbuild a landing page for local furniture.
- 44. Step 1: Whowould see it?
- 45. Lis$ng Search Shop Home Other “A significant fracDon” Traffic Percent (%) Browse
- 46. Lis$ng Search Shop Home Other Traffic Furniture Percent (%) Browse
- 47. Lis?ng Search Nota significant fracDon. Shop Home Other Traffic Items sold Percent (%) Browse
- 49. Step 2: Whatwould change imply?
- 50. visits × conversionrate × order value × liH ≈ ?
- 51. visits × conversionrate × order value × liH ≈ ? Es#mated Modeled
- 52. 50K × 0.25%× $400 × lig ≈ ? +2%? +1%? (fake numbers, by the way) $500,000 $375,000 $250,000 $125,000 $0 +0.5%? One Year
- 53. 50K × 0.25%× $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 +2%? +1%? +0.5%? One Year (fake numbers, by the way)
- 54. 50K × 0.25%× $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 +2% ≈ $360,000 +1%? +0.5%? (fake numbers, by the way) One Year
- 55. 50K × 0.25%× $400 × lig ≈ ? $500,000 $375,000 $250,000 $125,000 $0 $360,000 $13K (our cut) (fake numbers, by the way) One Year
- 56. Verdict: Let’s dosomething else.
- 57. Idea #2 Let’semail people that give up in the middle of purchases.
- 59. users × ordervalue × resurrec?on rate ≈ ?
- 60. 20K × $40× resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 +5%? +2%? +1%? One Year (fake numbers, again)
- 61. 20K × $40× resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 +5%? +2%? +1%? One Year (fake numbers, again)
- 62. 20K × $40× resurrecHon rate ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 $3,000,000! One Year (fake numbers, again)
- 63. 20K × $40× resurrec?on rate × 3.5% ≈ ? $20,000,000 $15,000,000 $10,000,000 $5,000,000 $0 $3,000,000 One Year $100K! (our cut) (fake numbers, again)
- 64.
- 65. Abributed Source ofEtsy’s Sales Other My email (epic)
- 66. Systemic improvement throughbasic arithmeHc!
- 67.
- 69. Who? ! Howmany?
- 70. Close is goodenough.
- 71.
- 72. Startup Mature Company Moonshots Strategic Projects Vebed OpportuniDes h/t Marc Hedlund
- 73.
- 74. “What do webuild?” photo: Wilhelm Joys Andersen on Flickr.
- 75. Many product ideas sound awesome. photo: mindonfire on Flickr.
- 76. “Upon what meatdoth this our Caesar feed, that he is grown so great?”
- 77. “He who refusesto do arithme;c is doomed to talk nonsense.” -‐ John McCarthy
- 78.