This document summarizes Dan McKinley's talk at the Lean Startup Conference about using data-driven decision making at Etsy. Some key points: 1) In the early days of Etsy, product decisions were based on "coolest ideas" but later shifted to using A/B testing and metrics to evaluate ideas. 2) Examples show how estimating metrics like conversion rates and order values can help determine if ideas are worth pursuing. 3) This approach of using basic math and estimates led to systemic improvements in Etsy's sales and product decisions over time.

1. Data Driven Done Right
Dan McKinley @mcfunley mcfunley.com
Lean Startup Conference -­‐ San Francisco, CA
December 10th, 2014

2. IntroducHons

4. 2007
2003
2014
1998

5. 2007
2003
2014
1998

6. Handmade & vintage marketplace.
$1.3 Billion in Gross Sales in 2013.
600ish Employees.

7. Less than 20 people in 2007.

8. photo: Ian Norman on Flickr.

9. “What do we build?”
photo: Wilhelm Joys Andersen on Flickr.

10. “We should be data-­‐driven.”

11. We should be data driven.
This is data.
∴ we should be driven by this.

12. Data-­‐Driven PrioriHes
This is a thing that’s possible.
!
I fell ass-­‐backwards into learning a thing
or two about it.

13. War Stories

14. In the Beginning

15. Skateboard ramp demoli;on, 2007.

18. Ideas that sound
awesome are awesome.
-­‐ Etsy product philosophy in the early days.
photo: srslyguys on Flickr.

19. photo: JusDn Brockie on Flickr.

20. Products in the Free-­‐Growth Universe
(They RULE)
Released a thing
Released a thing
Released a thing
Oh man that thing
we released, I tell
you what

21. Pimply, Awkward
Company

22. Product Release Plan
1. Build feature.
2. Rent warehouse for launch party.
3. Release feature.
4. Launch party.
5.Wait 20 months.
6.Delete feature for lack of usage.
photo: James Willamor on flickr.

23. 2008

24. 2011

25. A/B tesHng: a soluHon?

27. Product selecHon: sHll
“coolest idea.”
photo: nanpalmero on Flickr.

28. nanpalmero on Flickr.
Product selecHon: sHll
“coolest idea.”
But the measurement had changed!
photo: masterpunk.etsy.com

29. “Only Hope stayed
under the rim of the jar.”
photo: ketrin1407 on Flickr.

30. Something that Actually
Started to Work

31. Time
2005
Idea
Code
Release

32. 2005 2010
Idea
Code
Release
Idea
Code
A/B test
Release?

33. “Oh, crap”
2005 2010
Idea
Code
Release
Idea
Code
A/B test
Release?

34. Wasted effort (maybe).
THE HELL WITH IT!
2005 2010
Idea
Code
Release
Idea
Code
A/B test
Release?

35. 2005 2010 2012
Idea
Code
Release
Idea
Code
A/B test
Release?
Idea
Validate
Prototype
A/B Test
Refinement
A/B test
Release

36. Use data here.
2005 2010 2012
Idea
Code
Release
Idea
Code
A/B test
Release?
Idea
Validate
Prototype
A/B Test
Refinement
A/B test
Release

37. 2005 2010 2012
Idea
Code
Release
Idea
Code
A/B test
Release?
Idea
Validate
Prototype
A/B Test
Refinement
A/B test
Release
Possibly quite
crappy.

38. Made less
crappy here.
2005 2010 2012
Idea
Code
Release
Idea
Code
A/B test
Release?
Idea
Validate
Prototype
A/B Test
Refinement
A/B test
Release
Possibly quite
crappy.

39. 2005 2010 2012
Idea
Code
Release
Idea
Code
A/B test
Release?
Idea
Validate
Prototype
A/B Test
Refinement
A/B test
Release
Maybe give up.
Maybe give up.
Maybe give up
(hopefully not).

40. Examples

41. Here are two real ideas.
Neither should sound completely nuts.
photo: machineisorganic on Flickr.

42. Idea #1
Let’s build a landing page for local furniture.

44. Step 1:
Who would see it?

45. Lis$ng
Search
Shop
Home
Other
“A significant fracDon”
Traffic
Percent (%)
Browse

46. Lis$ng
Search
Shop
Home
Other
Traffic
Furniture
Percent (%)
Browse

47. Lis?ng
Search
Not a significant fracDon.
Shop
Home
Other
Traffic
Items sold
Percent (%)
Browse

49. Step 2:
What would change imply?

50. visits × conversion rate × order value × liH ≈ ?

51. visits × conversion rate × order value × liH ≈ ?
Es#mated Modeled

52. 50K × 0.25% × $400 × lig ≈ ?
+2%?
+1%?
(fake numbers, by the way)
$500,000
$375,000
$250,000
$125,000
$0
+0.5%?
One Year

53. 50K × 0.25% × $400 × lig ≈ ?
$500,000
$375,000
$250,000
$125,000
$0
+2%?
+1%?
+0.5%?
One Year
(fake numbers, by the way)

54. 50K × 0.25% × $400 × lig ≈ ?
$500,000
$375,000
$250,000
$125,000
$0
+2% ≈ $360,000
+1%?
+0.5%?
(fake numbers, by the way)
One Year

55. 50K × 0.25% × $400 × lig ≈ ?
$500,000
$375,000
$250,000
$125,000
$0
$360,000
$13K
(our cut)
(fake numbers, by the way)
One Year

56. Verdict:
Let’s do something else.

57. Idea #2
Let’s email people that give up in the
middle of purchases.

59. users × order value × resurrec?on rate ≈ ?

60. 20K × $40 × resurrecHon rate ≈ ?
$20,000,000
$15,000,000
$10,000,000
$5,000,000
$0
+5%?
+2%?
+1%?
One Year
(fake numbers, again)

61. 20K × $40 × resurrecHon rate ≈ ?
$20,000,000
$15,000,000
$10,000,000
$5,000,000
$0
+5%?
+2%?
+1%?
One Year
(fake numbers, again)

62. 20K × $40 × resurrecHon rate ≈ ?
$20,000,000
$15,000,000
$10,000,000
$5,000,000
$0
$3,000,000!
One Year
(fake numbers, again)

63. 20K × $40 × resurrec?on rate × 3.5% ≈ ?
$20,000,000
$15,000,000
$10,000,000
$5,000,000
$0
$3,000,000
One Year
$100K!
(our cut) (fake numbers, again)

64. Verdict:
Let’s build it!

65. Abributed Source of Etsy’s Sales
Other
My email
(epic)

66. Systemic improvement
through basic arithmeHc!

67. GeneralizaHons

69. Who?
!
How many?

70. Close is
good enough.

71. YMMV

72. Startup Mature Company
Moonshots Strategic Projects Vebed OpportuniDes
h/t Marc Hedlund

73. Conclusions

74. “What do we build?”
photo: Wilhelm Joys Andersen on Flickr.

75. Many product ideas
sound awesome.
photo: mindonfire on Flickr.

76. “Upon what meat doth this
our Caesar feed, that he is
grown so great?”

77. “He who refuses to do
arithme;c is doomed
to talk nonsense.”
-­‐ John McCarthy

78. mcfunley@gmail.com
mcfunley.com
@mcfunley