Dan siroker data testing and business business of software conference 2013

1. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO

2. ‘05 ‘06 ‘07 ‘08 ’09 ’10

3. ‘05 ‘06 ‘07 ‘08 ’09 ’10

4. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4

5. Millions $7 Team: 4 Customers: 644 Annual Run Rate: $0.1m $6 $5 $4 $3 $2 $1 Dan Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4

6. Millions $7 Team: 10 (+150%) Customers: 1,508 (+134%) Annual Run Rate: $1.2m(+982%) $6 $5 $4 $3 $2 Julio Ryan M $1 Ricky Meng Xiao Dan Mat Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4

7. Millions $7 Team: 42 (+320%) Customers: 3,460 (+129%) Annual Run Rate: $7.6m (+533%) $6 $5 Steve Tyle Jeff Z Arthu Amina Greg Sandy $4 Brendan Sophie Dogan Sonesh Shanann Tommy $3 Dan G Cara Byron Mik Alan Eddie Jodie $2 Dae-Ho Cyrus Julio Ryan M $1 Lauren Ashley Ricky Meng Xiao Dan Mat Cristian Travis Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Katy Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Ryan L Ian Jonathan Q4 Joanna Jon

10. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Q3 Q4 2013 Q2 Q1 Q3 Q4

11. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Team: 42 Tyle Jeff Z Arthu Amina Greg Sophie Dogan Cara Alan Dan G Shanann Byron Eddie Sandy Dae-Ho Cyrus Sonesh Brendan Cristian Mik Ricky Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Lauren Ashley Tommy Ryan M Jodie Mat Meng Xiao Julio Katy Travis Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Ryan L Q3 Q4 2013 Q2 Joanna Q1 Ian Jon Q3 Q4

12. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Steve Team: 130 (+210%) Tyle Marius Adam Bo Evan Junan Ash Grace Anshu Lucas Benson Bryce Darwish Jeff Z Arthu David Jaron Richar Tammie Khattaab Laura Asia Amanda Amina Pamela Brian Greg Nikhil Martjin Sophie Amelia Dogan Gary Marco Natarajan Ben Kasra Katie Andrea Daniel SG Dae-Ho Omari Andy Eric Andy Mustafa Ranjit Chris Waylon Pedro Bradley Neha Lauren Adrian Chrix Cristian Mik Ricky Robin Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Ashley Tommy Ryan M Mat Julio Katy Travis Lisa Jamie Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Nick Miyo Jodie Meng Xiao Ryan L Q3 Becca Oren Cyrus Brendan Megan Byron Eddie Sonesh Hemant Shanann Brett Sandy Ryan Michael James Etienne Alan Lacy Dan G Luke Cara Westly Aaron Adam Q4 2013 Q2 Helen Joanna Q1 Jen Ian Jon Nick Fabian Saum Courtney Dan Yonat Kent Ural Eduar Toby Oliver Grant Q3 Cate Christop Julia Q4 Danielle Tai Jonathan Chris Chris Rama

13. How?

14. Limited Life Experience

15. Limited Life Experience + Overgeneralization

16. Limited Life Experience + Overgeneralization = Advice -Paul Buchheit

17. Lesson #1: Deﬁne success.

18. Lesson #1: Deﬁne success .

19. Lesson #1: Deﬁne quantiﬁable success metrics.

20. visitors per day Visit sign up rate $ per recipient Email $

21. visitors per day Visit sign up rate $ per recipient Email $

22. Media Button

23. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springﬁeld Video 6. Sam’s Video

24. Button: “Sign Up”

25. Button: “Learn More”

26. Button: “Join Us Now”

27. Button: “Sign Up Now”

29. Media: “Get Involved”

30. Media: “Family”

31. Media: “Change”

32. Media: “Barack’s Video”

33. Media: “Springﬁeld Video”

34. Media: “Sam’s Video”

36. Splash page experiment

39. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000

42. Lesson #1: Deﬁne quantiﬁable success metrics.

43. Lesson #2: Less is more. Reduce choices.

44. vs. +15% conversion rate +10% $ per visitor

45. vs. +20% submissions

46. vs. +16.8% $ per visitor

47. vs. +43.8% purchases

48. Lesson #2: Less is more. Reduce choices.

49. Lesson #3: Words matter. Focus on your call to action.

50. Donation button experiment Variations

51. Donation button experiment Variations Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%

52. vs. +14.6% clicks

53. vs. +15.75% $ per visitor

54. Lesson #3: Words matter. Focus on your call to action.

55. Lesson #4: Seek the global maximum.

56. Source: http://blog.intercom.io/criticism-and-two-way-streets/

57. vs.

58. Source: http://blog.netﬂix.com/2011/06/new-look-and-feel-for-netﬂix-website.html

59. vs. +0.5% retention +1.4% engagement

60. vs. +600% engagement

61. vs. +49% conversion rate from 240 A/B tests

62. Lesson #4: Seek the global maximum.

63. Lesson #5: Start today.

64. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/

65. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/

66. 5,000+ Happy Customers!

67. Lesson #5: Start today.

68. Lessons Learned 1. Deﬁne quantiﬁable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Start today.

69. One more thing...

70. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO

Dan siroker data testing and business business of software conference 2013

You are reading a preview.

Check these out next

Dan siroker data testing and business business of software conference 2013

More Related Content

Similar to Dan siroker data testing and business business of software conference 2013 (20)

More from Business of Software Conference (20)

Recently uploaded (20)

Dan siroker data testing and business business of software conference 2013