Dan siroker data testing and business business of software conference 2013

537 views
446 views

Published on

Dan Siroker, CEO, Optimizely talks about Data Testing and Business Business of Software Conference 2013

Published in: Business, Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
537
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Dan siroker data testing and business business of software conference 2013

  1. 1. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO
  2. 2. ‘05 ‘06 ‘07 ‘08 ’09 ’10
  3. 3. ‘05 ‘06 ‘07 ‘08 ’09 ’10
  4. 4. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  5. 5. Millions $7 Team: 4 Customers: 644 Annual Run Rate: $0.1m $6 $5 $4 $3 $2 $1 Dan Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  6. 6. Millions $7 Team: 10 (+150%) Customers: 1,508 (+134%) Annual Run Rate: $1.2m(+982%) $6 $5 $4 $3 $2 Julio Ryan M $1 Ricky Meng Xiao Dan Mat Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  7. 7. Millions $7 Team: 42 (+320%) Customers: 3,460 (+129%) Annual Run Rate: $7.6m (+533%) $6 $5 Steve Tyle Jeff Z Arthu Amina Greg Sandy $4 Brendan Sophie Dogan Sonesh Shanann Tommy $3 Dan G Cara Byron Mik Alan Eddie Jodie $2 Dae-Ho Cyrus Julio Ryan M $1 Lauren Ashley Ricky Meng Xiao Dan Mat Cristian Travis Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Katy Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Ryan L Ian Jonathan Q4 Joanna Jon
  8. 8. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  9. 9. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q1 Q2 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  10. 10. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Q3 Q4 2013 Q2 Q1 Q3 Q4
  11. 11. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Team: 42 Tyle Jeff Z Arthu Amina Greg Sophie Dogan Cara Alan Dan G Shanann Byron Eddie Sandy Dae-Ho Cyrus Sonesh Brendan Cristian Mik Ricky Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Lauren Ashley Tommy Ryan M Jodie Mat Meng Xiao Julio Katy Travis Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Ryan L Q3 Q4 2013 Q2 Joanna Q1 Ian Jon Q3 Q4
  12. 12. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Steve Team: 130 (+210%) Tyle Marius Adam Bo Evan Junan Ash Grace Anshu Lucas Benson Bryce Darwish Jeff Z Arthu David Jaron Richar Tammie Khattaab Laura Asia Amanda Amina Pamela Brian Greg Nikhil Martjin Sophie Amelia Dogan Gary Marco Natarajan Ben Kasra Katie Andrea Daniel SG Dae-Ho Omari Andy Eric Andy Mustafa Ranjit Chris Waylon Pedro Bradley Neha Lauren Adrian Chrix Cristian Mik Ricky Robin Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Ashley Tommy Ryan M Mat Julio Katy Travis Lisa Jamie Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Nick Miyo Jodie Meng Xiao Ryan L Q3 Becca Oren Cyrus Brendan Megan Byron Eddie Sonesh Hemant Shanann Brett Sandy Ryan Michael James Etienne Alan Lacy Dan G Luke Cara Westly Aaron Adam Q4 2013 Q2 Helen Joanna Q1 Jen Ian Jon Nick Fabian Saum Courtney Dan Yonat Kent Ural Eduar Toby Oliver Grant Q3 Cate Christop Julia Q4 Danielle Tai Jonathan Chris Chris Rama
  13. 13. How?
  14. 14. Limited Life Experience
  15. 15. Limited Life Experience + Overgeneralization
  16. 16. Limited Life Experience + Overgeneralization = Advice -Paul Buchheit
  17. 17. Lesson #1: Define success.
  18. 18. Lesson #1: Define success .
  19. 19. Lesson #1: Define quantifiable success metrics.
  20. 20. visitors per day Visit sign up rate $ per recipient Email $
  21. 21. visitors per day Visit sign up rate $ per recipient Email $
  22. 22. Media Button
  23. 23. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  24. 24. Button: “Sign Up”
  25. 25. Button: “Learn More”
  26. 26. Button: “Join Us Now”
  27. 27. Button: “Sign Up Now”
  28. 28. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  29. 29. Media: “Get Involved”
  30. 30. Media: “Family”
  31. 31. Media: “Change”
  32. 32. Media: “Barack’s Video”
  33. 33. Media: “Springfield Video”
  34. 34. Media: “Sam’s Video”
  35. 35. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  36. 36. Splash page experiment
  37. 37. Splash page experiment
  38. 38. Splash page experiment
  39. 39. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  40. 40. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  41. 41. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  42. 42. Lesson #1: Define quantifiable success metrics.
  43. 43. Lesson #2: Less is more. Reduce choices.
  44. 44. vs. +15% conversion rate +10% $ per visitor
  45. 45. vs. +20% submissions
  46. 46. vs. +16.8% $ per visitor
  47. 47. vs. +43.8% purchases
  48. 48. Lesson #2: Less is more. Reduce choices.
  49. 49. Lesson #3: Words matter. Focus on your call to action.
  50. 50. Donation button experiment Variations
  51. 51. Donation button experiment Variations Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%
  52. 52. vs. +14.6% clicks
  53. 53. vs. +15.75% $ per visitor
  54. 54. Lesson #3: Words matter. Focus on your call to action.
  55. 55. Lesson #4: Seek the global maximum.
  56. 56. Source: http://blog.intercom.io/criticism-and-two-way-streets/
  57. 57. vs.
  58. 58. Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html
  59. 59. vs. +0.5% retention +1.4% engagement
  60. 60. vs. +600% engagement
  61. 61. vs. +49% conversion rate from 240 A/B tests
  62. 62. Lesson #4: Seek the global maximum.
  63. 63. Lesson #5: Start today.
  64. 64. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  65. 65. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  66. 66. 5,000+ Happy Customers!
  67. 67. Lesson #5: Start today.
  68. 68. Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Start today.
  69. 69. One more thing...
  70. 70. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO

×