Your SlideShare is downloading. ×
Best Practices & Lessons Learned From Over 223,628 A/B Tests
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Best Practices & Lessons Learned From Over 223,628 A/B Tests

741
views

Published on

Published in: Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
741
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
20
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Adding tax hurt performance
  • Transcript

    • 1. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
    • 2. ‘05‘06‘07‘08’09’10Wednesday, June 5, 13
    • 3. Wednesday, June 5, 13
    • 4. Wednesday, June 5, 13
    • 5. Lessons LearnedWednesday, June 5, 13
    • 6. 1.2. Less is more. Reduce choices.3. Words matter. Focus on your callto action.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedDefine quantifiable success metrics.Wednesday, June 5, 13
    • 7. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
    • 8. Visitvisitorsper dayWednesday, June 5, 13
    • 9. Visitvisitorsper dayEmailsign uprateWednesday, June 5, 13
    • 10. Visitvisitorsper dayEmailsign uprate$$ perrecipientWednesday, June 5, 13
    • 11. Visit Email $sign uprate$ perrecipientvisitorsper dayWednesday, June 5, 13
    • 12. MediaButtonWednesday, June 5, 13
    • 13. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoSplash page experimentVariations:Wednesday, June 5, 13
    • 14. Button: “Sign Up”Wednesday, June 5, 13
    • 15. Button: “Learn More”Wednesday, June 5, 13
    • 16. Button: “Join Us Now”Wednesday, June 5, 13
    • 17. Button: “Sign Up Now”Wednesday, June 5, 13
    • 18. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    • 19. Media: “Get Involved”Wednesday, June 5, 13
    • 20. Media: “Family”Wednesday, June 5, 13
    • 21. Media: “Change”Wednesday, June 5, 13
    • 22. Media: “Barack’s Video”Wednesday, June 5, 13
    • 23. Media: “Springfield Video”Wednesday, June 5, 13
    • 24. Media: “Sam’s Video”Wednesday, June 5, 13
    • 25. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    • 26. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    • 27. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    • 28. Wednesday, June 5, 13
    • 29. Wednesday, June 5, 13
    • 30. Wednesday, June 5, 13
    • 31. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    • 32. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    • 33. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    • 34. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
    • 35. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
    • 36. vs.Wednesday, June 5, 13
    • 37. vs.+15% conversion rate+10% $ per visitorWednesday, June 5, 13
    • 38. vs.Wednesday, June 5, 13
    • 39. vs.+8%engagementWednesday, June 5, 13
    • 40. vs.Wednesday, June 5, 13
    • 41. vs.+16.8%$ pervisitorWednesday, June 5, 13
    • 42. v.Wednesday, June 5, 13
    • 43. v.+14.8%conversion rateWednesday, June 5, 13
    • 44. v.Wednesday, June 5, 13
    • 45. v.Wednesday, June 5, 13
    • 46. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
    • 47. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
    • 48. Donation button experimentVariationsWednesday, June 5, 13
    • 49. NeverSigned UpSigned Up,NeverDonatedPreviouslyDonated0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%Donation button experimentVariationsWednesday, June 5, 13
    • 50. vs.Wednesday, June 5, 13
    • 51. vs.+14.6%clicksWednesday, June 5, 13
    • 52. vs.Wednesday, June 5, 13
    • 53. vs.+15.75%$ pervisitorWednesday, June 5, 13
    • 54. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
    • 55. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
    • 56. Source: http://blog.intercom.io/criticism-and-two-way-streets/Wednesday, June 5, 13
    • 57. vs.Wednesday, June 5, 13
    • 58. vs.Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
    • 59. vs.+0.5% retention+1.4% engagementComments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
    • 60. vs.Wednesday, June 5, 13
    • 61. vs.+600%engagementWednesday, June 5, 13
    • 62. vs.Wednesday, June 5, 13
    • 63. vs.+49% conversionrate from 240 A/BtestsWednesday, June 5, 13
    • 64. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
    • 65. Lesson #5:Fail fast.Wednesday, June 5, 13
    • 66. vs.Wednesday, June 5, 13
    • 67. vs.-92.3%videoclicksWednesday, June 5, 13
    • 68. vs.Wednesday, June 5, 13
    • 69. vs.-10% conversionsWednesday, June 5, 13
    • 70. vs.Wednesday, June 5, 13
    • 71. vs.fewerconversionsWednesday, June 5, 13
    • 72. Lesson #5:Fail fast.Wednesday, June 5, 13
    • 73. Lesson #6:Start today.Wednesday, June 5, 13
    • 74. Wednesday, June 5, 13
    • 75. Wednesday, June 5, 13
    • 76. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
    • 77. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
    • 78. 4,000+ Happy Customers!Wednesday, June 5, 13
    • 79. Lesson #6:Start today.Wednesday, June 5, 13
    • 80. 1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call toaction.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedWednesday, June 5, 13
    • 81. The Future of OptimizationWednesday, June 5, 13
    • 82. The Future of OptimizationWhat’s the best experiencefor the average user?One to ManyWednesday, June 5, 13
    • 83. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?Wednesday, June 5, 13
    • 84. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?One to OneWhat’s the best experiencefor each user?Wednesday, June 5, 13
    • 85. One to ManyWednesday, June 5, 13
    • 86. One to FewWednesday, June 5, 13
    • 87. One to OneWednesday, June 5, 13
    • 88. One to OneWednesday, June 5, 13
    • 89. One to OneWednesday, June 5, 13
    • 90. One to OneWednesday, June 5, 13
    • 91. One to OneWednesday, June 5, 13
    • 92. One to OneWednesday, June 5, 13
    • 93. One to OneWednesday, June 5, 13
    • 94. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
    • 95. AppendixWednesday, June 5, 13
    • 96. Wednesday, June 5, 13

    ×