Best Practices & Lessons Learned From Over 223,628 A/B Tests

  • 676 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
676
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
18
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Adding tax hurt performance

Transcript

  • 1. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
  • 2. ‘05‘06‘07‘08’09’10Wednesday, June 5, 13
  • 3. Wednesday, June 5, 13
  • 4. Wednesday, June 5, 13
  • 5. Lessons LearnedWednesday, June 5, 13
  • 6. 1.2. Less is more. Reduce choices.3. Words matter. Focus on your callto action.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedDefine quantifiable success metrics.Wednesday, June 5, 13
  • 7. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
  • 8. Visitvisitorsper dayWednesday, June 5, 13
  • 9. Visitvisitorsper dayEmailsign uprateWednesday, June 5, 13
  • 10. Visitvisitorsper dayEmailsign uprate$$ perrecipientWednesday, June 5, 13
  • 11. Visit Email $sign uprate$ perrecipientvisitorsper dayWednesday, June 5, 13
  • 12. MediaButtonWednesday, June 5, 13
  • 13. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoSplash page experimentVariations:Wednesday, June 5, 13
  • 14. Button: “Sign Up”Wednesday, June 5, 13
  • 15. Button: “Learn More”Wednesday, June 5, 13
  • 16. Button: “Join Us Now”Wednesday, June 5, 13
  • 17. Button: “Sign Up Now”Wednesday, June 5, 13
  • 18. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • 19. Media: “Get Involved”Wednesday, June 5, 13
  • 20. Media: “Family”Wednesday, June 5, 13
  • 21. Media: “Change”Wednesday, June 5, 13
  • 22. Media: “Barack’s Video”Wednesday, June 5, 13
  • 23. Media: “Springfield Video”Wednesday, June 5, 13
  • 24. Media: “Sam’s Video”Wednesday, June 5, 13
  • 25. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • 26. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • 27. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • 28. Wednesday, June 5, 13
  • 29. Wednesday, June 5, 13
  • 30. Wednesday, June 5, 13
  • 31. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • 32. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • 33. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • 34. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
  • 35. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
  • 36. vs.Wednesday, June 5, 13
  • 37. vs.+15% conversion rate+10% $ per visitorWednesday, June 5, 13
  • 38. vs.Wednesday, June 5, 13
  • 39. vs.+8%engagementWednesday, June 5, 13
  • 40. vs.Wednesday, June 5, 13
  • 41. vs.+16.8%$ pervisitorWednesday, June 5, 13
  • 42. v.Wednesday, June 5, 13
  • 43. v.+14.8%conversion rateWednesday, June 5, 13
  • 44. v.Wednesday, June 5, 13
  • 45. v.Wednesday, June 5, 13
  • 46. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
  • 47. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
  • 48. Donation button experimentVariationsWednesday, June 5, 13
  • 49. NeverSigned UpSigned Up,NeverDonatedPreviouslyDonated0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%Donation button experimentVariationsWednesday, June 5, 13
  • 50. vs.Wednesday, June 5, 13
  • 51. vs.+14.6%clicksWednesday, June 5, 13
  • 52. vs.Wednesday, June 5, 13
  • 53. vs.+15.75%$ pervisitorWednesday, June 5, 13
  • 54. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
  • 55. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
  • 56. Source: http://blog.intercom.io/criticism-and-two-way-streets/Wednesday, June 5, 13
  • 57. vs.Wednesday, June 5, 13
  • 58. vs.Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
  • 59. vs.+0.5% retention+1.4% engagementComments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
  • 60. vs.Wednesday, June 5, 13
  • 61. vs.+600%engagementWednesday, June 5, 13
  • 62. vs.Wednesday, June 5, 13
  • 63. vs.+49% conversionrate from 240 A/BtestsWednesday, June 5, 13
  • 64. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
  • 65. Lesson #5:Fail fast.Wednesday, June 5, 13
  • 66. vs.Wednesday, June 5, 13
  • 67. vs.-92.3%videoclicksWednesday, June 5, 13
  • 68. vs.Wednesday, June 5, 13
  • 69. vs.-10% conversionsWednesday, June 5, 13
  • 70. vs.Wednesday, June 5, 13
  • 71. vs.fewerconversionsWednesday, June 5, 13
  • 72. Lesson #5:Fail fast.Wednesday, June 5, 13
  • 73. Lesson #6:Start today.Wednesday, June 5, 13
  • 74. Wednesday, June 5, 13
  • 75. Wednesday, June 5, 13
  • 76. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
  • 77. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
  • 78. 4,000+ Happy Customers!Wednesday, June 5, 13
  • 79. Lesson #6:Start today.Wednesday, June 5, 13
  • 80. 1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call toaction.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedWednesday, June 5, 13
  • 81. The Future of OptimizationWednesday, June 5, 13
  • 82. The Future of OptimizationWhat’s the best experiencefor the average user?One to ManyWednesday, June 5, 13
  • 83. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?Wednesday, June 5, 13
  • 84. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?One to OneWhat’s the best experiencefor each user?Wednesday, June 5, 13
  • 85. One to ManyWednesday, June 5, 13
  • 86. One to FewWednesday, June 5, 13
  • 87. One to OneWednesday, June 5, 13
  • 88. One to OneWednesday, June 5, 13
  • 89. One to OneWednesday, June 5, 13
  • 90. One to OneWednesday, June 5, 13
  • 91. One to OneWednesday, June 5, 13
  • 92. One to OneWednesday, June 5, 13
  • 93. One to OneWednesday, June 5, 13
  • 94. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
  • 95. AppendixWednesday, June 5, 13
  • 96. Wednesday, June 5, 13