Your SlideShare is downloading. ×
0
Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
‘05‘06‘07‘08’09’10Wednesday, June 5, 13
Wednesday, June 5, 13
Wednesday, June 5, 13
Lessons LearnedWednesday, June 5, 13
1.2. Less is more. Reduce choices.3. Words matter. Focus on your callto action.4. Seek the global maximum.5. Fail fast.6. ...
Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
Visitvisitorsper dayWednesday, June 5, 13
Visitvisitorsper dayEmailsign uprateWednesday, June 5, 13
Visitvisitorsper dayEmailsign uprate$$ perrecipientWednesday, June 5, 13
Visit Email $sign uprate$ perrecipientvisitorsper dayWednesday, June 5, 13
MediaButtonWednesday, June 5, 13
Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Bara...
Button: “Sign Up”Wednesday, June 5, 13
Button: “Learn More”Wednesday, June 5, 13
Button: “Join Us Now”Wednesday, June 5, 13
Button: “Sign Up Now”Wednesday, June 5, 13
Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Bara...
Media: “Get Involved”Wednesday, June 5, 13
Media: “Family”Wednesday, June 5, 13
Media: “Change”Wednesday, June 5, 13
Media: “Barack’s Video”Wednesday, June 5, 13
Media: “Springfield Video”Wednesday, June 5, 13
Media: “Sam’s Video”Wednesday, June 5, 13
Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Bara...
Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Bara...
Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Bara...
Wednesday, June 5, 13
Wednesday, June 5, 13
Wednesday, June 5, 13
EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New...
EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New...
EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New...
Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.+15% conversion rate+10% $ per visitorWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.+8%engagementWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.+16.8%$ pervisitorWednesday, June 5, 13
v.Wednesday, June 5, 13
v.+14.8%conversion rateWednesday, June 5, 13
v.Wednesday, June 5, 13
v.Wednesday, June 5, 13
Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
Donation button experimentVariationsWednesday, June 5, 13
NeverSigned UpSigned Up,NeverDonatedPreviouslyDonated0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8...
vs.Wednesday, June 5, 13
vs.+14.6%clicksWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.+15.75%$ pervisitorWednesday, June 5, 13
Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
Lesson #4:Seek the global maximum.Wednesday, June 5, 13
Source: http://blog.intercom.io/criticism-and-two-way-streets/Wednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
vs.+0.5% retention+1.4% engagementComments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednes...
vs.Wednesday, June 5, 13
vs.+600%engagementWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.+49% conversionrate from 240 A/BtestsWednesday, June 5, 13
Lesson #4:Seek the global maximum.Wednesday, June 5, 13
Lesson #5:Fail fast.Wednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.-92.3%videoclicksWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.-10% conversionsWednesday, June 5, 13
vs.Wednesday, June 5, 13
vs.fewerconversionsWednesday, June 5, 13
Lesson #5:Fail fast.Wednesday, June 5, 13
Lesson #6:Start today.Wednesday, June 5, 13
Wednesday, June 5, 13
Wednesday, June 5, 13
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
4,000+ Happy Customers!Wednesday, June 5, 13
Lesson #6:Start today.Wednesday, June 5, 13
1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call toaction.4. Seek ...
The Future of OptimizationWednesday, June 5, 13
The Future of OptimizationWhat’s the best experiencefor the average user?One to ManyWednesday, June 5, 13
The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencef...
The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencef...
One to ManyWednesday, June 5, 13
One to FewWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
One to OneWednesday, June 5, 13
Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
AppendixWednesday, June 5, 13
Wednesday, June 5, 13
Upcoming SlideShare
Loading in...5
×

Best Practices & Lessons Learned From Over 223,628 A/B Tests

785

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
785
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Adding tax hurt performance
  • Transcript of "Best Practices & Lessons Learned From Over 223,628 A/B Tests"

    1. 1. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
    2. 2. ‘05‘06‘07‘08’09’10Wednesday, June 5, 13
    3. 3. Wednesday, June 5, 13
    4. 4. Wednesday, June 5, 13
    5. 5. Lessons LearnedWednesday, June 5, 13
    6. 6. 1.2. Less is more. Reduce choices.3. Words matter. Focus on your callto action.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedDefine quantifiable success metrics.Wednesday, June 5, 13
    7. 7. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
    8. 8. Visitvisitorsper dayWednesday, June 5, 13
    9. 9. Visitvisitorsper dayEmailsign uprateWednesday, June 5, 13
    10. 10. Visitvisitorsper dayEmailsign uprate$$ perrecipientWednesday, June 5, 13
    11. 11. Visit Email $sign uprate$ perrecipientvisitorsper dayWednesday, June 5, 13
    12. 12. MediaButtonWednesday, June 5, 13
    13. 13. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoSplash page experimentVariations:Wednesday, June 5, 13
    14. 14. Button: “Sign Up”Wednesday, June 5, 13
    15. 15. Button: “Learn More”Wednesday, June 5, 13
    16. 16. Button: “Join Us Now”Wednesday, June 5, 13
    17. 17. Button: “Sign Up Now”Wednesday, June 5, 13
    18. 18. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    19. 19. Media: “Get Involved”Wednesday, June 5, 13
    20. 20. Media: “Family”Wednesday, June 5, 13
    21. 21. Media: “Change”Wednesday, June 5, 13
    22. 22. Media: “Barack’s Video”Wednesday, June 5, 13
    23. 23. Media: “Springfield Video”Wednesday, June 5, 13
    24. 24. Media: “Sam’s Video”Wednesday, June 5, 13
    25. 25. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    26. 26. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    27. 27. Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
    28. 28. Wednesday, June 5, 13
    29. 29. Wednesday, June 5, 13
    30. 30. Wednesday, June 5, 13
    31. 31. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    32. 32. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    33. 33. EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
    34. 34. Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
    35. 35. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
    36. 36. vs.Wednesday, June 5, 13
    37. 37. vs.+15% conversion rate+10% $ per visitorWednesday, June 5, 13
    38. 38. vs.Wednesday, June 5, 13
    39. 39. vs.+8%engagementWednesday, June 5, 13
    40. 40. vs.Wednesday, June 5, 13
    41. 41. vs.+16.8%$ pervisitorWednesday, June 5, 13
    42. 42. v.Wednesday, June 5, 13
    43. 43. v.+14.8%conversion rateWednesday, June 5, 13
    44. 44. v.Wednesday, June 5, 13
    45. 45. v.Wednesday, June 5, 13
    46. 46. Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
    47. 47. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
    48. 48. Donation button experimentVariationsWednesday, June 5, 13
    49. 49. NeverSigned UpSigned Up,NeverDonatedPreviouslyDonated0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%Donation button experimentVariationsWednesday, June 5, 13
    50. 50. vs.Wednesday, June 5, 13
    51. 51. vs.+14.6%clicksWednesday, June 5, 13
    52. 52. vs.Wednesday, June 5, 13
    53. 53. vs.+15.75%$ pervisitorWednesday, June 5, 13
    54. 54. Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
    55. 55. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
    56. 56. Source: http://blog.intercom.io/criticism-and-two-way-streets/Wednesday, June 5, 13
    57. 57. vs.Wednesday, June 5, 13
    58. 58. vs.Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
    59. 59. vs.+0.5% retention+1.4% engagementComments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
    60. 60. vs.Wednesday, June 5, 13
    61. 61. vs.+600%engagementWednesday, June 5, 13
    62. 62. vs.Wednesday, June 5, 13
    63. 63. vs.+49% conversionrate from 240 A/BtestsWednesday, June 5, 13
    64. 64. Lesson #4:Seek the global maximum.Wednesday, June 5, 13
    65. 65. Lesson #5:Fail fast.Wednesday, June 5, 13
    66. 66. vs.Wednesday, June 5, 13
    67. 67. vs.-92.3%videoclicksWednesday, June 5, 13
    68. 68. vs.Wednesday, June 5, 13
    69. 69. vs.-10% conversionsWednesday, June 5, 13
    70. 70. vs.Wednesday, June 5, 13
    71. 71. vs.fewerconversionsWednesday, June 5, 13
    72. 72. Lesson #5:Fail fast.Wednesday, June 5, 13
    73. 73. Lesson #6:Start today.Wednesday, June 5, 13
    74. 74. Wednesday, June 5, 13
    75. 75. Wednesday, June 5, 13
    76. 76. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
    77. 77. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
    78. 78. 4,000+ Happy Customers!Wednesday, June 5, 13
    79. 79. Lesson #6:Start today.Wednesday, June 5, 13
    80. 80. 1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call toaction.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedWednesday, June 5, 13
    81. 81. The Future of OptimizationWednesday, June 5, 13
    82. 82. The Future of OptimizationWhat’s the best experiencefor the average user?One to ManyWednesday, June 5, 13
    83. 83. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?Wednesday, June 5, 13
    84. 84. The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?One to OneWhat’s the best experiencefor each user?Wednesday, June 5, 13
    85. 85. One to ManyWednesday, June 5, 13
    86. 86. One to FewWednesday, June 5, 13
    87. 87. One to OneWednesday, June 5, 13
    88. 88. One to OneWednesday, June 5, 13
    89. 89. One to OneWednesday, June 5, 13
    90. 90. One to OneWednesday, June 5, 13
    91. 91. One to OneWednesday, June 5, 13
    92. 92. One to OneWednesday, June 5, 13
    93. 93. One to OneWednesday, June 5, 13
    94. 94. Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
    95. 95. AppendixWednesday, June 5, 13
    96. 96. Wednesday, June 5, 13
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×