Best Practices & Lessons Learned From Over 223,628 A/B Tests
Upcoming SlideShare
Loading in...5
×
 

Best Practices & Lessons Learned From Over 223,628 A/B Tests

on

  • 848 views

 

Statistics

Views

Total Views
848
Views on SlideShare
848
Embed Views
0

Actions

Likes
2
Downloads
18
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Adding tax hurt performance

Best Practices & Lessons Learned From Over 223,628 A/B Tests Best Practices & Lessons Learned From Over 223,628 A/B Tests Presentation Transcript

  • Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
  • ‘05‘06‘07‘08’09’10Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • Lessons LearnedWednesday, June 5, 13
  • 1.2. Less is more. Reduce choices.3. Words matter. Focus on your callto action.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedDefine quantifiable success metrics.Wednesday, June 5, 13
  • Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
  • Visitvisitorsper dayWednesday, June 5, 13
  • Visitvisitorsper dayEmailsign uprateWednesday, June 5, 13
  • Visitvisitorsper dayEmailsign uprate$$ perrecipientWednesday, June 5, 13
  • Visit Email $sign uprate$ perrecipientvisitorsper dayWednesday, June 5, 13
  • MediaButtonWednesday, June 5, 13
  • Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoSplash page experimentVariations:Wednesday, June 5, 13
  • Button: “Sign Up”Wednesday, June 5, 13
  • Button: “Learn More”Wednesday, June 5, 13
  • Button: “Join Us Now”Wednesday, June 5, 13
  • Button: “Sign Up Now”Wednesday, June 5, 13
  • Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • Media: “Get Involved”Wednesday, June 5, 13
  • Media: “Family”Wednesday, June 5, 13
  • Media: “Change”Wednesday, June 5, 13
  • Media: “Barack’s Video”Wednesday, June 5, 13
  • Media: “Springfield Video”Wednesday, June 5, 13
  • Media: “Sam’s Video”Wednesday, June 5, 13
  • Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • Button:1. Sign Up2. Learn More3. Join Us Now4. Sign Up NowMedia:1. Get Involved Image2. Family Image3. Change Image4. Barack’s Video5. Springfield Video6. Sam’s VideoVariations:Splash page experimentWednesday, June 5, 13
  • Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • EmailSubscriptionsVolunteersAmountRaisedOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000New: 10,000,000 1,000,000 $200,000,000Splash page experimentWednesday, June 5, 13
  • Lesson #1:Define quantifiable success metrics.Wednesday, June 5, 13
  • Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+15% conversion rate+10% $ per visitorWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+8%engagementWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+16.8%$ pervisitorWednesday, June 5, 13
  • v.Wednesday, June 5, 13
  • v.+14.8%conversion rateWednesday, June 5, 13
  • v.Wednesday, June 5, 13
  • v.Wednesday, June 5, 13
  • Lesson #2:Less is more. Reduce choices.Wednesday, June 5, 13
  • Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
  • Donation button experimentVariationsWednesday, June 5, 13
  • NeverSigned UpSigned Up,NeverDonatedPreviouslyDonated0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%Donation button experimentVariationsWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+14.6%clicksWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+15.75%$ pervisitorWednesday, June 5, 13
  • Lesson #3:Words matter. Focus on your call to action.Wednesday, June 5, 13
  • Lesson #4:Seek the global maximum.Wednesday, June 5, 13
  • Source: http://blog.intercom.io/criticism-and-two-way-streets/Wednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
  • vs.+0.5% retention+1.4% engagementComments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.htmlWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+600%engagementWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.+49% conversionrate from 240 A/BtestsWednesday, June 5, 13
  • Lesson #4:Seek the global maximum.Wednesday, June 5, 13
  • Lesson #5:Fail fast.Wednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.-92.3%videoclicksWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.-10% conversionsWednesday, June 5, 13
  • vs.Wednesday, June 5, 13
  • vs.fewerconversionsWednesday, June 5, 13
  • Lesson #5:Fail fast.Wednesday, June 5, 13
  • Lesson #6:Start today.Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • Wednesday, June 5, 13
  • Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
  • Source: http://techcrunch.com/2013/04/10/optimizely-series-a/Wednesday, June 5, 13
  • 4,000+ Happy Customers!Wednesday, June 5, 13
  • Lesson #6:Start today.Wednesday, June 5, 13
  • 1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call toaction.4. Seek the global maximum.5. Fail fast.6. Start today.Lessons LearnedWednesday, June 5, 13
  • The Future of OptimizationWednesday, June 5, 13
  • The Future of OptimizationWhat’s the best experiencefor the average user?One to ManyWednesday, June 5, 13
  • The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?Wednesday, June 5, 13
  • The Future of OptimizationWhat’s the best experiencefor the average user?One to Many One to FewWhat’s the best experiencefor segments of users?One to OneWhat’s the best experiencefor each user?Wednesday, June 5, 13
  • One to ManyWednesday, June 5, 13
  • One to FewWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • One to OneWednesday, June 5, 13
  • Best Practices andLessons Learned fromDan SirokerCo-Founder & CEOA/B Tests223,628Wednesday, June 5, 13
  • AppendixWednesday, June 5, 13
  • Wednesday, June 5, 13