Best Practices & LessonsLearned from 30,000 A/B and Multivariate Tests       Dan Siroker    Co-Founder & CEO
Lesson #1:Nothing is sacred. Question assumptions.
MediaButton
Splash page experimentVariations:    Button:           Media:     1. Sign Up        1. Get Involved Image     2. Learn Mor...
Button: “Sign Up”
Button: “Learn More”
Button: “Join Us Now”
Button: “Sign Up Now”
Splash page experimentVariations:    Button:           Media:     1. Sign Up        1. Get Involved Image     2. Learn Mor...
Media: “Get Involved”
Media: “Family”
Media: “Change”
Media: “Barack’s Video”
Media: “Springfield Video”
Media: “Sam’s Video”
Splash page experimentVariations:    Button:           Media:     1. Sign Up        1. Get Involved Image     2. Learn Mor...
Splash page experimentVariations:    Button:           Media:     1. Sign Up        1. Get Involved Image     2. Learn Mor...
Splash page experimentVariations:    Button:           Media:     1. Sign Up        1. Get Involved Image     2. Learn Mor...
Splash page experiment results
Splash page experiment results
Splash page experiment results
Splash page experiment results               Email                            Volunteers   Amount Raised            Subscr...
Splash page experiment results               Email                            Volunteers   Amount Raised            Subscr...
Splash page experiment results               Email                            Volunteers   Amount Raised            Subscr...
vs.
vs.      +600%      engagement
Lesson #1:Nothing is sacred. Question assumptions.
Lesson #2:Less is more. Reduce choices.
vs.
vs.+8.4%signups
vs.
vs.      +10.97%      $ per visitor
vs.
vs.      +8%      engagement
Lesson #2:Less is more. Reduce choices.
Lesson #3:Words matter. Focus on your call to action.
Donation button experimentVariations
Donation button experiment results                  Never       Signed Up,    PreviouslyVariations      Signed Up   Never ...
vs.
vs.+10%clicks
vs.
vs.      +14.6%      clicks
vs.
vs.      +15.75%      $ per visitor
Lesson #3:Words matter. Focus on your call to action.
Lesson #4:Fail fast.
vs.
vs.      -92.3%      video      clicks
Lesson #4:Fail fast.
Lesson #5:Start today.
Happy CustomersFeatured In
Lesson #5:Start today.
Lessons Learned1. Nothing is sacred. Question assumptions.2. Less is more. Reduce choices.3. Words matter. Focus on your c...
Email me:        Follow me:dan@optimizely.com   @dsiroker
Best Practices & Lessons Learned from 30,000 A/B and Multivariate Tests
Upcoming SlideShare
Loading in …5
×

Best Practices & Lessons Learned from 30,000 A/B and Multivariate Tests

6,207 views

Published on

Most businesses know they should be A/B testing their website but few actually do it on a regular basis. One of their biggest challenges is knowing what to test. In this talk Dan will share the best practices & lessons learned from working with over 10,000 users who have created more than 30,000 experiments using a dead-simple A/B testing tool called Optimizely. Dan will share key insights from these experiments and leave you inspired to start testing today.

Published in: Technology, Business
1 Comment
25 Likes
Statistics
Notes
No Downloads
Views
Total views
6,207
On SlideShare
0
From Embeds
0
Number of Embeds
1,854
Actions
Shares
0
Downloads
178
Comments
1
Likes
25
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Best Practices & Lessons Learned from 30,000 A/B and Multivariate Tests

    1. 1. Best Practices & LessonsLearned from 30,000 A/B and Multivariate Tests Dan Siroker Co-Founder & CEO
    2. 2. Lesson #1:Nothing is sacred. Question assumptions.
    3. 3. MediaButton
    4. 4. Splash page experimentVariations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
    5. 5. Button: “Sign Up”
    6. 6. Button: “Learn More”
    7. 7. Button: “Join Us Now”
    8. 8. Button: “Sign Up Now”
    9. 9. Splash page experimentVariations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
    10. 10. Media: “Get Involved”
    11. 11. Media: “Family”
    12. 12. Media: “Change”
    13. 13. Media: “Barack’s Video”
    14. 14. Media: “Springfield Video”
    15. 15. Media: “Sam’s Video”
    16. 16. Splash page experimentVariations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
    17. 17. Splash page experimentVariations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
    18. 18. Splash page experimentVariations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
    19. 19. Splash page experiment results
    20. 20. Splash page experiment results
    21. 21. Splash page experiment results
    22. 22. Splash page experiment results Email Volunteers Amount Raised SubscriptionsOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
    23. 23. Splash page experiment results Email Volunteers Amount Raised SubscriptionsOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
    24. 24. Splash page experiment results Email Volunteers Amount Raised SubscriptionsOriginal: 7,120,000 712,000 $143,000,000+40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
    25. 25. vs.
    26. 26. vs. +600% engagement
    27. 27. Lesson #1:Nothing is sacred. Question assumptions.
    28. 28. Lesson #2:Less is more. Reduce choices.
    29. 29. vs.
    30. 30. vs.+8.4%signups
    31. 31. vs.
    32. 32. vs. +10.97% $ per visitor
    33. 33. vs.
    34. 34. vs. +8% engagement
    35. 35. Lesson #2:Less is more. Reduce choices.
    36. 36. Lesson #3:Words matter. Focus on your call to action.
    37. 37. Donation button experimentVariations
    38. 38. Donation button experiment results Never Signed Up, PreviouslyVariations Signed Up Never Donated Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%
    39. 39. vs.
    40. 40. vs.+10%clicks
    41. 41. vs.
    42. 42. vs. +14.6% clicks
    43. 43. vs.
    44. 44. vs. +15.75% $ per visitor
    45. 45. Lesson #3:Words matter. Focus on your call to action.
    46. 46. Lesson #4:Fail fast.
    47. 47. vs.
    48. 48. vs. -92.3% video clicks
    49. 49. Lesson #4:Fail fast.
    50. 50. Lesson #5:Start today.
    51. 51. Happy CustomersFeatured In
    52. 52. Lesson #5:Start today.
    53. 53. Lessons Learned1. Nothing is sacred. Question assumptions.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Start today.
    54. 54. Email me: Follow me:dan@optimizely.com @dsiroker

    ×