0
Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings

Thursday, November 28, 13
Is de (klant) reis...

belangrijker dan de bestemming?
Thursday, November 28, 13
2009!

Thursday, November 28, 13

2010!

2011!

2012!

2013!
Lessons Learned

Thursday, November 28, 13
Lesson #1:
Define success .

Thursday, November 28, 13
Thursday, November 28, 13
Lesson #1:
Define

Thursday, November 28, 13

success

.
Lesson #1:
Define quantifiable success metrics.

Thursday, November 28, 13
Thursday, November 28, 13
visitors
per day

Thursday, November 28, 13

Visit
visitors
per day

Thursday, November 28, 13

Visit

sign up
rate

Email
visitors
per day

Thursday, November 28, 13

Visit

sign up
rate

Email

$ per recipient

$
visitors
per day

Thursday, November 28, 13

Visit

sign up
rate

Email

$ per recipient

$
Media
Button

Thursday, November 28, 13
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Button: “Sign Up”
Thursday, November 28, 13
Button: “Learn More”
Thursday, November 28, 13
Button: “Join Us Now”
Thursday, November 28, 13
Button: “Sign Up Now”
Thursday, November 28, 13
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Media: “Get Involved”
Thursday, November 28, 13
Media: “Family”
Thursday, November 28, 13
Media: “Change”
Thursday, November 28, 13
Media: “Barack’s Video”
Thursday, November 28, 13
Media: “Springfield Video”
Thursday, November 28, 13
Media: “Sam’s Video”
Thursday, November 28, 13
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment

Thursday, November 28, 13
Splash page experiment

Thursday, November 28, 13
Splash page experiment

Thursday, November 28, 13
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Lesson #1:
Define quantifiable success metrics.

Thursday, November 28, 13
Lesson #2:
Less is more. Reduce choices.

Thursday, November 28, 13
vs.

Thursday, November 28, 13
vs.
+8% engagement
Thursday, November 28, 13
+43.8%
Purchases

Thursday, November 28, 13
vs.

Thursday, November 28, 13
vs.

+14.8% conversion rate

Thursday, November 28, 13
Lesson #2:
Less is more. Reduce choices.

Thursday, November 28, 13
Lesson #3:
Words matter. Focus on your call to action.

Thursday, November 28, 13
Donation button experiment
Variations

Thursday, November 28, 13
Donation button experiment
Previously
Donated

0.0%

0.0%

+2.3%

+27.8%

+16.3%

-27.8%

N/A

N/A

+15.2%

-24.6%

+11.9%...
vs.

Thursday, November 28, 13
vs.

Thursday, November 28, 13
vs.
+14.6% clicks

Thursday, November 28, 13
Lesson #3:
Words matter. Focus on your call to action.

Thursday, November 28, 13
Lesson #4:
Fail fast.

Thursday, November 28, 13
vs.

Thursday, November 28, 13
vs.

-10% conversions

Thursday, November 28, 13
vs.

Thursday, November 28, 13
vs.

-10% conversions
Thursday, November 28, 13
Lesson #4:
Fail fast.

Thursday, November 28, 13
Lesson #5:
Seek the Global Maximum.

Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
+0.5% Retention
+1.4% Engagement

Thursday, November 28, 13
Lesson #5:
Seek the Global Maximum.

Thursday, November 28, 13
Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call...
Thursday, November 28, 13
One more thing...

Thursday, November 28, 13
6000+ happy testers

Start Today!

Thursday, November 28, 13
Upcoming SlideShare
Loading in...5
×

Webshop Wednesday #11: Best Practices and Lessons Learned from 100,000 A/B Tests, Martijn Geerlings/Optimizely

186

Published on

The basics you should take a look at before starting A/B testing.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
186
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Webshop Wednesday #11: Best Practices and Lessons Learned from 100,000 A/B Tests, Martijn Geerlings/Optimizely"

  1. 1. Best Practices and Lessons Learned from 100,000 A/B Tests Martijn Geerlings Thursday, November 28, 13
  2. 2. Is de (klant) reis... belangrijker dan de bestemming? Thursday, November 28, 13
  3. 3. 2009! Thursday, November 28, 13 2010! 2011! 2012! 2013!
  4. 4. Lessons Learned Thursday, November 28, 13
  5. 5. Lesson #1: Define success . Thursday, November 28, 13
  6. 6. Thursday, November 28, 13
  7. 7. Lesson #1: Define Thursday, November 28, 13 success .
  8. 8. Lesson #1: Define quantifiable success metrics. Thursday, November 28, 13
  9. 9. Thursday, November 28, 13
  10. 10. visitors per day Thursday, November 28, 13 Visit
  11. 11. visitors per day Thursday, November 28, 13 Visit sign up rate Email
  12. 12. visitors per day Thursday, November 28, 13 Visit sign up rate Email $ per recipient $
  13. 13. visitors per day Thursday, November 28, 13 Visit sign up rate Email $ per recipient $
  14. 14. Media Button Thursday, November 28, 13
  15. 15. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Thursday, November 28, 13
  16. 16. Button: “Sign Up” Thursday, November 28, 13
  17. 17. Button: “Learn More” Thursday, November 28, 13
  18. 18. Button: “Join Us Now” Thursday, November 28, 13
  19. 19. Button: “Sign Up Now” Thursday, November 28, 13
  20. 20. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Thursday, November 28, 13
  21. 21. Media: “Get Involved” Thursday, November 28, 13
  22. 22. Media: “Family” Thursday, November 28, 13
  23. 23. Media: “Change” Thursday, November 28, 13
  24. 24. Media: “Barack’s Video” Thursday, November 28, 13
  25. 25. Media: “Springfield Video” Thursday, November 28, 13
  26. 26. Media: “Sam’s Video” Thursday, November 28, 13
  27. 27. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Thursday, November 28, 13
  28. 28. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Thursday, November 28, 13
  29. 29. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Thursday, November 28, 13
  30. 30. Splash page experiment Thursday, November 28, 13
  31. 31. Splash page experiment Thursday, November 28, 13
  32. 32. Splash page experiment Thursday, November 28, 13
  33. 33. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Thursday, November 28, 13 712,000 $143,000,000
  34. 34. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Thursday, November 28, 13 712,000 $143,000,000
  35. 35. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Thursday, November 28, 13 712,000 $143,000,000
  36. 36. Lesson #1: Define quantifiable success metrics. Thursday, November 28, 13
  37. 37. Lesson #2: Less is more. Reduce choices. Thursday, November 28, 13
  38. 38. vs. Thursday, November 28, 13
  39. 39. vs. +8% engagement Thursday, November 28, 13
  40. 40. +43.8% Purchases Thursday, November 28, 13
  41. 41. vs. Thursday, November 28, 13
  42. 42. vs. +14.8% conversion rate Thursday, November 28, 13
  43. 43. Lesson #2: Less is more. Reduce choices. Thursday, November 28, 13
  44. 44. Lesson #3: Words matter. Focus on your call to action. Thursday, November 28, 13
  45. 45. Donation button experiment Variations Thursday, November 28, 13
  46. 46. Donation button experiment Previously Donated 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% Thursday, November 28, 13 Signed Up, Never Donated 0.0% Variations Never Signed Up +2.9% +18.4%
  47. 47. vs. Thursday, November 28, 13
  48. 48. vs. Thursday, November 28, 13
  49. 49. vs. +14.6% clicks Thursday, November 28, 13
  50. 50. Lesson #3: Words matter. Focus on your call to action. Thursday, November 28, 13
  51. 51. Lesson #4: Fail fast. Thursday, November 28, 13
  52. 52. vs. Thursday, November 28, 13
  53. 53. vs. -10% conversions Thursday, November 28, 13
  54. 54. vs. Thursday, November 28, 13
  55. 55. vs. -10% conversions Thursday, November 28, 13
  56. 56. Lesson #4: Fail fast. Thursday, November 28, 13
  57. 57. Lesson #5: Seek the Global Maximum. Thursday, November 28, 13
  58. 58. Thursday, November 28, 13
  59. 59. Thursday, November 28, 13
  60. 60. Thursday, November 28, 13
  61. 61. Thursday, November 28, 13
  62. 62. Thursday, November 28, 13
  63. 63. +0.5% Retention +1.4% Engagement Thursday, November 28, 13
  64. 64. Lesson #5: Seek the Global Maximum. Thursday, November 28, 13
  65. 65. Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Fail fast. 5. Seek the Global Maximum. Thursday, November 28, 13
  66. 66. Thursday, November 28, 13
  67. 67. One more thing... Thursday, November 28, 13
  68. 68. 6000+ happy testers Start Today! Thursday, November 28, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×