Friday, January 31, 14
Friday, January 31, 14
Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
9:1
Friday, January 31, 14
Lessons Learned

Friday, January 31, 14
Lesson #1:
Define success .

Friday, January 31, 14
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Friday, January 31, 14
visitors
per day

Friday, January 31, 14

Visit
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
Media
Button

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Button: “Sign Up”
Friday, January 31, 14
Button: “Learn More”
Friday, January 31, 14
Button: “Join Us Now”
Friday, January 31, 14
Button: “Sign Up Now”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Media: “Get Involved”
Friday, January 31, 14
Media: “Family”
Friday, January 31, 14
Media: “Change”
Friday, January 31, 14
Media: “Barack’s Video”
Friday, January 31, 14
Media: “Springfield Video”
Friday, January 31, 14
Media: “Sam’s Video”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$5...
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
+43.8%
Purchases

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

+14.8% conversion rate

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+14.6% clicks

Friday, January 31, 14
Donation button experiment
Variations

Friday, January 31, 14
Donation button experiment
Previously
Donated

0.0%

0.0%

+2.3%

+27.8%

+16.3%

-27.8%

N/A

N/A

+15.2%

-24.6%

+11.9%...
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

-10% conversions
Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
+0.5% Retention
+1.4% Engagement

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call...
Thank You

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+8% engagement
Friday, January 31, 14
2009!

Friday, January 31, 14

2010!

2011!

2012!

2013!
Upcoming SlideShare
Loading in …5
×

Search Marketing Thursday - Januari 2014 - Martijn Geerlings

1,345 views

Published on

Search Marketing Thursday - Januari 2014 - Martijn Geerlings (Optimizely)

Published in: Marketing, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,345
On SlideShare
0
From Embeds
0
Number of Embeds
966
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Search Marketing Thursday - Januari 2014 - Martijn Geerlings

  1. 1. Friday, January 31, 14
  2. 2. Friday, January 31, 14
  3. 3. Best Practices and Lessons Learned from 100,000 A/B Tests Martijn Geerlings Friday, January 31, 14
  4. 4. Friday, January 31, 14
  5. 5. Friday, January 31, 14
  6. 6. Friday, January 31, 14
  7. 7. Friday, January 31, 14
  8. 8. Friday, January 31, 14
  9. 9. Friday, January 31, 14
  10. 10. 9:1 Friday, January 31, 14
  11. 11. Lessons Learned Friday, January 31, 14
  12. 12. Lesson #1: Define success . Friday, January 31, 14
  13. 13. Lesson #1: Define quantifiable success metrics. Friday, January 31, 14
  14. 14. Friday, January 31, 14
  15. 15. visitors per day Friday, January 31, 14 Visit
  16. 16. visitors per day Friday, January 31, 14 Visit sign up rate Email
  17. 17. visitors per day Friday, January 31, 14 Visit sign up rate Email $ per recipient $
  18. 18. visitors per day Friday, January 31, 14 Visit sign up rate Email $ per recipient $
  19. 19. Media Button Friday, January 31, 14
  20. 20. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Friday, January 31, 14
  21. 21. Button: “Sign Up” Friday, January 31, 14
  22. 22. Button: “Learn More” Friday, January 31, 14
  23. 23. Button: “Join Us Now” Friday, January 31, 14
  24. 24. Button: “Sign Up Now” Friday, January 31, 14
  25. 25. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Friday, January 31, 14
  26. 26. Media: “Get Involved” Friday, January 31, 14
  27. 27. Media: “Family” Friday, January 31, 14
  28. 28. Media: “Change” Friday, January 31, 14
  29. 29. Media: “Barack’s Video” Friday, January 31, 14
  30. 30. Media: “Springfield Video” Friday, January 31, 14
  31. 31. Media: “Sam’s Video” Friday, January 31, 14
  32. 32. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Friday, January 31, 14
  33. 33. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Friday, January 31, 14
  34. 34. Splash page experiment Variations: Button: Media: 1. Sign Up 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video Friday, January 31, 14
  35. 35. Splash page experiment Friday, January 31, 14
  36. 36. Splash page experiment Friday, January 31, 14
  37. 37. Splash page experiment Friday, January 31, 14
  38. 38. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Friday, January 31, 14 712,000 $143,000,000
  39. 39. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Friday, January 31, 14 712,000 $143,000,000
  40. 40. Splash page experiment Email Volunteers Amount Raised Subscriptions Original: 7,120,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Friday, January 31, 14 712,000 $143,000,000
  41. 41. Lesson #1: Define quantifiable success metrics. Friday, January 31, 14
  42. 42. Lesson #2: Less is more. Reduce choices. Friday, January 31, 14
  43. 43. +43.8% Purchases Friday, January 31, 14
  44. 44. vs. Friday, January 31, 14
  45. 45. vs. +14.8% conversion rate Friday, January 31, 14
  46. 46. Lesson #2: Less is more. Reduce choices. Friday, January 31, 14
  47. 47. Lesson #3: Words matter. Focus on your call to action. Friday, January 31, 14
  48. 48. vs. Friday, January 31, 14
  49. 49. vs. Friday, January 31, 14
  50. 50. vs. +14.6% clicks Friday, January 31, 14
  51. 51. Donation button experiment Variations Friday, January 31, 14
  52. 52. Donation button experiment Previously Donated 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% Friday, January 31, 14 Signed Up, Never Donated 0.0% Variations Never Signed Up +2.9% +18.4%
  53. 53. Lesson #3: Words matter. Focus on your call to action. Friday, January 31, 14
  54. 54. Lesson #4: Fail fast. Friday, January 31, 14
  55. 55. vs. Friday, January 31, 14
  56. 56. vs. Friday, January 31, 14
  57. 57. vs. -10% conversions Friday, January 31, 14
  58. 58. Lesson #4: Fail fast. Friday, January 31, 14
  59. 59. Lesson #5: Seek the Global Maximum. Friday, January 31, 14
  60. 60. Friday, January 31, 14
  61. 61. Friday, January 31, 14
  62. 62. Friday, January 31, 14
  63. 63. Friday, January 31, 14
  64. 64. Friday, January 31, 14
  65. 65. +0.5% Retention +1.4% Engagement Friday, January 31, 14
  66. 66. Lesson #5: Seek the Global Maximum. Friday, January 31, 14
  67. 67. Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Fail fast. 5. Seek the Global Maximum. Friday, January 31, 14
  68. 68. Thank You Friday, January 31, 14
  69. 69. vs. Friday, January 31, 14
  70. 70. vs. +8% engagement Friday, January 31, 14
  71. 71. 2009! Friday, January 31, 14 2010! 2011! 2012! 2013!

×