A/B Testing
If data can help win elections, what
can it help you do for your business?
Dan Siroker
Co-Founder & CEO
‘05
‘06
‘07
‘08
’09
’10
‘05
‘06
‘07
‘08
’09
’10
Millions

$7
$6
$5
$4
$3
$2
$1
$0
2010 Q2
Q1

Q3

Q4

2011
Q1

Q2

Q3

Q4 2012
Q1

Q2

Q3

Q4
Millions

$7

Team: 4
Customers: 644
Annual Run Rate: $0.1m

$6
$5
$4
$3
$2
$1

Dan

Pete

Eric

$0
2010 Q2
Q1

Elliot

Q3...
Millions

$7

Team: 10 (+150%)
Customers: 1,508 (+134%)
Annual Run Rate: $1.2m(+982%)

$6
$5
$4
$3
$2

Julio

Ryan M

$1

...
Millions

$7

Team: 42 (+320%)
Customers: 3,460 (+129%)
Annual Run Rate: $7.6m (+533%)

$6
$5

Steve
Tyle
Jeff Z
Arthu
Ami...
Millions

$7
$6
$5
$4
$3
$2
$1
$0
2010 Q2
Q1

Q3

Q4

2011
Q1

Q2

Q3

Q4 2012
Q1

Q2

Q3

Q4
Millions

$7
$6
$5
$4
$3
$2
$1
$0
2010
Q1

Q2

Q3

Q4

2011
Q1

Q2

Q3

Q4

2012
Q1

Q2

Q3

Q4
Millions

$28
$27
$26
$25
$24
$23
$22
$21
$20
$19
$18
$17
$16
$15
$14
$13
$12
$11
$10
$9
$8
$7
$6
$5
$4
$3
$2
$1
$0

2010 ...
Millions

$28
$27
$26
$25
$24
$23
$22
$21
$20
$19
$18
$17
$16
$15
$14
$13
$12
$11
$10
$9
$8
$7
$6
$5
$4
$3
$2
$1
$0

Steve...
Millions

$28
$27
$26
$25
$24
$23
$22
$21
$20
$19
$18
$17
$16
$15
$14
$13
$12
$11
$10
$9
$8
$7
$6
$5
$4
$3
$2
$1
$0

Steve...
How?
Limited Life Experience
Limited Life Experience
+
Overgeneralization
Limited Life Experience
+
Overgeneralization
=
Advice
-Paul Buchheit
Lesson #1:
Define success.
Lesson #1:
Define

success

.
Lesson #1:
Define quantifiable success metrics.
visitors
per day

Visit

sign up
rate

$ per recipient

Email

$
visitors
per day

Visit

sign up
rate

$ per recipient

Email

$
Media
Button
Splash page experiment
Variations:
Button:
1.

Sign Up

Media:
1.

Get Involved Image

2. Learn More

2. Family Image

3. ...
Button: “Sign Up”
Button: “Learn More”
Button: “Join Us Now”
Button: “Sign Up Now”
Splash page experiment
Variations:
Button:
1.

Sign Up

Media:
1.

Get Involved Image

2. Learn More

2. Family Image

3. ...
Media: “Get Involved”
Media: “Family”
Media: “Change”
Media: “Barack’s Video”
Media: “Springfield Video”
Media: “Sam’s Video”
Splash page experiment
Variations:
Button:
1.

Sign Up

Media:
1.

Get Involved Image

2. Learn More

2. Family Image

3. ...
Splash page experiment
Splash page experiment
Splash page experiment
Splash page experiment
Email
Volunteers
Subscriptions
712,000

Amount
Raised

Original:

7,120,000

$143,000,000

+40.6%

...
Splash page experiment
Email
Volunteers
Subscriptions
712,000

Amount
Raised

Original:

7,120,000

$143,000,000

+40.6%

...
Splash page experiment
Email
Volunteers
Subscriptions
712,000

Amount
Raised

Original:

7,120,000

$143,000,000

+40.6%

...
Lesson #1:
Define quantifiable success metrics.
Lesson #2:
Less is more. Reduce choices.
vs.
+15% conversion rate
+10% $ per visitor
vs.

+20% submissions
vs.
+16.8%
$ per visitor
vs.

+43.8% purchases
Lesson #2:
Less is more. Reduce choices.
Lesson #3:
Words matter. Focus on your call to action.
Donation button experiment
Variations
Donation button experiment
Variations

Never
Signed Up

Signed Up,
Never Donated

Previously
Donated

0.0%

0.0%

0.0%

+2...
vs.
+14.6% clicks
vs.

+15.75%
$ per
visitor
Lesson #3:
Words matter. Focus on your call to action.
Lesson #4:
Seek the global maximum.
Source: http://blog.intercom.io/criticism-and-two-way-streets/
vs.
Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html
vs.

+0.5% retention
+1.4% engagement
vs.
+600%
engagement
vs.

+49% conversion rate
from 240 A/B tests
Lesson #4:
Seek the global maximum.
Lesson #5:
Start today.
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
5,000+ Happy Customers!
Lesson #5:
Start today.
Lessons Learned
1.

Define quantifiable success metrics.

2.

Less is more. Reduce choices.

3.

Words matter. Focus on your...
One more thing...
A/B Testing
If data can help win elections, what
can it help you do for your business?
Dan Siroker
Co-Founder & CEO
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?
Upcoming SlideShare
Loading in …5
×

Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?

1,864 views
1,801 views

Published on

Dan Siroker's presentation at Business of Software 2013 about the universal startup algorithm and how to apply lessons learned from the 2008 Obama campaign and 5,000 Optimizely customers who have run over 300,000 experiments.

Published in: Business, Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,864
On SlideShare
0
From Embeds
0
Number of Embeds
289
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Business of Software 2013: A/B Testing - If data can help win elections, what can it help you do for your business?

  1. 1. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO
  2. 2. ‘05 ‘06 ‘07 ‘08 ’09 ’10
  3. 3. ‘05 ‘06 ‘07 ‘08 ’09 ’10
  4. 4. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  5. 5. Millions $7 Team: 4 Customers: 644 Annual Run Rate: $0.1m $6 $5 $4 $3 $2 $1 Dan Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  6. 6. Millions $7 Team: 10 (+150%) Customers: 1,508 (+134%) Annual Run Rate: $1.2m(+982%) $6 $5 $4 $3 $2 Julio Ryan M $1 Ricky Meng Xiao Dan Mat Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  7. 7. Millions $7 Team: 42 (+320%) Customers: 3,460 (+129%) Annual Run Rate: $7.6m (+533%) $6 $5 Steve Tyle Jeff Z Arthu Amina Greg Sandy $4 Brendan Sophie Dogan Sonesh Shanann Tommy $3 Dan G Cara Byron Mik Alan Eddie Jodie $2 Dae-Ho Cyrus Julio Ryan M $1 Lauren Ashley Ricky Meng Xiao Dan Mat Cristian Travis Jeff P Pete Eric $0 2010 Q2 Q1 Elliot Katy Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Ryan L Ian Jonathan Q4 Joanna Jon
  8. 8. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  9. 9. Millions $7 $6 $5 $4 $3 $2 $1 $0 2010 Q1 Q2 Q3 Q4 2011 Q1 Q2 Q3 Q4 2012 Q1 Q2 Q3 Q4
  10. 10. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 2010 Q2 Q1 Q3 Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Q3 Q4 2013 Q2 Q1 Q3 Q4
  11. 11. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Team: 42 Tyle Jeff Z Arthu Amina Greg Sophie Dogan Cara Alan Dan G Shanann Byron Eddie Sandy Dae-Ho Cyrus Sonesh Brendan Cristian Mik Ricky Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Lauren Ashley Tommy Ryan M Jodie Mat Meng Xiao Julio Katy Travis Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Ryan L Q3 Q4 2013 Q2 Joanna Q1 Ian Jon Q3 Q4
  12. 12. Millions $28 $27 $26 $25 $24 $23 $22 $21 $20 $19 $18 $17 $16 $15 $14 $13 $12 $11 $10 $9 $8 $7 $6 $5 $4 $3 $2 $1 $0 Steve Steve Team: 130 (+210%) Tyle Marius Adam Bo Evan Junan Ash Grace Anshu Lucas Benson Bryce Darwish Jeff Z Arthu David Jaron Richar Tammie Khattaab Laura Asia Amanda Amina Pamela Brian Greg Nikhil Martjin Sophie Amelia Dogan Gary Marco Natarajan Ben Kasra Katie Andrea Daniel SG Dae-Ho Omari Andy Eric Andy Mustafa Ranjit Chris Waylon Pedro Bradley Neha Lauren Adrian Chrix Cristian Mik Ricky Robin Dan Pete Eric 2010 Q2 Q1 Elliot Q3 Jeff P Ashley Tommy Ryan M Mat Julio Katy Travis Lisa Jamie Jonathan Q4 2011 Q2 Q1 Q3 Actual Q4 2012 Q2 Q1 Projection Nick Miyo Jodie Meng Xiao Ryan L Q3 Becca Oren Cyrus Brendan Megan Byron Eddie Sonesh Hemant Shanann Brett Sandy Ryan Michael James Etienne Alan Lacy Dan G Luke Cara Westly Aaron Adam Q4 2013 Q2 Helen Joanna Q1 Jen Ian Jon Nick Fabian Saum Courtney Dan Yonat Kent Ural Eduar Toby Oliver Grant Q3 Cate Christop Julia Q4 Danielle Tai Jonathan Chris Chris Rama
  13. 13. How?
  14. 14. Limited Life Experience
  15. 15. Limited Life Experience + Overgeneralization
  16. 16. Limited Life Experience + Overgeneralization = Advice -Paul Buchheit
  17. 17. Lesson #1: Define success.
  18. 18. Lesson #1: Define success .
  19. 19. Lesson #1: Define quantifiable success metrics.
  20. 20. visitors per day Visit sign up rate $ per recipient Email $
  21. 21. visitors per day Visit sign up rate $ per recipient Email $
  22. 22. Media Button
  23. 23. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  24. 24. Button: “Sign Up”
  25. 25. Button: “Learn More”
  26. 26. Button: “Join Us Now”
  27. 27. Button: “Sign Up Now”
  28. 28. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  29. 29. Media: “Get Involved”
  30. 30. Media: “Family”
  31. 31. Media: “Change”
  32. 32. Media: “Barack’s Video”
  33. 33. Media: “Springfield Video”
  34. 34. Media: “Sam’s Video”
  35. 35. Splash page experiment Variations: Button: 1. Sign Up Media: 1. Get Involved Image 2. Learn More 2. Family Image 3. Join Us Now 3. Change Image 4. Sign Up Now 4. Barack’s Video 5. Springfield Video 6. Sam’s Video
  36. 36. Splash page experiment
  37. 37. Splash page experiment
  38. 38. Splash page experiment
  39. 39. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  40. 40. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  41. 41. Splash page experiment Email Volunteers Subscriptions 712,000 Amount Raised Original: 7,120,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000
  42. 42. Lesson #1: Define quantifiable success metrics.
  43. 43. Lesson #2: Less is more. Reduce choices.
  44. 44. vs. +15% conversion rate +10% $ per visitor
  45. 45. vs. +20% submissions
  46. 46. vs. +16.8% $ per visitor
  47. 47. vs. +43.8% purchases
  48. 48. Lesson #2: Less is more. Reduce choices.
  49. 49. Lesson #3: Words matter. Focus on your call to action.
  50. 50. Donation button experiment Variations
  51. 51. Donation button experiment Variations Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4%
  52. 52. vs. +14.6% clicks
  53. 53. vs. +15.75% $ per visitor
  54. 54. Lesson #3: Words matter. Focus on your call to action.
  55. 55. Lesson #4: Seek the global maximum.
  56. 56. Source: http://blog.intercom.io/criticism-and-two-way-streets/
  57. 57. vs.
  58. 58. Source: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html
  59. 59. vs. +0.5% retention +1.4% engagement
  60. 60. vs. +600% engagement
  61. 61. vs. +49% conversion rate from 240 A/B tests
  62. 62. Lesson #4: Seek the global maximum.
  63. 63. Lesson #5: Start today.
  64. 64. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  65. 65. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/
  66. 66. 5,000+ Happy Customers!
  67. 67. Lesson #5: Start today.
  68. 68. Lessons Learned 1. Define quantifiable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Start today.
  69. 69. One more thing...
  70. 70. A/B Testing If data can help win elections, what can it help you do for your business? Dan Siroker Co-Founder & CEO

×