January	
  14,	
  2014	
  

P.O.P.	
  Copyright	
  2013,	
  All	
  Around	
  The	
  World	
  
A/B	
  TesAng 	
   	
  	
  
What	
  your	
  mother	
  didn’t	
  tell	
  you	
  
hDp://www.allaroundtheworld.fr/	
  

Janua...
What	
  is	
  A/B	
  TesAng?	
  
•  TesAng	
  customers,	
  not	
  code	
  
•  Show	
  different	
  versions	
  
•  Pick	
 ...
Who	
  Relies	
  On	
  A/B	
  TesAng?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Transform	
  Your	
  Company	
  
•  Test	
  your	
  customers,	
  not	
  your	
  code	
  
•  Stop	
  guessing	
  what	
  w...
Should	
  your	
  paycheck	
  
depend	
  on	
  my	
  hunch?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaro...
Conversion	
  Funnel	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Conversion	
  Funnel	
  

→	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
A	
  Basic	
  Test	
  
• 
• 
• 
• 

Your	
  current	
  behavior	
  is	
  the	
  base	
  
A	
  variant	
  is	
  a	
  new	
 ...
January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Generates	
  More	
  Clicks?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Generates	
  More	
  Clicks?	
  

10,017	
  Visits	
  

125	
  Conversions	
  

January	
  14,	
  2014	
  

9,983...
Which	
  Generates	
  More	
  Clicks?	
  

9,950	
  Visits	
  

125	
  Conversions	
  

January	
  14,	
  2014	
  

10,550...
The	
  Numbers	
  (95%	
  Confidence)	
  

Version	
  

Visits	
  

Conversion	
  

Rate	
  

Not	
  Random	
  

Red	
  

1...
The	
  Numbers	
  (95%	
  Confidence)	
  

Version	
  

Visits	
  

Conversion	
  

Rate	
  

Not	
  Random	
  

Red	
  

9...
Which	
  Sells	
  More	
  Product?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Creates	
  More	
  Profit?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
What	
  Confidence	
  Level?	
  

Is	
  99%	
  beDer	
  than	
  90%?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://ww...
The	
  Cycle	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Create	
  Experiment	
  

Only	
  one	
  difference	
  per	
  version!	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://...
Roll	
  the	
  Dice	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Track	
  Performance	
  

Sta:s:cs	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Choose	
  the	
  Best	
  

Pa:ence	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Create	
  Experiment	
  

Wash,	
  rinse,	
  repeat	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundthewo...
Choose	
  the	
  Best	
  
• 
• 
• 
• 

Ignore	
  the	
  first	
  day	
  
Full-­‐business	
  cycle	
  
What	
  if	
  you’re	...
LimitaAons	
  of	
  A/B	
  TesAng	
  
•  Doesn’t	
  tell	
  you	
  why	
  
•  Side	
  effects	
  
•  Doesn’t	
  tell	
  you...
Benefits	
  
• 
• 
• 
• 
• 
• 
• 
• 

Hyundai:	
  62%	
  increase	
  in	
  test	
  drive	
  requests	
  
Artsy	
  Editor:	
...
What	
  Are	
  They	
  TesAng?	
  
• 
• 
• 
• 
• 
• 
• 
• 

Add-­‐to-­‐cart	
  buDon	
  placement	
  
Call	
  to	
  acAon	...
Gemng	
  Started	
  
• 
• 
• 
• 

Vendor	
  or	
  in-­‐house?	
  
Go	
  for	
  quick	
  wins	
  
Run	
  an	
  AA	
  test	
...
Problems	
  

Ego	
  
January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Problems	
  
• 
• 
• 
• 
• 
• 
• 

Forgemng	
  caching	
  
Non-­‐random	
  selecAon	
  
Forgemng	
  selecAon	
  
TesAng	
 ...
Tips	
  
• 
• 
• 
• 
• 

Most	
  experiments	
  fail	
  
Don’t	
  roll	
  the	
  dice	
  prematurely	
  
A/B	
  test	
  bu...
SegmentaAon	
  
• 
• 
• 
• 
• 
• 

New	
  versus	
  returning	
  customers	
  
Different	
  browsers	
  
Accept-­‐Language	...
See	
  Also	
  
•  The	
  MulA-­‐Armed	
  Bandit	
  
hDp://stevehanov.ca/blog/index.php?id=132	
  

January	
  14,	
  2014...
Conclusion	
  
• 
• 
• 
• 
• 

Lots	
  of	
  money	
  lying	
  on	
  the	
  table	
  
Egos	
  are	
  bad	
  
Data	
  is	
 ...
QuesAons?	
  

hDp://www.allaroundtheworld.fr/	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.f...
Upcoming SlideShare
Loading in …5
×

A/B Testing - What your mother didn't tell you

6,358 views

Published on

Many people think A/B testing is merely showing customers two versions of something and choosing the one that customers respond to better.

It's much, much more complex than that.

Published in: Marketing, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,358
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
5
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

A/B Testing - What your mother didn't tell you

  1. 1. January  14,  2014   P.O.P.  Copyright  2013,  All  Around  The  World  
  2. 2. A/B  TesAng       What  your  mother  didn’t  tell  you   hDp://www.allaroundtheworld.fr/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  3. 3. What  is  A/B  TesAng?   •  TesAng  customers,  not  code   •  Show  different  versions   •  Pick  the  best  one   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  4. 4. Who  Relies  On  A/B  TesAng?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  5. 5. Transform  Your  Company   •  Test  your  customers,  not  your  code   •  Stop  guessing  what  works   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  6. 6. Should  your  paycheck   depend  on  my  hunch?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  7. 7. Conversion  Funnel   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  8. 8. Conversion  Funnel   →   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  9. 9. A  Basic  Test   •  •  •  •  Your  current  behavior  is  the  base   A  variant  is  a  new  version  ( just  one  change)   You  can  have  one  or  more  variants   But  we  assume  only  one  for  this  talk   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  10. 10. January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  11. 11. Which  Generates  More  Clicks?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  12. 12. Which  Generates  More  Clicks?   10,017  Visits   125  Conversions   January  14,  2014   9,983  Visits   132  Conversions   ©  2014  hDp://www.allaroundtheworld.fr/  
  13. 13. Which  Generates  More  Clicks?   9,950  Visits   125  Conversions   January  14,  2014   10,550  Visits   155  Conversions   ©  2014  hDp://www.allaroundtheworld.fr/  
  14. 14. The  Numbers  (95%  Confidence)   Version   Visits   Conversion   Rate   Not  Random   Red   10,017   125   1.25%   —   Blue   9,983   132   1.32%   51%   hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  15. 15. The  Numbers  (95%  Confidence)   Version   Visits   Conversion   Rate   Not  Random   Red   9,950   125   1.26%   —   Blue   10,550   155   1.47%   94%   hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  16. 16. Which  Sells  More  Product?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  17. 17. Which  Creates  More  Profit?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  18. 18. What  Confidence  Level?   Is  99%  beDer  than  90%?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  19. 19. The  Cycle   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  20. 20. Create  Experiment   Only  one  difference  per  version!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  21. 21. Roll  the  Dice   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  22. 22. Track  Performance   Sta:s:cs   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  23. 23. Choose  the  Best   Pa:ence   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  24. 24. Create  Experiment   Wash,  rinse,  repeat   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  25. 25. Choose  the  Best   •  •  •  •  Ignore  the  first  day   Full-­‐business  cycle   What  if  you’re  wrong?   False  posiAves   hDp://www.einarsen.no/is-­‐your-­‐ab-­‐tesAng-­‐effort-­‐just-­‐chasing-­‐staAsAcal-­‐ghosts/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  26. 26. LimitaAons  of  A/B  TesAng   •  Doesn’t  tell  you  why   •  Side  effects   •  Doesn’t  tell  you  loyalty   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  27. 27. Benefits   •  •  •  •  •  •  •  •  Hyundai:  62%  increase  in  test  drive  requests   Artsy  Editor:  47%  increase  in  click-­‐throughs   Visual  Website  OpAmizer:  50%  signup  increase   AssessmentDay:  62%  increase  in  sales   SmileyCookie:  41%  increase  in  sales   BaseKit:  25%  increase  in  conversions   Mobal:  27%  increase  in  sales   You  Need  A  Budget:  85%  increase  in  downloads   hDp://visualwebsiteopAmizer.com/case-­‐studies.php  
  28. 28. What  Are  They  TesAng?   •  •  •  •  •  •  •  •  Add-­‐to-­‐cart  buDon  placement   Call  to  acAon  buDons   Using  popup  forms   Removing  content   Link  color   Adding  tesAmonials   NavigaAon  menus   Removing  signup  forms   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  29. 29. Gemng  Started   •  •  •  •  Vendor  or  in-­‐house?   Go  for  quick  wins   Run  an  AA  test   Be  pa:ent!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  30. 30. Problems   Ego   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  31. 31. Problems   •  •  •  •  •  •  •  Forgemng  caching   Non-­‐random  selecAon   Forgemng  selecAon   TesAng  only  the  view   Performance   Dead  code   Profit  versus  conversion   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  32. 32. Tips   •  •  •  •  •  Most  experiments  fail   Don’t  roll  the  dice  prematurely   A/B  test  bugs   Test  “stupid”  ideas   Don’t  require  rollouts!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  33. 33. SegmentaAon   •  •  •  •  •  •  New  versus  returning  customers   Different  browsers   Accept-­‐Language   Time  of  day   IP  country   Gender?  Age?  Something  else?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  34. 34. See  Also   •  The  MulA-­‐Armed  Bandit   hDp://stevehanov.ca/blog/index.php?id=132   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  35. 35. Conclusion   •  •  •  •  •  Lots  of  money  lying  on  the  table   Egos  are  bad   Data  is  good   Customer  behavior  trumps  expert  opinion   Customer  behavior  trumps  code  behavior   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  36. 36. QuesAons?   hDp://www.allaroundtheworld.fr/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

×