Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
January	
  14,	
  2014	
  

P.O.P.	
  Copyright	
  2013,	
  All	
  Around	
  The	
  World	
  
A/B	
  TesAng 	
   	
  	
  
What	
  your	
  mother	
  didn’t	
  tell	
  you	
  
hDp://www.allaroundtheworld.fr/	
  

Janua...
What	
  is	
  A/B	
  TesAng?	
  
•  TesAng	
  customers,	
  not	
  code	
  
•  Show	
  different	
  versions	
  
•  Pick	
 ...
Who	
  Relies	
  On	
  A/B	
  TesAng?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Transform	
  Your	
  Company	
  
•  Test	
  your	
  customers,	
  not	
  your	
  code	
  
•  Stop	
  guessing	
  what	
  w...
Should	
  your	
  paycheck	
  
depend	
  on	
  my	
  hunch?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaro...
Conversion	
  Funnel	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Conversion	
  Funnel	
  

→	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
A	
  Basic	
  Test	
  
• 
• 
• 
• 

Your	
  current	
  behavior	
  is	
  the	
  base	
  
A	
  variant	
  is	
  a	
  new	
 ...
January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Generates	
  More	
  Clicks?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Generates	
  More	
  Clicks?	
  

10,017	
  Visits	
  

125	
  Conversions	
  

January	
  14,	
  2014	
  

9,983...
Which	
  Generates	
  More	
  Clicks?	
  

9,950	
  Visits	
  

125	
  Conversions	
  

January	
  14,	
  2014	
  

10,550...
The	
  Numbers	
  (95%	
  Confidence)	
  

Version	
  

Visits	
  

Conversion	
  

Rate	
  

Not	
  Random	
  

Red	
  

1...
The	
  Numbers	
  (95%	
  Confidence)	
  

Version	
  

Visits	
  

Conversion	
  

Rate	
  

Not	
  Random	
  

Red	
  

9...
Which	
  Sells	
  More	
  Product?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Which	
  Creates	
  More	
  Profit?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
What	
  Confidence	
  Level?	
  

Is	
  99%	
  beDer	
  than	
  90%?	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://ww...
The	
  Cycle	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Create	
  Experiment	
  

Only	
  one	
  difference	
  per	
  version!	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://...
Roll	
  the	
  Dice	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Track	
  Performance	
  

Sta:s:cs	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Choose	
  the	
  Best	
  

Pa:ence	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Create	
  Experiment	
  

Wash,	
  rinse,	
  repeat	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundthewo...
Choose	
  the	
  Best	
  
• 
• 
• 
• 

Ignore	
  the	
  first	
  day	
  
Full-­‐business	
  cycle	
  
What	
  if	
  you’re	...
LimitaAons	
  of	
  A/B	
  TesAng	
  
•  Doesn’t	
  tell	
  you	
  why	
  
•  Side	
  effects	
  
•  Doesn’t	
  tell	
  you...
Benefits	
  
• 
• 
• 
• 
• 
• 
• 
• 

Hyundai:	
  62%	
  increase	
  in	
  test	
  drive	
  requests	
  
Artsy	
  Editor:	
...
What	
  Are	
  They	
  TesAng?	
  
• 
• 
• 
• 
• 
• 
• 
• 

Add-­‐to-­‐cart	
  buDon	
  placement	
  
Call	
  to	
  acAon	...
Gemng	
  Started	
  
• 
• 
• 
• 

Vendor	
  or	
  in-­‐house?	
  
Go	
  for	
  quick	
  wins	
  
Run	
  an	
  AA	
  test	
...
Problems	
  

Ego	
  
January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.fr/	
  
Problems	
  
• 
• 
• 
• 
• 
• 
• 

Forgemng	
  caching	
  
Non-­‐random	
  selecAon	
  
Forgemng	
  selecAon	
  
TesAng	
 ...
Tips	
  
• 
• 
• 
• 
• 

Most	
  experiments	
  fail	
  
Don’t	
  roll	
  the	
  dice	
  prematurely	
  
A/B	
  test	
  bu...
SegmentaAon	
  
• 
• 
• 
• 
• 
• 

New	
  versus	
  returning	
  customers	
  
Different	
  browsers	
  
Accept-­‐Language	...
See	
  Also	
  
•  The	
  MulA-­‐Armed	
  Bandit	
  
hDp://stevehanov.ca/blog/index.php?id=132	
  

January	
  14,	
  2014...
Conclusion	
  
• 
• 
• 
• 
• 

Lots	
  of	
  money	
  lying	
  on	
  the	
  table	
  
Egos	
  are	
  bad	
  
Data	
  is	
 ...
QuesAons?	
  

hDp://www.allaroundtheworld.fr/	
  

January	
  14,	
  2014	
  

©	
  2014	
  hDp://www.allaroundtheworld.f...
Upcoming SlideShare
Loading in …5
×

A/B Testing - What your mother didn't tell you

6,662 views

Published on

Many people think A/B testing is merely showing customers two versions of something and choosing the one that customers respond to better.

It's much, much more complex than that.

Published in: Marketing, Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/i1K5P ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ http://1url.pw/i1K5P ◀ ◀ ◀ ◀
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

A/B Testing - What your mother didn't tell you

  1. 1. January  14,  2014   P.O.P.  Copyright  2013,  All  Around  The  World  
  2. 2. A/B  TesAng       What  your  mother  didn’t  tell  you   hDp://www.allaroundtheworld.fr/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  3. 3. What  is  A/B  TesAng?   •  TesAng  customers,  not  code   •  Show  different  versions   •  Pick  the  best  one   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  4. 4. Who  Relies  On  A/B  TesAng?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  5. 5. Transform  Your  Company   •  Test  your  customers,  not  your  code   •  Stop  guessing  what  works   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  6. 6. Should  your  paycheck   depend  on  my  hunch?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  7. 7. Conversion  Funnel   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  8. 8. Conversion  Funnel   →   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  9. 9. A  Basic  Test   •  •  •  •  Your  current  behavior  is  the  base   A  variant  is  a  new  version  ( just  one  change)   You  can  have  one  or  more  variants   But  we  assume  only  one  for  this  talk   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  10. 10. January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  11. 11. Which  Generates  More  Clicks?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  12. 12. Which  Generates  More  Clicks?   10,017  Visits   125  Conversions   January  14,  2014   9,983  Visits   132  Conversions   ©  2014  hDp://www.allaroundtheworld.fr/  
  13. 13. Which  Generates  More  Clicks?   9,950  Visits   125  Conversions   January  14,  2014   10,550  Visits   155  Conversions   ©  2014  hDp://www.allaroundtheworld.fr/  
  14. 14. The  Numbers  (95%  Confidence)   Version   Visits   Conversion   Rate   Not  Random   Red   10,017   125   1.25%   —   Blue   9,983   132   1.32%   51%   hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  15. 15. The  Numbers  (95%  Confidence)   Version   Visits   Conversion   Rate   Not  Random   Red   9,950   125   1.26%   —   Blue   10,550   155   1.47%   94%   hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  16. 16. Which  Sells  More  Product?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  17. 17. Which  Creates  More  Profit?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  18. 18. What  Confidence  Level?   Is  99%  beDer  than  90%?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  19. 19. The  Cycle   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  20. 20. Create  Experiment   Only  one  difference  per  version!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  21. 21. Roll  the  Dice   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  22. 22. Track  Performance   Sta:s:cs   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  23. 23. Choose  the  Best   Pa:ence   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  24. 24. Create  Experiment   Wash,  rinse,  repeat   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  25. 25. Choose  the  Best   •  •  •  •  Ignore  the  first  day   Full-­‐business  cycle   What  if  you’re  wrong?   False  posiAves   hDp://www.einarsen.no/is-­‐your-­‐ab-­‐tesAng-­‐effort-­‐just-­‐chasing-­‐staAsAcal-­‐ghosts/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  26. 26. LimitaAons  of  A/B  TesAng   •  Doesn’t  tell  you  why   •  Side  effects   •  Doesn’t  tell  you  loyalty   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  27. 27. Benefits   •  •  •  •  •  •  •  •  Hyundai:  62%  increase  in  test  drive  requests   Artsy  Editor:  47%  increase  in  click-­‐throughs   Visual  Website  OpAmizer:  50%  signup  increase   AssessmentDay:  62%  increase  in  sales   SmileyCookie:  41%  increase  in  sales   BaseKit:  25%  increase  in  conversions   Mobal:  27%  increase  in  sales   You  Need  A  Budget:  85%  increase  in  downloads   hDp://visualwebsiteopAmizer.com/case-­‐studies.php  
  28. 28. What  Are  They  TesAng?   •  •  •  •  •  •  •  •  Add-­‐to-­‐cart  buDon  placement   Call  to  acAon  buDons   Using  popup  forms   Removing  content   Link  color   Adding  tesAmonials   NavigaAon  menus   Removing  signup  forms   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  29. 29. Gemng  Started   •  •  •  •  Vendor  or  in-­‐house?   Go  for  quick  wins   Run  an  AA  test   Be  pa:ent!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  30. 30. Problems   Ego   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  31. 31. Problems   •  •  •  •  •  •  •  Forgemng  caching   Non-­‐random  selecAon   Forgemng  selecAon   TesAng  only  the  view   Performance   Dead  code   Profit  versus  conversion   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  32. 32. Tips   •  •  •  •  •  Most  experiments  fail   Don’t  roll  the  dice  prematurely   A/B  test  bugs   Test  “stupid”  ideas   Don’t  require  rollouts!   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  33. 33. SegmentaAon   •  •  •  •  •  •  New  versus  returning  customers   Different  browsers   Accept-­‐Language   Time  of  day   IP  country   Gender?  Age?  Something  else?   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  34. 34. See  Also   •  The  MulA-­‐Armed  Bandit   hDp://stevehanov.ca/blog/index.php?id=132   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  35. 35. Conclusion   •  •  •  •  •  Lots  of  money  lying  on  the  table   Egos  are  bad   Data  is  good   Customer  behavior  trumps  expert  opinion   Customer  behavior  trumps  code  behavior   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  
  36. 36. QuesAons?   hDp://www.allaroundtheworld.fr/   January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

×