Your SlideShare is downloading. ×
0
>	  Tes&ng	  for	  Success	  <	     Elements	  of	  a	  Successful	  Tes0ng	                   Program	  
>	  Agenda	  § Why	  Test?	  	   	     	   	   	  	  § Problem	  Diagnosis	  § Deciding	  what	  to	  Test 	     	  	  ...
10101101001001001010111101001001010101010000101111100101010101010010101100110001010010100110110100110100101010011100101001...
1.  Why	  does	  your	                 EVERYONE’S	      business/organisa0on	      exist?	     GOT	  AN	  2.  How	  can	  ...
>	  Why	  Test?	  1.  Systema0c	  Innova0on	  2.  Avoid	  costly	  mistakes	  3.  Know	  why	  things	  go	  right,	  know...
10101101001001001010111101001001010101010000101111100101010101010010101100110001010010100110110100110100101010011100101001...
>	  What	  is	  the	  business	  problem?	        Acquisi0on	          Up-­‐Sell	                            Reten0on	    ...
>	  Case	  Study	  December	  2011	       ©	  Datalicious	  Pty	  Ltd	     8	  
>	  Further	  Diagnosis	                           PROBLEM:	  Sales	  through	  online	                                 No...
>	  Further	  Diagnosis	  II	                         Source:	  www.feng-­‐gui.com	  December	  2011	                     ...
>	  Some&mes	  the	  small	  things	  count	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     11	  
>	  Further	  diagnosis	  III	                                         Wrong	  message?	                                  ...
>	  Tes&ng	  as	  risk	  mi&ga&on	                                                                       Roll-­‐out	  Chan...
>	  Tes&ng	  as	  standard	  prac&ce	                                                                       Test	  Market	...
10101101001001001010111101001001010101010000101111100101010101010010101100110001010010100110110100110100101010011100101001...
Don’t	  reinvent	  the	  wheel	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     16	  
>	  What	  are	  the	  solu&on(s)?	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     17	  
>	  Consumer	  Empathy	   What	  are	  your	  visitors	  trying	  to	  achieve	  by	  visi2ng	  your	  site?	  December	  ...
>	  Consumer	  Empathy	  1.  Make	  it	  visible	         –  People	  can’t	  convert	  if	  they	  can’t	  find	  your	   ...
>	  Start	  with	  the	  basics…	  1.	  The	  headline	          –  Have	  a	  headline!	          –  Headline	  should	  ...
>	  Start	  with	  the	  basics…	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     21	  
>	  Case	  Study	  December	  2011	       ©	  Datalicious	  Pty	  Ltd	     22	  
>	  Further	  Examples	                         TEST	  A	                                          EXISTING	  December	  2...
>	  Further	  Examples	       EXISTING	           TEST	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     24	  
>	  Deciding	  What	  to	  Test	                            Test	  Selec0on	  Checklist	  §    Is	  the	  measurement	  i...
>	  	  Do	  you	  have	  the	  repor&ng?	                For	  each	  of	  Segment	  X,	  Y	  and	  Z...	                 ...
>	  Offline	  conversions	  from	  online	   Tying	  offline	  conversions	  back	  to	  online	  campaign	  and	  research	  ...
>	  Search	  call	  to	  ac&on	  for	  offline	  	  December	  2011	      ©	  Datalicious	  Pty	  Ltd	     28	  
>	  OTP	  Response	          –  Different	  numbers	  for	  different	  media	  channels	          –  Different	  numbers	  f...
>	  Whose	  help	  do	  you	  need?	  Technology/IT	         UX Agency    Analytics!Your boss, Your boss’ boss Creative Ag...
>	  Proving	  the	  Value	   GO	  BIG	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     31	  
>	  How	  much	  sample	  do	  I	  need?	                                          BAU/Baseline	                          ...
>	  Sta&s&cal	  Significance	  Q.	  How	  much	  am	  I	  willing	  to	  accept	  that	  the	  	  	  	       difference	  in...
>	  Type	  I	  and	  Type	  II	  Error	  Type	  I:	   	  Accept	  result	  to	  be	  true	  when	  it’s	                 	...
>	  Es&ma&ng	  Sample	  Size	  (%s)	                             2 # p1 (1− p1 ) + p2 (1− p2 ) &          n = (1.645+1.282...
>	  Es&ma&ng	  Sample	  Size	  (%s)	      Typical	  Champion	  (control)	  vs.	  Challenger	  (test)	  A|B	  test,	  typic...
>	  Es&ma&ng	  Sample	  Size	  ($s)	                                   (1.645 +1.282)2 * (s12 + s2 )                      ...
>	  Standard	  Devia&on	      Standard	  devia0on	  is	  measure	  of	  the	  variability	  of	  your	  results,	  whether...
>	  Es&ma&ng	  Sample	  Size	  ($s)	      Typical	  Champion	  (control)	  vs.	  Challenger	  (test)	  A|B	  test,	  typic...
>	  Further	  Complexity	  I	      If	  we	  wanted	  to	  test	  the	  performance	  of	  Challenger	  vs.	  Champion	  f...
>	  Further	  Complexity	  II	      If	  we	  wanted	  to	  test	  the	  performance	  of	  Challenger	  vs.	  Champion	  ...
>	  Further	  Complexity	  III	      If	  we	  wanted	  to	  test	  the	  performance	  of	  Challenger	  crea0ve	  that	 ...
>	  Mul&variate	  Tes&ng	      Mul0variate	  Tes0ng	  (commonly	  called	  MVT)	  is	  a	  term	  used	  for	  tes0ng	  di...
>	  MVT	  –	  Full	  Factorial	      A	  full	  factorial	  design	  requires	  every	  unique	  combina0on	  of	  page	  ...
>	  MVT	  –	  Frac&onal	  Factorial	      The	  alterna0ve	  is	  called	  a	  frac0onal	  factorial	  design	  which	  is...
>	  Layout	  Before	  Content	  §  Phase	  #1:	  A|B	  test	          –  Test	  the	  same	  landing	                    ...
>	  Case	  Study	  §    Yes,	  the	  measurement	  infrastructure	  is	  in	  place	  §    I	  can	  readily	  execute	 ...
10101101001001001010111101001001010101010000101111100101010101010010101100110001010010100110110100110100101010011100101001...
Before	  you	  leap…	  December	  2011	              ©	  Datalicious	  Pty	  Ltd	     49	  
>	  Sample	  Selec&on	  §  Each	  sample	  needs	  to	  be	  alike	  in	  terms	  of	      their	  predisposi0on	  to	  c...
>	  Timing	  is	  Important	                            	  ‘Burst’	  Non	  BAU	  ATL	                          Ideal	  Tes...
>	  A|A	  Tes&ng	  §  Set	  a	  test	  that	  splits	  your	  visitors	  50/50	      between	  the	  same	  treatment	   ...
>	  Measuring	  your	  performance	  §  Propor0ons	  (conversion	  rates)	  §  Means	  (average	  $s)	  §  Variability	...
>	  Confidence	  Intervals	        Conversion	  Rate	                                                                      ...
>	  Confidence	  Intervals	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     55	  
>	  Confidence	  Interval	  (%s)	                                                           ˆ    ˆ                         ...
>	  Confidence	  Interval	  Es&ma&on	                         Typical	  Champion	  (control)	  vs.	  Challenger	  (test)	  ...
>	  Confidence	  Interval	  Es&ma&on	                                  p1 (1− p1 ) p2 (1− p2 )                p1 − p2 ±1.96...
>	  Confidence	  Interval	  Es&ma&on	                          Typical	  Champion	  (control)	  vs.	  Challenger	  (test)	 ...
>	  Control	  Group	  Sample	  Size	                                                      p1 (1− p1 ) p2 (1− p2 )         ...
>	  Control	  Group	  Sample	  Size	          We	  have	  50,000	  customers	  that	  we	  could	  include	  in	  our	  te...
>	  Confidence	  intervals	  ($s)	                                                                s                        ...
>	  Standard	  Devia&on	  (reminder)	      Standard	  devia0on	  is	  measure	  of	  the	  variability	  of	  your	  resul...
>	  Confidence	  intervals	  ($s)	                                                                                         ...
>	  Confidence	  intervals	  ($s)	                         Typical	  Champion	  (control)	  vs.	  Challenger	  (test)	  A|B...
>	  Case	  Study	  December	  2011	       ©	  Datalicious	  Pty	  Ltd	     66	  
>	  Main	  Effects	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     67	  
>	  Main	  Effects	                                                 Typical	  Landing	  Page	  Test	                       ...
>	  Main	  Effects	                                                          Typical	  Landing	  Page	  Test	              ...
>	  Main	  Effects	                                                   Typical	  Landing	  Page	  Test	                     ...
>	  Interac&on	  Effects	                                                     Typical	  Landing	  Page	  Test	             ...
>	  Interac&on	  Effects	                                                          Typical	  Landing	  Page	  Test	        ...
>	  Interac&on	  Effects	                                                             Typical	  Landing	  Page	  Test	     ...
>	  Interac&on	  Effects	                                                                   Typical	  Landing	  Page	  Test...
10101101001001001010111101001001010101010000101111100101010101010010101100110001010010100110110100110100101010011100101001...
Document	  Everything!	  December	  2011	     ©	  Datalicious	  Pty	  Ltd	     76	  
>	  1.	  Describe	  the	  test	  §  Describe	  the	  outcome(s)	  you’re	  trying	  to	      influence	  §  Describe	  yo...
>	  2.	  Jus&fy	  the	  test	  design	  §  Detail	  why	  you’ve	  chosen	  the	  par0cular	  	      outcome	  you’re	  t...
>	  3.	  Results	  &	  Conclusions	  §  Detail	  all	  the	  performance	  results	  §  Discuss	  your	  hypotheses	  §...
>	  The	  Scien&fic	  Method	                         Knowledge                             	     Establish	               ...
>	  Case	  Study	  December	  2011	       ©	  Datalicious	  Pty	  Ltd	     81	  
>	  List	  of	  (Some)	  Resources	  §  hRp://visualwebsiteop0mizer.com/case-­‐    studies.php	  §  hRp://www.whichtestw...
Contact	  us	                         msavio@datalicious.com	                                  	                          ...
Data	  >	  Insights	  >	  Ac&on	  
Upcoming SlideShare
Loading in...5
×

Testing for Success

196

Published on

The presentation discusses the significance of testing and how to execute a successful testing program.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
196
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Testing for Success"

  1. 1. >  Tes&ng  for  Success  <   Elements  of  a  Successful  Tes0ng   Program  
  2. 2. >  Agenda  § Why  Test?              § Problem  Diagnosis  § Deciding  what  to  Test      § Test  Execu0on  and  Measurement  § Test  Repor0ng  December  2011   ©  Datalicious  Pty  Ltd   2  
  3. 3. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010  >  Why  Test?  December  2011   ©  Datalicious  Pty  Ltd   3  
  4. 4. 1.  Why  does  your   EVERYONE’S   business/organisa0on   exist?   GOT  AN  2.  How  can  your  business/ OPINION   organisa0on  improve?  December  2011   ©  Datalicious  Pty  Ltd   4  
  5. 5. >  Why  Test?  1.  Systema0c  Innova0on  2.  Avoid  costly  mistakes  3.  Know  why  things  go  right,  know  why  things   go  wrong  4.  BeRer  employee  engagement  §  Requires  planning  and  governance!  December  2011   ©  Datalicious  Pty  Ltd   5  
  6. 6. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010  >  Problem  Diagnosis  December  2011   ©  Datalicious  Pty  Ltd   6  
  7. 7. >  What  is  the  business  problem?   Acquisi0on   Up-­‐Sell   Reten0on   Advocacy   Analy&cs  and  metrics  frameworks  December  2011   ©  Datalicious  Pty  Ltd   7  
  8. 8. >  Case  Study  December  2011   ©  Datalicious  Pty  Ltd   8  
  9. 9. >  Further  Diagnosis   PROBLEM:  Sales  through  online   Not  enough  site  traffic   High  home  page  bounce  rate   Low  conversion  on  product  page   Checkout  fallout  December  2011   ©  Datalicious  Pty  Ltd   9  
  10. 10. >  Further  Diagnosis  II   Source:  www.feng-­‐gui.com  December  2011   ©  Datalicious  Pty  Ltd   10  
  11. 11. >  Some&mes  the  small  things  count  December  2011   ©  Datalicious  Pty  Ltd   11  
  12. 12. >  Further  diagnosis  III   Wrong  message?   Wrong  channel?   Wrong  person?   Wrong  0me?  December  2011   ©  Datalicious  Pty  Ltd   12  
  13. 13. >  Tes&ng  as  risk  mi&ga&on   Roll-­‐out  Channel     Press   TV   Radio   Outdoor   Offer,   Crea&ve,   Call-­‐to-­‐ Offer,  Call-­‐ Offer,  Call-­‐ eDM/DM   Call-­‐to-­‐ Ac&on   to-­‐Ac&on   to-­‐Ac&on   Ac&on   Test   Paid   Channel   Search   Offer   Offer   Offer   Offer   Crea&ve,   Display   Offer,  Call-­‐ Offer,  Call-­‐ -­‐   Crea&ve   Media   to  Ac&on   to  Ac&on  December  2011   ©  Datalicious  Pty  Ltd   13  
  14. 14. >  Tes&ng  as  standard  prac&ce   Test  Market   Control  Market  (no  ATL)      %  Uplib  in  Sales    TimeDecember  2011   ©  Datalicious  Pty  Ltd   14  
  15. 15. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010  >  Deciding  what  to  Test  December  2011   ©  Datalicious  Pty  Ltd   15  
  16. 16. Don’t  reinvent  the  wheel  December  2011   ©  Datalicious  Pty  Ltd   16  
  17. 17. >  What  are  the  solu&on(s)?  December  2011   ©  Datalicious  Pty  Ltd   17  
  18. 18. >  Consumer  Empathy   What  are  your  visitors  trying  to  achieve  by  visi2ng  your  site?  December  2011   ©  Datalicious  Pty  Ltd   18  
  19. 19. >  Consumer  Empathy  1.  Make  it  visible   –  People  can’t  convert  if  they  can’t  find  your   ‘Buy  Now’  buRon  2.  Make  it  relevant   –  Need  to  resolve  consumer  reserva0ons/ ques0ons  3.  Make  it  easy   –  Easy  naviga0on,  easy  form  comple0on,  easy  to   read,  quick  page  load  December  2011   ©  Datalicious  Pty  Ltd   19  
  20. 20. >  Start  with  the  basics…  1.  The  headline   –  Have  a  headline!   –  Headline  should  be  concrete   –  Headline  should  be  first  thing  visitors  look  at  2.  Call  to  ac&on   –  Don’t  have  too  many  calls  to  ac0on   –  Have  an  ac0onable  call  to  ac0on   –  Have  a  big,  prominent,  visible  call  to  ac0on  3.  Social  proof   –  Logos,  number  of  users,  tes0monials,     case  studies,  media  coverage,  etc  December  2011   ©  Datalicious  Pty  Ltd   20  
  21. 21. >  Start  with  the  basics…  December  2011   ©  Datalicious  Pty  Ltd   21  
  22. 22. >  Case  Study  December  2011   ©  Datalicious  Pty  Ltd   22  
  23. 23. >  Further  Examples   TEST  A   EXISTING  December  2011   ©  Datalicious  Pty  Ltd   23  
  24. 24. >  Further  Examples   EXISTING   TEST  December  2011   ©  Datalicious  Pty  Ltd   24  
  25. 25. >  Deciding  What  to  Test   Test  Selec0on  Checklist  §  Is  the  measurement  infrastructure  in  place  already?    [  ✔    ]                §  Can  I  readily  execute  the  solu0on?    [  ✔    ]                §  Do  I  have  enough  sample  to  draw  valid  conclusions?    [  ✔    ]                §  Will  this  prove  the  value  of  tes0ng  in  the  business?    [  ✔    ]                December  2011   ©  Datalicious  Pty  Ltd   25  
  26. 26. >    Do  you  have  the  repor&ng?   For  each  of  Segment  X,  Y  and  Z...   Test  Channel     ATL   DM   eDM   Online   Online   ✔   ✔   Mailroom   ✔   Response   Call  Centre   Channel   Bricks  &   Mortar   Channels  in   ✔   Aggregate  December  2011   ©  Datalicious  Pty  Ltd   26  
  27. 27. >  Offline  conversions  from  online   Tying  offline  conversions  back  to  online  campaign  and  research  behavior  using   standard  cookie  technology  by  triggering  virtual  online  order  confirma0on   pages  for  offline  sales  using  email  receipts.   Website.com   Phone   Virtual  Order   Research   Orders   @   Confirma&on   Online  Ad   Website.com   Retail   Virtual  Order   Campaign   Research   Orders   @   Confirma&on   Website.com   Online   Online  Order   Virtual  Order   Research   Orders   Confirma&on   @   Confirma&on   Cookie   Cookie   Cookie  December  2011   ©  Datalicious  Pty  Ltd   27  
  28. 28. >  Search  call  to  ac&on  for  offline    December  2011   ©  Datalicious  Pty  Ltd   28  
  29. 29. >  OTP  Response   –  Different  numbers  for  different  media  channels   –  Different  numbers  for  different  product   categories   –  Different  numbers  for  different  conversion  steps   –  Call  origin  becoming  useful  to  shape  call  script   –  Feasible  to  pause  numbers  to  improve  integrity  December  2011   ©  Datalicious  Pty  Ltd   29  
  30. 30. >  Whose  help  do  you  need?  Technology/IT   UX Agency Analytics!Your boss, Your boss’ boss Creative Agency Customer Contact ManagementDecember  2011   ©  Datalicious  Pty  Ltd   30  
  31. 31. >  Proving  the  Value   GO  BIG  December  2011   ©  Datalicious  Pty  Ltd   31  
  32. 32. >  How  much  sample  do  I  need?   BAU/Baseline   Conversion  Rate   #  on  Segments,   #  of  Treatments   n   Expected  Δ   in  Conversion   Time  in  Market   [Digital  Only]  December  2011   ©  Datalicious  Pty  Ltd   32  
  33. 33. >  Sta&s&cal  Significance  Q.  How  much  am  I  willing  to  accept  that  the         difference  in  the  results  between  my  test   group  and  control  group  may  have  been  due   to  chance?    A.  Not  much.  I  want  to  be  confident  that  if  I   repeated  the  test  100  &mes,  then  I  would   observe  this  difference  95  &mes.       This  is  ‘95%  confidence’  December  2011   ©  Datalicious  Pty  Ltd   33  
  34. 34. >  Type  I  and  Type  II  Error  Type  I:    Accept  result  to  be  true  when  it’s    actually  false  (false  posi&ves)    Type  II:  Accept  result  to  be  false  when  it’s      actually  true  (false  nega&ves)  December  2011   ©  Datalicious  Pty  Ltd   34  
  35. 35. >  Es&ma&ng  Sample  Size  (%s)   2 # p1 (1− p1 ) + p2 (1− p2 ) & n = (1.645+1.282) * % 2 ( $ Δ Where:    n    =    es0mated  sample  size  for  each  group    p1  =    expected  conversion  rate  for  your  test  treatment    p2  =    expected  conversion  rate  for  your  control  treatment    Δ    =    expected  minimum  percentage  point  difference  between  test        and  control  results         The  value  of  1.645  reflects  that  we  accept  Type  I  error  probability  of  .05     The  value  of  1.282  reflects  that  we  accept  Type  II  error  probability  of  .10    December  2011   ©  Datalicious  Pty  Ltd   35  
  36. 36. >  Es&ma&ng  Sample  Size  (%s)   Typical  Champion  (control)  vs.  Challenger  (test)  A|B  test,  typical  champion   response  rate  of  2.5%.     •  Only  going  to  replace  Champion  with  Challenger  if  Challenger   response  rate  is  3.0%  (0.5%  is  a  meaningful  difference)       2 ! 0.025* 0.975 + 0.030 * 0.970 $ n = (1.645+1.282) * # 2 & " 0.005 % Sample  size  =  18,326  For  each  of  the  Champion  and  Challenger  groups     If  1.0%  our  meaningful  difference  then  sample  size  is  only  5,378  December  2011   ©  Datalicious  Pty  Ltd   36  
  37. 37. >  Es&ma&ng  Sample  Size  ($s)   (1.645 +1.282)2 * (s12 + s2 ) 2 n= Δ2 Where:    n    =    number  of  observa0ons  for  each  group    s1  =    expected  standard  devia0on  of  value  for  your  test  treatment    s2  =    expected  standard  devia0on  of  value  for  your  control  treatment    Δ    =    expected  minimum  difference  in  value  between  test        and  control  results         The  value  of  1.645  reflects  an  accepted  Type  I  error  probability  of  .05     The  value  of  1.282  reflects  an  accepted  Type  II  error  probability  of  .10    December  2011   ©  Datalicious  Pty  Ltd   37  
  38. 38. >  Standard  Devia&on   Standard  devia0on  is  measure  of  the  variability  of  your  results,  whether  some   your  results  are  quite  different  to  your  mean  (average)  result  or  whether  they   are  quite  similar.   n ∑(x − x ) i i=1 s= n −1 Where:    n    =    number  of  observa0ons    xi  =    the  result  for  the  ith  observa0on    x  =    mean  (average)  for  your  data  December  2011   ©  Datalicious  Pty  Ltd   38  
  39. 39. >  Es&ma&ng  Sample  Size  ($s)   Typical  Champion  (control)  vs.  Challenger  (test)  A|B  test,  typical  champion   mean  response  value  of  $20,  typical  response  rate  of  5%     •  Only  going  to  replace  Champion  with  Challenger  if  Challenger  mean   response  value  is  is  $30  ($10  is  a  meaningful  difference)   •  Standard  devia0on  of  Champion  results  is  $5  (based  on  past  results).   We’ll  assume  the  same  for  the  Challenger.         2 2 2   (1.645 +1.282) * (5 + 5 ) n= 2 10 Number  of  observa0ons  =  4.3  (~5)  for  each  of  the  Champion  and  Challenger   groups.     Then  divide  through  with  the  expected  response  rate  to  get  minimum  sample   size  of  86  for  each  of  Challenger  and  Control  groups  (4.3/0.05)  December  2011   ©  Datalicious  Pty  Ltd   39  
  40. 40. >  Further  Complexity  I   If  we  wanted  to  test  the  performance  of  Challenger  vs.  Champion  for  different   segments  of  consumers:   Response  Rate   Champion   Challenger   A   %   %   Segment   B   %   %   C   %   %   Using  same  assump0ons  as  in  earlier  example  need  18,326  per  cell,   18,326*6=109,956  in  total  .    December  2011   ©  Datalicious  Pty  Ltd   40  
  41. 41. >  Further  Complexity  II   If  we  wanted  to  test  the  performance  of  Challenger  vs.  Champion  for   difference  segments  of  consumers  AND  had  3  different  types  of  Champion   crea0ve:   Response  Rate   Challenger   Challenger   Challenger   Champion   #1   #2   #3   A   %   %   %   %   Segment   B   %   %   %   %   C   %   %   %   %   Using  same  assump0ons  as  in  earlier  example  need  18,326  per  cell,   18,326*12=219,912  in  total.    December  2011   ©  Datalicious  Pty  Ltd   41  
  42. 42. >  Further  Complexity  III   If  we  wanted  to  test  the  performance  of  Challenger  crea0ve  that  was   specifically  customised  for  difference  segments  of  consumers,  then  we’re   actually  only  running  6  tests   Response  Rate   Challenger   Challenger   Challenger   Champion   #1   #2   #3   A   %   %   Segment   B   %   %   C   %   %   Using  same  assump0ons  as  in  earlier  example  need  18,326  per  cell,   18,326*6=109.956  in  total.    December  2011   ©  Datalicious  Pty  Ltd   42  
  43. 43. >  Mul&variate  Tes&ng   Mul0variate  Tes0ng  (commonly  called  MVT)  is  a  term  used  for  tes0ng  different   varia0ons  of  typical  elements  of  a  landing  page,  direct  mail  leRer,  etc.    The  aim  is   to  determine  which  combina0on  delivers  the  best  result.   Element  #1:  Prominent   headline   §  Element  #1   Element  #2:     –  2  varia0ons  (1  exis0ng,  1  new)   Suppor0ng     Call  to   content   §  Element  #2   ac0on   –  2  varia0ons  (1  exis0ng,  1  new)   Element  #3:  Social  proof  /   §  Element  #3:   trust   –  2  varia0ons  (1  exis0ng,  1  new)   Terms  and  condi0ons  December  2011   ©  Datalicious  Pty  Ltd   43  
  44. 44. >  MVT  –  Full  Factorial   A  full  factorial  design  requires  every  unique  combina0on  of  page  elements  and   can  therefore  be  very  sample  hungry.     Element   To  calculate  the   Headline   Call  to  Ac&on   Social  Proof   number  of   1   H1   CTA1   SP1   treatments  just  need   2   H1   CTA1   SP2   to  mul0ply  the   3   H1   CTA2   SP1   number  of  varia0ons   4   H1   CTA2   SP2   Treatment   for  each  factor   5   H2   CTA1   SP1   together:   6   H2   CTA1   SP2     7   H2   CTA2   SP1   2  x  2  x  2  =    8     8   H2   CTA2   SP2  December  2011   ©  Datalicious  Pty  Ltd   44  
  45. 45. >  MVT  –  Frac&onal  Factorial   The  alterna0ve  is  called  a  frac0onal  factorial  design  which  is  some  smaller  set  of   elements  combina0ons.  The  design  should  be  ‘balanced’  -­‐  every  varia0on  is   tested  the  same  number  of  0mes  and  each  combina0on  of  varia0ons  occurs  the   same  number  of  0mes.   Element   Headline   Call  to  Ac&on   Social  Proof   1   2   H1   CTA1   SP2   Reduced  sample   3   H1   CTA2   SP1   requirements   Treatment   4   4x18,326=73,304   5   H2   CTA1   SP1   6   7   8   H2   CTA2   SP2  December  2011   ©  Datalicious  Pty  Ltd   45  
  46. 46. >  Layout  Before  Content  §  Phase  #1:  A|B  test   –  Test  the  same  landing   Element  #1:  Prominent  headline   page  content  in   completely  different   layouts  §  Phase  #2:  MV  test   Suppor0ng     Element  #2:     –  Then  test  different   content   Call  to  ac0on   content  element   combina0ons  within  the   winning  layout   Element  #3:  Social  proof  /  trust  §  Phase  #3:  MV  test  (if   req’d)   –  Test  with  reduced  set  of   Terms  and  condi0ons   elements  December  2011   ©  Datalicious  Pty  Ltd   46  
  47. 47. >  Case  Study  §  Yes,  the  measurement  infrastructure  is  in  place  §  I  can  readily  execute  the  test  design  §  I  have  enough  sample  to  draw  valid  conclusions  §  Yes,  this  design  will  prove  the  value  of  tes0ng  in  my   business  December  2011   ©  Datalicious  Pty  Ltd   47  
  48. 48. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010  >  Execu&on  &  Measurement  December  2011   ©  Datalicious  Pty  Ltd   48  
  49. 49. Before  you  leap…  December  2011   ©  Datalicious  Pty  Ltd   49  
  50. 50. >  Sample  Selec&on  §  Each  sample  needs  to  be  alike  in  terms  of   their  predisposi0on  to  conversion   Conversion:  low  rate  credit  card  applica0on  form  comple0on   TEST   CONTROL   18-­‐34   35-­‐64   Mostly  Male   Mostly  Female   Mostly  Low  Income   Mostly  High  Income  December  2011   ©  Datalicious  Pty  Ltd   50  
  51. 51. >  Timing  is  Important    ‘Burst’  Non  BAU  ATL   Ideal  Test  Window   Campaign   Sales    Time  December  2011   ©  Datalicious  Pty  Ltd   51  
  52. 52. >  A|A  Tes&ng  §  Set  a  test  that  splits  your  visitors  50/50   between  the  same  treatment   –  Check  that  sample  sizes  are  actually  50/50   –  Is  there  should  be  no  difference  in  your   conversion  rates   –  Are  volumes  of  conversions  matching  other   repor0ng?  December  2011   ©  Datalicious  Pty  Ltd   52  
  53. 53. >  Measuring  your  performance  §  Propor0ons  (conversion  rates)  §  Means  (average  $s)  §  Variability  of  Means  (standard  devia0on)   Would  my  winning  treatment  s2ll  be  the  winner   across  all  my  customers/visitors/consumers?      §  Use  confidence  intervals  December  2011   ©  Datalicious  Pty  Ltd   53  
  54. 54. >  Confidence  Intervals   Conversion  Rate   Revenue  per   Response   A   B   C   A   B   C    Treatments    Treatments  December  2011   ©  Datalicious  Pty  Ltd   54  
  55. 55. >  Confidence  Intervals  December  2011   ©  Datalicious  Pty  Ltd   55  
  56. 56. >  Confidence  Interval  (%s)   ˆ ˆ p(1− p) ˆ p ±1.96 * n Where:   ^    p    =    response  rate    n  =    sample  size  for  treatment     The  value  of  1.96  reflects  a  95%  confidence  level  December  2011   ©  Datalicious  Pty  Ltd   56  
  57. 57. >  Confidence  Interval  Es&ma&on   Typical  Champion  (control)  vs.  Challenger  (test)  A|B  Test   Treatment   Champion   Challenger   Mailed   60850   52812   Responses   1055   455   Response  Rate   1.7   0.9   .017(1−.017) .009(1−.009) 1.7% ±1.96 * 0.9% ±1.96 * 60850 52812 1.7% ± 0.10% 0.9% ± 0.08% 1.69%  ≤  Champion  ≤    1.71%   0.82%  ≤  Challenger  ≤    0.98%  December  2011   ©  Datalicious  Pty  Ltd   57  
  58. 58. >  Confidence  Interval  Es&ma&on   p1 (1− p1 ) p2 (1− p2 ) p1 − p2 ±1.96 * + n1 n2 Where:    p1   =    response  rate  for  challenger    p2   =    response  rate  for  champion      n1  =    sample  size  for  challenger    n2  =    sample  size  for  challenger     The  value  of  1.96  reflects  a  95%  confidence  level  December  2011   ©  Datalicious  Pty  Ltd   58  
  59. 59. >  Confidence  Interval  Es&ma&on   Typical  Champion  (control)  vs.  Challenger  (test)  A|B  Test   Treatment   Champion   Challenger   Mailed   60850   52812   Responses   1055   455   Response  Rate   1.7   0.9   .009(1−.009) .017(1−.017) 0.9 −1.7 ±1.96 * + 52812 60850 −0.8 ± 0.13 -­‐0.93%  ≤  Difference  Between  Challenger  and  Champion  ≤    -­‐0.67%  December  2011   ©  Datalicious  Pty  Ltd   59  
  60. 60. >  Control  Group  Sample  Size   p1 (1− p1 ) p2 (1− p2 ) p1 − p2 ±1.96 * + n1 n2 pc (1− pc ) Rearranged:   nc = 2 " m % pt (1− pt ) $ − # 1.96 & nt Where:    nc    =    sample  size  for  control  group    nt    =    sample  size  for  test  group    pc  =    forecast  response  rate  for  control  group    nt  =    forecast  response  rate  for  test  group    m  =    desired  level  of  precision  (%  that  is  a  meaningful  difference)       The  value  of  1.96  reflects  a  95%  confidence  level  December  2011   ©  Datalicious  Pty  Ltd   60  
  61. 61. >  Control  Group  Sample  Size   We  have  50,000  customers  that  we  could  include  in  our  test  design,  what   would  our  control  sample  need  to  be  if  we  tested  40,000  customers,  our   ‘natural’  cross-­‐sell  rate  was  1.0%  and  an  incremental  response  rate  of  1.0%   points  would  be  deemed  to  be  meaningful?   .01(1−.01) nc = 2 " .01 % .02(.02 −.02) $ − # 1.96 & 40, 000 nc = 387 This  result  suggests  we  could  actually  test  more  of  our  available  customer  base   than  we  might  have  ini0ally  expected  (~40,600).  December  2011   ©  Datalicious  Pty  Ltd   61  
  62. 62. >  Confidence  intervals  ($s)   s x ±1.96 * n Where:    x    =    mean  revenue  among  treatment  responders    s  =    standard  devia0on  of  revenue  among  some  treatment’s  responders    n  =    number  of  responders  to  the  treatment     The  value  of  1.96  reflects  a  95%  level  of  confidence.    December  2011   ©  Datalicious  Pty  Ltd   62  
  63. 63. >  Standard  Devia&on  (reminder)   Standard  devia0on  is  measure  of  the  variability  of  your  results,  whether  some   your  results  are  quite  different  to  your  mean  (average)  result  or  whether  they   are  quite  similar.   n ∑(x − x ) i i=1 s= n −1 Where:    n    =    number  of  observa0ons    xi  =    the  result  for  the  ith  observa0on    x  =    mean  (average)  for  your  data  December  2011   ©  Datalicious  Pty  Ltd   63  
  64. 64. >  Confidence  intervals  ($s)   2 2 s s 1 2 x1 − x2 ±1.96 * + n1 n2 Where:    x1  =    mean  value  among  among  responders  to  a  treatment    x2  =    mean  value  among  among  responders  to  a  different  treatment      s1  =    std.  dev.  of  value  among  one  treatment’s  responders    s2  =    std.  dev.  of  value  among  the  other  treatment’s  responders  n1  =    number  of  responders  to  the  treatment    n2  =    number  of  responders  to  the  other  treatment     The  value  of  1.96  reflects  a  95%  level  of  confidence.   n1  and  n2  is  sufficiently  large  to  es0mate  the  std.  dev.  in  the  popula0on  with   the  std.  dev.  of  the  sample.  December  2011   ©  Datalicious  Pty  Ltd   64  
  65. 65. >  Confidence  intervals  ($s)   Typical  Champion  (control)  vs.  Challenger  (test)  A|B  Test   Treatment   Champion   Challenger   Mailed   60850   52812   Responses   1055   455   Response  Rate   1.7   0.9   Total  Value   $36,925   $38,675   Mean  Value   $35   $85   Std  Dev   $30   $50   50 2 30 2 85 − 35 ±1.96 * + 50 ± 4.9 455 1055At  a  minimum,  we  should  expect  an  incremental  $45.1  if  we  rolled  out  the  Challenger  crea0ve  as  BAU  (although  our  total  amount  of  incremental  revenue  would  be  less).  December  2011   ©  Datalicious  Pty  Ltd   65  
  66. 66. >  Case  Study  December  2011   ©  Datalicious  Pty  Ltd   66  
  67. 67. >  Main  Effects  December  2011   ©  Datalicious  Pty  Ltd   67  
  68. 68. >  Main  Effects   Typical  Landing  Page  Test   Element   Results   Call  to   Visitors   Conversion   Headline   Social  Proof   Conversions   Ac&on   Tested   Rate   1   H1   CTA1   SP1   1237   456   37%   2   H1   CTA1   SP2   1456   345   24%   3   H1   CTA2   SP1   1245   234   19%   4   H1   CTA2   SP2   2123   432   20%   Treatment   5   H2   CTA1   SP1   1342   234   17%   6   H2   CTA1   SP2   1102   123   11%   7   H2   CTA2   SP1   1365   700   51%   8   H2   CTA2   SP2   1243   643   52%  Treatment  #7  and  #8  were  the  clear  winners  and  It  looks  as  if  the  Headline  and  Call-­‐to-­‐Ac0on  were  much  bigger  drivers  of  posi0ve  performance  than  the  Social  Proof.  Lets  check  this!  December  2011   ©  Datalicious  Pty  Ltd   68  
  69. 69. >  Main  Effects   Typical  Landing  Page  Test   Element   Results   Call  to   Social   Visitors   Conversion   Headline   Ac&on   Proof   Tested   Rate   1   H1   CTA1   SP1   1237   37%   2   H1   CTA1   SP2   1456   24%   Avg  H1=24%   3   H1   CTA2   SP1   1245   19%   4   H1   CTA2   SP2   2123   20%   Treatment   5   H2   CTA1   SP1   1342   17%   6   H2   CTA1   SP2   1102   11%   7   H2   CTA2   SP1   1365   51%   Avg  H2=33%   8   H2   CTA2   SP2   1243   52%  The  Main  Effect  of  the  Headline  is  simply  the  (weighted)  average  conversion  rate  for  Headline  2  less  the  (weighted)  average  conversion  rate  for  Headline  1    (33%-­‐24%=9%)  December  2011   ©  Datalicious  Pty  Ltd   69  
  70. 70. >  Main  Effects   Typical  Landing  Page  Test   Main  Effect   Headline   9.4%   Element   Call  to  Ac&on   11.1%   Social  Proof   5.3%  In  actual  fact,  it  was  varia0ons  in  Call  to  Ac0on  that  had  the  most  posi0ve  impact  on  our  results,  improving  conversions  by  11.1%  points.  December  2011   ©  Datalicious  Pty  Ltd   70  
  71. 71. >  Interac&on  Effects   Typical  Landing  Page  Test   Element   Results   Call  to   Social   Visitors   Conversion   Headline   Ac&on   Proof   Tested   Rate   1   H1   CTA1   SP1   1237   37%   2   H1   CTA1   SP2   1456   24%   7   H2   CTA2   SP1   1365   51%   8   H2   CTA2   SP2   1243   52%   Treatment   3   H1   CTA2   SP1   1245   19%   4   H1   CTA2   SP2   2123   20%   5   H2   CTA1   SP1   1342   17%   6   H2   CTA1   SP2   1102   11%  An  interac0on  effect  is  present  where  the  performance  of  one  element  is  dependent  on  which  varia0on  of  the  another  variable  is  present.  In  this  example,  we  are  looking  at  whether  the  results  for  each  of  the    Headlines  is  dependent  on  which  Call-­‐to-­‐Ac0on.  December  2011   ©  Datalicious  Pty  Ltd   71  
  72. 72. >  Interac&on  Effects   Typical  Landing  Page  Test   Element   Results   Call  to   Social   Visitors   Conversion   Headline   Ac&on   Proof   Tested   Rate   1   H1   CTA1   SP1   1237   37%   2   H1   CTA1   SP2   1456   24%   Wtd  Avg  H1CTA1=30%   3   H1   CTA2   SP1   1245   19%   Wtd  Avg  H1CTA2=20%   4   H1   CTA2   SP2   2123   20%   Treatment   5   H2   CTA1   SP1   1342   17%   Wtd  Avg  H2CTA1=14%   6   H2   CTA1   SP2   1102   11%   7   H2   CTA2   SP1   1365   51%   Wtd  Avg  H2CTA2=51%   8   H2   CTA2   SP2   1243   52%  The  first  step  is  to  create  weighted  average  response  rates  between  for  each  of  the  two  factors  (ignoring  Social  Proof).    December  2011   ©  Datalicious  Pty  Ltd   72  
  73. 73. >  Interac&on  Effects   Typical  Landing  Page  Test   Call  to  Ac&on   CTA1   CTA2   Diff   60%   H1   30%   20%   -­‐10%   40%   CTA1   20%   Headline   H2   14%   51%   37%   CTA2   0%   Diff   -­‐16%   31%   H1   H2  The  next  step  is  to  calculate  the  difference  in  performance  of  one  factor  across  different  variants  of  the  other  factor.  If  the  difference  of  this  difference  is  non-­‐zero  (or  not  very  close  to  zero),  then  you  have  an  interac0on  effect.      For  example,  there  is  an  interac0on  effect  between  the  Headline  and  Call  to  Ac0on  as  the  difference  in  the  difference  in  performance  is  non-­‐zero  (31%-­‐(-­‐16%)=47%).  This  is  very  large  interac0on  when  compared  to  the  Main  Effects!  December  2011   ©  Datalicious  Pty  Ltd   73  
  74. 74. >  Interac&on  Effects   Typical  Landing  Page  Test   Sociol  Proof   SP1   SP2   Diff   40%   H1   28%   22%   -­‐6%   SP1   20%   Headline   H2   34%   33%   -­‐1%   SP2   0%   Diff   -­‐6%   11%   H1   H2   Sociol  Proof   40%   SP1   SP2   Diff   CTA1   27%   18%   -­‐9%   20%   SP1   Call  to   SP2   CTA2   36%   32%   -­‐4%   Ac&on   0%   Diff   9%   14%   CTA1   CTA2  December  2011   ©  Datalicious  Pty  Ltd   74  
  75. 75. 101011010010010010101111010010010101010100001011111001010101010100101011001100010100101001101101001101001010100111001010010010101001001010010100100101001111101010100101001001001010  >  Repor&ng  December  2011   ©  Datalicious  Pty  Ltd   75  
  76. 76. Document  Everything!  December  2011   ©  Datalicious  Pty  Ltd   76  
  77. 77. >  1.  Describe  the  test  §  Describe  the  outcome(s)  you’re  trying  to   influence  §  Describe  your  target  audience  §  Describe  the  different  treatments  including   copies  of  crea0ve  December  2011   ©  Datalicious  Pty  Ltd   77  
  78. 78. >  2.  Jus&fy  the  test  design  §  Detail  why  you’ve  chosen  the  par0cular     outcome  you’re  trying  to  influence  §  Detail  why  you’ve  chosen  the  consumers   you  are  trying  to  influence  §  Detail  why  your  interven0on  should  work   –  Past  test  results/Useability  test/Case  studies   –  Marketers  intui0on/logic  December  2011   ©  Datalicious  Pty  Ltd   78  
  79. 79. >  3.  Results  &  Conclusions  §  Detail  all  the  performance  results  §  Discuss  your  hypotheses  §  Future  tests  §  ‘Meta’  repor0ng  of  your  test  program    December  2011   ©  Datalicious  Pty  Ltd   79  
  80. 80. >  The  Scien&fic  Method   Knowledge   Establish   Develop   Facts   Test(s)   Data  December  2011   ©  Datalicious  Pty  Ltd   80  
  81. 81. >  Case  Study  December  2011   ©  Datalicious  Pty  Ltd   81  
  82. 82. >  List  of  (Some)  Resources  §  hRp://visualwebsiteop0mizer.com/case-­‐ studies.php  §  hRp://www.whichtestwon.com/  §  hRp://www.feng-­‐gui.com  §  hRp://www.smashingmagazine.com/ 2010/06/24/the-­‐ul0mate-­‐guide-­‐to-­‐a-­‐b-­‐ tes0ng  December  2011   ©  Datalicious  Pty  Ltd   82  
  83. 83. Contact  us   msavio@datalicious.com     Learn  more   blog.datalicious.com     Follow  us   twizer.com/datalicious    December  2011   ©  Datalicious  Pty  Ltd   83  
  84. 84. Data  >  Insights  >  Ac&on  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×