Conversion Optimization: The World Beyond Headlines & Button Color
 

Conversion Optimization: The World Beyond Headlines & Button Color

on

  • 283 views

Patrick McKenzie, Software Developer at Kalzumeus Software ...

Patrick McKenzie, Software Developer at Kalzumeus Software

You've run A/B tests.

You're long past the point where "Guess Which Alternative Won" articles teach you something useful.

- How do you get to the next level of conversion optimization?
- How do you get it to be a repeatable team effort?
- How do you retain organizational know-how about previous tests?
- How do you track conversions over more involved funnels?
- How do you keep the team onboard with testing for years at a time?
- How do you manage the engineering aspects of changes which are more in--depth than changing the marketing site?

This session delves into the "Here There Be Dragons" parts of the testing map. Patrick McKenzie, Software Developer at Kalzumeus Software and Bingo Card Creator, has helped a dozen companies through testing maps, and he presents some of the mistakes he's made so that you don't have to repeat them—and also a success or two.

Statistics

Views

Total Views
283
Views on SlideShare
283
Embed Views
0

Actions

Likes
1
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Conversion Optimization: The World Beyond Headlines & Button Color Conversion Optimization: The World Beyond Headlines & Button Color Presentation Transcript

    • Conversion  Op,miza,on:     The  World  Beyond     Headlines  and  Bu:on  Color  Patrick   McKenzie   Founder   Kalzumeus  So>ware   @pa,o11    
    • Low  hanging  fruit  tastes  great,   but  the  rest  is  worth  the  work,  too.  
    • Test  Idea  #1   Quotas  /  Tier  Levels  /  etc  
    • @user.card_limit  =  ab_test("user_card_limit_May_2012",  [8,  15],  :conversion  =>  "purchase")
    • What  Did  We  Learn  From  That  Test?   •  ~60%  increase  in  sales  going  from  15  -­‐>  8  cards   •  Synergis,c  effects  can  occasionally  actually  happen.   – More  on  that  later.   •  You  don’t  know  what  you  don’t  know  un,l  you  test   for  it.    This  post  on  HN  was  me  losing  $100,000:  
    • Key  Points  With  TesPng  Quotas   •  Grandfather  in  exis,ng  users.   •  Set  quota  levels  intelligently.   – Quotas  should  segment  people  by  willingness  to  pay   – Break  out  a  histogram  of  exis,ng  accounts   •  Be  ready  to  tolerate  small  up,ck  in  support  load.   •  Your  product  is  s%ll  too  cheap.  
    • Test  Idea  #2   Payment  Methods  /  Checkout  Flow  
    • Threw  This  Together  In  An  Hour  (Oops)  
    • What  We  Learned  From  This  Test   •  A  single  A/B  test  can  help  your  boZom  line  more  than   a  product  launch.   •  Dealing  with  synergis,c  effects  in  tes,ng  can   occasionally  be  maddening.   –  First  a:empts  at  implemen,ng  on-­‐site  checkout  had  null   result  a>er  null  result   –  Combina,on  of  on-­‐site  checkout  plus  redesign  plus   aforemen,oned  card  limit  test  plus  hillclimbing  won  big   •  The  data  is  never  wrong,  but  some,mes  the  data  isn’t   telling  you  what  you  think  it  is.  
    • Key  Points  With  TesPng  Checkout  Flow   •  Make  absolutely  sure  you  don’t  break  this!   •  Have  UX-­‐focused  team  (product  /  marke,ng  /  etc)   go  over  every  step  of  checkout  funnel  with  zero   tolerance  for  bad  UX.   •  Checkout  flows  go  “stale”  at  many  successful   companies.    Schedule  a  deep  dive  on  them  twice  a   year,  at  least.   •  Upsells  and  cross  selling  are  classics  for  a  reason!  
    • Test  Idea  #3   Offline  Business  Processes  
    • TesPng  degree  of  sales  which   maximizes  conversion  of  leads  
    • What  We  Learned  From  This  Test   •  Important  to  communicate  with  sales/CS.   – Allay  fears  about  being  replaced,  leads  creamed,  etc.   – Demonstrate  it  directly  makes  them  money.   •  Lead  to  resource-­‐intensive  but  hugely  successful   lead-­‐scoring  efforts.   •  Jus,fied  very  expensive  interven,ons  like  e.g.   onboarding  help  for  everybody.  
    • Key  Points  With  TesPng  Offline  Processes   •  Your  so>ware/workflow  has  to  close  the  loop  back   to  the  tes,ng  so>ware.    This  can  be  tricky.   •  Virtually  no  CRM  or  A/B  tes,ng  so>ware  accurately   models  enterprise  decisionmaking  process.    You   might  be  forced  to  op,mize  for  proxy  conversions.   •  Volumes  required  for  sta,s,cal  significance  are   really  punishing,  so  some,mes  we  “proxy  and   pray.”  
    • Test  Idea  #4   Product  On-­‐boarding  /  FRE  
    • What  Engineers  Will  Ship  If  You  Let  Them  
    • Key  Goals  of  Onboarding  Process   •  Get  user  to  a  concrete  success  as  soon  as  possible.   •  For  more  complex  sales,  lay  groundwork  for  future   success  by  e.g.  invi,ng  in  team  or  gepng   permission  to  follow-­‐up.   •  For  B2C,  especially  in  social,  you  live  and  die  on   virality  during  the  onboarding  experience.  
    • What  We  Learned  From  This  Test   •  Users  who  start  onboarding  tour  25%  to  100%+   more  likely  to  purchase  so>ware   •  Users  some,mes  have  visceral  reac,on  to   elements  of  the  onboarding  experience.   – The  wise-­‐alecky  Kiwi   – “Beau,ful  So>ware  Lady  Susan”   •  Engineers  prefer  in-­‐app  onboarding  but  it  is   expensive  as  heck  to  build  and  does  not  always   jus,fy  itself  versus  e.g.  a  Youtube  video.  
    • Key  Points  With  TesPng  Offline  Processes   •  Your  so>ware/workflow  has  to  close  the  loop  back   to  the  tes,ng  so>ware.    This  can  be  tricky.   •  Virtually  no  CRM  or  A/B  tes,ng  so>ware  accurately   models  enterprise  decisionmaking  process.    You   might  be  forced  to  op,mize  for  proxy  conversions.   •  Volumes  required  for  sta,s,cal  significance  are   really  punishing,  so  some,mes  we  “proxy  and   pray.”  
    • Test  Idea  #5   Core  Product  InteracPons  
    • Too  Complicated
    • 80%  of  Users  Use  20%  of  Features
    • Try  Radically  Simplifying
    • What  We  Learned  From  This  Test   •  Just  hiding  the  advanced  features  increased  task   success  by  10%,  which  directly  improved  sales   •  Further  jump  of  15%  when  we  did  the  redesign   – Absolutely  no  increase  in  sales…   – …  but  we  saw  in  the  funnel  that  this  was  solely  due  to   less  people  hipng  the  trial  limits  …   – …  which  we  were  able  to  profitably  exploit  with  the   quota  limit  test  described  earlier  
    • Key  Points  With  TesPng  Core  InteracPons   •  Test  the  en,re  funnel.    Some,mes  task  success   doesn’t  (unfortunately)  help  bo:om  line  metrics.   •  A  good  understanding  of  where  people  spend  their   ,me/energy  is  vital  –  test  the  90%  interac,on,  not   the  2%.  
    • ROI  on  ~$500  A  Month  TesPng  Budget 0.00%   0.50%   1.00%   1.50%   2.00%   2.50%   3.00%   3.50%   February,  2010   March,  2010   April,  2010   May,  2010   June,  2010   July,  2010   August,  2010   September,  2010   October,  2010   November,  2010   December,  2010   January,  2011   February,  2011   March,  2011   April,  2011   May,  2011   June,  2011   July,  2011   August,  2011   September,  2011   October,  2011   November,  2011   December,  2011   January,  2012   February,  2012   March,  2012   April,  2012   May,  2012   June,  2012   July,  2012   August,  2012   September,  2012   October,  2012   November,  2012   December,  2012   January,  2013   February,  2013   March,  2013   April,  2013   May,  2013   June,  2013   July,  2013   August,  2013   September,  2013   October,  2013   November,  2013   December,  2013   January,  2014   February,  2014  
    • Let’s  Talk  TesPng  Challenges  
    • A/B  TesPng  Across  System   Boundaries  
    • Not  A  Hard  Problem   Our  Marke,ng  Site A/B
    • Gets  A  Wee  Bit  Trickier  With  2  Sites   Our  Marke,ng  Site A/B The  Web  Applica,on
    • Total  Freaking  Madness   Our  Marke,ng  Site A/B The  Web  Applica,on iPhone  app Offline  processes Android  app Partner  site Content  minisite Offsite  checkout Japanese  subsidiary
    • Sanity  Restored   Consumer  Applica,ons Iden,ty  &  A/B  Manager
    • Lessons  Learned  With  This   •  Think  about  strategy  for  transi,oning  people  between   systems  –  easier  to  build  into  design  earlier  than  later   •  Cookies  with  unique  iden,,es  are  a  stopgap,  not  a   solu,on.    You  need  to  understand  the  sales  process.   •  This  is  a  good  opportunity  to  standardize  on  an  A/B   tes,ng  tool  and  workflow…  or  you’ll  hate  your  life   •  This  is  a  “high  class  problem”  which  can  be  deferred   un,l  $50M+  in  revenue,  but  when  you  solve  it,  it  is   going  to  cost  serious  ,me/money.  
    • Increasing  A/B  TesPng  AdopPon  In   Engineering  and  MarkePng  Teams  
    • Single  Most  Important  Lesson:     One  Line  of  Code   @user.card_limit  =  ab_test("user_card_limit_May_2012",  [8,  15],  :conversion  =>  "purchase")
    • Other  Lessons  Learned   •  Organiza,on  should  learn  rather  than  guess  be:er.   – Best  investment  ever:  organiza,on-­‐wide  wiki.   – Next  best:  weekly  “What  we  tried  this  week”  email.   •  Engineers  love  A/B  tes,ng  when  you  describe  it  like   a  puzzle  or  a  video  game,  with  bragging  rights.   •  Marketers  love  A/B  tes,ng  when  it’s  posi,oned  as   a  way  to  increase  their  status  in  the  organiza,on   rather  than  a  replacement  for  their  talents.  
    • Things  To  Never  Be  Without   •  CS  should  be  able  to  force-­‐override  a  user’s   par,cipa,on  in  an  A/B  test  (and,  relatedly,  “see   what  users  is  seeing”  is  great  if  you  can  get  it).   •  A/B  tests  deeper  than  surface  level  need  unit/etc   tests.    Make  sure  your  framework  supports  it.   •  Test  switcher  UI  in  development/staging  is  a  huge   win.   •  URL  query  parameter  to  flip  tests  a  huge  win.  
    • Thanks  for  Listening!   •  I  love  talking  about  this  and  you  cannot  possibly   waste  my  ,me:  @pa,o11  or   patrick@kalzumeus.com   •  Free  email  mini-­‐course  on  CRO  in  so>ware   companies:  h:p://bit.ly/patrick-­‐op,mizely   •  Liked  this  presenta,on?    I  deliver  something  like   this  twice  a  month,  free,  to  my  email  list.     h:p://training.kalzumeus.com  to  sign  up  
    • Conversion  Op,miza,on:     The  World  Beyond     Headlines  and  Bu:on  Color   Patrick   McKenzie   Founder   Kalzumeus  So>ware   @pa,o11