My year in testing...Cool stuff, hard lessons, new insights<br />Craig Sullivan, Belron<br />@OptimiseOrDie<br />
Introduction<br /><ul><li>I optimise stuff:
Websites, processes, traffic, revenue, profit, costs
A/B and Multi-variate testing
PPC, Landing page and funnel optimisation
User experience design, prototyping, lab testing
Online self-servicing and contact deflection
Customer Research & Insight
Web Analytics
Site search analytics
Browser and email compatibility
Mobile UX and optimisation
Prove it!  I do UX with numbers
CX Manager for Belron®  (Carglass®)
35 websites, 19 languages, 8 figure revenue p.a.
Endured 18+ years of meetings with time wasters
I get to do stuff with crowds</li></li></ul><li>"If you go to the men's washrooms at the Schiphol airport in Amsterdam, yo...
 What is optimisation?<br />We’re a customer experience team, but focused on how we can influence customer behaviour in ni...
Insight - Inputs<br />
Insight - Inputs<br />
Small team, big challenge<br />Team of 3 – 2 x optimisers, 1 x part time designer<br />Higher ROI without spending more (o...
What’s our testing mix?<br />A/B testing – Homepage, Landing pages, Funnels<br />MVT testing – high traffic landing or hom...
Where do we get ideas from?<br />Analytics data<br />Customer insight – contact, customer feedback, research<br />Usabilit...
Killer app list<br />Analytics Talent				<br />Session recording			:	Clicktale/Tealeaf<br />Performance optimisation		:	Go...
Case StudyInternational Multi-Variate testing<br />Tools & Techniques:<br />ClicktaleGoogle AnalyticsAutonomy Optimost<br />
The challenge<br />Use MVT  to increase conversion internationally <br />35 websites, 19 languages, several bi or tri-ling...
 Language differences
 Different market conditions
 Huge variations in traffic size
Very different local marketing approaches (traffic mix)</li></li></ul><li>Where to test?<br />High traffic, High impact ar...
What about us?<br />Easy wins?  In priority order:<br />Homepage<br />Core funnel (common to all websites)<br />High traff...
How do you find your test areas?<br />Analytics talent, tools and instrumentation<br />Clicktale or Tealeaf – for session ...
Homepage testing<br />Run tests in key international hubs<br />Cascade using ‘hub and spoke’ model<br />Run A/B best hypot...
International Traffic Volumes<br />Hub and Spoke model – homepage traffic<br />
Typical test lifecycle<br />
Key learnings – on page elements<br />Less is more – simplicity, comprehension, clarity in design<br />Copy is king – not ...
Key learnings – test approach<br />Stop testing subtleties (e.g. Button colour)<br />Cultural factors are minimal<br />Mar...
Summary<br />Don’t overcook tests with too many variables<br />Don’t test things very similar (blue vs. green button?)<br ...
UK Wave 2 – 21/9 to 26/10 <br />
+12.5%<br />
+8.3%<br />
+13.9%<br />
+5.9%<br />
What is the ROI then?<br />+11.2%<br />
Another example<br />+1-2.5%<br />
People images that suck…<br />
Photo Guidelines<br />Groups, vans, stores, corporate guff = NO<br />Single person – not groups or even person + customer<...
Case Study #2New funnel implementation<br />Tools & Techniques:<br />ClicktaleGoogle AnalyticsSpeed Trap (analytics)Optimo...
The challenge<br />Without operational change, improve conversion<br />No systems or process changes<br />Reskinning rathe...
The process<br />
What happened?<br />€200k cost – user centered redesign (web/app/mobile)<br />Mobile and App – superb results<br />New fun...
The result<br />V 1.1 – Early funnel is a win, mid funnel great but end funnel sucks<br />V 1.2 – Early funnel – d’oh - in...
Takeaways<br />Usable in lab?  Doesn’t mean conversion gain<br />User testing doesn’t uncover bugs, compatibility, forms, ...
What’s the ROI like on this work?<br />Reduction in call time, questions, hassle<br />Faster booking time (< 10%)<br />Gre...
Cross channel optimisation<br />Chat : we removed 17 phone calls per week for 325 x 8m chat<br />Chat increased ringback v...
Cross channel optimisation #2<br />Call centre optimisation<br />Emergency phone switchoff<br />Telephony MVT<br />Chat te...
What’s next<br />Segmentation, Segmentation, Segmentation<br />E.g. - screen size work on Mobile site, iPads<br />Better t...
My advice<br />Invest in optimisation tools and talent<br />Tools may be low cost – expertise is worth paying for<br />Big...
Further reading…<br />Email	:   Sullivac@gmail.com<br />Twitter	:   @OptimiseOrDie<br />www.linkedin.com/in/craigsullivan<...
Resources…<br />
EFFORT--><br />(Size of bubble = ROI                                       COST --><br />
Upcoming SlideShare
Loading in …5
×

Conversion Summit Slides - 1 Sep 2011 - Craig Sullivan

9,237 views

Published on

Hi everyone. Great to see you all and totally maxed out on all the conversion advice. Thank you and in return, here are the slides with extra resources added.

Published in: Technology, Business

Conversion Summit Slides - 1 Sep 2011 - Craig Sullivan

  1. 1. My year in testing...Cool stuff, hard lessons, new insights<br />Craig Sullivan, Belron<br />@OptimiseOrDie<br />
  2. 2. Introduction<br /><ul><li>I optimise stuff:
  3. 3. Websites, processes, traffic, revenue, profit, costs
  4. 4. A/B and Multi-variate testing
  5. 5. PPC, Landing page and funnel optimisation
  6. 6. User experience design, prototyping, lab testing
  7. 7. Online self-servicing and contact deflection
  8. 8. Customer Research & Insight
  9. 9. Web Analytics
  10. 10. Site search analytics
  11. 11. Browser and email compatibility
  12. 12. Mobile UX and optimisation
  13. 13. Prove it! I do UX with numbers
  14. 14. CX Manager for Belron® (Carglass®)
  15. 15. 35 websites, 19 languages, 8 figure revenue p.a.
  16. 16. Endured 18+ years of meetings with time wasters
  17. 17. I get to do stuff with crowds</li></li></ul><li>"If you go to the men's washrooms at the Schiphol airport in Amsterdam, you may notice there's a fly in the urinals.<br />So what do you think most men do? That's right, they aim at the fly when they urinate. <br />They don't even think about it, and they don't need to read a user's manual; it's just an instinctive reaction. <br />The interesting feature of these urinals is that they're deliberately designed to take advantage of this inherent human male tendency.“<br />This is my job.<br />
  18. 18. What is optimisation?<br />We’re a customer experience team, but focused on how we can influence customer behaviour in nice ways! <br />So, our definition of Optimization:<br />“A set of techniques, implemented in order to influence customer behaviour towards these outcomes:”<br />Increased revenue or profit<br />Removing cost from the business or contact center<br />Increasing productivity or labor flexibility<br />Increasing NPS (Customer Sat) scores <br />Simply delighting customers<br />All work is challenged on this basis!<br />
  19. 19. Insight - Inputs<br />
  20. 20. Insight - Inputs<br />
  21. 21. Small team, big challenge<br />Team of 3 – 2 x optimisers, 1 x part time designer<br />Higher ROI without spending more (or lower costs)<br />Stuff that doesn’t hinder SEO or PPC<br />Measurable ROI on changes, not ego or opinion<br />Tools with low or zero IT footprint, easy to deploy<br />Low cost or fast return techniques<br />Specialised tools for specific optimisation work<br />Optimisation is a culture, not a tool though<br />We don’t let IT run our website!<br />MVT and A/B testing has the fastest ROI of *any* technique<br />
  22. 22. What’s our testing mix?<br />A/B testing – Homepage, Landing pages, Funnels<br />MVT testing – high traffic landing or home pages<br />Mini (less than 8 variables)<br />Midi (8-32)<br />Maxi (>32 variables)<br />And 5 types of specialised testing:<br />Funnel testing – bedding in new funnels<br />Landing page optimisation<br />Cross channel testing (e.g. Chat, Contact deflection)<br />New Feature testing<br />Telephony MVT<br />
  23. 23. Where do we get ideas from?<br />Analytics data<br />Customer insight – contact, customer feedback, research<br />Usability testing<br />Diary studies<br />Clicktale recordings<br />Business owners – market data, customer mix, local knowledge<br />Customer contact centre immersion (listening)<br />Books, Websites, Tips<br />Other optimisers<br />Test data – variables plotted on axis<br />Making lots of mistakes – Fail faster<br />
  24. 24. Killer app list<br />Analytics Talent <br />Session recording : Clicktale/Tealeaf<br />Performance optimisation : Google/Strangeloop<br />Feedback : 4Q/Kampyle<br />Usability testing : See resources<br />Remote testing : See resources<br />Forms and search analytics : Clicktale/Other<br />Testing tools : Browsercam<br />Deviceanywhere<br />Google speed test : Cool stuff!<br />Survey tools : See resources<br />Multi-variate and A/B testing : Free, Paid or Both<br />
  25. 25. Case StudyInternational Multi-Variate testing<br />Tools & Techniques:<br />ClicktaleGoogle AnalyticsAutonomy Optimost<br />
  26. 26. The challenge<br />Use MVT to increase conversion internationally <br />35 websites, 19 languages, several bi or tri-lingual websites<br />Core sites and pages are similar but there are:<br /><ul><li> Cultural differences
  27. 27. Language differences
  28. 28. Different market conditions
  29. 29. Huge variations in traffic size
  30. 30. Very different local marketing approaches (traffic mix)</li></li></ul><li>Where to test?<br />High traffic, High impact areas<br />High ‘bounce index’ landing pages<br />Key conversion points or creatives<br />Funnels / Forms<br />Registration, Signup processes<br />PPC/SEO high traffic landing<br />Low hanging fruit first please<br />Any page with a call to action! <br />
  31. 31. What about us?<br />Easy wins? In priority order:<br />Homepage<br />Core funnel (common to all websites)<br />High traffic landing pages (hero SEO and PPC terms)<br />Local landing pages<br />Influencing pages<br />Lower traffic landing pages<br />
  32. 32. How do you find your test areas?<br />Analytics talent, tools and instrumentation<br />Clicktale or Tealeaf – for session recording<br />Identify key loss or influence pages<br />Develop a leak model (see resources)<br />Usability testing & UCD<br />Contact centre feedback vital<br />Traffic * opportunity is the key<br />Grade targets by impact, time, resource, difficulty<br />Free evaluation – ask me!<br />
  33. 33. Homepage testing<br />Run tests in key international hubs<br />Cascade using ‘hub and spoke’ model<br />Run A/B best hypothesis for smallest traffic<br />Maxi (32+ variables), Midi (8-32), Mini (8), A/B (2+)<br />UK, DE, FR = Maxi<br />AUS, IT, ES = Midi<br />NZ, AT, IE = Mini or A/B<br />
  34. 34. International Traffic Volumes<br />Hub and Spoke model – homepage traffic<br />
  35. 35. Typical test lifecycle<br />
  36. 36.
  37. 37. Key learnings – on page elements<br />Less is more – simplicity, comprehension, clarity in design<br />Copy is king – not content, but key persuasive copy<br />Large, prominent, visible, button like CTA<br />CTA above fold, especially on landing page<br />Do not split attention on CTA (Save, Cancel, Print, Back, Help)<br />Use goal oriented keywords (Fix my glass!)<br />Where is the core value proposition?<br />Maintain a scent – PPC advert, Landing page, Funnel, Reflect the user<br />CTA wording, headlines, copy, framing = 50%+ of uplift<br />Think about emotional states and make use of them<br />Use badging, testimonials, social proofing and trust symbols<br />We run a betting pool – even the best people are no better than 50/50<br />Guidelines are good but every page and site will vary<br />
  38. 38. Key learnings – test approach<br />Stop testing subtleties (e.g. Button colour)<br />Cultural factors are minimal<br />Market and traffic composition are much larger drivers<br />Techniques work well, regardless of market or language<br />Huge similarities in testing, but not identical results<br />Hub and spoke works, but only at a traffic level<br />Faster cycles of smaller tests = better (fail faster)<br />A fast test that fails is better than a slow test<br />You must not test inside a bubble – keep challenging<br /> Use A/B to test radical approaches/disruptive change<br />
  39. 39. Summary<br />Don’t overcook tests with too many variables<br />Don’t test things very similar (blue vs. green button?)<br />Don’t be timid – fail faster, be bolder<br />International tests show little cultural differences<br />Market conditions & traffic mix are more important<br />Biggest drivers on page = <br />Copy, CTA and CTA copy, Images, Badging, Nav/Layout<br />Biggest external factors =<br />Page and site performance, Browser compatibility<br />We are now doing up to 7M page tests per month<br />
  40. 40. UK Wave 2 – 21/9 to 26/10 <br />
  41. 41. +12.5%<br />
  42. 42. +8.3%<br />
  43. 43. +13.9%<br />
  44. 44. +5.9%<br />
  45. 45. What is the ROI then?<br />+11.2%<br />
  46. 46. Another example<br />+1-2.5%<br />
  47. 47. People images that suck…<br />
  48. 48. Photo Guidelines<br />Groups, vans, stores, corporate guff = NO<br />Single person – not groups or even person + customer<br />Open, Friendly, Natural, Smiling, Engaged = YES<br />Uniform – has huge effect, even wearing branded ball cap improves conversion<br />Look straight at viewer or slightly towards CTA<br />Plain backgrounds only<br />No folded arms, arms on hips, pointing, scowling, cheesy smiles, hands behind back, hanging useless by the side, clasped by groin, fiddling with hands<br />Arms with prop instead – hold a clipboard, rag, tool – I’m working for *you*<br />Female images work best in almost every country – they slay the guys<br />Now we get slightly better images with the guidelines!<br />We still got a genital framing one last week<br />Photoshoots are now used to drive advertising strategy<br />Video tests of potential ‘actors’ as a trial<br />
  49. 49. Case Study #2New funnel implementation<br />Tools & Techniques:<br />ClicktaleGoogle AnalyticsSpeed Trap (analytics)OptimostBrowsercamSurvey toolsMarket researchDiary studies<br />
  50. 50. The challenge<br />Without operational change, improve conversion<br />No systems or process changes<br />Reskinning rather than repurpose<br />Minimal investment<br />
  51. 51. The process<br />
  52. 52. What happened?<br />€200k cost – user centered redesign (web/app/mobile)<br />Mobile and App – superb results<br />New funnel – converts 0.5% better<br />Ow! What happened?<br />Works worse than old funnel in key areas<br />
  53. 53. The result<br />V 1.1 – Early funnel is a win, mid funnel great but end funnel sucks<br />V 1.2 – Early funnel – d’oh - introduced page fold issue / layout / copy<br />V 1.2 – Mid funnel - Bugs, copy issues, alternative approach fails<br />V 1.2 – Split Customer details page into 2 pages helps end funnel<br />V 1.3 – Reverted most early/mid funnel changes, kept end funnel<br />V 1.4 – Optimisation on copy/buttons/messages (WIP)<br />
  54. 54. Takeaways<br />Usable in lab? Doesn’t mean conversion gain<br />User testing doesn’t uncover bugs, compatibility, forms, many persuasion and contextual issues<br />Use clicktale to bed new process in<br />No need to go back to lab or scrap the project<br />You can see fixes working in real-time!<br />Optimize and fix as you go<br />Rapid iterative changes, instead of SLED<br />You must watch channel shifts – phone, email etc.<br />
  55. 55. What’s the ROI like on this work?<br />Reduction in call time, questions, hassle<br />Faster booking time (< 10%)<br />Greater customer delight – feedback. Less questions and worries<br />12.4% from funnel work, 12.5% from MVT<br />3.5% increase in downstream conversion rate<br />5.5% increase in NPS (Customer sat) score <br />4% increase from mobile optimised site<br />Total increase in conversion (against control) = 32%<br />Actual increase in conversion = 41%<br />NO additional marketing spend needed<br />On one site = > several hundred thousand € per month<br />Yearly budget for usability, staff, tools – paid within weeks<br />
  56. 56. Cross channel optimisation<br />Chat : we removed 17 phone calls per week for 325 x 8m chat<br />Chat increased ringback volumes - didn’t increase conversion<br />Use dynamic phone numbering (PPC, SEO, Funnels, Tests)<br />Measure all contact channel volumes (contact forms, other)<br />Instrument A/B funnels with multiple phone numbers<br />Key funnels and channel influencing tests – instrument them!<br />Not applicable to all tests but critical to measure<br />If you make a mistake here, you’re doomed.<br />Anyway – who cares if it’s not your KPI?<br />
  57. 57. Cross channel optimisation #2<br />Call centre optimisation<br />Emergency phone switchoff<br />Telephony MVT<br />Chat testing<br />Usability testing of internal apps<br />Simple no training overspill systems<br />Contact deflection systems and self servicing<br />Customer reminders <br />Safelite – Technician profile and ‘on their way’ notice<br />
  58. 58. What’s next<br />Segmentation, Segmentation, Segmentation<br />E.g. - screen size work on Mobile site, iPads<br />Better testing tools, self administered A/B<br />More funnel optimisation, increased volumes of page tests<br />Move from 8 tests -> 20+ tests per month<br />International MVT – now onto landing pages #1, homepage #3<br />Telephony and Email MVT<br />Landing page generation – human and rules based<br />Hero term PPC, SEO + Local landing pages a priority<br />The perfect clothes store journey<br />
  59. 59. My advice<br />Invest in optimisation tools and talent<br />Tools may be low cost – expertise is worth paying for<br />Biggest ROI is high traffic landing page optimisation<br />Funnel is good but page is quicker (weeks vs 6 mth)<br />Work on images, copy, call-to-action, layout, hurry up messages, scarcity calls, badging, value proposition<br />Always measure these tests with other channels…<br />Chat and funnel test show cross channel is critical<br />Optimise low hanging fruit – then segments<br />Start a gambling club at work – have fun! Try crazy stuff!<br />More questions? Grab me after for a beer!<br />
  60. 60. Further reading…<br />Email : Sullivac@gmail.com<br />Twitter : @OptimiseOrDie<br />www.linkedin.com/in/craigsullivan<br />
  61. 61. Resources…<br />
  62. 62. EFFORT--><br />(Size of bubble = ROI COST --><br />
  63. 63. CRO and Testing resources<br />101 Landing page tips : slidesha.re/8OnBRh <br />544 Optimisation tips : bit.ly/8mkWOB<br />108 Optimisation tips : bit.ly/3Z6GrP<br />32 CRO tips : bit.ly/4BZjcW<br />57 CRO books : bit.ly/dDjDRJ<br />CRO article list :bit.ly/nEUgui<br />
  64. 64. Analytics – where to optimise?<br />It depends on your site...<br />High traffic entry pages<br />High dropout or bail pages<br />Key value pages (credit card)<br />High cost pages (PPC)<br />Conversion tipping points<br />Persuasion points<br />Anywhere there is traffic and CTA!<br />Use a leak model?<br />
  65. 65. How a leak model helps<br />How many people continue?<br />How many leave?<br />How many channel switch?<br />Tip! Dynamic site numbering!<br />Where do they go?<br />Are there influence pages?<br />Any error clusters?<br />What are my pain points?<br />
  66. 66. Leak Model Detail<br />
  67. 67. Value of leak model?<br />Confirmation page loses 8% visitors<br />One small change reduced this by 75% to 2%<br />1 page drives 60% traffic flow to phone<br />Quote page has low switch – big test opportunity<br />Bugs in address lookup, branch finder<br />Forms errors contribute hugely to ‘leavers’<br />Losses aren’t always big – they’re just switching<br />In most cases, over 50% of funnel ‘losses’ are switch<br />Must use dynamic phone numbering!<br />
  68. 68. Session recording<br />Vital for optimizers<br />Fills in a ‘missing link’ for insight<br />Rich source of data<br />Segment by attribute/source/behavior<br />Can be used to optimise in real time!<br />Tealeaf www.tealeaf.com<br />Clicktale www.clicktale.com<br />Clixpy www.clixpy.com<br />
  69. 69. Performance and Availability<br />Operate internationally? Across the USA?<br />Customers attention span very short<br />½ a second delay = <br /> 15% loss in conversion rate<br />On mobile, this is even more acute<br />Improves your Google quality score<br />Improves your conversion rate<br />International visitors?<br />Get a consumer perspective<br />Measure third party tags<br />Blame someone<br />
  70. 70. Performance resources?<br />Read anything by Steve Souders<br />Read Joshua Bixby (ace!) : bit.ly/8YB3ET<br />Use Keynote, Gomez for site monitoring<br />Enable google speed tags – get customer KPIs<br />Consider CDN acceleration (Akamai etc.)<br />Think of the customer context<br />Optimise key traffic/conversion pages first<br />Do A/B splits to confirm ROI<br />Optimise for ‘return control to user’<br />Image, video, browser, mobile -> squeeze!<br />Optimising batch images?<br />Who owns customer facing performance?<br />
  71. 71. Feedback<br />Simplest example – 4q – bail from site<br />Deeper sampling:<br />Light touch – micro conversion (social, newsletter)<br />Starts key process – registration/purchase<br />Gets to tipping point – payment page<br />Get feedback at different funnel points<br />Review the user session if captured<br />Like 4Q but think of an onion<br />
  72. 72. Onsite feedback<br />Onsite feedback, always available, immediate, contextual<br />Feedback tools:<br />Kampyle<br />Opinionlab<br />Foresee<br />Polldaddy<br />Crowdsourcing:<br />Uservoice<br />Ideascale<br />Google moderator<br />Kampyle has excellent international support – trial in Q4 2011.<br />
  73. 73. Usability Testing and UCD<br />Lab based testing<br />Remote testing tools<br />Eye tracking<br />Prototype testing<br />Low budget testing<br />Not what they say, watch what they do<br />Hurdles to conversion, comprehension, action<br />Excellent for ‘black box’ forms/funnels<br />Fills a serious gap in your knowledge<br />
  74. 74. Email Optimisation<br />Delivery rates<br />How the user sees it<br />Webmail & Clients<br />Spam/Junk filters<br />MVT testing<br />Experiment and learn<br />Returnpath<br />Lyris<br />
  75. 75. Remote testing, usability, feedback<br />Userzoom www.userzoom.com<br />Usabilla www.usabilla.com<br />Chalkmark www.chalkmark.com<br />Userfly www.userfly.com<br />Silverback www.silverbackapp.com<br />Usertesting www.usertesting.com<br />Loop11 www.loop11.com<br />Pidoco www.pidoco.com<br />Uservue www.techsmith.com<br />Ethnio www.ethnio.com<br />Feedback army www.feedbackarmy.com<br />Five second test www.fivesecondtest.com<br />Mechanical turk www.mechanicalturk.com<br />Open Hallway www.openhallway.com<br />
  76. 76. Forms and Search Analytics<br />Ignore at your peril<br />Forms are often seen as a ‘technical’ area<br />What are the common errors?<br />What errors precede abandonment?<br />Validation improvements<br />Form design – Luke Wroblewski<br />Good example – Postcode Lookup<br />CR0 1XA CRO lXA<br />2.5% loss due to strict validation<br />Search analytics – Hurol Inan<br />
  77. 77. Browser Compatibility<br />“Anyone who slaps a ‘this page is best viewed with Browser X’ label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network.”<br />Tim Berners-Lee<br />Money on the table<br />John Lewis test<br />Who doorsteps you?<br />Measure entry page stats only<br />Learn about the segments – they might be valuable<br />Create a browser matrix<br />Use a testing service, not simulators<br />Test mobile devices too<br />Try the ‘Checkout browser’ test<br />
  78. 78. Testing tools<br />Browser testing www.browsercam.com<br />Mobile devices www.perfectomobile.com<br />www.deviceanywhere.com<br />Performance testing www.keynote.com<br />www.gomez.com<br />New on the block strangeloopnetworks.com<br /> End browser and device discrimination!<br />
  79. 79. Survey tools<br />Dust off those survey tools and ask stuff:<br />My personal rating:<br />Surveymonkey www.surveymonkey.com<br />Zoomerang www.zoomerang.com<br />Surveygizmo www.surveygizmo.com<br />

×