The document discusses Craig Sullivan's experiences and insights from his work in optimization at Belron, focusing on strategies to enhance customer experience and engagement across multiple websites and languages. It covers various testing methods, including A/B and multivariate testing, emphasizes the importance of data-driven decisions, and highlights key learnings about website optimization techniques, such as simplifying design and optimizing copy. The takeaway is to foster a culture of continuous testing and to utilize various analytics tools to improve conversion rates and customer satisfaction.

1. My year in testing...Cool stuff, hard lessons, new insightsCraig Sullivan, Belron@OptimiseOrDie

2. IntroductionI optimise stuff:

3. Websites, processes, traffic, revenue, profit, costs

4. A/B and Multi-variate testing

5. PPC, Landing page and funnel optimisation

6. User experience design, prototyping, lab testing

7. Online self-servicing and contact deflection

8. Customer Research & Insight

9. Web Analytics

10. Site search analytics

11. Browser and email compatibility

12. Mobile UX and optimisation

13. Prove it! I do UX with numbers

14. CX Manager for Belron® (Carglass®)

15. 35 websites, 19 languages, 8 figure revenue p.a.

16. Endured 18+ years of meetings with time wasters

17. I get to do stuff with crowds"If you go to the men's washrooms at the Schiphol airport in Amsterdam, you may notice there's a fly in the urinals.So what do you think most men do? That's right, they aim at the fly when they urinate. They don't even think about it, and they don't need to read a user's manual; it's just an instinctive reaction. The interesting feature of these urinals is that they're deliberately designed to take advantage of this inherent human male tendency.“This is my job.

18. What is optimisation?We’re a customer experience team, but focused on how we can influence customer behaviour in nice ways! So, our definition of Optimization:“A set of techniques, implemented in order to influence customer behaviour towards these outcomes:”Increased revenue or profitRemoving cost from the business or contact centerIncreasing productivity or labor flexibilityIncreasing NPS (Customer Sat) scores Simply delighting customersAll work is challenged on this basis!

19. Insight - Inputs

20. Insight - Inputs

21. Small team, big challengeTeam of 3 – 2 x optimisers, 1 x part time designerHigher ROI without spending more (or lower costs)Stuff that doesn’t hinder SEO or PPCMeasurable ROI on changes, not ego or opinionTools with low or zero IT footprint, easy to deployLow cost or fast return techniquesSpecialised tools for specific optimisation workOptimisation is a culture, not a tool thoughWe don’t let IT run our website!MVT and A/B testing has the fastest ROI of *any* technique

22. What’s our testing mix?A/B testing – Homepage, Landing pages, FunnelsMVT testing – high traffic landing or home pagesMini (less than 8 variables)Midi (8-32)Maxi (>32 variables)And 5 types of specialised testing:Funnel testing – bedding in new funnelsLanding page optimisationCross channel testing (e.g. Chat, Contact deflection)New Feature testingTelephony MVT

23. Where do we get ideas from?Analytics dataCustomer insight – contact, customer feedback, researchUsability testingDiary studiesClicktale recordingsBusiness owners – market data, customer mix, local knowledgeCustomer contact centre immersion (listening)Books, Websites, TipsOther optimisersTest data – variables plotted on axisMaking lots of mistakes – Fail faster

24. Killer app listAnalytics Talent Session recording : Clicktale/TealeafPerformance optimisation : Google/StrangeloopFeedback : 4Q/KampyleUsability testing : See resourcesRemote testing : See resourcesForms and search analytics : Clicktale/OtherTesting tools : BrowsercamDeviceanywhereGoogle speed test : Cool stuff!Survey tools : See resourcesMulti-variate and A/B testing : Free, Paid or Both

25. Case StudyInternational Multi-Variate testingTools & Techniques:ClicktaleGoogle AnalyticsAutonomy Optimost

26. The challengeUse MVT to increase conversion internationally 35 websites, 19 languages, several bi or tri-lingual websitesCore sites and pages are similar but there are: Cultural differences

27. Language differences

28. Different market conditions

29. Huge variations in traffic size

30. Very different local marketing approaches (traffic mix)Where to test?High traffic, High impact areasHigh ‘bounce index’ landing pagesKey conversion points or creativesFunnels / FormsRegistration, Signup processesPPC/SEO high traffic landingLow hanging fruit first pleaseAny page with a call to action!

31. What about us?Easy wins? In priority order:HomepageCore funnel (common to all websites)High traffic landing pages (hero SEO and PPC terms)Local landing pagesInfluencing pagesLower traffic landing pages

32. How do you find your test areas?Analytics talent, tools and instrumentationClicktale or Tealeaf – for session recordingIdentify key loss or influence pagesDevelop a leak model (see resources)Usability testing & UCDContact centre feedback vitalTraffic * opportunity is the keyGrade targets by impact, time, resource, difficultyFree evaluation – ask me!

33. Homepage testingRun tests in key international hubsCascade using ‘hub and spoke’ modelRun A/B best hypothesis for smallest trafficMaxi (32+ variables), Midi (8-32), Mini (8), A/B (2+)UK, DE, FR = MaxiAUS, IT, ES = MidiNZ, AT, IE = Mini or A/B

34. International Traffic VolumesHub and Spoke model – homepage traffic

35. Typical test lifecycle

37. Key learnings – on page elementsLess is more – simplicity, comprehension, clarity in designCopy is king – not content, but key persuasive copyLarge, prominent, visible, button like CTACTA above fold, especially on landing pageDo not split attention on CTA (Save, Cancel, Print, Back, Help)Use goal oriented keywords (Fix my glass!)Where is the core value proposition?Maintain a scent – PPC advert, Landing page, Funnel, Reflect the userCTA wording, headlines, copy, framing = 50%+ of upliftThink about emotional states and make use of themUse badging, testimonials, social proofing and trust symbolsWe run a betting pool – even the best people are no better than 50/50Guidelines are good but every page and site will vary

38. Key learnings – test approachStop testing subtleties (e.g. Button colour)Cultural factors are minimalMarket and traffic composition are much larger driversTechniques work well, regardless of market or languageHuge similarities in testing, but not identical resultsHub and spoke works, but only at a traffic levelFaster cycles of smaller tests = better (fail faster)A fast test that fails is better than a slow testYou must not test inside a bubble – keep challenging Use A/B to test radical approaches/disruptive change

39. SummaryDon’t overcook tests with too many variablesDon’t test things very similar (blue vs. green button?)Don’t be timid – fail faster, be bolderInternational tests show little cultural differencesMarket conditions & traffic mix are more importantBiggest drivers on page = Copy, CTA and CTA copy, Images, Badging, Nav/LayoutBiggest external factors =Page and site performance, Browser compatibilityWe are now doing up to 7M page tests per month

40. UK Wave 2 – 21/9 to 26/10

41. +12.5%

42. +8.3%

43. +13.9%

44. +5.9%

45. What is the ROI then?+11.2%

46. Another example+1-2.5%

47. People images that suck…

48. Photo GuidelinesGroups, vans, stores, corporate guff = NOSingle person – not groups or even person + customerOpen, Friendly, Natural, Smiling, Engaged = YESUniform – has huge effect, even wearing branded ball cap improves conversionLook straight at viewer or slightly towards CTAPlain backgrounds onlyNo folded arms, arms on hips, pointing, scowling, cheesy smiles, hands behind back, hanging useless by the side, clasped by groin, fiddling with handsArms with prop instead – hold a clipboard, rag, tool – I’m working for *you*Female images work best in almost every country – they slay the guysNow we get slightly better images with the guidelines!We still got a genital framing one last weekPhotoshoots are now used to drive advertising strategyVideo tests of potential ‘actors’ as a trial

49. Case Study #2New funnel implementationTools & Techniques:ClicktaleGoogle AnalyticsSpeed Trap (analytics)OptimostBrowsercamSurvey toolsMarket researchDiary studies

50. The challengeWithout operational change, improve conversionNo systems or process changesReskinning rather than repurposeMinimal investment

51. The process

52. What happened?€200k cost – user centered redesign (web/app/mobile)Mobile and App – superb resultsNew funnel – converts 0.5% betterOw! What happened?Works worse than old funnel in key areas

53. The resultV 1.1 – Early funnel is a win, mid funnel great but end funnel sucksV 1.2 – Early funnel – d’oh - introduced page fold issue / layout / copyV 1.2 – Mid funnel - Bugs, copy issues, alternative approach failsV 1.2 – Split Customer details page into 2 pages helps end funnelV 1.3 – Reverted most early/mid funnel changes, kept end funnelV 1.4 – Optimisation on copy/buttons/messages (WIP)

54. TakeawaysUsable in lab? Doesn’t mean conversion gainUser testing doesn’t uncover bugs, compatibility, forms, many persuasion and contextual issuesUse clicktale to bed new process inNo need to go back to lab or scrap the projectYou can see fixes working in real-time!Optimize and fix as you goRapid iterative changes, instead of SLEDYou must watch channel shifts – phone, email etc.

55. What’s the ROI like on this work?Reduction in call time, questions, hassleFaster booking time (< 10%)Greater customer delight – feedback. Less questions and worries12.4% from funnel work, 12.5% from MVT3.5% increase in downstream conversion rate5.5% increase in NPS (Customer sat) score 4% increase from mobile optimised siteTotal increase in conversion (against control) = 32%Actual increase in conversion = 41%NO additional marketing spend neededOn one site = > several hundred thousand € per monthYearly budget for usability, staff, tools – paid within weeks

56. Cross channel optimisationChat : we removed 17 phone calls per week for 325 x 8m chatChat increased ringback volumes - didn’t increase conversionUse dynamic phone numbering (PPC, SEO, Funnels, Tests)Measure all contact channel volumes (contact forms, other)Instrument A/B funnels with multiple phone numbersKey funnels and channel influencing tests – instrument them!Not applicable to all tests but critical to measureIf you make a mistake here, you’re doomed.Anyway – who cares if it’s not your KPI?

57. Cross channel optimisation #2Call centre optimisationEmergency phone switchoffTelephony MVTChat testingUsability testing of internal appsSimple no training overspill systemsContact deflection systems and self servicingCustomer reminders Safelite – Technician profile and ‘on their way’ notice

58. What’s nextSegmentation, Segmentation, SegmentationE.g. - screen size work on Mobile site, iPadsBetter testing tools, self administered A/BMore funnel optimisation, increased volumes of page testsMove from 8 tests -> 20+ tests per monthInternational MVT – now onto landing pages #1, homepage #3Telephony and Email MVTLanding page generation – human and rules basedHero term PPC, SEO + Local landing pages a priorityThe perfect clothes store journey

59. My adviceInvest in optimisation tools and talentTools may be low cost – expertise is worth paying forBiggest ROI is high traffic landing page optimisationFunnel is good but page is quicker (weeks vs 6 mth)Work on images, copy, call-to-action, layout, hurry up messages, scarcity calls, badging, value propositionAlways measure these tests with other channels…Chat and funnel test show cross channel is criticalOptimise low hanging fruit – then segmentsStart a gambling club at work – have fun! Try crazy stuff!More questions? Grab me after for a beer!

60. Further reading…Email : Sullivac@gmail.comTwitter : @OptimiseOrDiewww.linkedin.com/in/craigsullivan

61. Resources…

62. EFFORT-->(Size of bubble = ROI COST -->

63. CRO and Testing resources101 Landing page tips : slidesha.re/8OnBRh 544 Optimisation tips : bit.ly/8mkWOB108 Optimisation tips : bit.ly/3Z6GrP32 CRO tips : bit.ly/4BZjcW57 CRO books : bit.ly/dDjDRJCRO article list :bit.ly/nEUgui

64. Analytics – where to optimise?It depends on your site...High traffic entry pagesHigh dropout or bail pagesKey value pages (credit card)High cost pages (PPC)Conversion tipping pointsPersuasion pointsAnywhere there is traffic and CTA!Use a leak model?

65. How a leak model helpsHow many people continue?How many leave?How many channel switch?Tip! Dynamic site numbering!Where do they go?Are there influence pages?Any error clusters?What are my pain points?

66. Leak Model Detail

67. Value of leak model?Confirmation page loses 8% visitorsOne small change reduced this by 75% to 2%1 page drives 60% traffic flow to phoneQuote page has low switch – big test opportunityBugs in address lookup, branch finderForms errors contribute hugely to ‘leavers’Losses aren’t always big – they’re just switchingIn most cases, over 50% of funnel ‘losses’ are switchMust use dynamic phone numbering!

68. Session recordingVital for optimizersFills in a ‘missing link’ for insightRich source of dataSegment by attribute/source/behaviorCan be used to optimise in real time!Tealeaf www.tealeaf.comClicktale www.clicktale.comClixpy www.clixpy.com

69. Performance and AvailabilityOperate internationally? Across the USA?Customers attention span very short½ a second delay = 15% loss in conversion rateOn mobile, this is even more acuteImproves your Google quality scoreImproves your conversion rateInternational visitors?Get a consumer perspectiveMeasure third party tagsBlame someone

70. Performance resources?Read anything by Steve SoudersRead Joshua Bixby (ace!) : bit.ly/8YB3ETUse Keynote, Gomez for site monitoringEnable google speed tags – get customer KPIsConsider CDN acceleration (Akamai etc.)Think of the customer contextOptimise key traffic/conversion pages firstDo A/B splits to confirm ROIOptimise for ‘return control to user’Image, video, browser, mobile -> squeeze!Optimising batch images?Who owns customer facing performance?

71. FeedbackSimplest example – 4q – bail from siteDeeper sampling:Light touch – micro conversion (social, newsletter)Starts key process – registration/purchaseGets to tipping point – payment pageGet feedback at different funnel pointsReview the user session if capturedLike 4Q but think of an onion

72. Onsite feedbackOnsite feedback, always available, immediate, contextualFeedback tools:KampyleOpinionlabForeseePolldaddyCrowdsourcing:UservoiceIdeascaleGoogle moderatorKampyle has excellent international support – trial in Q4 2011.

73. Usability Testing and UCDLab based testingRemote testing toolsEye trackingPrototype testingLow budget testingNot what they say, watch what they doHurdles to conversion, comprehension, actionExcellent for ‘black box’ forms/funnelsFills a serious gap in your knowledge

74. Email OptimisationDelivery ratesHow the user sees itWebmail & ClientsSpam/Junk filtersMVT testingExperiment and learnReturnpathLyris

75. Remote testing, usability, feedbackUserzoom www.userzoom.comUsabilla www.usabilla.comChalkmark www.chalkmark.comUserfly www.userfly.comSilverback www.silverbackapp.comUsertesting www.usertesting.comLoop11 www.loop11.comPidoco www.pidoco.comUservue www.techsmith.comEthnio www.ethnio.comFeedback army www.feedbackarmy.comFive second test www.fivesecondtest.comMechanical turk www.mechanicalturk.comOpen Hallway www.openhallway.com

76. Forms and Search AnalyticsIgnore at your perilForms are often seen as a ‘technical’ areaWhat are the common errors?What errors precede abandonment?Validation improvementsForm design – Luke WroblewskiGood example – Postcode LookupCR0 1XA CRO lXA2.5% loss due to strict validationSearch analytics – Hurol Inan

77. Browser Compatibility“Anyone who slaps a ‘this page is best viewed with Browser X’ label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network.”Tim Berners-LeeMoney on the tableJohn Lewis testWho doorsteps you?Measure entry page stats onlyLearn about the segments – they might be valuableCreate a browser matrixUse a testing service, not simulatorsTest mobile devices tooTry the ‘Checkout browser’ test

78. Testing toolsBrowser testing www.browsercam.comMobile devices www.perfectomobile.comwww.deviceanywhere.comPerformance testing www.keynote.comwww.gomez.comNew on the block strangeloopnetworks.com End browser and device discrimination!

79. Survey toolsDust off those survey tools and ask stuff:My personal rating:Surveymonkey www.surveymonkey.comZoomerang www.zoomerang.comSurveygizmo www.surveygizmo.com