1. Craig SullivanThe Multi-variate advantage Group CX Manager How testing saved my life Belron® @OptimiseOrDie
2. Naked self promotion @OptimiseOrDie• Conversion Rate Optimisation • Contact deflection and• A/B and Multi-variate testing online self service• Cross channel optimisation • Site search analytics• User Centered Design • Site, Page and Campaign optimisation• Usability testing • Test design and execution• Customer Research & Insight • A long usable site portfolio• Web Analytics • ROI on UX improvements• Browser and email compatibility • Over 18 years of boring meetings Group CX Manager for Belron® (Safelite) I get to do it with crowds
3. Outline• What is optimisation?• Belron® Testing Culture• What is MVT• Our testing program• Advantages of testing• The results• Cross channel optimisation• Case study : Chat ROI• Resources
4. What’s optimisation? @OptimiseOrDie
5. A set of techniques, implemented inorder to influence customerbehaviour towards these outcomes:• Increased revenue or profit• Increasing NPS (Customer Sat) scores• Lower cost for business or contact centres• Increasing productivity or labour flexibility• Simply delighting customersWe may fail with 90% of what we try.The 10% of winners let us shiftbehaviour hugely and measurably.
6. ®Belron Testing Culture @OptimiseOrDie
7. If you go to the mens washrooms at theSchiphol airport in Amsterdam, you maynotice theres a fly in the urinals. It’sscreen printed .So what do you think most men do? Thatsright, they aim at the fly when theyurinate.They dont even think about it, and theydont need to read a users manual; itsjust an instinctive reaction that means85% less spillage!The interesting feature of these urinals isthat theyre deliberately designed to takeadvantage of this inherent human maletendency.This is my job.
8. Small team, big challenge• Team of 3 - 2 Optimisers, 1 x part time designer• Higher ROI or lower costs, without spending more• Stuff that doesn’t hinder SEO, PPC, Performance• Measurable ROI - not ego or opinion• Fast return and low cost techniques• Over 26M page tests, 17 international tests live now• ROI is > 50 x optimisation costs• Total cost = ~800k for two years. ROI > $40M USD
9. Primer on testing =
10. A/B test Control Version A Version B
11. Multi variate Pizza Ingredients Experiment Recipes =
13. So what about ingredients? +20% -2% +5% +3% +2% +6% +6% +10% -18%
14. Weed out some losers… +2% +3% +9% +4% +5% +13%
15. The final… +10% +13%
16. Our testing program @OptimiseOrDie
17. Insight - Inputs Forms Remote Search analytics Depth testing analytics interviews Eye tracking Market research Customer A/B and contact MVT testing Socialanalytics Insight Unstructured data Web Clicktale analytics Usability Competitor testing Ops and CS Customer evals teams Surveys services
18. Killer idea generation sources• Analytics data• Customer insight – contact, customer feedback, research• Usability testing• Diary studies• Clicktale recordings• Business owners – market data, customer mix, local knowledge• Customer contact centre immersion (listening)• Books, Websites, Tips• Other optimisers• Test data – variables plotted on axis• Making lots of mistakes – Fail faster
19. Typical test lifecycle Other test Gather and results – good Brainstorm & Instrument review inputs variables & Hypotheses learnings Signoff (Legal, Design variables Instrument Test Plan Marketing, & creative analytics Document Brand) elements Next WaveWave design Results of wave Washup (repeat) Business led Do verificationRepeat from reporting (with = wave + 90/10% Declare winner start £x, without = £y) split
21. What’s our testing mix?• A/B testing – Homepage, Landing pages, Funnels• MVT testing – high traffic landing or home pages – Mini (less than 8 variables) – Midi (8-32) – Maxi (>32 variables)And 5 types of specialised testing:• Funnel testing – bedding in new funnels• Landing page optimisation• Cross channel testing• Iterative feature and product testing• Micro testing• Telephony MVT
22. Level 1 Level 2 Level 3 Level 4 Level 5 Starter Level Early maturity Serious testing Core business value You rock, awesomely Dedicated team Ninja Team Local Heroes Small team Cross silo team Culture Volume Testing in the Chaotic Good Low hanging fruit Systematic tests opportunities DNA_____________________________________________________________________________________________ _ Process Ad Hoc Outline process Well developed Streamlined Company wide _______________________________________________________________________________________________ +Funnel +Cross channel +Spread tool use Guessing + Multi variate optimisation testing Dynamic adaptive Testing A/B testing Session replay Call tracking Integrated CRO targeting focus Basic tools No segments Some segments and analytics Machine learning Micro testing Segmentation Realtime________________________________________________________________________________________________ Bounce rates + Funnel analysis + Funnel fixes + offline Multichannel Analytics Low converting integration funnels Big volume Forms analytics focus landing pages & High loss pages Channel switches Single channel Cross channel picture synergy________________________________________________________________________________________________ +Regular usability + All channel view Analytics + Customer sat testing/research +User Centered of customer Insight Surveys Prototyping Design scores tied to UX Driving offline Contact Centre Rapid iterative methods Low budget Session replay Layered feedback testing and using online usability Mini product tests design All promotion Onsite feedback driven by testing________________________________________________________________________________________________ Continual Mission Get buyin Prove ROI Scale the testing Mine value improvement
23. International testing examples @OptimiseOrDie
24. UK Wave 2 – 21/9 to 26/10
25. What is the ROI then?
26. Another example
27. Testing advantages @OptimiseOrDie
28. Your ‘design’ team? Web/Online Pesky Meddling Kids? Marketing Who designs this stuff anyway? Brand IT THE CEO CUSTOMER Legal
29. BENEFITS OF TESTING:• Reduced ego and opinion (execs + others)• Measurable ROI• Faster time to market• Increased conversion in the pocket early• Iterative and rapid product lifecycles• Faster, smaller, focused, measured releases• Continual testing = continual improvement• SLED vs. Agile = faster return, better focus• Less reliance on anecdotal evidence• Business owners part of testing culture• Introduces a testing culture• Large cross silo team now gets involved• Reduces timidity – be brave and curious!• Helps us hit the sweet spot...
30. Cross Channel Optimisation @OptimiseOrDie
31. Insight - Inputs Ringback Phone Email Retargeting Chat Mobile TV handsets Display AppsAffiliate Exploding Tablets Channels Desktops Direct Social & SEO Micro Partners PPC DM
32. TV advertising and WebTV - Off TV - On
33. People images that suck…
34. Photo Guidelines• Groups, vans, stores, corporate guff = • Female images work best in almost NO every country – they slay the guys• Single person – not groups or even person + customer • Natural, authentic and not ‘model’• Open, Friendly, Natural, Smiling, or ‘stock’ image types work best for Engaged = YES people• Uniform – has huge effect, even • We get better images now, but we wearing branded ball cap improves keep working on the guidelines! conversion • We still got a genital framing• Look straight at viewer or slightly picture the other week towards CTA • Photoshoots and tests are now• Plain backgrounds only being used to drive advertising• No folded arms, arms on hips, pointing, strategy scowling, cheesy smiles, hands behind back, hanging useless by the side, • Pre TV test of potential ‘actors’ on clasped by groin, fiddling with hands website. X factor meets split• Arms with prop instead – hold a testing. clipboard, rag, tool – I’m working for *you*
35. Cross channel mini studies• Mini studies: – Abandon testing – Emergency phone switchoff & Overspill systems – Contact deflection systems and self servicing – Scripting Experiments – APIs and other systems integration – Chat testing – Phone tracking
36. Abandon testingReduce unwanted phone hangups. Test plan – Remove corporate guffspeak – Two countries, big phone systems, – Don’t mention the website, esp. – Test variants of messages, music, if this is the web number (D’oh!) timings, amounts quoted – Don’t give a generic or corporate – Messages make a big impact – drone apology message experiment freely with volumes – Quote real KPIs and timings (ING – Opportunity is large due to high direct, average queue time) call volumes we have – Make routing emotionally driven – What people hate personally vs. – not confusing categories that what those same people have on map poorly onto customer needs their company lines, lol – Experiment with queue timing – – On a mission here! find sweet spot, don’t overquote
37. Switchoff OverspillIf you have spikes, have a throttle: Overspill systems: – Busiest TV & Radio campaign so far, Q2 – Get a UX designer to make you a fast 2011 for Autoglass® entry system – Designed a phone ‘switchoff’ system, – Make them simple, easy to learn and permanently on tap leverage your web platform – Routed majority of contact to a web form – Give API access to overspill – Prevented a backlog situation from companies developing – Sometimes heavy IT integration isn’t – Bought time and still exceeded needed customer expectations (<4h excellent) – “Nobody needs a freaking training – Used twice to dampen inbound call course to use facebook” - Make sure volumes your internal apps and call centre overspill systems are built for instant learnability – This could save millions in process efficiency. Same as web, really.
38. Contact deflection & self service• People like immediate gratification! • What about your measurement?• The iceberg of contact opportunity • Basics – Ratings, Did you find this• Make people like you for not talking answer helpful? to them on the phone –double win • Intermediate – Tracking behaviour• If you stick it in help, will it go away? and contact channels• No elephant graveyard, please (FAQ) • Add tracking of self service events or• Instrument for analytics and ROI behaviour• Set it up to measure and improve • Set up a game or reward structure for staff• Gamification of content? Adoption? • Continually improve and get proof• LOVEFiLM Help Centre: • Add phone tracking - make avoidable – Ratings, Deflection tracking, self contact visible service tracking • Remove all barriers to self servicing – V1 :Please view in your account for high cost/volume contacts secion, V2 : Please visit your account, V3 : Please check your • Try Amazon help – where is phone? next billing date, V4 - Final • Help experts? Sticky Content are version – to show the real info : brilliant (www.stickycontent.co.uk) “Your next billing date is 1 Feb”
39. Script Optimisation• How can we optimise phone conversion? – Dedicated test teams – Script testing often limited range/shift in companies now – Now – we can use A/B or MV testing, with API integration – Try out 3,4,5 different versions of key areas or entire script – ‘Wrapping’ sends XML message event, triggers analytics tag – Analyse in the same way as an A/B or multi-variate web test – Fast optimisation of handling scripts, using a mini lab – Should yield 3-5% higher performing scripts, minimum – Shows what variables drive script performance globally – These emotional and psychological triggers can be mined – Aim for Q2 2012 for a proof of concept
40. Platform APIs• Use of web platform APIs now for: – Contact centre messages – Facebook app – Mobile site and apps – Scheduling – Analytics tag firing• Just making APIs, automagically makes: – People use them – Improvements happen – New ideas happen, that you didn’t expect – Incompatible stuff get glued together – Complex IT integration projects go away
41. Chat Testing Ringback Contact Form Contact UsYou must measure behaviouracross contact channelsduring tests like these A - Chat B - Chat C - Chat Split A B C
43. Chat testing• 1 month trial – properly A/B tested • Customers happy, yes, but not in a• Measured all the contact channels for good business way movements – what happened? • Like putting your customer service• Over a week, chat saves 17 calls – desk into the street yay! • You must measure influences in• But swapped that for 329 chats @ contact volumes when trialling 8.25 minutes. Ouch. Chat• Increased ringback requests? Email form about the same. • Be very careful – prove your ROI on any pilot.• No statistically significant impact on conversion • Only 7-10% use and out of those,• Assumption was big impact, huge less than 50% are influencable etc. impact! etc.• Anecdotally, a wonderful tool. • The productivity Myth of chat Agreed. • Your situation may differ – please TEST!
44. Call trackingHow it works:• Rent a bank of numbers• Add a dynamic ‘tag’ to your page• Phone number appears automatically• Customer calls this number• Captures the keywords, landing page, converting page, traffic source• Geo mapping captures customer location• When the call is hung up, optionally record the outcome code (sale, loss)• Have a special number for Facebook, SEO, Adwords traffic etc.• Lets you analyse from web -> phone call -> wait time -> wrapping• Builds a link between web and call, finally
45. Call trackingWhy you should use it!• Most people under-report on web driven phone contact or sales• You can’t work out your advertising effectiveness, if you don’t include phone• If you don’t measure phone, your A/B tests may be wrong• You can capture all your web AND phone, in one easy interface• Allows you a multi-channel view of all your contact sources• Lets you record offline campaigns as well with specific numbers (print, TV etc.)• You can experiment with routing, messages, on hold music – much more easily• If you can’t afford lots of phone numbers, this is an easy way to get tracking
46. Company Website CoverageWho?Mongoose Metrics* www.mongoosemetrics.com UK, USA, Canada Ifbyphone* www.ifbyphone.com USA TheCallR* www.thecallr.com USA, Canada, UK, IT, FR, BE, ES, NL Call tracking metrics www.calltrackingmetrics.com USA Hosted Numbers www.hostednumbers.com USA Callcap www.callcap.com USA UK, SE, FI, NO, DK, LT, PL, IE, CZ, SI, Freespee* www.freespee.com AT, NL, DE Adinsight* www.adinsight.co.uk UK Infinity tracking* www.infinity-tracking.com UK Optilead* www.optilead.co.uk UK Switchboard free www.switchboardfree.co.uk UK Freshegg www.freshegg.co.uk UK Avanser www.avanser.com.au AUS Jet Interactive* www.jetinteractive.com.au AUS * I read up on these or talked to them. These are my picks .
47. Call trackingWhat data can you get?• Data to keyword, group or traffic source level – for Phone contact / conversions• Splits by location, traffic source, time, date• Data on the call – how long to answer, to abandon, call duration• Data post call – was there a sale? (call wrapping)• Linkage between marketing activity and phone ROI on web channels (very important)• Which parts of your site drive contact• Tagging supports GA, Omniture, Others• Complete call analytics (including recording, if wanted)
48. Tracking tips• Make contact points trackable (phone, email, form, chat etc.) For example:• Collapsible phone section on Contact Us form – tracks intent• Phone ‘reveal’ icon, button or widget – tracks intent• Tap to call wrapper on Phone• Use a smart email form instead of email address• Use ‘ringme’ ringback functionality• If email being generated, tweak subject line• Integrate call centre events for maximum coverage• If you can’t buy phone tracking, make your own (mail me!)• Use phone tracking for major traffic groups, say:• PPC groups, Organic, Print, PPC, email, TV, Brand keywords• See your phone traffic in GA, Omniture and most packages
49. Follow the money – ROI example• 12.4% from Web increase• 12.5% increase from Multi Variate Testing• 3.5% increase in downstream conversion rate• 4% increase in online conversion from mobile optimised site• Reduced call time• Faster booking time• 5.5% increase in NPS (Customer sat) score• Customer delight – feedback. Less questions and worries• Total increase in conversion (measured) = 32%• Actual increase in conversion = 41%• NO additional marketing spend needed• Web Channel : ROI positive within 2 weeks
50. More reading. Slides and resources on slideshare.net Email : firstname.lastname@example.org Twitter : @OptimiseOrDie : linkd.in/pvrg14 Slideshare : slidesha.re/nlCDm6