The document discusses Belron's testing culture and multi-variate testing program. Belron has a small in-house optimization team that runs dozens of tests across its sites annually. Their testing aims to increase key metrics like revenue and customer satisfaction while lowering costs. Belron gathers ideas from analytics, customer feedback, and usability research. They employ A/B testing as well as multi-variate or "MVT" testing with dozens of variables. Through continual testing, Belron has achieved over $40 million in ROI from optimizations costing around $800,000 over two years.
Conversion conference london nov 2011 - multi channel testing - craig sullivanCraig Sullivan
Cross channel testing tips including how to optimise your call tracking, call centres, contact methods, contact flows. Also how to deflect, manage or remove cost from contact handling.
This set of slides also covers our optimisation process, results and shows some case studies.
If you're a serious marketer, cross channel is going to be vital for you in 2012.
We take this opportunity to introduce you Centaur Research (Market Research Fieldwork Company) headquartered at Mumbai, India. Centaur Research is a full service marketing research consultancy passionate about helping companies to make right decisions in order to grow their brands and business making a difference to meet the objectives and the need for change.
Centaur Research specializes in qualitative and quantitative market research fieldwork. Our team of in-house recruiters, interviewers, moderators , project managers and research Analyst make us the perfect partner to handle any projects across the globe.
E-mail: info@c-research.in
www.c-research.in
Conversion conference london nov 2011 - multi channel testing - craig sullivanCraig Sullivan
Cross channel testing tips including how to optimise your call tracking, call centres, contact methods, contact flows. Also how to deflect, manage or remove cost from contact handling.
This set of slides also covers our optimisation process, results and shows some case studies.
If you're a serious marketer, cross channel is going to be vital for you in 2012.
We take this opportunity to introduce you Centaur Research (Market Research Fieldwork Company) headquartered at Mumbai, India. Centaur Research is a full service marketing research consultancy passionate about helping companies to make right decisions in order to grow their brands and business making a difference to meet the objectives and the need for change.
Centaur Research specializes in qualitative and quantitative market research fieldwork. Our team of in-house recruiters, interviewers, moderators , project managers and research Analyst make us the perfect partner to handle any projects across the globe.
E-mail: info@c-research.in
www.c-research.in
We will examine how to create a cost benefit analysis document that clarifies your customer’s problem, explains your intended solution, and details the work required. We will break down costs of the current state and future state compared to forecasted benefits to show payback of the solution over time. This session will guide you through a simple process to help you understand how to calculate and communicate the business value of your SharePoint solutions.
Predicting Customer Behavior - An Introduction to iSkyiSky
iSky helps companies and brand owners predict customer behavior.
Our industry experts and research specialists design fully-integrated customer engagement programs to capture the motives behind customer behavior, and analyze the financial implications of this behavior on brand health and enterprise profitability. Our solutions are designed to support customer acquisition and retention initiatives and impact brand strategy.
Excelerated Sourcing, SAP Sourcing Overviewdanchilds
Excelerated Sourcing assists organisations running SAP in become best-in-class procurement functions through our advisory, implementation and optimisation services. Our focused services will enable an organisation to increase control and compliance in the end to end process which will lead to increased cost reduction and improved measurable and sustainable performance.
Seminar 2: Testing The Water: Market Research and Validation
Date: Thursday 15 March 2012
Time: 5.30pm-7.00pm
Venue: Cambridge campus (LAB026) - Chelmsford campus (MAB221)
You have a great idea but will anyone actually want to buy it. Understanding the market you're in, what your customers actually want and who you're going to be up against is key if you want to be a success.
Guest speaker: Stewart McTavish, Innovation Director, IdeaSpace
Stewart McTavish is the Innovation and Support Director at the University of Cambridge ideaSpace. Before joining the ideaSpace he founded three ventures in web and media. At ideaSpace Stew helps advise and connect entrepreneurs.
E-mailmarketing Conferentie 2011 / Presentatie Sander van der BlonkRapidSugar
Tijdens de E-mailmarketing Conferentie op 7 september jl. gaf Sander een presentatie over 10 jaar e-mailmarketing. Een scherpe visie op het verleden, het heden en de toekomst.
We will examine how to create a cost benefit analysis document that clarifies your customer’s problem, explains your intended solution, and details the work required. We will break down costs of the current state and future state compared to forecasted benefits to show payback of the solution over time. This session will guide you through a simple process to help you understand how to calculate and communicate the business value of your SharePoint solutions.
Predicting Customer Behavior - An Introduction to iSkyiSky
iSky helps companies and brand owners predict customer behavior.
Our industry experts and research specialists design fully-integrated customer engagement programs to capture the motives behind customer behavior, and analyze the financial implications of this behavior on brand health and enterprise profitability. Our solutions are designed to support customer acquisition and retention initiatives and impact brand strategy.
Excelerated Sourcing, SAP Sourcing Overviewdanchilds
Excelerated Sourcing assists organisations running SAP in become best-in-class procurement functions through our advisory, implementation and optimisation services. Our focused services will enable an organisation to increase control and compliance in the end to end process which will lead to increased cost reduction and improved measurable and sustainable performance.
Seminar 2: Testing The Water: Market Research and Validation
Date: Thursday 15 March 2012
Time: 5.30pm-7.00pm
Venue: Cambridge campus (LAB026) - Chelmsford campus (MAB221)
You have a great idea but will anyone actually want to buy it. Understanding the market you're in, what your customers actually want and who you're going to be up against is key if you want to be a success.
Guest speaker: Stewart McTavish, Innovation Director, IdeaSpace
Stewart McTavish is the Innovation and Support Director at the University of Cambridge ideaSpace. Before joining the ideaSpace he founded three ventures in web and media. At ideaSpace Stew helps advise and connect entrepreneurs.
E-mailmarketing Conferentie 2011 / Presentatie Sander van der BlonkRapidSugar
Tijdens de E-mailmarketing Conferentie op 7 september jl. gaf Sander een presentatie over 10 jaar e-mailmarketing. Een scherpe visie op het verleden, het heden en de toekomst.
What’s working right now in email marketing? As email is opened more and more frequently on mobile, campaigns should focus on more than just responsive designs. Here are the current best practices for email, along with 5 key trends we're seeing in email campaigns right now.
Conversion Conference - What's in YOUR toolkit?Craig Sullivan
This set of slides lists 12 practical tools, techniques or services you can use to improve conversion rates.
There are handy lists of companies and websites that will make a welcome addition to the stuff that all marketers should be using.
Leveraging Analytics To Improve Your Marketing Efforts:BusinessOnline
When it comes to marketing…we should assume nothing, and test everything. There are hundreds of variables affecting your website and campaign performance. How do you know which elements are working and which ones are holding you back? You test.
Understanding user behaviour in the omni channel world- jwtB2B Marketing
How to best leverage data insights from multi-channel platforms to impact ongoing marketing strategies and revenues. This includes customer behaviour across digital platforms and integrated online and offline user experience journey.
This presentation includes advice on how behaviours impact customers’ decision-making processes, practical advice on how to garner insights and how to measure and structure marketing campaigns across multi-channel platforms.
The session will also feature a case study example of applying the above and increasing ROI for B2B brands.
Benefits
Practical advice on managing multi-channel marketing activities
Customer insights in the digital world
Real life examples
How this impacts business objectives and ROI
Measurement of ROI retained via digital and social media
This will be presented at the Optimizely's San Francisco User Group session on Oct 4th. As with any program, an A/B Testing Practice also follows a specific maturity curve. Since it is much more complex and spans across various domains and business units, it begins with a "Sell" phase focused on getting buy-in from various stakeholders but with a specific focus on Engineering & QA, followed by "Scale" phase with focus on building team, efficiency and program and then on to "Expand" phase focused on wider scope/complex tests and strengthen the platform, over to the "Deepen" phase where the focus is to ingrain testing within the company's DNA, i.e., within the backend/algorithms, cross pollinate learning and testing across various business units. The final phase is the "Sustain" phase where Algorithmic Test Management takes over Testing, and Testing is productized as a Value Add service for monetization and brand captial creation. We will walk the audience through our own journey so far along the maturity curve, the lessons learnt along the way, the challenges and what worked for us. The session will be rounded up with a working session with the audience on their own journey, lessons and advice for others.
eMetrics SF 2011 - Integrating Analytics and Testing at Delljoel_wright
Presentation from eMetrics SF 2011: Joel Wright, Global Analytics Manager, and Charles Hua, Global Testing Manager explain the concepts and processes that Dell uses to manage the testing pipeline/roadmap planning and execution to ensure high quality tests, learnings, and actionable outcomes.
SigOpt at O'Reilly - Best Practices for Scaling Modeling PlatformsSigOpt
Companies are increasingly building modeling platforms to empower their researchers to efficiently scale the development and productionalization of their models. Scott Clark and Matt Greenwood share a case study from a leading algorithmic trading firm to illustrate best practices for building these types of platforms in any industry. Join in to learn how Two Sigma, a leading quantitative investment and technology firm, solved its model optimization problem.
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...PRWD
Craig Sullivan, Senior Optimisation Consultant covers the top 10 reasons why your conversion rate might suck. Packed with actionable tips and resources this presentation is for anyone wanting to improve their Conversion Optimisation. Craig covers common problems and topic areas such as issues with Google Analytics setup, inputs, tools, testing, testing cycles, product cycles, Photo UX, how to analyse statistics / Data, segmentation, multiple channel optimisation. The resource pack also include a maturity model, Crowd-sourced UX, collaborative tools, testing tools for CRO & QA, Belron Methodology example, and CRO and testing resources.
Cross Device Optimisation - Google Analytics ShortcutsCraig Sullivan
In this session, we explain how to mine GA for broken device experiences, flows, funnel blocks and more... Using a new grid tool we've developed, you can pull multi-dimensional segmented funnel and metric data from Google Analytics - we explain how it works, why you need it and what problems it solves. Find where your site is leaking money through data
Product Design is Poo - And we're all going to dieCraig Sullivan
A humorous presentation about what is wrong with the current way of building digital products. Showing what is wrong, explaining the signs and giving you a checklist for reforming your company - are laid out with links, resources and further reading.
Product design is Poo - And how to fix it!Craig Sullivan
A look at why product design is still so poor, even after 22 years of digital design work. Why do these problems exist and how can we remove them from the way we build products? Lean corporate and startup growth models are explored in the solutions to this horrendous problem!
Slides to go with a talk on rapid, lightweight research you can do before tackling a landing page, funnel step or lead-gen form. Comes complete with all the Google Analytics reports you'll need to mine useful data to share! Less bullshit, more truth in meetings!
Web Analytics Wednesday - Session Replay Tools are VitalCraig Sullivan
Session Replay or Screen Recording tools are now part of an arsenal of discovery toolkits that can drive optimisation, bug fixes, funnel and journey analysis - using qual and quant techniques. Without these tools, the analytics data misses emotion, frustration, friction and more - I've collated the best tips, tricks, tools and approaches to yield the most valuable insights for CRO / Growth Hacking.
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015Craig Sullivan
My Slides from Reaktor Breakpoint 2015 - This is by far the best deck (and hopefully talk) I've done this year. Masses of info, reading, articles, useful reports and more.
Surviving the hype cycle Shortcuts to split testing successCraig Sullivan
In this talk, I show the key shortcuts to stop doing stupid testing and move towards innovative and transformative design & build methodologies, including innovation through split testing exploration
Myths and Illusions of Cross Device Testing - Elite Camp June 2015Craig Sullivan
A compendium of the most common mistakes and problems people encounter when trying to optimise or split test cross device experiences (mobile, tablet, desktop, app, tv etc.)
Myths, Lies and Illusions of AB and Split TestingCraig Sullivan
What are the common assumptions about AB (split) testing that are wrong? What are the lies told by vendors, consultants and the stuff you have convinced yourself about. What is illusory - what can you trust - what's it really all about. 20 top myths debunked after asking fellow CRO professionals what is on THEIR top list.
20 Ways to Shaft your Split Tesring : Conversion ConferenceCraig Sullivan
This talk is the latest deck showing common problems that will easily break or skew your ab and multivariate testing results. Avoid these problems by following the simple advice in this deck!
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeCraig Sullivan
An updated deck of a short talk (30m) given at the first Brighton CRO meetup. Contains useful AB testing tools as well as full speaker notes for most of the slides.
#Measurecamp : 18 Simple Ways to F*** up Your AB TestingCraig Sullivan
An expanded deck of the top 18 blockers to getting successful AB or Multivariate test results. In this deck, you get a complete checklist of the stuff you need to prepare, watch, launch and monitor your testing, so it gets you the *right* conclusions.
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014Craig Sullivan
A practical toolkit for getting inside customers heads, in order to design and create persuasive psychological approaches to copy, pages, buttons, designs and your entire service. Craig shows you here how to mine what you already have - to design a better bank balance and continuously improving future for your company, staff and your customers.
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...Craig Sullivan
A roundup of all the things to help you maintain a competitive edge in experience design and conversion optimisation. With examples of companies putting this stuff together, the tools they are using and their project management approaches, this presentation delves deeper into the cultural aspects of CRO.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
2. Naked self promotion
@OptimiseOrDie
• Conversion Rate Optimisation • Contact deflection and
• A/B and Multi-variate testing online self service
• Cross channel optimisation • Site search analytics
• User Centered Design • Site, Page and Campaign
optimisation
• Usability testing
• Test design and execution
• Customer Research & Insight
• A long usable site portfolio
• Web Analytics
• ROI on UX improvements
• Browser and email
compatibility • Over 18 years of boring
meetings
Group CX Manager for Belron® (Safelite)
I get to do it with crowds
3. Outline
• What is optimisation?
• Belron® Testing Culture
• What is MVT
• Our testing program
• Advantages of testing
• The results
• Cross channel optimisation
• Case study : Chat ROI
• Resources
5. A set of techniques, implemented in
order to influence customer
behaviour towards these outcomes:
• Increased revenue or profit
• Increasing NPS (Customer Sat) scores
• Lower cost for business or contact centres
• Increasing productivity or labour flexibility
• Simply delighting customers
We may fail with 90% of what we try.
The 10% of winners let us shift
behaviour hugely and measurably.
7. If you go to the men's washrooms at the
Schiphol airport in Amsterdam, you may
notice there's a fly in the urinals. It’s
screen printed .
So what do you think most men do? That's
right, they aim at the fly when they
urinate.
They don't even think about it, and they
don't need to read a user's manual; it's
just an instinctive reaction that means
85% less spillage!
The interesting feature of these urinals is
that they're deliberately designed to take
advantage of this inherent human male
tendency.
This is my job.
8. Small team, big challenge
• Team of 3 - 2 Optimisers, 1 x part time designer
• Higher ROI or lower costs, without spending more
• Stuff that doesn’t hinder SEO, PPC, Performance
• Measurable ROI - not ego or opinion
• Fast return and low cost techniques
• Over 26M page tests, 17 international tests live now
• ROI is > 50 x optimisation costs
• Total cost = ~800k for two years. ROI > $40M USD
17. Insight - Inputs Forms
Remote Search
analytics Depth
testing analytics
interviews
Eye tracking Market
research
Customer A/B and
contact MVT testing
Social
analytics
Insight Unstructured
data
Web
Clicktale
analytics
Usability Competitor
testing Ops and CS Customer evals
teams Surveys services
18. Killer idea generation sources
• Analytics data
• Customer insight – contact, customer feedback, research
• Usability testing
• Diary studies
• Clicktale recordings
• Business owners – market data, customer mix, local knowledge
• Customer contact centre immersion (listening)
• Books, Websites, Tips
• Other optimisers
• Test data – variables plotted on axis
• Making lots of mistakes – Fail faster
19. Typical test lifecycle
Other test
Gather and results – good Brainstorm &
Instrument
review inputs variables & Hypotheses
learnings
Signoff (Legal, Design variables
Instrument Test Plan
Marketing, & creative
analytics Document
Brand) elements
Next Wave
Wave design Results of wave Washup
(repeat)
Business led Do verification
Repeat from
reporting (with = wave + 90/10% Declare winner
start
£x, without = £y) split
21. What’s our testing mix?
• A/B testing – Homepage, Landing pages, Funnels
• MVT testing – high traffic landing or home pages
– Mini (less than 8 variables)
– Midi (8-32)
– Maxi (>32 variables)
And 5 types of specialised testing:
• Funnel testing – bedding in new funnels
• Landing page optimisation
• Cross channel testing
• Iterative feature and product testing
• Micro testing
• Telephony MVT
22. Level 1 Level 2 Level 3 Level 4 Level 5
Starter Level Early maturity Serious testing Core business value You rock, awesomely
Dedicated team Ninja Team
Local Heroes Small team Cross silo team
Culture Volume Testing in the
Chaotic Good Low hanging fruit Systematic tests
opportunities DNA
_____________________________________________________________________________________________ _
Process Ad Hoc Outline process Well developed Streamlined Company wide
_______________________________________________________________________________________________
+Funnel +Cross channel +Spread tool use
Guessing + Multi variate optimisation testing Dynamic adaptive
Testing A/B testing Session replay Call tracking Integrated CRO targeting
focus Basic tools No segments Some segments and analytics Machine learning
Micro testing Segmentation Realtime
________________________________________________________________________________________________
Bounce rates + Funnel analysis + Funnel fixes + offline Multichannel
Analytics Low converting
integration funnels
Big volume Forms analytics
focus landing pages & High loss pages Channel switches
Single channel Cross channel
picture synergy
________________________________________________________________________________________________
+Regular usability + All channel view
Analytics + Customer sat
testing/research +User Centered of customer
Insight Surveys
Prototyping
Design scores tied to UX
Driving offline
Contact Centre Rapid iterative
methods Low budget Session replay
Layered feedback
testing and
using online
usability
Mini product tests design All promotion
Onsite feedback driven by testing
________________________________________________________________________________________________
Continual
Mission Get buyin Prove ROI Scale the testing Mine value improvement
33. Your ‘design’ team?
Web/Online
Pesky Meddling Kids?
Marketing
Who
designs this
stuff
anyway?
Brand
IT
THE
CEO CUSTOMER Legal
34. BENEFITS OF TESTING:
• Reduced ego and opinion (execs + others)
• Measurable ROI
• Faster time to market
• Increased conversion in the pocket early
• Iterative and rapid product lifecycles
• Faster, smaller, focused, measured releases
• Continual testing = continual improvement
• SLED vs. Agile = faster return, better focus
• Less reliance on anecdotal evidence
• Business owners part of testing culture
• Introduces a testing culture
• Large cross silo team now gets involved
• Reduces timidity – be brave and curious!
• Helps us hit the sweet spot...
39. Photo Guidelines
• Groups, vans, stores, corporate guff = • Female images work best in almost
NO
every country – they slay the guys
• Single person – not groups or even
person + customer • Natural, authentic and not ‘model’
• Open, Friendly, Natural, Smiling,
or ‘stock’ image types work best for
Engaged = YES people
• Uniform – has huge effect, even • We get better images now, but we
wearing branded ball cap improves keep working on the guidelines!
conversion • We still got a genital framing
• Look straight at viewer or slightly picture the other week
towards CTA
• Photoshoots and tests are now
• Plain backgrounds only
being used to drive advertising
• No folded arms, arms on hips, pointing, strategy
scowling, cheesy smiles, hands behind
back, hanging useless by the side, • Pre TV test of potential ‘actors’ on
clasped by groin, fiddling with hands website. X factor meets split
• Arms with prop instead – hold a testing.
clipboard, rag, tool – I’m working for
*you*
40. Cross channel mini studies
• Mini studies:
– Abandon testing
– Emergency phone switchoff & Overspill systems
– Contact deflection systems and self servicing
– Scripting Experiments
– APIs and other systems integration
– Chat testing
– Phone tracking
41. Abandon testing
Reduce unwanted phone hangups. Test plan
– Remove corporate guffspeak – Two countries, big phone systems,
– Don’t mention the website, esp. – Test variants of messages, music,
if this is the web number (D’oh!) timings, amounts quoted
– Don’t give a generic or corporate – Messages make a big impact –
drone apology message experiment freely with volumes
– Quote real KPIs and timings (ING – Opportunity is large due to high
direct, average queue time) call volumes we have
– Make routing emotionally driven – What people hate personally vs.
– not confusing categories that what those same people have on
map poorly onto customer needs their company lines, lol
– Experiment with queue timing – – On a mission here!
find sweet spot, don’t overquote
42. Switchoff Overspill
If you have spikes, have a throttle: Overspill systems:
– Busiest TV & Radio campaign so far, Q2 – Get a UX designer to make you a fast
2011 for Autoglass® entry system
– Designed a phone ‘switchoff’ system, – Make them simple, easy to learn and
permanently on tap
leverage your web platform
– Routed majority of contact to a web
form – Give API access to overspill
– Prevented a backlog situation from companies
developing – Sometimes heavy IT integration isn’t
– Bought time and still exceeded needed
customer expectations (<4h excellent) – “Nobody needs a freaking training
– Used twice to dampen inbound call course to use facebook” - Make sure
volumes your internal apps and call centre
overspill systems are built for instant
learnability
– This could save millions in process
efficiency. Same as web, really.
43. Contact deflection & self service
• People like immediate gratification! • What about your measurement?
• The iceberg of contact opportunity • Basics – Ratings, Did you find this
• Make people like you for not talking answer helpful?
to them on the phone –double win • Intermediate – Tracking behaviour
• If you stick it in help, will it go away? and contact channels
• No elephant graveyard, please (FAQ) • Add tracking of self service events or
• Instrument for analytics and ROI behaviour
• Set it up to measure and improve • Set up a game or reward structure for
staff
• Gamification of content? Adoption?
• Continually improve and get proof
• LOVEFiLM Help Centre:
• Add phone tracking - make avoidable
– Ratings, Deflection tracking, self contact visible
service tracking
• Remove all barriers to self servicing
– V1 :Please view in your account for high cost/volume contacts
secion, V2 : Please visit your
account, V3 : Please check your • Try Amazon help – where is phone?
next billing date, V4 - Final • Help experts? Sticky Content are
version – to show the real info : brilliant (www.stickycontent.co.uk)
“Your next billing date is 1 Feb”
44. Script Optimisation
• How can we optimise phone conversion?
– Dedicated test teams
– Script testing often limited range/shift in companies now
– Now – we can use A/B or MV testing, with API integration
– Try out 3,4,5 different versions of key areas or entire script
– ‘Wrapping’ sends XML message event, triggers analytics tag
– Analyse in the same way as an A/B or multi-variate web test
– Fast optimisation of handling scripts, using a mini lab
– Should yield 3-5% higher performing scripts, minimum
– Shows what variables drive script performance globally
– These emotional and psychological triggers can be mined
– Aim for Q2 2012 for a proof of concept
45. Platform APIs
• Use of web platform APIs now for:
– Contact centre messages
– Facebook app
– Mobile site and apps
– Scheduling
– Analytics tag firing
• Just making APIs, automagically makes:
– People use them
– Improvements happen
– New ideas happen, that you didn’t expect
– Incompatible stuff get glued together
– Complex IT integration projects go away
46. Chat Testing Ringback
Contact
Form
Contact
Us
You must measure behaviour
across contact channels
during tests like these
A - Chat B - Chat C - Chat
Split
A B C
48. Chat testing
• 1 month trial – properly A/B tested • Customers happy, yes, but not in a
• Measured all the contact channels for good business way
movements – what happened?
• Like putting your customer service
• Over a week, chat saves 17 calls – desk into the street
yay!
• You must measure influences in
• But swapped that for 329 chats @
contact volumes when trialling
8.25 minutes. Ouch.
Chat
• Increased ringback requests? Email
form about the same. • Be very careful – prove your ROI
on any pilot.
• No statistically significant impact on
conversion • Only 7-10% use and out of those,
• Assumption was big impact, huge less than 50% are influencable etc.
impact! etc.
• Anecdotally, a wonderful tool. • The productivity Myth of chat
Agreed. • Your situation may differ – please
TEST!
49. Call tracking
How it works:
• Rent a bank of numbers
• Add a dynamic ‘tag’ to your page
• Phone number appears automatically
• Customer calls this number
• Captures the keywords, landing page, converting page, traffic source
• Geo mapping captures customer location
• When the call is hung up, optionally record the outcome code (sale, loss)
• Have a special number for Facebook, SEO, Adwords traffic etc.
• Lets you analyse from web -> phone call -> wait time -> wrapping
• Builds a link between web and call, finally
50. Call tracking
Why you should use it!
• Most people under-report on web driven phone contact or sales
• You can’t work out your advertising effectiveness, if you don’t include
phone
• If you don’t measure phone, your A/B tests may be wrong
• You can capture all your web AND phone, in one easy interface
• Allows you a multi-channel view of all your contact sources
• Lets you record offline campaigns as well with specific numbers (print, TV
etc.)
• You can experiment with routing, messages, on hold music – much more
easily
• If you can’t afford lots of phone numbers, this is an easy way to get
tracking
51. Company Website Coverage
Who?
Mongoose Metrics* www.mongoosemetrics.com UK, USA, Canada
Ifbyphone* www.ifbyphone.com USA
TheCallR* www.thecallr.com USA, Canada, UK, IT, FR, BE, ES, NL
Call tracking metrics www.calltrackingmetrics.com USA
Hosted Numbers www.hostednumbers.com USA
Callcap www.callcap.com USA
UK, SE, FI, NO, DK, LT, PL, IE, CZ, SI,
Freespee* www.freespee.com
AT, NL, DE
Adinsight* www.adinsight.co.uk UK
Infinity tracking* www.infinity-tracking.com UK
Optilead* www.optilead.co.uk UK
Switchboard free www.switchboardfree.co.uk UK
Freshegg www.freshegg.co.uk UK
Avanser www.avanser.com.au AUS
Jet Interactive* www.jetinteractive.com.au AUS
* I read up on these or talked to them. These are my picks .
52. Call tracking
What data can you get?
• Data to keyword, group or traffic source level – for Phone
contact / conversions
• Splits by location, traffic source, time, date
• Data on the call – how long to answer, to abandon, call
duration
• Data post call – was there a sale? (call wrapping)
• Linkage between marketing activity and phone ROI on web
channels (very important)
• Which parts of your site drive contact
• Tagging supports GA, Omniture, Others
• Complete call analytics (including recording, if wanted)
53. Tracking tips
• Make contact points trackable (phone, email, form, chat etc.)
For example:
• Collapsible phone section on Contact Us form – tracks intent
• Phone ‘reveal’ icon, button or widget – tracks intent
• Tap to call wrapper on Phone
• Use a smart email form instead of email address
• Use ‘ringme’ ringback functionality
• If email being generated, tweak subject line
• Integrate call centre events for maximum coverage
• If you can’t buy phone tracking, make your own (mail me!)
• Use phone tracking for major traffic groups, say:
• PPC groups, Organic, Print, PPC, email, TV, Brand keywords
• See your phone traffic in GA, Omniture and most packages
54. Follow the money – ROI example
• 12.4% from Web increase
• 12.5% increase from Multi Variate Testing
• 3.5% increase in downstream conversion rate
• 4% increase in online conversion from mobile optimised site
• Reduced call time
• Faster booking time
• 5.5% increase in NPS (Customer sat) score
• Customer delight – feedback. Less questions and worries
• Total increase in conversion (measured) = 32%
• Actual increase in conversion = 41%
• NO additional marketing spend needed
• Web Channel : ROI positive within 2 weeks
55. More reading. Slides and resources on slideshare.net
Email : sullivac@gmail.com
Twitter : @OptimiseOrDie
: linkd.in/pvrg14
Slideshare : slidesha.re/nlCDm6