Multivariate testing-buyers-guide (1)

6,598
-1

Published on

Published in: Technology, Business

Multivariate testing-buyers-guide (1)

  1. 1. Market Data / Supplier Selection / Event Presentations / User Experience Benchmarking / Best Practice / Template Files / Trends & InnovationMultivariate TestingBuyer‟s Guide
  2. 2. MultivariateTestingBuyer‟s GuidePublished May 2011 Econsultancy London Econsultancy New York 4th Floor, The Corner 41 East 11th St., 11th Floor 91-93 Farringdon Road New York, NY 10003 London EC1M 3LN United StatesAll rights reserved. No part of this publication may be United Kingdomreproduced or transmitted in any form or by any means, Telephone:electronic or mechanical, including photocopy, recording Telephone: +1 212 699 3626or any information storage and retrieval system, without +44 (0)20 7269 1450prior permission in writing from the publisher. http://econsultancy.comCopyright © Econsultancy.com Ltd 2011 help@econsultancy.com
  3. 3. Contents 1. Introduction ..................................................................... 3 1.1. About Econsultancy .................................................................... 4 1.2. Acknowledgements ..................................................................... 5 2. Executive Summary ......................................................... 6 3. The Market ....................................................................... 7 3.1. What is multivariate testing? ...................................................... 7 3.1.1. Types of multivariate tests ....................................................... 8 3.2. Market trends .............................................................................. 9 3.2.1. Testing market goes from strength to strength ....................... 9 3.2.2. Self-service solutions give market impetus ............................ 11 3.2.3. Vendors differentiate tools with self-learning technology .... 12 3.2.4. Companies need skill sets to complement technology .......... 13 3.2.5. Enlightened marketers cultivate culture of on-going testing ..................................................................................... 14 3.3. Return on investment ................................................................ 15 4. Strengths, Weaknesses, Opportunities and Threats (SWOT) .......................................................................... 16 4.1. Strengths .....................................................................................16 4.2. Weaknesses................................................................................. 17 4.3. Opportunities .............................................................................18 4.4. Threats ........................................................................................19 5. Costs and Pricing Models .............................................. 21 6. Tips and Pitfalls ............................................................. 22 6.1. Introduction .............................................................................. 22 6.2. How to find the right MVT technology ..................................... 22 6.3. How to manage internal expectations and requirements ........ 26 6.4. A few tips to avoid frustration and failure ................................ 27 7. Market Positioning Charts ............................................. 31 7.1. Explanation for Chart 1: Business model .................................. 31 7.2. Explanation for Chart 2: Type of company .............................. 32 7.3. Market positioning overview: Business model......................... 33 Multivariate Testing Buyer‟s Guide All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  4. 4. 7.4. Market positioning overview: Type of company ...................... 348. Supplier Matrix .............................................................. 359. Specialist Conversion Optimisation Agencies ............... 3710. Supplier Marketplace and Profiles ................................ 43 10.1. Accenture Interactive ................................................................ 43 10.2. Adobe Systems........................................................................... 52 10.3. Amadesa .................................................................................... 59 10.4. Autonomy .................................................................................. 65 10.5. GlobalMaxer ............................................................................... 71 10.6. Google Website Optimizer .........................................................77 10.7. HiConversion ............................................................................. 83 10.8. Maxymiser ................................................................................. 90 10.9. Monetate .................................................................................... 96 10.10. SiteSpect .................................................................................. 102 10.11. Vertster .................................................................................... 108 10.12. Visual Website Optimizer ........................................................ 114 10.13. Webtrends ............................................................................... 120Multivariate Testing Buyer‟s GuideAll rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storageand retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  5. 5. 1. Introduction Econsultancy‟s buyer‟s guides are the ideal starting place for anybody researching new suppliers in digital market sectors. They contain in-depth vendor profiles; to help you quickly evaluate suppliers and service providers, as well as market analysis to help you put things into perspective. Vendors are selected for the report based on a combination of factors, not limited to but including:  Analysis of capabilities (services / products)  Clients  Experience (qualifications / trade bodies / case studies)  Expertise (by sector / topic)  UK and/or USA status (occasional exceptions are made)  Ability to take on and fulfill new projects  Recommendations from trusted sources (or anecdotal evidence to the contrary)  Google visibility  Business model (a high proportion of turnover should be related to these services)  Company Web site Econsultancy does not explicitly recommend any of the suppliers featured in these guides, principally because it is impossible for us to work with all of them to form a first-hand opinion. But we do believe - based on an intensive and careful selection process - that the chosen vendors represent quality. Buyer‟s Guides are updated on an annual basis, so the information contained within is recent and thus valid. Send any questions or comments to Econsultancy‟s Research Director Linus Gregoriadis (linus@econsultancy.com). Multivariate Testing Buyer‟s Guide Page 3 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  6. 6. 1.1. About Econsultancy Econsultancy is a digital publishing and training group used by more than 200,000 Internet professionals every month. The company publishes practical and time-saving research to help marketers make better decisions about the digital environment, build business cases, find the best suppliers, look smart in meetings and accelerate their careers. Econsultancy has offices in New York and London, and hosts more than 100 events every year in the US and UK. Many of the world‟s most famous brands use Econsultancy to educate and train their staff. Some of Econsultancy‟s members include: Google, Yahoo, Dell, BBC, BT, Shell, Vodafone, Virgin Atlantic, Barclays, Deloitte, T-Mobile and Estée Lauder. Join Econsultancy today to learn what‟s happening in digital marketing – and what works. Call us to find out more on +1 212 699 3626 (New York) or +44 (0)20 7269 1450 (London). You can also contact us online.Other related Econsultancy reportsCustomer Experience and Engagement Statisticshttp://econsultancy.com/reports/customer-experience-statisticsConversion Reporthttp://econsultancy.com/reports/conversion-reportCheckout Optimisation Guidehttp://econsultancy.com/reports/checkout-optimization-guideOnline Measurement and Strategy Reporthttp://econsultancy.com/reports/online-measurement-and-strategy-reportUser Experience Buyer’s Guidehttp://econsultancy.com/reports/user-experience-buyers-guideUsability and User Experience: A Beginner’s Guidehttp://econsultancy.com/reports/usability-and-user-experience-a-beginner-s-guideMultichannel Customer Experience Reporthttp://econsultancy.com/reports/multichannel-customer-experience-reportCustomer Engagement Reporthttp://econsultancy.com/reports/customer-engagement-report Multivariate Testing Buyer‟s Guide Page 4 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  7. 7. 1.2. Acknowledgements Our thanks go out to all the vendors and agencies who contributed profiles and additional insights for this report. We would also like to thank the following client-side contributors:  Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron  Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco  Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph  Matthew Curry, Head of Ecommerce, Lovehoney  Sandra White, Head of Optimisation, The Financial Times Multivariate Testing Buyer‟s Guide Page 5 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  8. 8. 2. Executive Summary The focus of this report is multivariate testing, including a discussion of market trends and detailed profiles submitted by 12 leading vendors. Even though online testing is not a new concept, adoption of effective optimisation technology and processes by marketers has been relatively slow. However, spending on MVT solutions has been increasing rapidly in the last few months, mainly fuelled by the significant growth of e-commerce and improved testing tools. A large proportion of companies have started to understand the benefits of testing and senior management buy-in has generally become less of a concern. The multivariate testing market hasn‟t reached the maturity stage yet and still represents an emerging area for the majority of marketers, but there are plenty of new exciting opportunities that vendors can tap into. Many companies are planning to test across multiple channels and integrate their MVT solutions with other systems such as web analytics, web content management and CRM. Segmentation and targeting also are increasingly sought after features, enabling organisations to differentiate between various categories of visitors and identify which segments convert better. Trends covered in this guide include:  The testing market is going from strength to strength, as companies are investing more resources and planning to integrate testing into their optimisation efforts.  Self-service solutions are giving the market impetus, with a significant proportion of organisations demanding tools that remove the complexity from optimisation efforts.  Self-learning technology is used to differentiate tools, while segmentation and targeting are gradually becoming a natural addition to any MVT solution.  A knowledgeable, experienced human interface is needed to complement testing technology.  Enlightened marketers are cultivating a culture of on-going testing by setting aside a portion of their budgets for testing and involving more business units. The Supplier Marketplace and Profiles section (Section 10) contains profiles of the following MVT providers: Accenture Interactive, Adobe Systems, Amadesa, Autonomy, GlobalMaxer, Google, HiConversion, Maxymiser, Monetate, SiteSpect, Vertster, Visual Website Optimizer, Webtrends. Multivariate Testing Buyer‟s Guide Page 6 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  9. 9. 3. The Market While online testing is nothing new, the proliferation of digital channels has enabled it to evolve from a confusing and therefore untapped area into a discipline in its own right. Despite the fact that online testing is an essential component of an effective marketing optimisation strategy, adoption by marketers has been much slower than expected.3.1. What is multivariate testing? Multivariate (MVT) testing or multi-variable testing is the process through which the performance of more than one component of a website can be evaluated in a live environment. The aim is to experiment with different combinations of factors on a web page in order to establish the most effective way of driving users to carry out a specific action. The value of multivariate testing lies in understanding the effect of each variable (called „main effect‟), but also the combined effect of all variables included in the test, often referred to as an „interaction effect‟. Multivariate testing has its roots in the design of experiments (DoE) statistical method developed by Sir Ronald A. Fisher, who has been using it to determine how various factors affect subsequent processes and actions. Several statisticians such as Genichi Taguchi have been refining it for decades, and multivariate testing was first applied to online marketing in the late 1990s. Since then, the industry has grown from strength to strength and testing has become an integrated service offering. MVT is a powerful method that enables marketers to experiment with and discover calls-to- action, user interaction designs and user experience flows that motivate users to take a desired action. Multivariate tests enable them to modularise any web page and dynamically change what elements appear on the page, where they appear and the amount of traffic they receive. The difference between A/B testing and multivariate testing The core differentiator between A/B and multivariate tests is that MVT enables you to test multiple elements simultaneously. A/B testing is a common approach used by marketers to optimise the performance of a website. This testing method consists of creating two versions of a page; one acts as a „control‟ element, enabling you to establish a performance baseline, while the other serves as a „challenger‟, allowing you to determine if the new versions perform better than the control one. On the one hand, A/B testing can be the best option if you want to test broader page concepts and your targets are easily achievable and do not require a lot of effort. The amount of planning and traffic requirements for this approach are significantly lower than is the case with MVT, but it is much more difficult to control external factors. A/B tests also provide little insight into why a certain version performed better than the other. On the other hand, multivariate testing is more complex and enables marketers to understand the impact that each of the tested factors has on the conversion rate. Therefore, the insight obtained during a multivariate test can feed into future experiments. Typically, companies start with small scale A/B tests and gradually move towards more sophisticated testing methods. As they reach a maturity point in their testing practices, they become more interested to know which specific change within the A/B test has driven the most significant impact. At this stage, multivariate testing (MVT) becomes a critical tool and is Multivariate Testing Buyer‟s Guide Page 7 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  10. 10. integrated into their testing strategy. A/B testing is also sometimes used as a control mechanism to double-check the results of a multivariate test.3.1.1. Types of multivariate tests At a high level, there are two types of multivariate tests that you need to know about: Full factorial testing A full factorial multivariate test is the most widely known method, running every possible combination based on the number of factors being tested. For example, if you have five page factors with five options each you would have 125 possible combinations to test (5x5x5). Website traffic is distributed equally among all combinations, so each of them gets the same amount of traffic. Full factorial testing makes no assumptions regarding statistics or the mathematics of data and provides all the data you need to determine which particular combination and factor had the best performance. While this type of MVT provides better results, it may take longer to complete. Partial or fractional factorial testing By using advanced statistical models, a partial or fractional factorial test runs a subset of every possible combination and only a fraction of these combinations are exposed to website traffic. The conversion rate for the combinations that are not exposed is driven from the ones included in tests based on some assumptions and mathematical models. Although this method requires significantly less traffic, it is inherently based on assumptions and one can argue that hard data is always a better alternative than inference. Taguchi testing Taguchi testing is a particular type of fractional factorial testing. The statistical methods developed by Genichi Taguchi were originally used in the manufacturing industry, where specific assumptions were made to decrease the time spent on quality assurance processes and the number of combinations needing to be tested. This method enables marketers to test specific combinations against one another, compare the results of each of them and basically determine which combination performs best based on comparison. Taguchi-based tools typically draw conclusions from low traffic, as they are able to extrapolate results. Therefore, they usually seem a good choice for low-traffic websites. The Taguchi Method has been criticised by statisticians, economists and optimisation experts, debating whether it can be applied to online marketing and advertising reliably. It relies on a number of assumptions and most often there are other factors that need to be taken into consideration when running a test. By not testing every single combination, you can never be sure that you are selecting a „true‟ winner. In this case, experts recommend testing the winner against a control using A/B testing. Multivariate Testing Buyer‟s Guide Page 8 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  11. 11. 3.2. Market trends3.2.1. Testing market goes from strength to strength Interest in conversion optimisation has intensified in the last few months and online testing vendors have been reaping the benefits. The landscape we see today was shaped by a flurry of acquisitions in recent years:  As part of its strategy to deliver a comprehensive optimisation solution for online businesses, Omniture acquired Offermatica, an on-demand A/B testing and multivariate testing company, for $65 million. Omniture was acquired by Abobe in 2009.  CMS provider Interwoven acquired Optimost for $52 million and was then acquired by Autonomy in March 2009.  Accenture acquired Memetrics and Maxamine in 2008, two companies providing testing and optimisation services.  Webtrends purchased Widemile in 2009, adding multivariate testing and site optimisation to its portfolio of customer intelligence technologies and services. Evidence from all directions suggests the testing market is going from strength to strength. While web analytics still plays an important role in improving conversion rates, a significant proportion of companies are pouring more resources into testing and planning to integrate it into their optimisation efforts. According to Econsultancy‟s Conversion Report 2010, carried out in association with RedEye, almost half of client-side respondents (48%) said they were planning to carry out A/B testing and 42% said they intended to use multivariate testing [Figure 1] in order to improve conversion rates. Only 17% said they were currently using MVT, while 44% were already doing A/B testing. Figure 1: Which of the following methods do you plan to use for improving conversion rates? Respondents: 306 Multivariate Testing Buyer‟s Guide Page 9 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  12. 12. Despite the fact that a minority of organisations use these techniques, around half of responding organisations with experience in these methods consider A/B testing (57%) and multivariate testing (48%) to be highly valuable methods for improving conversion [Figure 2]. Figure 2: How valuable do you find the following methods for improving conversion rates? Respondents: 318 According to Tim Brown, CEO of Maxymiser, spending on testing solutions is increasing every year: “Our belief is that even today only 10 – 20% of marketers (depending on which geographic market you consider) that would generate great ROI from MVT are actively using MVT. We also see that once our customers initiate testing programmes the results are so compelling that they increase their spend significantly year-on-year and often start personalisation initiatives building on the test results. These are the two dominant trends we see to be driving spend.”What the experts say“Businesses are realising that MVT testing can no longer be considered a point solution, but rather as an integralcomponent of a marketing platform. Businesses are looking for more than content creation, delivery andoptimisation. Rather, they want an end-to-end solution that enables them to deliver personalisation acrossmultiple channels. Businesses will continue to invest in MVT capabilities, but more so to those that are part of aclosed-loop platform that enables marketers to leverage benefits across all customer touch points.” Autonomy“Foviance has worked with a number of the MVT vendors in this buyers guide and as with any data technologythere will be ones that fit your business requirements better than others. We find that once the technology issuesare solved that businesses still have challenges in achieving value from optimisation tools but that if they get itright the benefits can be enormous. Where else do you get these sorts of immediate uplifts?” John D’Arcy, Practice Director, Analytics & Insight, Foviance Multivariate Testing Buyer‟s Guide Page 10 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  13. 13. 3.2.2. Self-service solutions give market impetus As discussed above, the online testing market is growing at a fast pace as more organisations are beginning to realise the importance of investing in good testing technology and are actively looking to develop conversion rate improvement projects. This is mainly due to the fact that they increasingly understand the benefits of prioritising the improvement of conversion rates for existing traffic rather than trying to attract additional traffic that converts less. Driven by market demand, multivariate testing vendors often focus on a self-service model with professional, premium services on the side. More companies demand tools that remove the complexity from optimisation efforts so they can focus on what to test and how to test rather than what technology and algorithms to use. According to Dr. Karl Blanks, co-founder of Conversion Rate Experts: “The main trend is that the market is growing quickly. Several years ago, split testing was an obscure niche; now it’s relatively well known. New testing platforms are appearing each year, each with its own advantages.” No-frills, low-cost or free solutions that provide competent testing often represent the best starting point for inexperienced companies that are looking to get a taste of MVT and want to build a business case for a more advanced, ongoing programme. Although these tools often lack advanced functionality and may deliver sub-par performance if not used properly, they are continuing to bolster their market position and fuel the overall growth in the use of testing. Free tools such as Google‟s Website Optimizer are considered to be good enough testing tools for the majority of marketers, especially if they are not currently carrying out any tests, are not able to allocate sufficient financial resources or need to build a business case showing that testing works before committing to a large investment. They can often be implemented relatively easily, so the involvement of the IT department is minimal. However, companies that start out testing in-house are typically looking to outsource once they realise they need more external experience to refine and optimise their tests. According to Tim Brown of Maxymiser, fully managed services will become more prevalent due to the fact that self-service models are increasingly associated with hidden costs and high resource requirements: “The most important trend occurring in this market is the flight to quality and full service. MVT is now recognised as a major strategic initiative in leading online players. Customers want to deal with focused, specialist expert staff and solutions focused on conversion optimisation. “The hidden costs and high resource needs of self-serve MVT solutions are also now becoming well-known and the vast majority of even the largest enterprises have moved or are moving to providers who offer top quality fully managed services.”What the experts say “When MVT testing was only affordable to large corporations, the testing platforms tended to compete on„number of features‟. Many of the newer solutions focus on ease of use, which has been a big obstacle until now.” Dr. Karl Blanks, Conversion Rate Experts“More companies are becoming aware of conversion rate optimisation. This is leading to increased demand forfree or easy-to-use tools (e.g. Google Website Optimizer and Visual Website Optimizer) and is helping to keepcosts down. The cost of tools has been driven down. There‟s no need to spend £x,xxx on advanced testing anymore.” Stephen Pavlovich, Director, Conversion Factory Multivariate Testing Buyer‟s Guide Page 11 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  14. 14. What the experts say“[Choosing a vendor] depends on the level of expertise within the business and the level of access to yourbackend code. Some vendors offer a managed service, taking away the complexities involved in setting up testsand many also minimise the level of change required in your backend code for each MVT you run. These canhave a huge impact on the decision points. I would also stress that it is worth looking at the right vendor forwhere your business is and if choosing a more basic solution, you always have the opportunity to move up to oneof the more advanced (and usually more costly) platforms once you‟ve operationalised MVT within yourbusiness.” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“An emerging trend in the market is companies finding that a combination of enterprise grade technology andsupport, coupled with a full self-serve option sets them up perfectly to do both complex testing, in parallel withquick and simple testing.“The net result is a highly flexible testing culture which gives the ability to test both quickly and with complexitywithout necessarily having to work at the defined pace of your vendor. To meet the requirements of ourcustomers we provide both self-service and fully managed testing.” Tim Brown, CEO, Maxymiser3.2.3. Vendors differentiate tools with self-learning technology According to Stephen Pavlovich, Director of Conversion Factory, testing tools are becoming more differentiated as vendors target either companies with basic testing requirements or those with more advanced requirements: “We’re seeing more of a separation between the simple and advanced tools. More tools are targeting either end of this spectrum, rather than the whole market.” One of the key areas vendors are trying to differentiate in is the use of predictive algorithms and self-learning technologies, aiming to maximise conversions and revenue beyond the limitations of manually configured tests. These algorithms continuously analyse each test‟s performance to produce the best possible combinations, uncovering opportunities over time and adjusting variables accordingly. As all adjustments are made based on all tests, not just a specific selection, self-learning solutions enable organisations to adopt a big picture approach. They also benefit from full visibility and control; whenever last-minute changes occur in terms of business strategy and opportunities, companies are able to override the self-learning technology and adjust any testing elements manually. Self-learning technologies are often described as closed feedback loops, as they continuously identify combinations that are performing better than the control and thus begin converging towards the best solution. However, these platforms need some „learning time‟ to produce effective results. Data is increasingly considered a crucial asset in the online testing ecosystem and one of the essential components of any testing strategy. According to Autonomy, an increasing number of companies are planning to use data in order to perform more informed tests that can drive significant changes: “It is no longer about finding one winner but rather understanding what makes content a winner – how and why does content such as messaging, price, or creative improve conversion rates. To extract the most value from MVT testing, businesses are looking to better mine their data to drive faster, more informed testing.” Multivariate Testing Buyer‟s Guide Page 12 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  15. 15. Segmentation and targeting are gradually becoming a natural addition to any MVT solution, helping companies to differentiate between different categories of website visitors and serve each of these groups with the appropriate variation based on needs and relevance. They also enable them to identify if certain segments of visitors convert better than others, run separate tests for new and returning visitors and measure performance variations across multiple categories and metrics. As the demand for actionable data that help companies discover new audiences, optimise performance and uncover unique insights into customer behaviour increases, more and more vendors will focus on adding these features to their platforms.What the experts say“An important trend is the tight integration of targeting with testing. Different offers have different conversionrates depending on visitor segment, so it is imperative to find what works from whom (through testing) and thenimplement that logic on your website permanently (through targeting).” Paras Chopra, Founder, Wingify (Visual Website Optimizer)3.2.4. Companies need skill sets to complement technology Any successful testing programme requires at least four skill sets which can be rarely found in a department or even company – a project manager with a good knowledge of both marketing and statistics, a copywriter, an IT consultant and an analyst. A knowledgeable, experienced human interface is often the key differentiator between mediocre results and great performance lifts, as Stephen Pavlovich of Conversion Factory notes: “The MVT tool won’t increase your conversion rate on its own. An understanding of conversion is infinitely more important.” Therefore, relying on technology alone can have disastrous results for both the company‟s bottom line and overall morale. To a certain extent, testing is a part of every organisation‟s marketing strategy. However, only the largest companies have dedicated testing and optimisation teams. For the rest, this responsibility usually lies with the web analytics or even IT departments. While large enterprises have started to set up their own, fully staffed testing departments, companies at the other end of the spectrum tend to have just one person responsible for the entire process. Mid-sized companies usually consider the testing process too tedious and resource- intensive and find it difficult to organise a smoothly running team and allocate resource (often all members of the team are working to full capacity on other projects). Another challenge mentioned by an increasing number of organisations is the involvement of the IT department. However, as MVT vendors are focusing on easy-to-use, visual solutions, this will become less of a concern. The concept of the „get-in-line‟ IT department is gradually becoming a thing of the past as technical knowledge is no longer necessary and tests can be set up in a few minutes at the most. While technology is able to drive positive results without any human involvement, automation is not a silver bullet and can only be used to a certain extent. Therefore, having access to a dedicated, experienced account manager and support staff who can explain processes, providing advice on how to best design and carry out tests and dealing with any issues is the main reason companies choose a full-service solution over a self-service one. Multivariate Testing Buyer‟s Guide Page 13 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  16. 16. As multivariate testing technology is still evolving and not all solutions are the same, the need to complement the technology with a competent human interface is becoming a critical element of any testing initiative.What the clients say“Having worked with a number of MVT vendors, from free to the higher end spectrum, the key is knowing theroles required to support MVT and having these in place. Typical roles include that of a designer, developer andsomeone to oversee the MVT programme. If you‟re completely new to this, consider a managed service to get youup and running whilst you work out whether it is more cost effective to run this in-house and the roles you needfor your company.“Getting this right is the most important part of making MVT work. It is not a set once and forget machine whichchurns out improvements, it needs someone that understands your business, working with designers and codersthat can make this a reality. Assuming MVT then takes off it will become the foundation of a lot of your siteimprovement projects and will therefore need a sufficient level of internal support” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco3.2.5. Enlightened marketers cultivate culture of on-going testing To fully benefit from MVT and execute a successful strategy, testing needs to become a part of every company‟s marketing culture, not just a one-off experiment. Audiences are constantly shifting and their behaviour is changing every day, so testing should be an ongoing process, part of a complete optimisation cycle. While most organisations find it difficult to cultivate a culture of continuous testing, they are increasingly taking steps in this direction by setting aside a portion of their budgets for testing and involving more business units. According to Paras Chopra, Founder of Wingify, the company behind Visual Website Optimizer, one of the main challenges for vendors is to support these efforts by making sure their existing and potential clients understand the benefits of testing and how they can integrate it into their marketing processes: “The most important challenge for companies in this sector is to cultivate a culture of continuous testing. Even the most mature enterprises do A/B testing on and off and rarely have a dedicated resource in the organisation for testing. This results in testing taking a back seat as compared to other activities like SEO, web analytics and email marketing. “The challenge for us is to make sure companies are always testing something or the other by making sure the products get integrated in the client’s workflow and by helping them understand the benefits of continuous testing.” Although multivariate testing has gone mainstream in the last few months, the majority of marketers fail to embrace testing as a regular business practice and optimise the whole range of marketing processes and activities. However, as the need for smooth integration with internal systems increases, testing will become more embedded in every company‟s culture and used to drive decisions across multiple business functions. Multivariate Testing Buyer‟s Guide Page 14 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  17. 17. 3.3. Return on investment The potential return on investment and benefits of multivariate testing technology can be summarised as follows:  Save time and benefit from more robust results than A/B testing. – Depending on the MVT solution you use, your traffic volume and the number of elements you are testing, you may save a lot of time compared to A/B testing. – Although more sophisticated software and analysis is required and traffic requirements are high, insights generated through the use of MVT are more robust and actionable than those generated by a few A/B tests. – MVT solutions provide insight into what worked and what didn‟t, helping you to examine customer preferences and adjust your product and business strategy accordingly.  Increase sales by helping to drive the right user actions. – Optimised pages can result in a significant increase in conversion rates and average order value, having the potential to yield great gains. – Higher quality customer engagement on websites can also drive increased up-sell and cross-sell opportunities.  Quickly react to changes in customer demand and inventory availability.  Directly measure increases in revenue. – MVT solutions allow to you track conversion rates before and after each test, helping you to determine the exact impact on your bottom line. – This also gives you an indication of how much more you can generate through additional tests.  Decrease customer dissatisfaction and churn. – Customers are less likely to leave the website if specific elements of their journey are optimised. – Even limited testing can eliminate a significant amount of problems, leading to considerable revenue growth.  Avoid lost revenue. – If some changes are implemented without a structured, scientific approach, they will negatively impact sales.  Find the individual variable that causes the lift or decrease in your site performance. – Compared to A/B testing, a multivariate test provides the insight necessary to determine the single most significant factor that affects the performance of any page and thus the ability to make informed decisions. Multivariate Testing Buyer‟s Guide Page 15 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  18. 18. 4. Strengths, Weaknesses, Opportunities and Threats (SWOT) This section contains a „SWOT’ analysis of the multivariate testing market from the perspective of vendors. It also serves as a summary of some of the points made above.4.1. Strengths  Growing awareness of conversion rate optimisation and interest in testing. – In contrast to the weak sales of traditional retailers in 2010, e-commerce sales reached more than $200 billion in the US 1 and around £60 billion in the UK 2. This is a clear indicator of how strong consumer confidence is in the online retail channel and the extent of benefits companies can reap. – More organisations are starting conversion rate optimisation programmes hoping to ramp up their revenue and differentiate in over-crowded markets. The majority of companies now understand the benefits of online testing and its contribution to the bottom line, sometimes embedding it into their overall business strategy. – However, this increased demand of MVT solutions may also have an impact on pricing, driving down the cost of testing solutions and leading to a proliferation of free tools.  Technology makes testing easier. – As multivariate testing moves further upstream, marketers demand simple, easy to understand solutions with enough features to get them started. Vendors have started to respond to this growing demand by developing MVT tools that are more user-friendly and have visual, intuitive interfaces. – This simple approach to how tests are designed and monitored also translates into increased uptake, as testing is no longer viewed as a complex, tedious process. While solutions are getting more sophisticated in terms of capabilities, they are also making it easier to set up a test by using point-and-click features and integrating advanced functions to minimise or even eliminate involvement of the IT team.  Companies are more geared up internally to initiate testing programmes. – As online testing is becoming a more common practice and building a business case for a comprehensive optimisation programme is no longer a major challenge, companies are investing more resources to make sure they have the infrastructure and technical capabilities to use MVT. – As senior management buy-in is becoming less of an issue and measurement tools are increasingly sophisticated, MVT is finally attracting significant budget.  Vendors and companies alike are becoming more experienced and knowledgeable. – As more brands are diving into online testing and thus providing good case studies and best practice examples, both companies and vendors are becoming more experienced and knowledgeable. 1According to comScore, the value of US e-commerce sales in 2010 was $228 billion (February 2011). 2According to the IMRG Capgemini e-Retail Sales Index published in January 2011, £58.5 billion were spent online in the UK in 2010. Multivariate Testing Buyer‟s Guide Page 16 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  19. 19. What the experts say“I believe this sector will move from strength to strength. There are very few solutions available to onlinemarketers that deliver absolute clarity that all developments to the website are delivering proven lifts inconversion rates and strong insight as to why.“The marketing ROIs are transparent and huge and once a „test and improve‟ culture is embedded within anonline marketing group it is permanent. More and more PPC and SEO spend will transition to this sector with anincreased ROI.” Tim Brown, CEO, Maxymiser4.2. Weaknesses  Companies are slow in building a testing culture. – Some organisations still think of testing as a one-off experiment and a tactic that can boost their revenue quickly, instead of integrating it into their marketing processes and overall business strategy. The majority fail to embed this at every level of the company and tend to keep it contained in a single department/team.  Many solutions still lack integration capabilities. – There is an increasing demand for more advanced integration capabilities that can leverage the data generated via other channels. However, most vendors are still not focusing enough on multichannel optimisation programmes and are thus missing out on a tremendous opportunity.  Technology can only do so much. – As in other areas of digital marketing, the majority of companies rely too much on technology without investing enough financial resources to hire specialists and build a cohesive team dedicated to testing. – This is one of the first mistakes that inexperienced companies make, but they soon realise that human insight is still required to accurately interpret results.  Bad advice and self-proclaimed experts reduce the credibility of the industry. – Although there are more expert practitioners in the industry, there are also many self- proclaimed experts that have just started to dip their toes into testing and provide bad advice. This not only leads to erosion of client buy-in, but also has a negative impact on trust in platforms‟ capabilities.  High traffic requirements to get significant results. – Many smaller sites don‟t have enough traffic to warrant multivariate testing and obtain meaningful results. Testing not only requires a significant amount of resources, but also time and traffic to make sure that you choose the best performing combination.What the experts say“Companies regularly have to link test data to their internal backend data, as not all conversions can be trackedonline. This can still be a major challenge, and is likely to remain so until all backends are built with tracking inmind.“A new challenge is the increasing amounts of bad conversion rate optimisation advice. More so than before,we‟re seeing companies getting bad advice, implementing tests incorrectly, stopping tests too early, and thenmistakenly promoting the „winning‟ variation - often, ironically, lowering their conversion rate in the process.” Stephen Pavlovich, Director, Conversion Factory Multivariate Testing Buyer‟s Guide Page 17 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  20. 20. What the experts say“One of the largest weaknesses among almost all of the players is an inability to drive intelligent testing. Thesevendors focus more on usability features and less so on ensuring businesses are testing smart and faster to driveROI. It‟s more than usability. It‟s about leveraging the meaning of your customer data in real time to driveintelligent testing as part of a multichannel revenue optimisation programme.” Autonomy“The technology is only as smart as the information it can be given. To use it to its full potential, it needs to beintegrated with other systems and across site, so that this can be fed into the decision-making layer. This cantake some leg work.“But once you start a proper programme of testing and optimisation, though, the potential is almost endless,assuming you have decent traffic volumes.” Sandra White, Head of Optimisation, The Financial Times4.3. Opportunities  Multichannel testing and integration support. – While multivariate testing remains an emerging area for most marketers, many companies are looking to test across multiple channels and integrate their testing solutions with other systems such as web content management, web analytics and CRM. – Vendors need to work with each client to help them identify where testing sits in their overall marketing ecosystem and determine the extent of integration they need (either with native tools or third parties).  Segmentation and targeting. – The ability to differentiate between new and returning visitors, serve different variations to all categories of website visitors and identify the segments that convert better than others is an increasingly sought after feature, so vendors have the opportunity to gain a larger market share by focusing on this area.  Ease of use. – Many marketers are still pointing out that some of the current MVT solutions require them to get the IT department involved. Vendors need to take into consideration all types of users that may interact with the platform and design the user interface – especially the reporting and analysis modules – accordingly. – Particularly, information distribution features represent an essential component that vendors need to consider, as the majority of companies are looking for an easy way to get multiple departments involved and disseminate information throughout the entire organisation.  Market ripe for growth. – Although the fact that adoption has been slow is considered a weakness by many, it also means that the sector is ripe for growth. The online testing market hasn‟t reached the maturity stage yet and MVT providers are able to ramp up their revenue by helping organisations understand the benefits of testing and getting them on board. – Many client-side companies are building their own in-house testing teams, thus creating demand for specialist support and training that vendors can easily tap into. Multivariate Testing Buyer‟s Guide Page 18 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  21. 21. What the experts say“Most astute MVT buyers are already buying into not just the capability to MVT test, but also a strategic roadmapthat builds on the structural and design improvements delivered by the testing to provide targeted personalexperiences for each customer and prospect to further increase revenues and effectiveness of cross-sell and up-sell campaigns.” Tim Brown, CEO, Maxymiser“Segmentation and targeting remain a very key part of testing. The challenge facing marketers with theseopportunities is how they can understand their customer data better to accurately segment their customers andtarget them with a personalised experience. Customers need a solution that allows them to intelligentlyunderstand where the most actionable segments are that are critical to revenue generation.” Autonomy“The biggest opportunity of growth is testing for mobile sites because the challenges for optimizing mobile sitesare very different from their desktop equivalent. On mobiles you have small screen-sizes, tap, gestures and otherunnatural forms of interaction. Tighter integration of behavioural targeting with A/B and MVT is another bigopportunity in the market.” Paras Chopra, Founder, Wingify“The biggest opportunity is for simple testing platforms that integrate with the most common CMSs. In the next12 months, we should expect CMSs like WordPress and Magento to have more testing extensions, which willopen up conversion rate optimisation to a significantly larger audience.“Testing is also going to become more advanced, as conversion rate optimisers start demanding increasedfunctionality from their tools. One area I‟m keen to see developed is multi-channel testing: testing across ad,landing page, email and phone.” Stephen Pavlovich, Director, Conversion Factory4.4. Threats  Free tools may have a negative impact on the uptake of paid-for solutions. – If free tools become more sophisticated and start offering a range of advanced capabilities, this is a potential threat for paid-for vendors focusing on fully managed services. However, this is an unlikely scenario because free tools are usually implemented by inexperienced buyers that have just started to carry out testing. After an initial experimentation phase, companies often move to full-service solutions that are more scalable and are able to match more complex requirements.  Lack of compatibility with internal systems / platforms. – Some vendors tend to over-promise by implying that there is no need for IT involvement and that they can facilitate pain-free integration. Companies that choose point solutions for financial reasons without having a long-term strategy quickly find out that they are not compatible with the organisation‟s internal systems. This has a negative impact on senior management buy-in and will damage the credibility of the industry. Multivariate Testing Buyer‟s Guide Page 19 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  22. 22. What the experts say“Over promising by vendors regarding their deployment technologies. If you believe the vendor speak everyonehas a „one-time‟ integration and everyone implies zero IT is needed post deployment. This is simply not true andleads to slow testing progress and high Total Cost of Ownership issues which discredits the whole sector. Buyers -do your due diligence!” Tim Brown, CEO, Maxymiser“There will be a clear divergence between low end, throw away tools and enterprise class testing solutions. We‟reseeing this now – marketers looking to significantly impact revenue will want testing solutions.” Autonomy Multivariate Testing Buyer‟s Guide Page 20 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  23. 23. 5. Costs and Pricing Models There are different costs that you need to take into consideration before carrying out any multivariate tests. Internal costs such as effort, time, potential expense associated with more complex tests and creative (whether you have in-house resources to deal with this or you need to outsource) can be easily overlooked when you‟ve just started to get your feet wet. One of the first steps is to decide what type of service model is most appropriate for your goals and budget. As Paras Chopra of Wingify puts it: “Always double check if you want to take control of the MVT solution you are purchasing or you want to get dependent on a third party to get even a simple test done.” When choosing a self-service model, you need to carefully examine if the potential benefits outweigh the costs. The cost of creating and running tests internally is sometimes overlooked and challenges associated with setting up and managing a team can come as a total surprise. While a self-service model seems to currently be the best option, you may realise along the way that you can‟t keep up with the increasingly challenging business requirements and market conditions, and that you need specialist help. In the case of a fully-managed service, you are not only paying for someone to create and run tests, but also for strategic knowledge and expertise. The consulting component of full service models means that the vendor can help steer your direction in an advisory capacity. When shortlisting vendors, you need to make sure you are aware of all the cost breakdowns and what the percentage allocated to consulting is. As far as costs are concerned, multivariate testing tools can be classified into the following three high-level categories:  Free tools  Low-cost solutions that allow you to set up tests „visually‟  Powerful enterprise-grade solutions Although prices for MVT software and services have been gradually increasing in the last few years, increased competition in the sector means that they have remained fairly stable. The most commonly used method of charging is a subscription model, based on either a flat licence fee or server calls. Price ranges vary greatly, depending on the vendor‟s background and expertise, which often come at a premium. Performance-based pricing is sometimes used, but implementation of this method is still rare and doesn‟t seem to be a popular option among the vendors profiled in this guide. Although performance lifts are directly measurable, it is sometimes difficult to prove that these improvements were driven by the MVT platform alone. However, some vendors will agree to use this charging method if it is specifically requested by their clients. Multivariate Testing Buyer‟s Guide Page 21 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  24. 24. 6. Tips and Pitfalls6.1. Introduction In order to identify the right multivariate testing technology and get the best fit for your organisation, you need to understand that different providers offer different types of solutions and their technology and expertise varies a lot. The detailed company profiles and market positioning charts included in this guide will help you find the right fit. This section contains some key questions and considerations that you need to keep in mind – and some pitfalls to avoid.6.2. How to find the right MVT technology This section acts as a checklist, but should also draw your attention to some pitfalls to avoid. Before you start looking for a suitable MVT vendor, you need to have a very clear testing plan, including information related to what types of tests you intend to run, available in-house resources and expected results. As Emilija Vilkyte, Head of Digital Revenue Optimisation at The Telegraph notes: “Before choosing an MVT vendor you need to be very clear about the type of tests you are looking to run, how often, how much internal resource you have got and how quickly you expect to see the results, as there are a lot of different products and propositions in the market. Also be clear where the tests will run (same domain vs. subdomains and third parties).” Get an initial feel for what the multivariate testing market has to offer by identifying potential suppliers and spending sufficient time to understand their products. Another initial step is to decide what type of service model is most appropriate for your goals and budget.  A full-service solution may be suitable if you don‟t want to invest in additional resources and you want someone else to manage the entire process for you. In this scenario, you pay an ongoing fee rather than incur a one-off cost. Make sure that you review all charging structures and choose the most suitable one for your needs and available budget. You may want to consider investing in a full-service solution if any of the following is true about your organisation: – Your conversion process or sales cycle is extremely complex, making it difficult to identify which components of the sales funnel are worth optimising. – Your website visitors come from multiple sources, such as email, social media, offline campaigns and you want to capitalise on this data in your tests or segment them by traffic source. – You want to use advanced segmentation and targeting techniques to serve different combinations to your visitors. – You want to test multiple elements on the same page and avoid technical complications and overburdening your IT department. – You use an advanced analytics solution and want to integrate a testing package.  A self-service solution is a good choice if you feel comfortable enough to operate the MVT technology yourself. If you have a team in place with good knowledge of multivariate testing and want to run the tests in-house, this is probably more suitable for you. While a self-service model can seem a cheaper alternative in the short-term, you may soon come to the conclusion Multivariate Testing Buyer‟s Guide Page 22 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  25. 25. that associated costs far exceed what you expected and you don‟t have enough experienced people to manage the entire testing cycle internally.What the experts say“If budget is an issue, I‟d strongly recommend using a free tool and getting the best conversion advice you can.“And if budget isn‟t an issue, get the best consultant possible and map out everything you‟ll need the platform todo – then choose the tool. (I see companies spending £x,xxx per month on testing platforms which they‟re usingto the same level as GWO – which is free).” Stephen Pavlovich, Director, Conversion Factory“Be honest and realistic about the broad range of skill sets and availability of in-house resources that are neededto drive a high ROI testing roadmap if you only purchase a „self-serve‟ platform. Even if the resources areavailable within the team is this the best use of them? The right turn-key full service solution offers a very lowand transparent Total Cost of Ownership and guarantees you will be able to realise an ambitious testing roadmapeven in complex web environments.” Tim Brown, CEO, Maxymiser“Start simple and don‟t expect business changing uplifts early on. The beauty of MVT is the last letter of theacronym – Testing. MVT allows you to test many ideas within a relatively short space of time, speeding up thelearning process and development time. From experience, MVT never ends. In addition, you won‟t always createimprovements – learn and improve. There is always something to improve, something to tweak, new ideas to tryout and with any business, as the business dynamics change, as do the opportunities MVT can leverage. Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“Breadth of features is very important. Make sure that the solution is not only easy to use, but offers all thefeatures that you would need. The last thing you would want is to realise that the tool does not support creating aparticular test you have in mind.” Paras Chopra, Founder, Wingify“If you‟re doing testing, don‟t forget segmentation. If you can‟t use your testing tool to segment, you should setup your web analytics to track this. Although you can find one design that converts best for everyone, this isn‟tthe optimal way to get a lift – look at which creatives most appeal to different audience segments. Morecompanies now offer you the ability to then serve these to visitors – it means you are always showing the bestconverting creative to visitors, based on historical data.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron The most important step for a buyer is to establish if the supplier not only has a very comprehensive technical solution, but also sufficient training and expertise to run multivariate tests.  Does the vendor have a strong multivariate testing background or have they just added the solution to a broad range of technologies/services without much of a history or the appropriate credentials in this area? – Is multivariate testing a core competence? How long has the company been in the testing business? – Is the vendor adequately staffed to deal with advanced projects? Ask for case studies showing evidence of improved performance and ROI and make sure the solution is scalable enough to match your complex requirements. – Do they have a track record of working with a wide range of clients? Have they run tests for organisations with a similar business model to your own? Multivariate Testing Buyer‟s Guide Page 23 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  26. 26. – You should contact recent clients directly for references before you start working with a specific vendor. Recently departed clients can also provide a clearer picture of the vendor‟s capabilities than current clients.  How many employees are specifically dedicated to the support of MVT customers? Is there sufficient support available for the geographies you focus on? What is the average turnaround time? Paras Chopra of Wingify said: “Ideally, you would need a dedicated account manager whom you can contact any time you want to get your query sorted. So, making sure you get quality support from a knowledgeable person from the vendor’s team is very important. Do a background check on the vendor too – ask their existing customers if they are happy.”What the clients say“Think about what you want from MVT. In our case, MVT doesn‟t really cover what we do. It is more abouthaving a testing platform that allows us to optimise our site, which is an ongoing process. Having control overwhole pages is therefore preferable to being limited to „test areas‟.“I would also recommend looking at how much account management is included, and whether this covers helpwith testing strategy or any consultancy. Also, what are the workflow limits in terms of getting tests up? Cansimple tests be set up by anyone in-house with some basic technical knowledge?“Bear in mind that testing also requires design resource, which most vendors don‟t supply.” Sandra White, Head of Optimisation, The Financial Times“A few recommendations: Choose two front runners and do a proof of concept for a month or two. Be very clear what‟s expected from the solution and what you will use it for. Make sure it integrates with other platforms you use. Check tag load times from multiple geo locations. Negotiate on the price as much as possible – there are ways to save money by limiting a number of tests a month, doing some data or design work yourself, removing some feature from your account e.g. there is no point in going straight to personalisation if you site usability is fundamentally wrong. Make sure the solution is flexible in case your business strategy changes.” Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph Some additional questions to ask vendors are:  Would you be able to access real-time reports? Are reports customisable to match your requirements and internal policies? You should be able to view a sample report to become familiar with the typical statistics and numbers included and to see if you find the format/layout usable.  What are their annual downtimes and what backup methods do they use? How is the data related to your tests stored? Who has access to this data?  How smooth is the integration of their software with your systems? Would it have any impact on your website loading times? The best way to find the right MVT vendor that suits your needs and budget is to evaluate several providers by arranging a pre-contract trial of all systems and deploying a proof of concept (POC) test. A significant amount of work is involved for both you and the vendors, so don‟t embark on this route if you are not serious about hiring one of them at the end of the trial. Multivariate Testing Buyer‟s Guide Page 24 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  27. 27.  Narrow the list of suppliers – select up to five technologies by taking into account the key considerations outlined above and then choose two or three „front runners‟.  Put together at least two tests by defining meaningful and measurable goals and including multiple elements/pages. A good mix of simple and complex tests can help you identify the technology that best handles fundamentals as well as complex requirements.  Create a level-playing field by ensuring that all suppliers receive the same specifications and instructions.  Allocate sufficient time and resources to better evaluate the performance of each vendor – remember that the decision you make has a tremendous impact on your bottom line.  Evaluate each MVT provider against clear criteria, such as performance, technology, recommendations and service.What the clients say“When the salmon return from the sea to the river, they stop feeding until they spawn. If they‟re not interested infeeding, how do people catch them? It may take all day, with a salmon sitting behind a rock, to find the right lureand the perfect place to drop it. You might catch a fish the first time or spend all day with nothing to take home.You‟ll try endless combinations of lure, bait and placement to get any reaction and just when you are ready togive up – bang – a flash of line tells you that you‟ve finally got some attention.“MVT is like having a thousand lines in the water, to find the lure that will catch the most fish. You‟re welcome topatiently try everything in your repertoire but those that experiment, they get their fish quicker and more easilythan everyone else. If you have less budget, a competitive market or both – make sure you have the edge whenyou‟re out fishing.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron“Whilst MVT vendors look to features as a way of differentiating themselves, and what makes or breaks an MVTtool is the interface you use to construct tests. If it‟s painful, or limited, you‟ll find yourselves not performing asmany tests as you should. If possible, try to arrange a trial of the system, and try to put together both simple andcomplicated tests. Multipage and multi-element tests are normally a good place to start.“If ultimately you‟re looking to pass MVT responsibilities to your content team, then make sure that the tool hasa good test setup wizard or otherwise easy-to-use test construction.“If you‟re able, conduct tests that optimise on a monetary figure, such as revenue or, even better, profit. That wayyou can make an easier judgement on how much the tool is worth to you. MVT isn‟t a magic bullet, it will requirea lot of effort to get results. Don‟t budget on how much extra revenue you could theoretically make, remember tobe pragmatic.” Matthew Curry, Head of Ecommerce, Lovehoney “After hearing about six pitches we have chosen our two „front runners‟ to run a POC and measured them onperformance, technology, recommendations, future road map, time it took from brief to launch... giving differentweighting to each criteria. This made it really easy to choose a vendor at the end. Do not get „hooked‟ on upliftonly during the POC – a lot of partners are very flexible and this will improve as they get to know you sitestructure.“You need to ensure that you check tag load times from your key audience locations and not only the UK. Makesure your solution can be easily integrated with your analytics and ad server at least.” Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph Multivariate Testing Buyer‟s Guide Page 25 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  28. 28. 6.3. How to manage internal expectations and requirements Companies that set unrealistic goals and believe that a single test can revolutionise the way their website works and significantly boost their conversion rates are prone to failure from the beginning. To prevent this, start with a test roadmap that can help you set the right internal expectations and focus on achievable goals, but also ensure that the relationship with the MVT vendor is smooth and painless. As John D‟Arcy of Foviance notes: “A big challenge is to deliver a set of tests that will impact your business immediately and give you meaningful learning. To do that you need to take a step back and deliver a prioritised roadmap of tests before you start out. Understanding the potential impact of a test can be analysed with web analytics data but you get much more actionable information by bringing in learning from your user experience tests, your voice of the customer tools and from competitor benchmarks. Try prioritising your tests based on these data sources and you deliver lots more value. We find that clients who are able to do this coherently are able to deliver a test roadmap that provides both immediate commercial potential and long term customer experience impact.” The amount of work that is required internally when using multivariate or similar technology varies a lot, broadly depending on the number and complexity of the tests you are running, but also on the type of service model you choose (full-service versus self-service solution). Before choosing a vendor, you need to identify exactly what services and expertise you need, as this will greatly influence internal requirements. Seamless collaboration across the entire organisation is an essential prerequisite, as all departments need to get involved in this process.What the clients say“The key for any business is to involve the right stakeholders. MVT can make those less involved feel like they‟relosing control. Instead highlight the reduction in guesswork, in wasted development time trialling ideas andmoney saved in sifting out the bad ideas from the good as quickly as possible. Keep the key stakeholders engagedand involved and the whole process becomes easier.” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“The amount of technical work required internally is very much dependent on the underlying technology of themultivariate testing tool you use. For a content proxy tool, you‟ll need to make changes to your DNS, so you‟llhave to be careful that it‟s not disruptive. You may then need to add additional parameters to your URLs forgoals that can‟t be tracked by other means. Other than that initial configuration, further tests shouldn‟t requireany overheads.“For JavaScript based tools, you have two options. Tools either require a combination of JavaScript tags andareas of your site „cordoning off‟ for testing using HTML tags, which if you aren‟t in direct control of your sourcecan be a constant pain. Other tools require a single JavaScript tag placed on any page to be included in the test –so insertion into a header file makes sense.“For any tool you use, even going through a proxy, you may be affecting site speed, so work close with yourtechnical department to monitor the impact your testing has. You should also, as a matter of course, work withyour content and design departments when constructing any tests to make sure that you aren‟t drasticallyaltering the look and message of the site (unless that‟s the point of the test!).” Matthew Curry, Head of Ecommerce, Lovehoney Multivariate Testing Buyer‟s Guide Page 26 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011

×