Your SlideShare is downloading. ×
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Multivariate testing-buyers-guide (1)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Multivariate testing-buyers-guide (1)

6,056

Published on

Published in: Technology, Business
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,056
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
10
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Market Data / Supplier Selection / Event Presentations / User Experience Benchmarking / Best Practice / Template Files / Trends & InnovationMultivariate TestingBuyer‟s Guide
  • 2. MultivariateTestingBuyer‟s GuidePublished May 2011 Econsultancy London Econsultancy New York 4th Floor, The Corner 41 East 11th St., 11th Floor 91-93 Farringdon Road New York, NY 10003 London EC1M 3LN United StatesAll rights reserved. No part of this publication may be United Kingdomreproduced or transmitted in any form or by any means, Telephone:electronic or mechanical, including photocopy, recording Telephone: +1 212 699 3626or any information storage and retrieval system, without +44 (0)20 7269 1450prior permission in writing from the publisher. http://econsultancy.comCopyright © Econsultancy.com Ltd 2011 help@econsultancy.com
  • 3. Contents 1. Introduction ..................................................................... 3 1.1. About Econsultancy .................................................................... 4 1.2. Acknowledgements ..................................................................... 5 2. Executive Summary ......................................................... 6 3. The Market ....................................................................... 7 3.1. What is multivariate testing? ...................................................... 7 3.1.1. Types of multivariate tests ....................................................... 8 3.2. Market trends .............................................................................. 9 3.2.1. Testing market goes from strength to strength ....................... 9 3.2.2. Self-service solutions give market impetus ............................ 11 3.2.3. Vendors differentiate tools with self-learning technology .... 12 3.2.4. Companies need skill sets to complement technology .......... 13 3.2.5. Enlightened marketers cultivate culture of on-going testing ..................................................................................... 14 3.3. Return on investment ................................................................ 15 4. Strengths, Weaknesses, Opportunities and Threats (SWOT) .......................................................................... 16 4.1. Strengths .....................................................................................16 4.2. Weaknesses................................................................................. 17 4.3. Opportunities .............................................................................18 4.4. Threats ........................................................................................19 5. Costs and Pricing Models .............................................. 21 6. Tips and Pitfalls ............................................................. 22 6.1. Introduction .............................................................................. 22 6.2. How to find the right MVT technology ..................................... 22 6.3. How to manage internal expectations and requirements ........ 26 6.4. A few tips to avoid frustration and failure ................................ 27 7. Market Positioning Charts ............................................. 31 7.1. Explanation for Chart 1: Business model .................................. 31 7.2. Explanation for Chart 2: Type of company .............................. 32 7.3. Market positioning overview: Business model......................... 33 Multivariate Testing Buyer‟s Guide All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 4. 7.4. Market positioning overview: Type of company ...................... 348. Supplier Matrix .............................................................. 359. Specialist Conversion Optimisation Agencies ............... 3710. Supplier Marketplace and Profiles ................................ 43 10.1. Accenture Interactive ................................................................ 43 10.2. Adobe Systems........................................................................... 52 10.3. Amadesa .................................................................................... 59 10.4. Autonomy .................................................................................. 65 10.5. GlobalMaxer ............................................................................... 71 10.6. Google Website Optimizer .........................................................77 10.7. HiConversion ............................................................................. 83 10.8. Maxymiser ................................................................................. 90 10.9. Monetate .................................................................................... 96 10.10. SiteSpect .................................................................................. 102 10.11. Vertster .................................................................................... 108 10.12. Visual Website Optimizer ........................................................ 114 10.13. Webtrends ............................................................................... 120Multivariate Testing Buyer‟s GuideAll rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storageand retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 5. 1. Introduction Econsultancy‟s buyer‟s guides are the ideal starting place for anybody researching new suppliers in digital market sectors. They contain in-depth vendor profiles; to help you quickly evaluate suppliers and service providers, as well as market analysis to help you put things into perspective. Vendors are selected for the report based on a combination of factors, not limited to but including:  Analysis of capabilities (services / products)  Clients  Experience (qualifications / trade bodies / case studies)  Expertise (by sector / topic)  UK and/or USA status (occasional exceptions are made)  Ability to take on and fulfill new projects  Recommendations from trusted sources (or anecdotal evidence to the contrary)  Google visibility  Business model (a high proportion of turnover should be related to these services)  Company Web site Econsultancy does not explicitly recommend any of the suppliers featured in these guides, principally because it is impossible for us to work with all of them to form a first-hand opinion. But we do believe - based on an intensive and careful selection process - that the chosen vendors represent quality. Buyer‟s Guides are updated on an annual basis, so the information contained within is recent and thus valid. Send any questions or comments to Econsultancy‟s Research Director Linus Gregoriadis (linus@econsultancy.com). Multivariate Testing Buyer‟s Guide Page 3 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 6. 1.1. About Econsultancy Econsultancy is a digital publishing and training group used by more than 200,000 Internet professionals every month. The company publishes practical and time-saving research to help marketers make better decisions about the digital environment, build business cases, find the best suppliers, look smart in meetings and accelerate their careers. Econsultancy has offices in New York and London, and hosts more than 100 events every year in the US and UK. Many of the world‟s most famous brands use Econsultancy to educate and train their staff. Some of Econsultancy‟s members include: Google, Yahoo, Dell, BBC, BT, Shell, Vodafone, Virgin Atlantic, Barclays, Deloitte, T-Mobile and Estée Lauder. Join Econsultancy today to learn what‟s happening in digital marketing – and what works. Call us to find out more on +1 212 699 3626 (New York) or +44 (0)20 7269 1450 (London). You can also contact us online.Other related Econsultancy reportsCustomer Experience and Engagement Statisticshttp://econsultancy.com/reports/customer-experience-statisticsConversion Reporthttp://econsultancy.com/reports/conversion-reportCheckout Optimisation Guidehttp://econsultancy.com/reports/checkout-optimization-guideOnline Measurement and Strategy Reporthttp://econsultancy.com/reports/online-measurement-and-strategy-reportUser Experience Buyer’s Guidehttp://econsultancy.com/reports/user-experience-buyers-guideUsability and User Experience: A Beginner’s Guidehttp://econsultancy.com/reports/usability-and-user-experience-a-beginner-s-guideMultichannel Customer Experience Reporthttp://econsultancy.com/reports/multichannel-customer-experience-reportCustomer Engagement Reporthttp://econsultancy.com/reports/customer-engagement-report Multivariate Testing Buyer‟s Guide Page 4 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 7. 1.2. Acknowledgements Our thanks go out to all the vendors and agencies who contributed profiles and additional insights for this report. We would also like to thank the following client-side contributors:  Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron  Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco  Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph  Matthew Curry, Head of Ecommerce, Lovehoney  Sandra White, Head of Optimisation, The Financial Times Multivariate Testing Buyer‟s Guide Page 5 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 8. 2. Executive Summary The focus of this report is multivariate testing, including a discussion of market trends and detailed profiles submitted by 12 leading vendors. Even though online testing is not a new concept, adoption of effective optimisation technology and processes by marketers has been relatively slow. However, spending on MVT solutions has been increasing rapidly in the last few months, mainly fuelled by the significant growth of e-commerce and improved testing tools. A large proportion of companies have started to understand the benefits of testing and senior management buy-in has generally become less of a concern. The multivariate testing market hasn‟t reached the maturity stage yet and still represents an emerging area for the majority of marketers, but there are plenty of new exciting opportunities that vendors can tap into. Many companies are planning to test across multiple channels and integrate their MVT solutions with other systems such as web analytics, web content management and CRM. Segmentation and targeting also are increasingly sought after features, enabling organisations to differentiate between various categories of visitors and identify which segments convert better. Trends covered in this guide include:  The testing market is going from strength to strength, as companies are investing more resources and planning to integrate testing into their optimisation efforts.  Self-service solutions are giving the market impetus, with a significant proportion of organisations demanding tools that remove the complexity from optimisation efforts.  Self-learning technology is used to differentiate tools, while segmentation and targeting are gradually becoming a natural addition to any MVT solution.  A knowledgeable, experienced human interface is needed to complement testing technology.  Enlightened marketers are cultivating a culture of on-going testing by setting aside a portion of their budgets for testing and involving more business units. The Supplier Marketplace and Profiles section (Section 10) contains profiles of the following MVT providers: Accenture Interactive, Adobe Systems, Amadesa, Autonomy, GlobalMaxer, Google, HiConversion, Maxymiser, Monetate, SiteSpect, Vertster, Visual Website Optimizer, Webtrends. Multivariate Testing Buyer‟s Guide Page 6 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 9. 3. The Market While online testing is nothing new, the proliferation of digital channels has enabled it to evolve from a confusing and therefore untapped area into a discipline in its own right. Despite the fact that online testing is an essential component of an effective marketing optimisation strategy, adoption by marketers has been much slower than expected.3.1. What is multivariate testing? Multivariate (MVT) testing or multi-variable testing is the process through which the performance of more than one component of a website can be evaluated in a live environment. The aim is to experiment with different combinations of factors on a web page in order to establish the most effective way of driving users to carry out a specific action. The value of multivariate testing lies in understanding the effect of each variable (called „main effect‟), but also the combined effect of all variables included in the test, often referred to as an „interaction effect‟. Multivariate testing has its roots in the design of experiments (DoE) statistical method developed by Sir Ronald A. Fisher, who has been using it to determine how various factors affect subsequent processes and actions. Several statisticians such as Genichi Taguchi have been refining it for decades, and multivariate testing was first applied to online marketing in the late 1990s. Since then, the industry has grown from strength to strength and testing has become an integrated service offering. MVT is a powerful method that enables marketers to experiment with and discover calls-to- action, user interaction designs and user experience flows that motivate users to take a desired action. Multivariate tests enable them to modularise any web page and dynamically change what elements appear on the page, where they appear and the amount of traffic they receive. The difference between A/B testing and multivariate testing The core differentiator between A/B and multivariate tests is that MVT enables you to test multiple elements simultaneously. A/B testing is a common approach used by marketers to optimise the performance of a website. This testing method consists of creating two versions of a page; one acts as a „control‟ element, enabling you to establish a performance baseline, while the other serves as a „challenger‟, allowing you to determine if the new versions perform better than the control one. On the one hand, A/B testing can be the best option if you want to test broader page concepts and your targets are easily achievable and do not require a lot of effort. The amount of planning and traffic requirements for this approach are significantly lower than is the case with MVT, but it is much more difficult to control external factors. A/B tests also provide little insight into why a certain version performed better than the other. On the other hand, multivariate testing is more complex and enables marketers to understand the impact that each of the tested factors has on the conversion rate. Therefore, the insight obtained during a multivariate test can feed into future experiments. Typically, companies start with small scale A/B tests and gradually move towards more sophisticated testing methods. As they reach a maturity point in their testing practices, they become more interested to know which specific change within the A/B test has driven the most significant impact. At this stage, multivariate testing (MVT) becomes a critical tool and is Multivariate Testing Buyer‟s Guide Page 7 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 10. integrated into their testing strategy. A/B testing is also sometimes used as a control mechanism to double-check the results of a multivariate test.3.1.1. Types of multivariate tests At a high level, there are two types of multivariate tests that you need to know about: Full factorial testing A full factorial multivariate test is the most widely known method, running every possible combination based on the number of factors being tested. For example, if you have five page factors with five options each you would have 125 possible combinations to test (5x5x5). Website traffic is distributed equally among all combinations, so each of them gets the same amount of traffic. Full factorial testing makes no assumptions regarding statistics or the mathematics of data and provides all the data you need to determine which particular combination and factor had the best performance. While this type of MVT provides better results, it may take longer to complete. Partial or fractional factorial testing By using advanced statistical models, a partial or fractional factorial test runs a subset of every possible combination and only a fraction of these combinations are exposed to website traffic. The conversion rate for the combinations that are not exposed is driven from the ones included in tests based on some assumptions and mathematical models. Although this method requires significantly less traffic, it is inherently based on assumptions and one can argue that hard data is always a better alternative than inference. Taguchi testing Taguchi testing is a particular type of fractional factorial testing. The statistical methods developed by Genichi Taguchi were originally used in the manufacturing industry, where specific assumptions were made to decrease the time spent on quality assurance processes and the number of combinations needing to be tested. This method enables marketers to test specific combinations against one another, compare the results of each of them and basically determine which combination performs best based on comparison. Taguchi-based tools typically draw conclusions from low traffic, as they are able to extrapolate results. Therefore, they usually seem a good choice for low-traffic websites. The Taguchi Method has been criticised by statisticians, economists and optimisation experts, debating whether it can be applied to online marketing and advertising reliably. It relies on a number of assumptions and most often there are other factors that need to be taken into consideration when running a test. By not testing every single combination, you can never be sure that you are selecting a „true‟ winner. In this case, experts recommend testing the winner against a control using A/B testing. Multivariate Testing Buyer‟s Guide Page 8 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 11. 3.2. Market trends3.2.1. Testing market goes from strength to strength Interest in conversion optimisation has intensified in the last few months and online testing vendors have been reaping the benefits. The landscape we see today was shaped by a flurry of acquisitions in recent years:  As part of its strategy to deliver a comprehensive optimisation solution for online businesses, Omniture acquired Offermatica, an on-demand A/B testing and multivariate testing company, for $65 million. Omniture was acquired by Abobe in 2009.  CMS provider Interwoven acquired Optimost for $52 million and was then acquired by Autonomy in March 2009.  Accenture acquired Memetrics and Maxamine in 2008, two companies providing testing and optimisation services.  Webtrends purchased Widemile in 2009, adding multivariate testing and site optimisation to its portfolio of customer intelligence technologies and services. Evidence from all directions suggests the testing market is going from strength to strength. While web analytics still plays an important role in improving conversion rates, a significant proportion of companies are pouring more resources into testing and planning to integrate it into their optimisation efforts. According to Econsultancy‟s Conversion Report 2010, carried out in association with RedEye, almost half of client-side respondents (48%) said they were planning to carry out A/B testing and 42% said they intended to use multivariate testing [Figure 1] in order to improve conversion rates. Only 17% said they were currently using MVT, while 44% were already doing A/B testing. Figure 1: Which of the following methods do you plan to use for improving conversion rates? Respondents: 306 Multivariate Testing Buyer‟s Guide Page 9 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 12. Despite the fact that a minority of organisations use these techniques, around half of responding organisations with experience in these methods consider A/B testing (57%) and multivariate testing (48%) to be highly valuable methods for improving conversion [Figure 2]. Figure 2: How valuable do you find the following methods for improving conversion rates? Respondents: 318 According to Tim Brown, CEO of Maxymiser, spending on testing solutions is increasing every year: “Our belief is that even today only 10 – 20% of marketers (depending on which geographic market you consider) that would generate great ROI from MVT are actively using MVT. We also see that once our customers initiate testing programmes the results are so compelling that they increase their spend significantly year-on-year and often start personalisation initiatives building on the test results. These are the two dominant trends we see to be driving spend.”What the experts say“Businesses are realising that MVT testing can no longer be considered a point solution, but rather as an integralcomponent of a marketing platform. Businesses are looking for more than content creation, delivery andoptimisation. Rather, they want an end-to-end solution that enables them to deliver personalisation acrossmultiple channels. Businesses will continue to invest in MVT capabilities, but more so to those that are part of aclosed-loop platform that enables marketers to leverage benefits across all customer touch points.” Autonomy“Foviance has worked with a number of the MVT vendors in this buyers guide and as with any data technologythere will be ones that fit your business requirements better than others. We find that once the technology issuesare solved that businesses still have challenges in achieving value from optimisation tools but that if they get itright the benefits can be enormous. Where else do you get these sorts of immediate uplifts?” John D’Arcy, Practice Director, Analytics & Insight, Foviance Multivariate Testing Buyer‟s Guide Page 10 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 13. 3.2.2. Self-service solutions give market impetus As discussed above, the online testing market is growing at a fast pace as more organisations are beginning to realise the importance of investing in good testing technology and are actively looking to develop conversion rate improvement projects. This is mainly due to the fact that they increasingly understand the benefits of prioritising the improvement of conversion rates for existing traffic rather than trying to attract additional traffic that converts less. Driven by market demand, multivariate testing vendors often focus on a self-service model with professional, premium services on the side. More companies demand tools that remove the complexity from optimisation efforts so they can focus on what to test and how to test rather than what technology and algorithms to use. According to Dr. Karl Blanks, co-founder of Conversion Rate Experts: “The main trend is that the market is growing quickly. Several years ago, split testing was an obscure niche; now it’s relatively well known. New testing platforms are appearing each year, each with its own advantages.” No-frills, low-cost or free solutions that provide competent testing often represent the best starting point for inexperienced companies that are looking to get a taste of MVT and want to build a business case for a more advanced, ongoing programme. Although these tools often lack advanced functionality and may deliver sub-par performance if not used properly, they are continuing to bolster their market position and fuel the overall growth in the use of testing. Free tools such as Google‟s Website Optimizer are considered to be good enough testing tools for the majority of marketers, especially if they are not currently carrying out any tests, are not able to allocate sufficient financial resources or need to build a business case showing that testing works before committing to a large investment. They can often be implemented relatively easily, so the involvement of the IT department is minimal. However, companies that start out testing in-house are typically looking to outsource once they realise they need more external experience to refine and optimise their tests. According to Tim Brown of Maxymiser, fully managed services will become more prevalent due to the fact that self-service models are increasingly associated with hidden costs and high resource requirements: “The most important trend occurring in this market is the flight to quality and full service. MVT is now recognised as a major strategic initiative in leading online players. Customers want to deal with focused, specialist expert staff and solutions focused on conversion optimisation. “The hidden costs and high resource needs of self-serve MVT solutions are also now becoming well-known and the vast majority of even the largest enterprises have moved or are moving to providers who offer top quality fully managed services.”What the experts say “When MVT testing was only affordable to large corporations, the testing platforms tended to compete on„number of features‟. Many of the newer solutions focus on ease of use, which has been a big obstacle until now.” Dr. Karl Blanks, Conversion Rate Experts“More companies are becoming aware of conversion rate optimisation. This is leading to increased demand forfree or easy-to-use tools (e.g. Google Website Optimizer and Visual Website Optimizer) and is helping to keepcosts down. The cost of tools has been driven down. There‟s no need to spend £x,xxx on advanced testing anymore.” Stephen Pavlovich, Director, Conversion Factory Multivariate Testing Buyer‟s Guide Page 11 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 14. What the experts say“[Choosing a vendor] depends on the level of expertise within the business and the level of access to yourbackend code. Some vendors offer a managed service, taking away the complexities involved in setting up testsand many also minimise the level of change required in your backend code for each MVT you run. These canhave a huge impact on the decision points. I would also stress that it is worth looking at the right vendor forwhere your business is and if choosing a more basic solution, you always have the opportunity to move up to oneof the more advanced (and usually more costly) platforms once you‟ve operationalised MVT within yourbusiness.” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“An emerging trend in the market is companies finding that a combination of enterprise grade technology andsupport, coupled with a full self-serve option sets them up perfectly to do both complex testing, in parallel withquick and simple testing.“The net result is a highly flexible testing culture which gives the ability to test both quickly and with complexitywithout necessarily having to work at the defined pace of your vendor. To meet the requirements of ourcustomers we provide both self-service and fully managed testing.” Tim Brown, CEO, Maxymiser3.2.3. Vendors differentiate tools with self-learning technology According to Stephen Pavlovich, Director of Conversion Factory, testing tools are becoming more differentiated as vendors target either companies with basic testing requirements or those with more advanced requirements: “We’re seeing more of a separation between the simple and advanced tools. More tools are targeting either end of this spectrum, rather than the whole market.” One of the key areas vendors are trying to differentiate in is the use of predictive algorithms and self-learning technologies, aiming to maximise conversions and revenue beyond the limitations of manually configured tests. These algorithms continuously analyse each test‟s performance to produce the best possible combinations, uncovering opportunities over time and adjusting variables accordingly. As all adjustments are made based on all tests, not just a specific selection, self-learning solutions enable organisations to adopt a big picture approach. They also benefit from full visibility and control; whenever last-minute changes occur in terms of business strategy and opportunities, companies are able to override the self-learning technology and adjust any testing elements manually. Self-learning technologies are often described as closed feedback loops, as they continuously identify combinations that are performing better than the control and thus begin converging towards the best solution. However, these platforms need some „learning time‟ to produce effective results. Data is increasingly considered a crucial asset in the online testing ecosystem and one of the essential components of any testing strategy. According to Autonomy, an increasing number of companies are planning to use data in order to perform more informed tests that can drive significant changes: “It is no longer about finding one winner but rather understanding what makes content a winner – how and why does content such as messaging, price, or creative improve conversion rates. To extract the most value from MVT testing, businesses are looking to better mine their data to drive faster, more informed testing.” Multivariate Testing Buyer‟s Guide Page 12 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 15. Segmentation and targeting are gradually becoming a natural addition to any MVT solution, helping companies to differentiate between different categories of website visitors and serve each of these groups with the appropriate variation based on needs and relevance. They also enable them to identify if certain segments of visitors convert better than others, run separate tests for new and returning visitors and measure performance variations across multiple categories and metrics. As the demand for actionable data that help companies discover new audiences, optimise performance and uncover unique insights into customer behaviour increases, more and more vendors will focus on adding these features to their platforms.What the experts say“An important trend is the tight integration of targeting with testing. Different offers have different conversionrates depending on visitor segment, so it is imperative to find what works from whom (through testing) and thenimplement that logic on your website permanently (through targeting).” Paras Chopra, Founder, Wingify (Visual Website Optimizer)3.2.4. Companies need skill sets to complement technology Any successful testing programme requires at least four skill sets which can be rarely found in a department or even company – a project manager with a good knowledge of both marketing and statistics, a copywriter, an IT consultant and an analyst. A knowledgeable, experienced human interface is often the key differentiator between mediocre results and great performance lifts, as Stephen Pavlovich of Conversion Factory notes: “The MVT tool won’t increase your conversion rate on its own. An understanding of conversion is infinitely more important.” Therefore, relying on technology alone can have disastrous results for both the company‟s bottom line and overall morale. To a certain extent, testing is a part of every organisation‟s marketing strategy. However, only the largest companies have dedicated testing and optimisation teams. For the rest, this responsibility usually lies with the web analytics or even IT departments. While large enterprises have started to set up their own, fully staffed testing departments, companies at the other end of the spectrum tend to have just one person responsible for the entire process. Mid-sized companies usually consider the testing process too tedious and resource- intensive and find it difficult to organise a smoothly running team and allocate resource (often all members of the team are working to full capacity on other projects). Another challenge mentioned by an increasing number of organisations is the involvement of the IT department. However, as MVT vendors are focusing on easy-to-use, visual solutions, this will become less of a concern. The concept of the „get-in-line‟ IT department is gradually becoming a thing of the past as technical knowledge is no longer necessary and tests can be set up in a few minutes at the most. While technology is able to drive positive results without any human involvement, automation is not a silver bullet and can only be used to a certain extent. Therefore, having access to a dedicated, experienced account manager and support staff who can explain processes, providing advice on how to best design and carry out tests and dealing with any issues is the main reason companies choose a full-service solution over a self-service one. Multivariate Testing Buyer‟s Guide Page 13 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 16. As multivariate testing technology is still evolving and not all solutions are the same, the need to complement the technology with a competent human interface is becoming a critical element of any testing initiative.What the clients say“Having worked with a number of MVT vendors, from free to the higher end spectrum, the key is knowing theroles required to support MVT and having these in place. Typical roles include that of a designer, developer andsomeone to oversee the MVT programme. If you‟re completely new to this, consider a managed service to get youup and running whilst you work out whether it is more cost effective to run this in-house and the roles you needfor your company.“Getting this right is the most important part of making MVT work. It is not a set once and forget machine whichchurns out improvements, it needs someone that understands your business, working with designers and codersthat can make this a reality. Assuming MVT then takes off it will become the foundation of a lot of your siteimprovement projects and will therefore need a sufficient level of internal support” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco3.2.5. Enlightened marketers cultivate culture of on-going testing To fully benefit from MVT and execute a successful strategy, testing needs to become a part of every company‟s marketing culture, not just a one-off experiment. Audiences are constantly shifting and their behaviour is changing every day, so testing should be an ongoing process, part of a complete optimisation cycle. While most organisations find it difficult to cultivate a culture of continuous testing, they are increasingly taking steps in this direction by setting aside a portion of their budgets for testing and involving more business units. According to Paras Chopra, Founder of Wingify, the company behind Visual Website Optimizer, one of the main challenges for vendors is to support these efforts by making sure their existing and potential clients understand the benefits of testing and how they can integrate it into their marketing processes: “The most important challenge for companies in this sector is to cultivate a culture of continuous testing. Even the most mature enterprises do A/B testing on and off and rarely have a dedicated resource in the organisation for testing. This results in testing taking a back seat as compared to other activities like SEO, web analytics and email marketing. “The challenge for us is to make sure companies are always testing something or the other by making sure the products get integrated in the client’s workflow and by helping them understand the benefits of continuous testing.” Although multivariate testing has gone mainstream in the last few months, the majority of marketers fail to embrace testing as a regular business practice and optimise the whole range of marketing processes and activities. However, as the need for smooth integration with internal systems increases, testing will become more embedded in every company‟s culture and used to drive decisions across multiple business functions. Multivariate Testing Buyer‟s Guide Page 14 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 17. 3.3. Return on investment The potential return on investment and benefits of multivariate testing technology can be summarised as follows:  Save time and benefit from more robust results than A/B testing. – Depending on the MVT solution you use, your traffic volume and the number of elements you are testing, you may save a lot of time compared to A/B testing. – Although more sophisticated software and analysis is required and traffic requirements are high, insights generated through the use of MVT are more robust and actionable than those generated by a few A/B tests. – MVT solutions provide insight into what worked and what didn‟t, helping you to examine customer preferences and adjust your product and business strategy accordingly.  Increase sales by helping to drive the right user actions. – Optimised pages can result in a significant increase in conversion rates and average order value, having the potential to yield great gains. – Higher quality customer engagement on websites can also drive increased up-sell and cross-sell opportunities.  Quickly react to changes in customer demand and inventory availability.  Directly measure increases in revenue. – MVT solutions allow to you track conversion rates before and after each test, helping you to determine the exact impact on your bottom line. – This also gives you an indication of how much more you can generate through additional tests.  Decrease customer dissatisfaction and churn. – Customers are less likely to leave the website if specific elements of their journey are optimised. – Even limited testing can eliminate a significant amount of problems, leading to considerable revenue growth.  Avoid lost revenue. – If some changes are implemented without a structured, scientific approach, they will negatively impact sales.  Find the individual variable that causes the lift or decrease in your site performance. – Compared to A/B testing, a multivariate test provides the insight necessary to determine the single most significant factor that affects the performance of any page and thus the ability to make informed decisions. Multivariate Testing Buyer‟s Guide Page 15 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 18. 4. Strengths, Weaknesses, Opportunities and Threats (SWOT) This section contains a „SWOT’ analysis of the multivariate testing market from the perspective of vendors. It also serves as a summary of some of the points made above.4.1. Strengths  Growing awareness of conversion rate optimisation and interest in testing. – In contrast to the weak sales of traditional retailers in 2010, e-commerce sales reached more than $200 billion in the US 1 and around £60 billion in the UK 2. This is a clear indicator of how strong consumer confidence is in the online retail channel and the extent of benefits companies can reap. – More organisations are starting conversion rate optimisation programmes hoping to ramp up their revenue and differentiate in over-crowded markets. The majority of companies now understand the benefits of online testing and its contribution to the bottom line, sometimes embedding it into their overall business strategy. – However, this increased demand of MVT solutions may also have an impact on pricing, driving down the cost of testing solutions and leading to a proliferation of free tools.  Technology makes testing easier. – As multivariate testing moves further upstream, marketers demand simple, easy to understand solutions with enough features to get them started. Vendors have started to respond to this growing demand by developing MVT tools that are more user-friendly and have visual, intuitive interfaces. – This simple approach to how tests are designed and monitored also translates into increased uptake, as testing is no longer viewed as a complex, tedious process. While solutions are getting more sophisticated in terms of capabilities, they are also making it easier to set up a test by using point-and-click features and integrating advanced functions to minimise or even eliminate involvement of the IT team.  Companies are more geared up internally to initiate testing programmes. – As online testing is becoming a more common practice and building a business case for a comprehensive optimisation programme is no longer a major challenge, companies are investing more resources to make sure they have the infrastructure and technical capabilities to use MVT. – As senior management buy-in is becoming less of an issue and measurement tools are increasingly sophisticated, MVT is finally attracting significant budget.  Vendors and companies alike are becoming more experienced and knowledgeable. – As more brands are diving into online testing and thus providing good case studies and best practice examples, both companies and vendors are becoming more experienced and knowledgeable. 1According to comScore, the value of US e-commerce sales in 2010 was $228 billion (February 2011). 2According to the IMRG Capgemini e-Retail Sales Index published in January 2011, £58.5 billion were spent online in the UK in 2010. Multivariate Testing Buyer‟s Guide Page 16 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 19. What the experts say“I believe this sector will move from strength to strength. There are very few solutions available to onlinemarketers that deliver absolute clarity that all developments to the website are delivering proven lifts inconversion rates and strong insight as to why.“The marketing ROIs are transparent and huge and once a „test and improve‟ culture is embedded within anonline marketing group it is permanent. More and more PPC and SEO spend will transition to this sector with anincreased ROI.” Tim Brown, CEO, Maxymiser4.2. Weaknesses  Companies are slow in building a testing culture. – Some organisations still think of testing as a one-off experiment and a tactic that can boost their revenue quickly, instead of integrating it into their marketing processes and overall business strategy. The majority fail to embed this at every level of the company and tend to keep it contained in a single department/team.  Many solutions still lack integration capabilities. – There is an increasing demand for more advanced integration capabilities that can leverage the data generated via other channels. However, most vendors are still not focusing enough on multichannel optimisation programmes and are thus missing out on a tremendous opportunity.  Technology can only do so much. – As in other areas of digital marketing, the majority of companies rely too much on technology without investing enough financial resources to hire specialists and build a cohesive team dedicated to testing. – This is one of the first mistakes that inexperienced companies make, but they soon realise that human insight is still required to accurately interpret results.  Bad advice and self-proclaimed experts reduce the credibility of the industry. – Although there are more expert practitioners in the industry, there are also many self- proclaimed experts that have just started to dip their toes into testing and provide bad advice. This not only leads to erosion of client buy-in, but also has a negative impact on trust in platforms‟ capabilities.  High traffic requirements to get significant results. – Many smaller sites don‟t have enough traffic to warrant multivariate testing and obtain meaningful results. Testing not only requires a significant amount of resources, but also time and traffic to make sure that you choose the best performing combination.What the experts say“Companies regularly have to link test data to their internal backend data, as not all conversions can be trackedonline. This can still be a major challenge, and is likely to remain so until all backends are built with tracking inmind.“A new challenge is the increasing amounts of bad conversion rate optimisation advice. More so than before,we‟re seeing companies getting bad advice, implementing tests incorrectly, stopping tests too early, and thenmistakenly promoting the „winning‟ variation - often, ironically, lowering their conversion rate in the process.” Stephen Pavlovich, Director, Conversion Factory Multivariate Testing Buyer‟s Guide Page 17 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 20. What the experts say“One of the largest weaknesses among almost all of the players is an inability to drive intelligent testing. Thesevendors focus more on usability features and less so on ensuring businesses are testing smart and faster to driveROI. It‟s more than usability. It‟s about leveraging the meaning of your customer data in real time to driveintelligent testing as part of a multichannel revenue optimisation programme.” Autonomy“The technology is only as smart as the information it can be given. To use it to its full potential, it needs to beintegrated with other systems and across site, so that this can be fed into the decision-making layer. This cantake some leg work.“But once you start a proper programme of testing and optimisation, though, the potential is almost endless,assuming you have decent traffic volumes.” Sandra White, Head of Optimisation, The Financial Times4.3. Opportunities  Multichannel testing and integration support. – While multivariate testing remains an emerging area for most marketers, many companies are looking to test across multiple channels and integrate their testing solutions with other systems such as web content management, web analytics and CRM. – Vendors need to work with each client to help them identify where testing sits in their overall marketing ecosystem and determine the extent of integration they need (either with native tools or third parties).  Segmentation and targeting. – The ability to differentiate between new and returning visitors, serve different variations to all categories of website visitors and identify the segments that convert better than others is an increasingly sought after feature, so vendors have the opportunity to gain a larger market share by focusing on this area.  Ease of use. – Many marketers are still pointing out that some of the current MVT solutions require them to get the IT department involved. Vendors need to take into consideration all types of users that may interact with the platform and design the user interface – especially the reporting and analysis modules – accordingly. – Particularly, information distribution features represent an essential component that vendors need to consider, as the majority of companies are looking for an easy way to get multiple departments involved and disseminate information throughout the entire organisation.  Market ripe for growth. – Although the fact that adoption has been slow is considered a weakness by many, it also means that the sector is ripe for growth. The online testing market hasn‟t reached the maturity stage yet and MVT providers are able to ramp up their revenue by helping organisations understand the benefits of testing and getting them on board. – Many client-side companies are building their own in-house testing teams, thus creating demand for specialist support and training that vendors can easily tap into. Multivariate Testing Buyer‟s Guide Page 18 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 21. What the experts say“Most astute MVT buyers are already buying into not just the capability to MVT test, but also a strategic roadmapthat builds on the structural and design improvements delivered by the testing to provide targeted personalexperiences for each customer and prospect to further increase revenues and effectiveness of cross-sell and up-sell campaigns.” Tim Brown, CEO, Maxymiser“Segmentation and targeting remain a very key part of testing. The challenge facing marketers with theseopportunities is how they can understand their customer data better to accurately segment their customers andtarget them with a personalised experience. Customers need a solution that allows them to intelligentlyunderstand where the most actionable segments are that are critical to revenue generation.” Autonomy“The biggest opportunity of growth is testing for mobile sites because the challenges for optimizing mobile sitesare very different from their desktop equivalent. On mobiles you have small screen-sizes, tap, gestures and otherunnatural forms of interaction. Tighter integration of behavioural targeting with A/B and MVT is another bigopportunity in the market.” Paras Chopra, Founder, Wingify“The biggest opportunity is for simple testing platforms that integrate with the most common CMSs. In the next12 months, we should expect CMSs like WordPress and Magento to have more testing extensions, which willopen up conversion rate optimisation to a significantly larger audience.“Testing is also going to become more advanced, as conversion rate optimisers start demanding increasedfunctionality from their tools. One area I‟m keen to see developed is multi-channel testing: testing across ad,landing page, email and phone.” Stephen Pavlovich, Director, Conversion Factory4.4. Threats  Free tools may have a negative impact on the uptake of paid-for solutions. – If free tools become more sophisticated and start offering a range of advanced capabilities, this is a potential threat for paid-for vendors focusing on fully managed services. However, this is an unlikely scenario because free tools are usually implemented by inexperienced buyers that have just started to carry out testing. After an initial experimentation phase, companies often move to full-service solutions that are more scalable and are able to match more complex requirements.  Lack of compatibility with internal systems / platforms. – Some vendors tend to over-promise by implying that there is no need for IT involvement and that they can facilitate pain-free integration. Companies that choose point solutions for financial reasons without having a long-term strategy quickly find out that they are not compatible with the organisation‟s internal systems. This has a negative impact on senior management buy-in and will damage the credibility of the industry. Multivariate Testing Buyer‟s Guide Page 19 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 22. What the experts say“Over promising by vendors regarding their deployment technologies. If you believe the vendor speak everyonehas a „one-time‟ integration and everyone implies zero IT is needed post deployment. This is simply not true andleads to slow testing progress and high Total Cost of Ownership issues which discredits the whole sector. Buyers -do your due diligence!” Tim Brown, CEO, Maxymiser“There will be a clear divergence between low end, throw away tools and enterprise class testing solutions. We‟reseeing this now – marketers looking to significantly impact revenue will want testing solutions.” Autonomy Multivariate Testing Buyer‟s Guide Page 20 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 23. 5. Costs and Pricing Models There are different costs that you need to take into consideration before carrying out any multivariate tests. Internal costs such as effort, time, potential expense associated with more complex tests and creative (whether you have in-house resources to deal with this or you need to outsource) can be easily overlooked when you‟ve just started to get your feet wet. One of the first steps is to decide what type of service model is most appropriate for your goals and budget. As Paras Chopra of Wingify puts it: “Always double check if you want to take control of the MVT solution you are purchasing or you want to get dependent on a third party to get even a simple test done.” When choosing a self-service model, you need to carefully examine if the potential benefits outweigh the costs. The cost of creating and running tests internally is sometimes overlooked and challenges associated with setting up and managing a team can come as a total surprise. While a self-service model seems to currently be the best option, you may realise along the way that you can‟t keep up with the increasingly challenging business requirements and market conditions, and that you need specialist help. In the case of a fully-managed service, you are not only paying for someone to create and run tests, but also for strategic knowledge and expertise. The consulting component of full service models means that the vendor can help steer your direction in an advisory capacity. When shortlisting vendors, you need to make sure you are aware of all the cost breakdowns and what the percentage allocated to consulting is. As far as costs are concerned, multivariate testing tools can be classified into the following three high-level categories:  Free tools  Low-cost solutions that allow you to set up tests „visually‟  Powerful enterprise-grade solutions Although prices for MVT software and services have been gradually increasing in the last few years, increased competition in the sector means that they have remained fairly stable. The most commonly used method of charging is a subscription model, based on either a flat licence fee or server calls. Price ranges vary greatly, depending on the vendor‟s background and expertise, which often come at a premium. Performance-based pricing is sometimes used, but implementation of this method is still rare and doesn‟t seem to be a popular option among the vendors profiled in this guide. Although performance lifts are directly measurable, it is sometimes difficult to prove that these improvements were driven by the MVT platform alone. However, some vendors will agree to use this charging method if it is specifically requested by their clients. Multivariate Testing Buyer‟s Guide Page 21 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 24. 6. Tips and Pitfalls6.1. Introduction In order to identify the right multivariate testing technology and get the best fit for your organisation, you need to understand that different providers offer different types of solutions and their technology and expertise varies a lot. The detailed company profiles and market positioning charts included in this guide will help you find the right fit. This section contains some key questions and considerations that you need to keep in mind – and some pitfalls to avoid.6.2. How to find the right MVT technology This section acts as a checklist, but should also draw your attention to some pitfalls to avoid. Before you start looking for a suitable MVT vendor, you need to have a very clear testing plan, including information related to what types of tests you intend to run, available in-house resources and expected results. As Emilija Vilkyte, Head of Digital Revenue Optimisation at The Telegraph notes: “Before choosing an MVT vendor you need to be very clear about the type of tests you are looking to run, how often, how much internal resource you have got and how quickly you expect to see the results, as there are a lot of different products and propositions in the market. Also be clear where the tests will run (same domain vs. subdomains and third parties).” Get an initial feel for what the multivariate testing market has to offer by identifying potential suppliers and spending sufficient time to understand their products. Another initial step is to decide what type of service model is most appropriate for your goals and budget.  A full-service solution may be suitable if you don‟t want to invest in additional resources and you want someone else to manage the entire process for you. In this scenario, you pay an ongoing fee rather than incur a one-off cost. Make sure that you review all charging structures and choose the most suitable one for your needs and available budget. You may want to consider investing in a full-service solution if any of the following is true about your organisation: – Your conversion process or sales cycle is extremely complex, making it difficult to identify which components of the sales funnel are worth optimising. – Your website visitors come from multiple sources, such as email, social media, offline campaigns and you want to capitalise on this data in your tests or segment them by traffic source. – You want to use advanced segmentation and targeting techniques to serve different combinations to your visitors. – You want to test multiple elements on the same page and avoid technical complications and overburdening your IT department. – You use an advanced analytics solution and want to integrate a testing package.  A self-service solution is a good choice if you feel comfortable enough to operate the MVT technology yourself. If you have a team in place with good knowledge of multivariate testing and want to run the tests in-house, this is probably more suitable for you. While a self-service model can seem a cheaper alternative in the short-term, you may soon come to the conclusion Multivariate Testing Buyer‟s Guide Page 22 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 25. that associated costs far exceed what you expected and you don‟t have enough experienced people to manage the entire testing cycle internally.What the experts say“If budget is an issue, I‟d strongly recommend using a free tool and getting the best conversion advice you can.“And if budget isn‟t an issue, get the best consultant possible and map out everything you‟ll need the platform todo – then choose the tool. (I see companies spending £x,xxx per month on testing platforms which they‟re usingto the same level as GWO – which is free).” Stephen Pavlovich, Director, Conversion Factory“Be honest and realistic about the broad range of skill sets and availability of in-house resources that are neededto drive a high ROI testing roadmap if you only purchase a „self-serve‟ platform. Even if the resources areavailable within the team is this the best use of them? The right turn-key full service solution offers a very lowand transparent Total Cost of Ownership and guarantees you will be able to realise an ambitious testing roadmapeven in complex web environments.” Tim Brown, CEO, Maxymiser“Start simple and don‟t expect business changing uplifts early on. The beauty of MVT is the last letter of theacronym – Testing. MVT allows you to test many ideas within a relatively short space of time, speeding up thelearning process and development time. From experience, MVT never ends. In addition, you won‟t always createimprovements – learn and improve. There is always something to improve, something to tweak, new ideas to tryout and with any business, as the business dynamics change, as do the opportunities MVT can leverage. Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“Breadth of features is very important. Make sure that the solution is not only easy to use, but offers all thefeatures that you would need. The last thing you would want is to realise that the tool does not support creating aparticular test you have in mind.” Paras Chopra, Founder, Wingify“If you‟re doing testing, don‟t forget segmentation. If you can‟t use your testing tool to segment, you should setup your web analytics to track this. Although you can find one design that converts best for everyone, this isn‟tthe optimal way to get a lift – look at which creatives most appeal to different audience segments. Morecompanies now offer you the ability to then serve these to visitors – it means you are always showing the bestconverting creative to visitors, based on historical data.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron The most important step for a buyer is to establish if the supplier not only has a very comprehensive technical solution, but also sufficient training and expertise to run multivariate tests.  Does the vendor have a strong multivariate testing background or have they just added the solution to a broad range of technologies/services without much of a history or the appropriate credentials in this area? – Is multivariate testing a core competence? How long has the company been in the testing business? – Is the vendor adequately staffed to deal with advanced projects? Ask for case studies showing evidence of improved performance and ROI and make sure the solution is scalable enough to match your complex requirements. – Do they have a track record of working with a wide range of clients? Have they run tests for organisations with a similar business model to your own? Multivariate Testing Buyer‟s Guide Page 23 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 26. – You should contact recent clients directly for references before you start working with a specific vendor. Recently departed clients can also provide a clearer picture of the vendor‟s capabilities than current clients.  How many employees are specifically dedicated to the support of MVT customers? Is there sufficient support available for the geographies you focus on? What is the average turnaround time? Paras Chopra of Wingify said: “Ideally, you would need a dedicated account manager whom you can contact any time you want to get your query sorted. So, making sure you get quality support from a knowledgeable person from the vendor’s team is very important. Do a background check on the vendor too – ask their existing customers if they are happy.”What the clients say“Think about what you want from MVT. In our case, MVT doesn‟t really cover what we do. It is more abouthaving a testing platform that allows us to optimise our site, which is an ongoing process. Having control overwhole pages is therefore preferable to being limited to „test areas‟.“I would also recommend looking at how much account management is included, and whether this covers helpwith testing strategy or any consultancy. Also, what are the workflow limits in terms of getting tests up? Cansimple tests be set up by anyone in-house with some basic technical knowledge?“Bear in mind that testing also requires design resource, which most vendors don‟t supply.” Sandra White, Head of Optimisation, The Financial Times“A few recommendations: Choose two front runners and do a proof of concept for a month or two. Be very clear what‟s expected from the solution and what you will use it for. Make sure it integrates with other platforms you use. Check tag load times from multiple geo locations. Negotiate on the price as much as possible – there are ways to save money by limiting a number of tests a month, doing some data or design work yourself, removing some feature from your account e.g. there is no point in going straight to personalisation if you site usability is fundamentally wrong. Make sure the solution is flexible in case your business strategy changes.” Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph Some additional questions to ask vendors are:  Would you be able to access real-time reports? Are reports customisable to match your requirements and internal policies? You should be able to view a sample report to become familiar with the typical statistics and numbers included and to see if you find the format/layout usable.  What are their annual downtimes and what backup methods do they use? How is the data related to your tests stored? Who has access to this data?  How smooth is the integration of their software with your systems? Would it have any impact on your website loading times? The best way to find the right MVT vendor that suits your needs and budget is to evaluate several providers by arranging a pre-contract trial of all systems and deploying a proof of concept (POC) test. A significant amount of work is involved for both you and the vendors, so don‟t embark on this route if you are not serious about hiring one of them at the end of the trial. Multivariate Testing Buyer‟s Guide Page 24 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 27.  Narrow the list of suppliers – select up to five technologies by taking into account the key considerations outlined above and then choose two or three „front runners‟.  Put together at least two tests by defining meaningful and measurable goals and including multiple elements/pages. A good mix of simple and complex tests can help you identify the technology that best handles fundamentals as well as complex requirements.  Create a level-playing field by ensuring that all suppliers receive the same specifications and instructions.  Allocate sufficient time and resources to better evaluate the performance of each vendor – remember that the decision you make has a tremendous impact on your bottom line.  Evaluate each MVT provider against clear criteria, such as performance, technology, recommendations and service.What the clients say“When the salmon return from the sea to the river, they stop feeding until they spawn. If they‟re not interested infeeding, how do people catch them? It may take all day, with a salmon sitting behind a rock, to find the right lureand the perfect place to drop it. You might catch a fish the first time or spend all day with nothing to take home.You‟ll try endless combinations of lure, bait and placement to get any reaction and just when you are ready togive up – bang – a flash of line tells you that you‟ve finally got some attention.“MVT is like having a thousand lines in the water, to find the lure that will catch the most fish. You‟re welcome topatiently try everything in your repertoire but those that experiment, they get their fish quicker and more easilythan everyone else. If you have less budget, a competitive market or both – make sure you have the edge whenyou‟re out fishing.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron“Whilst MVT vendors look to features as a way of differentiating themselves, and what makes or breaks an MVTtool is the interface you use to construct tests. If it‟s painful, or limited, you‟ll find yourselves not performing asmany tests as you should. If possible, try to arrange a trial of the system, and try to put together both simple andcomplicated tests. Multipage and multi-element tests are normally a good place to start.“If ultimately you‟re looking to pass MVT responsibilities to your content team, then make sure that the tool hasa good test setup wizard or otherwise easy-to-use test construction.“If you‟re able, conduct tests that optimise on a monetary figure, such as revenue or, even better, profit. That wayyou can make an easier judgement on how much the tool is worth to you. MVT isn‟t a magic bullet, it will requirea lot of effort to get results. Don‟t budget on how much extra revenue you could theoretically make, remember tobe pragmatic.” Matthew Curry, Head of Ecommerce, Lovehoney “After hearing about six pitches we have chosen our two „front runners‟ to run a POC and measured them onperformance, technology, recommendations, future road map, time it took from brief to launch... giving differentweighting to each criteria. This made it really easy to choose a vendor at the end. Do not get „hooked‟ on upliftonly during the POC – a lot of partners are very flexible and this will improve as they get to know you sitestructure.“You need to ensure that you check tag load times from your key audience locations and not only the UK. Makesure your solution can be easily integrated with your analytics and ad server at least.” Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph Multivariate Testing Buyer‟s Guide Page 25 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 28. 6.3. How to manage internal expectations and requirements Companies that set unrealistic goals and believe that a single test can revolutionise the way their website works and significantly boost their conversion rates are prone to failure from the beginning. To prevent this, start with a test roadmap that can help you set the right internal expectations and focus on achievable goals, but also ensure that the relationship with the MVT vendor is smooth and painless. As John D‟Arcy of Foviance notes: “A big challenge is to deliver a set of tests that will impact your business immediately and give you meaningful learning. To do that you need to take a step back and deliver a prioritised roadmap of tests before you start out. Understanding the potential impact of a test can be analysed with web analytics data but you get much more actionable information by bringing in learning from your user experience tests, your voice of the customer tools and from competitor benchmarks. Try prioritising your tests based on these data sources and you deliver lots more value. We find that clients who are able to do this coherently are able to deliver a test roadmap that provides both immediate commercial potential and long term customer experience impact.” The amount of work that is required internally when using multivariate or similar technology varies a lot, broadly depending on the number and complexity of the tests you are running, but also on the type of service model you choose (full-service versus self-service solution). Before choosing a vendor, you need to identify exactly what services and expertise you need, as this will greatly influence internal requirements. Seamless collaboration across the entire organisation is an essential prerequisite, as all departments need to get involved in this process.What the clients say“The key for any business is to involve the right stakeholders. MVT can make those less involved feel like they‟relosing control. Instead highlight the reduction in guesswork, in wasted development time trialling ideas andmoney saved in sifting out the bad ideas from the good as quickly as possible. Keep the key stakeholders engagedand involved and the whole process becomes easier.” Depesh Mandalia, Senior Marketing Manager - Personalisation & MVT, Tesco“The amount of technical work required internally is very much dependent on the underlying technology of themultivariate testing tool you use. For a content proxy tool, you‟ll need to make changes to your DNS, so you‟llhave to be careful that it‟s not disruptive. You may then need to add additional parameters to your URLs forgoals that can‟t be tracked by other means. Other than that initial configuration, further tests shouldn‟t requireany overheads.“For JavaScript based tools, you have two options. Tools either require a combination of JavaScript tags andareas of your site „cordoning off‟ for testing using HTML tags, which if you aren‟t in direct control of your sourcecan be a constant pain. Other tools require a single JavaScript tag placed on any page to be included in the test –so insertion into a header file makes sense.“For any tool you use, even going through a proxy, you may be affecting site speed, so work close with yourtechnical department to monitor the impact your testing has. You should also, as a matter of course, work withyour content and design departments when constructing any tests to make sure that you aren‟t drasticallyaltering the look and message of the site (unless that‟s the point of the test!).” Matthew Curry, Head of Ecommerce, Lovehoney Multivariate Testing Buyer‟s Guide Page 26 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 29. What the clients say“Using MVT can be used to drive cultural change within a company. You‟ll have less meetings arguing with idiotsabout what should be agreed and you‟ll find out what stuff works (or sucks) with customers. It frees you from theyoke of a single design process and lets you innovate, test and learn. Don‟t ignore the cultural impact of this kindof testing as it makes staff more productive and driven by evidence rather than their opinions.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron“With regards to tests – as long as you don‟t change your mind and test spec in the middle of process everythingwill be fine. Document all possible user journeys and scenarios – this will help you understand main problemareas. Be very clear „where you are‟ and „where you want to be‟ when providing instructions to your MVT partner.Create a test roadmap to manage internal expectations. Make sure you test variants before test goes live on allbrowsers and platforms.” Emilija Vilkyte, Head of Digital Revenue Optimisation, The Telegraph6.4. A few tips to avoid frustration and failure Before embarking on the first multivariate test, you need to take a tactical and strategic approach – rather than simply relying on a specific technology, you need to begin with a very clear testing strategy and individual test plans.  Some initial key questions to ask before running multivariate tests are: – Does your site have enough traffic to yield significant results? – Do you know enough about the page to determine the most critical areas that need to be tested? – Are you able to set up a controlled test?  Identify the goals of your website and a real problem you need to solve. – An important step is to decide what success metrics you use for your website based on your overall business goals e.g. customer satisfaction, lead generation, revenue, etc. – This will help you identify the problem you want to solve, decide what the main goals for your MVT tests are and choose the most suitable testing areas. – Goals should be measurable and quantifiable. What is the degree of change you aim to achieve? What metrics will you use to measure the impact of your test?What the experts say“Here‟s a list of other common mistakes to avoid: Common Mistake 1: Launching a re-design of key pages without monitoring the effect on conversions. Common Mistake 2: Using guesswork to decide which pages to focus your efforts on. Instead, you should be using analytics. Common Mistake 3: Assuming you‟ll get the best results by testing millions of variations of your pages. Common Mistake 4: Designing tests that don‟t teach you anything. Common Mistake 5: Designing tests that have the wrong goals – for example ignoring offline orders or measuring click-throughs. For most businesses, the main goal is to maximise the profit-per-visitor.” Dr. Karl Blanks, Conversion Rate Experts Multivariate Testing Buyer‟s Guide Page 27 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 30. What the experts say“Think ahead. You are starting a journey. Once the basic navigation, journeys, landing pages, checkout processesare optimised you will want to get at the even higher ROIs and insight available through automated segmenttargeting, behavioural targeting and product recommendations. Are these all available from your MVT vendorvia the same one line of code already on your site? Do they share the same reporting and data infrastructures orare they a superficially „integrated‟ set of discrete acquired products? Don‟t just believe some fancy slide-ware –see a demo of the UIs and get references from clients who have called the vendor‟s bluff!” Tim Brown, CEO, Maxymiser  Make a list of hypotheses for the problem you identified in the previous step. – Leverage web analytics data, usability testing, best practice, experience and your own intuition to come up with a list of potential reasons for the problem you‟re trying to solve. Although these seem nothing more than an educated guess, they represent a great starting point in helping you decide how you can address these issues and what elements you want to test.  Choose testing areas carefully. – One of the main challenges of an MVT project is to know where to start. Testing experiments often become political battles within a company, each department having a vested interest in a different step of the customer‟s journey. – Too often, organisations solely focus on optimising the home page and are disappointed when test results fail to identify a single factor that can lift its performance. If you are just starting with multivariate testing, choose several test areas that are less visible but still have a significant impact on the overall user experience. It has been argued that landing pages and forms represent the best starting points, as they enable you to „fail faster‟ and learn more about the process in a short period of time. – Once you have an outline of all testing areas and possible combinations, make sure you preview all combinations to remove those with incompatible elements.  Become familiar with the basics before running sophisticated tests. – Many marketers embark blindly on complex MVT projects without fully understanding the mechanics behind basic tests. The best solution would be to start with simple A/B tests to narrow the number of possible combinations and deal with „low-hanging fruit‟ first.  Start with a single, specific and easy-to-measure goal and choose a page with a poor conversion rate.  Resist the temptation to end the test early, allowing enough conversions to be tested in a reasonable amount of time. If you stop running a test too early, you may end up using a version that is not the best performing one. However, if you stop it too late you risk wasting time that could be spent on other tests. – After you are comfortable with these tests, you can move to more complex options with multiple variables.  Estimate the traffic needed to get significant results. Take into consideration the elements that affect the time it takes to obtain significant results. The most common variables that impact testing time are: – Number of combinations tested (direct relationship) – Volume of traffic on the page (indirect relationship) – Conversion rate of the page being tested (indirect relationship) – Expected performance boost (direct relationship) Multivariate Testing Buyer‟s Guide Page 28 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 31. Useful tipRemember that the more combinations you include in the test, the more traffic you‟ll need to get meaningfulresults.Google‟s Website Optimizer Duration Calculator is a useful tool that can help you estimate the potential durationof your test.  Review the entire customer journey, not just one step. – You need to optimise at each stage of customer funnel in order to increased likelihood of purchase. No matter how much you optimise one page, the improvement in the overall customer experience may be hardly noticed.  Test the waters with a small traffic segment. – For example, run an initial test with only 5% of your traffic. Once you have a clear winning combination run a second one based on this initial insight, with a larger traffic segment. – If you have a large revenue-generating site, testing an underperforming combination with all your visitors can have a tremendous negative impact on your bottom line.What the clients say“When first approaching MVT, document. It‟s easy to get carried away and launch multiple tests, but when youcome back the results in six months, or a year‟s time, they won‟t mean a great deal. I‟ve found a simple TestSpecification Document, that records the purpose of the test, the variations within, the success criteria, andfollow-up ideas, really helps me put together a comprehensive plan for testing each element of the site.“There‟s a lot of MVT press releases about how companies exponentially increased their revenue through a singlesmall change. This doesn‟t often happen, so don‟t expect to recreate those successes, especially if you‟ve alreadyfollowed a lot of best practice. Instead, look to transformational testing: your message, your layout, photography,checkout. Change big things first, then when you think you‟ve got those all working, you can concentrate onsmaller changes.” Matthew Curry, Head of Ecommerce, Lovehoney“If you‟re doing testing, don‟t try to test too many variables, especially if your traffic isn‟t huge to your site. Forexample, does testing a different shade of blue on your button work? You‟re much more likely to understandwhat resonates with customers if you are bolder and try new things. Avoid testing subtleties or small changesthat will eat into your testing time. There isn‟t much point of taking eight months to test your homepage, asduring that time you‟ve not had the optimal test. Like Agile, do your testing in smaller batches and learn from theresults to drive your next test.” Craig Sullivan, Customer Experience Manager (eBusiness Group), Belron  Understand the factors that influence the reliability of test results. – The larger the volume of traffic on the page you‟re testing, the more reliable the results. – Pages with low conversion rates take a much longer period to generate statistically significant results. – Testing combinations that are very different and vary a lot in terms of performance is more reliable than using very similar ones.  Don’t panic if test variations perform worse than the control. – Multivariate testing is a continuous learning methodology and a test that returns negative results is definitely an opportunity to learn what doesn‟t work. It also enables you to identify what you need to change in your future tests and the direction you need to follow. – An important issue that you need to take into consideration is that sometimes a single variable can perform extremely poorly compared to the other ones and thus have a negative impact on all test variations. In this case, you need to stop the test, remove the poorly performing variable and start another test. Multivariate Testing Buyer‟s Guide Page 29 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 32.  Be aware of MVT’s limitations and use it wisely. – Always remember that multivariate testing is a complex process and sometimes it takes a lot of time to gain sufficient data to derive a meaningful conclusion. The more variations you include in the test, the more traffic and time you will need to complete it. – Do not just replicate the suggestions that worked for other websites and test a lot of elements just for the sake of testing. As with any business decision you need to make, adopt a strategic, structured approach to avoid frustration and failure.  Analyse results, create a recommendations report and circulate it in the organisation.  Based on the data generated, suggest additional tests and testing timelines.What the experts say“Here‟s the process that most newcomers seem to follow:1. Evaluate and implement software2. Test some randomly chosen things (like so-called „best practices‟).3. Discover that the improvement, if there is any, is too small to be measured, so the tests never reach statistical significance.4. Realise that the hard part is knowing what to test.5. Learn about how to design high-converting pages.” Dr. Karl Blanks, Conversion Rate Experts Multivariate Testing Buyer‟s Guide Page 30 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 33. 7. Market Positioning Charts7.1. Explanation for Chart 1: Business model Full-service Where we are going Where we are now Self-service We only do Range of MVT technologies The vertical axis charts to what extent the company provides a full-service solution, as opposed to a self-service one. The horizontal axis charts the extent to which the supplier is focused purely on multivariate testing as opposed to offering a portfolio of technologies which includes multivariate testing. For example, a vendor positioned in the bottom right quadrant will offer a portfolio of technologies including multivariate testing and will be more geared towards offering a self-service solution. Multivariate Testing Buyer‟s Guide Page 31 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 34. 7.2. Explanation for Chart 2: Type of company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus The vertical axis charts the size of the company compared to its competitors. The horizontal axis charts the extent to which the vendor is geared towards a specific market, as opposed to being a more global player. For example, a vendor positioned in the bottom right quadrant will be small compared to its competitors and have a global focus. Multivariate Testing Buyer‟s Guide Page 32 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 35. 7.3. Market positioning overview: Business model Multivariate Testing Buyer‟s Guide Page 33 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 36. 7.4. Market positioning overview: Type of company Multivariate Testing Buyer‟s Guide Page 34 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 37. 8. Supplier Matrix An at-a-glance overview of who‟s doing what… Google Website Adobe Systems Visual Website HiConversion GlobalMaxer Webtrends MaxymiserSupplier matrix: a fast Interactive Autonomy Accenture Optimizer Optimizer SiteSpect Amadesa Monetate guide to services VertsterService ModelFull-service solution           Self-service solution            Hosted            Other ServicesSegment targeting (rules)            Recommendation engine      (automated model)Behavioural targeting          (automated model)Type of Testing Methodology / TechnologyA/B Testing by URL            A/B Testing using multiple             content variantsMVT – Full factorial            MVT – Fractional          factorial/Taguchi MethodAbility to cull poor content as            test proceedsEngine TypeClient-side (genericJavaScript tag across all           pages)Client-side (single JavaScript         tag per page)Client-side (code embedded    in each page)Server-side    Inspection proxy   InterfaceIs HTML knowledge needed  to define tests?Live preview of variations              Multivariate Testing Buyer‟s Guide Page 35 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 38. Google Website Adobe Systems Visual Website HiConversion GlobalMaxer Webtrends MaxymiserSupplier matrix: a fast Interactive Autonomy Accenture Optimizer Optimizer SiteSpect Amadesa Monetate guide to services VertsterTest Segmentation MethodsIs the system able to limittests to specific audience            segments?Inbound search keywords            Referring URL            Landing page URL            Query string parameter            Cookie parameter           Visitor type            Time returning           Visit Metrics (Time/duration)          On-site search keywords           By device (e.g. iPad)            IntegrationWeb analytics            CRM data           E-commerce platform           CMS           Types of ReportingAbility to report against            multiple goalsE-commerce conversion            Page goal conversion             Bounce rate          Revenue per page generation          Revenue per visitor           Gross profit / gross margin            Average order value            Custom event conversions             Are reports available online?              Multivariate Testing Buyer‟s Guide Page 36 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 39. 9. Specialist Conversion Optimisation Agencies Below is a list of agencies specialising in conversion optimisation and the implementation of MVT technology. http://www.conversionworks.co.uk/ ConversionWorks increases your website profitability and conversion rates. We find opportunities for your website to increase your ROI and deliver you better results. We’re specialists in Google Analytics, A/B and Multivariate Testing and AdWords management and optimisation. MVT case study: http://www.conversionworks.co.uk/case-study/ Contact detailsConversionWorks Tel: 0845 5000 408 Fax: 01753 539 801 Email: info@conversionworks.co.uk Address: Nash House Datchet Road Slough Berkshire SL3 7LR http://www.conversioncounts.com/ Conversion Counts offers a complete array of online marketing services with one aim: To increase online conversions rates and revenue. Conversion Counts uses a unique approach to website e-commerce development combining Web Analytics, User Testing, Pay Per Click and Search Engine Optimisation (SEO). We also publish a successful blog where we regularly post useful tips and tricks designed toConversion Counts help our customers and readers achieve higher conversion rates and boost revenue. Contact details Tel: 0208 123 9908 Address: 923 Finchley Road London NW11 7PE http://conversionfactory.com/ We specialise in conversion rate optimisation for highly competitive niches. Our proven approach to conversion rate optimisation will increase your revenue and help you dominate your market. Conversion Factory was formed by Stephen Pavlovich in early 2007. We started life as Bony Toad Online Marketing, specialising in SEO and PPC. We started experimenting with conversion rate optimisation (CRO) and, within three months, had increased one client’sConversion Factory revenue by 600%. We now focus exclusively on CRO and work with clients ranging from small ecommerce companies to large gaming sites turning over close to a billion dollars a year in revenue. We love working in highly-competitive niches. Our proven approach to CRO means we’re always happy to guarantee our results – even working on a performance basis with selected clients. Contact details http://conversionfactory.com/contact-us Multivariate Testing Buyer‟s Guide Page 37 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 40. http://www.conversion-rate-experts.com/ Conversion Rate Experts is an international consultancy with the audacious opinion that marketing should be measurably effective. We pioneered a new approach to conversion rate optimisation called CRO®. It fuses science and marketing to increase revenue for online businesses. We have applied this methodology to companies like Apple, Sony, Google, Vodafone, SEOmoz, SEO Book and some fantastic small businesses – frequently with off- the-chart results. The company was founded in 2006 by Ben Jesson (an internet marketing expert) and Dr Karl Blanks (a former rocket scientist), and now has a team of experts throughout Europe and North America.Conversion Rate Experts Contact details London, UK Office New York, US Office 271 Regent Street 410 Park Avenue London 15th Floor pmb #334 W1B 2ES New York, NY 10022 UK USA Tel: 0800 043 2650 or 0203 368 6212 Tel: (212) 231-8319 Fax: 0870 838 1135 Fax: (212) 658-9097 http://www.foviance.com/ Foviance is a customer experience consultancy to the world’s leading brands. We help clients improve their business using media optimisation, data analytics, customer insight, usability testing and research services. We believe this combination of skills and knowledge are essential to the successful delivery of MVT, Optimisation and Test & Experimentation programmes. Contact detailsFoviance London office Asia office 14 Bonhill Street 29/F Shanghai Kerry Center London 1515 Nanjing Road West, JingAn EC2A 4BX Shanghai, China Tel: +44 (0)8450 546 500 Tel: +86 21 6103 7190 Email: info@foviance.com http://www.ga-experts.com GA-Experts.com is part of Omega Digital Media Ltd – a company founded by Brian Clifton in 1997. Built on a wealth of experience in complementary services including conversion- based website design and search marketing promotion, we now focus on advanced services for Google Analytics, Urchin 7 and Urchin 6 software. We drive in-depth knowledge into your organisation enabling you to maximise the potential of your web site ROI, whether through lead generation, marketing or pure ecommerce. Contact detailsGoogle Analytics Experts UK office Sweden offices Germany office(GA-Experts) 8 The Drive Brunnsgatan 21 B Kastanienallee 19 Brighton 111 38 Stockholm 10435 Berlin East Sussex, BN3 3JT Tel: +46 (0)8 559 21 356 Tel: +49 (0)308 093 5087 Tel: +44 (0)845 123 3898 Sporthallsvägen 2b 263 34 Helsingborg Tel: +46 (0)8 559 21 356 Email: mail@ga-experts.com Multivariate Testing Buyer‟s Guide Page 38 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 41. http://www.houseofkaizen.com/ House of Kaizen is an award-winning conversion rate optimisation agency based in London. We combine split testing technology, behavioural modelling and eye flow control with advanced analytics to maximise website revenue. We use empirical data, based on actual site traffic, to validate site modifications and user journey enhancements that lead to higher conversion rates. Our team of highly trained, gimlet-eyed analysts, developers and designers spend their days dissecting, analysing, learning, implementing, split testing, learning more and implementing again to optimise your conversion rates. We’ve helped clients increase online conversionHouse of Kaizen rates by as much as 105%. Contact details Tel: +44 (0)207 9543750 Address: Old Aske Hospital Hoffman Square, 50 Buttesland Street Hoxton, London N1 6BY UK http://www.invesp.com Since 1995 our founders and team members architected, designed and deployed 22 of the largest e-commerce websites in North America. During the last two years, we have successfully completed 200+ conversion optimisation projects. We founded Invesp in 2006 with a promise of delivering more conversion. Khalid Saleh (pronounced ka-lid) is the leading e-commerce architect in North America. He was the lead for 12 of the largest ecommerce operations including Motorola, AMEX, Levelor, Citrix, WWT, Nabors, and many others. Khalid managed all aspects of ecommerce design and implementation for these companies; from the initial stages of creating the concepts, through the grueling process ofInvesp implementation, and finally to releasing and marketing these sites. He continues to consult with companies all around the world advising them on e-commerce operations. Ayat Shukairy (pronounced i-yat) is a leading conversion optimisation and usability expert with hundreds of published articles. Ayat’s client list includes companies such as Skis.com, HowStuffWorks, ReMAX, Discovery, CSRI, and PGI Connect. Ayat led over 200 global conversion optimisation projects overseeing all aspects from market analysis and person creation to full optimisation implementation. She consistently delivers double digit increases in conversion rates for her clients. Contact details Phone: 1.800.421.7393 http://www.ioninteractive.com ion interactive’s LiveBall cloud-based landing page software platform creates, tests and optimises advanced landing experiences. Landing pages, microsites and conversion paths are quickly and easily created, optimised and managed using a seamless combination of content management, testing and analysis. The platform drives results that average three times industry averages. LiveBall was developed in-house by ion and was launched in Q2 2007. LiveBall clients range from Fortune 100s to small businesses. ion was founded in a merger of a web software company and a web marketing company in 1998. The company is privately held, headquartered in Florida, and has an office inion interactive Massachusetts. Contact details Florida office Massachusetts office 124 East Boca Raton Road One Broadway, 14th Floor Boca Raton Cambridge FL 33432 MA 02142 Tel (main): 1.888.466.4332 Tel (international): +01.561.394.9484 Fax: +01.561.394.9773 Email: info@ioninteractive.com Multivariate Testing Buyer‟s Guide Page 39 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 42. http://www.nemawashi.co.uk/ Created by Colin Tate, Nemawashi Ltd is a London-based consultancy with world-wide clientele. We opened up shop in 2009, and since then we’ve worked with companies as diverse as Orange and IG Group. We specialise in value-growth via improving website conversion, and by mentoring companies on how to align and get the most value from theirNemawashi online presence. We call this Conversion Optimisation and Company Conversion. Contact details http://www.nemawashi.co.uk/contact/ http://www.prwd.co.uk Founded by Paul Rouke, PRWD is a usability and conversion rate optimisation consultancy focusing on analysing and understanding user behaviour to increase online conversion. PRWD clients include Speedo, Lakeland, Pentland, Isabella Oliver, Stella McCartney and The Scout Association, who each benefit from services including moderated and remote user testing, heuristic evaluations, customer research, conversion rate optimisation (split and multivariate testing) and in-house consultancy. With 12 years experience working in blue chip retailing, Paul brings invaluable experience of user behaviour to each and every analysis and conversion optimisation project that PRWD deliver. Paul stays at the leading edge of the conversion optimisation industry allowing him to deliver both public and in-house usability and conversion rate optimisation training for Econsultancy for brands including River Island, Mobistar, Sage, Kwik Fit Insurance, Wiatrose and VisitBritain. In addition to helping clients improve the conversion rate of their existing website, both forPRWD lead generation and e-commerce, PRWD also work closely with businesses and their agencies when embarking on replatforming and redevelopment projects to ensure they benefit from a true user-centred approach. Conversion rate optimisation doesn’t just need to start once a website is live – key usability principles, best practice and user understanding should be a fundamental aspect of any new website platform build. PRWD also publish articles containing tips, techniques and case studies on conversion rate optimisation: http://bit.ly/CROresources Contact details Tel: +44 (0)161 228 0585 Email: enquiries@prwd.co.uk Address: 22 Lever Street Manchester M1 1EA http://www.redeye.com RedEye was established in 1997. In 2005 the company bought e-relationship marketing Ltd ‘e-rm’ and began the drive towards integrating email and web analytics in order to increase the return on investment for customers. Capabilities grew and in 2006 a unique behavioural email marketing solution became available. During this time optimum.web also became part of RedEye, expanding RedEye’s services to website usability. RedEye now offers an integrated service of web analytics, email marketing and website usability focusing on analysing user behaviour to increase online conversion.Red Eye Contact details Crewe office Milton Keynes office London office Oak House 26 Shenley Pavilions 26 Grosvenor Gardens Crewe Hall Farm Chalkdell Drive Victoria Crewe Shenley Wood London Cheshire Milton Keynes SW1W 0GT CW1 5UE MK5 6LB Tel: 020 7730 9958 Tel: 01270 848 490 Tel: 01908 340 990 Fax: 020 7824 9951 Fax: 01270 848 499 Fax: 01908 340 999 Multivariate Testing Buyer‟s Guide Page 40 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 43. http://sitetuners.com/ SiteTuners was launched in 2002 to improve the technology used for website conversion rate optimisation. The result became known as the TuningEngine, around which SiteTuners developed a comprehensive portfolio of both consulting and conversion optimisation testing services. Today, more than 650 companies have improved the performance of their online marketing programmes following a conversion optimisation engagement with SiteTuners, including major brands such as Facebook, Google, eHarmony, Sanyo, Real Networks, Nestle, Symantec, Canon, McAfee, Verizon Wireless, Rhapsody, Norton, Ed Hardy and many others.SiteTuners Contact details Tel (Pacific Time): (619) 223-8020 Fax: (619) 255-1540 Email: contact@SiteTuners.com Mailing address: P.O. Box 928500 San Diego, CA 92192-8500 2535 Kettner Blvd. Office location: Suite 3A3 San Diego, CA 92101 http://en.traffic4u.nl Traffic4u is an experienced, international online marketing agency, specialising in Result Driven Online Marketing. The company began small in 1999 and has since grown into one of the most professional players in the field of search engine marketing. With a team of specialist employees in the Netherlands and Spain, we concentrate on optimising your online results. Traffic4u is an active member of diverse branch organisations such as IPAN, IAB, SEMPO and WAA. Moreover we are accredited by both Google and Yahoo! as a professional search engine marketing agency. Traffic4u is an international company: we provide our services in eight languages across all of Europe. We work in English, Dutch, Spanish, Flemish, French, German, Portuguese and Italian. Contact details Groningen office Amsterdam office Salamanca officeTraffic4u Hereweg 85/B2 Overschiestraat 63-1 Plaza de Espaňa 5-6 9725 AC Groningen 1062 XD Amsterdam 37004 Salamanca Tel: +31 (0)50 - 360 46 64 Tel: +31 (0)20 - 462 60 80 Tel: +34 923 282951 Fax: +31 (0)50 - 589 23 66 Fax: +31 (0)20 - 694 49 32 Fax: +34 923 282953 Madrid office Phone: +31 (0)20 46 26 080 Edificio Eurocentro Email: info@traffic4u.nl C/ Capitán Haya 1, 15ª planta, oficina 19-20 28020 Madrid Tel: +34 91 417 69 39 Fax: +34 91 417 69 40 Multivariate Testing Buyer‟s Guide Page 41 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 44. www.web-arts.com Web Arts AG was founded in 1996 and is one of the leading providers of conversion optimisation. Web Arts has 35 employees and services a lead/retail volume of around three billion euros. The company is based in Germany. Contact detailsWeb Arts AG Frankfurt office Hamburg office München office Seifgrundstraße 2 Stadthausbrücke 1-3 Karlstraße 35 61348 Bad Homburg 20355 Hamburg 80333 München Tel.:+49 6172 68097-0 Fax:+49 6172 68097-77 Email: info@web-arts.com http://www.webstrategy360.com Web Strategy 360 is an independent digital strategy consultancy that is managed by Kamal El Agha. We are based in Abu Dhabi, UAE but work with clients across the world. We have worked with various client types including agencies such as Invesp, Fortune Cookie and Billington Cartmell and client-side appointments such as Disney and AutoTrader.Web Strategy 360 We have experience of various web analytics tools including Google Analytics, Google Website Optimizer, Webtrends Analytics, Omniture SiteCatalyst, Omniture DataWarehouse and Omniture Discover. Contact details Tel: +971 (0)566 222 890 Email: info@webstrategy360.com http://www.widerfunnel.com/ Co-founded by Raquel Hirsch and Chris Goward, WiderFunnel Marketing Optimisation is a full-service marketing agency and the developer of industry-first Kaizen Method™ (for continuous conversion rate improvement) as well as the LIFT Model ™ (for identifying test hypotheses). At WiderFunnel, we plan, design and deliver clients with web pages and landing pages that are statistically proven to convert more visitors into leads or customers. Plus, we guarantee our work. Since 2007, WiderFunnel Marketing has been on the leading edge of developing this newWiderFunnel and fast growing area of expertise called Conversion Optimisation. Contact details Tel: +1 (604) 569-0897 Fax: +1 (604) 648-8651 Email: iwant@widerfunnel.com Address: 10th Floor, 675 West Hastings Street Vancouver, BC V6B 1N2 Canada Multivariate Testing Buyer‟s Guide Page 42 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 45. 10. Supplier Marketplace and Profiles 10.1. Accenture InteractiveAccenture Interactivewww.accenture.comCompany Proposition COMPANY OWNERSHIPAccenture is a leading management consulting company helping organisations to overcome the Publically Listed NYSEchallenges they face in as they strive for high-performance.Brief Company HistoryAccenture, initially called Arthur Andersen, was born as a management consultancy division in CLIENTS1957. In 1989 a group of partners from the Consulting division of Arthur Andersen formed a neworganisation, Andersen Consulting, focused on consulting and technology services related to Accenture works with most of the Fortune 500managing large-scale systems integration and enhancing business processes. Over the next 10 companies.years, we evolved from a systems integrator to a global management consulting and technologyservices company, providing the full range of consulting, outsourcing and related technologyservices. On January 1, 2001 we changed our name to Accenture. Accenture became a publiccompany on July 19, 2001, when it listed on the New York Stock Exchange under the symbol ACN. OFFICE LOCATIONSAccenture is headquartered in Ireland and operates under the legal name Accenture Ltd. Global Office NetworkUSP – What sets you apart from competitors?Accenture‟s Digital Optimisation (ADO) capability has four separate axes of differentiation which setit apart from competing optimisation companies and software: HEAD OFFICE ADDRESS The quality of the testing methodology and mathematics (unique Nobel Prize winning algorithms). Plantation Place The power and granularity of the software‟s personalisation (behavioural targeting) delivery. 90 Fenchurch Street The range and flexibility of deployment and integration options (open framework approach). A choice of high quality engagement models (full service to self-service) supported by a highly London qualified global network of optimisation and statistics professionals.The combination of these unique capabilities allows us to arrive at the „optimal‟ experience to drive aparticular „outcome‟ with unmatched certainty and clarity. CONTACTCase study and/or testimonials Stuart BarkerOnline Retail Accenture InteractiveGlobal test and segmentation programme for an online retailer drove significant increases in digitalmarketing performance and uncovered key market insights to drive up to 60% uplift (from baseline 0207 844 4000conversion rate). Stuart.x.barker@accenture.cKey Highlights om Optimisation of keyword management, ad group wording and cost per click management. Challenged legacy marketing assumptions around communication for each customer type. Turned the risk of all new site rollout into an opportunity to learn and optimise functional pages towards business KPIs. Stronger pages for Visitors, Sellers and Buyers.TelecomUtilised the ADO multivariate testing platform for a leading communications provider to test 64email and eight landing page variations across approximately one million unique prospects. TheMVT was designed to optimise email opens, email clicks and total received applications to determinewhat connections, if any, exist between the initial customer contact and the subsequent onlineexperience. Uncovered a 15% uplift opportunity for Email Opens and a 200% uplift opportunity forEmail Clicks when customizing for visitor segment.Key Insights Optimisation based page configurations identified for email open, email click, and confirmation outcomes. Segmentation findings were centred around the „product offering‟ and „geography‟. A seasonal Subject Line and a neutral CTA where found to be the significant factors influencing recipient behaviour. Multivariate Testing Buyer‟s Guide Page 43 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 46. Service ModelFull-service solution Yes – Accenture offers full service end-to-end delivery of MVT experiments.Self-service solution Yes – clients can choose to operate the ADO technology themselves to develop and deliver MVT experiments. ADO Enterprise is the robust, high performance solution for ongoing experimentation, and can be fully deployed in client‟s data center. An Enterprise deployment can easily be integrated with real-time CRM applications and legacy data, using client‟s existing customer identifiers and first-party cookies for tracking.Hosted Yes – ADO can be hosted by Accenture in one of our global data centres. In this model, all of the ADO components are hosted and maintained by Accenture and delivered to the client under a framework of SLAs. The standard option for experiment implementation is JavaScript, where page content is varied and data is collected via a JavaScript tag that is placed on the page(s) involved in the experiment. Clients may also choose to employ the more robust features of the server-side ADO Toolkit while still operating in a hosted model. In this deployment, the ADO Toolkit is installed on the client‟s web server, but the calls are made over the Internet to the ADO components hosted in the Accenture data centre.Other (please specify) Bespoke Service Plan – Accenture is happy to develop a bespoke service plan to represent a blend of full and self-service options. As an example, the client could set up experiments but rely on Accenture to create the creative assets for the experiment. Alternatively, client could ask Accenture to gather testing hypothesis and execute the experiment, with the client taking responsibility for analysing results. Server-side integration – The entire ADO deployment is positioned in the client data centre, behind the client firewall.Other ServicesSegment targeting Segmentation exploration capabilities allow you to discover the one or more segments that drive(rules) behavioural differences, enabling actionable, meaningful and valuable personalisation tactics.Recommendation Powerful rules management within ADO allows fine-grain control over the visitor experience.engine (automated Implement experiment rules to test content variations and uncover the optimal experience for eachmodel) segment of visitors, optimal rules to automatically manage content delivery based on experiment results, and targeting rules to deliver personalised content to each visitor profile.Behavioural targeting ADO offers a comprehensive rules management system for complete control over the user-experience.(automated model) These rules allow for dynamic behavioural targeting by delivering specific content to specific visitor segments.Other related services Accenture Digital Optimization offers a comprehensive consulting and support services catalog ranging(please specify) from training, technical services, and operational consulting to strategic consulting. Recommended optional consulting services include The development of an Optimisation Roadmap establishes the priority and sequence of experiments that will be undertaken in the future based on ease, value and urgency. Experiment-specific consulting and support services cover each stage of the experiment delivery process:  Definition phase  Build phase  Run phase  Review phase Multivariate Testing Buyer‟s Guide Page 44 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 47. Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing using Yes – The ADO software is a mature, flexible platform that enables everything from simple, light-touchmultiple content A/B testing to be delivered efficiently.variantsMVT - full factorial Yes – ADO experiment programme leverages advanced multivariate and behavioural targeting capabilities to run „Experiments‟ aimed at improving the performance of individual „Touch-points‟ by delivering the optimal content across key customer segments.MVT - Fractional Yes – ADO has an extensive design library which includes fractional factorial designs which can be usedfactorial/Taguchi to reduce production cost, or in response to platform limitations. Fractional factorial design results in aMethod subset of total number of treatments (page variations) that are available in a full factorial design and comes at the expense of exploring interactions (impact of showing an attribute‟s variation in combination with another attribute).Ability to cull poor Yes – ADO models can be set to automatically explore experiment data, then direct incoming traffic tocontent as test proceeds better performing treatments for each customer profile. After an initial learning period, incoming traffic is automatically directed to high-performing treatments. All else equal, the outcome rate (or financial value) improves over time as the model becomes more certain of the optimal treatment.Other (please specify) Interaction detection as part of full factorial experimentation analysis.Engine TypeClient-side (generic Yes – ADO JavaScript is a very quick and simple-to-deploy solution.JavaScript tag across all The experiment is set up entirely within the ADO UI using the Campaign Editor. The Campaign Editor ispages) a WYSIWYG tool that enables you to mouse over and select attributes on the page to be varied, then select one or more actions such as changing HTML, changing URLs of images, or updating CSS properties. The Campaign Editor allows you to manually select and visually confirm all of the content variations prior to deploying any code. Segment and outcome data captures are also defined in the ADO UI. Values can be static, captured from the query string, or can be defined with custom JavaScript. Special tags can also be generated by ADO that will capture clicks on any specific DOM object, or will capture a value if a customer remains on a page for a specified period of time. When you are ready to deploy your experiment, ADO generates a single (one line) JavaScript tag that you place on the landing page and any subsequent outcome pages. Once you have placed the tag, you can go back into ADO and change any part of the experiment definition (content or data captures) without changing the code on the page.Client-side (single Yes – as above.JavaScript tag per page)Client-side (code N/A – ADO simplifies experiment delivery to the point where all that is need is a single line of codeembedded in each page) adding to each experiment touchpoint.Server-side An ADO server-side Toolkit enables your web server(s) to request content instructions directly from ADO before the page is rendered in the customer‟s browser. The setup of an ADO server-side experiment is distinct from ADO JavaScript in that no content is ever stored in ADO. Content variations are identified by simple key/value pairs which are used in your server-side page code logic to determine the content to be presented. ADO Toolkits are available off-the-shelf for PHP, JSP/J2EE, ASP and ASP.NET. If your environment is based on some other platform (such as Ruby on Rails), Accenture can provide details of the common XML-RPC API used in these Toolkits and help you to construct your own custom Toolkit.Inspection proxy Yes Multivariate Testing Buyer‟s Guide Page 45 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 48. Other (please specify) N/AInterfaceIs HTML knowledge No – The Campaign Editor will allow non-technical resources to define simple MVT test.needed to define tests?Live preview of Yes – The treatment explorer allows any possible combination of the attributes and levels coded into anvariations ADOtoolkit instrumented page to be displayed.Test Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) YesIntegrationWeb analytics Yes – ADO APIs are provided to enable ADO integration into web analytics solutions. We have executed many experiments on web pages also running web analytics tools like Adobe (Omniture), Webtrends, Google etc.CRM data Server side toolkit‟s can be used programmatically to integrate with CRM DB.E-commerce platform Server side toolkit‟s can be used programmatically to integrate with CRM DB.CMS ADO can access the content from CMS through content URL, content can be uploaded into ADO if you wish (when using ADO JavaScript only), its intended use is to provide content instructions rather than the content itself.Other (Please specify) N/A Multivariate Testing Buyer‟s Guide Page 46 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 49. Types of ReportingAbility to report against Yesmultiple goalsE-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) Any kind of value or volume based metric can be reported. Offline outcomes can also me reported within ADO if a suitable data feed can be constructed (A decrease in call centre call volumes as an example).PricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes – License costs between $150,000 and $250,000 per annum.(please specify typicalamount or range)Performance-basedpricing (please specify Yes – Accenture is happy to consider or propose performance based agreements.typical amount orrange)Set-up fee (pleasespecify typical amount Variable – $5,000 - $50,000.or range)Fee per number of„seats‟ (please specify Notypical amount orrange)Consultant daily rate(please specify typical $300 - $1,500 per day depending on experience and role.amount or range) Multivariate Testing Buyer‟s Guide Page 47 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 50. Other (please specify) Complemented by full consulting services and support – experienced consulting services are available as and when you need them. We can run your entire optimisation programme for you, or just give you access to the software and the training you need to build your own world-class optimisation practice.ServiceDedicated consultant / Yes – Local account management services are provided. These can be deployed as part of regional oraccount manager global account teams.Telephone support desk Yes – If required, available on a 24 / 7 basis.Other (please specify) Email supportNumber of staff  North America – 8specifically dedicated to  EU – 7the support of MVTcustomers (by  APAC – 4geography)  Other – 3Please give details of Standard technology support is included in the ADO license fee. This support is focused on maintaininghow you charge (or application availability and resolving application level issues that may be experienced by clients.don‟t charge) for An agreed level of account management support is also included in the license fee. Extended accountsupport management services will be agreed, as required, with individual clients. We have always believed that the quality of our advice and insight provided by our Account Managers is the primary driver through which we earn our reputation as a trusted partner. We take pride in staffing every engagement with an Optimisation Account Manager and then providing a framework of further specialists and experts to ensure that they are always best placed to exploit untapped value. This commitment to value creation and the specific actions that will ensure it takes place, will be defined with each client to suit their own needs.Additional Supporting InformationAny other relevantinformation Fig: Accenture History Multivariate Testing Buyer‟s Guide Page 48 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 51. USP – What sets you apart from competitors? (additional information) Advantages in testing methodology Unique mathematics – ADO is unique in harnessing the Nobel Prize winning „Choice Modelling‟ algorithm to drive multi-variant tests. This algorithm provides a number of specific benefits: Full-factorial experimentation – As well as a full range of A / B and fractional factorial (Taguchi) testing options, ADO is able to execute full-factorial experiments. Full-factorial experiments allow marketers to identify the most accurate optimal experiences, through the most effective use of sample (traffic) underpinned by a rich and extensive dataset. ‘Interaction effect detection’ – ADO is unique in being able to identify valuable „interaction effects‟ which may exist within the variables being tested in an MVT experiment. A simple illustration of interaction effects can be created by using a meal in a restaurant as an example. A diner‟s choice of Main Course, (Steak) will have an impact on their choice of side dish (French Fries). By extension, a diner‟s choice of starter (Smoked Salmon) will have an impact on their choice of Main (“I‟ve already had fish, so I‟ll take the Steak”). Understanding, at scale and with precision, the relationships that may exist between the choices offered to consumers is key to creating truly optimal experiences and Interaction Detection is a unique function of ADO. Multiple outcome optimisation – ADO is the only MVT technology that enables multiple outcomes to be defined for an MVT test. An example could be „increase current account applications without reducing the rate of credit card adoption‟ or „improve the traffic volume moving from the homepage but ensure that the ratio of driven traffic remains split 60% Flights, 30% Holidays and 10% Insurance.‟ The ability to drive „Multiple outcomes‟ increases optimisation flexibility and maximises value improvement whilst reducing risk. Multiple-touchpoint optimisation – As inferred in the previous „interaction effect detection section,‟ ADO is unique in being able to optimise customer journeys across and between touchpoints. MVT experiments can be designed that holistically optimise multiple touchpoints in the conversion funnel to ensure that there are no negative interaction effects and to improve the performance of individual touchpoints both individually and collectively. In addition, multiple touchpoint optimisation can work across channels. As an example, holistically optimising display media driving traffic to a product landing page, driving traffic into a check out process. Segmentation Discovery – As a result of an ADO experiment marketers are able, not only to identify the „overall‟ optimal experience to drive particular consumer behaviour, but also specific optimal experiences for individual segments. By interrogating the underlying data set, ADO is able to identify specific experiences which, whilst not being the best overall performer within the experiment, are the most effective at driving the desired outcome when delivered to a particular segment. Advantages in personalisation High volume personalisation delivery – Once ADO has discovered a „segment optimal‟ treatment (see Segmentation Discovery above), the personalisation module is able to continuously deliver the identified optimal treatment to individuals arriving to the experiment touchpoint that belong to the appropriate segment. Individuals can be identified as part of a segmentation by several methods; traffic source, cookie, time of day, geo-location look up, etc. Further to this, ADO personalisation can be used to aggregate multiple data-sources to enable ever more detailed real-time segmentations that are then served compelling optimal experiences. ADO makes segmentation and personalisation insights far more actionable and effective. While many MVT vendors have the ability to segment the audience, based on demographic or behavioural data, they fall short of providing true insight as to which segment dimensions are causing the behavioural differences and which segment dimensions are just „along for the ride‟. ADO‟s superior modelling approach allows the user to input multiple segment dimensions to uncover those that cause significant behavioural differences (while ignoring those that don‟t). That is, the model uncovers the segment dimensions that have a causal relationship with the data.Multivariate Testing Buyer‟s Guide Page 49All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storageand retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 52. Advantages in deployment and integration Wide range of integration options – ADO offers both JavaScript and server side integration options. JavaScript offers light touch „Software as a Service‟ type deployments and allow organisations to quickly execute MVT experiments with minimal technology effort. Server side integration enables the broadest and most comprehensive range of tests to be executed including those around page templates, relating to secure or https experiment sites. Often server side integration will provide advantages in delivering experiments that require integration with CRM databases or other internal services and systems (e.g. My Account). Clients are not limited to one mode of integration and it is not uncommon for clients to migrate between JavaScript and server side deployments as resources and business requirement change. Open and flexible interface – The ADO application has been designed so that a range of common elements and modules (testing, personalisation and reporting) can be configured and customised to suit individual business need. Whilst a comprehensive and simple to use GUI exists, (through which MVT experiments can be rapidly developed) the software can also be shaped to execute almost any kind of test through the development of customised integration modules. This gives clients the choice of working within a simple, wizard driven environment or creating a more advanced bespoke view of the technology. Transparent Dataset – ADO creates a fully transparent data set that can be exported to other systems and used to enhance other data-operations. Advantages in approach Proven methodology – As appropriate for one of the world‟s leading technology management consulting organisations, ADO is supported and underpinned by a robust, proven, delivery methodology. Technology deployment (JavaScript and server side) is delivered through a proven phased process guided by Accenture project management methodologies. Experiment design (by request, or for full service clients) leverages the knowledge and experience of highly skilled statisticians and analysts, coupled with optimisation experts and, critically, industry experts who help to identify the emerging opportunities within a client‟s sector. Global Account management – Where appropriate, Accenture can provide a global account management team able to deliver a unified programme of optimisation across geographies. 24 / 7 Global technical support – If required, Accenture can provide 24 / 7, technical support in multiple languages and to the highest SLAs of response.Multivariate Testing Buyer‟s Guide Page 50All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storageand retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 53. Accenture Interactive – Market Positioning ChartsChart 1: Business Model Full-service Where we are Where now we are going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared Where Where to competitors we we are are going now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 51 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 54. 10.2. Adobe SystemsAdobe Systems, Inc.www.adobe.comCompany Proposition COMPANY OWNERSHIPAdobe revolutionises how the world engages with ideas and information. Our award-winning Publicly held and traded onsoftware and technologies have set the standard for communication and collaboration for more than the NASDAQ: ADBE.25 years, bringing vital and engaging experiences to people across media and to every screen in theirlives, at work and at play. CLIENTSThe impact of Adobe® software is evident almost everywhere you look. Whether people arecollaborating at work, transacting online, or socializing with friends, businesses use Adobe software Adobe software is distributedand technologies to turn digital interactions into richer, high-value experiences that reach across in 34 languages on majorcomputing platforms and devices to engage people anywhere, anytime. platforms worldwide.With a reputation for excellence and a portfolio of many of the most respected and recognizablesoftware brands, Adobe is one of the world‟s largest and most diversified software companies. OFFICE LOCATIONSBrief Company History Adobe has dozens of officesAdobe was founded in 1982 by Charles Geschke and John Warnock and has produced such products throughout the world.as Adobe Acrobat, Illustrator, Photoshop and Flash. Today the company has matured into anenterprise software company that offers solutions for customer experience management, contentpublishing and online marketing optimisation. HEAD OFFICEUSP – What sets you apart from competitors? ADDRESS 345 Park AvenueAdobe has a worldwide presence and its software has become ubiquitous among enterprisebusinesses, content developers and publishers, and online marketers. In 2010, the company San Jose, California 95110achieved record revenues of $3.8 billion and reported its first billion dollar quarter. USACase study and/or testimonialsAdobe has published hundreds of customer case studies that highlight the success customers have CONTACTachieved in using Adobe products. Phone: +1-408-536-6000Adobe Test Target has gained significant market share and is in use by some of the world‟s mostsuccessful online brands, including Accuquote, CNET, Dell, Etrade, Experian, McAfee, SafeCo, Email: sales@adobe.comSalesforce.com, TD Ameritrade, and dozens of others.For a complete listing of customer case studies, please visit:http://www.omniture.com/en/products/conversion/testandtarget Multivariate Testing Buyer‟s Guide Page 52 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 55. Service ModelFull-service solution Yes. Adobe provides a full range of services to help Test&Target customers achieve success. This support includes a full-service offering where testing strategy, test design, creative development and execution as well as result interpretation and presentation are all included. Omniture Consulting Services are available to assist clients in getting the product installed and the first tests implemented. These same services are available for on-going test strategy and design as well as result interpretation on an on- going basis.Self-service solution Test&Target has been designed and implemented from the ground-up to be run directly by marketers rather than operated by our services teams. Many customers, large and small, use Test&Target with little or no service support from Adobe.Hosted Test&Target is only delivered as Software-as-a-Service. We do not offer an on premise version.Other (please specify) N/AOther ServicesSegment targeting Yes. Test&Target has multiple segmentation and targeting capabilities. Targeting is used to determine(rules) which content a visitor sees, how success metrics are recorded, and how campaigns performed. Targeting and segmentation is presented in a marketer-controlled UI that groups targeting criteria into several categories:  Site Pages – URLs, domains, URL parameters associated with the current page, landing page, or previous page.  Visitor Behaviour – built in behavioural characteristics such as new v. returning, browser type, operating system or custom visitor behaviours stored in the Test&Target profile.  Traffic Sources – search campaigns/queries and referring URLs.  Geo – country, city, state, zip, DMA, postal code, latitude, longitude.  Success Metrics.  Percentage Targeting.  Saved/custom segments (called the „Targets Library‟). Targeting and segmentation can be defined at the campaign level, the experience/recipe level, or the location level as well as for reporting. Test&Target can also target campaigns to common segments shared across the products of the Adobe Online Marketing Suite.Recommendation Yes. Test&Target provides a close integration with Adobe Recommendations within the Adobe Onlineengine (automated Marketing Suite. The same tags and anonymous visitor profile that power Test&Target are used directlymodel) by Recommendations.Behavioural targeting Yes. A key advanced capability is the breadth and flexibility of Test&Target‟s targeting and segmentation(automated model) capabilities. These components are even more tailored by marketers to their business requirements and marketing hypotheses using profile scripts – written by marketers and executed server-side by Test&Target to capture visitor characteristics. These characteristics are often combined to create very robust views of a customer‟s visitors. Any custom profile information is automatically accounted for by Test&Target‟s 1:1 predictive models giving a significant amount of marketer control and influence over targeting capability. The predictive modelling engine scores each visitor relative to content/offer options in order to serve the most relevant offer to each site visitor individually. This system also accounts in real-time for the changing preferences of site visitors thus delivering significant value and lift.Other related services Test&Target also delivers advanced capabilities through integrations with other Adobe products,(please specify) including SearchCenter, SiteCatalyst, Discover, Insight, Recommendations, SiteSearch, Publish, Dreamweaver , Search&Promote, CQ, and Flash Pro. Flash Pro integration offers a compelling capability that streamlines the delivery and testing of dynamic creative, both on-site and in display ads. Multivariate Testing Buyer‟s Guide Page 53 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 56. Type of Testing Methodology / TechnologyA/B Testing by URL Yes. We can test entirely different destination URLs. This technique is often employed by search marketers to determine the best landing page for an ad group for example or by display marketers for the same purpose. We offer this solution natively in Test&Target but also use it as the basis for our integration with Adobe SearchCenter+.A/B Testing usingmultiple content YesvariantsMVT - full factorial Yes. Test&Target supports full-factorial and optimal design of experiments.MVT - Fractional Yes. Test&Target supports the Taguchi method for multivariate testing. Test&Target provides a Taguchifactorial/Taguchi test design tool that automates test design. Marketers have the ability to modify or override the design ofMethod the campaign though the campaign still needs to match the valid Taguchi array.Ability to cull poor Yes. We offer manual paring of underperforming or otherwise undesirable content versions. We alsocontent as test proceeds offer an automated self-optimizing campaign that automatically reduces traffic to the variations that are underperforming. As response rates change, the system adjusts traffic accordingly in order to drive overall campaign lift. This form of automated campaign is different from our 1:1 behavioural targeting solution which makes a content serve decision based on predictive models for each individual site visitor.Other (please specify) By default, Test&Target splits randomly between experiences in a campaign. However, many alternative approaches are supported including different percentages of traffic to the various recipes in a test. Tests can include targeting criteria at the campaign level, at the recipe/experience level, or at the location level to make sure that the right audience is exposed to the test. These different approaches (targeting and percentage of traffic allocated) can be used in combination with one another.Engine TypeClient-side (generic Yes. In general, Test&Target can support a number of different deployment options ranging from aJavaScript tag across all generic tag/script that is the same on all site pages, single-line JavaScript deployed on specific pages, orpages) tags embedded in the page code on specific pages. Each approach allows for concurrent campaigns without page modification.Client-side (single YesJavaScript tag per page)Client-side (code Yesembedded in each page)Server-side NoInspection proxy NoOther (please specify) We also have modules for use in Flash and software development kits for mobile app optimisation for the iOS, Blackberry/RIM, and Android platforms.InterfaceIs HTML knowledge No. Test&Target offers several different approaches for mitigating the need for html or page-codeneeded to define tests? knowledge. First, Test&Target‟s first-of-its-kind visual editor (called Onsite and released in 2008) gives marketers a visual campaign creation tool that runs in the page or pages where the campaign will run. Secondly, we have a WYSIWYG HMTL editor. Third, through integrations with Scene7 and Day, rich content authored in those solutions is automatically deployed to Test&Target for use in tests/campaigns.Live preview of Yesvariations Multivariate Testing Buyer‟s Guide Page 54 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 57. Test Segmentation MethodsIs the system able to Yes. Targeting campaigns to specific visitors or visitor segments is an essential and fundamentallimit tests to specific capability of Test&Target. Segmentation definition using a variety of criteria and techniques is madeaudience segments? easy through the Test&Target interface designed for use by marketers.Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) Test&Target makes basing segmentation/targeting off of behavioural information easy through an open and marketer-defined profile that can be used to flexibly capture any type of on-site or off-site interaction data. We also have built-in geo segmentation and third party demographic data. And, through an integration with Adobe Insight, Test&Target makes it easy to include offline or enterprise data in the profile for use in targeted campaigns or for consideration by our predictive modelling engine.IntegrationWeb analytics Yes. Adobe Test&Target offers strong integration with Adobe SiteCatalyst, Discover, and Insight.CRM data Yes, with Salesforce.com.E-commerce platform Yes, we work in conjunction with the full spectrum of ecommerce platforms ranging from ATG and IBM WCS to Fry, GSI, and Marketlive to Magento to name a few.CMS Yes, we have a rich integration with Day Software (CQ 5.4) and also have a built-in integration with Crownpeak. We are used in conjunction with an array of other CMS solutions, including but not limited to, Fatwire, Vignette, and SDL Tridion.Other (Please specify) We have built-in integrations with ExactTarget and Responsys for content optimisation in email and have been used with a variety of other email packages including but not limited to SilverPop and Epsilon. We have integration solutions with a variety of third party data providers like Bluekai and Demandbase. We have a rich session replay integration with Tealeaf. Finally, Test&Target is strongly integrated with other products in the Adobe Online Marketing Suite as well as other content creation products across Adobe. Multivariate Testing Buyer‟s Guide Page 55 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 58. Types of ReportingAbility to report against Yes. Goals are configured on a per campaign basis and can include multiple and compound metrics.multiple goals Also, through integrations with Adobe Online Marketing Suite products like SiteCatalyst, Discover and Insight, a rich variety of success metrics can be used to evaluate campaign success. For example, not only can we evaluate bounce rate in Test&Target and SiteCatalyst, but we can look at the pathing or page-flow impact of campaigns through Discover.E-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yes, our reports are offered primarily online but are often exported using our extensive APIs.online?Other (please specify) Engagement metrics including time-on-site and a custom scoring solution.PricingSubscription modelbased on server calls Yes(please specify typicalamount or range)Subscription modelbased on flat licence fee No(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Optionally, typically through Adobe Consulting Services.or range)Fee per number of„seats‟ (please specify Notypical amount orrange) Multivariate Testing Buyer‟s Guide Page 56 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 59. Consultant daily rate(please specify typical N/Aamount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk Yes, 24/7 and in seven languages. Support also available via chat, email, Twitter, online, knowledge base, customer forum, etc.Other (please specify) N/ANumber of staff We have a worldwide staff of 400 individuals specifically dedicated to the support of MVT customers.specifically dedicated to This includes support of our global clients through all aspects of the products purchase,the support of MVT implementation, consulting, support, technical services, education and training, certification,customers (by development and engineering, consulting, marketing, etc.geography)Please give details of New contracts for Test&Target include consulting packages that include an initial time allotment forhow you charge (or implementation consulting and a monthly time allotment for test strategy and design consulting. Thesedon‟t charge) for hours can be augmented on a fee-based contractual basis. All other consulting services are available on asupport fee-based contractual basis.Additional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 57 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 60. Adobe – Market Positioning ChartsChart 1: Business Model Full-service Where we are now Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 58 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 61. 10.3. AmadesaAmadesahttp://www.amadesa.com/Company Proposition INVESTORSAmadesa is the first company to enterprise level software-as-a-service (SaaS) solution in a fullyintegrated suite spanning testing, targeting and automated targeting delivery. The Amadesa Carmel VenturesCustomer Experience Suite™ offers end-to-end testing and relevance targeting solutions to helponline businesses maximise revenues, enhance customer engagement, increase conversions andimprove marketing ROI. Our technology and services help companies like Progressive Insurance, CLIENTSShoebuy.com, myFICO and General Mills optimise the effectiveness and relevancy of theirwebsites, and convert site visitors into customers. Amadesa‟s customers include ProgressiveOur target customers are ecommerce (retailers, subscriptions) and ebusiness (travel, CPG, Insurance, General Mills,education, media, financial services, healthcare, etc.) companies who need to increase revenues,build their brands, improve conversions and expand relationships with visitors across all myFICO, Shoebuy.com, Newmediums. England Journal of Medicine, MountainBrief Company History Equipment Co-op, and manyAmadesa was founded in 2005. In 2008, it received Series A funding from Carmel Ventures. At others.first, Amadesa provided a full service platform focusing on optimizing the checkout process forcustomers. As of 2008, the company launched a SaaS platform offering end-to-end websitepersonalisation and optimisation tools. OFFICE LOCATIONSUSP – What sets you apart from competitors?Amadesa offers a fully integrated suite of onsite, e-mail and mobile testing, targeting and Headquarters / USApersonalisation solutions designed to meet different company needs regardless of previous testing Officeexperience. 630 Davis Street #300Amadesa is a marketer-friendly e-commerce solution and easy to use application with and Evanston, IL 60201 USAintuitive five-step process, and multiple integration options to reduce IT involvement. Amadesaoffers genuine support through product and client services experts that help support the most R&D / EMEA Officeeffective optimisation programme form end-to-end. Finally, Amadesa has no hidden fees. Instead,a flat rate is charged for unlimited optimisations and straightforward support options including 3 Hahilazon St. 9th Floorself-service, managed service or full service. Ramat-Gan 52522 IsraelCase study and/or testimonialsTRUSTe.com HEAD OFFICETRUSTe deployed Amadesa‟s self-service, easy-to-use A/B Testing platform to determine the ADDRESSbenefits of including the privacy seal on consumer websites, and obtained the following results: AchooAllergy.com achieved a $10 increase in average order value. Headquarters / USA Wedding service provider The Pros saw an 84% lift in purchases. Office Budshop.com realised a 13% revenue per visit lift. 630 Davis Street #300 Debnroo.com visitors who spotted the TRUSTe privacy seal were 29% more likely to buy than visitors who didn‟t see the seal. Evanston, IL 60201 USA2Checkout .comUtilizing the Amadesa platform, 2Checkout was able to optimise their website and yield increased CONTACTconversions including: Rebecca Oistad Hiding the automated order summary before the final page and enhancing security messages. Adding mouse-over pop-up reminders designed to provide data entry instructions and Marketing Director ultimately reduce user error and time on site. Redesigning the billing information page, splitting the form from one lengthy column into two 847-448-0086 smaller parallel columns to eliminate excessive scrolling. Rebecca@amadesa.com Shifting security logos beneath the „continue‟ button and into the eye path of the consumer.Shoebuy.comAfter implementing Amadesa, Shoebuy experienced significant increases in revenues per visit andaverage order values. With Amadesa, the company has been able to test and segment programmesto improve the effectiveness of marketing initiatives. For example, they used the Amadesa solutionto target mobile users for its recently launched Apple iPhone and iPod touch applications, todetermine the impact if security logos on its Web site, and to run design and layout tests across itscatalogue pages. Multivariate Testing Buyer‟s Guide Page 59 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 62. Service ModelFull-service solution YesSelf-service solution YesHosted Yes – Amadesa provides a hosted SaaS platform.Other (please specify) Managed Services: providing optimisation planning and implementation, test analysis and creative design services at an hourly rate with several pre-packaged offerings to suite common customer needs.Other ServicesSegment targeting Yes(rules)Recommendationengine (automated Nomodel)Behavioural targeting Yes(automated model)Other related services InPlace (In-context, real-time reporting interface).(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi NoMethodAbility to cull poor Yescontent as test proceedsOther (please specify) All testing methodologies are supported by a testing engine called „Pick the Winner‟ that facilitates automated analysis and elimination of underperforming variants to expedite test completion.Engine TypeClient-side (genericJavaScript tag across all Yespages)Client-side (single YesJavaScript tag per page)Client-side (code Yesembedded in each page)Server-side Yes – via API. Multivariate Testing Buyer‟s Guide Page 60 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 63. Inspection proxy NoOther (please specify) N/AInterfaceIs HTML knowledge Noneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter NoVisitor type YesTime returning YesVisit Metrics No(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) Geographic location (up to City level), Browser type and version, Time and Day of visit, Application- based and custom segmentation using tag parameters.IntegrationWeb analytics YesCRM data YesE-commerce platform YesCMS YesOther (Please specify) Site-search (Endeca)Types of ReportingAbility to report against Yesmultiple goals Multivariate Testing Buyer‟s Guide Page 61 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 64. E-commerce conversion YesPage goal conversion YesBounce rate NoRevenue per page NogenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) An on-site contextual optimisation management tool called „InPlace‟ provides a dashboard view as well as the complete set of detailed reports for all optimisations directly on top of the site being optimised.PricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes, based on total monthly page views.(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Noor range)Fee per number of„seats‟ (please specify No, we offer an unlimited amount of seats for no additional fee.typical amount orrange)Consultant daily rate(please specify typical Strategic Services start at $250 per hour and can be bundled at a lower rate.amount or range)Other (please specify) N/A Multivariate Testing Buyer‟s Guide Page 62 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 65. ServiceDedicated consultant / Yesaccount managerTelephone support desk YesOther (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT Allcustomers (bygeography)Please give details ofhow you charge (or Several different on-demand service options are available. We provide Professional Services across a range of needs, offering Full, Managed and A la Carte Services. We also offer a Quick Start programmedon‟t charge) for to help customers get up and running at the beginning of an engagement.supportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 63 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 66. Amadesa – Market Positioning ChartsChart 1: Business Model Full-service Where Where we are we are now going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 64 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 67. 10.4. AutonomyAutonomyhttp://www.autonomy.com/Company Proposition COMPANY OWNERSHIPAs the acknowledged leader in the rapidly growing area of Meaning Based Computing, Autonomy Public company listed on theallows its 20,000+ clients worldwide to automatically process and form a conceptual understanding London Stock Exchangeof all information, whether in the form of text, audio, images or video. Autonomy Optimost, as part LSE: AU or AU.L.of Autonomy‟s Meaning Based Marketing product suite, leverages advanced statistical analysis andpattern-matching in order to provide the most comprehensive, end-to-end offering for real timemarketing optimisation, whether that is increasing click-throughs for online ads, improvingcustomer engagement on a site, optimizing call center scripts or maximizing e-mail subscriptions. CLIENTSWith over a decade of experience in helping businesses optimise their marketing, Autonomy Belron, FT.com, John Lewis,Optimost is uniquely positioned as the premier vendor in an increasingly competitive market. AVIS, LexisNexis, BritishDelivered both on-premise and as a Software as a Service, Autonomy Optimost offers the most Airways, Look Voyages,advanced website testing and optimisation capabilities available and brings immediate ROI. Unicredit Banca, EarthLink, Gocompare.com, Butlins,Brief Company History Northwest Airlines etc.Founded in 1996 out of pioneering research at Cambridge University, Autonomy has experienced ameteoric rise to become the second largest pure software company in Europe. As a financially stablecompany, Autonomy is listed on the London Stock Exchange and has offices worldwide. OFFICE LOCATIONSUSP – What sets you apart from competitors? Global presence with officesForming a conceptual and contextual understanding of all data, Autonomy Optimost provides real- in Europe, North America,time testing, segmentation and targeting of site visitors to allow website optimisation and improved Latin America and Asiacustomer satisfaction while generating real and fast ROI. Unlike other existing offers in the market, Pacific.Autonomy Optimost MVT offers a quick and easy implementation. It provides a simple userinterface for building basic tests without the need for HTML or CSS knowledge, thus making it easyto use for non-technical users. Capable of testing unlimited combinations of content – Flash, HEAD OFFICEJavaScript, AJAX, video and audio – as well as pricing and offers for revenue optimisation, it ADDRESSremoves guesswork traditionally used to gauge customer preferences and intent. UK Headquarters:Autonomy‟s multivariable testing has no impact on load time, SEO or secure pages through the use Cambridge Business Park,of Akamai content delivery network. Its patented optimal design technology allows for very Cowley Road, Cambridge,complicated tests to produce valuable insights in a short time automatically with low manual tasks CB4 0WZand no reliance on IT. It can also be easily integrated into Autonomy‟s Meaning Based Marketingsuite including intelligent recommendations, ad serving and content retargeting. US Headquarters:Case study and/or testimonials One Market Plaza, Spear Tower, Suite 1900, SanLOVEFILM increases conversions with Autonomy Optimost MVT Francisco, CA 94105LOVEFILM wanted to convert more visitors into subscribers to their DVD rental service. It knew itshomepage played a critical part in achieving conversions, and prior to deploying AutonomyOptimost, it had been experimenting with multivariable testing for a year through another provider. CONTACTHowever, results had been few, and the testing process had proved disappointingly slow. Andrew HunterWorking with Autonomy Optimost, LOVEFILM had a very clear focus for tests – the top half of its Business Developmenthome page, with the objective of gaining an uplift in subscribers through multivariable testing on Managerthis specific area of the site. Tel: +44 1223 448 067Quickly up to speed and ready to roll within a day of sign-off on the project, Autonomy Optimostcreated a subject test plan and a custom-built page template. It tested six variables, including the Email:page header, the four-step section that explains how LOVEFiLM works, the free trial button and the andrewh@autonomy.comlayout of the page itself. Autonomy Optimost came up with 92 different page combinations, andtested 192 elements. Each element remained on the home page for a week to enable it to be testedwith a wide range of consumer types, and for sustainability. This enabled LOVEFiLM to see exactlywhat combination of elements affected conversion rates.The results were outstanding. In a four-week test period, conversions rose by 10 per cent – anincrease that will bring in a seven figure sum of revenue each year for the company. Multivariate Testing Buyer‟s Guide Page 65 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 68. Service ModelFull-service solution YesSelf-service solution YesHosted YesOther (please specify) N/AOther ServicesSegment targeting Yes(rules)Recommendationengine (automated Yesmodel)Behavioural targeting Yes(automated model)Other related services N/A(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) N/AEngine TypeClient-side (genericJavaScript tag across all Yespages)Client-side (single YesJavaScript tag per page)Client-side (code Yesembedded in each page)Server-side YesInspection proxy Yes Multivariate Testing Buyer‟s Guide Page 66 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 69. Other (please specify) N/AInterfaceIs HTML knowledge No, you can build tests in a drag and drop interface.needed to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) N/AIntegrationWeb analytics YesCRM data YesE-commerce platform YesCMS YesOther (Please specify) N/ATypes of ReportingAbility to report against Yesmultiple goalsE-commerce conversion Yes Multivariate Testing Buyer‟s Guide Page 67 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 70. Page goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) N/APricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Noor range)Fee per number of„seats‟ (please specify Notypical amount orrange)Consultant daily rate(please specify typical Noamount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk Yes Multivariate Testing Buyer‟s Guide Page 68 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 71. Other (please specify) N/ANumber of staffspecifically dedicated to  20 staff in the UKthe support of MVTcustomers (by  50 staff worldwidegeography)Please give details ofhow you charge (or Five hours of support per month is included in the license fee.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 69 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 72. Autonomy – Market Positioning Charts WhereChart 1: Business Model we are going Where Full-service we are now Self-service We only do Range of MVT technologies WhereChart 2: Type of Company we are going Where Large compared we are to competitors now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 70 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 73. 10.5. GlobalMaxerGlobalMaxerwww.globalmaxer.comCompany Proposition COMPANY OWNERSHIPGlobalMaxer is a software platform wholly owned, developed and supported by Oban Multilingual, Privately owned by Greigthe leading specialist international and multi-lingual web marketing agency. GlobalMaxer allows Holbrook and Grantweb marketers to undertake cultural multivariate testing (MVT), AB testing and segmentation tests Holbrookon their internet visitors and customers.Brief Company History CLIENTSEstablished in 2001, Oban Multilingual is a specialist in international internet marketing, being bestknown for our industry leading approach to SEO and PPC. We have worked with many global brands STA Travel, Mio Navman,in most industry sectors including Universal Music, Expedia, Manchester United and Proctor and Manchester Utd, AustinGamble. Since 2009, we have also been at the forefront of web testing with our unique cultural MVT Reed, Radley‟s of London,tool, GlobalMaxer. Poker HeavenUSP – What sets you apart from competitors?As a platform and service, GlobalMaxer is unique in a number of ways: OFFICE LOCATIONS Cultural Database of popular types of web design accessible by page type, industry sector and Brighton, UK country Cologne, Germany Ease of set up via our WYSIWYG. No hacking around in the code or expensive development required Flexible approach with both an agency managed or self managed option HEAD OFFICE ADDRESS Unique adaptive algorithm for maximum potential ROI Sussex Innovation Centre Industry leading consultancy including our 5 stage Landing Page Audit Science Park Square Real time analytics available 24/7/365 Falmer Platform neutral so works with all CMS Brighton Unique testing methodologies available including Sequential and Hypervariate testing BN1 9SB Configurable around multiple goals and objectivesCase study and/or testimonials CONTACTSince 2010, GlobalMaxer has been working the Sat Nav and GPS Manufacturer Mio on testing andoptimisation of their product and basket pages. Mio are globally the #3 in the GPS Market, and are Joe Dovetonbest known for the Navman brand. Head of Client ServicesFor Mio, we undertook Cultural Multivariate Testing on their product pages in order to find out if 01273 704504there were Cultural differences in the way that visitors across Europe responded to different types ofdesign. We tested a new version of their product page with a large heading, an orange heading (Mio‟s joed@obanmultilingual.combrand colours) and swapping the product image from right to left, in total making 12 combinations.The results were completely different across Europe: in both Spain and the UK, the original pagedesign was still the most popular. However in Germany, France and Italy the new designsconsiderably improved the page performance. In Germany the Orange header had an 83% chance ofbeating the original. In France and Italy all 11 new page versions beat the original page designs, within France the moved contents having a 70% chance of beating the original, whereas in Italy the largerorange title with the original contents was the winner.Follow on tests for Mio have included their basket processes, where our tests have created a high fivefigure Euro uplift in revenue simply by adjusting the colours on the call to action buttons through thecart. Multivariate Testing Buyer‟s Guide Page 71 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 74. Service ModelFull-service solution YesSelf-service solution YesHosted Yes, only hosted by GM available.Other (please specify) N/AOther ServicesSegment targeting (rules) Yes, by IP address, user variable (keyword/browser) and time stamp.Recommendation engine Yes, cultural database.(automated model)Behavioural targeting Yes, Sequential Testing option available.(automated model)Other related services Consultancy including Landing Page Audit; other testing methodologies include Hypervariate (HVT)(please specify) and Cognitive Load Testing through our tool Think Track.Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing using multiple Yescontent variantsMVT - full factorial YesMVT - Fractional Yesfactorial/Taguchi MethodAbility to cull poor content Yesas test proceedsOther (please specify) Hypervariate, Sequential, Cognitive LoadEngine typeClient-side (genericJavaScript tag across all Yespages)Client-side (single YesJavaScript tag per page)Client-side (code Noembedded in each page)Server-side NoInspection proxy NoOther (please specify) N/A Multivariate Testing Buyer‟s Guide Page 72 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 75. InterfaceIs HTML knowledge Noneeded to define tests?Live preview of variations YesTest Segmentation MethodsIs the system able to limittests to specific audience Yessegments?Inbound search keywords YesReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type Yes, by integration with eCRM data available on request as plug-in to basic package.Time returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) N/AIntegrationWeb analytics Yes, on request.CRM data Yes, on request.E-commerce platform Yes, on request.CMS Yes, on request.Other (Please specify) N/ATypes of reportingAbility to report against Yes, through our plug-ins to basic package.multiple goalsE-commerce conversion Yes, through our plug-ins to basic package.Page goal conversion Yes, through our plug-ins to basic package.Bounce rate Yes, through our plug-ins to basic package. Multivariate Testing Buyer‟s Guide Page 73 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 76. Revenue per page Yes, through our plug-ins to basic package.generationRevenue per visitor Yes, through our plug-ins to basic package.Gross profit / gross margin Yes, through our plug-ins to basic package.Average order value Yes, through our plug-ins to basic package.Custom event conversions Yes, through our plug-ins to basic package.Are reports available Yes, through our plug-ins to basic package.online?Other (please specify) N/APricingSubscription model basedon server calls (please No, but surcharges may apply on very large tests POA.specify typical amount orrange)Subscription model basedon flat licence fee (please Yes, approximately £1,000-£8,000 monthly depending on level of service required.specify typical amount orrange)Performance-based pricing(please specify typical On request following qualification period.amount or range)Set-up fee (please specify POAtypical amount or range)Fee per number of „seats‟(please specify typical N/Aamount or range)Consultant daily rate(please specify typical POAamount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk YesOther (please specify) Account team comprising Account Manager plus Technical Optimisation Manager (Usability and Design Manager andClient-Lead also available).Number of staff specificallydedicated to the support of 12 UK based.MVT customers (bygeography) Multivariate Testing Buyer‟s Guide Page 74 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 77. Please give details of howyou charge (or don‟t Usually included in contract bands; optional “unmanaged plus support contract” available on request.charge) for supportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 75 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 78. GlobalMaxer – Market Positioning ChartsChart 1: Business Model Full-service Where we are now Where we are going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where Small we are compared to now competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 76 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 79. 10.6. Google Website OptimizerGooglewww.google.com/websiteoptimizerCompany Proposition CLIENTSGoogle‟s mission is to organise the world‟s information and make it universally accessible and useful. Outrigger HotelsBrief Company History Mattress LiquidatorsStarted in 1996, Google began as a project by Stanford University graduate students Larry Page and Colonial CandleSergey Brin. Google has since grown by leaps and bounds, including various forms of advertising andweb applications for all kinds of tasks – in scores of languages – and thousands of employees and Team in Trainingoffices around the world.USP – What sets you apart from competitors? OFFICE LOCATIONSFree, powerful, flexible. Google Website Optimizer is one of the many products that Google offers for Globalfree that intend to make the web a better place for everyone. Along with Google Analytics andWebmaster Tools, Website Optimizer is great for the beginner and the expert alike, and can driveamazing results when put to good use. HEAD OFFICECase study and/or testimonials ADDRESSMattress Liquidators increased online leads 5,000% by improving their site with Google Website 1600 Amphitheatre PkwyOptimizer Mountain View, CA 94043The results Laidlaw has seen with Google Website Optimizer have given him the confidence to makea future leap into e-commerce – to turn the site into a retail store instead of just a lead generator.“We can use all of the Google tactics for tracking and advertising,” Laidlaw says. For now, he plans tokeep refining the site to get the best possible results for Mattress Liquidators, testing differentheadlines and colours of links. “I‟ll just keep tweaking,” he says. “I‟ll keep using Website Optimizeras long as my doors are open.”Doba.com increased conversions 70% and member sign-ups 50% with Google Website Optimizer“I‟m spending the same money on advertising but getting 50% more out of it by decreasing mytesting costs and increasing conversions,” Roberts says. “It‟s about letting your users guide you.”Jigsaw Health increased conversions 60% with Google Website Optimizer“Website Optimizer is a million times better than guessing, and that‟s what a lot of small startupcompanies like us used to have to do,” says Sullivan.Using insights acquired from using Google Website Optimizer, Moishe’s Moving Systems increasesits lead collection rate by 50%“After using Website Optimizer, we have a very good idea of what works on the site, what doesn‟t,and how to continue improving. Not using it would be almost criminal.”For more case studies, please visithttp://www.google.com/intl/en/websiteoptimizer/case_studies.html Multivariate Testing Buyer‟s Guide Page 77 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 80. Service ModelFull-service solution NoSelf-service solution YesHosted YesOther (please specify) N/AOther ServicesSegment targeting No (but can use a service like BT Buckets)(rules)Recommendationengine (automated Nomodel)Behavioural targeting No(automated model)Other related services N/A(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi NoMethodAbility to cull poor Yescontent as test proceedsOther (please specify) N/AEngine TypeClient-side (genericJavaScript tag across all Nopages)Client-side (single Code is inserted into test page and into goal page. Code must be modified for each test.JavaScript tag per page)Client-side (code Noembedded in each page)Server-side NoInspection proxy No Multivariate Testing Buyer‟s Guide Page 78 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 81. Other (please specify) N/AInterfaceIs HTML knowledge Yesneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Noaudience segments?Inbound search NokeywordsReferring URL NoLanding page URL NoQuery string parameter NoCookie parameter NoVisitor type NoTime returning NoVisit Metrics No(Time/duration)On-site search keywords NoBy device (e.g. iPad) NoOther (please specify) N/AIntegrationWeb analytics YesCRM data NoE-commerce platform NoCMS Yes – http://www.google.com/websiteoptimizer/wotep.htmlOther (Please specify) N/ATypes of ReportingAbility to report against Nomultiple goalsE-commerce conversion No Multivariate Testing Buyer‟s Guide Page 79 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 82. Page goal conversion YesBounce rate NoRevenue per page NogenerationRevenue per visitor NoGross profit / gross NomarginAverage order value NoCustom event YesconversionsAre reports available Yesonline?Other (please specify) N/APricingSubscription modelbased on server calls Free to use.(please specify typicalamount or range)Subscription modelbased on flat licence fee Free to use.(please specify typicalamount or range)Performance-basedpricing (please specify Free to use.typical amount orrange)Set-up fee (pleasespecify typical amount Free to use, authorised GWO partners available to perform implementation for a variable fee.or range)Fee per number of„seats‟ (please specify Free to use, can invite multiple users to have access.typical amount orrange)Consultant daily rate(please specify typical Dependent upon authorised GWO partner.amount or range)Other (please specify) N/AServiceDedicated consultant / Google has a network of authorised GWO partners who can provide support.account managerTelephone support desk No Multivariate Testing Buyer‟s Guide Page 80 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 83. Other (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT N/Acustomers (bygeography)Please give details ofhow you charge (or Varies by partner.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 81 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 84. Google – Market Positioning ChartsChart 1: Business Model Full-service WhereWhere we are we are now going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared Where to competitors Where we are we are going now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 82 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 85. 10.7. HiConversionHiConversion, Inc.www.hiconversion.comCompany Proposition COMPANY OWNERSHIPHiConversion‟s flagship e-Optimizer™ platform has replaced risky „trial-and-error‟ approachesemployed by traditional MVT solutions with a scientifically based optimisation methodology that Privately held.delivers predictable results in real-time and without risk.We solve common online marketing challenges: CLIENTS Web Traffic Monetisation: A revenue growth strategy based predominantly upon driving incremental web traffic is a game that ultimately results in diminishing returns. HiConversion focuses e-Optimizer™ delivers sustainable revenue growth without incurring the cost and risk primarily on e-commerce associated with a typical traffic generation strategy. merchants and has over 100 customers, including: Web Analytics Overload: Online marketers are overwhelmed by the amount of data provided Rockport, Hugo Boss, by web analytics. The numbers can reveal problems but they fall short in terms of providing Hallmark, Verio, Lifetime solutions. Brands, Eastbay, Applica, e-Optimizer™ acts in real time by collecting data, performing statistical analysis, and making Trend Micro, ModernAd adaptive website changes that serve to continually maximize conversion rates. Media, etc. IT Limitations: Marketing solutions requiring extensive IT support often fail. Our patented Virtual Enabling methodology eliminates the need for IT assistance. OFFICE LOCATIONSBrief Company HistoryHiConversion was founded in 2006 by a successful serial entrepreneur who holds a Ph.D. in Boca Raton, FL, USAoptimisation theory and whose scientific work has been published in leading engineering andmathematical journals and books. e-Optimizer™ is used by hundreds of e-commerce companies andit is protected by US Patent. HEAD OFFICEUSP – What sets you apart from competitors? ADDRESSUnder the umbrella of Conversion Rate Optimisation (CRO), competitors are selling MVT, 777 Yamato Rd, Suite 140segmentation and personalisation solutions that are in essence testing, or trial-and-error, tools. Boca RatonUnlike the competition, e-OptimizerTM legitimately delivers the “O” in CRO, a rigorous scientificoptimisation methodology that produces predictable, fast, and risk free results. FL 33431 USAThe key differentiators of our optimisation platform are: Real-Time ROI CONTACT Our patent-pending Adaptive Multivariate Optimization (AMVO) engine adapts to visitor behaviour in real time and delivers winning web pages and incremental revenues 10 to 100 Norman Matus times faster than alternatives. COO Risk-Free Optimisation 561-962-3692 Unlike traditional MVT solutions that generate un-advertised losses during the testing phase, AMVO starts lifting conversion rates even as it is still searching for top performing web pages. nmatus@hiconversion.com IT-Free Use By using our patented Virtual Enabler™ and its simple point-and-click interface our customers can quickly setup optimisation campaigns and start enjoying incremental revenues without dependency on the availability of IT resources.Customer testimonialsIncreased Conversion Rate by 19.8%“HiConversion is the single most amazing product I‟ve ever come across in my 14+ years of onlineexperience. It works better than the most expensive systems on the market, and best of all can bestarted without any need for IT department support. If you‟re not using HiConversion, you aresimply throwing away money.” (Mitchell Lieberman, CEO, www.OneWayFurniture.com)Note: For additional customer testimonials and a case study, see the “Additional SupportingInformation” section. Multivariate Testing Buyer‟s Guide Page 83 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 86. Service ModelFull-service solution YesSelf-service solution YesHosted YesOther (please specify) N/AOther ServicesSegment targeting Yes(rules)Recommendationengine (automated Nomodel)Behavioural targeting No(automated model)Other related services No(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL NoA/B Testing usingmultiple content YesvariantsMVT - full factorial NoMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) AMVO (patent pending adaptive multivariate optimisation – real time method that adapts to visitor behaviour in real time).Engine TypeClient-side (genericJavaScript tag across all Yes (US Patent #7,779,161).pages)Client-side (single YesJavaScript tag per page)Client-side (code Yesembedded in each page)Server-side NoInspection proxy No Multivariate Testing Buyer‟s Guide Page 84 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 87. Other (please specify) N/AInterfaceIs HTML knowledge Noneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning NoVisit Metrics No(Time/duration)On-site search keywords NoBy device (e.g. iPad) YesOther (please specify) N/AIntegrationWeb analytics NoCRM data NoE-commerce platform YesCMS NoOther (Please specify) N/ATypes of ReportingAbility to report against Yesmultiple goalsE-commerce conversion Yes Multivariate Testing Buyer‟s Guide Page 85 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 88. Page goal conversion YesBounce rate NoRevenue per page YesgenerationRevenue per visitor NoGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) N/APricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount $4k – $10k+or range)Fee per number of„seats‟ (please specify Notypical amount orrange)Consultant daily rate(please specify typical No (we provide fully managed services; flat fee includes all services).amount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk Yes Multivariate Testing Buyer‟s Guide Page 86 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 89. Other (please specify) Online support portalNumber of staffspecifically dedicated tothe support of MVT Fourcustomers (bygeography)Please give details ofhow you charge (or No chargedon‟t charge) forsupportAdditional Supporting InformationAny other relevant Case study and/or testimonialsinformation The best way to understand the value of e-Optimizer™ and its real-time optimisation capability is to see the solution in action. Example: Checkout Page Cart abandonment is a major e-commerce problem. Although the typical checkout page has a relatively plain layout, small changes to its elements can have a major impact. The graph below shows that after only few days of “training”, e-Optimizer™ was able to begin delivering versions of the web page that consistently outperformed the base line and ultimately produced a 30-day ROI of $359,000. AOV: ROI: Additionally, the below referenced “Variation Importance” report has provided valuable insight into customer preferences. Multivariate Testing Buyer‟s Guide Page 87 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 90. Customer testimonials Increased Conversion Rate by 10.33% “HiConversion‟s e-Optimizer™ solution relies on a real-time adaptive multivariate optimisation method that initially sounded far too complex and technical for our needs. Our initial perception couldn‟t have been further from the truth. It was easy to implement, quick to produce results, and best of all it provided us with deep insights into key customer behaviours and preferences.” (Paul Levett, CMO, www.liberatormedical.com) Increased Conversion Rate by 45% “With HiConversion‟s multivariate testing and the resulting optimisation of our checkout process, we were able to improve our conversion by 45%. This translates into double ROI in all of our marketing channels and half the customer acquisition cost without stretching our marketing budget. We are looking to further improve our overall conversion rate using HiConversion‟s adaptive multivariate testing solution. This has been the best investment we have made to our business.” (Nick Uresin, President, www.Tuccini.com)Multivariate Testing Buyer‟s Guide Page 88All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storageand retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 91. HiConversion – Market Positioning ChartsChart 1: Business Model Full-service Where we are now Where we are going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 89 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 92. 10.8. MaxymiserMaxymiserwww.maxymiser.comCompany Proposition COMPANY OWNERSHIPMaxymiser is the leading global provider of Conversion Management solutions, including A/B Maxymiser is a privately heldtesting, multivariate testing, web and mobile personalisation, and multichannel marketing. With independent company.offices in New York, London, San Francisco, Edinburgh, and Dusseldorf, and a global network ofpartners, its solutions enable brands such as Hertz, Hewlett-Packard, Sony, and Time Warner Cableto increase conversion rates across all marketing channels. Maxymiser empowers marketers to make CLIENTSsmarter, objective decisions, and deliver engaging customer experiences that improve the bottomline. Founded in 2006, Maxymiser is a rapidly expanding, private company whose best-in-class Our solutions enable brandstechnology is recognised by leading industry analysts. such as ASDA, Hertz,Brief Company History Hewlett-Packard and Sony to increase conversion ratesMaxymiser launched in 2006, becoming the first vendor to offer A/B and multivariate testing with optimisation andsolutions to many of Europe‟s leading online brands. As such, we‟ve been the catalyst for substantial personalisation across allchange in how companies develop customer experiences based on objective, data-driven decisions marketing channels.and continual website optimisation.USP – What sets you apart from competitors? OFFICE LOCATIONSMaxymiser offers a best-of-breed conversion optimisation platform: Offices in London, New York, Maxymiser products are deployed on a single platform, compared with point solutions San Francisco, Edinburgh assembled together through acquisition. Our clients benefit from lower TCO by turning on any and Dusseldorf. of our products without the need to change the site again. Maxymiser tests are made without additional IT requests, removing the burden placed on IT from traditional page tagging approaches. HEAD OFFICE Maxymiser‟s advanced statistics determine winning experiences for A/B through to complex ADDRESS multivariate tests, with an array of statistical models designed to meet any need. 121 Great Portland Street Maxymiser‟s MaxSEGMENT automatically suggests visitor segments for campaign targeting, London giving marketers the guidance needed to define targeting rules when necessary. United Kingdom Maxymiser‟s MaxPREDICT and MaxRECOMMEND automates the personalisation of recommendations, offers and promotions through self-learning, predictive models which drive W1W 6QL more precision with higher response rates and lower total cost of ownership. Maxymiser provides flexibility to combine campaigns as situations warrant, such as using CONTACT targeting rules for specific segments while predicting promotions for all other visitors. Wayne Morris (UK andCase study and/or testimonials Europe)Fragrance Direct added over £1 million to its revenue UK General Manager“The impact of our first six months working in partnership with Maxymiser has been a 14.5% + 44 203 375 0100increase in turnover, adding over £1m to revenue. This came from a 5% increase to average ordervalue and improved effectiveness at converting visitors. In year one, we forecast that we will benefit wayne.morris@maxymiser.cfrom 4.5x ROI on Maxymiser‟s fees.” (Katie Jowle, MD of Fragrance Direct) omASDA reduced bounce rate by 19%“We are continually looking to push site conversion performance harder and make visiting Mark Simpson (US)ASDA.com a better experience for customers. Maxymiser enables us to do this in a highlymeasurable way, making informed decisions on content based on live visitor response. Reducing our Founder and Presidentbounce rate 19% in this first test is a superb result.” +1 212 419 0394SEB Bank’s loan applications increased by 130% mark@maxymiser.com“Maxymiser enabled us to find out what users prefer and what makes them convert into customers.We no longer have to rely on gut feeling as our decisions can be based on statically significant dataon which content drives conversion and increases ROI sustainably.” Multivariate Testing Buyer‟s Guide Page 90 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 93. Service ModelFull-service solution Yes, Maxymiser provides a full range of services including optimisation strategy, creative design, web development for complex tests, and analysis of results.Self-service solution Yes, Maxymiser enables clients to take on each specialty area of services.Hosted Yes, Maxymiser‟s solution is hosted in multiple data centers globally.Other (please specify) N/AOther ServicesSegment targeting Yes, including the ability to suggest segments and combine offline CRM data for targeting online(rules) customers.Recommendationengine (automated Yes, with multiple recommendation models to be used for different applications.model)Behavioural targeting Yes, using any visitor attributes that can predict visitor behaviour.(automated model)Other related services Maxymiser suggests segments to replace manual analysis with intelligence that identifies key visitor(please specify) segments and the best performance experience for those segments.Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) Maxymiser‟s continuous optimisation determines winning experiences over time and automatically adapts and delivers those experiences without intervention. Marketers will not pick a winner too quickly in error and will limit the need to retest campaigns for seasonal changes.Engine TypeClient-side (generic Yes, Maxymiser‟s OneTouch platform requires only one change to the website, ever. IT is no longerJavaScript tag across all burdened with changes to the site when marketers need to perform a new test.pages)Client-side (single NoJavaScript tag per page)Client-side (code Noembedded in each page) Multivariate Testing Buyer‟s Guide Page 91 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 94. Server-side NoInspection proxy NoOther (please specify) Includes a high speed delivery network for all content assets.InterfaceIs HTML knowledge No, tests can be performed without knowledge of HTML.needed to define tests?Live preview of Yes, marketers can see what the variations will look like with Maxymiser‟s On Page Console whichvariations previews the site with content which is in a QA sandbox.Test Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) Yes, device and browser combinations.Other (please specify) Segmentation and targeting within tests are provided. Segments are automatically identified to remove the need to analyse reports across a wealth of visitor attribute combinations.IntegrationWeb analytics Yes, Maxymiser includes system interfaces and specific connectors for major web analytics companies.CRM data Yes, CRM Data such as propensity scores, products held, or offline segment can be used within the Maxymiser Inline Database to drive web decisions at high speed.E-commerce platform YesCMS YesOther (Please specify) Maxymiser is flexible at providing multiple integration methods to integrate with existing systems to make data collection actionable for decisions online. Multivariate Testing Buyer‟s Guide Page 92 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 95. Types of ReportingAbility to report against Yesmultiple goalsE-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross Yes, can be determined when combined with other data.marginAverage order value YesCustom event YesconversionsAre reports available Yes, reports are available online and can be scheduled and distributed through email.online?Other (please specify) Clickstream and revenue metrics are supported across experiences to track visitor level behaviour.PricingSubscription modelbased on server calls N/A(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes(please specify typicalamount or range)Performance-basedpricing (please specify N/Atypical amount orrange)Set-up fee (pleasespecify typical amount N/Aor range)Fee per number of„seats‟ (please specify N/Atypical amount orrange)Consultant daily rate(please specify typical N/Aamount or range) Multivariate Testing Buyer‟s Guide Page 93 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 96. Other (please specify) Maxymiser provides self-service through full-service options to meet the needs of each client and included in the flat license fee.ServiceDedicated consultant / Yesaccount managerTelephone support desk YesOther (please specify) Yes, Maxymiser provides full service to clients supporting specialty areas including strategic campaign planning, usability, creative design, web development and QA, and campaign and visitor analysis.Number of staffspecifically dedicated tothe support of MVT 60+customers (bygeography)Please give details ofhow you charge (or Support is included in self-service through to full service options.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 94 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 97. Maxymiser – Market Positioning ChartsChart 1: Business Model Full-service Where Where we are we are nowgoing Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 95 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 98. 10.9. MonetateMonetate Inc.www.monetate.comCompany Proposition COMPANY OWNERSHIPMonetate is the most comprehensive and sophisticated testing, targeting, and personalisation Monetate is a privately-heldsolution available for enterprise websites, offering built-in analytics, CRM, behavioural targeting, company.and automated catalog feeds. Monetate is uniquely simple to install with a single cut-and-paste ofone JavaScript tag that never changes. No internal IT resources are required. No external data feeds CLIENTSare needed. Complete analytics are provided via the browser-based Monetate dashboard. All tests Sports Authority, QVC,and campaign results can be tracked in real time. Urban Outfitters,Marketing departments at agile commerce companies like QVC, PETCO, L‟Occitane and Urban Anthropology, Aeropastle,Outfitters use Monetate to display and test personalized content and functionality, including PETCO, Brookstone,discounts and promotions that are automatically integrated into back-end systems. Monetate‟s full L‟Occitane,range of testing features includes MVT and A/B/n testing, plus automated and variable control Onlineshoes.com, Dicksgroups. Successful tests can be converted instantly into targeted and personalized messaging and Sporting Goods, GSIpromotional campaigns, anywhere on the website, with no additional cost or coding. Commerce, Isabella Oliver,Brief Company History ModCloth, NASCAR, NHL, Green Mountain Coffee, andFrom its founding in 2008, Monetate has quickly become the leading platform for testing onconsumer websites and provides a growing number of Internet Retailer Top 500 companies with an many more.unparalleled range of targeting and personalisation capabilities. Monetate is an IBM/Coremetrics OFFICE LOCATIONSPartner, 2011 eTail Rising Star, and member of Shop.org. Philadelphia, USAUSP – What sets you apart from competitors? HEAD OFFICEDescribed by marketing guru Bryan Eisenberg as “the most powerful and sophisticated testing and ADDRESStargeting tool available,” Monetate is enabled as a single-tag, SaaS enterprise marketing platform.Monetate is the first single-source solution that provides Marketing with actionable analytics for 100 Four Fall Corporatemaximizing website ROI through data-driven decision-making. Monetate is deployed and operated Centerwithout recourse to the IT department, HTML/CSS coders, or other technical experts. Suite 407Through its one tag, Monetate collects and analyses detailed data about site visitors, compiles a Conshohocken, PA 19428complete catalog of products on the site, serves up custom content to targeted traffic segments, andfeeds offers and promotions to back-end systems. Content and site features can be tested in multiple 1-877-666-3828ways, through MVT or A/B/n testing or automatic control groups. Results are trackable in real time CONTACTvia an elegant browser-based dashboard. Successful tests convert to ongoing campaigns with oneclick. Our full service offering includes access to our team of CRO experts. Scott SamiosCase study and/or testimonials VP SalesCase Study: L’Occitane’s Director of eCommerce, Matthew Kritzer, on Working with Monetate 1-646-334-0433“Implementation? Honestly, implementation took maybe 30 minutes. We put one tag in the global ssamios@monetate.comfooter and we were up and running within a week.“Favorite test? Badging, which was not on our radar, but because it‟s so easy with Monetate, wethought: Let‟s test it. Two weeks later, the data was like, Wow. We saw huge increases in conversionrates. Huge add-to-cart lifts. Across the board, there was tremendous engagement with the limitededition nature of our products, so we‟re playing it up in the stores now. It‟s an interesting scenariowhere the online has affected the offline world.“The Relationship? It‟s like having a vested partner beside you. Monetate‟s strategic services team isin the same boat with customers, trying to solve problems. And I really think that, together, we‟remore than we are individually. It‟s a community of learning and knowledge and sharing, and I wasnot expecting that at all.”Note: For additional testimonials see the “Additional Supporting Information” section. Multivariate Testing Buyer‟s Guide Page 96 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 99. Service ModelFull-service solution YesSelf-service solution NoHosted YesOther (please specify) Various levels of service are available.Other ServicesSegment targeting Yes(rules)Recommendationengine (automated Yesmodel)Behavioural targeting Yes(automated model)Other related services Product badging. Geo-targeting. Dynamic text rendering. Weather targeting.(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) Automatic and variable control groups on all targeted content and features, with true lift reporting and statistical significance displayed in real time.Engine typeClient-side (genericJavaScript tag across all Yespages)Client-side (single N/AJavaScript tag per page)Client-side (code N/Aembedded in each page)Server-side N/AInspection proxy N/A Multivariate Testing Buyer‟s Guide Page 97 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 100. Other (please specify) N/AInterfaceIs HTML knowledge Noneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) Geo-targeting, including international, regional, metro-area, shipping zones, distance to store). Technographic targeting, including device, screen size, browser. Mobile and smartphone specific features.IntegrationWeb analytics YesCRM data YesE-commerce platform YesCMS YesOther (Please specify) Third party applications can be implemented in test mode to determine effect on conversion rate and other key metrics.Types of ReportingAbility to report against Yesmultiple goals Multivariate Testing Buyer‟s Guide Page 98 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 101. E-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) Customer acquisition. Revenue per session.PricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes. From $5,000 per month to $60,000 per month and above.(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Typically equivalent to one month‟s fee.or range)Fee per number of„seats‟ (please specify Notypical amount orrange)Consultant daily rate(please specify typical N/Aamount or range)Other (please specify) No consultants needed. Installation and operation require no IT resources or coding skills. Multivariate Testing Buyer‟s Guide Page 99 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 102. ServiceDedicated consultant / Yesaccount managerTelephone support desk YesOther (please specify) NoNumber of staffspecifically dedicated tothe support of MVT Varies according to demand.customers (bygeography)Please give details ofhow you charge (or No charge.don‟t charge) forsupportAdditional Supporting InformationAny other relevant Clients typically conduct dozens of tests per month. Tests can be designed and managed from a wideinformation range of devices including iPad. Additional Customer Testimonials “The ability to seamlessly insert content anywhere in our site without having to modify any of our pages is phenomenal and an incredible resource saver for our teams.” (Alex Miller, VP eCommerce & eMarketing, QVC (8th largest online retailer in America) “Monetate gives us the ability to test everything in real time, against a control group, under the exact same conditions as the target group; and that gives us a lot more speed and confidence when making marketing decisions.” (Bob Myers, CEO, Sheplers) Multivariate Testing Buyer‟s Guide Page 100 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 103. Monetate – Market Positioning ChartsChart 1: Business Model Full-service Where we are Where going we are now Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared Where to competitors we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 101 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 104. 10.10. SiteSpectSiteSpectwww.sitespect.comCompany Proposition COMPANY OWNERSHIPSiteSpect provides the industry‟s only non-intrusive multivariate testing and behavioural targeting Not disclosed.solution, enabling web marketers to optimise their websites across any browser, including anymobile device. With SiteSpect, marketers enjoy unsurpassed speed and flexibility for optimizing allaspects of their site, without the need for ongoing IT involvement. CLIENTSSiteSpect‟s patent-pending technology and professional services are used by some of the world‟s Wal-Mart, Staples, Mozilla,leading online businesses, including Wal-Mart, Staples, Mozilla, Newegg, JCPenney, MTV, Cabela‟s, Newegg, JCPenney, MTV,Overstock.com, VEGAS.com, iStockphoto, and Unitrin Insurance. For more information, visit Cabela‟s, Overstock.com,www.SiteSpect.com or call +31208202082. VEGAS.com, iStockphoto, ING Bank, TUI, SantanderBrief Company History OFFICE LOCATIONSAt SiteSpect, we approached the challenges of web site optimisation with a clean slate, years oftechnological and marketing know-how, and the freedom to build a web optimisation solution from De Corridor 27the ground up based on precisely what web marketers need to succeed. Ours is a solution built based 3621 ZA Breukelenon our inside knowledge of web marketing along with a thorough understanding of the business andtechnical realities of web site testing. The result of our vision is a technological leap – one that‟s The Netherlandsbrought new power to web marketers, and has helped make us a leading provider of A/B andmultivariate testing solutions since 2004.USP – What sets you apart from competitors? HEAD OFFICE ADDRESSAs the first and only solution that enables you to perform A/B, split, and multivariate tests withouthaving to install or integrate software, SiteSpect offers advantages no competing web optimisation 11 Beacon Streettechnology can match. Suite 500Whereas other testing services require various forms of page-tagging and software integration, only Boston, MA 02108 USASiteSpect lets you run tests without changes to your underlying web site – and with no ITinvolvement needed.SiteSpect‟s patent-pending technology lets you track end-user behaviour and modify site content „on CONTACTthe fly‟ as it‟s served in real-time. This capability also extends to any partner site to which you may Frank Overbeeklink. Managing Director EuropeOptimise any type of content – not just static content, but also any dynamic content, Ajax, Flash,CSS, WAP (mobile), and more. +31208202082Track any type of user behaviour, anywhere on any site. foverbeek@sitespect.comCase study and/or testimonialshttp://www.sitespect.com/case-studies.shtml Multivariate Testing Buyer‟s Guide Page 102 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 105. Service ModelFull-service solution YesSelf-service solution YesHosted YesOther (please specify) Enterprise, fully managed in house solution, no external communication needed.Other ServicesSegment targeting Yes(rules)Recommendationengine (automated Nomodel)Behavioural targeting Yes(automated model)Other related services AMPS, a Web and mobile performance optimisation solution which dramatically improves the speed at(please specify) which your site presents content to users.Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) N/AEngine TypeClient-side (genericJavaScript tag across all Nopages)Client-side (single NoJavaScript tag per page)Client-side (code Noembedded in each page)Server-side NoInspection proxy Yes, therefore no tagging / scripting on any pages. Multivariate Testing Buyer‟s Guide Page 103 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 106. Other (please specify) N/AInterfaceIs HTML knowledge Yesneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) User Agent, IP address, Geo Location, Language, Mobile devices, Previous tests.IntegrationWeb analytics YesCRM data YesE-commerce platform N/ACMS N/AOther (Please specify) N/ATypes of ReportingAbility to report against Yesmultiple goalsE-commerce conversion Yes Multivariate Testing Buyer‟s Guide Page 104 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 107. Page goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) Fully configurable segmentation in reportPricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Yes(please specify typicalamount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Yesor range)Fee per number of„seats‟ (please specify Notypical amount orrange)Consultant daily rate(please specify typical € 1000,-amount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk Yes Multivariate Testing Buyer‟s Guide Page 105 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 108. Other (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT 10 in EUcustomers (bygeography)Please give details ofhow you charge (or N/Adon‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 106 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 109. SiteSpect – Market Positioning ChartsChart 1: Business Model Full-service Where Where we are we are now going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 107 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 110. 10.11. VertsterVertsterhttp://www.vertster.com/Company Proposition COMPANY OWNERSHIPVertster provides a SAAS suite of conversion rate optimisation tools including AB testing, MVT, Privately held.content targeting, personalisation and behavioural targeting. Although the tool was initially designedas a do-it-yourself platform, Vertster also provides professional services to companies that requirethem. One of the first providers in the space seven years ago, the Vertster suite is now in its‟ fifth CLIENTSmajor version. The software has been used to run thousands and thousands of experiments on sitesranging from huge (Alexa top 10) to tiny. The current platform is cloud based, making it infinitely Vertster has thousands ofscalable to handle the largest customers. Vertster also offers a complete private label solution for customers across the globe.interactive marketing agencies, along with attractive financials designed for the agency businessmodel.Brief Company History OFFICE LOCATIONSVertster emerged in 2004 as one of the original three solutions for split and multivariate testing. United StatesNow in the fifth generation, Vertster is based in Salt Lake City, UT. In 2008, Vertster added a pointof presence in Germany to better serve customers in central Europe. GermanyUSP – What sets you apart from competitors?As one of the oldest and most experienced providers of conversion optimisation software, Vertster HEAD OFFICEdifferentiates their technology in four areas: technology, experience, usability and reporting. ADDRESSCustomers of Vertster benefit from numerous technological advantages, including faster pagerendering – only one server call is required and the JavaScript library is 400% to 500% smaller and 12257 S. Business Park Dr.faster than others! Vertster also includes a robust set of APIs that give data access, record conversion Suite 125data after the fact, and create server side testing possibilities. Draper, UT 84020Site integration consists of a single JavaScript snippet that is placed globally on a site. Simple testscan be designed graphically with a point and click interface, while advanced users can easily testnested content, CSS rules, JavaScript, AJAX interactions and more. Vertster‟s reporting is presentedin near real time, and includes a real-time segmentation tool making it easy to investigate test results CONTACTfor individual niches of site traffic. Richard A. CracroftCase study and/or testimonials VP Sales and MarketingMojoPages.com provides a community oriented review and directory site for local markets across theU.S. As members review and recommend local business services, they are sharing their good „mojo‟ 801.571.8200 x 1000with the community, and earning status. One way MojoPages generates revenue is by selling local 801.326.4807 Skypetargeted advertising and listings. Rich@vertster.comThe SetupChief Marketing Officer, Lee Mills realised that MojoPages could provide increased value toadvertisers, because the topic search landing pages were ineffective at turning visitors into ad clicks.After talking with several vendors about optimizing this channel, the decision was made to go aheadwith Vertster.com, because of their „single page tracking technology‟. This allows clicks on outgoingads to be tracked as conversion actions, something no other vendor could provide.The TestThe first optimisation focused on the „above the fold‟ elements of the search landing page, withspecific attention given to making sure the page clearly communicated a relevant connectionbetween the users initial search and the landing page displayed. Vertster decided the best approachfor this test was to use a V-9 Taguchi Test. This design considered four variables with three optionseach yielding 81 possible page recipes.ResultsThe testing was a huge success, with a measured improvement in click yield of approximately 40%.MojoPages learned several interesting things as a result of the test – asking people to shareinformation and reviews on their first interaction with the site was reducing stickiness, a simplerdesign with ads appearing higher on the page resulted in more ad clicks. Lee credits Vertster‟sTaguchi testing technology for significantly cutting down the time to get enough traffic to achieve astatistically reliable result and improving click profitability faster than we could have done it withother solutions. Multivariate Testing Buyer‟s Guide Page 108 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 111. Service ModelFull-service solution Yes, Vertster offers three levels of consultancy (technical, managed and full).Self-service solution Yes, Vertster was designed as a do-it -yourself solution.Hosted Yes, Vertster operates as a hosted SAAS.Other (please specify) Vertster also offers a private label solution to interactive marketing agencies.Other ServicesSegment targeting Yes. Vertster can target recipes, content and tests by visitor segments.(rules)Recommendationengine (automated Nomodel)Behavioural targeting Yes. Vertster has the ability to recognise user attributes and serve a proven recipe for each user type.(automated model)Other related services Vertster also captures click data and generates heat maps per recipe.(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi YesMethodAbility to cull poor Yescontent as test proceedsOther (please specify) Vertster supports fractional factorial testing to significantly reduce test run times.Engine TypeClient-side (genericJavaScript tag across all Yespages)Client-side (single Yes. Vertster does not require any per variable tagging.JavaScript tag per page)Client-side (code Noembedded in each page)Server-side Vertster includes a server-side API which allows customers to integrate with our test server programmatically. Multivariate Testing Buyer‟s Guide Page 109 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 112. Inspection proxy NoOther (please specify) N/AInterfaceIs HTML knowledge No, but as you get into more advanced tests and more involved tests it is helpful.needed to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yes, Vertster has both fixed and user defined audience segmentation available.audience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes – requires use of custom variables.(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) N/AIntegrationWeb analytics YesCRM data YesE-commerce platform YesCMS Yes, Vertster has a built in CMS to facilitate test content creation.Other (Please specify) N/ATypes of ReportingAbility to report against Yes, Vertster supports unlimited goals per test, including page goals, on page events, form completionmultiple goals and form field interaction. Multivariate Testing Buyer‟s Guide Page 110 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 113. E-commerce conversion Yes, Vertster supports simple conversions as well revenue based metrics like RPV and AOVPage goal conversion YesBounce rate Yes, including form bounce rate.Revenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event Yes, user defined, on-page and browser events are supported.conversionsAre reports available Yes, rich graphical and numeric reports suitable for printing are provided.online? Vertster provides a comprehensive data export API, allowing customers to export dataOther (please specify) programmatically.PricingSubscription modelbased on server calls No(please specify typicalamount or range)Subscription modelbased on flat licence fee Vertster‟s pricing is based on a monthly fee and a bucket of monthly test participants. Our Agency(please specify typical pricing begins at $99 (US) monthly and our Enterprise pricing begins at $899 (US) per month.amount or range)Performance-basedpricing (please specify Notypical amount orrange)Set-up fee (pleasespecify typical amount Noneor range)Fee per number of„seats‟ (please specify None, Vertster can be deployed across any domain owned by the customer at no additional charge.typical amount orrange)Consultant daily rate(please specify typical Vertster consultancy rates are from $200 to $300 (US) per hour.amount or range)Other (please specify) Pricing above 10 million test participants per month are custom quoted,ServiceDedicated consultant / Yes, depending on account type, a dedicated account manager, consultant and technical support repaccount manager may be included. Multivariate Testing Buyer‟s Guide Page 111 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 114. Telephone support desk YesOther (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT 10customers (bygeography)Please give details ofhow you charge (or Basic support is included, consulting rates apply to advanced support.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 112 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 115. Vertster – Market Positioning ChartsChart 1: Business Model Full-service Where we are going Where we are now Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 113 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 116. 10.12. Visual Website OptimizerVisual Website Optimizerhttp://visualwebsiteoptimizer.com/Company Proposition COMPANY OWNERSHIPVisual Website Optimizer is world‟s easiest A/B and multivariate testing tool. The company Paras Chopra, CEOproposition is to offer a tool that marketers can use without getting reliant on technologydepartment for test setup and implementation. Using Visual Website Optimizer, sophisticated A/B Sparsh Gupta, CTOor Multivariate Tests can be created and made live in less than 10 minutes and absolutely noknowledge of HTML or JavaScript is required thanks to our intuitive „visual‟ test designer. CLIENTSIn spite of being easy to use, the tool offers all enterprise-level testing, targeting and analyticsfeatures. Some of the notable functionality includes heatmap/clickmap reports, revenue tracking, 500+ including Microsoft,segmentation and targeting, cross domain tracking, rich API, automatic optimisation, etc. Groupon, Rackspace,Brief Company History Foursquare, 99Designs, etc.The company was started in late 2009 by Paras Chopra, who is an engineer with deep interest inbusiness and marketing optimisation. Later in 2010 Sparsh Gupta, a graduate from Oxford OFFICE LOCATIONSUniversity, UK, joined the company to take over as its CTO.USP – What sets you apart from competitors? Delhi, IndiaUsability and ease of use. Visual Website Optimizer is world‟s easiest A/B and multivariate testingtool which most non-technical folks can start using right away without needing any training or help HEAD OFFICEfrom IT department at all. We also pride ourselves with the quality of customer support (which is ADDRESSonly done by the engineers who actually work on the product so support quality is top-notch andissue resolution time is absolutely minimal). 74, Vidya Vihar, Sector-9Case study and/or testimonials Rohini, Delhi-110085We usually publish one new case study every week. Our growing (20+) collection of case studies indifferent industry verticals can be seen online at http://visualwebsiteoptimizer.com/case- CONTACTstudies.php Paras ChopraAs can be seen from case studies, typically, our clients end up increasing sales by up to 20-40% andconversions up to 80-100%. CEO +91-9868221372 paras@wingify.com Multivariate Testing Buyer‟s Guide Page 114 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 117. Service ModelFull-service solution No, but we have a list of certified partners than can help with CRO strategy: http://visualwebsiteoptimizer.com/certified-partners.phpSelf-service solution YesHosted YesOther (please specify) N/AOther ServicesSegment targeting Yes(rules)Recommendationengine (automated Nomodel)Behavioural targeting No(automated model)Other related services Heatmaps and Clickmaps(please specify)Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractionalfactorial/Taguchi NoMethodAbility to cull poor Yescontent as test proceedsOther (please specify) Ability to show best content as test proceeds.Engine TypeClient-side (genericJavaScript tag across all Yespages)Client-side (single N/AJavaScript tag per page)Client-side (code N/Aembedded in each page)Server-side N/A Multivariate Testing Buyer‟s Guide Page 115 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 118. Inspection proxy N/AOther (please specify) N/AInterfaceIs HTML knowledge Noneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes, with some additional custom programming.(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) Browser, OS, JavaScript variable, Useragent string, etc.IntegrationWeb analytics Yes (including Google Analytics and SiteCatalyst).CRM data Yes, with some additional custom programming.E-commerce platform Yes (works with all e-commerce platforms).CMS Yes (WordPress, Drupal, Joomla plugins are available).Other (Please specify) N/ATypes of ReportingAbility to report against Yesmultiple goals Multivariate Testing Buyer‟s Guide Page 116 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 119. E-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per pagegenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify)PricingSubscription modelbased on server calls Yes (dependent on number of visitors tested). Typical pricing $1,300/month for 1 million visitors tested(please specify typical per month.amount or range)Subscription modelbased on flat licence fee N/A(please specify typicalamount or range)Performance-basedpricing (please specify N/Atypical amount orrange)Set-up fee (pleasespecify typical amount Noneor range)Fee per number of„seats‟ (please specify Nonetypical amount orrange)Consultant daily rate(please specify typical Technical support included in the cost.amount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount manager Multivariate Testing Buyer‟s Guide Page 117 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 120. Telephone support desk Yes, we have a phone number where customers can call though preferred mode is through email.Other (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT 5customers (bygeography)Please give details ofhow you charge (or We don‟t charge extra for support. It‟s all inclusive.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 118 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 121. Visual Website Optimizer – Market Positioning ChartsChart 1: Business Model Full-service Where Where wewe are are going now Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 119 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 122. 10.13. WebtrendsWebtrendshttp://www.webtrends.com/Company Proposition COMPANY OWNERSHIPWebtrends is the global leader in social and mobile analytics and is leading the way in multichannel Webtrends is a privately heldstrategic marketing, testing, segmentation and targeting, online advertisement measurement, mobile company. Once a publicapplication creation, and professional services. We are developing the most innovative marketing company on the NASDAQsolutions to help your organisation get the most out of your marketing investments and are pleased stock exchange, Franciscoto serve over half of the global and Fortune 500 organisations. Partners, one of the world‟sOur vision, roadmap and strategy have been focused on helping marketing leaders drive revenue largest technology-focusedthrough customer acquisition, engagement and retention, measuring marketing programme ROI, investment firms and anand improving sales and marketing workflow. We believe that we are the right partner to deliver organisation with over $5with the solutions that can easily be adopted to drive greater strategic value for your business. Wehave made significant investments in our solutions to provide best-in-class offerings that rapidly billion in capital underadapt to a changing marketplace, and we will continue to innovate in the best interests of our management, are now thecustomers. company‟s principal owners.Brief Company History CLIENTSWebtrends dates back to 1993. Webtrends went public in February of 1999, and was acquired by Webtrends counts Microsoft,NetIQ Corporation in March of 2001. In May of 2005, Webtrends was purchased from NetIQ by Domino‟s Pizza, 3M,Francisco Partners. In August of 2009, Webtrends acquired testing and optimisation services eMarketer,company Widemile. Webtrends employs over 350 individuals. MoneySupermarket.com,USP – What sets you apart from competitors? Phones4U, LateRooms, RSThe Webtrends Optimize platform provides a next generation intuitive interface and streamlined Components and others.user experience making the set-up and execution of tests, segmentation and customer targeting. OFFICE LOCATIONSIn addition to the intuitive, next generation platform, Webtrends provides comprehensive full Webtrends EMEAservice optimisation support including all of the essential roles: strategist, analyst, creative and web Headquartersdevelopment to help augment or extend a customer‟s organisational capabilities. Mallard CourtThe following is a high-level summary of the Webtrends Optimize advantages: Market Square Staines, Middlesex Universal Tag for Testing and Targeting – unified platform at back-end. TW18 4RH Specialised creative services support included as standard. United Kingdom Intuitive Platform designed for marketers, even if fully managed service offered – easy-to- +44 (0) 1784 415 700 understand QA/approvals and real-time reporting suite. Webtrends Seattle Office No performance issues (page load times). Puget Sound Plaza1325 Fourth Ave Transparent pricing model. Suite 1600 Self-service option with full training. Seattle, WA 98101Case study and/or testimonials United StatesClient Case Study: Microsoft Office Live Small Business HEAD OFFICESummary: Webtrends Optimize increases conversions for Microsoft Office Live Small Business ADDRESSthrough multivariate testing. Webtrends HeadquartersOverview: Webtrends executed a comprehensive optimisation programme for Microsoft Office Live 851 SW 6th Ave., Suite 1600Small Business, which included a new page layout design as part of a multi-step multivariate test. Portland, OR 97204Challenges: Though it is one of the fastest-growing small business services available with over United States850,000 customers, the Office Live Small Business team felt performance could improve further and 877-932-8736sought to understand specific elements that drive sign-ups. They partnered with Webtrends to CONTACTstrengthen click-through performance for traffic generated from PPC activities to the portal.Results: Hugh Kimber Overall, conversion rates improved more than 60% over the baseline control page. Regional Sales Manager The second multivariate test run improved results by introducing a product review quote from +44(0)1784 415720 the New York Times and honing the Call-to-Action Button to further boost conversions. Hugh.Kimber@webtrends.co Using a fractional factorial approach, test duration was 24 days for both runs. By comparison, a m full factorial approach would take 704 days. Multivariate Testing Buyer‟s Guide Page 120 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 123. Service ModelFull-service solution YesSelf-service solution YesHosted NoOther (please specify) We offer a combined managed-service and self-service offering which we call full-service.Other ServicesSegment targeting Yes(rules)Recommendationengine (automated Nomodel)Behavioural targeting Yes, but done through a rules based approach.(automated model)Other related services True personalisation through the combined Webtrends Optimize and Segments solutions. The(please specify) combined solution delivers the precision of Webtrends Optimize rules-based targeting with the rich profile-based segmentation of Webtrends Segments – the result is the most powerful onsite targeting solution available today.Type of Testing Methodology / TechnologyA/B Testing by URL YesA/B Testing usingmultiple content YesvariantsMVT - full factorial YesMVT - Fractional Yesfactorial/Taguchi Optimize proprietary Fractional Factorial approach leverages a highly tuned Design of ExperimentsMethod approach coupled with a standard predictive analytics engine to deliver actionable results in a fraction of the time. Our approach overcomes the inherent limitations of Taguchi adding an additional 200 plus test designs to better suit the needs of digital marketing optimisation. All of the test designs offered ensure balanced and orthogonal testing enabling results to provide accurate rather than directional conclusions. The solution also leverages predictive analytics to determine true optimals through Fractional Factorial testing – unlike some vendors that only expose presented experiments as optimals.Ability to cull poor Nocontent as test proceeds In order to provide the highest level of confidence and accuracy through Optimize testing, we embrace the scientific method which doesn‟t support the notion of changing tests in process. Note, users can simply stop a test, clone the test run, make any required modifications, and restart the test within a matter of minutes to remove underperforming elements. Users would then have a record of underperforming experiment/content.Other (please specify) N/A Multivariate Testing Buyer‟s Guide Page 121 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 124. Engine TypeClient-side (genericJavaScript tag across all Yespages)Client-side (single NoJavaScript tag per page)Client-side (code Noembedded in each page)Server-side NoInspection proxy NoOther (please specify) N/AInterfaceIs HTML knowledge Noneeded to define tests?Live preview of YesvariationsTest Segmentation MethodsIs the system able tolimit tests to specific Yesaudience segments?Inbound search YeskeywordsReferring URL YesLanding page URL YesQuery string parameter YesCookie parameter YesVisitor type YesTime returning YesVisit Metrics Yes(Time/duration)On-site search keywords YesBy device (e.g. iPad) YesOther (please specify) Any in-session attribute (e.g. IP, cookie, data object, etc.) can be leveraged for segmentation as can imported segment attributes. Attributes can be used individually or combined to create more sophisticated segmentation rules. Multivariate Testing Buyer‟s Guide Page 122 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 125. IntegrationWeb analytics YesCRM data YesE-commerce platform YesCMS YesOther (Please specify) N/ATypes of ReportingAbility to report against Yesmultiple goalsE-commerce conversion YesPage goal conversion YesBounce rate YesRevenue per page YesgenerationRevenue per visitor YesGross profit / gross YesmarginAverage order value YesCustom event YesconversionsAre reports available Yesonline?Other (please specify) N/APricingSubscription model Yesbased on server calls Optimize is tiered based on a server-call like approach called Optimize Events which only counts active,(please specify typical deployed experiments against the licensed tiers. Additionally, platform fees are usually bundled withamount or range) managed services in a full service contract. Contract value depends on client service requirements/needs.Subscription modelbased on flat licence fee N/A(please specify typicalamount or range)Performance-basedpricing (please specify N/Atypical amount orrange) Multivariate Testing Buyer‟s Guide Page 123 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 126. Set-up fee (pleasespecify typical amount N/Aor range)Fee per number of„seats‟ (please specify N/Atypical amount orrange)Consultant daily rate(please specify typical N/Aamount or range)Other (please specify) N/AServiceDedicated consultant / Yesaccount managerTelephone support desk YesOther (please specify) N/ANumber of staffspecifically dedicated tothe support of MVT Yes – we have full services teams in both Europe and North America dedicated to support customers.customers (bygeography)Please give details ofhow you charge (or N/A – support included.don‟t charge) forsupportAdditional Supporting InformationAny other relevant N/Ainformation Multivariate Testing Buyer‟s Guide Page 124 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011
  • 127. Webtrends – Market Positioning ChartsChart 1: Business Model Full-service Where Where we are we are now going Self-service We only do Range of MVT technologiesChart 2: Type of Company Large compared to competitors Where we are going Where we are now Small compared to competitors Single market Global focus focus Multivariate Testing Buyer‟s Guide Page 125 All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording or any information storage and retrieval system, without prior permission in writing from the publisher. Copyright © Econsultancy.com Ltd 2011

×