Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

December 2009, multivariate testing


Published on

I took the initial steps toward Multivariate testing across all media, not just the digital business. This kicked off the planning process

Published in: Business
  • Be the first to comment

  • Be the first to like this

December 2009, multivariate testing

  1. 1. MULTIVARIATE TESTING 2012 Planning
  2. 2. Confidential – Do Not Distribute TESTING INTRODUCTION • Marketers use testing to optimize the performance of their chosen tactical vehicle – Media – Creative – Messaging – Offer • Testing allows us to quantify the effects of design and user interface iterations – Does the “hard sell” copy really drive more sales? – Is the Flash navigation causing users to abandon the homepage? – Does the search box in the header drive more product views? • Online channels present an efficient testing opportunity due to the production efficiencies of the medium – Relatively short lead time – Accountable medium 2
  3. 3. Confidential – Do Not Distribute THREE TESTING METHODOLOGIES • A/B testing (Split run testing) – Tests a single design variation per test – Typically all users in the control group receive the baseline page and the users in the test group receive the design variation – Pros: Quick to deploy, simple test matrix, minimal resources need to be created – Cons: Doesn’t scale well • Multivariate testing – This test involves testing multiple items as one time – Pros: Can uncover success with multiple design elements combinations not tested with A/B, will almost always demonstrate improvement over control group – Cons: Doesn’t scale well. A small number of variables will cause the test matrix to quickly grow very large. Also many resources must be produced to fill the iterations in the matrix. • Multivariate testing using the Taguchi method – Similar to multivariate testing but it only uses a subset of the possible test iterations – Pros: Theoretically provides the optimal test strategy – Cons: Can be expensive as external vendors are recommended for optimal knowledge transfer 3
  4. 4. Confidential – Do Not Distribute ENSURE ACTIONABLE OBJECTIVES 1. Clearly identify success metrics before the test – “Increase customer service” is too broadly defined 1. Stakeholders must agree in advance to the test objectives – Ensure that all business owners understand the test and have given input 1. Ensure the test results can be acted upon – Don’t test variables that may be outside the control of the business unit 4
  5. 5. Confidential – Do Not Distribute ALIGN SCENARIOS WITH TEST OBJECTIVES • Scenario Design and use case development should inform testing objectives • Utilize existing scenario’s or use cases to guide the test – Testing scenarios are developed to test the effectiveness of a campaign to increase applications for “Student Visa Platinum” from the Student Banking Center landing page. BAC is specifically interested in growing this business on the west coast and in particular California. – If the campaign is geared specifically towards to drive users in the California market don’t test the effectiveness of the campaign for the East Coast. 5
  6. 6. Confidential – Do Not Distribute MINIMIZE VARIABLES TO ENSURE DATA INTEGRITY • Based on the objectives of the test determine if the users who will see the test are broadly representative of the overall audience segment • Ideally you can control the source of the traffic to the test via a media vehicle • Ensure a statistically reliable sample. This depends on the test setup but is typically at least 200 responses 6
  7. 7. Confidential – Do Not Distribute THE VENDOR LANDSCAPE Depending on the complexity of the test we recommend working with a partner to deal with the logistics of multivariate testing 1. Dedicated outsourced ASP based solution – Several vendors dedicated to multivariate testing exist. They can provide the infrastructure necessary for dealing with the complexities of running a multivariate test. 2. Existing transactional systems – Many application servers have testing modules that can be utilized. For example: ATG’s Campaign Optimizer 3. Custom solution – This is the most flexible but also the least efficient due to time and cost constraints 7
  8. 8. Confidential – Do Not Distribute PRIVACY ISSUES • Ensure that your privacy policy reflects the fact that you’re performing testing: – Additional cookies will likely be issued – Multiple users will get different versions of the site for purposes of the test. You want to avoid the scenario that Amazon ran into a few years ago where they offered different prices to different users as this was very unpopular with the general populous 8
  9. 9. Confidential – Do Not Distribute TESTING IS AN ONGOING PROCESS • Testing is not a one off project – It’s part of an ongoing optimization cycle • Learnings should be applied to future iterations of the test • Ideally you can test variables that can be applied to a larger body of work 9
  10. 10. Confidential – Do Not Distribute PREPARE THE TEAM FOR ORGANIZATIONAL CHALLENGES • People may be resistant to testing as the results often invalidate their assumptions • Create a case study to illustrate the value to the team • Strive to create a culture of testing • Start with simple A/B testing to develop internal processes for change management 10
  11. 11. Confidential – Do Not Distribute TESTING BEST PRACTISES • Start small – Start off testing manageable elements – Don’t try to solve all your problems on the first round of testing • Test actionable data – Testing will generate “best possible” success criteria – ensure that you can enact the changes the test has proven most successful • Get the creative teams involved early – Waiting until the results are complete will alienate the create teams • Test messaging, offers and user interface elements separately – Depending on your test methodology be sure to test offers and messaging in a separate test • Apply your test learnings to future projects – React to the information that the test has generated – Use this information in inform future testing 11
  12. 12. Confidential – Do Not Distribute 12 DISCUSSION