The document discusses different methodologies for online testing, including A/B testing, multivariate testing, and multivariate testing using the Taguchi method. It provides pros and cons of each approach and recommends best practices for online testing, such as clearly defining success metrics, aligning test scenarios with objectives, minimizing variables to ensure data integrity, involving stakeholders, and applying learnings to future tests. The goal is to use testing to optimize tactics like creative, messaging, and offers and quantify the effects of design changes on key metrics.

1. MULTIVARIATE
TESTING
2012 Planning

2. Confidential – Do Not Distribute
TESTING INTRODUCTION
• Marketers use testing to optimize the performance of their
chosen tactical vehicle
– Media
– Creative
– Messaging
– Offer
• Testing allows us to quantify the effects of design and user
interface iterations
– Does the “hard sell” copy really drive more sales?
– Is the Flash navigation causing users to abandon the homepage?
– Does the search box in the header drive more product views?
• Online channels present an efficient testing opportunity due to
the production efficiencies of the medium
– Relatively short lead time
– Accountable medium
2

3. Confidential – Do Not Distribute
THREE TESTING METHODOLOGIES
• A/B testing (Split run testing)
– Tests a single design variation per test
– Typically all users in the control group receive the baseline page and the users in the test group
receive the design variation
– Pros: Quick to deploy, simple test matrix, minimal resources need to be created
– Cons: Doesn’t scale well
• Multivariate testing
– This test involves testing multiple items as one time
– Pros: Can uncover success with multiple design elements combinations not tested with A/B, will
almost always demonstrate improvement over control group
– Cons: Doesn’t scale well. A small number of variables will cause the test matrix to quickly grow very
large. Also many resources must be produced to fill the iterations in the matrix.
• Multivariate testing using the Taguchi method
– Similar to multivariate testing but it only uses a subset of the possible test iterations
– Pros: Theoretically provides the optimal test strategy
– Cons: Can be expensive as external vendors are recommended for optimal knowledge transfer
3

4. Confidential – Do Not Distribute
ENSURE ACTIONABLE OBJECTIVES
1. Clearly identify success metrics before the test
– “Increase customer service” is too broadly defined
1. Stakeholders must agree in advance to the test objectives
– Ensure that all business owners understand the test and have given
input
1. Ensure the test results can be acted upon
– Don’t test variables that may be outside the control of the business
unit
4

5. Confidential – Do Not Distribute
ALIGN SCENARIOS WITH TEST OBJECTIVES
• Scenario Design and use case development should inform
testing objectives
• Utilize existing scenario’s or use cases to guide the test
– Testing scenarios are developed to test the effectiveness of a
campaign to increase applications for “Student Visa Platinum” from
the Student Banking Center landing page. BAC is specifically
interested in growing this business on the west coast and in
particular California.
– If the campaign is geared specifically towards to drive users in the
California market don’t test the effectiveness of the campaign for
the East Coast.
5

6. Confidential – Do Not Distribute
MINIMIZE VARIABLES TO ENSURE DATA INTEGRITY
• Based on the objectives of the test determine if the users who
will see the test are broadly representative of the overall
audience segment
• Ideally you can control the source of the traffic to the test via a
media vehicle
• Ensure a statistically reliable sample. This depends on the test
setup but is typically at least 200 responses
6

7. Confidential – Do Not Distribute
THE VENDOR LANDSCAPE
Depending on the complexity of the test we recommend working
with a partner to deal with the logistics of multivariate testing
1. Dedicated outsourced ASP based solution – Several vendors
dedicated to multivariate testing exist. They can provide the
infrastructure necessary for dealing with the complexities of
running a multivariate test.
2. Existing transactional systems – Many application servers have
testing modules that can be utilized. For example: ATG’s
Campaign Optimizer
3. Custom solution – This is the most flexible but also the least
efficient due to time and cost constraints
7

8. Confidential – Do Not Distribute
PRIVACY ISSUES
• Ensure that your privacy policy reflects the fact that you’re
performing testing:
– Additional cookies will likely be issued
– Multiple users will get different versions of the site for purposes of
the test. You want to avoid the scenario that Amazon ran into a few
years ago where they offered different prices to different users as
this was very unpopular with the general populous
8

9. Confidential – Do Not Distribute
TESTING IS AN ONGOING PROCESS
• Testing is not a one off project – It’s part of an ongoing
optimization cycle
• Learnings should be applied to future iterations of the test
• Ideally you can test variables that can be applied to a larger
body of work
9

10. Confidential – Do Not Distribute
PREPARE THE TEAM FOR ORGANIZATIONAL
CHALLENGES
• People may be resistant to testing as the results often invalidate
their assumptions
• Create a case study to illustrate the value to the team
• Strive to create a culture of testing
• Start with simple A/B testing to develop internal processes for
change management
10

11. Confidential – Do Not Distribute
TESTING BEST PRACTISES
• Start small
– Start off testing manageable elements – Don’t try to solve all your
problems on the first round of testing
• Test actionable data
– Testing will generate “best possible” success criteria – ensure that
you can enact the changes the test has proven most successful
• Get the creative teams involved early
– Waiting until the results are complete will alienate the create teams
• Test messaging, offers and user interface elements separately
– Depending on your test methodology be sure to test offers and
messaging in a separate test
• Apply your test learnings to future projects
– React to the information that the test has generated
– Use this information in inform future testing
11

12. Confidential – Do Not Distribute 12
DISCUSSION