Can I Test More Than One Variable
at a Time?
Statisticians answer some of the most commonly asked A/B
testing questions
We’re sharing on Twitter!
#Webclinic
Confusion around online testing
• Designing and managing
experiments is listed as the
most significant challenge
The MarketingExperiments Library
Business Software Suite
#1 On-Demand. 6459+ World Clients
Award-Winning Solution. Free Tr...
Today’s Team
Bob Kemper
Senior Director, Sciences
As Senior Director of Sciences, Bob manages enterprise experimentation a...
Today’s Team
Benjamin Filip
Manager, Data Sciences
Ben participates in high-level marketing analytics and optimization inc...
Today’s Team
Paul Cheney
Editorial Analyst
Paul works directly with Flint McGlaughlin to study, manage and catalogue the
M...
Today, we are going to answer some of the most
common questions marketers ask about online testing.
QUESTION #1: Can I test more than
one variable at a time?
How many variables were changed?
Treatment
Logo
Logo
Logo
Control
Logo
increase in conversion
Through a more professional ...
Logo
Control
Logo
Treatment
How many variables were changed?
increase in conversion
By clarifying the value, reducing fric...
How many variables were changed?
Control Treatment
increase in conversion
By clarifying the value proposition and simplify...
QUESTION #1: Can I test more than
one variable at a time?
When to test more than one variable
Radical Redesigns: The objective is to
challenge the control enough to generate a
sign...
What you need to understand
1. We should design our tests so they are useful.
2. A variable can be anything the test desig...
QUESTION #2: What is multivariate
testing and when should I use it?
Definition: Multivariate testing
Multivariate Testing
A number of approaches for the simultaneous testing of multiple fact...
1. Multivariate testing (in marketing) is simply an automated way to run many A/B
tests at once.
2. Consider using multiva...
QUESTION #3: How do I test if I
have low traffic?
1. Testing with lower traffic levels will always require a trade-off between time,
budget or certainty.
2. With that in mi...
QUESTION #4: How do I determine
what to test?
How do you determine what to test?
Using a clear optimization and
testing strategy is the least
popular method for determi...
QUESTION #4: How do I determine
what to test?
Where to test in a path?
The Channel
(Ocnn)
The Presentation
(Oprn)
The Product Offer
(Opr)
Opr > Oprn > Ocnn
Wherein:
Opr...
What to test on a page?
Wherein:
“C” = Probability of conversion
“m” = Motivation of user
“v” = Force of the value proposi...
Background: A website that sells retail and wholesale collector items.
Goal: To increase conversion rate.
Primary Research...
• When we analyzed the metrics, we realized
there were leaks throughout the checkout
process. The credit card submission p...
What might be causing the fallout?
• It is unclear why the credit card is
required when payment method is
different.
• The...
How we addressed the issues:
• Third-party security indicators have
been added.
• Clearer explanation of why a credit
card...
Design Conversion Rate
Control 82.33%
Treatment 86.04%
Relative Difference 4.51%
5% Increase in Total Conversion
The new c...
1. Knowing where to test requires knowing your data.
• As a general rule, you can use: Opr > Oprn > Ocnn
2. Knowing what t...
QUESTION #5: How do I know my
test is valid?
How do I know my test is valid?
• Only 15% of marketers
monitor test validity.
QUESTION #5: How do I know my
test is valid?
1. Use a standardized statistical method for determining validity.
2. There are four main validity threats marketers shoul...
QUESTION #6: When should I stop
a test?
When to stop a testLevelofConfidence
Samples Collected
Stop Test Here
95%
Required Samples
1. It is safe to stop a test when your data meet two criteria:
a) Sufficient sample size
b) Desired level of confidence
Wh...
QUESTION #7: What is the best
testing tool?
5 Factors to Consider
1. Budget
2. Testing objectives
3. Flexibility
4. Ease of use
5. In-house expertise
What you need to understand
Consider 5 factors when evaluating testing platforms:
a) Budget
b) Testing objectives
c) Flexi...
SAVE $100
Discount Code: 481-OC-3007
MECLABS.com/OnlineTesting
Additional Questions
MECLABS Research Partnership Opportunities
MECLABS conducts rigorous experiments in the new science of optimization.
We ap...
Upcoming SlideShare
Loading in...5
×

Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions

920

Published on

A/B testing on the Web has become incredibly sophisticated in the last few years. New software makes it easier than ever to have a test up and running on your site. Still, a software program can only take you so far, and many marketers find themselves with questions.

In our next Web clinic, statisticians and testing experts from the MECLABS research lab will be answering some of the most common questions associated with online testing:

• Can I test more than one variable at a time?
• What is a multivariate test?
• Is a multivariate testing better than an A/B split test?
• Which page element(s) should I test?

Published in: Business, Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
920
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
32
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions

  1. 1. Can I Test More Than One Variable at a Time? Statisticians answer some of the most commonly asked A/B testing questions
  2. 2. We’re sharing on Twitter! #Webclinic
  3. 3. Confusion around online testing • Designing and managing experiments is listed as the most significant challenge
  4. 4. The MarketingExperiments Library Business Software Suite #1 On-Demand. 6459+ World Clients Award-Winning Solution. Free Trial www.XXXXXXXXXX.com/Business 54%Conversion 21%Conversion 97%Conversion 109%Conversion 31%Conversion 124%Conversion 201%Conversion 29%Conversion 15%Conversion 100%Conversion 21%Conversion137%Conversion 63%Conversion Over 15 years of research, spanning more than 10,000 pages/paths and 1 billion emails
  5. 5. Today’s Team Bob Kemper Senior Director, Sciences As Senior Director of Sciences, Bob manages enterprise experimentation and discovery processes for the corporation and for the individual operating companies. He serves on the Curriculum and Alumni boards for the Executive MBA program of the Davis College of Business at Jacksonville University, and on the board of the Jacksonville chapter of the National Association of Business Economists (NABE).
  6. 6. Today’s Team Benjamin Filip Manager, Data Sciences Ben participates in high-level marketing analytics and optimization including profit analysis, data mining, customer profile analysis and Web metrics analysis to determine visitor behaviors in online marketing funnels.
  7. 7. Today’s Team Paul Cheney Editorial Analyst Paul works directly with Flint McGlaughlin to study, manage and catalogue the MECLABS library of case studies and experiments. Essentially, he spends most of his day scouring over reports and data from the experiments that come through the lab.
  8. 8. Today, we are going to answer some of the most common questions marketers ask about online testing.
  9. 9. QUESTION #1: Can I test more than one variable at a time?
  10. 10. How many variables were changed? Treatment Logo Logo Logo Control Logo increase in conversion Through a more professional design and a clearer process, the treatment generated 58% more conversions. 58%
  11. 11. Logo Control Logo Treatment How many variables were changed? increase in conversion By clarifying the value, reducing friction and mitigating anxiety, the treatment produced 262% more leads. 262%
  12. 12. How many variables were changed? Control Treatment increase in conversion By clarifying the value proposition and simplifying the form, the treatment generated 189% more leads. 189%
  13. 13. QUESTION #1: Can I test more than one variable at a time?
  14. 14. When to test more than one variable Radical Redesigns: The objective is to challenge the control enough to generate a significant difference Focused variable clusters: The objective is to test the highest performing variables and increase channel specificity Single variable testing: The objective is to determine relative impact by isolating variables. Optimal Testing Cycle
  15. 15. What you need to understand 1. We should design our tests so they are useful. 2. A variable can be anything the test designer defines. 3. First, define variables that test the category to “get in the zone.” 4. From there, focus on relative impact of individual page elements.
  16. 16. QUESTION #2: What is multivariate testing and when should I use it?
  17. 17. Definition: Multivariate testing Multivariate Testing A number of approaches for the simultaneous testing of multiple factors (variables), each having two or more possible levels (values) to determine which composite treatment (combination of factor values) yields the best overall performance. Value 1 of 4 Value 1 of 4 Val ue 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Val ue 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Valu e 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Valu e 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 4 Value 1 of 2 Value 1 of 2 Value 1 of 2 Variable Variable Variable • Each combination is presented to a portion of arriving traffic. • The best combination is detected. • Other insights about the relative ‘importance’ and ‘variable interaction’ are also possible.
  18. 18. 1. Multivariate testing (in marketing) is simply an automated way to run many A/B tests at once. 2. Consider using multivariate testing when: • You have a large amount of traffic. • You can interpret complicated test results. • You have the time to run a longer test. • You have already tested your way into the best offer category. What you need to understand
  19. 19. QUESTION #3: How do I test if I have low traffic?
  20. 20. 1. Testing with lower traffic levels will always require a trade-off between time, budget or certainty. 2. With that in mind, there are a few potential ways you can solve the low traffic problem: • Test in the channel. • Lower your desired level of confidence. • Design more radical treatments. What you need to understand
  21. 21. QUESTION #4: How do I determine what to test?
  22. 22. How do you determine what to test? Using a clear optimization and testing strategy is the least popular method for determining what to test.
  23. 23. QUESTION #4: How do I determine what to test?
  24. 24. Where to test in a path? The Channel (Ocnn) The Presentation (Oprn) The Product Offer (Opr) Opr > Oprn > Ocnn Wherein: Opr = Optimization of Product Offer Oprn = Optimization of Presentation Ocnn = Optimization of Channel
  25. 25. What to test on a page? Wherein: “C” = Probability of conversion “m” = Motivation of user “v” = Force of the value proposition “i” = Incentive (additional) to take action “f” = Friction elements present “a” = Anxiety elements present C = 4m + 3v + 2(i - f) - 2a ©
  26. 26. Background: A website that sells retail and wholesale collector items. Goal: To increase conversion rate. Primary Research Question: Which version of the second step in the conversion funnel will produce the highest conversion rate? Approach: A/B variable cluster split test that focused on reducing anxiety through credibility indicators, copy and re-organization of existing page elements. Experiment ID: (Protected) Location: MarketingExperiments Research Library Test Protocol Number: TP1305 Research Notes:  Experiment: Background
  27. 27. • When we analyzed the metrics, we realized there were leaks throughout the checkout process. The credit card submission page stood out as a low cost opportunity for immediate return. • When we analyzed the metrics even further, we saw this step also had the highest lost revenue per cart (more than double any other step). • From this, we hypothesized optimizing this step would have the highest potential return on our efforts. Fallout Report: New Customers Experiment: Background
  28. 28. What might be causing the fallout? • It is unclear why the credit card is required when payment method is different. • The complexity of the Purchase Agreement Terms causes confusion and concern. • There is no indication that the credit card information is secure. Control Experiment: Control C = 4m + 3v + 2(i - f) - 2a ©
  29. 29. How we addressed the issues: • Third-party security indicators have been added. • Clearer explanation of why a credit card is required and that it will not be charged. • “Satisfaction Guaranteed” promise is emphasized. Treatment Experiment: Treatment C = 4m + 3v + 2(i - f) - 2a ©
  30. 30. Design Conversion Rate Control 82.33% Treatment 86.04% Relative Difference 4.51% 5% Increase in Total Conversion The new credit card page increased conversion by 4.51%  What you need to understand: While it might seem like a small increase, choosing this specific step in the sales funnel to test resulted in a projected $500,000+ increase in revenue per year. This underscores the potential impact of a properly identified research question. Experiment: Results
  31. 31. 1. Knowing where to test requires knowing your data. • As a general rule, you can use: Opr > Oprn > Ocnn 2. Knowing what to test requires a systematic methodology. • As a general rule, you can use: C = 4m + 3v + 2(i-f) - 2a What you need to understand
  32. 32. QUESTION #5: How do I know my test is valid?
  33. 33. How do I know my test is valid? • Only 15% of marketers monitor test validity.
  34. 34. QUESTION #5: How do I know my test is valid?
  35. 35. 1. Use a standardized statistical method for determining validity. 2. There are four main validity threats marketers should be conscious of when testing: a. Sample Distortion b. Selection Effect c. History Effect d. Instrumentation Effect 3. Beware of only relying on a testing tool for validity. What you need to understand
  36. 36. QUESTION #6: When should I stop a test?
  37. 37. When to stop a testLevelofConfidence Samples Collected Stop Test Here 95% Required Samples
  38. 38. 1. It is safe to stop a test when your data meet two criteria: a) Sufficient sample size b) Desired level of confidence What you need to understand
  39. 39. QUESTION #7: What is the best testing tool?
  40. 40. 5 Factors to Consider 1. Budget 2. Testing objectives 3. Flexibility 4. Ease of use 5. In-house expertise
  41. 41. What you need to understand Consider 5 factors when evaluating testing platforms: a) Budget b) Testing objectives c) Flexibility d) Ease of use e) In-house expertise
  42. 42. SAVE $100 Discount Code: 481-OC-3007 MECLABS.com/OnlineTesting
  43. 43. Additional Questions
  44. 44. MECLABS Research Partnership Opportunities MECLABS conducts rigorous experiments in the new science of optimization. We apply our discoveries to help leaders optimize the financial performance of their sales and marketing programs. Learn more about how you may be a fit for a MECLABS research partnership: • Select Research Partnership Opportunities on the post- webinar survey • Contact us directly info@MECLABS.com 1-877-635-0565 x
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×