Multivariate DIY - Daigneault

1,076 views

Published on

If you're not doing multivariate (MVT) optimization tests on your landing pages, you could be leaving hundreds of thousands of dollars on the table.

But for those that have checked out MVT, all those tools and consultants can get expensive pretty darn quick. No matter how great the return on investment, sometimes there just isn't any money.

Enter MVT DIY-style! or, multivariate optimization "do it yourself" style. This session will show you how to use Google's free tools so that you can setup your own tests.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,076
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • M+R Powerpoint Template
  • M+R Powerpoint Template
  • M+R Powerpoint Template
  • M+R Powerpoint Template
  • M+R Powerpoint Template
  • Multivariate DIY - Daigneault

    1. 1. Multivariate Optimization Do-It-Yourself Edition [150] April 9, 2010 Steve Daigneault
    2. 2. Goals <ul><li>Solid understanding of multivariate optimization and why it’s so great </li></ul><ul><li>Overview of how Google’s free optimization tool works </li></ul><ul><li>Examples of recent tests and results </li></ul><ul><li>Resources and suggested next steps for you to run your own tests </li></ul>
    3. 3. What is multivariate optimization? <ul><li>We all know A/B tests </li></ul><ul><li>Create two versions </li></ul><ul><li>Split traffic </li></ul><ul><li>Watch response rates </li></ul>
    4. 4. Your landing page submit Trust Mrs. Peacock. She knows how to light candlesticks without killing anyone. Give to the Library
    5. 5. Things you want to test submit <ul><li>HEADLINE: 2 alternatives </li></ul><ul><ul><li>Save the Conservatory </li></ul></ul><ul><ul><li>Colonel Mustard Needs You </li></ul></ul>PICTURE: 2 alternatives <ul><li>FIELDS: 1 alternative </li></ul><ul><ul><li>Remove 1 field </li></ul></ul>Donate now! <ul><li>BUTTON: 1 alternative </li></ul><ul><ul><li>Bigger and orange </li></ul></ul>
    6. 6. How would you A/B test this? <ul><li>Set-up different landing pages for: </li></ul><ul><ul><li>Headline: 2 alternatives + 1 control = 3 </li></ul></ul><ul><ul><li>Picture: 2 alternatives + 1 control = 3 </li></ul></ul><ul><ul><li>Fields: 1 alternative + 1 control = 2 </li></ul></ul><ul><ul><li>Button: 1 alternative + 1 control = 2 </li></ul></ul><ul><li>3 x 3 x 2 x 2 = 36 pages </li></ul><ul><li>36 – 1 control = 35 new pages to build </li></ul>
    7. 7. <ul><ul><li>Test everything at once </li></ul></ul><ul><ul><ul><li>Input all elements at once </li></ul></ul></ul><ul><ul><ul><li>Software creates many page variations </li></ul></ul></ul><ul><ul><ul><li>Traffic split among page variations </li></ul></ul></ul><ul><ul><ul><li>Takes into account interactions between all the elements </li></ul></ul></ul><ul><ul><li>Fancy math tracks the effect each individual item has on conversions </li></ul></ul>Multivariate > A/B testing
    8. 8. HOW DOES OPTIMIZER WORK?
    9. 9. <ul><ul><li>Actually builds all page variations </li></ul></ul><ul><ul><ul><li>Some software will build fewer pages + use predictive modeling </li></ul></ul></ul><ul><ul><ul><li>Optimizer = free + great for basic tests </li></ul></ul></ul><ul><ul><li>Statistically significant results using the same traffic but using it more wisely </li></ul></ul><ul><ul><li>Optimizes conversions (not average gift*) </li></ul></ul>Google Optimizer
    10. 10. <ul><ul><li>Add Google Optimizer code around elements you want to test.* </li></ul></ul><ul><ul><li>Upload variations of those elements into Google Optimizer. </li></ul></ul><ul><ul><li>Optimizer then builds page variations, splits traffic and calculates changes in behavior. </li></ul></ul><ul><ul><li>*requires you have direct access to page code </li></ul></ul>Overview
    11. 11. <ul><ul><li>Choose the page you want to improve.* EX: donation page </li></ul></ul><ul><ul><li>Identify your conversion page.* EX: confirmation page </li></ul></ul><ul><ul><li>Identify page elements you want to test. EX: button and header image *these pages will get tracking code. </li></ul></ul>7 Steps to Test
    12. 12. <ul><ul><li>Create variations. EX: new graphic buttons and headers </li></ul></ul><ul><ul><li>Add Optimizer code around the elements you’re testing on your test page. </li></ul></ul><ul><ul><li>Upload the variations to those elements into Google Optimizer. </li></ul></ul><ul><ul><li>Review and launch. </li></ul></ul>7 Steps to Test
    13. 13. SHOW ME RESULTS
    14. 14. Sample results: Page variations
    15. 15. <ul><li>Estimated conversion rate range </li></ul><ul><li>provides the most immediate insight into overall performance. This column shows you how well each combination is performing relative to your original content. </li></ul>
    16. 16. <ul><li>Chance to beat original </li></ul><ul><li>the probability that a particular combination will be more successful than your original content. Above 95% (or below 5%) is considered significant. Above 99% (or below 1%) is considered “high confidence”. </li></ul>
    17. 17. <ul><li>Chance to beat all </li></ul><ul><li>the probability that a combination will be more the successful than all the other combinations running in the experiment. Usually a lower number because it’s competing against many page variations vs. just the original. </li></ul>
    18. 18. <ul><li>Observed improvement </li></ul><ul><li>displays the percent improvement over the original combination. </li></ul>
    19. 19. <ul><li>Conversions/impressions </li></ul><ul><li>the raw data of how many conversions and page views a particular combination generated. (You want 200+) </li></ul>
    20. 20. Sample results: Page elements
    21. 21. <ul><li>Relevance rating </li></ul><ul><li>shows how much impact a particular section has on your experiment. 0 means it had little to no impact. 5 means it was a major distinguishing factor. </li></ul>
    22. 22. RECENT TEST RESULTS
    23. 23. vs.
    24. 24. vs.
    25. 25. vs.
    26. 26. <ul><ul><li>#6: new nav and new button </li></ul></ul><ul><ul><li>original gift string </li></ul></ul><ul><ul><li>#5: new nav and new gift string </li></ul></ul><ul><ul><li>original button </li></ul></ul>Winner:
    27. 27. MORE ANSWERS
    28. 28. <ul><li>How much traffic do I need? </li></ul><ul><li>How many things can I test? </li></ul><ul><li>www.google.com/analytics/siteopt/siteopt/help/calculator.html </li></ul>Traffic Calculator
    29. 29. <ul><li>How much could improvements help? </li></ul>Value Calculator www.bplans.com/business_calculators/conversion_rate.cfm
    30. 30. Installation Guide https://www.google.com/analytics/siteopt/siteopt/help/install.html
    31. 31. Optimizer Community www.google.com/support/forum/p/websiteoptimizer?hl=en
    32. 32. Optimizer Engineer Blog www.gwotricks.com/default.htm
    33. 33. <ul><li>Frequently Asked Questions </li></ul><ul><ul><li>www.google.com/analytics/siteopt/siteopt/help/FAQ.html </li></ul></ul><ul><li>Technical overview gives you a deeper understanding of the tool </li></ul><ul><ul><li>www.google.com/analytics/siteopt/siteopt/help/techoverview.html </li></ul></ul><ul><li>Comparisons of different tools to help you decide if Google Optimizer is right for you </li></ul><ul><ul><li>www.whichmvt.com / </li></ul></ul>Resources
    34. 34. How Was this Session? Call In Text Online Call 404.939.4909 Enter Code 150 Text 150 to 69866 Visit nten.org/ntc-eval Enter Code 150 Tell Us and You Could Win a Free 2011 NTC Registration!

    ×