Multivariate Testing at Orange

Driving more revenue online
Greg Kelton
Managing Director, Autonomy Optimost EMEA




    ...
Quiz!
          Why Multivariate Conversion Analysis
          Case Studies (Orange, Delta, etc.)
          How it works
 ...
1
    A
    A


Control




    B
    B


+75%




          3
Generating a 75% increase in Clickthroughs
             •      14 template permutations analyzed




Control (creative 2) ...
2        A
         A



     +6.97%




        B
        B


    Control




              5
Generating a 53.42% lift (99% confidence)
              Total Recipes Analysed: 2,938




                                ...
3


    Flash (animated)



    A
    A                  B
                       B
            Control        +4.1%


   ...
Winning creative beat control by 10.0% on UAR-Congratulations
and 8.2% on UAR-Profile Information

Over 1 billion template...
Who should control your marketing?

                  Shareholders/
                  Shareholders/
                    In...
Who should control your marketing?

                  Shareholders/
                  Shareholders/
                    In...
Who should control your marketing?
     Your marketing should be driven by your customers




                         You...
14
Quiz!
           Why Multivariate Conversion Analysis
           Case Studies (Orange, Delta, etc.)
           How it work...
The Content Selection Process is Complex & Faulty
How do you make web content decisions?

                          • Spec...
17
Old Facebook




               New Facebook




                              18
Top facebook Applications See 25
Percent Drop In Traffic Since
Redesign


facebook Poll: 94% Of Users
Don’t Like Redesign
...
20
22
23
24
25
Quiz!
           Why Multivariate Conversion Analysis
           Case Studies (Orange, Delta, etc.)
           How it work...
But…. What’s the point?

• Your online competitive advantage
   • Your Sales Funnel:
      • Ease of buying & customer exp...
Affiliate Marketing Page


Simplify the Buying Process


• Simple version increased
  conversion by 3.86%




            ...
Putting business in control of your online channel
                    Merchandising Challenge - Brands, price
           ...
Shop Home Page




                 35
Shop Home Page




       C

                 A, B




                        36
30 total Page Recipes analysed
Quiz!
                                             B   Control
        A   +17%




       ...
Product Page




               40
Product Page




               41
60 total Page Recipes analysed
Quiz!
        A   -9%                               B   +13%




                          ...
Quiz!
           Why Multivariate Conversion Analysis
           Case Studies (Orange, Delta, etc.)
           How it work...
1. Pick a Page to Test




                         48
2. Add Single Javascript Callout to Page


<script
optimost.displayModule("main");
</script>




                         ...
3. Brainstorm ideas to test




                       1



                 2
                               3
     vs


...
4. Optimost Generates Page Recipes




                                     52
5. Run the Experiment Live on your site!




                                           53
6. Optimost declares the Best Recipe




                         This Version Increases
      Original Version
          ...
7. REPEAT



            55
Summary – MVT Allows you to:

Listen to your Customers
Improve Revenue… consistently
Avoid Cost and Risk of Redesigns




...
Thank You!

  Greg Kelton
  gkelton@interwoven.com




                           58
Upcoming SlideShare
Loading in …5
×

Greg Kelton - "Multivariate Testing at Orange"

510
-1

Published on

Published in: Design
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
510
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Greg Kelton - "Multivariate Testing at Orange"

  1. 1. Multivariate Testing at Orange Driving more revenue online Greg Kelton Managing Director, Autonomy Optimost EMEA Interwoven Confidential
  2. 2. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 2 2
  3. 3. 1 A A Control B B +75% 3
  4. 4. Generating a 75% increase in Clickthroughs • 14 template permutations analyzed Control (creative 2) Winner (creative 11) UAR-Shop = x% UAR-Shop = x% +75% 4
  5. 5. 2 A A +6.97% B B Control 5
  6. 6. Generating a 53.42% lift (99% confidence) Total Recipes Analysed: 2,938 B4 F=1 Control generating a 53.42% lift (99% confidence) 6
  7. 7. 3 Flash (animated) A A B B Control +4.1% 7
  8. 8. Winning creative beat control by 10.0% on UAR-Congratulations and 8.2% on UAR-Profile Information Over 1 billion template permutations analyzed Control (creative 121) Winner (creative 271) UAR-Congratulations = x% UAR-Congratulations = x% +10.0% UAR-Profile Information = x% UAR-Profile Information = x% +8.2% 8
  9. 9. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 11
  10. 10. Who should control your marketing? Shareholders/ Shareholders/ Investors Investors Product Product Marketing Marketing Managers Managers Dept Dept None of the above Creative Creative CEO CEO Agencies Agencies Executives Executives Analysts Analysts 12
  11. 11. Who should control your marketing? Your marketing should be driven by your customers Your Your Customers Customers 13
  12. 12. 14
  13. 13. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 15 15
  14. 14. The Content Selection Process is Complex & Faulty How do you make web content decisions? • Speculation • HiPPO • Intuition • Agency experience • Argument • Quasi-objective • Focus groups What if Your Customers • Usability testing Could Tell You What • Web Surveys Content They Like? 16
  15. 15. 17
  16. 16. Old Facebook New Facebook 18
  17. 17. Top facebook Applications See 25 Percent Drop In Traffic Since Redesign facebook Poll: 94% Of Users Don’t Like Redesign 19
  18. 18. 20
  19. 19. 22
  20. 20. 23
  21. 21. 24
  22. 22. 25
  23. 23. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 29 29
  24. 24. But…. What’s the point? • Your online competitive advantage • Your Sales Funnel: • Ease of buying & customer experience? • Multi-channel influences? • Technology investment (Flash, AJAX, Adobe AIR?) • Your Operational Advantage: • Change management? • Cost reduction? • Putting business in control of your site? • Speed to market? • Your Brand & Offering: • Regional customers • Price sensitivity? By region? By demographic? 30
  25. 25. Affiliate Marketing Page Simplify the Buying Process • Simple version increased conversion by 3.86% 32
  26. 26. Putting business in control of your online channel Merchandising Challenge - Brands, price brackets, colours vary depending on product category •How many price brackets? •Should the number of products in stock be displayed? •How many brands? •Which order is best? •Are collapsible menus preferred? Benefit –understand how your customers prefer to interact with site technologies eg. reviews, search, navigation, shopping carts. 33
  27. 27. Shop Home Page 35
  28. 28. Shop Home Page C A, B 36
  29. 29. 30 total Page Recipes analysed Quiz! B Control A +17% 39
  30. 30. Product Page 40
  31. 31. Product Page 41
  32. 32. 60 total Page Recipes analysed Quiz! A -9% B +13% 43
  33. 33. Quiz! Why Multivariate Conversion Analysis Case Studies (Orange, Delta, etc.) How it works Q&A Slide 47 47
  34. 34. 1. Pick a Page to Test 48
  35. 35. 2. Add Single Javascript Callout to Page <script optimost.displayModule("main"); </script> 49
  36. 36. 3. Brainstorm ideas to test 1 2 3 vs None 4 51
  37. 37. 4. Optimost Generates Page Recipes 52
  38. 38. 5. Run the Experiment Live on your site! 53
  39. 39. 6. Optimost declares the Best Recipe This Version Increases Original Version Conversions by 28.4% 54
  40. 40. 7. REPEAT 55
  41. 41. Summary – MVT Allows you to: Listen to your Customers Improve Revenue… consistently Avoid Cost and Risk of Redesigns 57
  42. 42. Thank You! Greg Kelton gkelton@interwoven.com 58
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×