From good to great beijing WAA


Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • We have one of the largest websites in the worldLast Year, we had…1Billion VisitsFrom Half a Billion VisitorsWho viewed 8 Billion PagesWe Have ComplexityDell has an online presence in over 100 countries and is translated into 34 languages4 Global Business UnitsThousands of System Configurations Thousands of Online Stores
  • Focus on how centralized, global online platform, implementation, vendor management, allow BU’s to collaborate more productively.
  • Talk about having different calculations in different BU’s for same metrics. Used SiteCatalyst implementation to standardize. Dealt with resistance during implementation – would not implement with different definitions for different BU’s.
  • Customized Personal Experience: Not only personalization and behavioral targeting driven transactions, but… Customer choice focused ; Intent & context based Multi-dimensional ; Cross-platformCustomized social experience: Not only social media integration and ratings & reviews, but…Embedded in broader experienceContextual; Cross-platformCustomized Solution Experience: Not only rich product content and bundles, but… Configurable solutions; Global solutions development platform; Cloud
  • When we formed our global team and got better at sharing information with each other, we started to learn things that have made us much better at getting the most out of our testing program. I’d like to take a few minutes to share with you some of the most important lessons we’ve learned.A while back, we weren’t very organized or focused, we just tested a lot of pages.We figured, the more pages we test, the more winners we’ll find, and we’ll improve our site faster. But we noticed that our most impactful tests throughout the world tended to be on the same pages – just a few for each country.We began to realize that we would be better off testing a few pages multiple times throughout the quarter than to test as many pages as possible.Another strategic focus area for us is basket and checkout pages.Even though these pages get a tenth of the volume of our most visited pages, they have the most revenue going through them. That shouldn’t be surprising because every purchase must go through these pages.The other reason these are valuable is that this is where we have are highest quality customers. This is especially true for Dell because we have a more detailed shopping experience than most stores. Many of our customers spend a lot of time choosing components and accessories to build the system that is exactly what they want. Customers who are willing to do this are seriously considering purchasing. Some of our greatest successes recently has been in cart and checkout tests.Our last focus area is behavioral targeting. The reason this is important to us is that we have a very high success rate with targeting.In fact, most of the targeting campaigns we try are successful.Whenever we focus on a customer’s needs and show them relevant content, they have a better experience and are more likely to buy.
  • Its not enough just to know where on your site to test and which pages to focus on.You need to know what you should try – what you should change and how to change it.A common mistake by marketers is to think too much about what we want our customers to do. We’ll design a site around the way we want them to shop.It’s a good idea to turn that around and think about why your customer is on your site and exactly what they want to do. That’s an easy concept, but in reality it is very complicated because we all have so many different kinds of customers.A page that is designed perfectly for one customer might not meet the needs of another customer.Here’s one example of this from our site:Customers who are brand loyal and come to our site with the intention of purchasing expect a simple shopping experienceDon’t want a sales pitch, just want to see the product offersOther visitors are open to buying from Dell or one of our competitorsThey want to take their time making a decision and want reasons why Dell is better than our competitorsThey would prefer a site with clear navigation that makes it easy to find the information they are looking forIf we tested these two options against each other, one would likely come out on topSuppose the option on the left came out with a 3% advantage in revenue per visitorThat would be good, but doesn’t tell the whole storyIf we dig deeper we find that the option on the left works 10% better for Efficient Shoppers but 7% worse for our Cautious ShoppersThe best solution is to use a targeting campaign to provide the best experience to both group.Target Cautious Shoppers based on:Clicks on Help Me Choose and other learning contentView of customer reviewsVideosOther content that suggests a high level of customer engagement with the siteA great way to learn what works best for different types of customers is to do some segmentation analysis on your test results to see if the results vary for different kinds of customers.Sometimes a simple A/B test can result in a very successful targeting campaign. At the very least, you’ll learn more about your various customers, which will help you design better tests in the future.
  • I wanted to show you a really simple of example of targeting we do to show our customers more relevant content based on what we know about them.This is our homepage in the United Kingdom. When a new customer comes to this page, they see a rotational banner with products across all of our segments, including consumer and business products.However, once a customer browses one of our customer segment sites, when they return to the country homepage, they will only see banners related to that segment.For example, if a visitor views pages in the Consumer site then returns, they will see banners featuring our consumer brands. Our Small and Medium Business customers will see products that would be relevant to them.We’ve done similar things with banners inside our customer segment sites. For example targeting specific product families based on prior browsing behavior.You should look to target based on any criteria you find that seems to differentiate the way customers shop. That might be clues from prior browsing history, traffic source, first time visitors vs repeat visitors, or based on prior purchases.The bottom line is, if we know what content would be relevant to a certain type of customer, we should use targeting campaigns to make them feel like the site is designed around their needs.
  • One of the most significant changes to our test process is what we do with successful tests.Previously, if we had a successful test, we would implement the winner, have a short celebration, and then move on to some other idea.We now realize that we were missing out on a huge opportunity to drive more value out of every test we run.There are now two additional things we do every time we have a successful test.First, we take the things we learned from a test and attempt to design a new test that will perform even better.Even with a test that has positive lift, further analysis can reveal some things about the design that might not have worked like we intended. So, we’ll redesign the test and run it again to see if we can do even better.A good example of this was when we ran a series of tests related to adding a trustmark in our checkout pages.Our early tests revealed that we had an idea that worked, but we continued to test different trustmarks and different locations until we found a solution that worked best.The final solution had more than double the lift that we saw in our first successful trustmark test.The second thing we do with our winners is that we run similar tests in most or all of our key markets throughout the world. Once we found the best Trustmark solution in the US, we tested it in our other major markets.In essence, instead of running a bunch of stand-alone tests, we think of our testing program as an iterative process for providing a global solution.From a single test, we generate many new test ideas and designs, and we take our best ideas and test them throughout the world.
  • Not only are we trying to get more out of our successful tests, but we now try to learn as much as possible from our mistakes as well.This happened in our checkout testing program when we tested the use of a simplified cart. We learned to not change too much at once.This slide illustrates the concept of the test.The hypothesis of this test was sound: We believed that customers would prefer a simplified cart with as little text as possible.So, we designed a test that removed or simplified any content we felt was unnecessary. Then we ran an A/B test.We were confident this test would work, so we were surprised when we found out the new recipe was performing worse than the original.It appeared that we removed some content that our customers thought was helpful. But we changed so much, how could we know which changes were positive and which were negative.Fortunately, in this case we got lucky. There was a change that caused significant negative performance that involved navigation, so we were able to detect it through some detailed pathing analysis of our test results. The change that hurt us was one of the smallest changes that we made, at least visually. We moved the Continue Shopping link from the top of the page next to the continue button to the very bottom of the page.This caused a frustrating shopping experience for customers who wanted to add other product to their cart, or go back to do more research before finalizing their order.We learned a couple of good lessons from this testFirst, if you are going to make a change that could significantly impact the way a customer browses your site – like changes to links to other pages, it is best to test those independently. Moving this link didn’t really fit with the original hypothesis of this test.Second, if you want to understand the impact of changing lots of different elements on the page, it is best to set up the test so that you can understand the impact of each change independently. Either multivariate test or an A, B, N test
  • In closing, I’d like to say that even if you have the right resources in place, the smartest people, and great processes, getting the most out of your testing program takes time. It will take a lot of failures and successes for you to learn the things you need to know to optimize your site. So be patient and stay committed to it, and it will definitely pay off.
  • From good to great beijing WAA

    1. 1. From Good to GreatHow web analytics & testing is evolving at Dell<br />Web Analytics Wednesday<br />Beijing, China<br />Dec. 22nd, 2010<br />
    2. 2. Agenda<br />2<br />Overview<br />Best-in-class analytics & testing<br />What’ve learned?<br />
    3. 3. Overview<br />3<br />
    4. 4. Our journey started more than a decade ago…<br />4<br />Early Adoption <br />Standardization<br />Maturity<br />Regional implementation<br />Global implementation<br />Standard <br />KPI’s – global dashboard<br />Regional KPI<br />1:1 BT experience <br />Industry leading testing capability<br />Internal testing capability<br />Full funnel testing<br />2008-2009<br />2010-<br />1996-2007<br />In-house clickstream tool<br />Advanced segmentation<br />VOC solution<br />Global team across BU<br />Regional team<br />Global team by BU<br />Limited expertise and talent bench<br />Work closely with vendors & consultants<br />Grow in-house expertise<br />Low<br />
    5. 5. Online:<br /> + Premier + Global Portal + B2B + Channel Portal + eSupport + Community<br />725K B2Btransactionsper year<br />34 supported languages<br />Googled 1M timesper day<br />170M visitsto community properties<br />58Kchannelpartnersonline<br />1Bvisits per year to<br />190KPremierPages<br />13Bpage viewsin FY’09<br />Dell Online Today<br />5<br />An online orderis placedevery2 seconds<br />
    6. 6. Public<br />Services<br />SMB<br />Large<br />Enterprise<br />CSMB<br />Consumer<br />Scale & Complexity Bring Opportunity<br />6<br />Global Online Team<br />
    7. 7. Centralized Analytics Operations<br />7<br />Tool Strategy/Support<br />Global Online Team<br />Governance<br />Implementation<br />BU / IT Partnering<br />Vendor Management<br />Training / Adoption<br />Global Online PlatformDynamic, Scalable & ReliableeCommerce Platform<br />
    8. 8. 8<br />100+ Report Suites. <br />1 Solution Design.<br />Confidential<br />
    9. 9. Define “Visits”<br />
    10. 10. Define “Visits”<br />Visits (all visits to site)<br />Segment Visits<br />SnP Visits<br />Cart<br />Browse<br />Accessories<br />Config<br />Support Visits<br />Store Visits<br />Checkout<br />Communities<br />Search<br />Support<br />Cart Visits<br />?<br />
    11. 11. Best-in-class Analytics & Testing<br />
    12. 12. 12<br />Strong back-office support<br />Good people<br />Our Strengths<br />Great technology<br />Committed to testing<br />
    13. 13. 13<br />“GOOD IS THE ENEMY OF GREAT”<br />Jim Collins<br />
    14. 14. 14<br /><br />Online Analytics, Testing & Targeting<br />To be thetechnologye-commerce leaderby offering the mostrelevant experienceand solutions todrive high customerloyalty<br />To become an <br />innovation engine through insightful analytics &the best-in-class testingcapability to enable constant and rapid changes<br />
    15. 15. People<br />15<br />Double the size of the team<br />On-boarding process to provide support and create connections<br />On-going training & development plan<br />If I were running a company today, I would have one priority above all others: to acquire as many of the best people as I could. <br />Jim Collins<br />
    16. 16. Team<br />Global Analytics<br />Global Testing<br />PMO<br />eDell Programs<br />Reporting<br />16<br />
    17. 17. Accountability<br />17<br />2x Revenue upside Y/Y<br />400-600 tests globally<br />Monetization model based on test results with statistical significance<br />Complete transparency & alignment with finance & business<br />Because they lack accountability, they fail to achieve credibility, and they have no authenticity.<br />Jim Collins<br />
    18. 18. Process<br />18<br />pipeline<br />roadmap<br />Brainstorm Recipes<br />Institutionalize learning<br />implement<br />Communicate<br />Analytics Manager Lead<br />Test Manager Lead<br />Sign off required<br />Documentation required<br />
    19. 19. Lesson Learned<br />19<br />
    20. 20. Focusing on Strategic Areas<br />20<br />Key Pages<br />Cart and Checkout<br />Targeting<br />
    21. 21. Understanding Customer Intent<br />21<br />Specific product offers for customers desiring an efficient shopping experience<br />Improved site navigation for customers want to take more time to find the right product<br />
    22. 22. Targeting Relevant Content<br />22<br />Consumer Customers<br />Small & Medium Business Customer<br />
    23. 23. Checkout Testing - July<br />Checkout Testing - August<br />Checkout Testing - Sept<br />CA Cons<br />US Cons<br />UK Cons<br />US Cons<br />DE Cons<br />US SMB<br />MX Cons<br />CN Cons<br />US Cons<br />US Cons<br />US SMB<br />JP Cons<br />AU Cons<br />CN Cons<br />CN SMB<br />JP SMB<br />AU SMB<br />MX SMB<br />BR Cons<br />CA SMB<br />US Cons<br />US Cons<br />AU Cons<br />FR Cons<br />US Cons<br />MX SMB<br />BR SMB<br />US Cons<br />AU SMB<br />Testing Globally…an Iterative Process<br />23<br />
    24. 24. Understand Why Something Works (or Doesn’t)<br />24<br />Simplified Cart Test – We changed too much<br />vs.<br />
    25. 25. Wrap it Up<br />25<br />
    26. 26. Embracing a New Culture:testing as a “religion”<br />26<br />Testing baked into projects from beginning<br />Tests are run rigorously with clearly defined process<br />Results captured and shared widely, good or bad<br />
    27. 27. 27<br />“It took Einstein ten years of groping through the fog to get the theory of special relativity, and he was a bright guy.”<br />Jim Collins<br />
    28. 28. 28<br />WAA Austin Symposium - the art & science of personalization<br />January 24th<br />
    29. 29. Join the winning team!<br />I am hiring a testing manager for Greater China region<br />There are a number of analytics/strategic positions open as well<br />Send your resumes to <br />29<br />