Downloaded 10 times
Uploaded byEd Wu
From good to great emetrics_dc_keynote
This document summarizes the vision and progress of Dell's online analytics and testing program. It discusses establishing a global testing team, standardizing processes, and prioritizing strategic areas like key pages and checkout testing. It emphasizes the importance of acquiring top talent, holding teams accountable, establishing iterative testing processes, understanding why tests succeed or fail, and embracing a culture where testing is a priority from the start of projects. The goal is to become a leader in e-commerce innovation by offering the most relevant customer experiences and driving high loyalty through insightful analytics and best-in-class testing capabilities.
More Related Content
![Navigate the Future: Roadmapping that Works [webinar]](https://cdn.slidesharecdn.com/ss_thumbnails/2016-10-19roadmappingwebinarsopheon-161024185021-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to From good to great emetrics_dc_keynote
From good to great emetrics_dc_keynote
- 1. From Good toGreatHow the testing program is evolving at DellDell Online Analytics & Testing TeamOct. 6th, 2010
- 2. Agenda2What’s the vision?(EdWu – Senior Manager, Global Testing)Got process?(Carol Gohl – PMO)What’ve learned?(Dave Nelson – Global Testing Manager)
- 3.
- 4. Online:Dell.com + Premier+ Global Portal + B2B + Channel Portal + eSupport + Community725K B2Btransactionsper year34 supported languagesGoogled 1M timesper day170M visitsto community properties58Kchannelpartnersonline1Bvisits per year to Dell.com190KPremierPages13Bpage viewsin FY’09Dell Online Today4An online orderis placedevery2 seconds
- 5. Our journey startedmore than a decade ago…5Early Adoption StandardizationMaturityRegional implementationGlobal implementationStandard KPI’s – global dashboardRegional KPI1:1 BT experience Industry leading testing capabilityInternal testing capabilityFull funnel testing2008-20092010-1996-2007In-house clickstream toolAdvanced segmentationVOC solutionGlobal team across BURegional teamGlobal team by BULimited expertise and talent benchWork closely with vendors & consultantsGrow in-house expertiseLow
- 6. 6Strong back-office supportGood peopleOur StrengthsGreat technologyCommitted to testing
- 7. 7“GOOD IS THEENEMY OF GREAT”Jim Collins
- 8. 8Dell.comOnline Analytics, Testing& TargetingTo be thetechnologye-commerce leaderby offering the mostrelevant experienceand solutions todrive high customerloyaltyTo become an innovation engine through insightful analytics &the best-in-class testingcapability to enable constant and rapid changes
- 9. People9Double the sizeof the teamOn-boarding process to provide support and create connectionsOn-going training & development planIf I were running a company today, I would have one priority above all others: to acquire as many of the best people as I could. Jim Collins
- 10. TeamGlobal AnalyticsGlobal TestingPMOeDellProgramsReporting10
- 11. Accountability112x Revenue upsideY/Y400-600 tests globallyMonetization model based on test results with statistical significanceComplete transparency & alignment with finance & businessBecause they lack accountability, they fail to achieve credibility, and they have no authenticity.Jim Collins
- 12.
- 13. Holy Grail ofWeb Analytics13
- 14. Entering Carol Gohl…10years experience in PMO’s Information technology deploymentsSales force transformationConstruction and infrastructureRetail salesProduct launchesBut, knew nothing about web analytics!14
- 15. Key Findings inthe First 30 DaysEntrepreneurial vs organizational strategySame people ….different roles!Multiple “unique” processes!Great testing ideas…capability challengesInvestment in Analytics tools – but not process and workflow tools15Look… Listen… Learn…
- 16. 16Planning cadencePipeline managementStandardizedmethodologiesPRIORITIZE the elephants and the cows….“We’ve never done that”“They” wont let you do that “That’s too risky” “We can’t do that”
- 17. Process Transformation Strategy17ArchitectedCollaborativeFasterdeploymentNear term benefit Not readily adoptedHard to sustainSlower deploymentLasting benefitMore readily adoptedBecomes organic
- 18. Big Picture First18pipelineroadmapBrainstormRecipesInstitutionalize learningimplementCommunicateAnalytics Manager LeadTest Manager LeadSign off requiredDocumentation required
- 19. …and then getinto Weeds19
- 20. Analytics Manager LeadCheckMajor MilestonesTesting Manager LeadWK11WK2WK6WK11WK13WK4WK8Strategic Areas (NQ)
- 21. List of testideas (NQ)
- 22.
- 23.
- 24. List of testideas (NQ)
- 25. Annualized upside (NQ)AnalyticsManagerM3 testing priorities– wireframes / feasibilityM2 testing priorities– wireframes / feasibilityM1 testing priorities– wireframes / feasibilityM1 testing priorities– wireframes / feasibilityIdea GenerationLock M1 schedule; First test in M1 ready to launchLock M2 schedule; First test in M2 ready to launchLock M3 schedule; First test in M3 ready to launchTesting ManagerTest ExecutionWeekly Optimization Meeting20
- 26. Establish a FeedbackMechanism21Overall SatisfactionImprovement Areas
- 27. Process Change isIterative…just like testing!22“Don't let the perfect be the enemy of the good” Voltaire
- 28.
- 29. Focusing on StrategicAreas24Key PagesCart and CheckoutTargeting
- 30. Understanding Customer Intent25Specificproduct offers for customers desiring an efficient shopping experienceImproved site navigation for customers want to take more time to find the right product
- 31. Targeting Relevant Content26ConsumerCustomersSmall & Medium Business Customer
- 32. Checkout Testing -JulyCheckout Testing - AugustCheckout Testing - SeptCA ConsUS ConsUK ConsUS ConsDE ConsUS SMBMX ConsCN ConsUS ConsUS ConsUS SMBJP ConsAU ConsCN ConsCN SMBJP SMBAU SMBMX SMBBR ConsCA SMBUS ConsUS ConsAU ConsFR ConsUS ConsMX SMBBR SMBUS ConsAU SMBTesting Globally…an Iterative Process27
- 33. Understand Why SomethingWorks (or Doesn’t)28Simplified Cart Test – We changed too muchvs.
Editor's Notes
- #9 Customized Personal Experience: Not only personalization and behavioral targeting driven transactions, but… Customer choice focused ; Intent & context based Multi-dimensional ; Cross-platformCustomized social experience: Not only social media integration and ratings & reviews, but…Embedded in broader experienceContextual; Cross-platformCustomized Solution Experience: Not only rich product content and bundles, but… Configurable solutions; Global solutions development platform; Cloud
- #14 Organizational design is about people firstGood people + robust process and the right tool sets – we had the people and the tools….
- #15 No preconcieved notions – smart people - working hard – delivering results
- #16 Running like small biz vs scaled org of a fortune 100 companyGeneralization of skills vs specialization of roles – lacks accountability/promotion of expertise and COCGeographic process disparity Dynamic funnel test ideas – but limited funnel accessRobust analytical tools ….and excel and powerpoint!Assemble the findings
- #17 Identify the 5 big opportunities – the “elephants” in the room that if solved will bring the most immediate value to the orgTackle the obvious – its probably obvious to other too!Don’t pick and choose - No sacred cows - Listen for “we can’t change that” – “we’ve always done that” – “we can’t do that” – sometimes these are the hardest areas to affect and bring he most rewardChange the things that matter – go big or go home!Ask questions: Quantify and qualify the process opportunity – with the experts, the stakeholders AND the criticsCommunicatecommunicate - over communicate – change is scary Make process planning and change relevant – how will it benefit the team, or function - …it will make it more interesting to others!
- #18 Organizationally decide how you want to change - Architected – 100 processes in 100 days – fast – quick resultsCollaborative – peel away the layers – slower, more painful, requires investment by allNo “not invented here”
- #20 WARNING!Tests the sticktuitiveness of your team – process definition work can clear a room very quicklyFor key activities of each work area - answer the question: What, how, who and whenTrain on the key activities; Performance plans and quality plans should capture the required activities and deliverables
- #21 Activities must align to a cadence or schedule (check durations or time to do the work)Define when the work starts and endsCheck process synergiesProcess is about who, what, how and WHEN
- #22 Surveys, conversations and dataSet goals; measure to them – assess the effectiveness of the processesEstablish a feedback forum outside and inside your teamsExecute, checkand change
- #23 a parting thought….In process design the 80/20 rule applies - Be secure in change control – it works for that 20%!Use the same discipline in rolling out a change in process as implementing a new process – changes are actually harder due to legacy knowledge
- #25 When we formed our global team and got better at sharing information with each other, we started to learn things that have made us much better at getting the most out of our testing program. I’d like to take a few minutes to share with you some of the most important lessons we’ve learned.A while back, we weren’t very organized or focused, we just tested a lot of pages.We figured, the more pages we test, the more winners we’ll find, and we’ll improve our site faster. But we noticed that our most impactful tests throughout the world tended to be on the same pages – just a few for each country.We began to realize that we would be better off testing a few pages multiple times throughout the quarter than to test as many pages as possible.Another strategic focus area for us is basket and checkout pages.Even though these pages get a tenth of the volume of our most visited pages, they have the most revenue going through them. That shouldn’t be surprising because every purchase must go through these pages.The other reason these are valuable is that this is where we have are highest quality customers. This is especially true for Dell because we have a more detailed shopping experience than most stores. Many of our customers spend a lot of time choosing components and accessories to build the system that is exactly what they want. Customers who are willing to do this are seriously considering purchasing. Some of our greatest successes recently has been in cart and checkout tests.Our last focus area is behavioral targeting. The reason this is important to us is that we have a very high success rate with targeting.In fact, most of the targeting campaigns we try are successful.Whenever we focus on a customer’s needs and show them relevant content, they have a better experience and are more likely to buy.
- #26 Its not enough just to know where on your site to test and which pages to focus on.You need to know what you should try – what you should change and how to change it.A common mistake by marketers is to think too much about what we want our customers to do. We’ll design a site around the way we want them to shop.It’s a good idea to turn that around and think about why your customer is on your site and exactly what they want to do. That’s an easy concept, but in reality it is very complicated because we all have so many different kinds of customers.A page that is designed perfectly for one customer might not meet the needs of another customer.Here’s one example of this from our site:Customers who are brand loyal and come to our site with the intention of purchasing expect a simple shopping experienceDon’t want a sales pitch, just want to see the product offersOther visitors are open to buying from Dell or one of our competitorsThey want to take their time making a decision and want reasons why Dell is better than our competitorsThey would prefer a site with clear navigation that makes it easy to find the information they are looking forIf we tested these two options against each other, one would likely come out on topSuppose the option on the left came out with a 3% advantage in revenue per visitorThat would be good, but doesn’t tell the whole storyIf we dig deeper we find that the option on the left works 10% better for Efficient Shoppers but 7% worse for our Cautious ShoppersThe best solution is to use a targeting campaign to provide the best experience to both group.Target Cautious Shoppers based on:Clicks on Help Me Choose and other learning contentView of customer reviewsVideosOther content that suggests a high level of customer engagement with the siteA great way to learn what works best for different types of customers is to do some segmentation analysis on your test results to see if the results vary for different kinds of customers.Sometimes a simple A/B test can result in a very successful targeting campaign. At the very least, you’ll learn more about your various customers, which will help you design better tests in the future.
- #27 I wanted to show you a really simple of example of targeting we do to show our customers more relevant content based on what we know about them.This is our homepage in the United Kingdom. When a new customer comes to this page, they see a rotational banner with products across all of our segments, including consumer and business products.However, once a customer browses one of our customer segment sites, when they return to the country homepage, they will only see banners related to that segment.For example, if a visitor views pages in the Consumer site then returns, they will see banners featuring our consumer brands. Our Small and Medium Business customers will see products that would be relevant to them.We’ve done similar things with banners inside our customer segment sites. For example targeting specific product families based on prior browsing behavior.You should look to target based on any criteria you find that seems to differentiate the way customers shop. That might be clues from prior browsing history, traffic source, first time visitors vs repeat visitors, or based on prior purchases.The bottom line is, if we know what content would be relevant to a certain type of customer, we should use targeting campaigns to make them feel like the site is designed around their needs.
- #28 One of the most significant changes to our test process is what we do with successful tests.Previously, if we had a successful test, we would implement the winner, have a short celebration, and then move on to some other idea.We now realize that we were missing out on a huge opportunity to drive more value out of every test we run.There are now two additional things we do every time we have a successful test.First, we take the things we learned from a test and attempt to design a new test that will perform even better.Even with a test that has positive lift, further analysis can reveal some things about the design that might not have worked like we intended. So, we’ll redesign the test and run it again to see if we can do even better.A good example of this was when we ran a series of tests related to adding a trustmark in our checkout pages.Our early tests revealed that we had an idea that worked, but we continued to test different trustmarks and different locations until we found a solution that worked best.The final solution had more than double the lift that we saw in our first successful trustmark test.The second thing we do with our winners is that we run similar tests in most or all of our key markets throughout the world. Once we found the best Trustmark solution in the US, we tested it in our other major markets.In essence, instead of running a bunch of stand-alone tests, we think of our testing program as an iterative process for providing a global solution.From a single test, we generate many new test ideas and designs, and we take our best ideas and test them throughout the world.
- #29 Not only are we trying to get more out of our successful tests, but we now try to learn as much as possible from our mistakes as well.This happened in our checkout testing program when we tested the use of a simplified cart. We learned to not change too much at once.This slide illustrates the concept of the test.The hypothesis of this test was sound: We believed that customers would prefer a simplified cart with as little text as possible.So, we designed a test that removed or simplified any content we felt was unnecessary. Then we ran an A/B test.We were confident this test would work, so we were surprised when we found out the new recipe was performing worse than the original.It appeared that we removed some content that our customers thought was helpful. But we changed so much, how could we know which changes were positive and which were negative.Fortunately, in this case we got lucky. There was a change that caused significant negative performance that involved navigation, so we were able to detect it through some detailed pathing analysis of our test results. The change that hurt us was one of the smallest changes that we made, at least visually. We moved the Continue Shopping link from the top of the page next to the continue button to the very bottom of the page.This caused a frustrating shopping experience for customers who wanted to add other product to their cart, or go back to do more research before finalizing their order.We learned a couple of good lessons from this testFirst, if you are going to make a change that could significantly impact the way a customer browses your site – like changes to links to other pages, it is best to test those independently. Moving this link didn’t really fit with the original hypothesis of this test.Second, if you want to understand the impact of changing lots of different elements on the page, it is best to set up the test so that you can understand the impact of each change independently. Either multivariate test or an A, B, N test
- #31 People firstTo have a successfultesting program, you’ll need good people, processes, and tools. Most important is people. Tools don’t give you anything on their own unless you have smart people to use them.Plus, smart people can help you define the best processes.Align people to tasks(Identify the tasks, then identify people to the tasks)(Specialists, accountability)Process is a necessity, not an optionFrom out experience we know that it is possible to run A/B tests without effective processes.However, you will need a streamlined process if you want to ensure you are consistently running the right test, executing them flawlessly, and maximizing the amount that you are learning from your testing program.Scalability is key to successEffective processes allow you to scale your testing program.Scale of people, tools, process, learningsFailure is as valuable as success
- #33 In closing, I’d like to say that even if you have the right resources in place, the smartest people, and great processes, getting the most out of your testing program takes time. It will take a lot of failures and successes for you to learn the things you need to know to optimize your site. So be patient and stay committed to it, and it will definitely pay off.