Optimizing eBay - Improving customer experience at the world’s online marketplace - eMetrics 2008 Keynote

Uploaded on


  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Optimizing eBay Improving customer experience at the world’s online marketplace Elissa Darnell, Director, User Experience Research Deepak Nadig, eBay Principal Architect eMetrics Marketing Optimization Summit May 06, 2008
  • 2. In the beginning… “I started eBay as an experiment” — Pierre Omid 2 eBay Inc. confidential
  • 3. An Amazing Story 2008 Registered eBay users: 276 Million 1996 Registered eBay users: 41,000 eBay Inc. confidential
  • 4. The eBay context eBay manages … – Over 276,000,000 registered users – Over 1 Billion photos – eBay users worldwide trade more than $2039 worth of goods every second An SUV sells every 2 seconds A sporting goodis sold every 5 minutes – eBay averages well over 1 billion page views per day – At any given time, there are over 113 million items for sale on the site in more than 50,000 categories – eBay stores over 2 Petabytes of data – over 200 times the size of the Library of Congress! – eBay analytics processes over 25 Petabytes of data on any day – The eBay platform handles 4.4 billion API calls per month In a dynamic environment – 300+ features per quarter – We roll 100,000+ lines of code every two weeks In 39 countries, in seven languages. >44 Billion SQL executions/day! eBay Inc. confidential Over ½ Million pounds of Kimchi are sold every year!
  • 5. Site Statistics: in a typical day… June Q1 1999 2007 Outbound Emails 1M 41 M 41x Total Page Views 54 M >1 B 19x 16 Gbps 59x 0 150 M N/A ~97% 99.94% 50x Peak Network Utilization API Calls Availability 268 Mbps 43 mins/day 5 eBay Inc. confidential 50 sec/day Growth
  • 6. Meet the Buyers and Sellers Some people buy on eBay and other people sell, and some do both eBay users trade in more than 50,000 categories eBay’s buyers want a fun shopping experience that provides a great deal Approximately 1.3M people around the world make all or part of their living selling on eBay 6 eBay Inc. confidential
  • 7. Who is eBay’s Community? “eBay Community” is everyone who has a relationship with eBay, Inc. Collectors New Buyers Frequent buyers Casual Sellers Business Sellers Hobby Sellers Store Sellers One-time buyers Top Buyers Power Sellers Small Businesses New Sellers Experienced Buyers In-active buyers Buy online Merchant Accounts Pay a friend 7 eBay Inc. confidential Personal Accounts
  • 8. What we mean by the whole user experience My overall feeling about a brand/ product (in the abstract) is good. It starts by being useful… I like the way the product looks and feels. Functionally, people must be able to use it… The way it looks must be pleasing… This helps create an overall brand experience Executing well on all of these areas is what creates a great user experience. Research is needed for each. 8 eBay Inc. confidential I am able to use the product easily (I can perform the tasks it supports). It is useful to me. It meets my needs.
  • 9. Assortment of User Research methods Lab Testing Surveys 9 eBay Inc. confidential Field Visits Eye Tracking Participatory Design Card sorting
  • 10. Lab-based Usability Testing •  Usability or “lab-testing” –  We bring in representative users individually to our usability labs –  Observe them while they perform assigned tasks –  Use prototypes or the live site •  Enables direct observation of target users as they interact with our web site or a design prototype –  Observe what users ACTUALLY do, rather than relying on what they SAY they do –  Understand WHY people do what they do, not just what they do •  Identify areas that are confusing and potential fixes •  Usability testing done iteratively, throughout the design process 10 eBay Inc. confidential
  • 11. Variations on the standard lab study Some variations on standard lab tests •  Low Fidelity (Paper) Lab Testing –  Designs are shown on paper –  Researcher or Designer acts as the computer –  Participant uses their finger as the mouse •  RITE Lab Testing –  Stands for Rapid Iterative Test and Evaluation –  Focuses more on rapidly iterating and re-testing the design based on very small sample –  Can be performed on lo-fi (paper) or high fidelity interactive prototypes 11 eBay Inc. confidential
  • 12. Visits Seeing through the eyes of our customers 12 eBay Inc. confidential
  • 13. Visits: What is it? •  Also known as a Field Study or Ethnography •  Involves going into people’s home, office, where ever they use eBay •  Spending two to three hours with them, observing them use eBay (or shop online) and listening to them, and sometimes conducting structured interview •  Unlike Lab Testing, typically formal tasks are not given •  Data is collected through videotaping and taking extensive notes •  Findings are summarized across participants 13 eBay Inc. confidential
  • 14. Visits allow us to observe how people use eBay in a natural context 14 eBay Inc. confidential
  • 15. We get to see people use eBay… •  On their own computers (PCs, laptops, old, new) •  With various connection speeds (some dial-up) •  To perform their own tasks (such as selling a camera or book) •  With their own cameras and workspace (living room, office) •  With life’s normal interruptions (talking parrot, cluttered desk) They tend to let their guard down and we learn things we might not otherwise learn online or in the lab 15 eBay Inc. confidential
  • 16. The focus of Visits •  Questions we focus on in Visits are: –  “What is the larger context of use?” –  “What issues exist, and WHY?” –  “What can we do to address the issues?” –  Visits are NOT about the numbers, or the question, “How many users experienced that?” –  If pervasiveness of a particular issue is of interest, we supplement with survey or analytics data to help quantify the issue 16 eBay Inc. confidential
  • 17. Research methods span the Product Development Strategic research to inspire Understand Field Visits Participatory Design Competitive Evaluation 17 eBay Inc. confidential Conceive Design research to inform Design Lab Testing RITE Testing Paper Prototypes Eye Tracking Card sorting Assessment research to track Develop Launch Surveys Live-to-site Testing Longitudinal Studies Diary Studies
  • 18. Research methods Self-reported (stated) Focus Groups / “Voices” Desirability studies Intercept Surveys Cardsorting Attitudes | What people believe Email Surveys Phone Interviews Diary/Camera Study Message Board Mining DATA SOURCE (Onsite interviews) Mixture Why How to fix What How much “Visits” / Ethnographic Field Studies Usability Lab Studies (task-based) (Extended observation) / Quantitative user experience assessments (e.g., Keynote/Vividence) Usability benchmarking (in lab) Observed Behavior Eyetracking Behaviors | What people do Data mining Experimentation Clickstreams Qualitative APPROACH KEY – Context of data collection with respect to product use eBay Inc. confidential De-contextualized / not using product Natural use of product Scripted or lab-based use of product Combination / hybrid Quantitative
  • 19. Case Study Using Qualitative and Quantitative Research to create a New “View Item” Page Why redesign the “View Item” Page? • Increase BID/BIN efficiency • Improve the user experience: Reduce complexity and clutter and inspire buyers to convert Research Approach • Qualitative and quantitative research to understand user needs and inform design decisions • Incorporate User Experience Research, Market Research, and Analytics • Institute research as an integral component of the redesign process (from start to finish) View ItemInc. confidential 2000 eBay 19 View Item 2003 View Item 2006 View Item 2008
  • 20. Research Overview Understanding User Needs (Qualitative) •  “Compelled” lab study (US) to understand the current experience and how it may be improved •  Participatory design (US, UK, DE) to understand the “ideal” experience to gather user requirements and inform product design and innovation Concept Testing (Qualitative) •  Focus groups (US) to gauge user reaction to early design concepts and isolate aspects of alternative designs that resonate with users Iterative Design (Qualitative & Quantitative) •  Rapid Iterative Testing (RITE) (US, DE, IT) to gauge user reaction to an early View Item prototype and make rapid improvements to the design based on user feedback and behavior •  Maximum Differential Survey (US) to determine the relative perceived usefulness of features to Buyers in an effort to streamline and simplify the experience •  Analytics (US) to understand current usage of existing features Visual Design Research (Quantitative) •  Desirability study (US) to determine which visual design approach best conveys target brand attributes (e.g. convenient, clean, safe, fun) •  Tab Visual Design Eyetracking study (US) to explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data Longitudinal Research (Q2 2008) (Qualitative) •  Longitudinal Diary Study (US) with functional prototype to understand “real world” usage of View Item over time eBay Inc. confidential 20 and determine areas for improvement prior to launch
  • 21. Compelled Lab Study Objective Surface usability issues and perceived strengths and weaknesses of the current View Item page and larger transaction flow Approach A “compelled” lab study with users genuinely interested in purchasing items of interest on eBay. Users were asked to find and purchase items of interest on eBay with their real account and payment information in the lab setting Participants 12 - Mix of experienced and less experienced eBay buyers All participants were pre-screened to ensure they had a genuine interest in purchasing specific items of interest on eBay in the coming days 21 eBay Inc. confidential
  • 22. Feature Prioritization Maximum Differential Survey Background and Objectives •  A key goal was site simplification: Provide buyers with the most important information they need to make their purchase decision •  Traditional surveys focused on understanding feature usefulness typically do not effectively differentiate among features (e.g. majority of features considered useful by 90% or more of buyers) •  A Max Differential survey was conducted in the U.S. to determine the relative usefulness of View Item features to understand which features to include and the relative prominence that should be assigned •  Approximately 5000 buyers and 5000 sellers completed the survey which involved asking users to choose the most useful and least useful feature among different combinations of features to yield a score for each feature EXAMPLE Total Buyers Current bid Auction countdown End time Time left Real-time updating of bid Questions and Answers Your maximum bid Number of unique bidders 22 eBay Inc. confidential Top 30% Buyers Lower 70% Buyers
  • 23. Tab Visual Design Eyetracking Study Goal: To explore different visual design treatments for View Item page tabs and determine which best captures and maintains attention as measured by eyetracking data On Hover Off Sample Heat map summarizing overall viewing pattern On Hover Off On Hover Off Sample scanning pattern 23 eBay Inc. confidential
  • 24. How do we know the new “View Item” page will be a success? Research was conducted throughout the product lifecycle to evaluate the current strategy and design at each stage (prior to further dedication of design or engineering resources) •  Focus groups on early concepts •  Rapid Iterative Testing of early prototype •  Max Differential survey (and Analytics data) to ensure that reduction of page complexity targeted the least useful features •  Quantitative “Desirability” study to ensure that the chosen design approach best conveyed target brand attributes (such as “fun” and “unique”) •  Quantitative Eyetracking study of tab design alternatives to ensure the chosen treatment would best capture and maintain user attention •  A longitudinal diary study (planned in Q2 2008) to gather qualitative real-world usage feedback to ensure a good user experience prior to launch (to be interpreted in conjunction with A/B Test data) 24 eBay Inc. confidential
  • 25. Experimentation •  Metrics •  Reporting •  Idea (!) •  Learning 7. Analysis & Results 1. Hypothesis • Tracking • Monitoring 5. Measurement eBay Experimentation Platform 2. Experimental Design •  DOE •  Define Samples, Treatments, Factors 4. Launch Experiment •  User  (Experiment, Treatment) •  Serve Treatment 25 eBay Inc. confidential 3. Setup Experiment •  Setup Experiment Samples Treatments, Factors •  Implementation
  • 26. Automation •  Dynamically adapt experience –  Choose page, modules, and inventory which provide best experience for that user and context –  Order results by combination of demand, supply, and other factors (“Best Match”) •  Feedback loop enables system to learn and improve over time –  Collect user behavior –  Aggregate and analyze offline –  Deploy updated metadata –  Decide and serve appropriate experience •  Best Practices –  “Perturbation” for continual improvement –  Dampening of positive feedback 26 eBay Inc. confidential
  • 27. Experimentation Platform Access Access Design Results Observations Experiment Metadata Results Observations Metrics / Experience 27 eBay Inc. confidential Page, Module Experience Metrics / Experience Experience Response
  • 28. What we think about Fidelity of Experiments Extent to which the model and its conditions represent the final feature or product Cost of Experiments The total cost of designing, building, running, and analyzing an experiment Iteration time The time from hypothesis to when the analyzed results are available for planning the next iteration Concurrency Number of experiments that can be run at the same time Signal/Noise Ratio The extent to which the metrics are obscured by experimental noise Type/Level of Experiment Supporting different types (A/B, 1-FAT, DOE) and levels (page, module, page flow) of experiments 28 eBay Inc. confidential
  • 29. Challenges … •  Sticky-ness to user •  What, not why •  Duration and long term effects •  Minor vs. major differences •  Extent of generalization 29 eBay Inc. confidential
  • 30. The Power of Combining Data Qualitative research such as lab tests and field visits give us rich data about •  •  •  •  •  Usability problems Discoverability Navigation Terminology More complex problems Quantitative research such as surveys, usage studies, and log analysis tells us •  How pervasive a problem is •  What features on the site are actually being used The combination of qualitative and quantitative data give us a more complete picture that is most powerful 30 eBay Inc. confidential
  • 31. How we get our data differs… Analytics Research Measure what customers do through serving experiences and tracking responses Work directly with customers by… • Talking to them • Observing them • Surveying them The real power is from looking at our user’s 31 eBay Inc. confidential from both sides!
  • 32. Summary •  Understanding the customer experience requires insights into –  What they do –  Why they do it –  Attitudes, motivations and behaviors •  A variety of research methods, both qualitative and quantitative, can be used. •  Each of the methods – qualitative and quantitative - have their advantages and limitations. •  Using them together continues to help eBay gain a holistic understanding of the user experience and aspects that need to be improved. 32 eBay Inc. confidential
  • 33. Questions Elissa Darnell edarnell@ebay.com Deepak Nadig dnadig@ebay.com 33 eBay Inc. confidential