Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply



Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Note that this was a collaboration between Uzanto and eBay Design Labs
  • Transcript

    • 1. Rapid User Mental Modeling (RUMM) at A Case Study Jonathan Boutelle Rashmi Sinha & Kirsten Swearingen Uzanto Consulting Larry Cornett & Ido Dan eBay, Inc.
    • 2. Background to Information Architecture Redesign of
      • eBay website had grown in organic fashion, leading to a somewhat inconsistent user experience
      • Although much design and usability work is done on site itself, this project was first effort to delineate user mental models for the entire website.
      • Basic eBay facts
        • World’s largest online marketplace
        • 16 million listings on any given day (as of Q2, 2003)
        • Almost 34 million active users worldwide
          • Buyers and sellers
          • New and experienced
          • Occasional and full-time
          • Individuals and corporations
          • Different countries and cultures
    • 3. Challenges
      • New IA would need to:
        • Support site’s vision of global economic democracy
        • Serve a large and diverse user population with varying goals and mental models
        • Be future-oriented, allowing for planned change and growth
        • Occur through evolution not revolution -- must accommodate current users who are very used to the site as it exists today
      • Senior management at eBay needed to see proof of the need for a drastic restructuring.
      • First phase of project (background research) to be completed in only 6 weeks
    • 4. Overview of Design Research Methodology Stage 1: Explore Interviews and free-listing with users and stakeholders Stage 3: Verify / Refine Large-sample closed card-sorting study Rapid User Mental Modeling Stakeholders User mental models Preliminary IA Stage 2: Understand Open card-sorting study Verified categorization Top-level categories Scope of content & functionality Top Category 3 Top Category 1 Top Category 2 Sub cat1 SubCat2 SubCat3 xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xx xxx xx xxx xxx xxx SubCat4 SubCat5 SubCat1 SubCat2 xxx xx xx
    • 5. Our Approach to Design Research: Rapid User Mental Modeling (RUMM)
      • Rigorous, three-stage design research methodology to lay foundation for IA redesign
        • Bring together established techniques in a novel way
          • Three stages: Exploration  Understanding  Verification
          • Individual components complement and build upon each other
        • Process moves from open-ended exploration to focused information-gathering and hypothesis testing
        • Addresses individual and group differences in a systematic way
        • Produces reliable results (can be reproduced)
        • Uses mix of qualitative and quantitative methods
          • Qualitative: rich, open-ended understanding of users and domain
          • Quantitative: generates recommendations firmly grounded in data (can help make case to management)
    • 6. Towards Persuasive, Defensible Design Research
      • Important to create not just good design, but successful design:
        • Serves user
        • Satisfies the business goals
        • Is accepted by management
      • Results of design research phase must provide rich picture of user, but also be persuasive and defensible.
        • Defensible: firmly grounded in data, can stand up to scrutiny
        • Persuasive: persuades stakeholders that your solution makes sense and will provide benefit to the business
    • 7. Stage 1: Explore Domain Through Stakeholder Interviews
      • Goals:
        • Explore domain in open-ended fashion
        • Identify both current and planned content and functionality
      • Method:
        • Interviews with eBay users about specific incidents of site usage
        • Interviews with stakeholders: top management, members of design team etc.
          • Identify problems, constraints, and specific plans for future
          • Generate list of current tasks and “Horizon Tasks” (tasks planned for future)
        • Stakeholder analysis
        • Task list development
    • 8. Stakeholder Interviews
      • Spoke to 16 eBay managers and designers
      • Asked 3 questions:
        • What do you imagine a user doing on the eBay site two to five years from now that they can’t do today?
        • What one problem, if it were solved today, would cause a large increase in the number of transactions executed on the eBay platform?
        • What user population is least well served by the current site?
    • 9. Purpose of Stakeholder Analysis
      • Gain valuable perspectives for design research:
        • Insure that design aligns with business strategy.
        • Tap into a company’s existing knowledge in a systematic fashion.
        • Ensure success of project by getting input at the research phase from those who have ability to say yes or no.
          • Helps stakeholders feel involved in and understand the design research and process.
      • Generate task list : helps map the scope and boundaries of the information domain
    • 10. Generating the Task List Using Free-Listing
      • Decided not to conduct inventory of site
        • Incredibly time-consuming
        • Might tie new IA too closely to current site’s design
      • Focused on tasks, since all eBay content is in context of completing tasks:
        • Pay for an item
        • List an item
      • Generated initial list of 300 tasks by using free-listing exercises with:
        • Users
        • Designers
        • Stakeholders (especially important for identifying future tasks)
    • 11. The Free-Listing Exercise
      • This simple, semi-structured method helps:
        • Determine the scope of a domain
        • Gain some insight into how the domain is structured
      • Example: “List all the animals you can think of”
      • Results:
        • Indices of salience:
          • Frequency
          • Rank
        • Indirect indices of similarity:
          • Co-occurrence. How often are cat and dog both mentioned?
          • Average rank distance. Are cat and dog mentioned together more closely than cat and horse?
      • In this case, we asked “List all the tasks at a person can do / might be able to do in the future.”
    • 12. Refining the Task List
      • Consolidated task list, removing redundant items
      • Divided tasks into groups:
        • Current Tasks (can be done on existing eBay website):
          • Core Tasks form core functionality of website
          • Boundary Tasks mark boundary of functionality offered
        • Horizon Tasks
          • Might be done on future versions of website
          • Important for creating a future-oriented IA
      • Selected representative items from each group
      • Result: comprehensive list of 100 tasks marking scope and boundary of what users can accomplish on the eBay site.
      • We were ready to begin studying user mental models!
    • 13. Stage 2: Understand User Mental Models
      • Goals: Gain further insight into user mental models and identify top-level categories for site content and functionality
      • Method: open card sorting
        • Used list of 100 tasks (including 20 Horizon Tasks) generated in previous stage.
        • 28 participants, representing the main types of eBay users (sellers and buyers; new and advanced users).
          • 22 completed card-sorts individually
          • 6 worked as a group to create a structure (mix of buyers, sellers, new, and advanced)
    • 14. Identifying the Top Levels of the Hierarchy
      • Cluster analysis used to generate aggregate mental maps.
      • These were directly used to create the hierarchical structure.
      • Looked at mental maps for four major segments.
        • Identified the inconsistencies between the mental maps of different groups.
        • Understood how to reconcile those to create a new map.
      • Result: preliminary IA, including 5 top-level categories that accounted for all site content and functionality.
      Aggregate maps from cluster analysis
    • 15. Stage 3: Verify and Refine 5-Category Scheme
      • Goals:
        • Verify that the five top-level groupings cover all 100 tasks/concepts.
        • Identify tasks/concepts that do not fit in, as well as where users do expect to find such information.
        • Examine differences in groupings among various user groups.
        • Further develop IA for eBay site.
        • Establish method for future verification and refinement of eBay IA.
    • 16. Stage 3: Verify and Refine 5-Category Scheme
      • Method: closed card sorting
        • Large sample study with more than 1,000 participants representing all segments of eBay users
        • Study conducted online.
        • Participants asked to categorize tasks to one of the five top-level categories (or to select Other if it didn’t belong)
      • Result: site-map showing the structure for the main parts of the eBay site.
    • 17. Analysis of Closed Card Sort Results
      • Category-belonging scores:
        • Percent of users who think a task belongs to a particular category
        • Lack of consensus or high scores in “Other” category may signal a design issue
      A color-coded visualization of closed card sorting data
    • 18. Reconciling Different Mental Models
      • Results suggest structural possibilities--they do not prescribe one solution
      • In both open and closed card sorting, found differences in mental models
        • Reconcile through design, if possible (revisit dendograms)
        • The most important, contentious issues may ultimately require a decision driven by company brand and business goals (revisit stakeholder analysis)
    • 19. Project Summary
      • Three-stage RUMM methodology used to:
        • Understand business context
        • Delineate hierarchical structure of user mental models
      • Final deliverable: site-map showing structure for main parts of entire eBay website.
        • This map is currently being used to guide the stages of the IA redesign. Top Category 3 Top Category1 Top Category 2 Top Category 4 SubCat1 Sub cat1 SubCat2 xx SubCat3 xx xx SubCat2 SubCat3 SubCat5 xx xx xx xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xxx xx xx xxx xx xxx xxx xxx SubCat7 SubCat4 SubCat5 SubCat6 SubCat1 SubCat2 xx xx xxx xx xxx xx xx SubCat4 xx Preliminary IA
    • 20. Impact of Project
      • Helped convince management that current site did not match user mental models and that IA redesign was needed.
      • Provided basis for a long-term redesign plan to implement a structured site-wide model.
      • Provided immediate guidance to near-term projects in order to begin moving the site in the long-term direction.
      • Broadened eBay designers’ perspective on overall design direction and goals by providing:
        • Access to user mental models
        • Site-map for all content and functionality of
      • Cleared the road towards a more consistent and unified user experience.