MindCanvas: Reimagining online research Rashmi Sinha Uzanto www.uzanto.com www.themindcanvas.com

Duct-tape research! (in the trenches - Information Architecture Redesign of eBay.com)

Background to IA redesign eBay.com has grown in organic fashion – inconsistent, fractured user experience Separate design / usability efforts for parts of eBay first effort to focus on user experience for entire site Basic eBay facts World’s largest online marketplace (79 million in US) Types of users Buyers & sellers New & experienced Individuals & corporations Different countries & cultures

eBay.com has grown in organic fashion – inconsistent, fractured user experience

Separate design / usability efforts for parts of eBay

first effort to focus on user experience for entire site

Basic eBay facts

World’s largest online marketplace (79 million in US)

Types of users

Buyers & sellers

New & experienced

Individuals & corporations

Different countries & cultures

Challenges New IA would need to Serve diverse user group varying goals / mental models Be future-oriented, allow for change & growth Changes needed to be evolutionary not revolutionary accommodate users familiar with current site Senior management at eBay needed proof Is there need for such drastic restructuring? Research to be completed in 6 weeks!

New IA would need to

Serve diverse user group

varying goals / mental models

Be future-oriented, allow for change & growth

Changes needed to be evolutionary not revolutionary

accommodate users familiar with current site

Senior management at eBay needed proof

Is there need for such drastic restructuring?

Research to be completed in 6 weeks!

Goals of research Research has multiple goals Need successful design, not just good design Serves user Satisfies business goals Is accepted by management 3-stage research method Stages: Exploration  Understanding  Verification Stages complement & build upon each other Mix of qualitative & quantitative methods Qualitative: rich, open-ended understanding of users / domain Quantitative: recommendations grounded in data (help make case to management)

Research has multiple goals

Need successful design, not just good design

Serves user

Satisfies business goals

Is accepted by management

3-stage research method

Stages: Exploration  Understanding  Verification

Stages complement & build upon each other

Mix of qualitative & quantitative methods

Qualitative: rich, open-ended understanding of users / domain

Quantitative: recommendations grounded in data (help make case to management)

3-stage user research Stage 1: Explore Interviews, free-listing Stage 3: Verify / Refine Large-sample closed card-sorting 3 stages Stakeholders User thinking / categorization Preliminary IA Stage 2: Understand Open card-sorting Verify categorization Top-level categories Scope of content & functionality ebay.com Community Buy Sell Sub cat1 SubCat2 SubCat3 xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xx xxx xx xxx xxx xxx SubCat4 SubCat5 SubCat1 SubCat2 xxx xx xx

Stage 1: Explore domain with Freelisting Goals Explore domain in open-ended fashion, map scope. Generate list of current & “Horizon Tasks” (planned for future) Method Interviews & Freelisting With eBay users, stakeholders & designers Freelisting tasks (eBay is about tasks!) What are all the tasks users can do on eBay.com List an item, Pay for it What else will users be able to do on eBay.com 2-5 years that they can’t do today? Results: List of 100 representative tasks (25 Horizon)

Goals

Explore domain in open-ended fashion, map scope.

Generate list of current & “Horizon Tasks” (planned for future)

Method

Interviews & Freelisting

With eBay users, stakeholders & designers

Freelisting tasks (eBay is about tasks!)

What are all the tasks users can do on eBay.com

List an item, Pay for it

What else will users be able to do on eBay.com 2-5 years that they can’t do today?

Results: List of 100 representative tasks (25 Horizon)

Stage 2: Explore user categorizations Goals: Gain insight into user thinking about eBay Identify top-level categories for ALL site content and functionality Method: Interviews & Open card sorting List of 100 tasks (including 20 Horizon Tasks). 35 participants (including sellers & buyers; new and advanced users).

Goals:

Gain insight into user thinking about eBay

Identify top-level categories for ALL site content and functionality

Method: Interviews & Open card sorting

List of 100 tasks (including 20 Horizon Tasks).

35 participants (including sellers & buyers; new and advanced users).

Identify top levels of hierarchy Hierarchical Cluster analysis to generate aggregate user categorizations. Separate analysis for 4 user groups Identify inconsistencies in categorizations Reconcile to create one scheme. Result: Preliminary IA 5 top-level categories accounting for all site content and functionality. Dendogram

Hierarchical Cluster analysis to generate aggregate user categorizations.

Separate analysis for 4 user groups

Identify inconsistencies in categorizations

Reconcile to create one scheme.

Result: Preliminary IA

5 top-level categories accounting for all site content and functionality.

Stage 3: Verify & refine scheme Goals: Verify 5-category scheme covers all of eBay. What tasks/concepts don’t fit? Where do users expect this info? Input from larger, more diverse set of users. Establish method for future verification of IA. and to generalize scheme to international eBays. Method: closed card sorting Large sample (~ 1,000) representing all eBay user types. Study conducted using online survey. Participants categorized tasks into one of categories (or selected Other if it didn’t belong) Result Site-map showing the structure for all main parts of the eBay.com

Goals:

Verify 5-category scheme covers all of eBay.

What tasks/concepts don’t fit? Where do users expect this info?

Input from larger, more diverse set of users.

Establish method for future verification of IA.

and to generalize scheme to international eBays.

Method: closed card sorting

Large sample (~ 1,000) representing all eBay user types.

Study conducted using online survey.

Participants categorized tasks into one of categories (or selected Other if it didn’t belong)

Result

Site-map showing the structure for all main parts of the eBay.com

Sell - Before

Sell Hub - After

eBay redesigned successfully! Three-stage methodology Understand business context Delineate user categorizations Create IA blueprint Deliverable: site-map with structure for main parts of eBay.com. map used as blueprint of IA redesign (took 2 years). Generalized to international eBays. Positive business implications (ROI)! Users did not protest! site.com Top Category 3 Top Category1 Top Category 2 Top Category 4 SubCat1 Sub cat1 SubCat2 xx SubCat3 xx xx SubCat2 SubCat3 SubCat5 xx xx xx xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xxx xx xx xxx xx xxx xxx xxx SubCat7 SubCat4 SubCat5 SubCat6 SubCat1 SubCat2 xx xx xxx xx xxx xx xx SubCat4 xx Preliminary IA

Three-stage methodology

Understand business context

Delineate user categorizations

Create IA blueprint

Deliverable: site-map with structure for main parts of eBay.com.

map used as blueprint of IA redesign (took 2 years).

Generalized to international eBays.

Positive business implications (ROI)!

Users did not protest!

Duct-tape research! There has to be a better way - fast, cheap (relatively) & online…

Why we built MindCanvas Qualitative research is great, takes a lot of time Online tools suck, get used anyway (users not engaged) Business stakeholders respond to quantitative analysis & large sample sizes Deliverables do not cater to designers Research findings remain locked up with analyst

Qualitative research is great, takes a lot of time

Online tools suck, get used anyway (users not engaged)

Business stakeholders respond to quantitative analysis & large sample sizes

Deliverables do not cater to designers

Research findings remain locked up with analyst

Inspiration Luis Von Ahn’s ESP Game Rely on games to label the web PopCap Games Quick, short, engaging games Visualization research bringing statistics to designers

Luis Von Ahn’s ESP Game

Rely on games to label the web

PopCap Games

Quick, short, engaging games

Visualization research

bringing statistics to designers

back to the future! MindCanvas interpretation

What is MindCanvas? Remote research methods Statistical Analysis & Data Mining Interactive Visualizations loosely coupled, use as qualitative or quantitative

Game-like elicitation methods Surveys do not engage people. Games are fun! People engage in complex tasks willingly Do game-like methods work better? MindCanvas has game-like elements (its not a game) Interactive, visual, screen build up, fun For both qualitative & quantitative Remote moderated for qualitative Remote unmoderated for quantitative

Surveys do not engage people. Games are fun!

People engage in complex tasks willingly

Do game-like methods work better?

MindCanvas has game-like elements (its not a game)

Interactive, visual, screen build up, fun

For both qualitative & quantitative

Remote moderated for qualitative

Remote unmoderated for quantitative

Rich interactive visualizations Large datasets need statistical analysis & data mining Visualizations-to-go Self-contained files - embedded into PowerPoint, emailed, shared Interactive visualizations with slides & knobs! Easy to share with team Visualization posters Designers want printouts to hang on walls, and scribble on

Large datasets need statistical analysis & data mining

Visualizations-to-go

Self-contained files - embedded into PowerPoint, emailed, shared

Interactive visualizations with slides & knobs!

Easy to share with team

Visualization posters

Designers want printouts to hang on walls, and scribble on

MindCanvas research focuses on the mind Inspiration from cognitive anthropology, psychology, market research Semi-structured methods Quantitative aggregation & visualization Categorizations Preferences Language

Inspiration from cognitive anthropology, psychology, market research

Semi-structured methods

Quantitative aggregation & visualization

Who we built for: designer persona Needs convenient methods to do research Does not want to spend time learning statistics The busy designer / researcher who wants to look at patterns visually Want to directly interact with users, does not have time

Who we built for: participant persona “ This was the most fun I have had in a while!” “ It is very easy to complete the task. In fact, I had fun doing this.” The not-tech savvy person who occasionally takes surveys “ I think there are too many cards. But it was kind of fun.”

Where MindCanvas could help Product Development: Finding the right feature-set for the right users (or personas) Information Architecture: Understanding how people think for information design. Early visual design validation: Understanding what people think & feel about a design when its just an image.

Product Development: Finding the right feature-set for the right users (or personas)

Information Architecture: Understanding how people think for information design.

Early visual design validation: Understanding what people think & feel about a design when its just an image.

MindCanvas in product development Initial understanding of domain What’s important to people? What words do they use? Method: Freelisting Deliverable: ListMap Prioritizing features / outcomes users care about Method: Divide-the-dollar Deliverable: WeightMap ClusterBrowser

Initial understanding of domain

What’s important to people? What words do they use?

Method: Freelisting

Deliverable: ListMap

Prioritizing features / outcomes users care about

Method: Divide-the-dollar

Deliverable:

WeightMap

ClusterBrowser

MindCanvas in Information Architecture How do people think about a domain? Method: OpenSort Deliverable Dendogram SimilarityBrowser VocabularyBrowser 1, 2 Is your information architecture effective? Is one scheme more effective than another? Method: TreeSort Deliverable: ClusterBrowser

How do people think about a domain?

Method: OpenSort

Deliverable

Dendogram

SimilarityBrowser

VocabularyBrowser 1, 2

Is your information architecture effective? Is one scheme more effective than another?

Method: TreeSort

Deliverable:

ClusterBrowser

Questions? MindCanvas team is… Delhi Kapil Mohan Kamalmeet Singh Debasish Biswas Sunil Gupta Amit Ranjan Mountain View Jon Boutelle Rashmi Sinha [email_address]

MindCanvas team is…

Delhi

Kapil Mohan

Kamalmeet Singh

Debasish Biswas

Sunil Gupta

Amit Ranjan

Mountain View

Jon Boutelle

Rashmi Sinha

Methods & Deliverables Current Methods OpenSort: Dendogram, VocabBrowser, SimilarityBrowser TreeSort: ClusterBrowser, SimilarityBrowser Divide-the-Dollar: WeightMap, ClusterBrowser Free Listing: ListMap Utility Questions: MultipleChoice, EssayType Upcoming Methods: MenuTrace, MindMontage, Likert Sort Upcoming Designables: OnionMap

Current Methods

OpenSort: Dendogram, VocabBrowser, SimilarityBrowser

TreeSort: ClusterBrowser, SimilarityBrowser

Divide-the-Dollar: WeightMap, ClusterBrowser

Free Listing: ListMap

Utility Questions: MultipleChoice, EssayType

Upcoming Methods: MenuTrace, MindMontage, Likert Sort

Upcoming Designables: OnionMap

Collaborative Research Process High quality research needs expertise research design + statistics Service should enable high quality research Structured research process Templates for study design Help with data collection

High quality research needs expertise

research design + statistics

Service should enable high quality research

Structured research process

Templates for study design

Help with data collection

What is MindCanvas An innovative research platform for high quality research Game-like elicitation methods Interactive visualizations (designables-to-go) Research expertise

An innovative research platform for high quality research

Game-like elicitation methods

Interactive visualizations (designables-to-go)

Research expertise