Qualitative: rich, open-ended understanding of users / domain
Quantitative: recommendations grounded in data (help make case to management)
3-stage user research Stage 1: Explore Interviews, free-listing Stage 3: Verify / Refine Large-sample closed card-sorting 3 stages Stakeholders User thinking / categorization Preliminary IA Stage 2: Understand Open card-sorting Verify categorization Top-level categories Scope of content & functionality ebay.com Community Buy Sell Sub cat1 SubCat2 SubCat3 xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xx xxx xx xxx xxx xxx SubCat4 SubCat5 SubCat1 SubCat2 xxx xx xx
Stage 1: Explore domain with Freelisting
Explore domain in open-ended fashion, map scope.
Generate list of current & “Horizon Tasks” (planned for future)
Interviews & Freelisting
With eBay users, stakeholders & designers
Freelisting tasks (eBay is about tasks!)
What are all the tasks users can do on eBay.com
List an item, Pay for it
What else will users be able to do on eBay.com 2-5 years that they can’t do today?
Results: List of 100 representative tasks (25 Horizon)
Stage 2: Explore user categorizations
Gain insight into user thinking about eBay
Identify top-level categories for ALL site content and functionality
Method: Interviews & Open card sorting
List of 100 tasks (including 20 Horizon Tasks).
35 participants (including sellers & buyers; new and advanced users).
Identify top levels of hierarchy
Hierarchical Cluster analysis to generate aggregate user categorizations.
Separate analysis for 4 user groups
Identify inconsistencies in categorizations
Reconcile to create one scheme.
Result: Preliminary IA
5 top-level categories accounting for all site content and functionality.
Stage 3: Verify & refine scheme
Verify 5-category scheme covers all of eBay.
What tasks/concepts don’t fit? Where do users expect this info?
Input from larger, more diverse set of users.
Establish method for future verification of IA.
and to generalize scheme to international eBays.
Method: closed card sorting
Large sample (~ 1,000) representing all eBay user types.
Study conducted using online survey.
Participants categorized tasks into one of categories (or selected Other if it didn’t belong)
Site-map showing the structure for all main parts of the eBay.com
Sell - Before
Sell Hub - After
eBay redesigned successfully!
Understand business context
Delineate user categorizations
Create IA blueprint
Deliverable: site-map with structure for main parts of eBay.com.
map used as blueprint of IA redesign (took 2 years).
Generalized to international eBays.
Positive business implications (ROI)!
Users did not protest!
site.com Top Category 3 Top Category1 Top Category 2 Top Category 4 SubCat1 Sub cat1 SubCat2 xx SubCat3 xx xx SubCat2 SubCat3 SubCat5 xx xx xx xx SubCat1 Subcat4 Subcat3 Subcat2 xx xx xxx SubCat3 SubCat5 SubCat4 SubCat6 SubCat4 xxx xx xx xxx xx xxx xxx xxx SubCat7 SubCat4 SubCat5 SubCat6 SubCat1 SubCat2 xx xx xxx xx xxx xx xx SubCat4 xx Preliminary IA
Duct-tape research! There has to be a better way - fast, cheap (relatively) & online…
Why we built MindCanvas
Qualitative research is great, takes a lot of time
Online tools suck, get used anyway (users not engaged)
Business stakeholders respond to quantitative analysis & large sample sizes
Deliverables do not cater to designers
Research findings remain locked up with analyst
Luis Von Ahn’s ESP Game
Rely on games to label the web
Quick, short, engaging games
bringing statistics to designers
back to the future! MindCanvas interpretation
What is MindCanvas? Remote research methods Statistical Analysis & Data Mining Interactive Visualizations loosely coupled, use as qualitative or quantitative
Game-like elicitation methods
Surveys do not engage people. Games are fun!
People engage in complex tasks willingly
Do game-like methods work better?
MindCanvas has game-like elements (its not a game)
Interactive, visual, screen build up, fun
For both qualitative & quantitative
Remote moderated for qualitative
Remote unmoderated for quantitative
Rich interactive visualizations
Large datasets need statistical analysis & data mining
Self-contained files - embedded into PowerPoint, emailed, shared
Interactive visualizations with slides & knobs!
Easy to share with team
Designers want printouts to hang on walls, and scribble on
MindCanvas research focuses on the mind
Inspiration from cognitive anthropology, psychology, market research
Quantitative aggregation & visualization
Categorizations Preferences Language
Who we built for: designer persona Needs convenient methods to do research Does not want to spend time learning statistics The busy designer / researcher who wants to look at patterns visually Want to directly interact with users, does not have time
Who we built for: participant persona “ This was the most fun I have had in a while!” “ It is very easy to complete the task. In fact, I had fun doing this.” The not-tech savvy person who occasionally takes surveys “ I think there are too many cards. But it was kind of fun.”
Where MindCanvas could help
Product Development: Finding the right feature-set for the right users (or personas)
Information Architecture: Understanding how people think for information design.
Early visual design validation: Understanding what people think & feel about a design when its just an image.
MindCanvas in product development
Initial understanding of domain
What’s important to people? What words do they use?
Prioritizing features / outcomes users care about
MindCanvas in Information Architecture
How do people think about a domain?
VocabularyBrowser 1, 2
Is your information architecture effective? Is one scheme more effective than another?