Taxonomy Validation
Upcoming SlideShare
Loading in...5
×
 

Taxonomy Validation

on

  • 5,644 views

This presentation covers validation techniques for testing taxonomy and metadata with users. Four approaches are covered: Delphi card sorting, online card sorting, usability testing, and search term ...

This presentation covers validation techniques for testing taxonomy and metadata with users. Four approaches are covered: Delphi card sorting, online card sorting, usability testing, and search term analysis. The presentation also contains a list of online card sorting tools.

Statistics

Views

Total Views
5,644
Slideshare-icon Views on SlideShare
5,191
Embed Views
453

Actions

Likes
9
Downloads
113
Comments
0

3 Embeds 453

http://www.scoop.it 429
http://www.slideshare.net 19
http://www.linkedin.com 5

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Taxonomy Validation Taxonomy Validation Presentation Transcript

    • Taxonomy Validation Dave Cooksey Principal Consultant IA Summit 2009 - Memphis March 20, 2009
    • Today’s Presentation ! About Dave ! Testing and Taxonomy | Metadata ! Taxonomy Validation Techniques: ! Delphi Card Sorting ! Remote Card Sorting ! Usability Testing ! Search Analysis ! Discussion ! Resources
    • Dave Cooksey Founder & Principal Consultant saturdave UX Consulting Affiliations ! Chair of PhillyCHI, the Philadelphia region’s chapter of the ACM SIGCHI ! Information Architecture Institute (IAI) ! American Society for Information Science & Technology (ASIS&T) Education ! M.S. Information Systems, Drexel University ! M.A. Political Science, University of Illinois Previous Work Experience ! GSI Commerce, Inc.: User Experience Lead; Information Architect ! LandAmerica Financial Group: (Web) Application Developer
    • Testing and Taxonomy | Metadata
    • Taxonomy ! Is an expert activity ! Comes from a privileged view ! Is more closely aligned to business & technology goals than user goals Taxonomies & Metadata ! Hold implicit meanings through ! Hierarchy & relationships ! Labeling conventions used ! Are interpreted from the user’s experience, context, & goals But the experience, context, & goals of the users are rarely our own!
    • Taxonomy Validation Is Needed ! Ensures that information is organized for the user ! Justifies the cost of taxonomy research & analysis ! Reassures the business that project focus is on success ! Helps alleviate organizational You’ve been warned: Princeton Record Exchange. conflict ! Vets untried ideas with real users
    • Taxonomy Validation Techniques
    • Taxonomy Validation Techniques ! Delphi Card Sorting ! Direct user testing method ! Test proposed design with real users ! Remote Card Sorting ! Direct user testing method ! Examine specific categories | items ! Usability Testing ! Direct user testing method ! Focus on interactions ! Search Analysis ! Indirect testing method ! Adjust taxonomy | metadata terms to match user vocabulary
    • Delphi Card Sorting Qualitative | Small Sample Based on the Delphi Method (RAND Corporation) Developed by Professor Kathryn Summers and Celeste Lyn Paul at University of Baltimore Hierarchy is laid out in cards - seeded or not Test for both categories and individual items Participants modify the structure one at a time Runs for 15 - 20 participants until hierarchy “stabilizes” Watch and learn - Interview for detail Can also present navigation schema, images for labeling, filters | facets
    • Step-by-Step Delphi Card Sorting 1. Decide to seed deck or let first user create seed 2. Think of other pertinent test points: navigation, facets, images for labeling 3. Put categories on cards (You or first participant) 4. Put representative items on cards | script (You) 5. Interview participant a. Explain exercise b. Participant organizes and comments c. Watch and interview 6. Repeat with participants until you are satisfied 7. Analyze data
    • Performing Delphi Card Sorting 1. User starts with deck from previous participant 2. Participant modifies card deck a. Removing category - turning the card over b. Changing labels - creating new card & place over old one c. Moving category - turn card over & create new card in new place 3. Participant examines changes made to card deck a. Looks at existing cards & placement b. Turns cards over to see previous changes to labels & placement 4. Moderator interviews informally during session
    • Different than Traditional Card Sorting Targets to group understanding, not to individual understandings Participants pick up where previous participant left off Assumes some level of “expertise” The end result is the “final” hierarchy Changes path reduces analysis time Informs the “black box” (Thanks, Michael)
    • The Question of Expertise Delphi Method designed for use with groups of experts Each expert contributes her knowledge to building a consensus view Outliers are removed through iteration Consider... What kind of users are you testing? Do they have domain expertise? Do you have multiple user types? Do you have a general population (public website)?
    • Tips for Delphi Card Sorting Plan for lots of table space for cards Allow for plenty of time between participants Create a “starter” sheet to capture notes Take pictures of table after each participant Detail is great, but keep your eye on patterns Video recording is essential Use questions to explore what the user is doing Employ multiple methods to solicit feedback
    • Remote Card Sorting Quantitative | Large Sample ! Online tools allow testing hierarchy | labels ! Either open or closed ! Closed - place items in hierarchy - tests structure ! Open - create categories for items - gathers many perspectives ! Recruit via intercept or email ! Easily run for large sample sizes ! Captures national sample ! Good for identifying patterns ! Use with in-person card sorting for balance
    • Step-by-Step Remote Card Sorting 1. Decide on open or closed card sort 2. Decide population target (online, email, other) 3. Determine sample size needed 4. Choose software package 5. Focus each sort to 15 - 20 items 6. Put items | categories on cards 6. Run study 7. Analyze the data
    • Tips for Remote Card Sorting ! Run a few qualitative interviews to vet if possible ! Scale the test appropriately - run multiples if needed ! Pick important categories | items to test ! Significant to the business but hard to classify ! Stereotypical cases that justify taxonomy guidelines ! Give yourself time to dig into the raw data ! Read the juicy, juicy comments! ! Deliverables depend on software used ! Your satisfaction depends on software used
    • Usability Testing Qualitative | Small Sample | Task-based ! Piggyback on system testing ! From paper prototyping to existing systems ! Great for testing navigation step-by-step (as opposed to birds eye view) ! Helps test other important aspects of taxonomy: facets, tagging, non- hierarchical relationships ! Great for examining the contextual interaction of multiple taxonomy | metadata points ! Demonstrates explicitly the value of taxonomy ! Good for reassuring the business ! Way easy
    • Tips for Usability Testing ! If someone else is testing, simply explain your goals & offer tasks ! Keep your tasks simple ! Pick tasks for the business (most significant) & for you (test the taxonomy) ! Give multiple tasks so moderator can pick for the situation ! Watch the live sessions and ask follow ups ! Take the opportunity to show to other stakeholders how the data is driving the experience ! Offer to create an overview for the study organizer
    • Search Analysis Quantitative | Interpretive | Focused on Patterns ! Uses existing reporting ! In your hands becomes a barometer of user satisfaction, requests, & complaints ! Easily shows you ! What folks want from the system (search results) ! What folks need and are not getting (no search results) ! The vocabulary folks use to describe things ! Great for pointing out opportunities for business & technology ! Easy but time consuming
    • Step-by-Step Search Analysis 1. Find the owner of analytics and ask for advice 2. Determine what’s available to you: search results, no search results, etc. 3. Use spreadsheet to filter and order results in order to highlight patterns 4. Identify: like terms & misspellings, adjectives, frequent terms both returned & not 5. Create summary 6. Discuss summary with business to get contextual detail 7. Create recommendations brief
    • Tips for Search Analysis ! Know the data ! Know the system ! Keep your interpretations simple ! Plan extra time for analysis - it’s time consuming ! When delivering results, create 2 views: overview & detail ! Pass results to teams that can benefit: business, search, technology, marketing ! Works best when used with other methods / techniques
    • Discussion
    • Resources
    • Validation Tools Recruitment Ethnio: http://ethnio.com Sample Size Calculator Creative Research Systems Calculator: http://www.surveysystem.com/sscalc.htm Online Card Sorting Tools The Mind Canvas: http://themindcanvas.com OptimalSort: http://optimalsort.com Socratic CardSortsm: http://www.sotech.com/ WebCat (NIST): http://zing.ncsl.nist.gov/WebTools/WebCAT/overview.html WebSort: http://websort.net/ STC’s Usability & User Experience List of Card Sorting Tools: http://www.stcsig.org/usability/topics/cardsorting.html Others?
    • For Your Reading Pleasure The Delphi Method for Card Sorting “A Modified Delphi Approach to a New Card Sorting Methodology” by Celeste Lyn Paul: http://www.upassoc.org/upa_publications/jus/ 2008november/paul1.html “A Delphi Approach to Card Sorting” by Celeste Lyn Paul (PDF): http://iasummit.org/proceedings/2007/files/Delphi_Card_Sorting.pdf
    • That’s All, Folks! Dave Cooksey saturdave UX Consulting 713 Pine Street 1R Philadelphia, PA 19106 +1.215.219.8960 dave@saturdave.com © 2009 saturdave