Testing Taxonomies


Published on

Conference presentation at Taxonomy Boot Camp, 2013, Washington, DC. Part of the session on Evaluation and Testing Taxonomies.

Published in: Technology, Education

Testing Taxonomies

  1. 1. Testing Taxonomies Heather Hedden Hedden Information Management Taxonomy Boot Camp, Washington, DC November 5, 2013
  2. 2. About Heather Hedden Independent taxonomy consultant, Hedden Information Management Continuing education online workshop instructor, Simmons College Graduate School of Library and Information Science Author of The Accidental Taxonomist (Information Today, Inc., 2010) Previously taxonomy consultant employed by a consulting firm taxonomy manager taxonomist for enterprise search tool vendor controlled vocabulary editor at a library database vendor (in Vocabulary and Quality Management department) 2 © 2013 Hedden Information Management
  3. 3. Outline Taxonomy testing overview Types of tests for taxonomies Card sorting A/B testing User/use case testing Indexing testing QA testing Conclusions 3 © 2013 Hedden Information Management
  4. 4. Taxonomy Testing Overview Taxonomies serve a purpose, and that purpose should be tested. All taxonomies, whether created by subject matter experts or taxonomists, should be tested. Testing involves participants, as sample or representative users. Testing can be simple and basic, or elaborate and thorough, depending on budget. Different types of tests are appropriate for different stages of taxonomy development. An inappropriate test or inappropriately timed text can be a waste of time and money. 4 © 2013 Hedden Information Management
  5. 5. Taxonomy Testing Overview Different tests for different stages of taxonomy development Design and development phase: to test ideas Card sorting A/B testing Draft completion phase: to test usability/functionality (may also be considered “validation”) User/use case content retrieval testing Content indexing testing Implemented taxonomy: to periodically test quality QA testing 5 © 2013 Hedden Information Management
  6. 6. Card Sorting Method common in information architecture for website menu label organization Term names/label/topics are written down each one to a card, and the cards can be sorted into groups. Traditionally done with actual index cards. Now usually done through software, usually drag-and-drop and online to allow remote access. Involves participation of multiple stakeholders or test-user subjects Two types: 1. Open sort: participants group terms and assign the groups category names of their own choosing 2. Closed sort: Categories are pre-defined, and participants place terms in the appropriate categories Subscription-based web services: www.optimalsort.com http://uxpunk.com/websort/ (formerly known as Websort) 6 © 2013 Hedden Information Management
  7. 7. Open Sort © 2013 Hedden Information Management
  8. 8. Card Sorting Closed Sort © 2013 Hedden Information Management
  9. 9. Card Sorting Although sometimes called “card sort testing” this is for testing ideas for taxonomy development, not for testing taxonomy functionality. Open sort - early in the taxonomy development process Closed sort - later in the process Additional, functionality testing is still needed. Subtle differences in wording, such as part of speech, can effect sorting choices. Example: Same or different categories? Scuba diving Racquet ball courts Aerobics classes Consider changing term names to make them similar in a group. 9 © 2013 Hedden Information Management
  10. 10. Card Sorting – Issues More suited for website navigation design than for taxonomies: Open card sort – rarely needed for taxonomies Unlike website navigation menus, taxonomies are rarely created from scratch. Usually there are top categories to start from. Closed card sort – purpose of finding the desirable broader category is often not needed Unlike website navigation menus, a topic may have more than one broader category (polyhierarchy) in a taxonomy. Card sorting is useful for a hierarchy of only one narrower level. Impractical to test multi-level hierarchies, which require multiple tests Card sorting is no useful for faceted taxonomies. 10 © 2013 Hedden Information Management
  11. 11. A/B Testing Test subjects are people who will use the taxonomy to find content. Test-users are presented with two different possible scenarios (“A” and “B”) and asked which they prefer Like closed card sort, done at a similar stage to test taxonomy ideas For taxonomy A/B testing, alternative could be: Different wordings of category labels Different ordering/arrangement of categories Different location of subcategories Often done in a user interface mock-up / web page wireframe. Can be performed any time in the taxonomy design and build process. Useful when undertaking a taxonomy redesign. 11 © 2013 Hedden Information Management
  12. 12. A/B Testing – Example Do you like A? (logically grouped) Do you like B? OR (more revealed at top level) © 2013 Hedden Information Management
  13. 13. A/B Testing – Issues Most suitable to compare proposed top-level categories Not practical to conduct a detailed term-by-term comparison Most effective if making use of graphical user interface design Existing or proposed design can be altered in a drawing program Test users may not have the time or patience for numerous A/B tests. Need to decide what to compare and how many comparison tests to make, to conserve time and resources. 13 © 2013 Hedden Information Management
  14. 14. User/Use Case Testing A form of taxonomy validation Tests to see if taxonomy will perform as hoped in search/retrieval Test-users are people who will use the taxonomy to find content. Particularly applicable for users of internal/enterprise taxonomies For public/subscriber access to taxonomies, instead of actual users, subject matter experts or customer support may suggest “typical” use cases. Can test deep hierarchical or faceted taxonomies 14 © 2013 Hedden Information Management
  15. 15. User/Use Case Testing – Procedure 1. Test-users are asked to prepare several use cases (information seeking scenarios), through interviews or written descriptions Most are “typical” scenarios One or two may be recent challenging scenarios 2. Test-users are asked to browse the draft offline taxonomy to look for terms under which the content for each scenario might be found Test users perform the test, either: a) in the test administrator’s (taxonomist’s) physical presence b) via screen-sharing with verbal narration explaining choices made c) independently in an offline worksheet, and then reviewed in a verbal meeting 3. Test administrator takes notes regarding problems in finding taxonomy terms for the use cases. 15 © 2013 Hedden Information Management
  16. 16. User/Use Case Testing – Examples Use case scenarios are initially narrative For licensed content subscribers: Jeff has a question about which employees, within the bank, meet the SAFE Act definition of a “loan originator” and therefore would be required to register. For internal web content management A staff member of the Content Group would like to find a seasonal banner ad for a specific brand to upload it. Minimally must answer: who, what (which can be complex), and for what purpose 16 © 2013 Hedden Information Management
  17. 17. User/Use Case Testing – Outcomes Noted findability problems should be considered as indications for: Additional taxonomy terms Or if the terms exist: Additional nonpreferred terms (synonyms) to point to existing terms More polyhierarchy (multiple drill-down paths to the same specific term) More associative (See also) relationships (if supported) to help guide the user to find the desired concepts © 2013 Hedden Information Management
  18. 18. User/Use Case Testing – Issues Too complex for non-employee test subjects recruited from the general public (unlike card-sorting and A/B testing). Requires advance planning and preparation Test users may have difficulty formulating use cases Taxonomist should develop use cases out of stakeholder interviews Important use cases could still be overlooked. 18 © 2013 Hedden Information Management
  19. 19. Indexing Testing A form of taxonomy validation Tests to see if taxonomy is suitable to index/tag/classify intended content For manual indexing, test-users are the indexers or content creators who assign the tags/categories to content. For auto-classification, taxonomist tests indexing. If implemented in the system, conduct testing in the system If not yet implemented in the system, conduct testing in a similar manner as for manual indexing. (Additional testing, once implemented, will still be needed.) Can test deep hierarchical or faceted taxonomies 19 © 2013 Hedden Information Management
  20. 20. Indexing Testing – Procedure 1. Test-user indexers, content creators, or stakeholders identify a set of varied sample documents/content assets that need indexing. 2. Test-indexers are asked to browse the draft offline taxonomy to look for terms to classify/tag each document or content asset. (Usually done independently) 3. Taxonomist does the same. 4. A meeting between the taxonomist and the test-users discusses the choices of indexing terms made separate meetings, if each test their own individual set of documents joint meeting if the same content was test-indexed by all participants 20 © 2013 Hedden Information Management
  21. 21. Indexing Testing – Example worksheets (Faceted taxonomy) 21 © 2013 Hedden Information Management
  22. 22. Indexing Testing – Outcomes Inability to find a taxonomy term may indicate the need for... Additional taxonomy terms Additional nonpreferred terms (synonyms) to point to existing terms More polyhierarchy (multiple drill-down paths to the same specific term) More associative (See also) relationships (if supported) Uncertainties or inconsistencies may indicate that... a taxonomy term should be reworded for clarity or nonpreferred terms need to be added (usually the case for dedicated indexers) The vocabulary size is larger than it needs to be and could be shortened and simplified (more often the case for content authors for whom indexing is an undesired task) Use of N/A may indicate that an indexing policy should not require the use of a certain facet. 22 © 2013 Hedden Information Management
  23. 23. Indexing Testing – Issues Task is time-consuming: sufficient participation of enough participants with a significant number of documents can be difficult Aim for 4-5 documents/assets per participant Not simulating the indexing/tagging user interface Inconclusive results, that may just require policies and training 23 © 2013 Hedden Information Management
  24. 24. QA Testing Test searches (or browsing) to test taxonomy in retrieval results. Performed after the taxonomy and content is implemented in the system and periodically thereafter. Taxonomist performs random searches Selects random terms from the taxonomy to see if appropriate content is retrieved (to test precision) Identifies content items, and checks to see if appropriate taxonomy terms retrieve it (to test recall) Issues: Usually cannot discern if errors are due to taxonomy problems or indexing problems. Need to investigate both. Not always easy to test recall. 24 © 2013 Hedden Information Management
  25. 25. Conclusions Test Testers Purpose Card sorting Stakeholders or sample users/searchers To test taxonomy structure ideas - but often not practical for taxonomies (in contrast to testing website menu navigation) A/B Testing Sample users/searchers Use case Sample Testing users/searchers To test suitability to retrieve desired content Indexing testing Sample indexers/taggers or taxonomist To test suitability to tag content and appropriateness for content scope QA testing Taxonomist To test continued precision and recall in retrieval results © 2013 Hedden Information Management
  26. 26. Conclusions Evaluating vs. Testing Taxonomies Evaluating a taxonomy to determine if it’s well designed and constructed does not always require having sample content or sample users may be done on an existing or implemented taxonomy done by an expert taxonomist Testing a taxonomy focuses on the specific application and use of the taxonomy involves using sample content and sample users part of the taxonomy development process done by sample or representative users, facilitated by a taxonomist Testing (aside from idea-testing in card-sorting and A/B testing) should come after initial taxonomy evaluation and revisions. © 2013 Hedden Information Management
  27. 27. Questions/Contact Heather Hedden Hedden Information Management Carlisle, MA heather@hedden.net 978-467-5195 www.hedden-information.com www.linkedin.com/in/hedden twitter.com/hhedden accidental-taxonomist.blogspot.com 27 © 2013 Hedden Information Management