Integrating Behavior User Studies with Log Analysis

Integrating Behavior User Studies with Log Analysis

• 1.
  Integrating Behavior UserStudies with Log Analysis Tao Zhang Assistant Professor Library Science Purdue University Xi Niu Assistant Professor Software and Information Systems University of North Carolina at Charlotte
• 2.
  • Sample logrecord Log analysis 50.117.41.253 - - [01/Sep/2014:00:08:46 -0400] "GET /primo_library/libweb/action/dlSearch.do?institution=PURDUE&vid=PURDUE&ind x=1&bulkSize=20&search_scope=everything&highlight=true&query=any,contains ,hard+time+for+soft+balancing HTTP/1.1" 200 45345 2373 03456FAEC7526F199BD42BEAE95030A5 - 50.117.41.253 • Request URL parsed into components: • Session ID • Search field • Query string • Facets IP Date & Time Request URL Status Bytes Sent Referring URL User Agent
• 3.
  • Data extraction Loganalysis Original log files Sessioned coded components Perl Python Import SAS R • Search behavior metrics: • Search fields, facets • Number of queries in session • Query length • Query formulation and reformulation
• 4.
  + Big data(for all user base) + Unobtrusive + Established metrics + Efficient Log analysis - Task context - System response - User needs and perceptions - User actions and preferences Behavioral user study
• 5.
  • Comparing twodiscovery tools (Nov. 8 to Dec. 7, 2012) Case study Niu, X., Zhang, T., & Chen, H. L. (2014). Study of user search activities with two discovery tools at an academic library. International Journal of Human- Computer Interaction, 30(5), 422-433.
• 6.
  • Search field Loganalysis results Percentage of keyword searches: VuFind: 68.4%, Primo: 88.2%
• 7.
  • Percentage offacet operations in all search actions • VuFind: 8.4% • Primo: 9.7% • Top used facets • VuFind: Format, Access, Topic, Building, Author • Primo: Show Only (Online, Peer-Review, On Shelf), Format, Subject, Publication Date, Library • Nested facet selections are rare Log analysis results
• 8.
  • Query resultsfor Primo Log analysis results Non-electronic resources Mean (SD) Electronic resources Mean (SD) Query length 5.1(5.4) 4.1(4.0) Number of query submissions 3.6 (5.4) 2.6(2.3) Percentages of searches that were reformulated 61.0% 57.8%
• 9.
  • Visualization ofquery reformulation Log analysis results Narrowing: Parallel: Mixed:
• 10.
  • Users predominantlyuse keyword (default) search • Use of facets is relatively low • Most search sessions involve fewer than 4 queries • Average number of words per query is generally less than 3 • More than half of search sessions reformulate queries by adjusting original keywords Summary of log analysis results
• 11.
  • One-and-one usertest • Understand the search context • Designed tasks • Lab observations of user interaction with discovery tools • Query • Search field • Facet • Search results list • Individual item Behavioral user study
• 12.
  Usability lab
• 13.
  Type Instruction Observation Close-endedtask Find the book Introduction to Algorithms by Thomas H. Cormen Query Search field Determine if the library has the book The Machine that Changed the World: The Story of Lean Production by James Womack Query Facet Find the book and video of Wizard of Oz Facet Open-ended task Find a recent journal article on soap operas (as a sociology student) Query Facet Find an e-book on Supply Chain Management Facet Locate the book No Impact Man in a library closest to you Facet User study tasks
• 14.
  • General behaviorpattern: • Start with default search and keyword from instruction • Browse first page of results • Low usage of facets • Reformulate keywords from instruction when target not in top results, not using facets • Users’ difficulties with search results: • Scan potentially large number of results • Identify material type (book, article, journal, video) • Identify format (print, online access) General observations
• 15.
  • Query formulation •Short queries for both open-ended and close-ended searches • Users want more initial search results (just in case …) • Primo may return 0 results for long queries • Query reformulation • When top results not relevant • Limited effect of number of search results • Users preferred reformulating queries than using facets (adding keywords like “book”, “article”, “journal”, etc.) • No clear search strategy, but users tended to narrow a search than to broaden one Log results & observation
• 16.
  • Reasons forlow facet usage: • Interface design • Users’ awareness of available facets • Facet combinations not intuitive • Users’ understanding of the terminology • Users used facets for: • Refine results (Online, Peer-Review, On Shelf) • Exclude unwanted results (publication date) • Library location Log results & observation
• 17.
  Facet UI change BeforeAfter
• 18.
  Search results UIchange Before
• 19.
  Search results designchange After
• 20.
  • User studyinformed by log analysis results • Test tasks targeting certain discovery tool features (facet) • User behavior to observe • Questions about the context • Behavior observations complement mining of big data • Task context • Potential usability issues • Underlying user needs • Data-driven design changes • Search results for visual scanning • Simplified facets display for exploration and interaction What we’ve learned
• 21.
  • Zhang, T., Niu, X., Zhu, L. & Chen, H. (2015). Search in One’s Hand: How Users Search a Mobile Library Catalog. Paper presented at the 17th International Conference on Human- Computer Interaction, Los Angeles, CA. August 2-7, 2015. • Zhang, T., Niu, X., & Promann, M. (in review). Assessing User Experience of E-Books in Academic Libraries. Paper submitted to College and Research Libraries. Subsequentstudies
• 22.
  Questions? zhan1022@purdue.edu xniu2@uncc.edu