Presentation at the 2015 ASIS&T Annual Meeting as part of the panel: Library Assessment and Data Analytics in the Big Data Era: Practice and Policies

1. Integrating Behavior User Studies
with Log Analysis
Tao Zhang
Assistant Professor
Library Science
Purdue University
Xi Niu
Assistant Professor
Software and Information Systems
University of North Carolina at Charlotte

2. • Sample log record
Log analysis
50.117.41.253 - - [01/Sep/2014:00:08:46 -0400] "GET
/primo_library/libweb/action/dlSearch.do?institution=PURDUE&vid=PURDUE&ind
x=1&bulkSize=20&search_scope=everything&highlight=true&query=any,contains
,hard+time+for+soft+balancing HTTP/1.1" 200 45345 2373
03456FAEC7526F199BD42BEAE95030A5 - 50.117.41.253
• Request URL parsed into components:
• Session ID
• Search field
• Query string
• Facets
IP Date & Time Request URL Status Bytes Sent Referring URL User Agent

3. • Data extraction
Log analysis
Original
log files
Sessioned
coded
components
Perl
Python
Import SAS
R
• Search behavior metrics:
• Search fields, facets
• Number of queries in session
• Query length
• Query formulation and reformulation

4. + Big data (for all user base)
+ Unobtrusive
+ Established metrics
+ Efficient
Log analysis
- Task context
- System response
- User needs and perceptions
- User actions and
preferences
Behavioral user study

5. • Comparing two discovery tools (Nov. 8 to Dec. 7, 2012)
Case study
Niu, X., Zhang, T., & Chen, H. L. (2014). Study of user search activities with
two discovery tools at an academic library. International Journal of Human-
Computer Interaction, 30(5), 422-433.

6. • Search field
Log analysis results
Percentage of keyword searches:
VuFind: 68.4%, Primo: 88.2%

7. • Percentage of facet operations in all search actions
• VuFind: 8.4%
• Primo: 9.7%
• Top used facets
• VuFind: Format, Access, Topic, Building, Author
• Primo: Show Only (Online, Peer-Review, On Shelf), Format,
Subject, Publication Date, Library
• Nested facet selections are rare
Log analysis results

8. • Query results for Primo
Log analysis results
Non-electronic
resources
Mean (SD)
Electronic resources
Mean (SD)
Query length 5.1(5.4) 4.1(4.0)
Number of query
submissions
3.6 (5.4) 2.6(2.3)
Percentages of searches
that were reformulated
61.0% 57.8%

9. • Visualization of query reformulation
Log analysis results
Narrowing:
Parallel:
Mixed:

10. • Users predominantly use keyword (default) search
• Use of facets is relatively low
• Most search sessions involve fewer than 4 queries
• Average number of words per query is generally less than
3
• More than half of search sessions reformulate queries by
adjusting original keywords
Summary of log analysis results

11. • One-and-one user test
• Understand the search context
• Designed tasks
• Lab observations of user interaction with discovery tools
• Query
• Search field
• Facet
• Search results list
• Individual item
Behavioral user study

12. Usability lab

13. Type Instruction Observation
Close-ended task Find the book Introduction to
Algorithms by Thomas H.
Cormen
Query
Search field
Determine if the library has the
book The Machine that
Changed the World: The Story
of Lean Production by James
Womack
Query
Facet
Find the book and video of
Wizard of Oz
Facet
Open-ended task Find a recent journal article on
soap operas (as a sociology
student)
Query
Facet
Find an e-book on Supply Chain
Management
Facet
Locate the book No Impact Man
in a library closest to you
Facet
User study tasks

14. • General behavior pattern:
• Start with default search and keyword from instruction
• Browse first page of results
• Low usage of facets
• Reformulate keywords from instruction when target not in top
results, not using facets
• Users’ difficulties with search results:
• Scan potentially large number of results
• Identify material type (book, article, journal, video)
• Identify format (print, online access)
General observations

15. • Query formulation
• Short queries for both open-ended and close-ended searches
• Users want more initial search results (just in case …)
• Primo may return 0 results for long queries
• Query reformulation
• When top results not relevant
• Limited effect of number of search results
• Users preferred reformulating queries than using facets (adding
keywords like “book”, “article”, “journal”, etc.)
• No clear search strategy, but users tended to narrow a search
than to broaden one
Log results & observation

16. • Reasons for low facet usage:
• Interface design
• Users’ awareness of available facets
• Facet combinations not intuitive
• Users’ understanding of the terminology
• Users used facets for:
• Refine results (Online, Peer-Review, On Shelf)
• Exclude unwanted results (publication date)
• Library location
Log results & observation

17. Facet UI change
Before After

18. Search results UI change
Before

19. Search results design change
After

20. • User study informed by log analysis results
• Test tasks targeting certain discovery tool features (facet)
• User behavior to observe
• Questions about the context
• Behavior observations complement mining of big data
• Task context
• Potential usability issues
• Underlying user needs
• Data-driven design changes
• Search results for visual scanning
• Simplified facets display for exploration and interaction
What we’ve learned

21. • Zhang, T., Niu, X., Zhu, L. & Chen, H. (2015). Search in One’s
Hand: How Users Search a Mobile Library Catalog. Paper
presented at the 17th International Conference on Human-
Computer Interaction, Los Angeles, CA. August 2-7, 2015.
• Zhang, T., Niu, X., & Promann, M. (in review). Assessing User
Experience of E-Books in Academic Libraries. Paper submitted
to College and Research Libraries.
Subsequent studies

22. Questions?
zhan1022@purdue.edu
xniu2@uncc.edu