SlideShare a Scribd company logo
1 of 44
Montreal Canada | | 29 October 2019
From Discovery to Fulfilment:
Improving the User Experience at Every Stage
Christopher Cyr, Ph.D.
Associate Research Scientist, OCLC
Two Related Projects
Discovery and Access 2018-2019
• How users navigate path from discovery to access of library
resources
Fulfillment 2019
• Experience when library requests are fulfilled from the user
and the librarians point of view
https://www.flickr.com/photos/lucasartoni/14941319645
Lynn Silipigni Connaway
Director of Library Trends
and User Research
connawal@oclc.org
@LynnConnaway
Peggy Gallagher
Market Analysis
Manager
gallaghp@oclc.org
@PeggyGal1
Christopher Cyr
Associate Research
Scientist
cyrc@oclc.org
@ChrisCyr19
Erin Hood
Research Support
Specialist
hoode@oclc.org
@ErinMHood1
Brittany Brannon
Research
Support Specialist
brannonb@oclc.org
Research team
DISCOVERY AND ACCESS
Discovery and Access Project
• How do academic library users navigate the path
from discovery to access?
• What do academic users do when searches don't result in
fulfillment?
• What differentiates searches that lead to access from
searches that don’t?
• What demographic characteristics influence the access of
users?
• How does access correlate with success?
https://www.flickr.com/photos/75487768@N04/6786332997
Theoretical Background
• Log analysis to collect large amounts of unbiased user data
(Jansen 2006, Connaway and Radford 2017)
• Logs used to study how people use online systems
• Catalog Search failure rates
• Behavior of digital library users
• Use of e-journals
• User experience with video and music streaming services
(Hunter 1991; Jamali, Nicholas, and Huntington 2005; Lamkhede and
Das 2019; Nouvellet, et al. 2019)
https://www.flickr.com/photos/75487768@N04/6786397239
Theoretical Background
• Problems with log analysis
• Ambiguity of log events
• Actions not captured in logs
• Combining log analysis with user interviews
• Users matched up with their search session
• Asked questions about search and analyzed transaction logs
(Connaway, Budd, and Kochtanek 1995)
• No indication that combining search logs with individual interviews
has been used since
https://www.flickr.com/photos/75487768@N04/7598096750
WORLDCAT DISCOVERY
SEARCH LOG ANALYSIS
“Log analysis is everything that a lab study is not.”
(Jansen 2017, 349)
1. Did a keyword search but mistyped
it
- Had 0 results
2. Redid keyword search with correct
spelling
- Had 759,902 results
3. Began typing in additional keyword
4. Selected one of the autosuggested
keyword phrases
- Had 1,761 results
What do the raw logs tell us?
USER INTERVIEWS
“User interviews can help capture search and discovery behavior as the
user understands it, rather than as a computer system understands it.”
(Connaway, Cyr, Brannon, Gallagher, and Hood 2019)
• Search session reconstructed with search log
• Interview protocol developed using critical incident
technique
• Users asked to reflect on specific searches and
requests during the search sessions
Interview Protocol Development
https://www.flickr.com/photos/gpoo/293086443
Example Questions
• “Please tell us what you were looking for and why you
decided to do an online search.”
• “Did the item you were searching for come up in your
search results? In other words, did you find it?”
• “I’d like to understand how you felt about your search
experience overall. Would you say you were delighted with
your search experience?”
https://www.flickr.com/photos/enekobidegain/11327202173
What the logs told us:
• Began keyword search but mistyped it
o Had 0 results
• Redid keyword search with correct
spelling
o Had 759,902 results
• Began typing in additional keyword
• Selected one of the autosuggested
phrases
o Had 1,761 results
• Just starting work on a paper on a broad topic;
didn’t yet have a direction for the paper
• Was overwhelmed with number of search
results
• Abandoned “library search” to do “Google
searching” to better determine a direction for
the paper
• Later came back to the library search and
found it useful
• Also received help from student workers in the
library
• Felt “prepared” to use the library search due to
1st-year library instruction
What Do The Interviews Tell Us?
https://www.flickr.com/photos/eric_duquenoy/7244639090
Interview coding themes
Item formats
Search
strategies
Decision-making
factors
Liked or
desired features
Evaluation of
resources
Feelings of
frustration and
delight
Influence of
librarian
BULK LOG ANALYSIS
Summary of Results
• Average of 5 minutes per session
• Average of 2.2 searches per session
• Average of 5.1 words per search
• 12% of sessions had search refinements
• 33% of sessions had multiple searches
n=282,307 sessions
https://www.flickr.com/photos/eric_duquenoy/7244639090
Types of Requests
Search results
Physical access
options
Online access
attempt
Attempt to save
Physical access
attempt
The user made a request for search results. This could include a new
search, refinement of an existing search, or the addition of limiters.
The user clicked an item or made a request to digitally access
the full text of the item.
The user attempted to export or otherwise save the citation.
The user clicked an item or made a request to place a hold
on a physical copy of the item.
Some users left the system after looking at holding, where they were
able to identify the physical item call number and/or location. These users
were categorized as having the option to physically access the item.
39%
54%
20%
19%
30%
16%
5%
6%
2%
2%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Last requests
(n=274,346
requests)
All click events
(n=1,961,168
events)
All click events vs. Last requests by type of request
Search results Physical access option Online access attempt Attempt to save Physical access attempt
While search results account for over half (54%) of all click
events, they account for just over a third (39%) of last requests
Probability of fulfillment
Number of searches 2
Number of search refinements 0
Words per search 2
Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 0
Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 69.09%
Number of searches 2
Number of search refinements 0
Words per search 7
Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 0
Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 70.32%
Number of searches 2
Number of search refinements 0
Words per search 2
Results per search 1000
Keyword limiter (1 if yes, 0 if no) 1
Author limiter (1 if yes, 0 if no) 1
Title limiter (1 if yes, 0 if no) 0
Chance of Fulfillment 84.76%
Impact of Study
• Identify why and what users did during the search and
when acquiring resources
• Develop a new methodology for studying user behaviors
– Winner of ALISE methodology award
• Influence product and system development
https://www.flickr.com/photos/boklm/37726606441
FULFILLMENT PROJECT
Rationale
• Purpose of the research: Influence and guide strategic
direction and product development of OCLC delivery
services.
• Methodology: The project will entail focus group
interviews with librarians and semi-structured
interviews with faculty members and
graduate/postgraduate students at select OCLC
Research Library Partnership institutions
Image:https://www.flickr.com/photos/satanoid/5851823198/by satanoid/CC BY 2.0
FOCUS GROUP INTERVIEWS
Focus Group Research Questions
Image: https://www.flickr.com/photos/rexness/4513981469/ by Rexness/CC BY-SA 2.0
• RQ1: How do resource sharing/ILL librarians currently spend their
time?
• RQ2: How would they like to spend their time?
• RQ3: What aspects of lending and fulfilment do they believe could be
unmediated?
• RQ4: Where are the inefficiencies in their workflow and what would
help them overcome these and streamline their processes?
• RQ5: What are their expectations, if any, for standardized, shared ILL
and resource sharing policies?
General Information
ACRL
Six participants (n=6)
Cleveland, Ohio
April 2019
ALIA
Nine participants (n=9)
Sydney, Australia
February 2019
Colorado ILL
Fifteen participants (n=15)
Westminster, Colorado
April 2019
Resource Sharing Conference
Ten participants (n=10)
Jacksonville, Florida
March 2019
Total n=40
Image:https://www.flickr.com/photos/thetejon/4051985062/ by Jon/(CC BY-SA 2.0)
Key Librarian Issues
Standardization
Image: https://www.flickr.com/photos/56001877@N04/8126838599/by runmonty/CC BY-NC-ND 2.0
FundingRequest Volume
Standardization
• High number of consortia
• Variation between library systems
• Cross consortia sharing
• Makes logistics difficult to manage
Image: https://www.flickr.com/photos/strausser/10926019275/ by free range jace/CC BY-NC 2.0
Request Volume
• High number of requests
• Many last minute or short notice requests
• Often held responsible for system
problems and user error
Image: https://www.flickr.com/photos/loloieg/3459011184/ by Loloieg/CC BY-ND 2.0
Funding
• Keeping ILL within budget
• Increasing number of ILL holdings
• Cost of licensing
Image: https://www.flickr.com/photos/hsin-yen/35173724402/by Hsin-Yen Lin/CC BY-NC-ND 2.0
SEMI-STRUCTURED INTERVIEWS
Gender
Female: 18
Male:11
Age Ranges
19-25: 5
25-34: 5
35-44: 9
45-54: 6
55-64: 4
Academic Level
Faculty: 15
Graduate/Postgraduate
students: 14
Countries
Australia: 21
USA: 8
Institutions
University of Adelaide: 9
Monash University: 4
Australian National University: 2
University of Melbourne: 6
Montana State University: 5
Swarthmore College: 3
N = 29
Semi-Structured Interview Demographics
Image: https://www.flickr.com/photos/thaths/4314445838/ by thaths/(CC BY-NC 2.0)
Semi-Structured Interview Research Questions
Image: https://www.flickr.com/photos/skellner/4203154536/ by Stefan Kellner/CC BY-NC 2.0
• RQ1: How do graduate/postgraduate students and faculty get access
to resources?
• RQ2: How do they prefer to get access to resources?
• RQ3: Under what circumstances and for what purposes do they
choose ways of accessing resources from the library, e.g., borrowing,
downloading/accessing online., ILL or requests-to-purchase?
• RQ4: Under what circumstances and for what purposes do they
choose ways of accessing resources beyond the library, e.g.,
purchasing from Alibris; purchasing from Amazon, Barnes & Noble,
etc.; borrowing from a friend or a colleague?
Key User Considerations
Image: https://www.flickr.com/photos/duncan/15262071481/by Duncan c/CC BY-NC 2.0
Speed Cost Format
Accessibility
Communication
Speed
• Heavy focus on fast fulfilment
• Speed overshadows quality
• Slow items are considered unavailable/not worth the time
• ILL perceived as a “slower option”
• Digital access regarded as quickest
• “High speed” mentioned in 21 out of 29 total interviews
Image:https://www.flickr.com/photos/fransdewit/4052612802/by Frans de Wit/CC BY-NC-ND 2.0
Cost
• Little mention of specific price points
• Free vs paid rather than cheap vs expensive
• Users wish items were free even when institutions
cover the cost
• Users will ignore paid options if free ones are
available
Image: https://www.flickr.com/photos/davem/14803922/ by Dave & Lorelle/CC BY-NC 2.0
Format
• Speed and cost important to users
• Dependent on context
• Some users have strong preference for one format
Image:https://www.flickr.com/photos/laurie_brooker/354514974/by Laurie Brooker/CC BY-NC-ND 2.0
Digital
• Journals, eBooks and other academic materials
• Flexibility of access and use
• Users access digital materials from multiple sources
• Google Scholar
• Databases
• Amazon
Image: https://www.flickr.com/photos/130540439@N05/16544443591/by LT Photography/CC BY-NC-ND 2.0
Physical
• Almost always prefer physical books
• For both personal and professional use
• For immersion and accessibility
• Frequent reference to tactility and “feel”
Image: https://www.flickr.com/photos/ncole458/5251071720/ by Nicholas Cole/CC BY 2.0
ILL and Communication Clarity
• Users tend to avoid ILL
• Concerns about clarity of communication
between library and user
• Prefer correspondence via email or automated
messages
• Puts user’s mind at ease for time sensitive
materials
• Users don’t want unnecessary updates, but want
to be informed of status of requests
Image: https://www.flickr.com/photos/huangce/8390725539/ by TripNotice.com/CC BY 2.0
Faculty and Graduate/Postgraduate Student Responses
• Graduate/postgraduate students appear to be more sensitive to
speed than faculty
• Graduate/postgraduate students have greater preference for PDFs
than faculty
• Open content mentioned more by Australians than Americans
• Faculty more interested in open content than
graduate/postgraduate students
• Americans want more clarity in communication
• No difference in faculty and graduate/postgraduate student need
for clarity in communication
• Most users search Google to find information and resources
Recommendations Based on User Interviews
Image:https://www.flickr.com/photos/majiksbox/24923950387/by Dave – resting/CC BY-SA 2.0
• Clarity of communication (Tracking)
• Convenience/Ease of use
• Consolidation/One stop shopping
• Citation tools
• Speed/Timeliness
#ALAAC19
• Path from discovery to access to fulfilment a complex,
interlinked process
• While there are many points of frustration along the
way…
• …Users are generally understanding and appreciative
of library services
Overall Conclusions
https://www.flickr.com/photos/axelrd/4019174359
ACKNOWLEDGEMENTS
We would like to thank Nick Spence for his assistance in
preparing this presentation.
Questions &
Discussion
Christopher Cyr, Ph.D.
Associate Research Scientist, OCLC
cyrc@oclc.org
@ChrisCyr19

More Related Content

What's hot

Assessing user experience of e-books in academic libraries
Assessing user experience of e-books in academic librariesAssessing user experience of e-books in academic libraries
Assessing user experience of e-books in academic librariesTao Zhang
 
Ux Research Portfolio
Ux Research PortfolioUx Research Portfolio
Ux Research PortfolioTao Zhang
 
Research Portfolio - Josh LaMar
Research Portfolio - Josh LaMarResearch Portfolio - Josh LaMar
Research Portfolio - Josh LaMarJosh LaMar
 
Get Better Content with Analytics and User Testing
Get Better Content with Analytics and User TestingGet Better Content with Analytics and User Testing
Get Better Content with Analytics and User TestingMichael Powers
 
Search Analytics for Content Strategists
Search Analytics for Content StrategistsSearch Analytics for Content Strategists
Search Analytics for Content StrategistsLouis Rosenfeld
 
[Jaalouk, Vivas-Thomas] SR15 Poster
[Jaalouk, Vivas-Thomas] SR15 Poster[Jaalouk, Vivas-Thomas] SR15 Poster
[Jaalouk, Vivas-Thomas] SR15 PosterLuciana Jaalouk
 

What's hot (6)

Assessing user experience of e-books in academic libraries
Assessing user experience of e-books in academic librariesAssessing user experience of e-books in academic libraries
Assessing user experience of e-books in academic libraries
 
Ux Research Portfolio
Ux Research PortfolioUx Research Portfolio
Ux Research Portfolio
 
Research Portfolio - Josh LaMar
Research Portfolio - Josh LaMarResearch Portfolio - Josh LaMar
Research Portfolio - Josh LaMar
 
Get Better Content with Analytics and User Testing
Get Better Content with Analytics and User TestingGet Better Content with Analytics and User Testing
Get Better Content with Analytics and User Testing
 
Search Analytics for Content Strategists
Search Analytics for Content StrategistsSearch Analytics for Content Strategists
Search Analytics for Content Strategists
 
[Jaalouk, Vivas-Thomas] SR15 Poster
[Jaalouk, Vivas-Thomas] SR15 Poster[Jaalouk, Vivas-Thomas] SR15 Poster
[Jaalouk, Vivas-Thomas] SR15 Poster
 

Similar to “From Discovery to Fulfillment: Improving the User Experience at Every Stage.”

Hot topic: User experience.
Hot topic: User experience.Hot topic: User experience.
Hot topic: User experience.Lynn Connaway
 
SPSBE14 Intranet Search #fail
SPSBE14 Intranet Search #failSPSBE14 Intranet Search #fail
SPSBE14 Intranet Search #failBen van Mol
 
SharePoint Saturday Belgium 2013 Intranet search fail
SharePoint Saturday Belgium 2013 Intranet search failSharePoint Saturday Belgium 2013 Intranet search fail
SharePoint Saturday Belgium 2013 Intranet search failBIWUG
 
UCISA Learning Anaytics Pre-Conference Workshop
UCISA Learning Anaytics Pre-Conference WorkshopUCISA Learning Anaytics Pre-Conference Workshop
UCISA Learning Anaytics Pre-Conference WorkshopMike Moore
 
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...Auraria Library Website Redesign Project Needs Assessment: What Did Students ...
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...Neena Weng
 
Managing Ireland's Research Data - 3 Research Methods
Managing Ireland's Research Data - 3 Research MethodsManaging Ireland's Research Data - 3 Research Methods
Managing Ireland's Research Data - 3 Research MethodsRebecca Grant
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopLynn Connaway
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopOCLC
 
Cutting the Commute: Assess Authentically and Still Arrive on Time
Cutting the Commute:  Assess Authentically and Still Arrive on TimeCutting the Commute:  Assess Authentically and Still Arrive on Time
Cutting the Commute: Assess Authentically and Still Arrive on TimeToni Carter
 
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...Nebraska Library Commission
 
Ethnographically-inspired usability testing
Ethnographically-inspired usability testingEthnographically-inspired usability testing
Ethnographically-inspired usability testingLeah Emary
 
Owning the Discovery Experience for Your Patrons
Owning the Discovery Experience for Your PatronsOwning the Discovery Experience for Your Patrons
Owning the Discovery Experience for Your PatronsRobert H. McDonald
 
OCLC Research update: Active engagement.
OCLC Research update: Active engagement.OCLC Research update: Active engagement.
OCLC Research update: Active engagement.Lynn Connaway
 
Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Lynn Connaway
 
Use of a follow-up survey for improvement of a digital library
Use of a follow-up survey for improvement of a digital libraryUse of a follow-up survey for improvement of a digital library
Use of a follow-up survey for improvement of a digital libraryKathryn Brockmeier
 
User research for Product Managers - Product Tank London Jan 17
User research for Product Managers - Product Tank London Jan 17User research for Product Managers - Product Tank London Jan 17
User research for Product Managers - Product Tank London Jan 17Morag McLaren
 
Online Catalogs: What Users and Librarians Want
Online Catalogs: What Users and Librarians WantOnline Catalogs: What Users and Librarians Want
Online Catalogs: What Users and Librarians WantKaren S Calhoun
 
Wc Usability Online Catalogs Combined August2009 Rev1 Ch
Wc Usability Online Catalogs Combined August2009 Rev1 ChWc Usability Online Catalogs Combined August2009 Rev1 Ch
Wc Usability Online Catalogs Combined August2009 Rev1 ChOCLC LAC
 
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar
 

Similar to “From Discovery to Fulfillment: Improving the User Experience at Every Stage.” (20)

Hot topic: User experience.
Hot topic: User experience.Hot topic: User experience.
Hot topic: User experience.
 
SPSBE14 Intranet Search #fail
SPSBE14 Intranet Search #failSPSBE14 Intranet Search #fail
SPSBE14 Intranet Search #fail
 
SharePoint Saturday Belgium 2013 Intranet search fail
SharePoint Saturday Belgium 2013 Intranet search failSharePoint Saturday Belgium 2013 Intranet search fail
SharePoint Saturday Belgium 2013 Intranet search fail
 
UCISA Learning Anaytics Pre-Conference Workshop
UCISA Learning Anaytics Pre-Conference WorkshopUCISA Learning Anaytics Pre-Conference Workshop
UCISA Learning Anaytics Pre-Conference Workshop
 
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...Auraria Library Website Redesign Project Needs Assessment: What Did Students ...
Auraria Library Website Redesign Project Needs Assessment: What Did Students ...
 
Managing Ireland's Research Data - 3 Research Methods
Managing Ireland's Research Data - 3 Research MethodsManaging Ireland's Research Data - 3 Research Methods
Managing Ireland's Research Data - 3 Research Methods
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
 
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive WorkshopUsing Qualitative Methods for Library Evaluation: An Interactive Workshop
Using Qualitative Methods for Library Evaluation: An Interactive Workshop
 
Cutting the Commute: Assess Authentically and Still Arrive on Time
Cutting the Commute:  Assess Authentically and Still Arrive on TimeCutting the Commute:  Assess Authentically and Still Arrive on Time
Cutting the Commute: Assess Authentically and Still Arrive on Time
 
Ucr summon
Ucr summonUcr summon
Ucr summon
 
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
NCompass Live: ACRL Outcome Measurement Made Easy: Project Outcome for Academ...
 
Ethnographically-inspired usability testing
Ethnographically-inspired usability testingEthnographically-inspired usability testing
Ethnographically-inspired usability testing
 
Owning the Discovery Experience for Your Patrons
Owning the Discovery Experience for Your PatronsOwning the Discovery Experience for Your Patrons
Owning the Discovery Experience for Your Patrons
 
OCLC Research update: Active engagement.
OCLC Research update: Active engagement.OCLC Research update: Active engagement.
OCLC Research update: Active engagement.
 
Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.Digging into assessment data: Tips, tricks, and tools of the trade.
Digging into assessment data: Tips, tricks, and tools of the trade.
 
Use of a follow-up survey for improvement of a digital library
Use of a follow-up survey for improvement of a digital libraryUse of a follow-up survey for improvement of a digital library
Use of a follow-up survey for improvement of a digital library
 
User research for Product Managers - Product Tank London Jan 17
User research for Product Managers - Product Tank London Jan 17User research for Product Managers - Product Tank London Jan 17
User research for Product Managers - Product Tank London Jan 17
 
Online Catalogs: What Users and Librarians Want
Online Catalogs: What Users and Librarians WantOnline Catalogs: What Users and Librarians Want
Online Catalogs: What Users and Librarians Want
 
Wc Usability Online Catalogs Combined August2009 Rev1 Ch
Wc Usability Online Catalogs Combined August2009 Rev1 ChWc Usability Online Catalogs Combined August2009 Rev1 Ch
Wc Usability Online Catalogs Combined August2009 Rev1 Ch
 
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018 Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
Ria Sankar - How to Build Winning Products - Product School Bellevue - 83018
 

More from Lynn Connaway

The Evolving Collection and Shift to Open
The Evolving Collection and Shift to OpenThe Evolving Collection and Shift to Open
The Evolving Collection and Shift to OpenLynn Connaway
 
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...Researching Students’ Information Choices (RSIC): Determining Identity and Ju...
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...Lynn Connaway
 
How Research and Community Inputs Fuel the Library On-Demand.
How Research and Community Inputs Fuel the Library On-Demand.How Research and Community Inputs Fuel the Library On-Demand.
How Research and Community Inputs Fuel the Library On-Demand.Lynn Connaway
 
OCLC delivery services: The library on-demand.
OCLC delivery services: The library on-demand.OCLC delivery services: The library on-demand.
OCLC delivery services: The library on-demand.Lynn Connaway
 
Interdisciplinary approaches to research methods in information behavior stu...
 Interdisciplinary approaches to research methods in information behavior stu... Interdisciplinary approaches to research methods in information behavior stu...
Interdisciplinary approaches to research methods in information behavior stu...Lynn Connaway
 
Take action: Using and presenting research findings to make your case.
Take action: Using and presenting research findings to make your case.Take action: Using and presenting research findings to make your case.
Take action: Using and presenting research findings to make your case.Lynn Connaway
 
User-centered assessment: Leveraging what you know and filling in the gaps.
User-centered assessment: Leveraging what you know and filling in the gaps. User-centered assessment: Leveraging what you know and filling in the gaps.
User-centered assessment: Leveraging what you know and filling in the gaps. Lynn Connaway
 
Public libraries respond to the opioid crisis in collaboration with their com...
Public libraries respond to the opioid crisis in collaboration with their com...Public libraries respond to the opioid crisis in collaboration with their com...
Public libraries respond to the opioid crisis in collaboration with their com...Lynn Connaway
 
"I like interlibrary loans a lot. I don’t that three- or four-day turnaround...
 "I like interlibrary loans a lot. I don’t that three- or four-day turnaround... "I like interlibrary loans a lot. I don’t that three- or four-day turnaround...
"I like interlibrary loans a lot. I don’t that three- or four-day turnaround...Lynn Connaway
 
"That little bit of information can go a long way." The importance of library...
"That little bit of information can go a long way." The importance of library..."That little bit of information can go a long way." The importance of library...
"That little bit of information can go a long way." The importance of library...Lynn Connaway
 
From research to reality: Transforming libraries for a global information world.
From research to reality: Transforming libraries for a global information world.From research to reality: Transforming libraries for a global information world.
From research to reality: Transforming libraries for a global information world.Lynn Connaway
 
Authority, context, and containers: Student perceptions and judgments when us...
Authority, context, and containers: Student perceptions and judgments when us...Authority, context, and containers: Student perceptions and judgments when us...
Authority, context, and containers: Student perceptions and judgments when us...Lynn Connaway
 
“It [library tour] wasn’t what do you do when you need to make a literature r...
“It [library tour] wasn’t what do you do when you need to make a literature r...“It [library tour] wasn’t what do you do when you need to make a literature r...
“It [library tour] wasn’t what do you do when you need to make a literature r...Lynn Connaway
 
Demystifying assessment: Crafting a message to communicate results that addr...
 Demystifying assessment: Crafting a message to communicate results that addr... Demystifying assessment: Crafting a message to communicate results that addr...
Demystifying assessment: Crafting a message to communicate results that addr...Lynn Connaway
 
Container collapse: Student search choices and implications for instructional...
Container collapse: Student search choices and implications for instructional...Container collapse: Student search choices and implications for instructional...
Container collapse: Student search choices and implications for instructional...Lynn Connaway
 
Public libraries respond to the opioid crisis collaboration with their commun...
Public libraries respond to the opioid crisis collaboration with their commun...Public libraries respond to the opioid crisis collaboration with their commun...
Public libraries respond to the opioid crisis collaboration with their commun...Lynn Connaway
 
A user-centered perspective: Integrating qualitative research methods into th...
A user-centered perspective: Integrating qualitative research methods into th...A user-centered perspective: Integrating qualitative research methods into th...
A user-centered perspective: Integrating qualitative research methods into th...Lynn Connaway
 
Using qualitative methods for library and information science research: An in...
Using qualitative methods for library and information science research: An in...Using qualitative methods for library and information science research: An in...
Using qualitative methods for library and information science research: An in...Lynn Connaway
 
OCLC strategy and vision: The library on-demand.
OCLC strategy and vision: The library on-demand.OCLC strategy and vision: The library on-demand.
OCLC strategy and vision: The library on-demand.Lynn Connaway
 
Re(Casting) call: Sculpting services & strategies for cultivating online sch...
 Re(Casting) call: Sculpting services & strategies for cultivating online sch... Re(Casting) call: Sculpting services & strategies for cultivating online sch...
Re(Casting) call: Sculpting services & strategies for cultivating online sch...Lynn Connaway
 

More from Lynn Connaway (20)

The Evolving Collection and Shift to Open
The Evolving Collection and Shift to OpenThe Evolving Collection and Shift to Open
The Evolving Collection and Shift to Open
 
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...Researching Students’ Information Choices (RSIC): Determining Identity and Ju...
Researching Students’ Information Choices (RSIC): Determining Identity and Ju...
 
How Research and Community Inputs Fuel the Library On-Demand.
How Research and Community Inputs Fuel the Library On-Demand.How Research and Community Inputs Fuel the Library On-Demand.
How Research and Community Inputs Fuel the Library On-Demand.
 
OCLC delivery services: The library on-demand.
OCLC delivery services: The library on-demand.OCLC delivery services: The library on-demand.
OCLC delivery services: The library on-demand.
 
Interdisciplinary approaches to research methods in information behavior stu...
 Interdisciplinary approaches to research methods in information behavior stu... Interdisciplinary approaches to research methods in information behavior stu...
Interdisciplinary approaches to research methods in information behavior stu...
 
Take action: Using and presenting research findings to make your case.
Take action: Using and presenting research findings to make your case.Take action: Using and presenting research findings to make your case.
Take action: Using and presenting research findings to make your case.
 
User-centered assessment: Leveraging what you know and filling in the gaps.
User-centered assessment: Leveraging what you know and filling in the gaps. User-centered assessment: Leveraging what you know and filling in the gaps.
User-centered assessment: Leveraging what you know and filling in the gaps.
 
Public libraries respond to the opioid crisis in collaboration with their com...
Public libraries respond to the opioid crisis in collaboration with their com...Public libraries respond to the opioid crisis in collaboration with their com...
Public libraries respond to the opioid crisis in collaboration with their com...
 
"I like interlibrary loans a lot. I don’t that three- or four-day turnaround...
 "I like interlibrary loans a lot. I don’t that three- or four-day turnaround... "I like interlibrary loans a lot. I don’t that three- or four-day turnaround...
"I like interlibrary loans a lot. I don’t that three- or four-day turnaround...
 
"That little bit of information can go a long way." The importance of library...
"That little bit of information can go a long way." The importance of library..."That little bit of information can go a long way." The importance of library...
"That little bit of information can go a long way." The importance of library...
 
From research to reality: Transforming libraries for a global information world.
From research to reality: Transforming libraries for a global information world.From research to reality: Transforming libraries for a global information world.
From research to reality: Transforming libraries for a global information world.
 
Authority, context, and containers: Student perceptions and judgments when us...
Authority, context, and containers: Student perceptions and judgments when us...Authority, context, and containers: Student perceptions and judgments when us...
Authority, context, and containers: Student perceptions and judgments when us...
 
“It [library tour] wasn’t what do you do when you need to make a literature r...
“It [library tour] wasn’t what do you do when you need to make a literature r...“It [library tour] wasn’t what do you do when you need to make a literature r...
“It [library tour] wasn’t what do you do when you need to make a literature r...
 
Demystifying assessment: Crafting a message to communicate results that addr...
 Demystifying assessment: Crafting a message to communicate results that addr... Demystifying assessment: Crafting a message to communicate results that addr...
Demystifying assessment: Crafting a message to communicate results that addr...
 
Container collapse: Student search choices and implications for instructional...
Container collapse: Student search choices and implications for instructional...Container collapse: Student search choices and implications for instructional...
Container collapse: Student search choices and implications for instructional...
 
Public libraries respond to the opioid crisis collaboration with their commun...
Public libraries respond to the opioid crisis collaboration with their commun...Public libraries respond to the opioid crisis collaboration with their commun...
Public libraries respond to the opioid crisis collaboration with their commun...
 
A user-centered perspective: Integrating qualitative research methods into th...
A user-centered perspective: Integrating qualitative research methods into th...A user-centered perspective: Integrating qualitative research methods into th...
A user-centered perspective: Integrating qualitative research methods into th...
 
Using qualitative methods for library and information science research: An in...
Using qualitative methods for library and information science research: An in...Using qualitative methods for library and information science research: An in...
Using qualitative methods for library and information science research: An in...
 
OCLC strategy and vision: The library on-demand.
OCLC strategy and vision: The library on-demand.OCLC strategy and vision: The library on-demand.
OCLC strategy and vision: The library on-demand.
 
Re(Casting) call: Sculpting services & strategies for cultivating online sch...
 Re(Casting) call: Sculpting services & strategies for cultivating online sch... Re(Casting) call: Sculpting services & strategies for cultivating online sch...
Re(Casting) call: Sculpting services & strategies for cultivating online sch...
 

Recently uploaded

Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designMIPLM
 
How to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPHow to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPCeline George
 
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxGrade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxChelloAnnAsuncion2
 
ACC 2024 Chronicles. Cardiology. Exam.pdf
ACC 2024 Chronicles. Cardiology. Exam.pdfACC 2024 Chronicles. Cardiology. Exam.pdf
ACC 2024 Chronicles. Cardiology. Exam.pdfSpandanaRallapalli
 
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxEPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxRaymartEstabillo3
 
Crayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon ACrayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon AUnboundStockton
 
Full Stack Web Development Course for Beginners
Full Stack Web Development Course  for BeginnersFull Stack Web Development Course  for Beginners
Full Stack Web Development Course for BeginnersSabitha Banu
 
ROOT CAUSE ANALYSIS PowerPoint Presentation
ROOT CAUSE ANALYSIS PowerPoint PresentationROOT CAUSE ANALYSIS PowerPoint Presentation
ROOT CAUSE ANALYSIS PowerPoint PresentationAadityaSharma884161
 
Atmosphere science 7 quarter 4 .........
Atmosphere science 7 quarter 4 .........Atmosphere science 7 quarter 4 .........
Atmosphere science 7 quarter 4 .........LeaCamillePacle
 
What is Model Inheritance in Odoo 17 ERP
What is Model Inheritance in Odoo 17 ERPWhat is Model Inheritance in Odoo 17 ERP
What is Model Inheritance in Odoo 17 ERPCeline George
 
How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17Celine George
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Celine George
 
Earth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatEarth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatYousafMalik24
 
Solving Puzzles Benefits Everyone (English).pptx
Solving Puzzles Benefits Everyone (English).pptxSolving Puzzles Benefits Everyone (English).pptx
Solving Puzzles Benefits Everyone (English).pptxOH TEIK BIN
 
Roles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceRoles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceSamikshaHamane
 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentInMediaRes1
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxAnupkumar Sharma
 
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...JhezDiaz1
 
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPT
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPTECONOMIC CONTEXT - LONG FORM TV DRAMA - PPT
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPTiammrhaywood
 

Recently uploaded (20)

Keynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-designKeynote by Prof. Wurzer at Nordex about IP-design
Keynote by Prof. Wurzer at Nordex about IP-design
 
How to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERPHow to do quick user assign in kanban in Odoo 17 ERP
How to do quick user assign in kanban in Odoo 17 ERP
 
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptxGrade 9 Q4-MELC1-Active and Passive Voice.pptx
Grade 9 Q4-MELC1-Active and Passive Voice.pptx
 
ACC 2024 Chronicles. Cardiology. Exam.pdf
ACC 2024 Chronicles. Cardiology. Exam.pdfACC 2024 Chronicles. Cardiology. Exam.pdf
ACC 2024 Chronicles. Cardiology. Exam.pdf
 
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptxEPANDING THE CONTENT OF AN OUTLINE using notes.pptx
EPANDING THE CONTENT OF AN OUTLINE using notes.pptx
 
Crayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon ACrayon Activity Handout For the Crayon A
Crayon Activity Handout For the Crayon A
 
Full Stack Web Development Course for Beginners
Full Stack Web Development Course  for BeginnersFull Stack Web Development Course  for Beginners
Full Stack Web Development Course for Beginners
 
ROOT CAUSE ANALYSIS PowerPoint Presentation
ROOT CAUSE ANALYSIS PowerPoint PresentationROOT CAUSE ANALYSIS PowerPoint Presentation
ROOT CAUSE ANALYSIS PowerPoint Presentation
 
Atmosphere science 7 quarter 4 .........
Atmosphere science 7 quarter 4 .........Atmosphere science 7 quarter 4 .........
Atmosphere science 7 quarter 4 .........
 
What is Model Inheritance in Odoo 17 ERP
What is Model Inheritance in Odoo 17 ERPWhat is Model Inheritance in Odoo 17 ERP
What is Model Inheritance in Odoo 17 ERP
 
How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17How to Configure Email Server in Odoo 17
How to Configure Email Server in Odoo 17
 
Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17Field Attribute Index Feature in Odoo 17
Field Attribute Index Feature in Odoo 17
 
Raw materials used in Herbal Cosmetics.pptx
Raw materials used in Herbal Cosmetics.pptxRaw materials used in Herbal Cosmetics.pptx
Raw materials used in Herbal Cosmetics.pptx
 
Earth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice greatEarth Day Presentation wow hello nice great
Earth Day Presentation wow hello nice great
 
Solving Puzzles Benefits Everyone (English).pptx
Solving Puzzles Benefits Everyone (English).pptxSolving Puzzles Benefits Everyone (English).pptx
Solving Puzzles Benefits Everyone (English).pptx
 
Roles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in PharmacovigilanceRoles & Responsibilities in Pharmacovigilance
Roles & Responsibilities in Pharmacovigilance
 
Alper Gobel In Media Res Media Component
Alper Gobel In Media Res Media ComponentAlper Gobel In Media Res Media Component
Alper Gobel In Media Res Media Component
 
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptxMULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
MULTIDISCIPLINRY NATURE OF THE ENVIRONMENTAL STUDIES.pptx
 
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
ENGLISH 7_Q4_LESSON 2_ Employing a Variety of Strategies for Effective Interp...
 
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPT
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPTECONOMIC CONTEXT - LONG FORM TV DRAMA - PPT
ECONOMIC CONTEXT - LONG FORM TV DRAMA - PPT
 

“From Discovery to Fulfillment: Improving the User Experience at Every Stage.”

  • 1. Montreal Canada | | 29 October 2019 From Discovery to Fulfilment: Improving the User Experience at Every Stage Christopher Cyr, Ph.D. Associate Research Scientist, OCLC
  • 2. Two Related Projects Discovery and Access 2018-2019 • How users navigate path from discovery to access of library resources Fulfillment 2019 • Experience when library requests are fulfilled from the user and the librarians point of view https://www.flickr.com/photos/lucasartoni/14941319645
  • 3. Lynn Silipigni Connaway Director of Library Trends and User Research connawal@oclc.org @LynnConnaway Peggy Gallagher Market Analysis Manager gallaghp@oclc.org @PeggyGal1 Christopher Cyr Associate Research Scientist cyrc@oclc.org @ChrisCyr19 Erin Hood Research Support Specialist hoode@oclc.org @ErinMHood1 Brittany Brannon Research Support Specialist brannonb@oclc.org Research team
  • 5. Discovery and Access Project • How do academic library users navigate the path from discovery to access? • What do academic users do when searches don't result in fulfillment? • What differentiates searches that lead to access from searches that don’t? • What demographic characteristics influence the access of users? • How does access correlate with success? https://www.flickr.com/photos/75487768@N04/6786332997
  • 6. Theoretical Background • Log analysis to collect large amounts of unbiased user data (Jansen 2006, Connaway and Radford 2017) • Logs used to study how people use online systems • Catalog Search failure rates • Behavior of digital library users • Use of e-journals • User experience with video and music streaming services (Hunter 1991; Jamali, Nicholas, and Huntington 2005; Lamkhede and Das 2019; Nouvellet, et al. 2019) https://www.flickr.com/photos/75487768@N04/6786397239
  • 7. Theoretical Background • Problems with log analysis • Ambiguity of log events • Actions not captured in logs • Combining log analysis with user interviews • Users matched up with their search session • Asked questions about search and analyzed transaction logs (Connaway, Budd, and Kochtanek 1995) • No indication that combining search logs with individual interviews has been used since https://www.flickr.com/photos/75487768@N04/7598096750
  • 8. WORLDCAT DISCOVERY SEARCH LOG ANALYSIS “Log analysis is everything that a lab study is not.” (Jansen 2017, 349)
  • 9. 1. Did a keyword search but mistyped it - Had 0 results 2. Redid keyword search with correct spelling - Had 759,902 results 3. Began typing in additional keyword 4. Selected one of the autosuggested keyword phrases - Had 1,761 results What do the raw logs tell us?
  • 10. USER INTERVIEWS “User interviews can help capture search and discovery behavior as the user understands it, rather than as a computer system understands it.” (Connaway, Cyr, Brannon, Gallagher, and Hood 2019)
  • 11. • Search session reconstructed with search log • Interview protocol developed using critical incident technique • Users asked to reflect on specific searches and requests during the search sessions Interview Protocol Development https://www.flickr.com/photos/gpoo/293086443
  • 12. Example Questions • “Please tell us what you were looking for and why you decided to do an online search.” • “Did the item you were searching for come up in your search results? In other words, did you find it?” • “I’d like to understand how you felt about your search experience overall. Would you say you were delighted with your search experience?” https://www.flickr.com/photos/enekobidegain/11327202173
  • 13. What the logs told us: • Began keyword search but mistyped it o Had 0 results • Redid keyword search with correct spelling o Had 759,902 results • Began typing in additional keyword • Selected one of the autosuggested phrases o Had 1,761 results • Just starting work on a paper on a broad topic; didn’t yet have a direction for the paper • Was overwhelmed with number of search results • Abandoned “library search” to do “Google searching” to better determine a direction for the paper • Later came back to the library search and found it useful • Also received help from student workers in the library • Felt “prepared” to use the library search due to 1st-year library instruction What Do The Interviews Tell Us? https://www.flickr.com/photos/eric_duquenoy/7244639090
  • 14. Interview coding themes Item formats Search strategies Decision-making factors Liked or desired features Evaluation of resources Feelings of frustration and delight Influence of librarian
  • 16. Summary of Results • Average of 5 minutes per session • Average of 2.2 searches per session • Average of 5.1 words per search • 12% of sessions had search refinements • 33% of sessions had multiple searches n=282,307 sessions https://www.flickr.com/photos/eric_duquenoy/7244639090
  • 17. Types of Requests Search results Physical access options Online access attempt Attempt to save Physical access attempt The user made a request for search results. This could include a new search, refinement of an existing search, or the addition of limiters. The user clicked an item or made a request to digitally access the full text of the item. The user attempted to export or otherwise save the citation. The user clicked an item or made a request to place a hold on a physical copy of the item. Some users left the system after looking at holding, where they were able to identify the physical item call number and/or location. These users were categorized as having the option to physically access the item.
  • 18. 39% 54% 20% 19% 30% 16% 5% 6% 2% 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Last requests (n=274,346 requests) All click events (n=1,961,168 events) All click events vs. Last requests by type of request Search results Physical access option Online access attempt Attempt to save Physical access attempt While search results account for over half (54%) of all click events, they account for just over a third (39%) of last requests
  • 19. Probability of fulfillment Number of searches 2 Number of search refinements 0 Words per search 2 Results per search 1000 Keyword limiter (1 if yes, 0 if no) 1 Author limiter (1 if yes, 0 if no) 0 Title limiter (1 if yes, 0 if no) 0 Chance of Fulfillment 69.09% Number of searches 2 Number of search refinements 0 Words per search 7 Results per search 1000 Keyword limiter (1 if yes, 0 if no) 1 Author limiter (1 if yes, 0 if no) 0 Title limiter (1 if yes, 0 if no) 0 Chance of Fulfillment 70.32% Number of searches 2 Number of search refinements 0 Words per search 2 Results per search 1000 Keyword limiter (1 if yes, 0 if no) 1 Author limiter (1 if yes, 0 if no) 1 Title limiter (1 if yes, 0 if no) 0 Chance of Fulfillment 84.76%
  • 20. Impact of Study • Identify why and what users did during the search and when acquiring resources • Develop a new methodology for studying user behaviors – Winner of ALISE methodology award • Influence product and system development https://www.flickr.com/photos/boklm/37726606441
  • 22. Rationale • Purpose of the research: Influence and guide strategic direction and product development of OCLC delivery services. • Methodology: The project will entail focus group interviews with librarians and semi-structured interviews with faculty members and graduate/postgraduate students at select OCLC Research Library Partnership institutions Image:https://www.flickr.com/photos/satanoid/5851823198/by satanoid/CC BY 2.0
  • 24. Focus Group Research Questions Image: https://www.flickr.com/photos/rexness/4513981469/ by Rexness/CC BY-SA 2.0 • RQ1: How do resource sharing/ILL librarians currently spend their time? • RQ2: How would they like to spend their time? • RQ3: What aspects of lending and fulfilment do they believe could be unmediated? • RQ4: Where are the inefficiencies in their workflow and what would help them overcome these and streamline their processes? • RQ5: What are their expectations, if any, for standardized, shared ILL and resource sharing policies?
  • 25. General Information ACRL Six participants (n=6) Cleveland, Ohio April 2019 ALIA Nine participants (n=9) Sydney, Australia February 2019 Colorado ILL Fifteen participants (n=15) Westminster, Colorado April 2019 Resource Sharing Conference Ten participants (n=10) Jacksonville, Florida March 2019 Total n=40 Image:https://www.flickr.com/photos/thetejon/4051985062/ by Jon/(CC BY-SA 2.0)
  • 26. Key Librarian Issues Standardization Image: https://www.flickr.com/photos/56001877@N04/8126838599/by runmonty/CC BY-NC-ND 2.0 FundingRequest Volume
  • 27. Standardization • High number of consortia • Variation between library systems • Cross consortia sharing • Makes logistics difficult to manage Image: https://www.flickr.com/photos/strausser/10926019275/ by free range jace/CC BY-NC 2.0
  • 28. Request Volume • High number of requests • Many last minute or short notice requests • Often held responsible for system problems and user error Image: https://www.flickr.com/photos/loloieg/3459011184/ by Loloieg/CC BY-ND 2.0
  • 29. Funding • Keeping ILL within budget • Increasing number of ILL holdings • Cost of licensing Image: https://www.flickr.com/photos/hsin-yen/35173724402/by Hsin-Yen Lin/CC BY-NC-ND 2.0
  • 31. Gender Female: 18 Male:11 Age Ranges 19-25: 5 25-34: 5 35-44: 9 45-54: 6 55-64: 4 Academic Level Faculty: 15 Graduate/Postgraduate students: 14 Countries Australia: 21 USA: 8 Institutions University of Adelaide: 9 Monash University: 4 Australian National University: 2 University of Melbourne: 6 Montana State University: 5 Swarthmore College: 3 N = 29 Semi-Structured Interview Demographics Image: https://www.flickr.com/photos/thaths/4314445838/ by thaths/(CC BY-NC 2.0)
  • 32. Semi-Structured Interview Research Questions Image: https://www.flickr.com/photos/skellner/4203154536/ by Stefan Kellner/CC BY-NC 2.0 • RQ1: How do graduate/postgraduate students and faculty get access to resources? • RQ2: How do they prefer to get access to resources? • RQ3: Under what circumstances and for what purposes do they choose ways of accessing resources from the library, e.g., borrowing, downloading/accessing online., ILL or requests-to-purchase? • RQ4: Under what circumstances and for what purposes do they choose ways of accessing resources beyond the library, e.g., purchasing from Alibris; purchasing from Amazon, Barnes & Noble, etc.; borrowing from a friend or a colleague?
  • 33. Key User Considerations Image: https://www.flickr.com/photos/duncan/15262071481/by Duncan c/CC BY-NC 2.0 Speed Cost Format Accessibility Communication
  • 34. Speed • Heavy focus on fast fulfilment • Speed overshadows quality • Slow items are considered unavailable/not worth the time • ILL perceived as a “slower option” • Digital access regarded as quickest • “High speed” mentioned in 21 out of 29 total interviews Image:https://www.flickr.com/photos/fransdewit/4052612802/by Frans de Wit/CC BY-NC-ND 2.0
  • 35. Cost • Little mention of specific price points • Free vs paid rather than cheap vs expensive • Users wish items were free even when institutions cover the cost • Users will ignore paid options if free ones are available Image: https://www.flickr.com/photos/davem/14803922/ by Dave & Lorelle/CC BY-NC 2.0
  • 36. Format • Speed and cost important to users • Dependent on context • Some users have strong preference for one format Image:https://www.flickr.com/photos/laurie_brooker/354514974/by Laurie Brooker/CC BY-NC-ND 2.0
  • 37. Digital • Journals, eBooks and other academic materials • Flexibility of access and use • Users access digital materials from multiple sources • Google Scholar • Databases • Amazon Image: https://www.flickr.com/photos/130540439@N05/16544443591/by LT Photography/CC BY-NC-ND 2.0
  • 38. Physical • Almost always prefer physical books • For both personal and professional use • For immersion and accessibility • Frequent reference to tactility and “feel” Image: https://www.flickr.com/photos/ncole458/5251071720/ by Nicholas Cole/CC BY 2.0
  • 39. ILL and Communication Clarity • Users tend to avoid ILL • Concerns about clarity of communication between library and user • Prefer correspondence via email or automated messages • Puts user’s mind at ease for time sensitive materials • Users don’t want unnecessary updates, but want to be informed of status of requests Image: https://www.flickr.com/photos/huangce/8390725539/ by TripNotice.com/CC BY 2.0
  • 40. Faculty and Graduate/Postgraduate Student Responses • Graduate/postgraduate students appear to be more sensitive to speed than faculty • Graduate/postgraduate students have greater preference for PDFs than faculty • Open content mentioned more by Australians than Americans • Faculty more interested in open content than graduate/postgraduate students • Americans want more clarity in communication • No difference in faculty and graduate/postgraduate student need for clarity in communication • Most users search Google to find information and resources
  • 41. Recommendations Based on User Interviews Image:https://www.flickr.com/photos/majiksbox/24923950387/by Dave – resting/CC BY-SA 2.0 • Clarity of communication (Tracking) • Convenience/Ease of use • Consolidation/One stop shopping • Citation tools • Speed/Timeliness
  • 42. #ALAAC19 • Path from discovery to access to fulfilment a complex, interlinked process • While there are many points of frustration along the way… • …Users are generally understanding and appreciative of library services Overall Conclusions https://www.flickr.com/photos/axelrd/4019174359
  • 43. ACKNOWLEDGEMENTS We would like to thank Nick Spence for his assistance in preparing this presentation.
  • 44. Questions & Discussion Christopher Cyr, Ph.D. Associate Research Scientist, OCLC cyrc@oclc.org @ChrisCyr19

Editor's Notes

  1. Image https://www.flickr.com/photos/lucasartoni/14941319645 by Luca Sartoni CC BY-SA 2.0
  2. Jay Where is the content that we could throw these documents into? Caro? Took all user journeys and mapped them out onto this board to see what conceptual model could fit the workflows. This is what we’ve come up with. Informed by Clutch research. See https://insideoclc.sharepoint.com/:b:/s/products/GPM/Operations/EZhEKZvwlpxNuMcMw0gSk8UBR3ANgUUw4jCeOUhj9VALqg?e=yasZQy See also core discovery user journeys: https://insideoclc.sharepoint.com/:w:/s/products/GPM/Operations/Ech0Yj1d4VlNi7y5Hxz6iwEBMgDGZije13zB3nBi3mFBhw?e=PNA8wI
  3. Image: https://www.flickr.com/photos/75487768@N04/6786332997 by barnyz CC BY-NC-ND 2.0 Lynn Add heading – show that this is the scope of this study more succinctly. OCLC embarked on a research project to identify how academic library users navigate the path from discovery to access. The project analyzed all WorldCat Discovery search logs from April 2018 in aggregate and analyzed search logs from specific users combined with interviews with those users. Research Questions Last Updated February 1, 2018 1. How do users navigate the path from discovery to access?      a. Where do people leave the system/discovery?      b. How long are user sessions?      c. With what features do the different types of academic library users interact? 2. What do academic users do when searches don’t result in fulfillment?      a. Go elsewhere for same thing?           i. Where?          ii. Why?      b. Get something different here instead? 3. What differentiates searches that lead to access from searches that don’t lead to access?      a. How many and what percentages of searches lead to access (are delivered/accessed)?      b. How many search terms are used for those searches that lead to access?      c. Do people refine their searches, if they do not find something of interest to them? If so, how do they refine their searches?      d. Does search refinement affect access rates? If so, what is the access rate after search refinement and does that differ between the access rate for searches not refined? 4. What demographic characteristics (personal and institutional) influence the access of users?      a. How are small academic users versus large ARL users using the system the same and differently? 5. To what extent does access to sources correlate/define a successful or unsuccessful search?
  4. Image https://www.flickr.com/photos/75487768@N04/6786397239 by barnyz CC BY-NC-ND 2.0 Lynn Jansen, Bernard J. 2006. “Search Log Analysis: What It Is, What’s Been Done, How to Do It.” Library and Information Science Research 28, no. 3: 407–432.      Connaway, Lynn Silipigni, and Marie L. Radford. 2017. Research Methods in Library and Information Science, 6th ed. Santa Barbara, CA: Libraries Unlimited.    Hunter, Rhonda N. 1991. “Successes and Failures of Patrons Searching the Online Catalog at a Large Academic Library: A Transaction Log Analysis.” RQ 30, no. 3: 395–402.  https://www.jstor.org/stable/25828813.     Jamali, Hamid R., David Nicholas, and Paul Huntington. 2005. “The Use and Users of Scholarly E-Journals: A Review of Log Analysis Studies.” Aslib Proceedings 57, no. 5: 554–571. https://doi.org/10.1108/00012530510634271.     Lamkhede, Sudarshan, and Sudeep Das. 2019. “Challenges in Search on Streaming Services: Netflix Case Study.” In the Proceedings of SIGIR ’19, July 21–25, 2019, Paris, France. https://arxiv.org/pdf/1903.04638.pdf.     Nouvellet, Adrien, Florence D’Alché-Buc, Valérie Baudouin, Christophe Prieur, and François Roueff. 2019. “A Quantitative Analysis of Digital Library User Behaviour Based on Access Logs.” Qualitative and Quantitative Methods in Libraries 7, no.  1: 1–13.  
  5. Image https://www.flickr.com/photos/75487768@N04/7598096750 by barnyz Lynn Connaway, Lynn Silipigni, John M. Budd, and Thomas R. Kochtanek. 1995. “An Investigation of the Use of an Online Catalog: User Characteristics and Transaction Log Analysis.” Library Resources and Technical Services 39, no. 2: 142–152.  
  6. Brittany We started by analyzing the data that was already available to us—search logs from WorldCat Discovery. Benefit: “One gets real behaviors from real users using real systems interacting with real information. Log analysis is everything that a lab study is not.” (Jansen 2017, 349) Caveat: Log data is trace data, meaning it is a trace left behind of what a user did, and it must be interpreted. Jansen, Bernard J. 2017. “Log Analysis.” In Research Methods for Library and Information Science, 6th ed., edited by Lynn Silipigni Connaway and Marie L. Radford, 348-349. Santa Barbara, CA: Libraries Unlimited. 
  7. Brittany One of the first things that we had to establish was what the search logs could tell us about user behavior. So we looked manually at some logs to determine what type of information we could glean. The logs could tell us: Type of search (index) Search terms/string Elements of the system interacted with, in this case the autosuggest feature Evolution of searches in a search session Part of this process was working with libraries to recruit participants so we could connect logs with the users who created them. Logs in the system are anonymous—they can’t be tied to the user who generated them. The participants we recruited gave us the ID from their search, which we then used to pull all of the log events associated with that search. On the left-hand side of this slide you can (maybe) see the search log for this participant, CBU09. What we found in interpreting the logs was that this participant started with a mistyped keyword search that returned 0 results. They corrected their spelling and returned over 700,000 results. They then began typing an additional word into their keyword search and selected one of the autosuggested keyword phrases, returning just under 2,000 results. Once we had interpreted the participant logs, we used that information to build an interview protocol. Peggy will talk about the interview process a little later. CBU09 - Female, 19-25, undergraduate, humanities, Christian Brothers University
  8. Peggy As mentioned earlier, our methodology consisted of both bulk log analysis and user interviews. We conducted semi-structured interviews, using the critical incident technique. How many of you are familiar with the critical incident technique? For those of you who aren't familiar with it, the critical incident technique is a methodology in which a subject is asked to reflect on a specific incident and discuss the cause, description and outcome of the incident as well as his/her feelings and perceptions of the situtation. Connaway, Lynn Silipigni, Chris Cyr, Brittany Brannon, Peggy Gallagher, and Erin Hood. 2019. “Speaking on the Record: Combining Interviews with Search Log Analysis in User Research.” ALISE/ProQuest Methodology Paper Competition Award Winner.    Go over methodology (semi-structured interviews with critical incident technique) Connaway, Lynn Silipigni, and Marie L. Radford. 2018. Survey Research. Webinar presented by ASIS&T, January 23. https://www.slideshare.net/LynnConnaway/survey-research-methods-with-lynn-silipigni-connaway and https://www.youtube.com/watch?v=4dlpAT7MXh0 .
  9. Image https://www.flickr.com/photos/gpoo/293086443 by German Poo-Caamano CC BY 2.0
  10. Image https://www.flickr.com/photos/enekobidegain/11327202173 by Eneko Bidegain CC BY 2.0 Peggy We developed specific protocols for each of our interviewees based on what we learned from their search logs. The protocols included some standard questions such as what they were looking for and why, whether they were successful – for example, if they were doing a known-item search, did the item appear in their search results, and whether they were delighted and/or frustrated with their search experience.
  11. Image https://www.flickr.com/photos/eric_duquenoy/7244639090 by Eric Duquenoy CC BY-NC-ND 2.0 Peggy What we learned from her interview, however, was that she was just getting started searching for resources for a paper and hadn't yet narrowed down her topic. She was overwhelmed by how many search results she got, and so she did indeed abandon the library search. However, she came back to the library search tool at a later time, after she had done some Googling and honed in on her topic. She told us how she found the library search tool very helpful once she knew what she wanted to write about, and that she appreciated the help she received from some student workers. She also mentioned that because of the library instruction she had received in her first year, she felt confident in using the library.  CBU09 - Female, 19-25, undergraduate, humanities, Christian Brothers University
  12. Peggy So, once we had conducted all of our interviews, we reviewed the interview transcripts, developed a code book and then had two different team members code each of the interviews. The themes that emerged from the interviews tended to fall into seven different buckets. Interviewees talked about the various search strategies they used and the factors that went into their decision-making process on which resources they wanted to access. As we've learned in several other studies over the years, convenience and ease of access are major factors. They talked about the formats they were most interested in – PDFs being the most mentioned. They talked about how they evaluate resources such as considering the type of study involved from a particular article and whether the article was peer-reviewed. They mentioned features they like about the current library search tool as well as features they wished it had. Many of them talked about being satisfied with their search experience but wouldn't go as far as saying they were delighted by it and spoke of some of their frustrations with the top frustration being too many search results. And many of them spoke favorably about the impact of working with a librarian for receiving library instruction on their confidence in their information-seeking abilities.  So, that's a quick review of what we did and how we did it. I'm going to turn things back over to Lynn who's going to talk about some of the challenges and benefits of the methodology we used for this study.
  13. Image https://www.flickr.com/photos/eric_duquenoy/7244639090 by Eric Duquenoy CC BY-NC-ND 2.0 Brittany Once we had established those categories, we could begin looking at the overall patterns in our search logs. For the bulk log analysis, we looked at all of the logs in the Discovery system for April of 2018. We found that on average, a search session lasted 5 minutes and contained just over two searches, with an average of about 5 words per search. Twelve percent of sessions showed search refinements and 33% of sessions showed multiple searches.
  14. Christopher Log events for last requests and for the full sessions were categorized in the following way: Online Access Attempt: The user clicked an item or made a request to digitally access the full text of the item.  Physical Access Attempt: The user clicked an item or made a request to place a hold on a physical copy of the item.  Physical Access Option: Some users left the system after looking at holdings. In these cases, users were in a place in the system where they were able to identify the physical item call number and/or location. The interviews revealed that at least some of the participants looked for the item on the library shelf at this point. Since logs cannot reveal what users did after leaving the system, these users were categorized as having the option to physically access the item.  Attempt to Save: The user made an attempt to export or otherwise save the citation.  Search Results: The user made a request for search results. This could include a new search, refinement of an existing search, or the addition of limiters.  Other: These requests and click events did not fit cleanly into any category, and there were no overarching themes to them.
  15. Christopher While 54% of the total click events were to generate search results, only 39% of users left on a search results page. More than 50% of users left the system with a request to access an item online, an attempt to access an item physically, or an option to find its physical location. This discrepancy between all request categories and last request categories suggests that users were more likely to leave the system on a successful note. However, it cannot be determined whether the access attempts were successful, so conclusions must be made cautiously.  
  16. Christopher
  17. Image https://www.flickr.com/photos/boklm/37726606441 by Nicolas Vigier CC0 1.0 Lynn It’s changing the way we think, the way we work together. Research goes beyond the academic… influencing product development We earn the right to share our insights by listening.
  18. Image:https://www.flickr.com/photos/satanoid/5851823198/by satanoid/CC BY 2.0
  19. Image: https://www.flickr.com/photos/rexness/4513981469/ by Rexness/CC BY-SA 2.0
  20. Image:https://www.flickr.com/photos/thetejon/4051985062/ by Jon/(CC BY-SA 2.0)
  21. Image: https://www.flickr.com/photos/56001877@N04/8126838599/by runmonty/CC BY-NC-ND 2.0 Standardization: parsing multiple systems that don’t interact with one another. Makes fast fulfilment difficult for users. Request Volume: Librarians are frustrated by a high volume of short notice requests. Usually from students working on assignments. Funding: Librarians operate on limited funds to meet user fulfilment requests
  22. Image: https://www.flickr.com/photos/strausser/10926019275/ by free range jace/CC BY-NC 2.0 Libraries do business with multiple ILL consortia that have high variation between their systems that makes both the logistics of shipping items and giving projected transit time difficult.
  23. Image: https://www.flickr.com/photos/loloieg/3459011184/ by Loloieg/CC BY-ND 2.0 Librarians are held responsible by users when something goes wrong at any point in the ILL process even though the shipping phase can be prone to errors and slowdown outside of librarian control
  24. Image: https://www.flickr.com/photos/hsin-yen/35173724402/by Hsin-Yen Lin/CC BY-NC-ND 2.0 Many librarians want to expand the resources offered through ILL but funding is a limiting factor.
  25. Image: https://www.flickr.com/photos/thaths/4314445838/ by thaths/(CC BY-NC 2.0)
  26. Image: https://www.flickr.com/photos/skellner/4203154536/ by Stefan Kellner/CC BY-NC 2.0
  27. Image: https://www.flickr.com/photos/duncan/15262071481/by Duncan c/CC BY-NC 2.0
  28. Image:https://www.flickr.com/photos/fransdewit/4052612802/by Frans de Wit/CC BY-NC-ND 2.0 Speed stood out as the most important aspect of ILL even overshadowing quality. ILL is frequently seen as not worth the time needed and unless the item is mandatory it will be skipped by the user. Digital items are preferred when high speed is a necessity.
  29. Image: https://www.flickr.com/photos/oprior/64409980/by Owen Prior/CC BY-SA 2.0
  30. Image:https://www.flickr.com/photos/jhlau/15969832976/ by a.canvas.of.light/CC BY 2.0
  31. Image: https://www.flickr.com/photos/davem/14803922/ by Dave & Lorelle/CC BY-NC 2.0 While price points weren’t mentioned as specifically high or low the issue of paid access vs free access was frequently mentioned. Paid options were unanimously ignored in the presence of a free alternative. If the first choice item turns out to be paid only the user will pass it up in favor of a different item. Users broadly wanted more items to be available for free even outside of universities.
  32. Image: https://www.flickr.com/photos/101643643@N08/15749180169/ by WANG-HSIN PEI/CC BY 2.0
  33. Image:https://www.flickr.com/photos/laurie_brooker/354514974/by Laurie Brooker/CC BY-NC-ND 2.0 Item format (digital physical) was dependent on the situation and the needs of the user except for a few “format loyalists” who overwhelmingly preferred one format type to another.
  34. Image: https://www.flickr.com/photos/130540439@N05/16544443591/by LT Photography/CC BY-NC-ND 2.0 Journals and eBooks were the most frequently mentioned digital resources, accessed through google scholar, specialized databases, and amazon. Physical journals were never mentioned with the exception of remarking that the user had never worked with them or worked with them a long time ago but not anymore. Speed, flexibility and portability were commonly mentioned as the best aspects of digital media.
  35. Image:https://www.flickr.com/photos/kool_skatkat/795393193/by kool_skatkat/(CC BY-NC-ND 2.0)
  36. Image: https://www.flickr.com/photos/ncole458/5251071720/ by Nicholas Cole/CC BY 2.0 Books were overwhelmingly the most popular physical media with immersion into the text (sitting down and reading for an extended period) and the ability to pick up and read the book offline were frequently mentioned. Additionally, books were mentioned for both use in personal and professional settings. While the individual materials didn’t often overlap, the medium of print books was still ubiquitous.
  37. Image:https://www.flickr.com/photos/thetejon/4182681980/by Jon/(CC BY-SA 2.0)
  38. Image: https://www.flickr.com/photos/huangce/8390725539/ by TripNotice.com/CC BY 2.0 Many users have a connotation that ILL is slow and unreliable, this is reinforced by their own avoidance of ILL (self-fulfilling prophecy). Those who do use ILL tend to be satisfied with it but see where improvements could be made. The issue of clarity is important. Many users complain about anxiety in not knowing when their item will arrive or if it will arrive on time. In contrast to this users also mentioned not wanting to be swamped with updates across multiple channels. Emails at key points in shipping were most often cited as ideal.
  39. Image: https://www.flickr.com/photos/rogermeyer/5244084417/ by Rodger Meyer/CC BY-NC-ND 2.0
  40. Image:https://www.flickr.com/photos/fernando/3992060782/by Fernando de Sousa/CC BY-SA 2.0
  41. Image: https://www.flickr.com/photos/147621582@N02/35993391071/by Staffan Andersson/CC0 1.0 Open access was occasionally mentioned but for a handful of interviews it was a key issue they spoke about at length. Open access wasn’t commonly talked about as a present issue but more so as a future goal. Something that would be nice to pursue eventually, once other issues were addressed.
  42. Image: https://www.flickr.com/photos/karlnorling/14668687253/by karlnorling/CC BY 2.0
  43. Image: https://www.flickr.com/photos/duncan/39921909493/by Duncan c/CC BY-NC 2.0
  44. While both groups cared about convenience and speed, graduate/postgraduate students appear to be more sensitive to it. Because of this, they have a much bigger preference for PDFs than faculty. Faculty are more interested in open content than graduate/postgraduate students While Americans and Australians differed in their wish for clarity in communication, faculty and graduate/postgraduate students do not appear to differ at all. Image: https://www.flickr.com/photos/shiridenovo/5318874705/ by Carey Ciuro/CC BY-NC 2.0
  45. Image:https://www.flickr.com/photos/majiksbox/24923950387/by Dave – resting/CC BY-SA 2.0
  46. Image https://www.flickr.com/photos/axelrd/4019174359 by Axel Drainville CC BY-NC 2.0