This study compared the search results and usability of a discovery layer system (Ex Libris Primo) to Google Scholar. 12 graduate students performed search tasks on both systems and rated the relevance of the top 10 results. While the relevance of results was generally comparable, users rated Google Scholar significantly higher than Primo in usability and preference. Analysis found usability issues with Primo's workflow between search, results, and full text as well as consistency. The study concluded that discovery layers need improved user interfaces to address these usability problems.

1. User-Centered Evaluation of a Discovery Layer
System with Google Scholar
Tao Zhang
Digital User Experience Specialist
Purdue University Libraries
zhan1022@purdue.edu
Thursday, July 25, 13

2. Library Website is still Important
Breadth of scholarly resources
Credible, highly relevant
Guaranteed full text access
Thursday, July 25, 13

3. User Expectation
“The problem with the databases is that you
can’t content yourself with just searching
one place, like you can with Google. It is
hard for the students to differentiate
between the different databases.”
Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006
Thursday, July 25, 13

4. Thursday, July 25, 13

5. Thursday, July 25, 13

6. Discovery Layer System - Ex Libris Primo®
• Unified search interface for (almost) all library resources
• Catalogs, databases, e-journals, digital collections, and repositories
• Increased discoverability
• Easier, faster, and more efficient search interface
• Faceted browsing/filtering of search results (e.g., by format, date,
subject, topic, etc.)
• Record grouping (e.g., multiple editions or versions of the same
material)
• Consistent“Get it”option
Thursday, July 25, 13

7. Thursday, July 25, 13

8. Study Goals
• Compare users’subjective assessment of search
results from a discovery layer system (Ex Libris
Primo) and Google Scholar
• Evaluate the discovery layer system for usability
issues
Thursday, July 25, 13

9. Method
• 12 graduate students (mean age = 26.3, SD = 3.2)
• Search tasks on Primo and Google Scholar
• Given topic for first three tasks: classroom assessment techniques, food
addition, natural language processing
• Participant own keywords for fourth search task
• Rate first ten results from 1 (highly irrelevant) to 7 (highly relevant)
• Within-subject design, balanced order of search systems, randomized tasks
• After search tasks: System Usability Scale (SUS) ratings, preference ratings,
open questions
Thursday, July 25, 13

10. Relevancy Ratings
Search 1: Article 9 and 10 significantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"
Ra,ng"
Ar,cle"2"
Ra,ng"
Ar,cle"3"
Ra,ng"
Ar,cle"4"
Ra,ng"
Ar,cle"5"
Ra,ng"
Ar,cle"6"
Ra,ng"
Ar,cle"7"
Ra,ng"
Ar,cle"8"
Ra,ng"
Ar,cle"9"
Ra,ng"
Ar,cle"10"
Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13

11. Relevancy Ratings
Search 2: Article 10 significantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"
Ra,ng"
Ar,cle"2"
Ra,ng"
Ar,cle"3"
Ra,ng"
Ar,cle"4"
Ra,ng"
Ar,cle"5"
Ra,ng"
Ar,cle"6"
Ra,ng"
Ar,cle"7"
Ra,ng"
Ar,cle"8"
Ra,ng"
Ar,cle"9"
Ra,ng"
Ar,cle"10"
Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13

12. Relevancy Ratings
Search 3: Article 3 significantly different
1"
2"
3"
4"
5"
6"
7"
8"
9"
Ar-cle"1"
Ra-ng"
Ar-cle"2"
Ra-ng"
Ar-cle"3"
Ra-ng"
Ar-cle"4"
Ra-ng"
Ar-cle"5"
Ra-ng"
Ar-cle"6"
Ra-ng"
Ar-cle"7"
Ra-ng"
Ar-cle"8"
Ra-ng"
Ar-cle"9"
Ra-ng"
Ar-cle"10"
Ra-ng"
Google"Scholar"
Primo"
Thursday, July 25, 13

13. Relevancy Ratings
Search 4: Article 3 significantly different
1"
2"
3"
4"
5"
6"
7"
8"
Ar,cle"1"
Ra,ng"
Ar,cle"2"
Ra,ng"
Ar,cle"3"
Ra,ng"
Ar,cle"4"
Ra,ng"
Ar,cle"5"
Ra,ng"
Ar,cle"6"
Ra,ng"
Ar,cle"7"
Ra,ng"
Ar,cle"8"
Ra,ng"
Ar,cle"9"
Ra,ng"
Ar,cle"10"
Ra,ng"
Google"Scholar"
Primo"
Thursday, July 25, 13

14. Results
• No significant difference of relevancy ratings between
given and participant chosen searches
• Usability ratings and preference of Google Scholar
significantly higher than Primo
• SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43)
• Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38)
• Usability correlated to preference ratings (r = 0.59, p <
0.01)
Thursday, July 25, 13

15. Usability Issues of Primo - Workflow
Search Results List
Identify Article
Link Resolver
Article Page
Thursday, July 25, 13

16. Usability Issues of Primo
!
Thursday, July 25, 13

17. Usability Issues - Consistency
Thursday, July 25, 13

18. Conclusion
Primo’ssearch relevancy may be on par with
Google Scholar, but usability issues have
affected users’perceived usability and
preference.
User interface of discovery layer systems needs
to address workflow and consistency issues.
Thursday, July 25, 13

19. What’s More
• “Blind”test on relevancy ratings
• Test large sample of search keywords
• Server log analysis on keywords, facets, session
length, etc.
• Better (Google Scholar-like?) link resolver
• UI improvement
Thursday, July 25, 13

20. Questions?
zhan1022@purdue.edu
Thursday, July 25, 13