User-Centered Evaluation of a Discovery Layer System with Google Scholar

  • 114 views
Uploaded on

Presentation at the Human-Computer Interaction International (HCII) 2013 Conference

Presentation at the Human-Computer Interaction International (HCII) 2013 Conference

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
114
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
6
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. User-Centered Evaluation of a Discovery Layer System with Google Scholar Tao Zhang Digital User Experience Specialist Purdue University Libraries zhan1022@purdue.edu Thursday, July 25, 13
  • 2. Library Website is still Important Breadth of scholarly resources Credible, highly relevant Guaranteed full text access Thursday, July 25, 13
  • 3. User Expectation “The problem with the databases is that you can’t content yourself with just searching one place, like you can with Google. It is hard for the students to differentiate between the different databases.” Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006 Thursday, July 25, 13
  • 4. Thursday, July 25, 13
  • 5. Thursday, July 25, 13
  • 6. Discovery Layer System - Ex Libris Primo® • Unified search interface for (almost) all library resources • Catalogs, databases, e-journals, digital collections, and repositories • Increased discoverability • Easier, faster, and more efficient search interface • Faceted browsing/filtering of search results (e.g., by format, date, subject, topic, etc.) • Record grouping (e.g., multiple editions or versions of the same material) • Consistent“Get it”option Thursday, July 25, 13
  • 7. Thursday, July 25, 13
  • 8. Study Goals • Compare users’subjective assessment of search results from a discovery layer system (Ex Libris Primo) and Google Scholar • Evaluate the discovery layer system for usability issues Thursday, July 25, 13
  • 9. Method • 12 graduate students (mean age = 26.3, SD = 3.2) • Search tasks on Primo and Google Scholar • Given topic for first three tasks: classroom assessment techniques, food addition, natural language processing • Participant own keywords for fourth search task • Rate first ten results from 1 (highly irrelevant) to 7 (highly relevant) • Within-subject design, balanced order of search systems, randomized tasks • After search tasks: System Usability Scale (SUS) ratings, preference ratings, open questions Thursday, July 25, 13
  • 10. Relevancy Ratings Search 1: Article 9 and 10 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
  • 11. Relevancy Ratings Search 2: Article 10 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
  • 12. Relevancy Ratings Search 3: Article 3 significantly different 1" 2" 3" 4" 5" 6" 7" 8" 9" Ar-cle"1" Ra-ng" Ar-cle"2" Ra-ng" Ar-cle"3" Ra-ng" Ar-cle"4" Ra-ng" Ar-cle"5" Ra-ng" Ar-cle"6" Ra-ng" Ar-cle"7" Ra-ng" Ar-cle"8" Ra-ng" Ar-cle"9" Ra-ng" Ar-cle"10" Ra-ng" Google"Scholar" Primo" Thursday, July 25, 13
  • 13. Relevancy Ratings Search 4: Article 3 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
  • 14. Results • No significant difference of relevancy ratings between given and participant chosen searches • Usability ratings and preference of Google Scholar significantly higher than Primo • SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43) • Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38) • Usability correlated to preference ratings (r = 0.59, p < 0.01) Thursday, July 25, 13
  • 15. Usability Issues of Primo - Workflow Search Results List Identify Article Link Resolver Article Page Thursday, July 25, 13
  • 16. Usability Issues of Primo ! Thursday, July 25, 13
  • 17. Usability Issues - Consistency Thursday, July 25, 13
  • 18. Conclusion Primo’ssearch relevancy may be on par with Google Scholar, but usability issues have affected users’perceived usability and preference. User interface of discovery layer systems needs to address workflow and consistency issues. Thursday, July 25, 13
  • 19. What’s More • “Blind”test on relevancy ratings • Test large sample of search keywords • Server log analysis on keywords, facets, session length, etc. • Better (Google Scholar-like?) link resolver • UI improvement Thursday, July 25, 13
  • 20. Questions? zhan1022@purdue.edu Thursday, July 25, 13