• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
User-Centered Evaluation of a Discovery Layer System with Google Scholar
 

User-Centered Evaluation of a Discovery Layer System with Google Scholar

on

  • 254 views

Presentation at the Human-Computer Interaction International (HCII) 2013 Conference

Presentation at the Human-Computer Interaction International (HCII) 2013 Conference

Statistics

Views

Total Views
254
Views on SlideShare
254
Embed Views
0

Actions

Likes
1
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    User-Centered Evaluation of a Discovery Layer System with Google Scholar User-Centered Evaluation of a Discovery Layer System with Google Scholar Presentation Transcript

    • User-Centered Evaluation of a Discovery Layer System with Google Scholar Tao Zhang Digital User Experience Specialist Purdue University Libraries zhan1022@purdue.edu Thursday, July 25, 13
    • Library Website is still Important Breadth of scholarly resources Credible, highly relevant Guaranteed full text access Thursday, July 25, 13
    • User Expectation “The problem with the databases is that you can’t content yourself with just searching one place, like you can with Google. It is hard for the students to differentiate between the different databases.” Akselbo et al., The Hybrid Library: From the Users’ Perspective, 2006 Thursday, July 25, 13
    • Thursday, July 25, 13
    • Thursday, July 25, 13
    • Discovery Layer System - Ex Libris Primo® • Unified search interface for (almost) all library resources • Catalogs, databases, e-journals, digital collections, and repositories • Increased discoverability • Easier, faster, and more efficient search interface • Faceted browsing/filtering of search results (e.g., by format, date, subject, topic, etc.) • Record grouping (e.g., multiple editions or versions of the same material) • Consistent“Get it”option Thursday, July 25, 13
    • Thursday, July 25, 13
    • Study Goals • Compare users’subjective assessment of search results from a discovery layer system (Ex Libris Primo) and Google Scholar • Evaluate the discovery layer system for usability issues Thursday, July 25, 13
    • Method • 12 graduate students (mean age = 26.3, SD = 3.2) • Search tasks on Primo and Google Scholar • Given topic for first three tasks: classroom assessment techniques, food addition, natural language processing • Participant own keywords for fourth search task • Rate first ten results from 1 (highly irrelevant) to 7 (highly relevant) • Within-subject design, balanced order of search systems, randomized tasks • After search tasks: System Usability Scale (SUS) ratings, preference ratings, open questions Thursday, July 25, 13
    • Relevancy Ratings Search 1: Article 9 and 10 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
    • Relevancy Ratings Search 2: Article 10 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
    • Relevancy Ratings Search 3: Article 3 significantly different 1" 2" 3" 4" 5" 6" 7" 8" 9" Ar-cle"1" Ra-ng" Ar-cle"2" Ra-ng" Ar-cle"3" Ra-ng" Ar-cle"4" Ra-ng" Ar-cle"5" Ra-ng" Ar-cle"6" Ra-ng" Ar-cle"7" Ra-ng" Ar-cle"8" Ra-ng" Ar-cle"9" Ra-ng" Ar-cle"10" Ra-ng" Google"Scholar" Primo" Thursday, July 25, 13
    • Relevancy Ratings Search 4: Article 3 significantly different 1" 2" 3" 4" 5" 6" 7" 8" Ar,cle"1" Ra,ng" Ar,cle"2" Ra,ng" Ar,cle"3" Ra,ng" Ar,cle"4" Ra,ng" Ar,cle"5" Ra,ng" Ar,cle"6" Ra,ng" Ar,cle"7" Ra,ng" Ar,cle"8" Ra,ng" Ar,cle"9" Ra,ng" Ar,cle"10" Ra,ng" Google"Scholar" Primo" Thursday, July 25, 13
    • Results • No significant difference of relevancy ratings between given and participant chosen searches • Usability ratings and preference of Google Scholar significantly higher than Primo • SUS: 4.27 (SD = 0.29) vs. 3.59 (SD = 0.43) • Preference: 8.33 (SD = 0.65) vs. 7.08 (SD = 1.38) • Usability correlated to preference ratings (r = 0.59, p < 0.01) Thursday, July 25, 13
    • Usability Issues of Primo - Workflow Search Results List Identify Article Link Resolver Article Page Thursday, July 25, 13
    • Usability Issues of Primo ! Thursday, July 25, 13
    • Usability Issues - Consistency Thursday, July 25, 13
    • Conclusion Primo’ssearch relevancy may be on par with Google Scholar, but usability issues have affected users’perceived usability and preference. User interface of discovery layer systems needs to address workflow and consistency issues. Thursday, July 25, 13
    • What’s More • “Blind”test on relevancy ratings • Test large sample of search keywords • Server log analysis on keywords, facets, session length, etc. • Better (Google Scholar-like?) link resolver • UI improvement Thursday, July 25, 13
    • Questions? zhan1022@purdue.edu Thursday, July 25, 13