Your SlideShare is downloading. ×
“Usability Testing of a Library Web Site: Librarians and Anthropologists Working Together”
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

“Usability Testing of a Library Web Site: Librarians and Anthropologists Working Together”


Published on

Ethnography, Libraries, Library, Web_Design, Usability

Ethnography, Libraries, Library, Web_Design, Usability

Published in: Education
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Allison: Rochester Study…10 minutes Amanda: Fresno State Study… 10 minutes Amanda: Usability Study…20 minutes Allison: Talking Points (develop and lead discussion)…30 minutes Allison: Conclusion…5minutes
  • Focus on design anthropology in which ethnographic study serves to aid in the design of products and services. Sought to engage students in their natural contexts. Library was closed and only offering limited services for the first half of the Library Study at Fresno State. New building opened in February, 2009. Study included observations of student life. Examples of some of the methods used were: “Floating Reference”—in which students acted out the role of the student and librarian at the reference desk in the Free Speech Area“Visions of Student Scholarship”—asked students to photograph student scholarship wherever they found it“Day Mapping” – with campus maps, disposable cameras, and notebooks. Ethnographic interviews—includinghome interviews (picture of lives beyond campus)“Bootlegging Workshops”—students brainstormed ideas about library services and acted out various scenes “Student Theatre Workshops”—research team devised a plot around a library service situation and directed the actors through the first half of a skit“Interior Design Workshops” –allowed students to design their own library space“Web Design Workshops”—from scratch or using “patchwork prototyping” (existing elements in new combinations)
  • Images: Set up a large poster board in the Free Speech Area in front of the library and asked students to draw how they feel when they write a paper. Mostly negative responses indicating a great deal of stress and frustration. Negative impressions of the library and librarians were demonstrated in several of the skits and other activities that students performed. Students expect to have difficulty when doing research and writing assignments and generally don’t see the library or librarians as being very helpful.
  • Some of the recommendations, particularly from the web design workshops, were useful for informing our redesign, including: separate search boxes for different types of holdings (doing this now) less institutional, more youthful and content rich feel move to a 2.0 model—general interest in personalization, class tie-ins, information about required texts for classes. Also requested online collaboration spaces like Google Docs--not practical. Integration with My Fresno State and FacebookGenerally—more “open-ended” observations and interactions. Did NOT seek statistical significance of findings. Observations about student life. Still needed to know…(read slide)
  • Decided to form a subcommittee of the main library web group to investigate what we could do to get more concrete data.Decided that a usability test was the best option. At first, only comprised of library faculty but we knew we wanted to build upon what was discovered in the Library Study. This led to further discussion and involvement with Professor Delcore, one of the main authors of the Library Study.Volunteered his students.
  • At first, we considered the use of focus groups, surveys, analysis of site usage logs—provided only limited information. Decided on usability testing. Previous decisions based on mostly anecdotal information, assumptions about user behavior and what students “should” know, and staff and faculty personal preferences. Needed data to support change
  • Purpose:Data rather than assumptions about user behavior. User group: Juniors (not experts yet, but may have some familiarity with the site) Pre-Survey/Consent form: consent to be videotaped Tasks: 10 tasks requiring basic research (finding a book, article, interlibrary loan, etc…); 30-45 minutes Script/Moderator Instructions: had to be very precise, offer very specific instruction to moderators. Demonstrate speak aloud method“Please let me know what you’re thinking.” How to move on, if necessary. “Thank you for trying. Some of these questions are difficult to answer. Would you like to move on to the next question.” What to say if they went off site Really stressed that it was not a test of them but of the site Balance between establishing rapport and consistencyTest design: what data do we want? (audio & video? Screen capture?), what software to use? how participants would be recruited, where to test, what software to use to record data, what incentives to provide… Pre-test: (10) Very important. Learned a lot and changed the wording and our approach to several things based on this. Recruit & Test: where to find the participants? Who would recruit? Where will they go and when? Analyze the results: working on this now. Think big at first, then streamline to what is practical
  • Subjects: Fresno State Juniors FTE: 4,081 95% Confidence LevelConfidence Interval of 15 Methods: On-site recruitment --$10 Starbucks gift cards—everyone recruited (but mostly Anthro. students). Did not take much time and was much easier than we thought it would be. Pre-Survey/Consent Form—Had to be over 18, a Junior, and had to give consent to be videotaped. Also asked about how often they used the library web site and how often, their major. Introduction Script—created a draft and then tweaked it throughout the process and through the pre-testing phase. This was read by one of 5 anthropology students who were recruited and trained as moderators. Usability Test Activity –consisted of 10 tasks that could be completed using the library’s web site. Consisted of such things as finding books, using interlibrary loan, searching for journal articles, etc… All were tasks that we expect most users to be able to accomplish with very little difficulty. This also took several meetings to draft and revise.
  • “Lostness” calculated using: Number of different pages visited Total number of pages visited Minimum (or optimum) number of pages that must be visited to complete the taskFrom “Measuring the User Experience” Tullis and Albert
  • Transcript

    • 1. Usability Testing of a Library Web Site: Librarians and Anthropologists Working Together
      Allison Cowgill & Amanda Dinscore
      Henry Madden Library
      California State University, Fresno
      California Association of Research Libraries
      April, 2010
      Sacramento, CA
    • 2. University of Rochester Study
      Ethnographic approach to studying campus libraries and their users
    • 3. The Library Study at Fresno State
      September, 2008 - May, 2009 Goal: “to discover information about student life that campus librarians could use to both increase library usage and improve student users’ experience of library services.”
    • 4. “Draw How You Feel When You Write a Paper.”
    • 5. Library Study Recommendations
      Provide different search boxes for different kinds of holdings.
      Attempt a redesign that aims for a less institutional, more youthful and content-rich feel.
      Move toward a Web 2.0 model.
      Still Needed to know…
      How do students use the site?
      What tasks can students accomplish?
      What assumptions are we making?
      What can we discover that will inform our re-design?
    • 6. Web Usability Workgroup (WUW)
      Decision to conduct a usability study and the formation of the Web Usability Workgroup (WUW)
      Involvement of Anthropology professor, Hank Delcore, and his students in ANTH 111B (Intermediate Ethnographic Fieldwork)
      “How people develop and use technological solutions (like the library's website) is one of the most complicated and important areas of modern life, so it's a natural anthropological attention.” Delcore, 2010
    • 7. Usability Testing
      Usability Testing demonstrates how users interact with the system.
      Participants are asked to complete a series of pre-defined tasks and to “think aloud” while completing them.
      See what real users actually do (rather than what they say they do)
      Provides quantitative data rather than opinion or conjecture
      Participants complete common tasks (finding a book, article, etc…)
    • 8. Process
      Develop a plan
      Determine the user group to be tested
      Pre-survey/Consent form
      Tasks to be performed (number & time)
      Script & Moderator Instructions—ensures consistency
      Test Design—how it will be conducted and how results are to be recorded
      Recruit & test
      Analyze the results
    • 9. Nuts & Bolts
      Subjects: Juniors
      Sample Size: 42 (plus 10 pre-tests)
      Recruitment: On-site
      People: 5 student moderators, 5 WUW members (Anthropology and library faculty)
      Facilities: 4 small rooms for testing; welcome area
      Equipment: 4 Macs
      Software: Screenium
      Test Documents:
      Pre-Survey/Consent Form
      Introduction Script
      Moderator Instructions
      Usability Test Activity
      Cost: $636 ($520 for Starbucks gift cards; Screenium software site licenses $29 per computer)
    • 10. Benefitted from pre-testing and multiple run-throughs with team members and moderators
      Challenges with script, wording of questions, issues that we hadn’t previously considered (e.g. using Google)
      Benefits of using student moderators (objective; offered valuable insight)
      Talk aloud method was sometimes challenging
      Testing took less time than we thought
      On-site recruitment worked better than expected
      The $10 Starbucks card incentive worked well
      Subjects took it seriously
      Software (Screenium) worked well
    • 11. Coding & Data Entry
      Subject Number, time stamp, task number (1-10)
      Link Label (words clicked) & URL
      Notes (including quotes from subjects)
      Codes for task completion
      Codes for positive & negative comments
      Codes for non-verbal behaviors such as smiling, frowning, laughing
      Data Analysis (in process)
    • 12. Discussion