Presentation from a workshop given at ACRL 2011 conference, Data-Driven Library Web Design: Making Usability Testing Work with Collaborative Partnerships

1. Data-Driven Library Web Design:
Making Usability Testing Work with
Collaborative Partnerships
Allison Cowgill, Head of Reference
Amanda Dinscore, Public Services Librarian
Patrick Newell, AUL for Information Technology and Electronic Resources
Henry Madden Library
California State University, Fresno
All documents available at: http://www.slideshare.net/adinscore

2. Background
•
•
The Library Study at Fresno State —Ethnographic study conducted by two
anthropology professors
Study recommended that the Library’s web site should be should be redesigned
“Draw How You Feel When You Write a Paper.”

3. Drafting the Research Plan
Step 1: Create a Purpose Statement and
Objectives
Purpose Statement: Should encapsulate the
goals the team hopes to accomplish.
Example: “The purpose of this study is to determine
if users can easily accomplish tasks required for
research using the library’s web site.”
--------------------------------------------------------------------------Objectives: Use to develop the user tasks.
Should reflect actual user needs.
Example: “Determine the number of study
participants who are able to search for and locate a
book using the library’s web site.”
Activity:
Create a draft
purpose
statement and
at least 3
objectives.

4. Drafting the Research Plan
Step 2: Form the Team
Identify librarians and library staff who are
sincerely interested in participating and
support change.
Find collaborators outside of the library
from academic departments such as
Anthropology, Business, Computer Science,
or Education.
Everyone should be fully aware of the time
and effort required.
Activity:
Brainstorm
potential
collaborators
from both within
and outside your
library.

5. Drafting the Research Plan
Step 3: Identify User Tasks & Develop
Questions
Consult with others (public service librarians, web team,…)
• What is easy/difficult for users to do on our web site?
• What do you spend time helping users do on our web site?
Create a list of tasks users are expected to perform and
relate the tasks to the study objectives
Task: An
Example tasks from our study:
activity that
• Find a book title in our library
fulfills an
• Find a book title through our patron-initiated borrowing system
information need
• Find a newspaper article
• Find an article from a scholarly journal
Keep in mind the type of data you will collect & use

6. Drafting the Research Plan
Step 3: Identify User Tasks & Develop
Questions
Types of Data:
Independent Variables: Variables you manipulate. Choose these based on your
research questions.
Dependent Variables (a.k.a. Outcome/Response Variables): Something
you measure as the result of (based on the response to) the independent variables.
Quantitative Data: can be counted or expressed numerically
Qualitative Data: nonnumeric information such as conversation, text, audio, or
video.
“All qualitative data can be coded
quantitatively.”
http://www.socialresearchmethods.net/kb/qualdeb.php

7. Drafting the Research Plan
Step 3: Identify User Tasks & Develop
Questions
Data Type
Common Metrics
Statistical Procedures
Nominal (categories)
Task success (binary), errors
(binary)
Frequencies, crosstabs, Chisquare
Ordinal (ranks)
Severity ratings, rankings
(designs)
Frequencies, crosstabs, chisquare, Wilcoxon rank sums,
Spearman rank correlation
Interval
Likert scale data, SUS scores
All descriptive statistics, ttests, ANOVAs, correlation,
regression analysis
Ratio
Completion time, time (visual All descriptive statistics, tattention), average task
tests, ANOVAs, correlation,
success (aggregated)
regression analysis

8. Drafting the Research Plan
Step 3: Identify User Tasks & Develop
Questions
Examples:
Using the library web site, find one journal article
on swine flu.
Show me where on the web site you can find help
using the library.
The library has a page with resources organized
by subject. Show me how to find the page with
history resources.
Activity:
Brainstorm at
least 3 tasks
based on the
objectives you
created.
What data will
you use to
measure the
tasks?

9. Drafting the Research Plan
Step 4: Determine the Study Population
Who?
How
many?

10. Drafting the Research Plan
Step 4: Determine the Study Population
The Size of Your Population or Sub-Group
Why Sample?
• To say something about a population
• A statistically valid sample size allows you to generalize to a population from a sample
Confidence Level
• Tells you how sure you can be
• Represents how often the true percentage of the population who would pick an answer lies within
the confidence interval
Confidence Interval
• a.k.a. “margin of error”
• A range that estimates the true population for a statistic

11. Drafting the Research Plan
Step 4: Determine the Study Population
Technique
Advantages
Disadvantages
Random sampling
•Theoretically most accurate.
•Influenced only by chance.
Sometimes a list of the entire
population is unavailable or
practical considerations or
prevent random sampling.
Systematic sampling
•Similar to random sampling.
•Often easier than random
sampling.
The system can sometimes be
biased.
Quota sampling
•Can be used when random
sampling is impossible.
•Quick to do.
There may still be biases not
controlled by the quota system.
Stratified sampling
•Ensures large enough sample to
subdivide on important variables.
•Needed when population is too
large to list.
•Can be combined with other
techniques.
Can be biased if strata are given
false weights, unless weighting
procedure is used for overall
analysis.

12. Drafting the Research Plan
Step 4: Determine the Study Population
Your
Recruitment
Strategy
Consider:
•Advertising Needs
•Recruitment Location(s)
•Incentives

13. Drafting the Research Plan
Step 5: Room/Technology/Data Capture
Considerations
Intake/subject data gathering location
Testing location
Equipment/staff to record the data
•
•
Hardware
Software
•
Who configures/operates/troubleshoots?
Privacy/data security considerations
•
•
Privacy and personal consent
Data back up and security

14. Drafting the Research Plan
Step 5: Room/Technology/Data Capture
Considerations
Back up procedures
Equipment/staff to code the data
Equipment/staff to analyze the data
Activity:
Make a list of
the resources
available at
your own
library.
What might you
need?

15. Drafting the Research Plan
Step 6: Develop Scripts/Instructions
& Train Moderators
Creating a script and instructions for
moderators:
• Helps them to clearly explain study procedures to
subjects
• Ensures uniformity throughout the process
http://www.flickr.com/photos/lwr/4124641930/sizes/
l/in/faves-61999692@N00/
Take advantage of collaborating
teaching faculty’s expertise by
enlisting their help to train student
moderators.
Links: Moderator Instructions &
Introduction Script

16. Drafting the Research Plan
Step 7: Pre-test & Refine
Pre-test using moderators as subjects
and have moderators run each other
through the protocols/tasks.
Identify any unclear or skewed
questions and revise accordingly
Critical to remember that we are
focusing on web site usability, not
student ability or experience.

17. Drafting the Research Plan
Step 8:Test
Schedule rooms
Schedule helpers (moderators, recruiters,
supervisors, etc.)
Verify all parts of the web site are working
(surprise!)
Assure pre-tests and consent forms are present
Assure incentives are present
Back up data nightly (multiple times, on different
media, if possible)

18. Drafting the Research Plan
Step 9: Analyze the Data
Prepare forms/spreadsheets for data
processing
Code the qualitative data from
video/audio
• Develop codes beforehand
• Review data/develop codes/apply codes
Calculate useful statistics from the data
• Time on task (mean)
• Completion rate (percentages)
Activity:
Practice
Coding

19. Drafting the Research Plan
Step 10: Communicate the Data
Meet with library colleagues, showing them video excerpts and
sharing preliminary findings.
Develop clearly understandable graphs and other visuals that
show how students navigate the web site and the difficulties
they encounter.
Analyze and communicate qualitative data to stakeholders and
address issues with an eye towards internal sensitivities.

20. Drafting the Research Plan
Step 10: Communicate the Data
• Add graphs here

21. Drafting the Research Plan
Step 10: Communicate the Data

22. Drafting the Research Plan
Step 10: Communicate the Data

23. Drafting the Research Plan
Step 10: Communicate the Data

24. Drafting the Research Plan
Step 10: Communicate the Data

25. Drafting the Research Plan
Step 10: Communicate the Data

26. Drafting the Research Plan
Step 11: Make Revisions to the Site
Meet with web team to discuss
findings
Old site:
Identify design elements that create
information-seeking difficulties
Development site:
Review other university web sites
New site:
Determine how problematic
elements should be changed and
redesign accordingly
http://www.csufresno.edu/library/arch
http://labs.lib.csufresno.edu/
http://www.csufresno.edu/library/

27. Conclusion
Don’t be discouraged by the time or
effort this will take…the results are
worth it.
Pay attention to the internal political
situation with your library web site
development.
Use the collected data to overcome
resistance to change.
Subsequent testing allows to validate
changes and to identify areas for
ongoing improvement.

28. Questions?
Comments?