ACRL 2011 Data-Driven Library Web Design

985 views
956 views

Published on

Presentation from a workshop given at ACRL 2011 conference, Data-Driven Library Web Design: Making Usability Testing Work with Collaborative Partnerships

Published in: Education, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
985
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

ACRL 2011 Data-Driven Library Web Design

  1. 1. Data-Driven Library Web Design: Making Usability Testing Work with Collaborative Partnerships Allison Cowgill, Head of Reference Amanda Dinscore, Public Services Librarian Patrick Newell, AUL for Information Technology and Electronic Resources Henry Madden Library California State University, Fresno All documents available at: http://www.slideshare.net/adinscore
  2. 2. Background • • The Library Study at Fresno State —Ethnographic study conducted by two anthropology professors Study recommended that the Library’s web site should be should be redesigned “Draw How You Feel When You Write a Paper.”
  3. 3. Drafting the Research Plan Step 1: Create a Purpose Statement and Objectives Purpose Statement: Should encapsulate the goals the team hopes to accomplish. Example: “The purpose of this study is to determine if users can easily accomplish tasks required for research using the library’s web site.” --------------------------------------------------------------------------Objectives: Use to develop the user tasks. Should reflect actual user needs. Example: “Determine the number of study participants who are able to search for and locate a book using the library’s web site.” Activity: Create a draft purpose statement and at least 3 objectives.
  4. 4. Drafting the Research Plan Step 2: Form the Team Identify librarians and library staff who are sincerely interested in participating and support change. Find collaborators outside of the library from academic departments such as Anthropology, Business, Computer Science, or Education. Everyone should be fully aware of the time and effort required. Activity: Brainstorm potential collaborators from both within and outside your library.
  5. 5. Drafting the Research Plan Step 3: Identify User Tasks & Develop Questions Consult with others (public service librarians, web team,…) • What is easy/difficult for users to do on our web site? • What do you spend time helping users do on our web site? Create a list of tasks users are expected to perform and relate the tasks to the study objectives Task: An Example tasks from our study: activity that • Find a book title in our library fulfills an • Find a book title through our patron-initiated borrowing system information need • Find a newspaper article • Find an article from a scholarly journal Keep in mind the type of data you will collect & use
  6. 6. Drafting the Research Plan Step 3: Identify User Tasks & Develop Questions Types of Data: Independent Variables: Variables you manipulate. Choose these based on your research questions. Dependent Variables (a.k.a. Outcome/Response Variables): Something you measure as the result of (based on the response to) the independent variables. Quantitative Data: can be counted or expressed numerically Qualitative Data: nonnumeric information such as conversation, text, audio, or video. “All qualitative data can be coded quantitatively.” http://www.socialresearchmethods.net/kb/qualdeb.php
  7. 7. Drafting the Research Plan Step 3: Identify User Tasks & Develop Questions Data Type Common Metrics Statistical Procedures Nominal (categories) Task success (binary), errors (binary) Frequencies, crosstabs, Chisquare Ordinal (ranks) Severity ratings, rankings (designs) Frequencies, crosstabs, chisquare, Wilcoxon rank sums, Spearman rank correlation Interval Likert scale data, SUS scores All descriptive statistics, ttests, ANOVAs, correlation, regression analysis Ratio Completion time, time (visual All descriptive statistics, tattention), average task tests, ANOVAs, correlation, success (aggregated) regression analysis
  8. 8. Drafting the Research Plan Step 3: Identify User Tasks & Develop Questions Examples: Using the library web site, find one journal article on swine flu. Show me where on the web site you can find help using the library. The library has a page with resources organized by subject. Show me how to find the page with history resources. Activity: Brainstorm at least 3 tasks based on the objectives you created. What data will you use to measure the tasks?
  9. 9. Drafting the Research Plan Step 4: Determine the Study Population Who? How many?
  10. 10. Drafting the Research Plan Step 4: Determine the Study Population The Size of Your Population or Sub-Group Why Sample? • To say something about a population • A statistically valid sample size allows you to generalize to a population from a sample Confidence Level • Tells you how sure you can be • Represents how often the true percentage of the population who would pick an answer lies within the confidence interval Confidence Interval • a.k.a. “margin of error” • A range that estimates the true population for a statistic
  11. 11. Drafting the Research Plan Step 4: Determine the Study Population Technique Advantages Disadvantages Random sampling •Theoretically most accurate. •Influenced only by chance. Sometimes a list of the entire population is unavailable or practical considerations or prevent random sampling. Systematic sampling •Similar to random sampling. •Often easier than random sampling. The system can sometimes be biased. Quota sampling •Can be used when random sampling is impossible. •Quick to do. There may still be biases not controlled by the quota system. Stratified sampling •Ensures large enough sample to subdivide on important variables. •Needed when population is too large to list. •Can be combined with other techniques. Can be biased if strata are given false weights, unless weighting procedure is used for overall analysis.
  12. 12. Drafting the Research Plan Step 4: Determine the Study Population Your Recruitment Strategy Consider: •Advertising Needs •Recruitment Location(s) •Incentives
  13. 13. Drafting the Research Plan Step 5: Room/Technology/Data Capture Considerations Intake/subject data gathering location Testing location Equipment/staff to record the data • • Hardware Software • Who configures/operates/troubleshoots? Privacy/data security considerations • • Privacy and personal consent Data back up and security
  14. 14. Drafting the Research Plan Step 5: Room/Technology/Data Capture Considerations Back up procedures Equipment/staff to code the data Equipment/staff to analyze the data Activity: Make a list of the resources available at your own library. What might you need?
  15. 15. Drafting the Research Plan Step 6: Develop Scripts/Instructions & Train Moderators Creating a script and instructions for moderators: • Helps them to clearly explain study procedures to subjects • Ensures uniformity throughout the process http://www.flickr.com/photos/lwr/4124641930/sizes/ l/in/faves-61999692@N00/ Take advantage of collaborating teaching faculty’s expertise by enlisting their help to train student moderators. Links: Moderator Instructions & Introduction Script
  16. 16. Drafting the Research Plan Step 7: Pre-test & Refine Pre-test using moderators as subjects and have moderators run each other through the protocols/tasks. Identify any unclear or skewed questions and revise accordingly Critical to remember that we are focusing on web site usability, not student ability or experience.
  17. 17. Drafting the Research Plan Step 8:Test Schedule rooms Schedule helpers (moderators, recruiters, supervisors, etc.) Verify all parts of the web site are working (surprise!) Assure pre-tests and consent forms are present Assure incentives are present Back up data nightly (multiple times, on different media, if possible)
  18. 18. Drafting the Research Plan Step 9: Analyze the Data Prepare forms/spreadsheets for data processing Code the qualitative data from video/audio • Develop codes beforehand • Review data/develop codes/apply codes Calculate useful statistics from the data • Time on task (mean) • Completion rate (percentages) Activity: Practice Coding
  19. 19. Drafting the Research Plan Step 10: Communicate the Data Meet with library colleagues, showing them video excerpts and sharing preliminary findings. Develop clearly understandable graphs and other visuals that show how students navigate the web site and the difficulties they encounter. Analyze and communicate qualitative data to stakeholders and address issues with an eye towards internal sensitivities.
  20. 20. Drafting the Research Plan Step 10: Communicate the Data • Add graphs here
  21. 21. Drafting the Research Plan Step 10: Communicate the Data
  22. 22. Drafting the Research Plan Step 10: Communicate the Data
  23. 23. Drafting the Research Plan Step 10: Communicate the Data
  24. 24. Drafting the Research Plan Step 10: Communicate the Data
  25. 25. Drafting the Research Plan Step 10: Communicate the Data
  26. 26. Drafting the Research Plan Step 11: Make Revisions to the Site Meet with web team to discuss findings Old site: Identify design elements that create information-seeking difficulties Development site: Review other university web sites New site: Determine how problematic elements should be changed and redesign accordingly http://www.csufresno.edu/library/arch http://labs.lib.csufresno.edu/ http://www.csufresno.edu/library/
  27. 27. Conclusion Don’t be discouraged by the time or effort this will take…the results are worth it. Pay attention to the internal political situation with your library web site development. Use the collected data to overcome resistance to change. Subsequent testing allows to validate changes and to identify areas for ongoing improvement.
  28. 28. Questions? Comments?

×