Presented at LibTech Conference, March 15, 2018. Creating and maintaining research guides that students use and find helpful is an ongoing challenge. On deciding that our subject guides were due for a significant update, librarians at Kennesaw State University realized we needed to learn what our students wanted from these resources. We conducted a study focused on learning what information students expect to find on research guides, as well as how they would organize it. During this presentation I will share the study results and how that information was used to design a new subject guide template in LibGuides CMS. Additionally, I will explain beta-testing the new template prior to updating all subject guides for the fall 2018 semester. Attendees will take away recommendations for transferable design characteristics for your own guides, card sorting, and usability testing methodologies you can use to learn what your own community is looking for.
2. Background Information
• Kennesaw State
University
– 35,000 students
• Library System
– Two libraries
– 22 librarians managing 60
undergraduate subject
guides
3. Background Information
• Why update design?
– An accessibility audit had revealed significant
issues with library content
– Some thoughts of moving away from a template
– Anecdotally, librarians disliked the template
– And…
5. Why do a Study?
• Not much information out there on version 2
of LibGuides, especially the side-navigation
layout option
• Needed to find support for continuing to use a
template
• When using a template, making significant
changes is a big project – wanted to get it
right
6. ResearchTeam
• Current:
– Ashley Hoffman, eLearning Librarian
– Michael Luther, Assessment Librarian
– Adam Olsen,Web Services Librarian
– Jon Hansen, Director ofVirtual Services
• Former:
– Rita Spisak, Strategic Marketing Librarian
– Xueying Chen, Donations Coordinator
– Ashley Dupuy, former Director of Research and
Instructional Services
7. ProjectTimeline
Study design,
IRB approval
• May - June
Phase 1:
Cardsorting
• July -
November
Beta design, IRB
approval
• December -
January
Phase 2:
UsabilityTesting
• January -
April
9. Methodology
• Individual, open card sort
• 60-minute session
– 40 minute card-sorting
– 10 minute discussion
• Discussion focused on
design (navigation and
visual preferences) and
predicted use (mobile,
multiple subjects)
10. Cardsorting: Issues andTips
• Recruitment! (or rather, attendance)
• If doing individual sorting, add a think-aloud
protocol and record
• Consider time carefully – 40 cards in 40 minutes was
probably the upper limit
• Streamline data collection/entry – number cards,
use envelopes
• Take photos immediately when finished
11. Cardsorting Analysis
• Complicated, messy data!
• Basic Process:
1. Identify similar categories (e.g.
“Search Effectively,” “SearchTips”)
2. Look for correlation: which cards
were frequently sorted into that
category?Which categories did a
specific card get sorted into?
3. Decide on a structure you’ll use
12. Cardsorting Results
• Process-oriented structure
• About 1/3 of students put search
tools first
• 50% of students preferred tabbed
navigation
• Preferred consistent layout across
subjects
• Average of 6 categories
13. NewTemplate
Design:
version 1
Major Design
Choices
• More images,
less text
• Clean,
uncluttered
feel
• Process-
oriented
• Less jargon
• Larger fonts
• Adaptable to
mobile
https://libguides.kennesaw.edu/engresearchguidebeta
15. Methodology
• Iterative process with 4 rounds
of testing, updates to the
design between each round
• 15-minute task-based, think-
aloud protocol
• One-on-one
• Recording screen and audio
for each session using
BlackBoard Collaborate Ultra
16. Tasks
• Your English professor has assigned you a research paper
onVirginiaWoolf’s novel To the Lighthouse. Using this
site, how would you get started?
• Your professor is requiring you to use a scholarly article
for the paper.Where would you go on this site to get
started?
• Your professor suggests you use literary criticism as well.
If you had never heard of this before, could you use this
site to find out more about it?
• You will need to cite your sources, but you aren’t familiar
with MLA style. Using this site, what would you do?
17. UsabilityTesting:Tips
• Emphasize that the participant can’t do anything wrong
• Don’t answer participant questions during testing
– If they can’t complete a task, the fault is with the design
• Create tasks that mimic actual use, align to the
different areas you want to test, and don’t use leading
phrasing
• Consider varying what order you ask the tasks in
• 15 minutes really is enough if you plan right!
• Recruiting “live” has worked well!
18. Round 1 Results
• Links to citation guides needed to be moved off
the help page
• Search page was too cluttered
• Page names were not always clear
• In general, needed to reduce the amount of text
and remove some content
• Navigation was sometimes confusing, partly due
to page length
20. NewTemplate
Design:
Version 2
Major Changes:
• Deleted content
• Renamed pages
• Moved citation
info to “Find
Sources”
• Reorganized
information
from “Search”
and “Sources”
into “Find
Sources” and
“SearchTips”
• All boxes now
display without
outlines
• Updated
navigation
21. Round 2 Results
• Mostly positive feedback on the visuals, layout,
and navigation
• Citation guide links were easier to find, but some
students still had problems
• All tasks could be completed on the “Find
Sources” page – which students seemed to like
• Definite preference for the layout with the
tabbed box, instead of a long page
22. UsabilityTesting:WhereWe’re at Now
• Conducting Round 3 onWednesday
– Updating design to use tabbed boxes more,
continuing to reduce length
– Creating new tasks to collect feedback on specific
pages
• Planning for Round 4 – final round
– Creating subject-specific content for Political Science
– Creating the new CMS SystemTemplate (as opposed
to aTemplateGuide)
23. Transferable Design Characteristics
• Identify your target population – who are you
designing for?
• Process-oriented organization
• Minimize scrolling
• Keep it clean and simple – but not too simple!
• Avoid text-only sections
• Provide navigation options
24. Future Plans @ KSU
• Launching completed design across all
undergraduate subject guides in July
• Embedding guides into D2L and marketing to
faculty and students
• Collecting usage data, etc. to assess impact
• Peer training on how to apply the lessons
learned here to other KSU guides
25. Recommended Resources
• Krug, S. (2010). Rocket surgery made easy:The do-it-yourself guide to finding and fixing
usability problems. Berkeley, CA: New Riders.
• Krug, S. (2014). Don’t make me think, revisited: A common sense approach toWeb usability
(Third edition.). San Francisco, California: New Riders, Peachpit, Pearson Education.
• Kuniavsky, M. (2003). Observing the user experience: A practitioner’s guide to user
research. San Francisco, Calif.: Morgan Kaufmann.
• Sinkinson,C., Alexander, S., Hicks,A., & Kahn, M. (2012). Guiding Design: Exposing
Librarian and Student Mental Models of Research Guides. Portal : Libraries and the
Academy; Baltimore, 12(1), 63–84.
• Spencer, D. (2009). Card sorting: Designing usable categories. NewYork: Rosenfeld Media.
• Spencer, D. (n.d.).Card sorting: Resources. Retrieved December 13, 2017, from
http://rosenfeldmedia.com/books/card-sorting/
• Thorngate, S., & Hoden, A. (2017). Exploratory usability testing of user interface options
in LibGuides 2. College & Research Libraries, 78(6), 844–861.
https://doi.org/https://doi.org/10.5860/crl.78.6.844
Good morning! As she said, my name is Amy Gratz, and I’m the Learning & Teaching Services Librarian at Kennesaw State University. One of my main duties there is to coordinate the content and design of our online guides.
Before I jump into my main topic, I just wanted to give you a little bit of context since I imagine most of you aren’t familiar with Kennesaw. We’re currently the 3rd largest university in Georgia with around 35,000 students; we have 32 librarians working to provide them with services, and 22 of us manage at least one undergraduate subject guide, which is the type of guide the project I’m talking about is focused on.
I started at KSU a year ago January, and one of the first things I decided to tackle was updating these guides. Partly because of accessibility issues, partly because I was hearing from my new colleagues that they wanted to move away from having a template at all. KSU has a template because it facilitates maintenance, ensures consistency of design and quality, and it helps those librarians who don’t have enough time or subject expertise to create one on their own. Regardless, though, I heard from several of my new colleagues that they disliked our current template.
…personally, I agree with them – I think our current design looks dated. This is our current English subject guide, which uses a design created in LibGuides v1 back in 2011 – and similar to what Audrey and Heather mentioned in their presentation yesterday, not everything transferred well with the change! This design is a template that librarians don’t have a lot of say in, and although guide owners can make changes with approval, almost all information is identical from one guide to the next, except for different databases listed under “find articles” and different featured books. I have included the URL there if anyone would like to have a look
I decided that before we could start making changes, we needed to do a study for 3 main reasons. I wasn’t seeing much in the literature about version 2 of LibGuides, especially the side-navigation option, although there was plenty about guide design and website design in general. I also wanted to find support for continuing to use a template, since I think it makes the most since at KSU for our situation. Although if it turned out students really didn’t like or need a template, I was open to that. And finally, since we do have a template, making significant changes is a big project, and I wanted to get it right.
Since this is a big project, there’s no way I could do it alone! That said, you might not need such a large team, but I think you need at least 3 just to provide different perspectives.
Although they’re not here with me, I wanted to take a minute to acknowledge everyone else who has been involved in the project. The current Research Team has been involved the entire way through; those you see listed as former members had to leave the team for various reasons, but were very helpful during the first phase of the study.
So this study has had two different main phases, both of which I’ll be talking about in more detail, but to give you a sense of the overall timeline, we started planning this project almost a year ago, and decided to break it into two main phases, each of which went through the IRB approval process separately. We completed the first phase late fall, and we’re currently in the second phase. The goal is to launch our new guides this summer.
Okay, so – Phase 1. How many if you are familiar with cardsorting? It’s a testing method that lets you create an overall organizational structure. Participants had a deck of cards with information on them, and were asked to create groups of similar cards – basically to organize the information in ways that made sense to them. This gives us a theoretical structure for the guide.
We decided to do an individual open card sort with decks of 40 cards – you can see an example on the right. Each card described a LibGuides box – the title and a brief description of the content in it. Most of these were from our current template, with some additions from course guides. “Open sort” means that each student was able to create their own categories – as many or as few as they wanted, with whatever names they wanted. These were 60 minute sessions, 40 minutes of card-sorting, 10 minutes of discussion, and 10 minutes for consent/instructions/etc. The discussion at the end was focused on the guide design and predicted use.
Recruitment: Used a one-question survey (What would you expect to find on a “research guide”?) with the option of signing up for the focus groups. Many students who said they would attend failed to show up, enough that we had to turn group sorting into individual sorting and add an additional round of testing
Think-Aloud: We couldn’t, but a helpful substitute was an accidental addition, requesting participants share anything they thought we should know while we analyzed their data
Timing: In analysis, it did appear that many cards were sorted solely based on the title of the card, not the description, so more time might have helped
Streamline data collection/entry: we numbered the backs of the cards so we could quickly enter data into the spreadsheet, and had students write their group names on envelopes so we could put all the cards in that group into the envelope, in case we needed to check anything later
Photos – SO helpful for going back to the original! We took photos exactly as the students left everything, then follow-ups with the numbers of cards showing
The data you get is complicated and messy, and it will take time to work through – you can get a sense of that from the screenshot here. I found an Excel template available as a free download (link at end of presentation), which I strongly recommend using, as it was very helpful. Ended up also creating alternatives so we could sort the data about 100 different ways
Basic process has 3 steps:
Identify similar categories and give them a standardized name (this name can change later). 4 of us worked on this – after familiarizing ourselves with the data, we found it very effective to do the first step together as a group. You will need to make adjustments – it can be easy to misinterpret what they said
Look for correlations – cards in each cat, cats for each card.
Decide on a structure you’ll use
We ended up with the structure you see on the left, which I’ll explain more in a moment – I know it’s too small to read! The general themes from our 18 participants in this phase were (read points). …we decided to use 4 categories, although two of these had sub-categories, as well.
Start
Sub-groups: Learn about the Research Process, Find & Develop a Topic, Get the Basics. Basic library/research information and definitions and information on working with a research topic
Search
Sub-groups: Start Searching, Search Tips. Strategies for working with keywords and other search tips, database and other resource links.
Sources
Information about different types of sources and places to search for them. Intended to vary by discipline.
Help
Contact information for the library and writing center, FAQ and citation guides.
From that, we created this Beta guide design. We decided on side-navigation because there wasn’t a strong preference among participants, it mimics the KSU website, and requires our librarians to think less about mobile design. Other Major design choices were to use more images, less text; aim for a clean, uncluttered feel; process-oriented; less jargon.
At this point we were ready to move on to the next phase of our study:
Usability testing! This is our current phase – and I my opinion, the more fun part!
Before I get into this, I do just want to say that I don’t think you have to do the cardsorting – it was useful for us in giving us some overall guidance, but I think the usability testing is the more useful of the two. And you could use a lot of different ways to get ready for doing usability testing!
We’re using an iterative design process, where we tweak the design between each round of testing. Doing 1 round of testing a month, about 3 weeks between for analysis and updates.
Our testing is short – just 15 minutes with one participant and the moderator. Because only the moderator is in the room, we are recording these for analysis.
Could really use any software that lets you share a screen and audio, and record a session.
These are the tasks we used in Round 1 of testing – essentially, we give the students an example assignment to get started on, then ask them to start finding a scholarly article, learn about literary criticism, and find information about MLA citations.
We did change the first task in Round 2 to remove the specific novel, but otherwise these are the tasks we’ve tested so far. Our goal was to create tasks that mimic actual use of the guide.
Before I start getting into the results, I want to share some tips on conducting usability testing (read from slide)
15 minutes – most students have only needed 10
Recruitment – we’re in a high-traffic area – everyone walks past as they enter or exit the building. One of the research team is actively soliciting participation – I’ve found it effective to look for students who are alone, will make eye contact, and don’t look like they’re in a rush. You do have to be very accepting of “no,” though!
All of that said, I imagine you want to know what we’ve found!
11 participants in this round – more than enough to see consistent trends. Our goal here is NOT a representative sample!
Citations – only about 2 students found those on their own, and it wasn’t the first place they looked
Search page – students had a hard time finding what they wanted
Page names were too ambiguous (Sources)
Too much text – which we expected. I’m going to get this quote wrong, but Steve Krug, whose books we’ve used a lot in this process, suggests writing what you want to say, then cutting it in half, and in half again. And then a third time. We’ve got a few more halves to go!
Navigation – length of pages and those ambiguous names were both issues
So, before I show you what version 2 looked like, I wanted to refresh your memory of the first version of the guide. Which turned into…
…This – (slide info). Tried to avoid changing some content we hadn’t received feedback on.
Additionally, we created 2 versions of the “Find Sources” page due to the length, the second of which had the same content in a tabbed box. Participants were asked to view this version during the Q&A at the end, rather than mixing it into the actual testing. Decision was made partly based on the amount of time we had left before testing.
13 participants in this round. Usability seemed to improve, so we’re headed in the right direction!
So I don’t have another version to show you, because we’re still making changes. We’ll be doing Round 3 next week – our time-table slightly condensed so we can finish testing before students get too caught up in finals. For this round, we’re focusing on reducing scrolling by using tabbed boxes, editing content, etc. 2 librarians working on content – switched which pages we’re looking at this time so we can edit one-another’s work. We’re also updating our tasks to make sure we get feedback on the entire guide – slight flaw in our methods for round 2!
We’re also starting to prepare for Round 4, scheduled for April 11. We’ll be testing a different subject, since this is intended to work across disciplines. We’re also planning to use the actual CMS System Template option, instead of creating custom code for this guide, since that is what we’ll be doing going forward. Couldn’t start creating the System Template until we were fairly confident what design changes we were going to keep
Even though we’re not done, we have seen some trends I don’t think will change in terms of overall design.
I know, the first point is more about content than design. But it’s important to identify this at the beginning! Who you’re creating content for will dictate the design to an extent.
Process oriented structure – our students don’t necessarily think the way we do!
Minimizing scrolling – there are a lot of ways to do that; I really like the accordion structure Audrey and Heather showed in their presentation yesterday. We’re making choices that work based on what LibGuides gives us, but you can certainly be more creative!
Keep it clean and simple – our students expect to be able to find answers in seconds, and whatever we might think about it, we have to work with that expectation. But do make sure your labels actually have meaning!
Avoid text-only sections – the students aren’t reading them, anyway!
Provide navigation options – our students really used the menu a lot in the last round of testing, and the option to turn on the next/prev page links were handy, too.
Once we finish our project, we’ll be training colleagues and creating updated versions of all 60 guides, to be launched in July. Hopefully buy-in won’t be a big issue for us!
We’re also working on plans to embed these directly into D2L and market them more directly to faculty and students to improve visibility. We’ll be collecting some data to assess the impact – I can’t be more specific because I don’t know exactly what that will look like!
Finally, I’m planning to do some peer training with my colleagues on those transferrable design characteristics, because the rest of our LibGuides also need a change!
I really used all of these, but I want to particularly draw attention to Steve Krug’s books on DIY usability testing and web design, Donna Spencer’s work if you’re interested in the card sorting, and finally, the article by Thorngate and Hoden – their model really inspired a lot of what we’ve done.
