4. 4
Executive Summary
Our team analyzed the Roca, Inc. website using three different usability
tests: a card sort, a heuristic evaluation, and a think-aloud protocol. We
determined the way users would organize the main topics on the website, the
most severe issues on the website according to a set of guidelines, and the way
users navigate the current website to complete important tasks.
4 out of 7 participants created 5 groups.
4 out of 7 participants did not name their groups.
No two cards were sorted together in more than 6 out of 7 card sorts.
Next, each team member evaluated the website according to a set of
guidelines. Our combined heuristic evaluation determined that Roca had a
good layout, color scheme, and had a great amount of information. However,
we noted the following issues:
The website does not have a clearly labeled “About Us” section.
It is unclear when the content on the website was last updated.
The website does not have a site map.
The website does not have a liquid layout, meaning that the
design does not adjust to different screen sizes.
Some content is a bit difficult to find.
Visited links and unvisited links look exactly the same.
Some link labels are not meaningful to users.
The only way for users to alter their current search or search again is to
return to the original search page.
First, our card sort allowed the 7 participants to sort each card, or major topic
on the website, into categories that made sense to them.
5. 5
Lastly, our think-aloud protocol allowed us to observe 3 participants
completing important tasks on the website, including how they navigated the
site and what they were thinking as they did so. The tasks we asked them to
complete involved: first, finding Roca’s financial information; second, finding
Roca’s Intervention Model; and third, finding where to donate and volunteer.
We found that most participants
Initially had difficulty finding the financial information
Had difficulty finding and reading the Intervention Model
Easily found where to donate and volunteer
Based on the results of our usability tests, we recommend making the
following changes:
Use a liquid layout to make sure users can navigate the website
on any size device.
Change “Who We Are” or “What We Do” to “About Us” or
“About Roca” so that users can easily locate information about
the organization.
Reorganize content on the website so that it is easier for users to find,
especially the financial information and the intervention model.
Use meaningful link labels and page titles so the users know
where they are going when they click on something.
Redesign the Intervention Model so that it is easier for users to
read online.
Allow users to easily search again or refine their search.
We also created a two-minute video summarizing the results of our usability
testing. You can view the video at https://youtu.be/zaBnf1Sf9Q4.
6. 6
Introduction
Our team highly supports Roca, Inc.’s mission of disrupting the cycle of
incarceration and poverty by helping young people transform their lives.
Because we support Roca’s mission, we delved far into the website to find
both where it succeeds and where it can be improved. We found that the
content on Roca’s website was informative and useful for users. Through
conducting card sorts, expert evaluations, and think-aloud protocols, we were
able to find minor errors that can be easily corrected to create a more cohesive
and usable website, as well as major errors that when fixed will improve the
website as a whole.
We understand that the website is intended for the press, politicians, advocacy
groups, prospective employees and volunteers, potential business partners,
and supporters (especially donors). We also understand that the website is
designed to do the following:
Inform users about Roca’s program and model
Inform users about how Roca differs from other recidivism
reduction programs
Encourage users to donate money to Roca
Next, we will discuss our cart sort, heuristic evaluation, think-aloud protocol,
and recommendations to improve the website.
7. 7
Card Sort
Each team member conducted a card sort to help determine how the
audience, or the users, would organize the current content on the Roca
website. Card sorts are a fast, easy, and inexpensive way to allow users to
organize the main topics on a website the way that they expect to see them
organized.
For the card sort, we numbered and labeled each major section of the website,
usually using page titles such as “Financial Profile” and “Roca Works.” In
some instances, we added clarification to the card, for example changing
“History” to “Organization History.” We did not use the titles of the main
tabs—“Who We Are,” “What We Do,” “News & Events,” “Get Involved,”
and “Donate”—because we wanted to allow the participants in the card sort
to label each category they created themselves. Each participant was asked to
sort the cards into categories that made sense to them; the only other thing we
asked them to do was limit their categories to 9 or fewer.
In addition to asking the participants to complete a card sort, we also
gathered demographic information about them.
8. 8
were white females
were 25 years old or younger
lived in the Dallas-Fort Worth area
were employed full time
had democratic political leanings or preferred not to say
were religious
volunteered three or more times a month
had multiple social media accounts
The majority of the participants
Participants
56+
41-55
26-40
<25
9. 9
Results
In the left column, each card topic is numbered and grouped according
to the current organization of the website, which depicts how closely the
current organization matches the results of the card sort. In the top row, the
number of each card is listed. The rest of the matrix uses percentages and
colors to show how frequently two cards were sorted together. For example,
Terms of Use (card 1) and Privacy Policy (card 2) were sorted together 71%
of the time, or in 5 out of 7 card sorts.
Overall, 4 of our participants created 5 categories and 4 did not name
them. A portion of the correlation matrix is shown below:
10. 10
The majority of the participants created five categories
overall. We noticed the following trends:
Not one pair of cards was sorted together 100% of the time.
No two cards were sorted together more than 86% of the time,
or for 6 of 7 card sorts.
The most commonly correlated cards are as follows:
• Donate Online and Donate by Mail
• Donate Online and Join Newsletter Mailing List
• Videos and Media Coverage Archive
• Roca Works and Roca Young Mothers
• Facebook Feed and Twitter Feed
The next most commonly correlated cards are as follows:
• Terms of Use and Privacy Policy
• Terms of Use and BD Login
• Volunteers and Join Newsletter Mailing List
• Donate by Mail and Join Newsletter Mailing List
• Financial Profile and Annual Report
• Staff and Leadership
• Organization History and Mission Statement
• Stats in 2015 and Awards Received
The graphic on the following page shows the stronger
correlations from our card sorts. You can find a more
detailed correlation matrix in Appendix A.
12. 12
Heuristic Evaluation
Each team member individually performed an expert evaluation of the
website. The evaluation chart we referenced provided multiple sections of
relevant issues, of varying importance, for which we could rate the severity.
Collectively, compared which issues we all found to be the most severe.
Performing several individual evaluations is helpful in examining all aspects
of the website, even parts that often go unnoticed. Comparing our individual
evaluations helped us narrow down the most severe issues on the website.
13. 13
Homepage
We found 2 main issues on the homepage:
The site does not have a clearly labeled “About Us” section.
Instead, it has the tabs “Who We Are” and “What We Do,”
which are unclear: Neither tell the users directly that the tabs
lead to a page that is about Roca.
It is unclear when the content on the site was updated. Perhaps
the content was updated three days ago, or perhaps it was
updated three years ago. The users do not know; therefore, they
do not know if the content is still relevant.
We found 1 main issue with the navigation:
The website has no site map. Site maps are a list of links to every
page on a website, and they are structured in a way that reflects
the organization of the site. They are not essential, but many
users find them helpful when they want to go directly to the
page they need, instead of exploring or searching the site.
Results
Navigation
14. 14
We found 1 main issue with the design:
The website does not have a liquid layout, meaning that the
design does not adjust to different screen sizes. Since so many
people are using the internet on their phones and tablets now, it
is essential that a site has a liquid layout. Having a liquid layout
will make the site easier for mobile users to navigate.
We found 1 main issue with the content:
Some content, such as Roca’s financial information, is a bit
tricky to find. Information should be organized in a way that is
easy for users to understand.
Design
Content
15. 15
We found 2 main issues with the links:
Visited links and unvisited links look exactly the same. Visually
distinguishing the two will help users track where they have been
so they do not feel like they are wasting time on your site.
Some link labels are not meaningful to users. For example,
“Roca Works” could be confusing. A more descriptive title
would help users understand its meaning and purpose.
We found 1 main issue with the search function:
Currently, the only way for users to alter their current search or
search again is to return to the search page. Allowing users to
easily refine their search will help them find the information they
need as efficiently as possible.
Links
Search
16. 16
Think-Aloud Protocol
The purpose of a think-aloud protocol is to learn how users complete the
most common or important tasks on a site and to understand their thought
processes as they do so. Therefore, we asked three participants to perform
tasks that required them to navigate the website based on three different
scenarios. We also asked them to vocally express their thoughts and emotions
as they navigated the site.
During the protocol, one team member interacted directly with the
participant by explaining the scenarios, administering tasks, and reminding
them to think out loud. One or two other team members observed the
participant and their screen and recorded what the participant said and
how they interacted with the site. We also recorded the amount of time the
participants spent completing the scenarios and whether they used a laptop or
a desktop computer. After the protocols were completed, we combined and
analyzed the results.
17. 17
Results
The first participant completed the protocol on a desktop in 20 minutes. The
second participant also completed the protocol on a desktop, but in only 11
minutes, and the third participant completed the protocol on a laptop in 10
minutes.
20
05
10
15
Time Spent
(In Minutes)
Participant
1 2 3
Time Spent
18. 18
Scenario One
We wrote the following scenario to focus on how users would most
likely determine whether Roca is credible:
“You’re looking for a non-profit organization involving recidivism
to which you can donate your time or money. Your friend told you
to check any non-profit organization’s website for their financial
information to confirm their credibility. Find Roca Inc.’s financial
information.”
Below are the most significant points about this scenario:
All participants found the financial information, specifically the
“Financial Profile” page.
66% (2 of 3) of the participants clicked on “What We Do” first.
When they discovered that the financial information was not
in that section, they navigated back to the home page. Then
they clicked on “Who We Are” and easily found the “Financial
Profile” page in the left navigation bar.
What
We Do
Home
Page
Who
We Are
Financial
Profile
Most Common Navigation Path
19. 19
Scenario Two
We focused on Roca’s Intervention Model for the second scenario:
You have decided that Roca Inc. is trustworthy and would now like to
see their model for preventing recidivism. Find this model.
Below are the most significant points about this scenario:
66% (2 of 3) of the participants found the Intervention Model
using the most direct path, shown below.
33% (1 of 3) of the participants did not find the Intervention
Model but felt successful and satisfied with the information
found.
What
We Do
Roca
Works
Intervention
Model
Most Common Navigation Path
Most participants found the model difficult to find because the
link was too small or unrecognizable.
Most participants initially found the model hard to read.
20. 20
Scenario Three
The third scenario focused on donating and volunteering:
After reading more, you agree with Roca’s methods. You’d like to
donate $5 a month and volunteer 10 hours a month. Find the best
way to do this.
Below are the most significant points about this scenario:
All participants were able to navigate to the “Donate” page.
66% (2 of 3) of the participants like the option to donate
through PayPal because they felt it was secure.
Donate Donate
Today
Donate
Online
Most Common Navigation Path
33% (1 of 3) of the participants clicked on “Donate Today.”
33% (1 of 3) of the participants clicked on “Donate Online”
because the website did not appear to be secure—it did not
have a lock by the URL—so they were uncomfortable donating
directly through Roca.
33% (1 of 3) of the participants went to the “Donate” page
but did not click on anything else because they felt no need to
navigate further.
21. 21
Recommendations
Based on the results from our usability tests, we recommend making 6
changes to rocainc.org. We will explain our recommendations on the
following pages.
22. 22
Use a liquid layout to make sure users can navigate roca.com from any
size device.
Because so many people are using the internet on their phones
and tablets, it is essential that the website is easy to use even on
small devices.
This is what rocainc.org currently looks like on some screen sizes.
This is what rocainc.org would look like with a liquid layout.
Recommendation One
23. 23
Change “Who We Are” or “What We Do” to “About Us” or “About
Roca” so that users can easily locate information about Roca.
Users expect to see a section on any website labeled “About Us,”
and they are sometimes confused when a section is not labeled
as such. Additionally, most of the participants in the think-aloud
protocol had difficulty finding Roca’s financial information
because they were unsure whether it would be under “Who We
Are” or “What We Do.”
Recommendation Two
24. 24
Reorganize content on the website so that it is easier for users to find,
especially the financial information and the intervention model.
In our heuristic evaluation, we found that some content, such as
the financial information, was a bit difficult to find. Our think-
aloud protocol supported this finding regarding both the financial
information and the intervention model.
Recommendation Three
25. 25
Use meaningful link labels and page titles so the users know where they
are going when they click on something.
Some link labels and page titles were confusing, such as “Roca
Works.” As users, we did not know whether “Roca Works” meant
that Roca is effective or Roca completes tasks. More descriptive
titles and labels help users understand the meaning and purpose
of a link or page.
Recommendation Four
26. 26
Redesign the Intervention Model so that it is easier for users to read
online.
Most of our think-aloud participants found the Intervention
Model difficult to read and disliked the format of the model.
Redesigning the model so it is more pleasing to users and easier
for them to read will help encourage users to take the time to
learn about Roca’s Intervention Model.
Recommendation Five
27. 27
Allow users to easily search again or refine their search.
Allowing users to refine their search will help them find the
information they need as efficiently as possible.
Searches take users to a new page with no search function.
Recommendation Six
28. 28
Conclusion
These recommendations are based on the most important issues that we
identified after analyzing the results of our card sort, heuristic evaluation, and
think-aloud protocol. Following these recommendations will help users more
easily understand and navigate the website, even on mobile devices. To find
other ways to improve the website, you could conduct usability tests with
participants in the Boston area.
