3. Cornell Alumni Magazine Website Usability/UX Evaluation 3
Executive Summary
During the fall 2015 semester, Cornell Alumni Magazine (CAM) engaged Master of Professional
Studies (MPS) Project students to update and remodel the old website, in order to grow and
better serve its online audience. With a new website drafted, our client wished to conduct a
usability study to evaluate whether the website shares content in a more compelling way, takes
advantage of new web technologies, and updates to newer, cleaner design innovations. Their
audience ranges from alumni who graduated in the 1950s through the present, who read the
magazine for varying reasons. Consequently, they wished to evaluate their audience more clearly
and to ensure that the digital presentation of CAM’s content is effective.
Our team started out conducting a preliminary data analysis on the old website and new website
as well as comparing the old website design to other Ivy League alumni magazines. We found
that other university magazine websites varied in how clear and descriptive their design was
compared to CAM. Next, we conducted an online Qualtrics survey to understand how alumni felt
about the design of the new and new development website compared to the old website. Most of
our responses came from individuals aged 61-70, which makes sense considering this is an
alumni magazine targeted towards alumni subscribers. Finally we emailed several participants,
inviting them to participate in an in-depth usability study.
We designed several tasks asking participants to look through and comment on the design of all
three websites and search for an article. During the process, we analyzed what they said and did
and found that most participant did not like the pop-up advertisements on the home page, could
not find the article, and did not understand why there were news articles under the Noteworthy
section. Our team compared the result of our in-depth usability study with the online survey and
found that most people that preferred the old website switched to preferring the new website
once they interacted with it. We also worked out improved suggestions ranging from forcing
participants to click “Never see this again” on the pop-up advertisement and adding suggested
words to search to optimize the user interface design and user experience of the redesign.
Introduction
CAM is a non-profit publication that is independent of Cornell, which publishes six print
versions over a year with an online website update with print version contents each two months.
CAM was first created by alumni for alumni and published in 1899. The online website includes
articles in current issues and past issues. Most magazine subscribers are graduated from the
1950s to the present. Last year, a group of MPS students created a new website design and a new
website development design. The new design websites are similar, but they are in very different
4. Cornell Alumni Magazine Website Usability/UX Evaluation 4
design compared to the original website. Now CAM wishes to gather feedback from its online
website audience to improve its overall website design. Our client wants to know how to share
more compelling content, include new web technology, and update to newer, cleaner design
innovations.
Consequently, CAM wished to conduct a usability study to identify the audience based on all
three versions of the website as well as evaluate their impressions and preferences towards the
redesign. For the new redesigns, they wanted to know how the website conveys stories to their
audience, how to create a web space that Cornell alumni view as a resource to provide value to
their overall Cornell experience, and how to better utilize the existing magazine content for
readers.
Activities
Preliminary Analysis
We started off looking at the old and new website versions for CAM before performing an
analysis on the look and feel. We chose this initial approach because we wanted to understand
how and whether the new redesign shares content in a compelling way with newer, cleaner
design innovations. One team member opened up and compared the layout of the old and new
website versions, while other team members compared the design of CAM to seven other Ivy
League magazines. Originally, we thought we were only comparing the old and new website
versions. We did not learn about the new development version until after we finished conducting
our analysis and met with our client.
Online Survey
Afterwards, we created an online Qualtrics survey (Figure 1) to understand how and if alumni
use the CAM website and their opinions on the look and feel of the new and new development
website versions compared to the old website. Once we finished drafting our questionnaire, our
client helped send emails out with the survey to 1738 magazine subscribers. We chose to
interview a wide variety of alumni of different ages, genders, nationalities, and occupations to
identify our main user demographic and see how different factors impact their enjoyment of
reading the magazine online.
5. Cornell Alumni Magazine Website Usability/UX Evaluation 5
We started off thanking alumni for taking
their time to complete this short survey
about changes to Cornell Alumni
Magazine’s website before asking them
discovery questions about their age, what
media they prefer when reading magazines
like laptop or paper, and how often they
visited the CAM website in the previous six
months. Our client mentioned that most
CAM readers range in age from alumni who
graduated in the 1950s through the present,
and who read the magazine for various reasons. We wanted to understand and evaluate who our
main demographic or readers were and ensure that the digital presentation of Cornell Alumni
Magazine’s content is effective.
Then, we moved onto in-depth questions that
showed alumni homepage screenshots and
asked them to compare the look and feel of the
old website to the new website version (Figure
2) and select which they preferred and why. If
alumni selected the old website version, we
showed them other homepage screenshots for
the old version and new development version.
If alumni selected the new website version, we
showed them other homepage screenshots for
the new version and new development version.
Afterwards, we asked them about which
article layout they preferred between the
new and new development versions (Figure
3). We chose to limit their choices to see if
and how their opinions change when they
are presented with the new development
version. The new development version is
based on the new version and looks very
similar minus a few color changes, addition
of a top story section, and text first then
image placement in articles.
Figure 1: Online survey discovery questions
Figure 2: Comparison between old and new website
Figure 3: Comparison between articles in new and new
development website
6. Cornell Alumni Magazine Website Usability/UX Evaluation 6
We concluded with follow-up questions that asked alumni to provide their email address if they
wished to participate in an in-depth evaluation of the CAM website (Figure 4). If they selected
yes, then we asked them if they are in the Ithaca area in March. Lastly, we thanked alumni for
their time and asked if they had any additional comments. We chose to collect email addresses,
so we could contact interested alumni for an in-depth usability study that allowed them to look
and interact with the three website versions instead of just viewing screenshots.
Figure 4: Follow-up question for people who are interested in participate in in-depth usability study
Chart Data Analysis
Two weeks later, we closed our online survey and started data analysis. We chose to export our
data as a report and as an Excel spreadsheet to help us select different data points and create bar
graphs for our discovery and in-depth questions. We sorted each column alphabetically and
clustered our discovery questions based on participant media preferences and visiting frequency
together across age demographics ranging from Under 30, 31-40, 41-50, 51-60, 61-70, 71-80,
and Over 80. On the Y axis, we counted participants in each particular age demographic.
WAAD Data Analysis
Then, we created fifteen work activity affinity diagram (WAAD) to analyze different ideas and
factor out comments to our design questions for the old, new, and new development websites.
We chose this method of data analysis to help us cluster all comments into different groups for
layout, navigation, readability, and graphics and find a relationship amongst them.
Originally, we used light green post-its and pen for our
topic groupings like overall readability, organization, and
layout (Figure 5), but then switched to blue marker due to
readability issues (Figure 6). We also used purple Post-Its
for our individual comments and counted how many
participants wrote down a similar comment but then
switched to black marker.
Figure 5: WAAD analysis with Post-Its and pen
7. Cornell Alumni Magazine Website Usability/UX Evaluation 7
Once we realized we could not read comments on the
post-outs, so we switched to writing out the comments
on the whiteboard (Figure 6). Moreover, we decided
to add a headline with the name of the question and its
corresponding age group to distinguish between
different questions. Each team member wrote out and
clustered corresponding comments for one question
for one age demographic in green or red marker.
In-Depth Usability Study
Two weeks later, we conducted an in-depth usability study to explore how participants interacted
with each of the websites. To gather participants, we sent out 96 emails to all the participants left
their email address in the online survey. In the end, 7 participants volunteered for our study.
We conducted seven interviews over the span of one week. We interviewed two of them through
Skype video chat, another two through GoToMeeting’s video chat, one through an in-depth
study in an empty room in the MPS Lab (Figure 7), one through WebEx video chat, and one
audio interview through GoToMeeting. Each interview took approximately 30-40 minutes long.
When participants couldn’t meet with us in person, we asked them to share their screen and
recorded audio in our Skype session and video and audio in all GoToMeeting sessions.
Questionnaire Protocol
Our usability study consisted of four sections.
One of our team members started off greeting
the participant to help them feel relaxed and
familiar with us before explaining the purpose
of website, the reason behind usability study,
and obtaining informed consent. Another team
member carefully observing the participant’s
emotions and nonverbal communication cues,
offering encouragement or reassurance. Our
last team member sat a foot away and recorded
all pertinent information throughout the
process, while making every effort to remain
unbiased. Since he/she has the least amount of
interaction with the participant, the user found it easier to believe that the recorder was merely a
part of the setting.
Figure 6: WAAD analysis on white board
Figure 7: Test environment set-up
8. Cornell Alumni Magazine Website Usability/UX Evaluation 8
Afterwards, we moved onto preliminary questions that
asked the participant if they visited CAM before. Then,
we conducted our two tasks. Participants first explored
the old version, new version and new development
version websites and the Top Story section (Figure 8).
Then, participants found, skimmed, and talked about
the design and look of the Snow Birds article in all
three website versions. If participants could not find
the article, we still asked them to open up a random
article and discuss how they felt about the social media
icons, image placement, and layout. Lastly, we asked participants about their overall impression
of the redesign, how valuable the magazine is staying connected to Cornell, which website they
prefer most, and if there are any features participant want changed in the new redesign.
Data Analysis
One week later, we typed up notes about each usability study based on their responses to our
question and nonverbal communication through eye movements, mouse movements, and facial
expressions. Then, we read through all our interview notes for seven participants and compared
their comments in the in-depth usability study to responses in the online survey.
Outcomes
Preliminary Data Analysis
On the old website version, we found the navigation tab confusing (Figure 9). Users also had to
click on the Home link to return to the Home page instead of simply clicking the logo. Moreover,
we did not understand the meaning and location behind navigational links like home, tags, search
and feed.
Figure 9: Screenshot of top navigation bar on old website
On the new website version, we noticed that users get a notification to update their class dues
when they go to the homepage (Figure 10). However, this notification keeps popping up every
time the user refreshes the homepage or navigates back to the homepage, which is annoying for
people not interested in joining the club.
Figure 8: Screenshot of video chat survey
9. Cornell Alumni Magazine Website Usability/UX Evaluation 9
Figure 10: Pop-up on new website
We found that other Ivy League alumni magazines varied in how clear and comprehensive their
content and design was compared to Cornell Alumni Magazine. We liked the clear and clean
menu bar on the Harvard website (Figure 11) and breadcrumb navigation links on the Princeton
website that allows users to go back to a higher page above a subpage.
Figure 11: Screenshot of Harvard University Alumni Magazine navigation bar
On the other hand, we found the random advertisement located at the top of the Dartmouth
(Figure 12) and Yale websites distracting and did not like how the Yale website does not adjust
properly across different screen dimensions.
Figure 12: Screenshot of Dartmouth University Alumni Magazine advertisement
10. Cornell Alumni Magazine Website Usability/UX Evaluation 10
Online Survey
Out of 1738 emails sent out about our survey, 813 participants opened the email. Only 311
started the online survey (Figure 13). Out of those that started it, 258 participants completed the
online survey. On the other hand, we had 925 participants that did not open the survey link.
Most of our responses in our online survey came from individuals aged 61-70 who represented
28% of total participants.
Only 2% of individuals under 30 participated in our
survey, which makes sense considering this is an
alumni magazine targeted towards alumni
subscribers. Almost all our participants that left their
emails were aged 50 and over. This might have
skewed our data results and caused a higher
percentage of elderly people to like each version
more than others because of their greater
representation compared to younger age
demographics.
As Figure 14 to the right shows, out of 258 participants,
55% preferred the new website design because it
looked less busy with less visual clutter and 32% that
preferred the old website found it more graphically
pleasing and easier to read information. They found it
easier to read, more eye-catching, informative, and
traditional. A lot of elderly participants aged 61 to Over
80 Years Old (34 participants) found the old version
easier to read and find information.
As Figure 15 to the left shows, out of 134
participants that selected the new version in question
4, 58% of participants liked the new development
version more because they found the layout less
cluttered and liked the Top Story feature. More
middle-aged individuals preferred the new version
and new development version more than individuals
less than 30 years old or older individuals greater
than 71 years old.
Figure 13: Demographic of Online Survey
Figure 14: Old Website vs New Website
Figure 15: New vs New Development Website
11. Cornell Alumni Magazine Website Usability/UX Evaluation 11
As Figure 16 to the right shows, out of 132 participants
55% noticed and liked the Top Story section in the new
development version because they liked the larger image
that gets the reader’s attention. 18% found it too
distracting and cluttered the top header. 20% did not think
it mattered because they would rather choose their own top
story based on interests. 7% did not notice that section
because their brain processed the large image underneath
as more important.
As Figure 17
to the left shows, out of 250 participants 72% liked
the new website article layout and found the article
width wider, easier to read, and more eye catching.
18% liked the new development version because
they liked the text first so they didn’t need to wait
for the picture to load when they viewed the story.
Many people like the image at the top. Individuals
aged 61-70 most preferred the article layout in the
new version because they found the font larger and
easier to read to their eyes.
In-Depth Usability Survey
Out of seven total participants three preferred the old website, three preferred the new, and one
neither. Two participants found the four-column layout bad. Also, three participants found the
website navigation confusing and bottom section busy. Three participants found the Top Story
section useful for people that live far from campus since it is about news from Cornell.
Participants that liked the new version liked the fact that two thirds of the page was used. They
mentioned that they could still see the advertisements, which were presented well. They also
liked the easy navigation. Three participants found the popup annoying and wanted to remove it
since it is the first thing people saw for the website. Four participants thought the Feature Story
section was good to show all the past issues on the magazine.
Participants that liked the new development version found it great to read and much cleaner. One
participant pointed out that this version nicely blends the org and com versions and the
Noteworthy section was good because it provided information about noteworthy events and
features. Three participants liked the two column layout and found the sub-navigation bar useful.
Figure 17: Comparison between article layout
in New and New Development websites
Figure 16: Top Story in New Development
website
12. Cornell Alumni Magazine Website Usability/UX Evaluation 12
Two participants found this version the best in terms of image placement and layout, while three
participants preferred the Top Story and Most Read sections that draw your attention.
Participants all found it difficult to find the Snow Bird article online on all website versions when
they used the search bar. They tried inputting “SnowBird” into the search bar and saw a number
of articles referencing class notes that called their classmates “snowbirds” in the old version or
nothing in the newer websites. Four participants liked the text first before the image in the new
development website. Four participants found it easy to find the article on the homepage of the
new development website. Six participants liked and found it easy to find the social media icons
on the new development website, while four participants found it difficult to find them under the
“+” in the old website.
Design Improvements
Overview
One week later, our team came back together to offer suggestions for design improvements. We
compared comments across all three websites mainly based on our in-depth usability study
before ranking our improvements from most important to least important. Our suggestions are
very useful to optimize the design layout and the user experience of the CAM website. Some of
them are very detailed, while others are trivial but actually play important roles in the website
design. Without messy, irrelevant information on a website, it’s easier for users to find the
information they need. Adding features like “email to your friends”, “share on Facebook” and
“print this article” will also give users more convenient channels when they use the website.
Other new web technologies could also be applied into the redesign. During our in-depth
usability study, many participants mentioned that the search function does not work well.
Sometimes people can’t find the article simply because of not typing space between two words.
A lot of search engine optimization widget plugins could help optimize search. This is crucial
since many users who go to CAM website want to find specific articles, and the searching
functionality is one of the most important functions for Cornell Alumni Magazine.
Suggestions
1) Ensure users always click on “Never See Again” when they see the pop-up on the home
page by removing the “x” on the top right side and making sure users cannot click out of
it without actually pressing “Never See Again”
a) In our in-depth usability study, three out of seven participants found the popup
annoying because it still shows up even after participants open up an article.
2) Fix up search functionality and suggest alternative spellings
13. Cornell Alumni Magazine Website Usability/UX Evaluation 13
a) In our in-depth usability study, two out of seven participants mentioned that they
want to make the search engine function works better (like Google to return a
similar result even if searching words are incorrect`). All participants found it
difficult to find the Snow Bird article online.
3) Rename the Noteworthy section into something more meaningful for alumni interested in
connecting to campus and learning about research opportunities, especially since users do
not mainly use CAM to read up on daily news at Cornell University
a) In our in-depth usability study, one out of seven participants mentioned the
magazine should not include news in the Noteworthy section since the purpose of
the magazine is not for news.
4) Add an email option in article sharing to account for users that do not have social media
profiles on Facebook or Instagram
a) In our in-depth usability study, one participant mentioned that although she read
the print version most time, the CAM is the fastest way for her to share interesting
news to her friends (by Email), that is the way she feel more connected. She also
mentioned that, “I like the social media buttons, and I use Facebook a lot. But I
often forward articles through email to my friends. I think if there should be a
button for email that will be great.”
5) Add the class notes section back into the newer website versions.
a) In our online survey, one participant mentioned that, “I’m eager to see if any of
my friends are mentioned in the class notes.”
b) Similarly, in our in-depth usability study, another two participants felt the online
website should focus on information like the feature article and class report for
class year. These participants found the class notes section very valuable for them,
since they are interested in information from alumni they know about.
6) Make the images of Features on homepage automatically rotate instead of static and add
number on the corner of each image
a) In our in-depth usability study, one participant found the images under features
confusing because they are function like a static graph and he cannot see how
many images lie there. However, he felt that automatic rotation should not be
applied on mobile device version since the users can easily click on a wrong page.
7) Several participants mentioned that the length of the text line in each article may be too
long and they don’t know where to start when they start reading a new line.
a) Holst (2010) found out the optimal length for reading is 50 - 75 characters per
line. Within this range, readers will feel like they are actually reading a magazine.
b) Some participants in our in-depth user study mentioned that there should always
be a headline for each article to draw readers’ attention, but the length of the
headline should not be too long.
14. Cornell Alumni Magazine Website Usability/UX Evaluation 14
Conclusion
Our team helped CAM identify and understand who their main magazine subscribers are, i.e.
mostly elderly alumni aged sixty and above. Moreover, our online and in-depth usability studies
found that most users enjoyed the new website and new development website more than the old
website. Participants found both these versions less cluttered and more visually appealing
compared to the traditional print magazine layout in the old version. More importantly,
participants that said they preferred the old version in our online survey switched to preferring
the new version in our in-depth usability study once they interacted with the website.
Improvement Impacts
Many Cornell alumni do find this website useful for providing information about research
opportunities and ways of staying connected to Cornell. CAM can use the information we
collected to minimize distracting advertisements, especially since the majority of our users never
clicked “Never see this again” and always saw the pop-up on every page they loaded on the new
version and new development version. Our client can also fix up search functionality and offer
suggestions for alternative spellings, especially since all of our users in our in-depth study could
not find the Snow Bird article when they searched “Snow Bird” in the search box. Lastly, our
client can rename the Noteworthy section into something meaningful for alumni interested in
connecting to campus and learning about research opportunities, especially since most users do
not use CAM to read up on daily news at Cornell University.
Next Steps
As a follow up, we would help our client solve some technical problems inside the website that
influence the overall user experience, like the search functionality. In this project, we mainly
focused on the usability of the homepage and article web pages, which are most frequently used
by users. Moreover, as a result of our usability study, we found some users expect to see new
features inside the new website like class notes. Our next steps are to test other usability pages
and help our client build the class notes section inside the redesign.
References
Holst, C. (2010, November 01). Readability: The Optimal Line Length. Retrieved May 18, 2016,
from http://baymard.com/blog/line-length-readability
