eFolioMinnesota Text-Based Usability Test Findings and Analysis Report

eFolioMinnesota™
Text-Based Version:
Usability Test Findings and Analysis Report
SHAUN BIBO, KEVIN GLENZ,
RON HECK, SUE ANN RODRIQUEZ
Metropolitan State University
Dr. Victoria Sadler
Sponsored by:
TM
Prepared for:
PaulWasko and Avenet
April27, 2012
This work was completed by the following Metropolitan State University students:
Shaun Bibo, Kevin Glenz, Ron Heck, and Sue Ann Rodriquez under the guidance of
Associate Professor Victoria Sadler – April 27, 2012.

Executive Summary Page 1 of 92
eFolioMinnesota™
Text-Based Version:
Usability Test Findings and Analysis Report
Executive Summary
Background and Test Objective
The eFolioMN Text-based Usability Group
(eTUG) is a group of four students in the
M.S. in Technical Communication program
at Metropolitan State University. eTUG
conducted usability testing on the text-based version of the eFolioMN website
(www.efoliomn.com). The stated purpose of this usability project was “to recommend how
eFolioMN can improve the user experience for the type of user called Professional Profile:
Careers. That is, users who are interested in establishing a professional portfolio for finding
employment, promoting skills (e.g., self-employed; artists), or for periodic assessments
(e.g., employment review).”
The four members of the eTUG team include:
Shaun Bibo – test logger, technical support
Ron Heck – project manager, test moderator, and test logger
Kevin Glenz – test moderator and test logger
Sue Ann Rodriquez – test logger
The team conducted usability testing on the eFolioMN website during the 2012 spring
semester under the guidance of Dr. Victoria Sadler, Associate Professor of Technical
Communication. This project was completed in four phases:
1. Performing a heuristic evaluation of the website against a standardized set of
criteria.
2. Creating Personas of users of the website.

eFolioMinnesota™ Text-Based Version: Usability Test Findings and Analysis Report
3. Conducting usability testing with evaluators recruited from Minnesota State
Services for the Blind and Vision Loss Resources.
4. Analyzing the results and providing recommendations based on findings from
usability testing.
All phases of the project were completed between January 2012 and April 2012. Phase
three, usability testing, was conducted between March 28 and April 6, 2012.
Prior usability testing of the text-based version of eFolioMN was conducted three years
prior to this test. According to the client, the site has changed dramatically since that time,
and a new usability test was warranted. The eTUG team worked directly with project
sponsor Paul Wasko, Director for eStudent Services at Minnesota State Colleges and
Universities, to understand the goals and desired outcomes of usability testing. Wasko has
worked with eFolioMN since its inception in 2002.
Results
The most significant findings of the evaluation are listed below. The findings were sorted
into three categories:
 Content –content that can inhibit users due to it being inaccurate, misleading, and/or
missing.
 Terminology – words that can be confusing to users because they are not clear or defined.
 User Interface – software design issues that can interfere with users’ ability to navigate
and complete tasks efficiently and effectively.
The findings were also ranked on a three-point severity scale based on the how
significantly the issue impacted the user.
Severity Scale
 High – user cannot complete a task; intervention by the moderator is required; or user
expresses extreme irritation and frustration; issue will definitely affect other users.
 Medium – user can complete a task, but is frustrated; issue is highly likely to impact
other users.
 Low – user can complete a task, but expresses minor irritation or annoyance; issue may
impact other users.

ResultsTable
Usability Problem
Number
Affected Severity
1. Content:Directionsineach step of the To Dolist actually describe the
steps you wouldtake on the mouse-based version
2/5 High
2. User Interface: Signuppage 8: Captcha difficulties withaudio challenge 4/5 High
3. User Interface: Signuppage 9: Login link takes user to mouse-based
login page
5/5 High
4. User Interface: Text-BasedVersionnot mentioned on home page at all. 5/5 High
5. User Interface: “Graphic AboutEquals” login link 5/5 High
6. User Interface: Extremely difficulttoaccess text-based version 5/5 High
7. User Interface: Mostusersneeded to be prompted by moderator to
switch totext-based version to avoid logging into the mouse-based
version
4/5 High
8. Content:Helpinformationnot present on all pages 1/5 Medium
9. Terminology: Confusionaboutoptions (My Content, Add New Content,
Manage Sites)
5/5 Medium
10. Terminology: Confusionoverwhat“ToDo” means. Help on the To Do
page is not helpful
5/5 Medium
11. User Interface: Theassigned shortcut keys in the tooldo not workfor
JAWS users (JAWS already uses these keys and overrides the tool)
3/5 Medium
12. User Interface: Inconsistency inbottom menu bar inside the tool 2/5 Medium

Usability Problem
Number
Affected Severity
13. User Interface: Lackof aconfirmation message when successfully
uploading content or adding it to a page
2/5 Medium
14. User Interface: Desiretoadd newly created/uploaded content to a page
directly after clickingSave or even during the create/upload process
2/5 Medium
15. User Interface: Userinterfaceis not intuitive and requires user to
become familiar with the toolto use it effectively and efficiently
5/5 Medium
16. Content:“Readmore” link doesn’t lead to more content 1/5 Low
17. Terminology: “Revert” wordcauses confusionon some pages, including
New Basic Content page.
1/5 Low
18. Terminology: Step2of the sign up process: Terms of Agreement. Button
options are “I do not accept” and “Continue…”
4/5 Low
19. Terminology: Finalstepof sign up process button says “Save” instead of
“continue” or “complete”
1/5 Low
20. Terminology: Signup reads as Sig-nup during sign up process 1/5 Low
21. User Interface: Beginningof sign up process: “ClickHere” link followed
by text ‘to sign up’
2/5 Low
22. User Interface: Formfields:Difficulty knowingrequired fields when
listed in Bold (JAWS does not indicate text is bold)
2/5 Low
23. User Interface: Passwordfielddoes not immediately appear on login
page
3/5 Low

Usability Problem
Number
Affected Severity
24. Content:Inconsistency inwhether sites have Headings or not 4/5 Low
25. User Interface: Someusers thought clicking “Sign Up” on Home Page
automatically started the sign up process
3/5 Low
26. User Interface: Toomany steps in sign up process 1/5 Low
Report Overview
This report includes the following major sections:
 Introduction and Project Background – a brief overview of the project including the test
objective and steps taken throughout the process.
 Methodology – an overview of the test process and equipment, scenarios used in testing,
and usability testing evaluators.
 Findings – information obtained from usability testing, including what the evaluators
and like and disliked, and analyses of the post-task questions and the post-test
questionnaire.
 Recommendations – presented in three categories – Content, Terminology, and User
Interface – and sorted by high, medium, and low severity.
 Conclusion – a summary of recommendations and explanation of the need for further
usability testing.
Several appendices present information that informed our testing procedures and the raw
data from usability testing. These include:
 Appendix A: Persona – a composite of professional users of the eFolioMN website.
 Appendix B: PersonasMemo – describes the process used to create personas.
 Appendix C: Heuristic Evaluation – an expert review of the website based on Jakob
Nielsen’s 10 Heuristics for User Interface Design.

 Appendix D: Jakob Nielsen’s10 Heuristics for UserInterfaceDesign – ten general
principles for effective user interface design used for the Heuristic Evaluation.
 Appendix E: Usability Test Plan – documents the process forconducting usability testing
on the E4C website. A copy of the Usability Test Plan was provided to the client on
March 29, 2012.
 Appendix F: Recruitment Email – used to recruit evaluators.
 Appendix G: Screening Questionnaire –used to recruit evaluators. Also included are the
answers from our five evaluators.
 Appendix H: Pre-Test Questionnaire – given to evaluators on arrival and prior to testing.
Also included are the results from our five evaluators.
 Appendix I: ModeratorScript – read to each evaluator at the beginning of each session.
 Appendix J: Post-Test Questionnaire – given to evaluators after the testing session was
completed.
 Appendix K: Audio RecordingConsent Forms – Release forms signed by all evaluators
allowing eTUG to recording their voice and share it with eFolioMN representatives.

Table of Contents
Executive Summary..............................................................................................................................................................1
Background and Test Objective ...............................................................................................................................1
Results......................................................................................................................................................................................2
Report Structure................................................................................................................................................................5
List of Figures...........................................................................................................................................................................9
Methodology...........................................................................................................................................................................10
Testing Overview ...........................................................................................................................................................10
User Persona.....................................................................................................................................................................11
Usability Testing Evaluators ...................................................................................................................................11
Description of Scenarios............................................................................................................................................12
Findings.....................................................................................................................................................................................14
What the Evaluators Liked.......................................................................................................................................15
What the Evaluators Disliked.................................................................................................................................16
Post-Task Questionnaire Results .........................................................................................................................27
Post-Test Questionnaire Analysis........................................................................................................................29
Recommendations..............................................................................................................................................................32
Content.................................................................................................................................................................................32
Terminology......................................................................................................................................................................33
User Interface...................................................................................................................................................................33
Conclusion................................................................................................................................................................................35
Appendix A: User Persona.............................................................................................................................................36

Appendix B: Personas Memo........................................................................................................................................38
Appendix C: Heuristic Evaluation..............................................................................................................................41
Appendix D: Jakob Nielsen’s 10 Heuristics for User Interface Design................................................63
Appendix E: Usability Test Plan..................................................................................................................................65
Appendix F: Recruitment Email .................................................................................................................................78
Appendix G: Screening Questionnaire....................................................................................................................79
Appendix H: Pre-Test Questionnaire ......................................................................................................................82
Appendix I: Moderator Script......................................................................................................................................83
Appendix J: Post-Test Questionnaire ......................................................................................................................85
Appendix K: Audio Recording Consent Forms..................................................................................................86

List of Figures Page 9 of 92
List of Figures
Figure 1: Captcha Box.......................................................................................................................................................17
Figure 2: Switch to text-based version link.........................................................................................................18
Figure 3: EfolioMN Home Page...................................................................................................................................18
Figure 4: Login Version 2 Link....................................................................................................................................19
Figure 5: Main Menu Shortcut Keys.........................................................................................................................19
Figure 6: Pre-Upload and Post-Upload Pages....................................................................................................20
Figure 7 : Bottom Menu Bars.......................................................................................................................................21
Figure 8: Getting Started Screen................................................................................................................................22
Figure 9: Login Bar Missing Password Field ......................................................................................................23
Figure 10: To Do List.........................................................................................................................................................24
Figure 11: To Do List Step 2: Create a welcome message..........................................................................26
Figure 12: Post-Test Questionnaire Responses................................................................................................31

Methodology Page 10 of 92
Methodology
The methodology section provides an overview of the testing process and environment
used to complete usability testing, a description of the recruiting process and evaluators,
and a description of the scenarios used in testing.
Testing Overview
The eTUG team conducted formal usability testing with a total of five evaluators at three
locations: a classroom on the Metropolitan State University campus in St. Paul, MN; a
computer lab at the offices of the Minnesota State Services for the Blind in St. Paul, MN; and
a computer lab at Vision Loss Resources in Minneapolis, MN.
All of the test sessions lasted between 80 and 90 minutes and included:
 A pre-test questionnaire delivered in person
 A moderator-led description of the testing process
 Six scenarios for evaluators to work through
 A post-test questionnaire delivered in person
 The primary equipment used for testing included:
 Evaluator computer
 Digital audio recording device
Each session was led by a moderator who delivered a description of each scenario and
asked several post-task questions. The moderator sat with evaluators through the entire
session and intervened when necessary to provide assistance.
For a full description of the test environment and equipment used, refer to the Usability
Test Plan located in the Appendix E.
The team evaluated the website through a heuristic evaluation (expert review) and user
evaluation. The heuristic evaluation informed our approach to the testing and influenced
our testing plan and decisions. Due to time constraints, we were not able conduct user
evaluations for all tasks and issues uncovered in the heuristic evaluation. For details about
the team’s findings from our expert review, please refer to the Heuristic Evaluation in
Appendix C.

User Persona
The team originally developed two user personas – 1) a visually-impaired and/or blind
users; 2) a mobile device user. The two personas were developed because the text-based
version of eFolioMN is intended for both user types. However, in consultation with Dr.
Sadler and the client, the team decided to focus solely on visually-impaired and/or blind
users. The team found many tasks could not be performed using a mobile device (e.g.,
uploading a resume), and a test evaluator using a mobile device would not have the
possibility of succeeding to complete tasks. There were also several complications involved
in recruiting this user group and conducting the user evaluations.
Usability Testing Evaluators
This section describes the process for recruiting evaluators and provides an overview of
the individuals that were selected to complete usability testing. We sought evaluators that
shared characteristics similar to the persona in Appendix A – a visually-impaired and/or
blind individual who utilizes the screen reading software JAWS (Job Access with Speech),
goes on the Internet, and who would benefit from an online portfolio in their academic and
career goals.
RecruitingProcess
The disability services department at Metropolitan State University was contacted for
potential evaluators. The Technical Communication department also posted a constructed
recruitment email (see Appendix F) on a state wide disability coordinators list serve. A
member of our team, who herself is blind, contacted people that she knew through school
and work. Some of the people she contacted forwarded the recruitment email to others that
they knew through various avenues. Vision Loss Resources, a rehabilitation center for
visually-impaired and blind individuals, was contacted via phone to inquire about potential
evaluators. Six respondents were contacted to answer questions that were presented in a
more in-depth evaluator screening questionnaire (see Appendix G). Evaluators were
selected to participate in usability testing based on a combination of their responses to the
screening questionnaire and their availability for testing.
Although six evaluators were recruited and participated in test sessions, the team
discovered during a test session one evaluator did not possess the requisite intermediate-
to-advanced level skill with using JAWS software. As a beginning JAWS user, this evaluator
faced basic navigation challenges that impeded their ability to complete assigned tasks.
Consequently, the team decided against including this user in our report. We based our

findings on the five evaluators who met the user requirements that were defined in the
screening profile.
The five evaluators included one female and four males. Their ages ranged from 21 to 55.
EvaluatorOverview
All of the evaluators selected were either visually-impaired or blind and use JAWS
software. Additionally, they share some combination of the following characteristics:
 At least twenty-one years old
 Have some college education
 Familiarity with online forms
 Use of Internet on a regular basis
Description of Scenarios
Evaluators were asked to work through a set of six scenarios designed around the usability
testing goals outlined in the Executive Summary. A summary of each of the scenarios is
listed below, while the original text of the scenarios can be found in the Usability Test Plan
(see Appendix E).
Scenario 1 – IntroductiontotheeFolioMNwebsite
Evaluators were asked to explore the eFolioMN homepage and give their first impressions.
This scenario was designed to gain the following input from the evaluators:
 How do evaluators respond to the overall look and feel of the site?
 Are evaluators able to easily tell what eFolioMN is about?
 What tasks do evaluators think they can perform on the eFolioMN website?
Scenario2 – Creatingan eFolioMNaccount
In scenario 2, evaluators were asked to take the necessary steps to create an account. This
scenario was designed to answer the following questions:

 Are evaluators easily able to find the sign up link?
 How easy or difficult is the sign up process for evaluators?
Scenario3 – Creatingyourwebsite
Scenario 3 had two parts:
1. Log in to your eFolio Minnesota account.
2. Review the main menu home page of the tool and give us your impressions.
This scenario was designed to gain the following input from the evaluators:
 Are evaluators able to log in to the text-based version of the website?
 How do evaluators respond to the overall look and feel of the Main Menu page?
 Are evaluators able to easily tell what the Main Menu page of the tool is about?
 What tasks do evaluators think they can perform on the Main Menu page?
Scenario4 – Craftinga welcomemessage
1. Create a short welcome message for people who visit your site.
2. Add your welcome message to your site’s home page in the main column.
Scenario 4 answered the following questions:
 What areas of the tool do evaluators visit to create and add their welcome message?
 Are the eFolioMN link titles easy to understand?
 Is this task easy or difficult to complete?
 Do the tool’s features work like users expect?
If the evaluators did not click on the correct link initially, they were asked what type of
information they expected to find under specific links.

Findings Page 14 of 92
Scenario5 – Uploadingresume
1. Upload your resume to your content.
2. Add your uploaded resume to the Employment page.
 What pages do evaluators visit to upload their resume and add it to their site?
If the evaluators did not click on the correct link initially, they were asked what type of
information they expected to find under specific links.
Scenario6 – Makingyour websitepublic
In Scenario 6, evaluators were asked to make their website public so that other people
could view it.
 What pages do evaluators visit to make their website public?
Findings
This section presents participant responses and reactions to each of the six scenarios
completed during usability testing. The team examined the findings using a bottom-up
method called affinity matching whereby we listed our findings, then placed them into

categories and rated their severity level. Analysis and specific recommendations based on
these findings are in the section that follows.
The findings were sorted into the following three categories and then by severity level
within each category:
 Content –content that can inhibit users due to it being inaccurate, misleading, and/or
missing.
 Terminology – words that can be confusing to users because they are not clear or defined.
 User Interface – software design issues that can interfere with users’ ability to navigate
and complete tasks efficiently and effectively.
The findings were also ranked on a three-point severity scale based on the how
significantly the issue impacted the user.
Severity Scale
 High – user cannot complete a task; intervention by the moderator is required; or user
expresses extreme irritation and frustration; issue will definitely affect other users.
 Medium – user can complete a task, but is frustrated; issue is highly likely to impact
other users.
 Low – user can complete a task, but expresses minor irritation or annoyance; issue may
impact other users.
To view the findings sorted by severity level, please see the Executive Summary.
What the Evaluators Liked
All of the evaluators had positive reactions to the first impression of the eFolioMN home
page. They stated that the home page was clear and easy to navigate, citing that there were
not too many links or images. Our one visually-impaired evaluator who had some vision
commented that the links on the home page were large, and the page had a good contrast in
colors.
Whenever an evaluator encountered an error message, the message provided the evaluator
with specific information that assisted them in successfully correcting the error(s). When
evaluator 3 missed entering a title, the error was explained well. When evaluator 3 missed

filling in the required first and last name form fields, the error message clearly indicated
that the first and last name form fields were not filled in.
The overall impressions of the form fields on the website were good:
“Good job on radio buttons” – E1
“Liked how the form fields were labeled.” – E5
All evaluators expressed an interest in using the website to display their portfolio:
“I would use it…seems like a good place to gather all of the things you would want to
showcase yourself.” – E5
What the Evaluators Disliked
Based on the feedback received from the evaluators as well as our analysis of the
quantitative data from testing, we identified 26 significant usability issues with the
eFolioMN website. The following information lists these issues first by category and then
by severity, indicating their impact on the usability of the site, along with the number of
evaluators who experienced each problem.

User Interface:High
Sign up page 8: Captchadifficulties with audio challenge
All of the evaluators needed to listen to the audio challenge multiple times because they
were unable to hear the words clearly. Background noise in the audio challenge made it
hard to discern the challenge words, and the evaluators could not recall all of the six words
spoken. Additionally, they were unable to type the words while the audio challenge was
playing. When they began typing the challenge words, JAWS would verbalize what was
being typed while the audio challenge was being played – both the audio challenge and
JAWS would be speaking simultaneously.
“What was the mess in the background? There was some language in the
background that was very distracting for me trying to hear the Captcha options. I got
a couple of the words. It gave me five words and I don’t know if I should use all of
them.” – E5
“Oh, that’s bad…I can’t type while listening. I need to wait until it’s done and then
type
it” – E3
Figure 1: Captcha Box
Sign up page 9: Login link takes them to mouse-basedlogin page
Upon creating an account via the text-based version of the website, evaluators were taken
to a page that displayed a login link for the mouse-based version of the website. During all
of the testing sessions, evaluators needed to be prompted to switch to the text-based
version in order to avoid logging into the mouse-based version of the website.
“If I was coming from the text-based creation… if they have two sites like a text and
a graphical version or whatever, I would kind of expect if I’m coming from the text-
based creation I would go to the text-based login.” – E2

Text-basedversion not mentionedon home page
All evaluators were informed that they were evaluating the text-based version of the
website at the beginning of each session. There is no mention of a text-based option on the
site’s home page.
Figure 2: Switch to text-based version link
Figure 3: EfolioMN Home Page

“Graphic About Equals” referring to login link
All evaluators needed to be prompted to select the graphic login link that JAWS reads as
“Graphic About Equals” in order to reach the login page. JAWS reads the content in the
alternative text tag within the HTML which is coded as ‘alt=“ about=”’.
Figure 4: Login Version 2 Link
Extremely difficult to access text-based version
All evaluators needed to be directed to the log in page for the text-based version of the
website as they attempted to log in after finishing the sign up process.
User Interface: Medium
The assigned shortcut keys in thetool do not work forJAWSusers(JAWS already usesthese
keys and overrides thewebsite tool)
Three evaluators attempted to use the shortcut keyboard commands, but the commands
did not work as they expected. One evaluator pressed the letter “m” to activate the main
menu link, but JAWS said “There are no frames on this page.”
“It says all you have to do is press a letter, and it didn’t work. The main page “m.” I
pressed M and nothing happened.” – E3
Figure 5: Main Menu Shortcut Keys

Lack of confirmationafter successfully uploading content or adding it to their site
Two evaluators were unsure whether the task was completed successfully.
“Theoretically that got added so…”, but then proceeded to preview site to make sure.
– E2
“Doesn’t say ‘content added’ anywhere.” – E4
Figure 6: Pre-Upload and Post-Upload Pages
Desireto add newly created/uploaded content to a page directly afterclickingSave oreven
during the create/upload process
Two evaluators searched for the option to add their welcome message or resume file to the
assigned pages immediately from the screen first seen upon pressing the save button.
User interfaceis not intuitive
All evaluators struggled in determining which link to select in order to perform a task. They
frequently needed to be directed back to the main menu page to select a different link in
order to complete the task.
When adding the welcome message to their website, evaluators debated between the My
Content, Add New Content, and Manage Sites links.
“Jumping around” “cumbersome path” “convoluted process” – E2

Inconsistency in bottom menu barinsidethetool
Two evaluators were affected by this issue. They were expecting the same options to be
available in the menu bar, such as a link to the Main Menu, but were surprised when the
menu bar options changed and they could no longer return to the Main Menu. The users
were forced to go back to previous pages to find a link to the Main Menu.
Figure 7 : Bottom Menu Bars
User Interface:Low
Beginning ofsign upprocess: “Click Here” link followed by text “to sign up”
Two evaluators were confused by the “click here” link to begin the sign up process. They
needed to be prompted to review the page again in order to attempt to understand what
the link led to. They simply searched the site for links instead of reading all of the content
and therefore missed the “to sign up” wording.
“ ‘Click here’? For what I don’t know. I’ll just take your word for it” – E4
Form fields: Difficulty knowing required fields whenlisted in bold (JAWS does not indicate
text is bold)
None of the evaluators had difficulty with this issue due to the fact that they were provided
with the required information for the sign up process during testing. However, two
evaluators commented how this would be an issue for JAWS users since JAWS does not
automatically indicate whether text is bold.
“Use * instead of bold to show which fields are required.” – E1
“Difficult to know which fields are bold, use * instead.” – E5

Thought that clicking “SignUp” on homepage automatically started the sign up process
Three evaluators assumed that once they pressed on the sign up link on the home page, the
sign up process would begin. Two evaluators ignored the graphic sign up link (once on the
sign up page) when first read by JAWS and needed to review the page again to locate the
link.
“I’ve come not to pay attention to graphics. Some users turn off graphics and all
that’s left is the alternate text. Maybe distinguish it from the sign up link, maybe
‘continue to sign up’ or something.” – E5
Figure 8: Getting Started Screen
Too many steps in signup process
Evaluators needed to visit 10 web pages in order to complete the sign up process.
“Could have combined some things so there would be less steps.” – E5

Password field does not immediately appear onlogin page
Three evaluators were affected by this finding:
 One evaluator searched for the password field on the login screen, unsure if they should
click “Login (continue…)” without typing in password first.
 One evaluator was confused by lack of the password field.
 One evaluator expected the password field to be present immediately after the email
field.
Figure 9: Login Bar Missing Password Field

Terminology:Medium
“To Do” meaning
None of the evaluators selected the “To Do” link to complete the tasks before prompted.
When asked what “To Do” means, some evaluators commented:
“ ‘To Do’, I’m not exactly sure what that is, unless it’s a reminders thing.” – E1
“ ‘To Do’ means a reminder list.” – E5
Figure 10: To Do List
Confusion about options (MyContent, Add New Content, ManageSites)
All evaluators struggled in determining what link(s) to select to complete various tasks. In
addition, all evaluators needed prompting to try a certain link or were directed back to the
main menu page.
Two evaluators first selected Add New Content link first when attempting to create the
welcome message.
Two evaluators selected the My Content link first when trying to add their welcome
message to the home page.

Terminology:Low
“Revert” phrasingis confusingon somepages, including New Basic Content page
While no evaluators were affected by this, this issue was determined as a finding during
our heuristic evaluation. It is more common practice to use the word “clear” on online
forms.
Step 2 of thesignup process: Termsof Agreement. Button optionsare “I do not accept” and
“Continue…”
Four evaluators hesitated when they heard JAWS read the “continue” button, so they went
back to review the two button options before selecting the “continue” button.
Sign up readsas oneword, “Sig-nup,” during sign up process
One evaluator commented about this error and this was also determined as a finding
during our heuristic evaluation.
“A hyphenated word would definitely not be dependent on your synthesizer. It
would be user-friendly for all user environments.” – E5
Final stepof sign up process button says “Save”instead of “continue” or“complete”
One evaluator commented regarding this finding.

Content:High
Directionsin eachstepof the To Do list actually describethesteps that a user of the mouse-
basedversion would take
Two evaluators commented about this finding. Examples include “Begin by taking time to
watch the Overview video” which only describes how to use the mouse-based version and
“When finished, drag item to HOME page.” Step 12 also describes a process to make your
site public that is not actually possible to complete while on the text-based version.
“ ‘Drag item to home page’ is confusing.” – E2
“Drag item to home page? There wouldn’t really be a way to do that with a
keyboard.” – E1
Figure 11: To Do List Step 2: Create a welcome message
Content: Medium
Help information not present onall pages
One evaluator attempted to obtain help, but there was no help information on the page. Our
heuristic evaluation found that many pages within the tool lack any kind of Help
information.
“On the ‘Tag’ screen, I’m gonna look for online help…and I’m not finding it.” – E4

Content: Low
Inconsistency in whethersiteshave Headings or not
Four evaluators commented about the inconsistency regarding headings throughout the
website.
“Headings are really helpful when used correctly to get to main content.” – E1
“Should use headings to start sections.” – E2
“No headings on page to help navigation.” – E4
“Read more” link doesn’t lead to more content
One evaluator selected the “Read more” link and discovered that it did not contain any
content. It only contained the navigation links.
Post-Task Questionnaire Results
At the completion of scenarios 2, 4, 5, and 6, evaluators were asked to rate how easy the
task was to complete and to comment on what was particularly easy or difficult. Due to
time constraints, not all evaluators completed every scenario. The numbers within each
table reflect what each evaluator answered.
Scenario2 – Creatingan eFolioMNaccount
The responses varied widely for how easy scenario 2 was to complete.
Strongly
DISAGREE
Strongly
AGREE
1 2 3 4 5
I thought this task was easy to
complete.
4 3 5 2
Please comment on what was particularly easy or difficult:
"That wasn't bad at all"
Navigating with JAWS didn’t exactly go where expected; the audio Captcha was
interesting

Captcha, and not easy to find how to start sign up process
Scenario4 – Craftinga welcomemessage
The responses were neutral regarding how easy scenario 4 was to complete.
Strongly
DISAGREE
Strongly
AGREE
1 2 3 4 5
complete.
1 2 4 3
Unclear what the options were/what the links meant
Didn’t initially know he had to add it to the site after creating it
It was fairly simple, just the one thing that caught him up was getting back to the
main menu, it said push M, but even after tabbing to that link pushing M still didn’t
work
Difficulty knowing task has been completed
Scenario5 – Uploadingresume
Two evaluators agreed that scenario 5 was easy to complete.
Strongly
DISAGREE
Strongly
AGREE
1 2 3 4 5
complete.
2 3
"Because it's just if you remember all of the steps, kind of. There was a little
ambiguity to it."
Based mostly on remembering the process, once that’s memorized it’s easy.
Wouldn’t change any of the wording.

Scenario6 – Makingyour websitepublic
The responses varied for how easy scenario 6 was to complete, but two evaluators strongly
agreed with the statement.
Strongly
DISAGREE
Strongly
AGREE
1 2 3 4 5
complete.
5 2 3
Establishing familiarity with the site helped
Very easy
"It was not as intuitive as I would have thought"
Post-Test Questionnaire Analysis
A post-test questionnaire was delivered at the end of testing that gave evaluators an
opportunity to respond directly to their thoughts about the site, including their favorite
and least favorite parts and potential areas for improvement.
Evaluators were receptive to the overall look and feel of the site’s home page. They
expressed an interest in using the site to display their portfolio:
“The home page was clear and easy to navigate as there were not too many links or
images.” – E1, E2, E4
“The links on the home page were large and there was good contrast in the colors on
the page.” – E3
“I would use it, seems like a good place to gather all of the things you would want to
showcase yourself.” – E5

Several common themes emerged through the participant responses to the post-test
questionnaire:
 4/5 surveyed did not find the website to be complex.
 5/5 surveyed found the website was easy to use.
 4/5 surveyed did not need the support of a technical person to be able to use the
website.
 5/5 surveyed found the various functions in the website to be well integrated.
 4/5 surveyed did not think there was too much inconsistency in the website.
 5/5 surveyed imagined that most people would learn to use the website very quickly.
 4/5 surveyed did not find the website very cumbersome to use.
 4/5 surveyed felt very confident using the website three of five surveyed did not
believe that they needed to learn a lot of things before could comfortably interact with
the website.
Some eTUG members had a misperception about how evaluators would rate the site. We
assumed evaluators would rate the product poorly in the post-test questionnaire, but
surprisingly they mostly gave it high marks. Some factors to explain the seeming
discrepancy between the challenges they faced when performing tasks and their favorable
perceptions include:
 Evaluators perceived testers as the site’s developers and wanted to please them.
 Evaluators did not want to appear unpleasant on recording or in front of others.
 People tend to avoid extremes on a scale – the error of central tendency.
 People tend to rate responses consistently – responses to previous questions will be
repeated in later questions.
 The timing of the questionnaire.

We adapted the post-test questionnaire from the System Usability Scale (SUS) that was
developed by John Brooke at Digital Equipment Corporation. The test uses Likert-type
statements. The responses are based on a 5-point scale which ranges as follows:
1 – Strongly disagree
2 – Disagree
3 – Neither agree nor disagree
4 – Agree
5 – Strongly agree
Figure 12: Post-Test Questionnaire Responses
Shaded cells indicate the point on the rating scale that was picked the most for the
corresponding statement.
Strongly
DISAGREE
Strongly
AGREE
1 2 3 4 5
1. I think that I would like to use this
website to display my portfolio
1 2 4 5 3
2. I found the website unnecessarily
complex
3 5 1 4 2
3. I thought the website was easy to use 1 2 3 4 5
4. I think I would need the support of a
technical person to be able to use this
website
3 5 1 2 4
5. I found the various functions in this
website to be well integrated
1 2 3 4 5
6. I thought there was too much
inconsistency in this website
5 1 2 4 3
7. I would imagine that most people
would learn to use this website very
quickly
2 4
5
1 3
8. I found the website very cumbersome
to use
1 3 5 2 4
9. I felt very confident using this website 4 1 2 3 5
10. I needed to learn a lot of things before
I could comfortably interact with this
website
1 2 5 3 4

Recommendations Page 32 of 92
Recommendations
This section includes recommendations to improve the usability of the eFolioMN website
and support the organization’s overall goals for the site. The recommendations are based
on the eTUG team’s findings from user testing. The recommendations are organized into
three major categories:
 Content
 Terminology
 User Interface
Some recommendations can be characterized as cosmetic in nature, but will go a long way
to improving the overall usability of the text-based version of the website.
Recommendations are assigned a severity ranking of high, medium, or low.
Content
Content:High
Creating directions for how to use the text-based version of the website will assist users –
especially new users – in completing the necessary steps both effectively and efficiently
when creating their portfolio.
Adding relevant help information to each page would assist in the navigation and
completion of tasks. The information could include what steps to perform and in what
order to perform them in, thus removing the “guess work” from the process.
Content:Low
Adding content to the “read more” page will give users the opportunity to read additional
content. If no content is desired for the page, removing this link/page will help users not
unnecessarily select it only to find there is nothing more to read.
Consider adding headings to all pages since some JAWS users look specifically for headings
to aid in their navigation of content.

Terminology
Terminology:Medium
Change the “To Do” link name to something that users would understand the meaning of,
such as “New User Tasks”. Possibly set it apart from the others, making it clear it is only for
new users since the tasks disappear upon completion.
To decrease confusion and potential frustration when attempting to select from the
navigational links on the main menu home page, consider changing the navigational link
names of “add new content,” “my content,” and “manage sites,” or add help text to explain
what each section does.
Terminology:Low
Change “revert” to a word more commonly used on the Internet such as “clear” on all web
pages where this terminology is found.
Change the “continue” button on step 2 of the sign up process to “I Accept” in order to
correspond with the “I do not accept” button.
On the final step of the sign up process, change the “Save” button to “Continue” or
“Complete” in order to minimize hesitation in the user’s response.
Add a hyphen or a space to the word “Signup” on all corresponding sign up pages. This will
improve the quality of the website as grammatical errors can be viewed as lack of pride or
interest in the site’s reputation.
User Interface
User Interface:High
To eliminate the need for users to attempt to listen and understand the Captcha challenge:
 Consider asking a question such as “what is the color of the sky?” or a simple math
question instead of using Captcha.
 Consider including a confirmation link in the email instead of using Captcha.

Take users to the text-based login page after completing the text-based sign up process.
This would ensure that users who desire or need to use the text-based version can do so
without inadvertently logging into the mouse-based version.
Mention the text-based version on the home page and wherever else deemed relevant. This
would ensure that users who desire or need to use the text-based version will know about
it and be able to access it.
For the login link on the home page, change the “alt” text to “Login” instead of “Graphic
About Equals.” This will ensure that all users of various types of screen reading software
such as JAWS will hear “login.”
Add a “login” link for the text-based version of the site on the home page. This will make it
easy for text-based users to be aware of this option and access the site with ease.
Additionally, users will be less likely to login to the mouse-based version.
User Interface:Medium
Include a sentence on the initial page of the tool mentioning that JAWS users will not be
able to effectively use these shortcut keys since JAWS overrides them.
Consider using a consistent bottom menu bar inside the tool. This will increase the
usability of the website as users will instinctively know where to find common links.
Consider adding a confirmation message after completion of tasks to allow users the
satisfaction of knowing the task has been completed successfully and eliminating any doubt
in their mind.
Include relevant help information and consider including some text on the Main Menu page
that explains the overall process of how to use eFolioMN. Doing so would assist in
decreasing the desire to add newly created/uploaded content to a page directly after
clicking Save or even during the create/upload process. This would also make the site more
intuitive.
User Interface:Low
Change the “click here” link to include the text “sign up” at beginning of sign up process on
the login screen. This will ensure users who are focusing only on the links for the sign up
link, will be able to locate it quickly. This follows basic SEO (search engine optimization)
techniques.

Conclusion Page 35 of 92
Use asterisks (*) before each required field instead of bold on all forms. This will allow all
users to quickly recognize what fields are required and will decrease the probability of
missing a required field.
Add the password field immediately after the email address field to eliminate any
confusion and hesitation of users attempting to login.
Since some evaluators thought selecting the “Sign Up” button on the home page
automatically started the sign up process, consider changing the wording of sign up button
to “Begin Sign Up Process” instead.
Consider combining some sign up pages to reduce the number of pages users must click
through to complete the sign up process.
Conclusion
A combination of usability testing and an expert review of the text-based version of the
eFolioMN website conducted by the eTUG team has revealed strong points of the website in
addition to a number of opportunities for improvements. The experience of our evaluators
indicates that the website does not fully meet eFolioMN’s goal of having a site that is fully
accessible and usable for visually impaired and/or blind users.
It is critical that direct access to the text-based version and information about how to use
the site be added. It is also important to add relevant help instructions on every page of the
site, particularly for new users.
eFolioMN’s online portfolio development tool features many more pages and tasks than
could be evaluated within the scope of this evaluation. Additional usability testing of other
areas of the website will likely reveal further areas where further improvements can be
made.
Our recommendation is to make immediate short-term changes and begin strategizing
ways to implement long-term changes in order to help users achieve their goals in a more
efficient manner.

Appendix A: User Persona Page 36 of 92
Appendix A: UserPersona
Elsie Manning
“After working for nearly 30 years in the engineering field, I
would now like to give back to society, using my skills in a
meaningful way.”
Demographic:
1. Gender: Female
2. Age: 52
3. Occupation: Electrical Engineer
4. Income: 125k
5. Technical Profile: High proficiency with engineering technology and personal
computers, less with mobile social applications
6. Internet use: 4 hours daily for work collaboration.
7. Hobbies: Gardening, ecology activities, hiking, traveling to see children
Elsie takes pride in the fact that she has been able to balance raising her children with a
rewarding engineering career. The children are now grown and Elsie is interested in
strengthening her connection to the engineering community at large as well as exploring
more activities outside of work.
Elsie regularly communicates with other engineers via peer groups to make sure her skills
remain current and to provide new insight for her projects.

Appendix A: User Persona Page 37 of 92
Elsie volunteers when she can. In college, and for the majority of her career, Elsie
successfully adapted to working in a male-dominated industry. Overcoming that career-
challenge motivated her to mentor young women in the field whenever possible. She is
interested in continuing that pursuit. Elsie also has a keen interest in ecology and
environmental issues. She thinks that her engineering skills and desire to improve
efficiency transcends to other engineering fields. In the past, Elsie has volunteered for
Habitat for Humanity by building houses and designing landscapes. She may want to
continue that, but also plans to explore various fields to see where she can provide the
most impact.
Elsie uses the Web at work as a research and collaboration tool. She typically bookmarks
useful sites. To Elsie, good sites allow her to find the specific information she needs quickly.
She will commonly visit a site for answers to specific questions (e.g., “what is ‘this thing’ on
a system board called?”), or what are the specifications for a manufacturer’s part. She gets
frustrated when documents are not available on a site or are difficult to find. Under those
conditions, she returns to sites with a logical, intuitive path to what she needs.
Overall, Elsie’s goals are to devote more time using her electrical engineering skills and
growing interests in other areas of engineering in a charitable way, perhaps even on a
global scale. Using the internet to assist in that goal would save her time and expand the
impact she can make.

Appendix B: Personas Memo Page 38 of 92
Appendix B: Personas Memo
Memorandum
To: Dr. Victoria Sadler
From: Team eTUG
Date: February 19, 2012
Subject: Personas for eFolioMN.com (text-based version)
Team eTUG has created two distinct personas in order to help us better understand typical
eFolioMN users. Learning as much as possible about target users is essential to building a strong
foundation for an effectiveusability evaluation. The steps taken to gather this information are detail
below. The twopersonas eTUG has created can be found attached to this memorandum.
Step 1: GatherInformationfromeFolioMN
The first step in our process was to gather as much information about the typicaleFolioMN user
from the creators of eFolioMN, specifically PaulWasko. Our team leader sent an email to Mr. Wasko
with a variety of questions with the hopes of obtaining more information about the typicaluser.
Unfortunately,eFolioMN does not collectvery much demographic information about its users, so
Mr. Wasko admitted he couldn’t be of much help. They do not collectany data on how many users
with visual impairments access the site, nor do they collectdata on how the site was accessed (e.g.,
from a laptop or a mobile device).Mr. Waskodid tell us that eFolioMN is promoted on the
disability.gov website, so the information is out there forthose whowish to find it. He also stated
that he believes the majority of mobile deviceuse is via the iPad whichcan largely be attributed to
the growing use of iPads in schools.
Step 2: GatherInformationfromprospectiveusers
As previously stated, eFolioMN collectsvery little demographic data about its users. This means
that eTUG did not have a user database from whichto draw information. However,since eTUG is
investigating the usability of the text-based version of the eFolioMN site, we already had
established twomain groups forprospective users: visually impaired users and mobile device
users. In order to find out more about each group of prospective users, it was necessary foreTUG to
conductseveral personal interviews. To reach mobile device users, eTUG sent an email to the
undergraduate and graduate technical communication students at Metro State asking for
volunteers. Six people volunteered to be interviewed, and the interviews were quickly conducted.

To reach visually impaired users, a vision-impaired member of out team reached out to her
networkand was able to obtain three interviews.
To provide informed consent for the interviewees, each interviewee was assured of their
anonymity and that answering each question was optional. Along withcollecting demographic
information, some examples of the questions asked are:
1. How much time do you spend on the web in a typical day?
2. What do you use the web for?
3. How experienced are you at creating a web page?
4. How interested are you in using a free tool that will help youbuild an online portfolio?
Mobile Device User Findings
Of the six people interviewed as mobile device users, three were male and three were female. The
average age was 27. Professions ranged from full-time positions as a technical writer to part-time
employment or seeking work.Fiveof the respondents identified their technical expertise level as
average or above,with one respondent answering below average. Eachrespondent admitted to
spending large amounts of time on the internet daily, with at least half an hour each day spent using
a mobile device. Their online goals varied from entertainment to shopping to work(professional or
school).
Visually Impaired User Findings
The three visually impaired respondents included twofemales and one male. The average age was
46. Professions ranged fromIT Specialist to Assistive Technology Expert. All three had experience
using assistive technology such as JAWS to aid their computer experience. Also, each respondent
stated that his or her level of technical expertise is above average. Typicalactivities on the internet
included communicating via email or Facebook,shopping, consuming information (news,
entertainment or new product research) and banking.
Step 3: IdentifySignificant BehaviorPatterns
Mobile Device Users
All but one of the respondents claimed that the idea of a free toolto help them create an online
portfoliowas very interesting. The one respondent who dissented was employed and used
LinkedIn for their online professional presence. All six respondents stated that having a mobile
version of a website was very valuable and/or useful. This shows that convenienceand ease of use
are important to this prospective user group.
Visually Impaired Users
All three of the respondents were unsure of their interest level in using a free tool to help them
create an online portfolio. Eachrespondent stated that he or she would like to read more
information about the tool before committing to an interest level. They also each said that they
would consider using such a tool in order toreview someone’s credentials. Interestingly, despite

(or perhaps because of) their high levels of technical expertise, each respondent was unwilling to
commit to an interest level of the free toolbefore learning more information. This couldmean
eFolioMN should consider adding accessible and helpful information on the home page so that
these users can learn more about the tool before they sign up.
Step 4: CreatePersonas
After significant behavior patterns were identified, we gathered the information from the
interviews to create two personas: a mobile deviceuser and a visually impaired user. The resulting
synthesis of this information can be found in the attached personas file.
Conclusion
The four steps discussed above helped eTUG understand the needs of prospective users of the text-
based version of eFolioMN. Twopersonas were created based largely on data obtained through
interviewing potential users due to the lack of demographic data collectedby eFolioMN.The
creation of these personas willbe used to select participants and develop scenarios forthe usability
testing portion of our project.

Appendix C: Heuristic Evaluation Page 41 of 92
Appendix C: HeuristicEvaluation
Heuristic Evaluation of
eFolioMN.com
eFolioMN Text Version for Visually Impaired and Hand-Held Technology Users
Completed by Team eTUG
(eFolioMN Text-Based Usability Group)
Ron Heck
Sue Ann Rodriquez
Shaun Bibo
Kevin Glenz
Writ 375 Sadler – February 13, 2012

Table of Contents
Introduction......................................................................................................................................................................44
Purpose................................................................................................................................................................................44
Methodology.....................................................................................................................................................................44
Heuristic Evaluation Findings................................................................................................................................45
1. Visibility of System Status ......................................................................................................................................45
2. Match between system and the real world ........................................................................................................46
3. User Control & freedom ..........................................................................................................................................47
4. Consistency & standards.........................................................................................................................................49
5. Error Prevention.......................................................................................................................................................51
6. Recognition rather than recall...............................................................................................................................52
7. Flexibility & efficiency of use .................................................................................................................................53
8. Aesthetic & minimalist design...............................................................................................................................55
9. Help users recognize, diagnose, & recover from errors .................................................................................56
10. Help & documentation .........................................................................................................................................57
Summary of Findings...................................................................................................................................................58
A. Layout Design ................................................................................................................................................................58
B. Utilization of Portfolio Tool........................................................................................................................................59
C. Navigation.......................................................................................................................................................................60
D. Terminology...................................................................................................................................................................60
Conclusion..........................................................................................................................................................................61
References..........................................................................................................................................................................62

Heuristic Evaluation of eFolioMN.com
Introduction
The heuristic evaluation of eFolio MN text-based version was completed on February 10th,
2012, by eTUG (eFolioMN Text-Based Usability Group) as the initial stage for a usability
test of the website. Preliminary information provided by the client revealed that “eFolio is a
free tool available online for any Minnesota resident who wants to create an electronic
portfolio [and]…is used primarily for educational purposes.” The client went on to reveal
their interest in attracting more career-focused, professional users.
The goal of potential participants using eFolio would be to successfully navigate through
the entire site utilizing the text-based tool with as little additional support or clarification
as possible. For the purposes of this heuristic evaluation there are two types of users who
will navigate the text-based eFolio MN site: visually impaired users and mobile device
users. Therefore, two eTUG members accessed the site utilizing a tool for the visually
impaired (the screen reader JAWS) and two members accessed the site utilizing mobile
devices (an iPad and an iPhone).
Purpose
The purpose of this heuristic evaluation is to provide objective feedback about the
experiences of eTUG while navigating throughout the text-based site. The outcome of the
collective experiences will be compiled and used to design scenarios for further usability
testing.
Methodology
eTUG evaluated the eFolio website from two perspectives: a professional user accessing
eFolio from a hand held device and a professional user with a visual impairment. Initial
group analysis of the site helped to determine the scenario. The professional user was to
simply upload content and review the public end result. This would force the participant to
navigate through the entire site, sign up for an eFolio account and complete enough steps
to display content on his or her portfolio. Each member of eTUG evaluated eFolio
individually using Jakob Nielsen’s 10 Heuristics and Severity Ratings for Usability Problems
(0-4) as found in Barnum’s Usability Testing Essentials (Barnum, 2011). The group then
compiled its findings to form this report.

Heuristic Evaluation Findings
1. Visibility of System Status
The system should always keep users informed about what is going on, through
appropriate feedback within reasonable time.
A. Link to blank page
The “Read more” link on the
Home page leads to an
unnecessary page that has
no extra content for the
user to read.
Suggestions:
1. Create content that
explains what Login
Version 2 means.
2. Consider eliminating
this link and page
completely.
B. Visibility oftext-basedoption
When logging in, user must remember to click “Switch to text-based version” every time
Suggestion:
Add notification such as “You previously logged in using the text-based version. Would you
like to use the text-based version again?”

2. Matchbetweensystemand therealworld
The systemshouldspeak theusers’language, withwords, phrasesandconcepts
familiartotheuser, ratherthansystem-orientedterms. Followreal-world
conventions,makinginformationappearin a naturalandlogicalorder.
A. Confusing word
choice
The Revert
command is
confusing – usually
means to undo all
revisions and start
over.
Suggestions:
1. Define use of
Revert in the
Help text.
2. Perhaps use the
word “Clear”
instead.
B. Confusingword choice
“Change Sorting” is confusing and sounds like a noun
rather than a verb.
Suggestion:
Change to “Sort My Content”.

C. Confusing word choice
“Postal Code” is used during the
Sign Up process and may be
confusing to some users,
especially since it is changed to
“Zip” in the Account Summary.
Suggestion:
Use “Zip Code” throughout the site
to avoid confusion.
3. User Control& freedom
Usersoftenchoosesystemfunctionsby mistakeand willneeda clearlymarked
“emergencyexit”toleavetheunwanted statewithouthavingtogothroughan
extendeddialogue.Supportundoandredo.
A. Not clearwhereto login
The Home page does not make it clear immediately
where to login. The button “Login Version 2” may be
unclear to new users.
Suggestion:
Add dedicated and prominent link/button simply
called “Login” near the top of the page.

B. No clear link to
text-based version
It is difficult to find a
link to the text-
based version as a
new user. There is
also no mention of a
text-based version
on the Home page.
This could cause
frustrated searching
for the link or
relevant
information before
giving up and
leaving unsatisfied.
Suggestions:
1. Add link to text-
based version on
Home page.
2. Add description
of text-based
version to Home
page.

4. Consistency &standards
Usersshouldnothavetowonderwhether differentwords, situations,or actions
meanthesamething. Followplatformconventions.
A. Missing alt-text for images
Not all images have alternative
text for screen readers.
Suggestion:
Add alternative text for all
images.
B. “Support Home”
incomplete pagetitle
When using a screen
reader, the page title reads
as “eFolioMinnesotaTM”
with no mention of it
being the support home
page. After selecting “Find
Answers” or any other tab
the title then becomes
“eFolio Support Home”
Suggestion:
Add “Support Home” to the
title of first page to avoid
confusion.

C. Inconsistent link layout
The four buttons “About Us,
Resources, Showcase, Contact Us”
move to appear vertically on the
right side after appearing
horizontally on every other Home
page link.
Suggestion:
Maintain consistency – place
buttons horizontally to match
other pages.

5. Error Prevention
Evenbetter thangooderror messagesisa carefuldesignwhichpreventsa
problem fromoccurringin thefirstplace.Either eliminateerror-prone
conditionsor check for themandpresentuserswith a confirmationoption
beforetheycommittotheaction.
A. Copy and Cancel error
Highlighting text and using Copy command
(Control-C or Command-C) activates the “c”
shortcut for Cancel and cancels the user’s
actions.
Suggestions:
1. Include a logout confirmation such as “are
you sure you want to log out?” or “would
you like to save this before logging out?”.
2. Consider changing logout shortcut key to
“l” or something else.
3. Place logout option on Main Menu page
only.
B. Firefox keyboard shortcut problem
In Firefox, typing a keyboard shortcut like “m” for main
menu initiates a search and places the characters typed
into the browser’s search field.
Suggestion:
Instruct Firefox users to turn off the “Search for text
when I stop typing” feature in Firefox’s accessibility
preferences.

6. Recognitionratherthan recall
Minimizetheuser’smemoryload by makingobjects, actions, andoptions
visible. Theuser should nothavetoremember informationfromonepartof the
dialoguetoanother. Instructionsfor useof thesystem shouldbevisibleor
easilyretrievablewhenever appropriate.
A. Tutorial video
issues
Text says to watch
overview of Version 2
video, but there is no
clear link or directions
for how to do this.
Users must click
“Complete This Item”
to view video. The
video only talks about
the mouse-based
version.
Suggestion:
1. Add link to or
directions for how
to view overview
video.
2. Consider creating a
video (or audio)
tutorial to help
users with the text-
based version.
B. Login link not visible on eachpage
“Login Version 2” link is only visible on
home page.
Suggestion:
Login link should be available on each
page.

7. Flexibility& efficiencyof use
Accelerators–-unseenby thenoviceuser –-mayoftenspeeduptheinteraction
for theexpertusersuchthatthesystemcancater tobothinexperiencedand
experiencedusers. Allowuserstotailorfrequentactions.
A. Multiple clicks to sign up make it difficult for
visually impairedusers
Continue button must be clicked 7 times to
complete sign up process and the user cannot
back out of first screen.
Suggestion:
1. Shorten the number of times a user must
click to complete sign up process by adding
more content to each page.
2. Enable users to “complete at a later time”
and a way to back out of every page that will
not negatively disrupt their experience.
B. Cannot upload files from
mobile device
It does not appear to be
possible to upload an image
or file from a mobile device.
Suggestion:
1. Find a way for hand held
devices to upload
material.
2. Inform users of the
limitations of using the
text-based version on a

mobile device.
C. Unnecessary extraclick
to login
When a user first arrives
at the login screen only
the “Email Address” field
is visible. The user must
enter his or her email and
click “Login (continue…)”
to make the “Password”
field appear.
Suggestion:
Make the “Password” field
visible immediately.

8. Aesthetic& minimalist design
Dialoguesshouldnotcontain informationwhichisirrelevantor rarelyneeded.
Every extraunitof informationin a dialoguecompeteswiththerelevantunits
of informationanddiminishestheir relativevisibility.
A. Returning Userspage seems
unnecessary
This page appears to serve no purpose.
It does not offer needed information
and the user must click another link to
reach the same login screen as “Login
Version 2” button.
Suggestion:
Consider eliminating this page.
B. Duplicate Links
The Resources page contains
duplicate links for Student, Educator,
and Career within a small space.
Suggestion:
Reduce clutter and have only one link
per user type.

9. Help usersrecognize, diagnose, &recover from errors
Error messagesshouldbeexpressedin plainlanguage(no codes), precisely
indicatetheproblem, andconstructively suggesta solution.
A. YouTube
tutorial video issue
Nowhere on the
page does it say
that clicking the
link of the still
picture in the
middle opens a
YouTube tutorial
video. The link
takes the user to a
new page with no
clear way to
return to the
eFolio site.
Suggestions:
1. Open the video
in a new page.
2. Create a link on
the YouTube
page (in
description
area, in video
itself) to get
back to eFolio.
3. Embed the
video on the
Home page.

10. Help & documentation
Eventhoughit isbetter if thesystemcanbeusedwithoutdocumentation, it
may benecessarytoprovidehelpanddocumentation. Anysuchinformation
shouldbeeasytosearch, focusedon theuser’stask, listconcretestepstobe
carriedout, and notbetoolarge.
A. Missing help text
On the Content Operations page within the tool no help
text appears and clicking the Help link or typing “h”
does nothing.
Suggestion:
Add help text or link to help page on pages that are
currently missing it.
B. Help text in wrong order
On some pages within the tool the
help text is listed in opposite order of
items on the page, e.g., “Full, Brief,
Title” vs. “Title, Brief, Full”.
Suggestion:
Place content in consistent order.

Summaryof Findings
Our findings based on the evaluation of the website are prioritized into four categories:
Layout Design, Utilization of Portfolio Tool, Navigation and Terminology. These four
categories are further broken down into Major Issues, Minor Issues and Cosmetic Issues.
A. LayoutDesign
The website contains various layout design issues that interfere with a user’s ability to
easily locate log in options and the text-based version of the website.
MajorIssues
There is no option for users to select the text-based version of the website on the home
page. This may result in the user concluding that the website does not contain a text-based
version. In addition, the user can only select the text-based option on the log in page before
logging in. Again, this may appear to a user that the website does not have a text-based
version if the option is overlooked (i.e., not seen).
The user must search to find out where to log in since there is no screen reader friendly log
in option on the home page. The user must select the “Returning Users” link to log in; on
the subsequent page, only the log in option appears. Not having an accessible log in option
on the home page forces the user to search for it and possibly navigate to an unnecessary
page.
Minor Issues
The horizontal navigational buttons (About Us, Resources, Showcase, and Contact Us) are
not consistent throughout the website. For example, on the “Sign Up Here” page, the
buttons appear vertically on the right. This inconsistency can cause users to be confused
and require them to search for these buttons.
In order to complete the sign up process users must select the “Continue” button seven
times. This can be frustrating to users as there is only a little bit of information required
each time.
On the log in page, only the email field is visible at first. Users must enter an email address
and select “Login (continue...)” in order for the password field to appear. As a result, the
current layout of these fields causes users to perform two steps instead of just one.

On the “eFolio Resources” page, there are duplicate links for Student, Educator, and Career
users. Having links in one control location will enable any type of user to locate the various
links more effectively.
On the “New Image Content” page, the help text does not follow the order of the page
elements. This can be confusing to users since all of the other pages follow this design
layout.
B. Utilizationof PortfolioTool
There are issues found on the website that can inhibit users from successfully learning and
utilizing the portfolio tool.
MajorIssues
Users of mobile devices cannot upload any files to their portfolio. The inability to perform
this task forces users to locate and use devices that will allow this.
Users are unable to use the copy command (control + c) to complete tasks due to the fact
that the letter c is the shortcut command to cancel. For example, on the “New Basic
Content” page, if the copy keyboard command is attempted, the user’s action is cancelled.
This makes it impossible for users who are unable to utilize a mouse to use the copy
shortcut keyboard command.
The log out and cancel actions both use “c” as their shortcuts on different pages. This can be
frustrating to users who may inadvertently press the letter “c” by accident and an
undesired action takes place.
On the “To Do Step 1” page there is not information about where or how to view the
“overview video”. This can be frustrating to users as they will need to spend time searching
for the video in order to learn about how to utilize the portfolio tool.
Some of the pages do not contain any help text. Users may not be able to complete tasks
successfully if they are unsure of what must be done to complete the task. As a result, users
will need to take the time to contact eFolio for assistance or decide not to continue on with
attempting to complete the task.
Minor Issues
Firefox can affect the successful interaction that users have with the portfolio tool because
in Firefox, typing a shortcut command such as m for main menu will initiate a search. As a

result, the characters typed are placed into the browser’s search field. This will take place
all while users are logged into eFolio.
C. Navigation
The website contains issues that affect the consistency of the layout. A user must take extra,
unnecessary steps to navigate the site successfully.
MajorIssues
On the log in page, the user must remember to select the text-based version option before
logging in as this is the only page that contains this option. If users forget to select this
option, they will log in to the mouse-based version. Users must then log out and return to
the log in page and select the text-based version in order to access the text-based version to
navigate the website and utilize the portfolio tool effectively.
When selecting to view the tutorial video via the image link in the middle of the home page,
users leave the eFolio website and are brought to the YouTube website. This greatly effects
the navigation of the website as there is no clear way to return to the eFolio website.
Minor Issues
Not all images on the website have alternative descriptive text associated with them. This
has the potential of making it difficult for screen reading software users to navigate the
website.
The “Read more” link on the home page encourages users to find more information.
Instead, users are taken to a page that contains nothing other than the overall website
navigation links with no additional content.
D. Terminology
Throughout the website there are words that can be confusing to users as they are not
clear or defined.
MajorIssues
The word “revert” for the revert feature on the website can be confusing as it is more
commonly used to mean to undo all revisions and start over. Users who desire to return to
the previous page will more than likely search for the “back” button as this word is
commonly used on websites (if not familiar with the way to go back manually).

Minor Issues
Users can become confused upon entering the eFolio support home page as the title of the
page is “eFolioMinnesotaTM” and, thus, does not indicate to users clearly that they are on
the support home page.
On the “Choose Link For” page, the “Choose Sorting” option is confusing; the phrase sounds
like a noun rather than a verb and what will be sorted is unclear as it is not defined. Users
are left to determine the meaning of the option. This can lead users not to utilize this
feature or unnecessarily use the feature if the results are not what they thought they would
be.
Different terminology is used for the same purpose and may confuse users. For many pages
including “New Image Content”, users choose “Cancel (c)” to leave the page. However, on
the “Account Summary” page, users must choose “Exit (x)” to leave the page (there is no
“cancel” option)
Cosmetic Issues
The phrase “postal code” is inconsistently used throughout the website and can be
confusing to users. In the Sign Up process on page 7, “postal code” is used, but on the
“Account Summary” page, “zip code” is used. The phrase “postal code,” can be confusing as
it is commonly referred to as “zip code” on websites.
Conclusion
Most of the issues team eTUG discovered are minor, but we uncovered several glaring
usability issues that pose major hurdles for users.
We found that eFolio is a great tool for those who choose to display their work on the site.
We also found the site provides prompt customer service. For instance, one team member
emailed eFolio on a Sunday afternoon at 3:19 p.m. and received a response at 3:26 p.m.
Such great customer service is just one of the many perks that this site has to offer.
However, since eFolio is a site focused on students, educators and professionals, the many
minor issues can impact the credibility of the site. Negative navigation experiences could
turn off potential participants.
The first major reaction of our collective experiences to testing this site using the text-
based version was finding this version. We also were confused by what Version 2 is. As
completely new users, we found that the phrase “How to access the eFolio Owner site
version 2 (launched May 4, 2009)…” is information that is not helpful to new users. Users

must make a hunch that if they click on the “Login Version 2” button they will get to the
regular log in page.
Referring to key usability concerns outlined by Carol Barnum in Usability Testing
Essentials, we find that the site will likely not leave a good first impression for new users.
Barnum states that users decide right away if a site can be trusted and has credibility, and
that first impressions are critical to users’ perceptions of how easy or difficult their web
experience will be. We believe new users would be frustrated and confused unless they are
provided with thorough introductory training. Even with training, the inconsistencies of
the site will be troublesome and likely lead to an unsatisfying user experience. We look
forward to testing this site with actual eFolio users and further improving our
recommendations.
References
Barnum, C. (2011). Usability Testing Essentials: Ready, Set…Test!. Burlington, MA:
Elsevier.
http://www.efoliominnesota.com

Appendix D: Jakob Nielsen’s 10 Heuristics for User Interface Design Page 63 of 92
Appendix D: Jakob Nielsen’s 10 Heuristics forUser Interface
Design
1. Visibility of
system status
The system should always keep users informed about what is going
on, through appropriate feedback within reasonable time.
2. Match between
system and the
real world
The system should speak the users’ language, with words, phrases
and concepts familiar to the user, rather than system-oriented terms.
Follow real-world conventions, making information appear in a
natural and logical order.
3. User control and
freedom
Users often choose system functions by mistake and will need a
clearly marked “emergency exit” to leave the unwanted state without
having to go through an extended dialogue. Support undo and redo.
4. Consistency and
standards
Users should not have to wonder whether different words, situations,
or actions mean the same thing. Follow platform conventions.
5. Error prevention Even better than good error messages is a careful design which
prevents a problem from occurring in the first place. Either eliminate
error-prone conditions or check for them and present users with a
confirmation option before they commit to the action.
6. Recognition
rather than
recall
Minimize the user’s memory load by making objects, actions, and
options visible. The user should not have to remember information
from one part of the dialogue to another. Instructions for use of the
system should be visible or easily retrievable whenever appropriate.
7. Flexibility and
efficiency of use
Accelerators –- unseen by the novice user –- may often speed up the
interaction for the expert user such that the system can cater to both
inexperienced and experienced users. Allow users to tailor frequent
actions.
8. Aesthetic and
minimalist
design
Dialogues should not contain information which is irrelevant or
rarely needed. Every extra unit of information in a dialogue competes
with the relevant units of information and diminishes their relative
visibility.

Appendix D: Jakob Nielsen’s 10 Heuristics for User Interface Design Page 64 of 92
9. Help users
recognize,
diagnose, and
recover from
errors
Error messages should be expressed in plain language (no codes),
precisely indicate the problem, and constructively suggest a solution.
10. Help and
documentation
Even though it is better if the system can be used without
documentation, it may be necessary to provide help and
documentation. Any such information should be easy to search,
focused on the user’s task, list concrete steps to be carried out, and
not be too large.

Appendix E: Usability Test Plan Page 65 of 92
Appendix E: Usability Test Plan
Usability Test Plan for
eFolioMN.com
eFolioMN Text Version for Visually Impaired and Hand-Held Technology Users
Completed by Team eTUG
(eFolioMN Text-Based Usability Group)
Ron Heck
Sue Ann Rodriquez
Shaun Bibo
Kevin Glenz
Prepared for Paul Wasko, MNSCU/eFolioMN
CC: Tori Sadler, Writ 574/G
Metropolitan State University
March 18, 2012

Table of Contents
Executive Summary...........................................................................................................................................................67
Problem Statement and Test Objectives...............................................................................................................68
User Profile..............................................................................................................................................................................69
Methodology...........................................................................................................................................................................69
Preparation........................................................................................................................................................................70
Testing ..................................................................................................................................................................................70
Think aloud protocol....................................................................................................................................................70
Session length...................................................................................................................................................................70
User Incentive ..................................................................................................................................................................70
Scenarios...................................................................................................................................................................................70
Test Environment/Equipment....................................................................................................................................74
Location/Evaluation room/Facility information .......................................................................................74
Equipment..........................................................................................................................................................................74
Project Timeline ...................................................................................................................................................................74
Team Roles and Responsibilities ...............................................................................................................................75
Moderator...........................................................................................................................................................................75
Logger....................................................................................................................................................................................75
Observer..............................................................................................................................................................................75
Technical support..........................................................................................................................................................75
Evaluation Methods...........................................................................................................................................................76

Quantitative Data ...........................................................................................................................................................76
Qualitative Data...............................................................................................................................................................76
Deliverables............................................................................................................................................................................77
ExecutiveSummary
The purpose of conducting this usability test on the text-based version of the eFolioMN
website is to gain knowledge of how users who are visually impaired (including blind)
interact with the website utilizing assistive technology. The usability test will specifically
focus on users who have the following:
 Intermediate to advanced computer skills
 Beginner to intermediate web experience
 Potential interest in using an online portfoliotool
 Access to the Internet via a screen reading software program such as JAWS(Job Access with
Speech)
The usability test results will assist eFolioMN in determining whether the text-based
version of the website is accessible for people with disabilities, specifically those who have
a visual impairment. The results collected from the usability test will also be reviewed and
taken into consideration when the text-based version of the eFolioMN website is further
tested and redesigned. In addition, the results will assist the client in understanding how
best to modify various aspects of the website in order to make the website more accessible
to the visually impaired user population.
The usability of the eFolioMN website – the effectiveness and efficiency of the users’
experience as they perform various tasks and navigate to specific areas within the website
– is directly tied to its success in terms of the following objectives (based upon Nielsen’s 10
Heuristics for User Interface Design):
 Determine if users are able to navigate various areas of the website
 Determine if users are able to utilize the online portfolio tool
 Determine if users are able to locate necessary information they need in order to complete
tasks withoutassistance

This usability test plan, with its scenarios, tasks, and questionnaires, has been designed
around these objectives. We have adapted these general objectives to the specific visually
impaired JAWS screen reading software program user who has the potential to utilize the
eFolioMN online portfolio tool.
ProblemStatementandTestObjectives
The usability test of the text-based version of the eFolioMN website by visually impaired
users will provide qualitative and quantitative data addressing eFolioMN’s interest in
understanding the visually impaired users’ experience. We will assess the participants’
experience and satisfaction by having them perform various tasks and navigate to specific
areas of the site. The results should help to inform the creative force behind eFolioMN of
areas that need consideration to improve the overall user experience as it pertains to
reasonable, realistic, and logical assumptions about how, why and what visually impaired
users would be trying to do while accessing eFolioMN.
We will investigate the following aspects of the eFolioMN site:
 The general feeling/layout of the site
 Logging in and site navigation:
- Any issues with logging in?
- Are there obvious waysto expedite navigation to make the process as efficientas
possible?
 Basic/Advanced search:
- Does it meet expectations?
- Does it work well?
- Is it possible to accomplish goals using this tool?
 Hot Keys:
- Do they workas expected?
- Are there any concerns?
 Language/Terminology:
- Does the wording help users accomplish their tasks?
 Help Menus:
- Can a user use the help menu to effectively answerquestions?
 Perceivedreliability of the site:
- Do users upload files and use them to display their portfolio?
 User Goals:
- What tasks wouldusers want to achieve or be likely to do?
 Satisfaction:

- Whichaspects do users like/not like?
These questions were devised using information supplied by our client and our assessment
of potential usability problems resulting from our heuristic evaluation of the site.
User Profile
To recruit prospective test participants who represent the site’s potential users, we
identified two primary user groups and created a description for each group. The two
personas that were identified were mobile device users and visually impaired users.
Personas for the two user profiles were submitted previously.
This usability test will focus on visually impaired users. It was determined that this group
would benefit the most from improvements to the text-based version of the eFolioMN site.
We will look to recruit an equal mix of male and female participants.
A general description of the user characteristics is as follows:
 Must have and use JAWS screen reading softwareprogram
 Must have experience interacting with websites – specifically uploading files, filling out
forms, etc.
 Must speak/read/write English
 Must be a Minnesota resident
 Must be at least 21 years old
 Representative education can be from high schooldegree and up
 Must be interested in using an electronic portfolio foremployment or self-marketing
Methodology
Usability testing of the eFolioMN site will be based on concerns identified during group
discussions as well as the findings of our heuristic report. We will test the website with five
participants who meet the screening criteria. Their responses and reactions will be
recorded and documented in a final report.
This usability test will be designed to allow us to obtain mostly qualitative data. The focus
of the evaluation will be task-oriented and directed toward how the user subjectively
responds to the issues previously listed. Participants will be given a pre-test questionnaire
to get information about their computer and website navigational experience and
expectations, followed by scenarios that direct them to perform specific tasks. A post-test
questionnaire has been designed to obtain detailed and specific feedback about eFolioMN.

Preparation
In preparation for the usability testing, we will conduct a walk-through test with a tolerant
participant. This tolerant participant will not necessarily meet all of the screening criteria,
but will allow us to practice our scenarios, moderating skills and the entire process. After
conducting the walk-through, changes will be made accordingly to any necessary aspects of
the Usability Test. We will also be conducting a pilot test under the guidance of Dr. Victoria
Sadler.
Testing
Testing will take place between March 29th and April 6th in a place to be determined.
Testing will likely take place at the Midway campus of Metropolitan State University. We
will travel to necessary places in order to accommodate participants than cannot make it to
Metro State.
Think aloudprotocol
Participants will be encouraged to think aloud while going through the scenarios. They will
be asked to describe what they are thinking, doing and feeling during each of the tasks. This
will allow our team to document their thoughts about and reactions to the eFolioMN
website.
Session length
Each session will last approximately two hours to complete the following components:
 Introduction, consent forms and pre-test Questionnaire – 15 minutes
 Scenarios (possibly including post-test questions after completing each one) – 90 minutes
 Closing including post-test questionnaire and final thoughts or comments – 15 minutes
User Incentive
Each participant will receive a gift card to Target. This will aid the recruiting of participants
and will help to ensure them that it is the website being tested, not the person.
Scenarios
The moderator will read the following before beginning the scenarios:

“Before we begin with the scenarios, I would like you to make any necessary adjustments
to JAWS that you wish. Once you are ready, or if the default settings are ok let me know and
we will begin.”
Scenario 1 – ~7 minutes
You have decided to create an online portfolio of your recent work to aid in your job search.
A friend told you about eFolioMN, so you decide to check out the site for yourself.
(At this time either ask the participant to refresh the page or do it for them so that JAWS
will start reading from the beginning).
1. Take some time to get to know the home page. Give yourfirst impression of the site. Tell us
what youlike and don’t like, and summarize briefly what you think youcan do using the
eFolioMN website.
Now that you’ve checked out the eFolioMN home page, take the necessary steps to create
an account using information provided to you.
1. Create your accountusing the information provided to you.
(If the participant goes to the “Returning Users” section and tries to sign up via that
link, stop them and say “I’m going to stop you here. The page youare on now will
only let yousign up using the mouse-based version of this website. This is an issue
we have already documented, and your actions have helped verify that it is indeed
an issue. Please go back to the home page, and select the link that reads as “Graphic
about equals. Now lookfor the “Switchto text-based version” link and continue.)
Step 3 No Sign up code
Step 4
Professional
Step 5
Yes
Step 6
Email Varies
(for example: etug1@yahoo.com)
Step 7
First name: John or Jane
Last name: Smith
Country: United States
Postal Code: 54321
SKIP PRIMARY USE
Organization: None of the above

SKIP GENDER
Password: usability
Post-Task Questions:
(1)Strongly disagree – (5)Strongly agree
1 2 3 4 5
1. I thought this task was easy to complete.
2. Please comment on what was particularly easy or difficult:
Now that you have an eFolioMN account, let’s begin creating your website.
1. Log in to youreFolioMN account.
2. Give yourfirst impression of this home page/menu.
First you should create a short welcome message for people who visit your site.
1. Create a short welcomemessage. Title should be “WelcomeMessage”. Text should be
“Welcometo my website.”
(If they go to “Add New Content” instead of the “ToDo” list, let them lookaround
briefly,then say: “I’m going to stop you here. Please go back to the previous page
(the Main Menu). What do you think wouldhappen if you clicked“ToDo”? Let’s try
it and find out.”)
2. Add yourwelcome message to the “Home” page in the main column.
1 2 3 4 5

Scenario 5 – ~6.5 minutes
Upload your resume. (File provided: located on Desktop, file is titled “PDF Resume”).
1. Upload your resume to yourcontent.
2. Add yourresume to the “Employment” page in the main column.
1 2 3 4 5
You decide your website is ready to show to others.
1. Make your eFolioMN site public.
(If they find the task in the “ToDo” list, let them see that it is will not work.Say “I’m
going to stop you here. You have come across the directions for how to do this using
the mouse-based version. Your actions have verified forus that this is an issue.
Please go backto the main menu. What do youthink the link “Manage Sites” would
lead to? Let’s try it and find out.”)
(From here on out the correctchoiceis “Properties”.)
1 2 3 4 5

Note: Return to Moderator Script
TestEnvironment/Equipment
Location/Evaluationroom/Facilityinformation
Depending upon our recruitment success and availability of our participants there are two
possibilities for our testing environment:
1. A classroom at Metropolitan State University midway campus.
2. “In the field,” going to the participants’ environment, at a locationthat suits their needs and
availability (p. 111 Barnum).
Equipment
Chairs
Desks
Writing utensils
Notebooks
A PC equipped with JAWS (or a computer the participant brings)
An audio recording device – Sony ICD-AX412 Digital Flash Memory Voice Recorder
Computers to log information after tests
ProjectTimeline
Date Day Time Task
3/12 Monday Morning Recruit and screen participants
3/16 Friday 5:00 PM Conduct walkthrough test with tolerant user
3/28 Wednesday 6:00 PM Conduct Pilot Test with instructor
3/30 –
4/12
TBD TBD Conduct user tests with participants and debrief,
analyze, document findings, and prioritize

findings. Prepare oral presentation of results.
4/22 Sunday Midnight Final Test Report due
4/27 Friday 1:00 PM Oral presentation of results with highlights/clips
in class, to client, and electronic files, final test
hard copy due to instructor and client.
TBD TBD TBD Meet with eFolio/Paul and Avenet to discuss
results in more detail
Team RolesandResponsibilities
Our team would like to allow each member to experience operating in every role. If we
have at least four participants this will be possible. If there are more or less than four,
depending upon the desires of team members, we will decide, as our charter states, by
either majority or allowing the team lead to decide. These are examples of roles that each
team member will perform.
Moderator
Ron is the moderator, and as such handles all interactions with the participant on test day.
If the participant has a technical problem, Ron will pass the participant on to technical
support.
Logger
Kevin will perform logger duties. He will record screen activity and navigation choices the
participant makes.
Observer
Sue Ann will record memorable quotes made both by the participant and JAWS. Sue Ann
will also provide any necessary technical support as needed with JAWS.
Technical support
Shaun will control the audio recording device, open JAWS and set up the website before the
test and keep track of the time taken for each segment of the test.

EvaluationMethods
Quantitative Data
During each test, we will collect both quantitative and qualitative data. The quantitative
data will provide measureable feedback based on users’ responsesto questions asked after
the test. Questionnaires, requiring participants to rank ease or difficulty, will be
administered at the end of the test. The first part of the post-test protocol includes
administering System Usability Scale (SUS) questions.
SUS questions are a set of standardized questions used widely in usability studies. They ask
the user to respond to questions by rating them on a scale of 1 to 5, where 1 is strongly
disagree, and 5 is strongly agree. One example of a SUS question is, “I thought the system
was easy to use”. The full set of SUS questions is included as an appendix to this document.
Our moderator will administer the questions to the participants. The SUS test will be
scored later.
The benefit of using SUS questions is that the average SUS scores for all participants in the
evaluation can provide a baseline for comparison to other website usability studies for the
future studies of the same site.
We will also be timing how long the user spends on each scenario. This information will
help us get a sense of which tasks are more difficult than others and could aid in assigning
severity ratings to issues encountered.
Qualitative Data
We will collect qualitative feedback through observations during each test, including users’
remarks, non-verbal body language, and facial expressions. We will administer a pre-test
questionnaire asking open-ended questions. We will also use a post-test interview at the
end of each test session allowing participants to convey their impression of the eFolioMN
user experience.
The post-test interview will be a brief Q & A session. During the testing session our
observer, logger and moderator will construct questions based upon our participants’
interaction with various scenarios and tasks. Our observers are allowed to write questions
during the test and give them to the moderator to ask the participant.
This combination of quantitative and qualitative data will aid the analysis to be presented
in the final report and presentation.
Pre-test and post-test questionnaires are included as Appendices to this document.

Deliverables
Screening questionnaires completed by March 28th, 2012.
eFolioMN test prototype to team members for scenarios; freeze date on prototype for the
test walkthrough March 16th.
All materials for walkthrough must be reviewed by team and completed by March 23rd.
Walkthrough version of test scenario on March 16th and March 19th.
Revision to materials as needed; finalized by March 23rd.
Pilot March 28th, 2012.
Revision to materials as needed; same day.
Testing Dates: (3/26-4/6 actual days and times to be determined per recruitment)
At the conclusion of this evaluation the delivery of these artifacts will include:
 a formal written report
 an oral presentation
 a meeting withthe eFolio website personnel
If, after reviewing the test and presenting our findings proves more insightful, we plan to
allow eFolio to use this information to make their text-based website more user friendly to
the blind and visually impaired.

Appendix F: Recruitment Email Page 78 of 92
Appendix F: Recruitment Email
Hello,
I’m a graduate student at Metropolitan State University. I’m in a group project for my
Usability Testing for Technical Communication course.
We are conducting a usability evaluation to determine how accessible a text-based version
of a website is to visually impaired/blind users.
Would you be interested in assisting my group with this test? The test will be conducted
between March 29th and April 6th. The test from beginning to end should take about two
hours (at the most) upon arrival. If possible, we would like the test to be conducted at
Metropolitan State’s Midway campus location (1450 Energy Park Drive), but would be
willing to consider other possible locations. Participants will receive a $10.00 Target gift
card for their participation.
Participant requirements:
 Must be a JAWS user
 Must have experience interacting with websites (e.g., uploading files, filling out forms)
 Must be a Minnesota resident
 Must be at least 21 years old
Please let me know if you have questions or if you would like to be a participant. Thank you
in advance for your consideration.
Sue Ann
rodrsu@metrostate.edu

Appendix G: Screening Questionnaire Page 79 of 92
Appendix G: Screening Questionnaire
Usability Test of eFolioMN Website
Test date:
Test time:
Recruiting Goal: 5 participants, plus backups
Special notes for recruiters:
 Equal gender distribution (if possible)
 Must speak/read/write English
 Representative education can be from high school degree and up
 User of JAWS (Job Access with Speech) screen reading software program
 Experience using JAWS interacting with websites
Candidate Name:
Recruited by:
Qualifies? (Yes or No)
Maybe depends on:
Recruitment introduction
Hello, my name is ___________. I’m a graduate student at Metropolitan State University and
I’m in a group project for my Usability for Technical Communication course.
We are conducting a usability test to determine how accessible the text-based version of an
online portfolio building website is to visually impaired users.
Screenerquestions
Because the usability test involves accessing the website with a screen reading software
program, we are looking for people who own JAWS and who have experience interacting
with websites using this program.

1. Do youhave experience interacting with websites using JAWS?
Yes (continue) No (terminate)
2. How would you rate yourself as a JAWSuser (read options)?
 Beginner
 Intermediate
 Advanced
3. Do youknow whatan online portfoliois?
Yes (continue) No (Read, then continue: An online portfolio is a place
to display your education and work history electronically on the internet. Online
portfolios could include resumes, photos, documents or any other content you
would want to show a potential employer or client.)
4. Would youhave any interest in using a free online portfoliotool?
Yes orMaybe(continuetoquestion 5) No (continueto question 4a)
4a. Would you be interested in testing a free online portfolio tool in order to help
improve it for other users?
Yes (continue to question 5) No (terminate)
Demographic relatedQuestions:
5. Record gender from voice,name or physical appearance (depending how recruiting):
Male
Female
6. We would like to recruit candidates from a variety of age groups. Please tell me whichrange
your age falls into.
 Under 21 (terminate)
 [21-25]
 [26-34]
 [35-45]
 [46 and above]

7. Are youa Minnesota Resident?
Yes No
8. What is yourhighest education level?
 High SchoolDegree
 Some College
 Bachelor’s Degree
In what field?
 Graduate Degree or above
In what field?
Thank you! This concludes our questionnaire.
If you are selected for this usability evaluation, you will receive a $10.00 Target gift card for
your participation. If you are selected, what’s the best way to reach you to schedule your
session?
Phone:
E-mail:
Scheduled session day:
Scheduled session time:

Appendix H: Pre-Test Questionnaire Page 82 of 92
Appendix H: Pre-Test Questionnaire
Screener: Thank you for participating in our usability test. Your participation will provide
valuable feedback for our evaluation. Before we start testing, I would like to ask you a few
questions about your experience with the internet.
1. Are youcurrently employed?
Yes No
a. If yes, whattype of workdo you do? Full-time or Part-time?
 Full-time
 Part-time
b. If no, whattype of workare youseeking?
2. Would youconsider uploading a video if it were pertinent to your career goals? (Example:
of a presentation you have given, a skill, a power point presentation)
3. Have you ever been particularly frustrated by a website? Why? Describe that experience.
4. Have you ever been particularly pleased by a website? Why?Describe that experience.
5. Is there any additional information you wouldlike to share before we begin your
participation in this review?

Appendix I: Moderator Script Page 83 of 92
Appendix I: Moderator Script
Introduction
Hi, my name is Ron Heck. I am a graduate student and a member of the group evaluating
the EfolioMN website. I would like to begin by introducing the other team members. This is
Kevin Glenz, Sue Ann Rodriquez and Shaun Bibo. They will be observing and taking notes
but they will not speak, interact or leave the room during the session. We have a work
station prepared for this session, would you like assistance getting to your seat?
We appreciate you helping us understand how the site works for you, and we want you to
be comfortable and relaxed. You’ll notice that I’ll be reading most of the time. This may
seem strange or awkward, but we do this to ensure that we give the same information to
everyone. I will be sitting next to you during this session and guiding you through the
necessary steps.
Audio equipment
There is an audio recording device that will pick up your voice as you speak. We are asking
your permission to record this session, which will help us to go back and review what
happened just in case we miss something in our notes. I will read you this form and let me
know if you have any questions or concerns.
(Read the Audio Recording Consent form to the participant and have them sign it if they
agree)
Test introduction
I’ll explain now why we’ve asked you to come here today: we’re testing the text-based
version of the eFolioMN website to see what it is like for people to use it. We want to learn
from you about what works well and what gives you problems with this website, so we
need to know what you honestly think.
As we go along, I’m going to ask you to think aloud, to tell us what’s going through your
mind. You may say something like “I am clicking on…”, “I chose this option because…”, “I
don’t like…”, or “Oh, that’s right where I thought it would be.”
During the test, I’ll read aloud a task for you to perform. Please repeat the task or make
sure that you understand what it is that I am asking you to do, and then begin the task.
Once you’re finished with the task, I will ask you some questions. Then I will tell you your

Appendix I: Moderator Script Page 84 of 92
next task, and we will repeat in this way until all the tasks are concluded. At that time we
will have a closing interview. Feel free to ask if you need to be reminded about the task.
Before we begin the first task, please answer a few questions.
(Complete pre-test questionnaire)
Now we can begin. This is the first task I would like you to complete. Please remember to
think aloud as you give your impression of the website. When you’re finished with the task,
just say, “I’m done” and we will proceed with the next step.
[Note: Return to the Scenarios script.]
AFTER COMPLETION OF SCENARIOS
(After participant has completed the last scenario and associated post-task questionnaires,
the moderator will explain the SUS questions.)
Thank you so much, we are almost done. Please respond to these ten questions. (Read SUS
questions.)
Final post-test questionnaire
The last thing we will ask of you today is to respond to this final questionnaire. Then we’re
going to have a brief interview to allow you to share your thoughts and for us to have a
chance to address anything that we would like you to explain further.
Aftertesting
Thank you very much for coming in today. We really learned a lot from your feedback. Here
is your gift card for taking the time to help us today. We really appreciate it.

Appendix J: Post-Test Questionnaire Page 85 of 92
Appendix J: Post-Test Questionnaire
The post-test questionnaire will be modeled after the System Usability Scale (SUS)
developed by John Brooke.
(1) Strongly disagree – (5) Strongly agree
1 2 3 4 5
1. I think that I wouldlike to use this website to display my
portfolio.
2. I found the website unnecessarily complex.
3. I thought the website was easy to use.
4. I think I would need the support of a technical person to
be able to use this website.
5. I found the various functions in this website to be well
integrated.
6. I thought there was too much inconsistency in this
website.
7. I would imagine that most people wouldlearn to use this
website very quickly.
8. I found the website very cumbersome to use.
9. I felt very confident using this website.
10. I needed to learn a lot of things before I couldcomfortably
interact with this website.
Total Score=00 SUS Score=00x2.5=00
Final Question:
Do you have any final thoughts or comments about your experience with this website
before we conclude?

Appendix K: Audio Recording Consent Forms Page 86 of 92
Appendix K: Audio Recording Consent Forms
I hereby give my permission to have my voice recorded as part of my participation in a
usability evaluation of eFolioMN.com conducted by Metro State students as part of a class
project in Writ 574/574G. I understand that if for any reason I do not want to continue I
can leave at any time during this session.
I understand and consent to the use and release of the recordings to the Writ 574/574G
Team and to eFolioMN representatives. I further understand that the recordings and any
highlights from the recordings may be used within presentations to the eFolioMN staff,
during classroom presentations, and as part of eFolioMN’s improvements of the website. I
relinquish any rights to the recordings and understand that the recordings may be used for
the purposes described in this release form without further permission.
Please sign both copies of this form. Keep one for your records and return the other to the
moderator. If you have questions about this project, you may contact Dr. Victoria Sadler
(Victoria.sadler@metrostate.edu).
Print
name
Signature
Date

eFolioMinnesota Text-Based Usability Test Findings and Analysis Report

More Related Content

Viewers also liked

Similar to eFolioMinnesota Text-Based Usability Test Findings and Analysis Report

eFolioMinnesota Text-Based Usability Test Findings and Analysis Report