Usability testing is an accepted process from the domain of user experience (UX) to gauge how easy to learn or use a product is. Users are asked to work on typical scenarios or tasks on the product with the goal of trying to identify challenges in the design of the user interface, interaction or content. In this paper I will discuss the benefits of usability testing in conjunction with accessibility testing and how they can work together. In particular, I will review some recent usability testing projects where we had people with disabilities participate and demonstrate how these findings can augment and support accessibility testing efforts.

1. Why Usability Testing Should be Part of
your Accessibility Testing Strategy
2018 Boston Accessibility Conference
October 27, 2018
Peter McNally
Email: pmcnally@Bentley.edu
Twitter: @pmcnallyux
User Experience Center
Bentley University
Waltham, MA 02452
USA

2. 2
Table of Contents
Section Slide
Opening 3
Intro to Usability Test 13
Real World Examples 18
Benefits of Usability Testing 24
Recommended Approaches 35
Moving Forward 39

3. Opening
3

4. 4
Question
Can anyone think of any examples of products
or “things” from everyday life that you have
access to, but are hard to use?
4

5. 55
• Is this door accessible?
Doors

6. 6
Doors
6
• Is this door accessible?

7. 77
• Is this door accessible?
• Is this door usable?
Doors

8. 88
• Is this door accessible?
• Is this door usable?
Doors

9. 9
Doors - Video
• Norman doors
• https://www.youtube.com/watch?v=yY96hTb
8WgI
9

10. 10
Doors
• What do doors have to do with usability
testing or accessibility?
• You can access door by using the handle or
pressing the automatic opener button.
• However, as we have seen the door may not
be usable.
10

11. 11
Design of Everyday Things by
Don Norman
11

12. 12
Goal
• Discuss the benefits of usability testing in
conjunction with accessibility testing and
how they can work together.
• Review some recent usability testing projects
where we had people with disabilities
participate.
• Demonstrate how these findings can
augment and support accessibility testing
efforts.
12

13. Introduction to Usability Testing
13

14. 14
Usability Testing Overview
• User Experience (UX) method.
• Ask an individual to try and use
something without instructions.
• Can be performed on anything that
a human has to use.
• including digital products such as a
web site or a mobile app.
14
When the process of refining is complete, the next step is to implement the changes live.

15. 15
Usability Testing Overview
• Most common one used in the UX
industry follows a qualitative
approach:
• 5-12 participants tested over 1 or 2 days.
• Each session lasts 30 to 90 minutes.
• Participants given multiple tasks to
accomplish during the session.
• Tasks types may range between open
ended and specific.
• Typical to identify many common
issues.
• Provide design recommendations
15

16. 16
Usability Testing Tasks
• Examples of usability testing tasks
More specific - Less Open Ended
16
When the process of refining is complete, the next step is to implement the changes live.
Example Product
Type
Example Task
Clothing web site You saw a friend’s daughter
wearing this Summer Green Surf
Floral Swimsuit and are considering
getting one for your daughter.
Please, find it and add it to your
cart

17. 17
Usability Testing Participants
• Each participant conducts a test
session one-on-one with a moderator.
• Participants are asked to think-aloud.
• The goal is to see if people can use the
product “cold” as w/ consumer web
sites
• Participants should have a desire, need
or goal in using the product tested.
• Anyone fitting this definition including
people with disabilities can participate.
17
When the process of refining is complete, the next step is to implement the changes live.

18. Real World Examples
18

19. 19
User Experience Center (UXC)
• Consulting group
• Provides user experience research,
evaluation, and design services
• Goal of understanding and improving the
experiences our clients provide to their
customers.
19

20. UXC Experience
75yrs
Rich & diverse
collective team
experience in design
and UX research
A breadth of
methodologies tailored to
Client’s market needs to
improve customer
experience
Integration with award-
winning graduate
program in Human
Factors in Information
Design
20

21. 21
4 Projects - Participants
• Large well known US companies.
• Requested participants with disabilities.
When the process of refining is complete, the next step is to implement the changes live.
Industry/
Disability
Blind Low
Vision
Color
Blind
Motor Learning/
Cognitive
Total
Banking 4 1 3 2 10
Publishing 5 3 8
Chemical 4 1 1 4 10
Pharmaceutical 1 1 3 4 9
Total 14 6 1 6 10 37

22. 22
Issues Found
• Found both usability and accessibility
issues
• 43% - Accessibility issues = that should have been
found in a thorough accessibility review (WCAG 2.0).
• 57% - Usability issues = not accessibility related, but
have a negative impact on the experience.
• Most interesting are usability issues
that are more likely to be found in
context of usability testing with people
with disabilitiesWhen the process of refining is complete, the next step is to implement the changes live.

23. 23
Issues Found
When the process of refining is complete, the next step is to implement the changes live.
Issue Category Accessibility Usability Total
Content/readability/Page Layout 12 26 38
Search 21 11 32
Forms/interactive pages 16 10 26
Process/Understanding Concepts 0 19 19
Navigation/Menus, Orientation 7 10 17
Registration/Log in/Captcha 5 7 12
Error handling/Required fields 5 4 9
General/Other 5 3 8
Information Architecture/Site
Structure
0 4 4
Total 71 (43%) 94(57%) 165

24. Benefits of Usability Testing
24

25. 25
Benefits of Usability Testing
• Usability Testing tasks can focus on
the end-to-end process (big picture)
• Focused on user goals
• May or may not line up with accessibility
issues.
• Accessible does not necessarily mean
usable.

26. 26
Focusing Resources
• Focusing design and development
resources.
• Ideally, all accessibility should be
fixed.
• Prioritize resources around what blocks
users from accomplishing their goals.
26
When the process of refining is complete, the next step is to implement the changes live.

27. 27
Context of Use
• Discovery of problems in context of
use.
• Usability testing can identify problems
that may not be identified during
accessibility testing.
• A real user may do things very
differently than in any test case.
• Talking to users may give you insights
into their motivation and allow you to
ask “why” questions.
27
When the process of refining is complete, the next step is to implement the changes live.

28. 28
Context of Use – Example
• Chemical Company Job Description
• Some participants using screen readers
missed some the job description
because when they heard the “About
<Company>” they assumed there
were at the end of the job description
and did not continue on
• Therefore missing important information
about company benefits
28
When the process of refining is complete, the next step is to implement the changes live.

29. 29
Context of Use – Example
• Chemical Company Job Description
29

30. 30
Context of Use – Example
• Chemical Company Job Description.
• Video clip with Joe.
• Quote:
• “Right now it’s a big long list and it’s pretty extensive. … You
don’t know if you’re missing part of the content…right now
once you reach Qualifications, and you hit H, it throws you
back at the top and you don’t know what you’re missing.”
30
When the process of refining is complete, the next step is to implement the changes live.

31. 31
Learning/Cognitive Disabilities
• Many issues that are considered
usability related deal with learning and
cognition.
• Present more challenges for individuals
with learning and cognitive disabilities.
• Accessibility guidelines such WCAG 2.0
focus more on code based issues.
• Design guidelines such as Jacob
Nielsen’s usability heuristics and from
the Center for Plain Language are a
good start
31

32. 32
Learning/Cognitive Disabilities
• Nielsen’s Heuristics:
• Visibility of system status
• The system should always keep users
informed about what is going on, through
appropriate feedback within reasonable time.
• Recognition rather than recall
• Minimize the user's memory load by making
objects, actions, and options visible. The user
should not have to remember information from
one part of the dialogue to another. Instructions
for use of the system should be visible or easily
retrievable whenever appropriate.
32

33. 33
Getting the Word Out
• Having stakeholders (clients,
developers, managers, etc.) watch
usability testing sessions puts a face
on accessibility for some who were not
aware (or didn’t care).
• Inviting developers to watch usability
testing can help them understand the
importance of their impact for
everyone.
33

34. 34
Accessibility Program Jump Start
• Provides a good way to “jump start” an
accessibility program.
• A “kick in the pants”.
34
When the process of refining is complete, the next step is to implement the changes live.

35. Recommended Approaches
35

36. 36
Recommended Approaches
• Conduct accessibility reviews by using
automatic and manual processes.
• After the “big” issues resolved conduct
usability testing with individuals with
disabilities.
• Can test in office, conference room (or
specialized lab (but not necessary)).
• Best if participants can bring in their
own AT to usability test.
• Tools such as Zoom offer promise of
remote usability testing with screen
reader users.
36
When the process of refining is complete, the next step is to implement the changes live.

37. 37
Recommended Approaches
• Approaches to recruiting participants
• Don’t test with all experts
• Participants should match your
user/customer profile – domain
knowledge.
• Ideally part of your standard UX
approach.
• Work with your UX team or local
disability organization.
• Range of disabilities: visual, learning,
motor, etc. Don’t forget older users…
37

38. 38
Recommended Approaches
• Steve Krug –
Rocket Surgery
Made Easy
• http://www.sen
sible.com/rsme
.html
38
When the process of refining is complete, the next step is to implement the changes live.

39. Moving Forward
39

40. 40
Moving Forward
• Typical to usability test after
accessibility test, however…
• Once coding starts UX issues are
baked in. Expensive to change.
• Accessibility should be integrated into
entire design and development
process.
• Developing user personas and reviewing
wireframes, e.g., asking how will
someone only using the keyboard or a
screen reader to access the primary
navigation menu, etc.
40
When the process of refining is complete, the next step is to implement the changes live.

41. 41
Moving Forward
• Sarah Horton
and Whitney
Quesenbery –
A Web for
Everyone.
• http://rosenfel
dmedia.com/bo
oks/a-web-for-
everyone/
41
When the process of refining is complete, the next step is to implement the changes live.

42. 42
Moving Forward
• UX Personas
with
individuals
with
disabilities
42
When the process of refining is complete, the next step is to implement the changes live.

43. 43
Your Experience
• How do you incorporate usability
testing along with accessibility testing?
• How do you conduct user research
with people with disabilities during the
design process (before coding)?
• Can you share some of your best
practice or recommendations?
43
When the process of refining is complete, the next step is to implement the changes live.

44. Thank You!
Questions?
Peter McNally
Email: pmcnally@Bentley.edu
Twitter: @pmcnallyux
User Experience Center
Bentley University
Waltham, MA 02452
USA
Questions?
Peter McNally
Email: pmcnally@Bentley.edu
Twitter: @pmcnallyux
User Experience Center
Bentley University
Waltham, MA 02452
USA
44