May 17, 2017 presentation at AccessU: The best way to learn how your website or app will work for people with disabilities is to observe them using it. Usability testing let users try out your product so you can learn what works for them for usable accessibility. You will learn how to adapt a usability test plan for different assistive technology and different disabilities. Best of all, you’ll practice running a usability test with users with disabilities.

1. 1
Usability Testing with People with Disabilities
Kate Walser
CX Insights
kwalser@cxinsights.com • @kwalser
Creative Commons – Some Rights
Reserved

2. 2
» 19 years in usability, accessibility and
user experience design
» Member of TEITAC, or Section 508 /
Section 255 refresh committee
» Principal consultant at CX Insights, the
user experience division of Tritus
Technologies, Inc.
» Originally from upstate NY, now call
Fairfax, VA (near DC) home
Creative Commons – Some Rights Reserved
Kate Walser
kate@cxinsights.com
571.281.2626
2

3. Accessibility in a Nutshell
Creative Commons – Some Rights Reserved 3

4. 4
ACCESSIBLE
3 Key Principles
It’s accessible if I can…
1. Get there
2. Know where I am
3. Know what I can do
- Jim Thatcher
Creative Commons – Some Rights Reserved 4

5. Demo – JAWS
Creative Commons – Some Rights Reserved 5

6. Navigation
Megamenus, Repetitive menus, Hover / Mouse-only access, Clear focus, Tap targets
Creative Commons – Some Rights
Reserved
6

7. Creative Commons – Some Rights
Reserved
Forms
Cues, Formats, Feedback
7

8. Dynamic Areas
Autocompletes, Data-dependent forms, Pop-ups, Surveys, “How can I help” chat
Creative Commons – Some Rights
Reserved
8

9. Multimedia & Visual Elements
Maps, Charts, Audio-video
Creative Commons – Some Rights
Reserved
9

10. VIDEO
Blind film critic
Play video
Source: https://youtu.be/c0nvdiRdehw
Creative Commons – Some Rights Reserved 10

11. » Usability testing helps us test the expected use –
what we anticipate, design, and build, against
actual use.
» This picture of the converging sidewalks of two
angled streets tells an important story – the
developers built the sidewalks to specification so
users could walk down each street. What they
didn’t anticipate or build was a shortcut between
the two, so pedestrians have made their own path
across the grassy area.
» Our goal is to make things accessible AND usable
for participants.Actual use vs. expected use
Creative Commons – Some Rights
Reserved
11

12. Why we test
» Observe
– Representative users trying site (vs. just asking)
– How easily they meet their goals
– What causes confusion or problems
» Inform design
– Confirm or challenge design assumptions
– Find and fix issues before launching
» Make an impression
– On us – Remember what participants did and said
– On them – Act as a representative of organization, product
Creative Commons – Some Rights Reserved
12

13. WHAT WE LEARN
Behavior & Actions
» “Oh by the way…”’s
» Environmental factors
» Shortcuts, preferences
» Power user habits
Creative Commons – Some Rights Reserved
13

14. USABILITY TESTING WITH PEOPLE WITH DISABILITIES
What’s the user experience like?
» How will a person with disabilities use the product?
– Assistive technology use and settings
– Other resources they use
» How usable is the product by people with disabilities?
– Content, forms without surrounding visual context, proximity
– Navigation
– Truly accessible vs. just meets the standards
» How engaging and effective is your content strategy?
– Images only vs. engaging context + images
– Search engine readiness (headings, link labels)
– Great content flow
Creative Commons – Some Rights Reserved
14

15. USABILITY TESTING WITH PEOPLE WITH DISABILITIES
Make it accessible first
» Usability testing ≠ accessibility audit
» Usability testing requires…
– All elements reachable
– Context, feedback available
– Site / app can be used with assistive technology
– Focus indicator (i.e., remove outline:0 CSS rules)
Creative Commons – Some Rights Reserved
15

16. When to Conduct Usability Testing
Apply UX,
design
principles
Fix a11y
problems
Conduct
usability
testing
Conduct
testing
with PWDs
Creative Commons – Some Rights Reserved
16

17. Planning, Logistics & Moderating
Creative Commons – Some Rights Reserved 17

18. Choosing Participants
Example Component Senses Include users who are…
Website,
Mobile
- Visual interface
- Links, menus
- Forms
- Vision
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Mobility impaired
Multimedia - Visual interface
- Sound
- Way to select
options
- Vision
- Hearing
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Deaf
- Hearing impaired
- Speech challenges
Phone - Speech and audio
feedback
- Hearing
- Speech (or way to
indicate option)
- Deaf
- Hearing impaired
- Speech challenges
Creative Commons – Some Rights Reserved
18

19. 19
Location Options
In-person
» Pros
– Interact directly with user
– See set-up, body language
easily
» Cons
– Time for participant (travel,
session)
– Time to plan, recruit, conduct
– Local participants only
» Considerations
– Who will provide assistive
technologies?
Remote, online
» Pros
– Occurs at user’s convenience
– Makes recruiting easier
– Further geographic reach
– Involves fewer logistics
– Schedule, complete quickly
» Cons
– Requires accessible screen-
sharing / testing tools
– Miss body language
– May not see person’s face
Creative Commons – Some Rights Reserved 19

20. IN-PERSON TESTING
Environment & Setup
» Location becomes even more critical
– Is it reachable by public transportation?
– Is the building itself accessible?
– Are the signs to find the room accessible?
» Room setup matters
– Is there space to navigate with wheelchairs or walking devices? Is there
space for a service animal to lie down near its human?
– Are tables or desks wheelchair accessible?
– If visually impaired, can the person sit close enough to the monitor to
view the screen? Can you mirror the screen on another monitor for the
moderator to view?
– How close are other workstations if screen readers will be used?
Creative Commons – Some Rights Reserved
20

21. IN-PERSON TESTING
Technology
» Will participant bring their device?
– Lets them use and you see their preferred settings
– May wish to bring earphones
» Will participant need device / AT supplied?
– Ask about OS, browser, AT preferences during recruiting
– Leave some time for set-up once participant arrives
Creative Commons – Some Rights Reserved
21

22. 22
REMOTE TESTING
Technology
Moderated
» Set up a web conference
– Use an accessible web
conferencing tool – GoToMeeting,
WebEx, Skype (Premium for
screen share + video)
– Set up phone conference or use
included audio
» Leave time for participant to
download plugins
» Ask about video
Unmoderated
» Use an accessible testing
tool
– E.g., Loop11
Creative Commons – Some Rights Reserved

23. Recruiting
» Allow more time to find participants
– Screener + disabilities
» Talk with community and outreach groups early
– Advocacy groups
– Employment non-profit groups
– Colleges and universities
– Occupational therapists, Vocational counselors
– Meetup, Facebook, Twitter
» Use specialized recruiting services
– Knowbility AccessWorks portal
Creative Commons – Some Rights Reserved
23

24. Scheduling
» Allot longer session times
– AT / device set-up, adjustments
– Listening with screen reader
» Schedule breaks between sessions
– AT can be overwhelming for users
who don’t typically use it
Creative Commons – Some Rights Reserved
24

25. SCHEDULING > CONFIRMING
Give participants a familiar voice
» If possible, confirm the session yourself
» Whether phone or email, introduce yourself
– “Hi, this is Suzy Thompson from the Acme usability testing. I’ll be
your moderator and wanted to give you more info about the
session and see if you have any questions…”
Creative Commons – Some Rights Reserved
25

26. SCHEDULING > CONFIRMING
Details
» Explain what will happen and ask about specifics
– “We usually provide handouts for users to read that describe the
scenario. What’s the best way to provide that to you?”
» Describe the setup, and include accessibility specifics in
your confirmation emails
– “We’re on the 3rd floor. There’s a ramp to the right of the front doors. The
doors automatically open. There’s an elevator bay in the lobby to get to
our floor.”
» Ask if they will bring anything
– Service animal (would they like a rest or water stop for the animal?)
» If food will be served, ask about dietary restrictions
Creative Commons – Some Rights Reserved
26

27. Additional logistical considerations
» Is an area available to walk service animals?
» Will you have pre- / post-session surveys?
Creative Commons – Some Rights Reserved
27

28. 28
Make participants feel valued and important
» Welcome and guide
participants
» Memorize the script
» Learn negotiation tactics
– Maintain an open mind
– Manage challenges that arise
(Handle surprises without disruption)
Photo credit: “No Cavities!” daveparker
http://www.flickr.com/photos/daveparker/
Creative Commons – Some Rights Reserved 28

29. 29
Moderating Change #1
Anticipate AT’s impact on the session
» Practice listening to JAWS
or other AT while someone
is talking
» Know when the AT will
affect how you interact with
the participant
» Expect user to listen first
» Anticipate user AT shortcuts
» Be patient and don’t expect
to understand everything
Creative Commons – Some Rights Reserved 29

30. Moderating Change #2
Provide accessible materials electronically
» If user will bring own device…
– Option A: Email tasks, survey links ahead of time
– Option B: Post the tasks, survey and give user an easy link to
access them
» If you supply the device…
– Place text file with the tasks on desktop
– Add browser shortcut to survey
» Use alternate versions to randomize task order
– Each participant may have different task order in their file
Creative Commons – Some Rights Reserved
30

31. Moderating
Otherwise, little changes
» Act like you normally would
– Treat them like you would any other colleague or friend
– Ask first before assuming they need help
– “See” and “look” are still ok
» Ask user to tell you what the task is after reading
it
» Remind them not to explore (unless that’s the
task)
Creative Commons – Some Rights Reserved
31

32. 32
Observing
Watch for usability problems
» General issues
» Issues made worse by
poor accessibility or
interaction with AT
– Missing information, problems
navigating
– Look for issues caused by the
presentation
» “Surprises”
Creative Commons – Some Rights Reserved 32

33. 33
Observing
Watch for accessibility hints
»
Special settings?
»
Macros, bookmarks, style
sheets?
»
Check whole page /
context first?
Creative Commons – Some Rights Reserved 33
»
Use AT shortcuts?
– Links list, headings
»
If speech recognition,
how do they navigate?
– Grid coordinates, Words

34. Observing
Tools help
» Add web conferencing option
– Participant accepts invite, shares screen
– Start sessions on other machines
» Use AirServer (or other tools) for mobile
sessions to screencast to other device
Creative Commons – Some Rights Reserved
34

35. 35
» Give them time
– Time to listen to AT
– Time to respond
» Clear browser cache between users!
» Remove outline:0
Creative Commons – Some Rights Reserved
Observing
Additional tips
35

36. GROUP EXERCISE #1
Create your test plan
1. Find partners
2. Pick your test site, application
3. Choose 1-2 tasks you want to test
Creative Commons – Some Rights Reserved 36

37. 37
GROUP SESSIONS – 15 minutes each
Practice moderating
Creative Commons – Some Rights
Reserved
37
1. Welcome and thank the participant for coming
2. Describe the task as briefly as possible
3. Ask them about their AT / set-up
4. Run thru the site or app you picked
1. Can you reach everything? Can you reach things if you zoom in?
2. Is the focus indicator outlined?
3. Are there any dynamic elements that will be trouble for the participant?

38. What did we learn?
Creative Commons – Some Rights Reserved 38

39. Wrap-up
1. Usability testing with people with disabilities is not the
same as an accessibility audit.
2. When including people with disabilities, the main
adjustments involve recruiting and logistics.
3. Understanding and prepping with AT will help sessions
go smoothly.
Creative Commons – Some Rights Reserved 39

40. Thank you!
Creative Commons – Some Rights Reserved 40
Kate Walser
kwalser@cxinsights.com

41. Windows / Screen Reader Tips
1. Tab / Shift+Tab to tab through links, buttons
2. Enter key to activate
3. Spacebar to select checkboxes
4. For *accessible* megamenus, try left / right arrow keys
More info
http://www.microsoft.com/enable/products/windows7/
Creative Commons – Some Rights Reserved
41

42. Apple / VoiceOver Tips
1. Command + F5 to activate
2. Ctrl to stop speech
3. Tab / Shift+Tab to tab through links, buttons
4. Enter key to activate
5. Spacebar to select checkboxes
6. For *accessible* megamenus, try left / right arrow keys
7. Ctrl+Option+F2 to hear window summary
8. Ctrl+Option+Command+H to move to next heading
9. Ctrl+Option+Command+J to move to next control
More info
http://help.apple.com/voiceover/vo/en/VOKeysColor_1.html
http://www.applevis.com/guides/voiceover-keyboard-shortcuts-mac-os-x
Creative Commons – Some Rights Reserved
42

43. iPhone VoiceOver Tips
1. Set the Accessibility Shortcut to Voiceover
2. Toggle on / off VoiceOver: Triple tap the Home button
3. Mute or unmute VoiceOver: Three-finger double-tap
More info
http://www.apple.com/accessibility/ios/voiceover/
Creative Commons – Some Rights Reserved
43

44. Android Talkback Tips
1. Press and hold the power button until you hear a sound
or feel a vibration.
2. Release the power button.
3. Touch and hold two fingers until you hear audio
confirmation (about 5 seconds).
More info
https://support.google.com/accessibility/android/answer/6007100?hl=en
Creative Commons – Some Rights Reserved
44

45. TEST PLAN
What do we want to test?
Creative Commons – Some Rights
Reserved
Website / app?
Desktop, mobile, both?
What senses are in play?
(vision, hearing, speech)
Which populations should
we include?
When do we want to
test?
What features, functions
do we want to test?
45

46. TEST PLAN
Where and how will we conduct testing?
Creative Commons – Some Rights
Reserved
In-person or remote?
In-person
What location is
accessible?
Can it be reached
via public transit?
Is there a room with
adequate space?
Do we have AT
available?
Remote
Do we have SW
licenses?
How can we find
participants?
How can we advertise /
promote the testing?
46

47. TEST PLAN
What logistical considerations do we need?
Creative Commons – Some Rights
Reserved
Who will schedule &
confirm participants?
If food will be served, ask
about dietary needs.
Who will greet them?
(Ideally same person)
Can we send any pre-test
survey ahead of time?
How / will we randomize
the tasks?
How will we provide the
tasks to users? (e.g.,
Text file? link to list of
tasks?)
Will we use a survey tool
for the post-test survey?
If remote, have we made
an easy link between the
test and survey?
47

48. TEST PLAN
Do we need anything else for in-person moderating / observing?
Creative Commons – Some Rights
Reserved
Who will moderate the
sessions?
Do we need to record the
sessions?
Have we set up any web
conferencing sessions so
others can observe on
their machines?
Do we have new / clean
earphones available?
48