For users with disabilities visiting your website, accessibility is essential for site and app use. But how usable is your product? Why not ask people who use assistive technologies to use your site and find out? Learn tips for creating a test plan, recruiting, and moderating usability test sessions with people with disabilities.

1. 1
Usability Testing with People with Disabilities
Image: xxxxx
Kate Walser
CX Insights
kwalser@cxinsights.com • @kwalser
Slides: http://www.slideshare.net/kwalser
Creative Commons – Some Rights
Reserved

2. 2
» 18 years in usability, accessibility and
user experience design
» Member of TEITAC, or Section 508 /
Section 255 refresh committee
» Principal consultant at CX Insights, the
user experience division of Tritus
Technologies, Inc.
» Originally from upstate NY, now call
Fairfax, VA (near DC) home
Creative Commons – Some Rights
Reserved
Kate Walser
kate@cxinsights.com
571.281.2626
2

3. SCHEDULE
Learning & Prep
Time Duration Topic
2:15 – 2:25 10 minutes Introductions, Objectives
2:25 – 2:35 10 minutes Accessible & usable
2:35 – 2:50 15 minutes Planning & logistics
2:50 – 3:00 10 minutes GROUP EXERCISE
Plan your test
3:00 – 3:10 10 minutes Moderating & observing
3:10 – 3:20 10 minutes GROUP EXERCISE
Practice
3:20 – 3:30 10 minutes Break
3:30 – 3:40 10 minutes Welcome participants
Creative Commons – Some Rights
Reserved
3

4. SCHEDULE
Applying What You Learn
Time Duration Topic
3:40 – 3:55 15 minutes Session #1
↳ 3:55 – 4:05 10 minutes Debrief, switch
4:05 – 4:20 15 minutes Session #2
↳4:20 – 4:25 5 minutes Debrief, switch
4:25– 4:40 15 minutes Session #3
↳4:40 – 4:45 5 minutes Debrief, switch
4:45 – 5:00 15 minutes Session #4
↳5:00 – 5:05 5 minutes Debrief, switch
5:05 – 5:20 15 minutes Session #5
5:20 – 5:30 10 minutes Wrap-up, Questions, Final thoughts
Creative Commons – Some Rights
Reserved
4

5. Accessibility in a Nutshell
Creative Commons – Some Rights
Reserved
5

6. 6
ACCESSIBLE
3 Key Principles
It’s accessible if I can…
1. Get there
2. Know where I am
3. Know what I can do
- Jim Thatcher
Creative Commons – Some Rights
Reserved
6

7. DEMO
Using a screen reader
Creative Commons – Some Rights
Reserved
7

8. Navigation
Megamenus, Repetitive menus, Hover / Mouse-only access, Clear focus, Tap targets
Creative Commons – Some Rights
Reserved
8

9. Creative Commons – Some Rights
Reserved
Forms
Cues, Formats, Feedback
9

10. Dynamic Areas
Autocompletes, Data-dependent forms, Pop-ups, Surveys, “How can I help” chat
Creative Commons – Some Rights
Reserved
10

11. Multimedia & Visual Elements
Maps, Charts, Audio-video
Creative Commons – Some Rights
Reserved
11

12. VIDEO
Blind film critic
Play video
Source: https://youtu.be/c0nvdiRdehw
Creative Commons – Some Rights
Reserved
12

13. GOAL
Accessible and usable
Creative Commons – Some Rights
Reserved
Photo Credit: Actions vs. Intentions: 20111219_326, by Gerald Fittipaldi, https://flic.kr/p/kvnGo2
13

14. Usability Testing
SUMMED UP
Creative Commons – Some Rights
Reserved
14

15. Why we test
» Observe
– Representative users trying site (vs. just asking)
– How easily they meet their goals
– What causes confusion or problems
» Inform design
– Confirm or challenge design assumptions
– Find and fix issues before launching
» Make an impression
– On us – Remember what participants did and said
– On them – Act as a representative of organization, product
Creative Commons – Some Rights
Reserved
15

16. Expected Use vs. Actual Use
Creative Commons – Some Rights
Reserved
16

17. WHAT WE LEARN
Behavior & Actions
» “Oh by the way…”’s
» Environmental factors
» Shortcuts, preferences
» Power user habits
Creative Commons – Some Rights
Reserved
17

18. USABILITY TESTING WITH PEOPLE WITH DISABILITIES
Make it accessible first
» Usability testing ≠ accessibility audit
» Usability testing requires…
– All elements reachable
– Context, feedback available
– Site / app can be used with assistive technology
Creative Commons – Some Rights
Reserved
18

19. When to Conduct Usability Testing
Apply UX,
design
principles
Fix a11y
problems
Conduct
usability
testing
Conduct
testing
with PWDs
Creative Commons – Some Rights
Reserved
19

20. USABILITY TESTING WITH PEOPLE WITH DISABILITIES
What’s the user experience like?
» How will a person with disabilities use the product?
– Assistive technology use and settings
– Other resources they use
» How usable is the product by people with disabilities?
– Content, forms without surrounding visual context, proximity
– Navigation
– Truly accessible vs. just meets the standards
» How engaging and effective is your content strategy?
– Images only vs. engaging context + images
– Search engine readiness (headings, link labels)
– Great content flow
Creative Commons – Some Rights
Reserved
20

21. USABILITY TESTING WITH PEOPLE WITH DISABILITIES
What We Can Learn
» Different methods for finding information
» Whether content, navigation are usable (not just
accessible)
» Adequate context for completing tasks
Creative Commons – Some Rights
Reserved
21

22. Planning & Logistics
2:35 – 2:50
Creative Commons – Some Rights
Reserved
22

23. Choosing Participants
Example Component Senses Include users who are…
Website,
Mobile
- Visual interface
- Links, menus
- Forms
- Vision
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Mobility impaired
Multimedia - Visual interface
- Sound
- Way to select
options
- Vision
- Hearing
- Way to navigate
(mobility, speech)
- Blind
- Vision impaired
- Deaf
- Hearing impaired
- Speech challenges
Phone - Speech and audio
feedback
- Hearing
- Speech (or way to
indicate option)
- Deaf
- Hearing impaired
- Speech challenges
Creative Commons – Some Rights
Reserved
23

24. 24
Location Options
In-person
» Pros
– Meet user & interact directly
– Can see body language
– Observe user’s set-up more
easily
» Cons
– Takes more time for participant
(travel + session), recruiting
– Requires accessible space
– Need more time to arrange,
complete
– Include local participants only
Remote, online
» Pros
– Occurs at user’s convenience
– Makes recruiting easier
– Invite users from further
– Involves fewer logistics
– Schedule, complete quickly
» Cons
– Requires accessible tools
– Miss body language
– May not see person’s face
Creative Commons – Some Rights
Reserved
24

25. IN-PERSON TESTING
Environment & Setup
» Location becomes even more critical
– Is it reachable by public transportation?
– Is the building itself accessible?
– Are the signs to find the room accessible?
» Room setup matters
– Is there space to navigate with wheelchairs or walking devices? Is there
space for a service animal to lie down near its human?
– Are tables or desks wheelchair accessible?
– If visually impaired, can the person sit close enough to the monitor to
view the screen? Can you mirror the screen on another monitor for the
moderator to view?
– How close are other workstations if screen readers will be used?
Creative Commons – Some Rights
Reserved
25

26. IN-PERSON TESTING
Technology
» Will participant bring their device?
– Lets them use and you see their preferred settings
– May wish to bring earphones
» Will participant need device / AT supplied?
– Ask about OS, browser, AT preferences during recruiting
– Leave some time for set-up once participant arrives
Creative Commons – Some Rights
Reserved
26

27. 27
REMOTE TESTING
Technology
Moderated
» Set up a web conference
– Use an accessible web
conferencing tool – GoToMeeting,
WebEx, Skype (Premium for
screen share + video)
– Set up phone conference or use
included audio
» Leave time for participant to
download plugins
» Ask about video
Unmoderated
» Use an accessible testing
tool
– E.g., Loop11
Creative Commons – Some Rights
Reserved
27

28. Recruiting
» Allow more time to find participants
– Screener + disabilities
» Look for community groups and ask
– Meetup, Facebook, Twitter
» Use specialized recruiting services
– Knowbility AccessWorks portal
» Work with outreach groups
– Advocacy groups
– Employment non-profit groups
– Colleges and universities
Creative Commons – Some Rights
Reserved
28

29. Scheduling
» Allot longer session times
– AT / device set-up, adjustments
– Listening with screen reader
» Schedule breaks between sessions
– AT can be overwhelming for users
who don’t typically use it
Creative Commons – Some Rights
Reserved
29

30. SCHEDULING > CONFIRMING
Give participants a familiar voice
» If possible, confirm the session yourself
» Whether phone or email, introduce yourself
– “Hi, this is Suzy Thompson from the Acme usability testing. I’ll be
your moderator and wanted to give you more info about the
session and see if you have any questions…”
Creative Commons – Some Rights
Reserved
30

31. SCHEDULING > CONFIRMING
Details
» Explain what will happen and ask about specifics
– “We usually provide handouts for users to read that describe the
scenario. What’s the best way to provide that to you?”
» Describe the setup, and include accessibility specifics in
your confirmation emails
– “We’re on the 3rd floor. There’s a ramp to the right of the front doors. The
doors automatically open. There’s an elevator bay in the lobby to get to
our floor.”
» Ask if they will bring anything
– Service animal (would they like a rest or water stop for the animal?)
» If food will be served, ask about dietary restrictions
Creative Commons – Some Rights
Reserved
31

32. Additional logistical considerations
» Is an area available to walk service animals?
» Will you have pre- / post-session surveys?
Creative Commons – Some Rights
Reserved
32

33. GROUP EXERCISE #1 – 2:50 – 3:00
Create your test plan
1. Find partners
2. Pick your test site, application
3. Choose 1-2 tasks you want to test
Creative Commons – Some Rights
Reserved
33

34. Moderating & Observing
3:00 – 3:10
Creative Commons – Some Rights
Reserved
34

35. 35
Make participants feel valued and important
» Welcome and guide
participants
» Memorize the script
» Learn negotiation tactics
– Maintain an open mind
– Manage challenges that arise
(Handle surprises without disruption)
Photo credit: “No Cavities!” daveparker
http://www.flickr.com/photos/daveparker/
Creative Commons – Some Rights
Reserved
35

36. 36
Moderating Change #1
Anticipate AT’s impact on the session
» Practice listening to JAWS
or other AT while someone
is talking
» Know when the AT will
affect how you interact with
the participant
» Expect user to listen first
» Anticipate user AT shortcuts
» Be patient and don’t expect
to understand everything
Creative Commons – Some Rights
Reserved
36

37. Moderating Change #2
Provide accessible materials electronically
» If user will bring own device…
– Option A: Email tasks, survey links ahead of time
– Option B: Post the tasks, survey and give user an easy link to
access them
» If you supply the device…
– Place text file with the tasks on desktop
– Add browser shortcut to survey
» Use alternate versions to randomize task order
– Each participant may have different task order in their file
Creative Commons – Some Rights
Reserved
37

38. Moderating
Otherwise, little changes
» Act like you normally would
– Treat them like you would any other colleague or friend
– Ask first before assuming they need help
– “See” and “look” are still ok
» Ask user to tell you what the task is after reading
it
» Remind them not to explore (unless that’s the
task)
Creative Commons – Some Rights
Reserved
38

39. 39
Observing
Watch for usability problems
» General issues
» Issues made worse by
poor accessibility or
interaction with AT
– Missing information, problems
navigating
– Look for issues caused by the
presentation
» “Surprises”
Creative Commons – Some Rights
Reserved
39

40. Observing
Tools help
» Add web conferencing option
– Participant accepts invite, shares screen
– Start sessions on other machines
» Use AirServer (or other tools) for mobile
sessions to screencast to other device
Creative Commons – Some Rights
Reserved
40

41. 41
» Give them time
– Time to listen to AT
– Time to respond
» Clear browser cache between users!
» Remove outline:0
Creative Commons – Some Rights
Reserved
Observing
Additional tips
41

42. GROUP EXERCISE #2 – 3:10 – 3:20
Practice moderating
1. Welcome and thank the participant for coming
2. Describe the task as briefly as possible
3. Ask them about their AT / set-up
4. Run thru the site or app you picked
1. Can you reach everything? Can you reach things if you zoom in?
2. Is the focus indicator outlined?
3. Are there any dynamic elements that will be trouble for the
participant?
Creative Commons – Some Rights
Reserved
42

43. Windows / Screen Reader Tips
1. Tab / Shift+Tab to tab through links, buttons
2. Enter key to activate
3. Spacebar to select checkboxes
4. For *accessible* megamenus, try left / right arrow keys
More info
http://www.microsoft.com/enable/products/windows7/
Creative Commons – Some Rights
Reserved
43

44. Apple / VoiceOver Tips
1. Command + F5 to activate
2. Ctrl to stop speech
3. Tab / Shift+Tab to tab through links, buttons
4. Enter key to activate
5. Spacebar to select checkboxes
6. For *accessible* megamenus, try left / right arrow keys
7. Ctrl+Option+F2 to hear window summary
8. Ctrl+Option+Command+H to move to next heading
9. Ctrl+Option+Command+J to move to next control
More info
http://help.apple.com/voiceover/vo/en/VOKeysColor_1.html
http://www.applevis.com/guides/voiceover-keyboard-shortcuts-mac-os-x
Creative Commons – Some Rights
Reserved
44

45. iPhone VoiceOver Tips
1. Set the Accessibility Shortcut to Voiceover
2. Toggle on / off VoiceOver: Triple tap the Home button
3. Mute or unmute VoiceOver: Three-finger double-tap
More info
http://www.apple.com/accessibility/ios/voiceover/
Creative Commons – Some Rights
Reserved
45

46. Android Talkback Tips
1. Press and hold the power button until you hear a sound
or feel a vibration.
2. Release the power button.
3. Touch and hold two fingers until you hear audio
confirmation (about 5 seconds).
More info
https://support.google.com/accessibility/android/answer/6007100?hl=en
Creative Commons – Some Rights
Reserved
46

47. TEST PLAN
What do we want to test?
Creative Commons – Some Rights
Reserved
Website / app?
Desktop, mobile, both?
What senses are in play?
(vision, hearing, speech)
Which populations should
we include?
When do we want to
test?
What features, functions
do we want to test?
47

48. TEST PLAN
Where and how will we conduct testing?
Creative Commons – Some Rights
Reserved
In-person or remote?
In-person
What location is
accessible?
Can it be reached
via public transit?
Is there a room with
adequate space?
Do we have AT
available?
Remote
Do we have SW
licenses?
How can we find
participants?
How can we advertise /
promote the testing?
48

49. TEST PLAN
What logistical considerations do we need?
Creative Commons – Some Rights
Reserved
Who will schedule &
confirm participants?
If food will be served, ask
about dietary needs.
Who will greet them?
(Ideally same person)
Can we send any pre-test
survey ahead of time?
How / will we randomize
the tasks?
How will we provide the
tasks to users? (e.g.,
Text file? link to list of
tasks?)
Will we use a survey tool
for the post-test survey?
If remote, have we made
an easy link between the
test and survey?
49

50. TEST PLAN
Do we need anything else for in-person moderating / observing?
Creative Commons – Some Rights
Reserved
Who will moderate the
sessions?
Do we need to record the
sessions?
Have we set up any web
conferencing sessions so
others can observe on
their machines?
Do we have new / clean
earphones available?
50

51. Session 1 – 3:40 – 3:55
Creative Commons – Some Rights
Reserved
Product tested
Participant
User device, OS
AT used
Special settings?
Macros, bookmarks, style sheets?
Check whole page /
context first?
Use AT shortcuts?
Links list, headings
If speech recognition,
how do they navigate?
Grid coordinates, Words
51

52. SESSION #1 DEBRIEF – 3:55 – 4:05
What did we learn?
Creative Commons – Some Rights
Reserved
52

53. Session 2 – 4:05 – 4:20
Creative Commons – Some Rights
Reserved
Product tested
Participant
User device, OS
AT used
Special settings?
Macros, bookmarks, style sheets?
Check whole page /
context first?
Use AT shortcuts?
Links list, headings
If speech recognition,
how do they navigate?
Grid coordinates, Words
53

54. SESSION #2 DEBRIEF – 4:20 – 4:25
What did we learn?
Creative Commons – Some Rights
Reserved
54

55. Session 3 – 4:25 – 4:40
Creative Commons – Some Rights
Reserved
Product tested
Participant
User device, OS
AT used
Special settings?
Macros, bookmarks, style sheets?
Check whole page /
context first?
Use AT shortcuts?
Links list, headings
If speech recognition,
how do they navigate?
Grid coordinates, Words
55

56. SESSION #3 DEBRIEF – 4:40 – 4:45
What did we learn?
Creative Commons – Some Rights
Reserved
56

57. Session 4 – 4:45 – 5:00
Creative Commons – Some Rights
Reserved
Product tested
Participant
User device, OS
AT used
Special settings?
Macros, bookmarks, style sheets?
Check whole page /
context first?
Use AT shortcuts?
Links list, headings
If speech recognition,
how do they navigate?
Grid coordinates, Words
57

58. SESSION #4 DEBRIEF – 5:00 – 5:05
What did we learn?
Creative Commons – Some Rights
Reserved
58

59. Session 4 – 5:05 – 5:20
Creative Commons – Some Rights
Reserved
Product tested
Participant
User device, OS
AT used
Special settings?
Macros, bookmarks, style sheets?
Check whole page /
context first?
Use AT shortcuts?
Links list, headings
If speech recognition,
how do they navigate?
Grid coordinates, Words
59

60. SESSION #5 DEBRIEF – 5:20 – 5:30
What did we learn?
Creative Commons – Some Rights
Reserved
60
THANK YOU PARTICIPANTS!

61. Wrap-up
1. Usability testing with people with disabilities is not the
same as an accessibility audit.
2. When including people with disabilities, the main
adjustments involve recruiting and logistics.
3. Understanding and prepping with AT will help sessions
go smoothly.
Creative Commons – Some Rights
Reserved
61

62. Thank you!
Creative Commons – Some Rights
Reserved
62
Kate Walser
kate@cxinsights.com
571.281.2626