First presented on the Technica11y.org podcast in May, 2019, Dr. Michele A. Williams, Senior UX Researcher - Accessibility at Pearson, explains what is needed to conduct inclusive user research: Know the Basics, Accessible Artifacts, and Accessible Testing.

1. Conducting
Accessibility
User Research:
What’s Really
Needed
Michele A. Williams, PhD
Senior UX Researcher -
Accessibility, Pearson
Technica11y.org | May 1, 2019

2. WHAT’S REALLY NEEDED:
1. Knowing the basics
2. Accessible artifacts
3. Accessible testing
2

3. 1.
KNOWING THE
BASICS

4. Understand disability
Learn the diversity and manifestations of disability
Photo Credit: International Labour Organization
4

5. Know the laws & standards
Web & Applications
Section 508
Web Content
Accessibility
Guidelines (WCAG)
Hardware
Section 508
Americans with
Disabilities Act
(ADA)
Voice
Section 508
21st Century
Communications and
Video Accessibility Act
(CVAA)
Note: this information is primarily U.S.-based 5

6. Get immersed
Assistive
Technology
Centers
Government-
sponsored
statewide centers
focused on
matching clients to
assistive
technology
Disability
Organizations
Disability-specific
national & local
organizations
Local library
Meetup.com
Email lists/groups
Industry
Professionals
Examples:
Orientation & Mobility
Specialists
Assistive Technology
Specialists
Rehabilitation
Specialists
6

7. Know the current state
Related Work
Search for
accessibility
publications such as
ACM’s ACCESS
or higher education’s
AHEAD
The Competition
Note accessibility
from similar or
competitor
products
Your Baseline
Employ an
accessibility
consulting firm to
perform a formal
audit of your product
7

8. User testing
is not an audit
You are responsible for testing accessibility,
your participants are testing usability
8

9. 2.
ACCESSIBLE
ARTIFACTS

10. Research occurs at all project
phases
10
Abstract Concrete
Ideas ----- Sketches ----- Prototypes ----- Product
Project
Phases
& Artifacts
What’s needed to conduct successful inclusive research
with these artifacts?

11. Considerations for your testing artifact
To use your artifact, does it require…?
▸ Clear vision
▸ Sharp hearing
▸ Talking
▸ Walking
▸ Staying focused
▸ Grasping objects
If so, start to pivot…
11

12. Ideas – No artifacts to test
This is more concerned with
method which we’ll visit in Section 3
12

13. Sketches – Go beyond “paper”
Web & Application
Leverage accessible
technologies such as
Microsoft Office
Suite to increase
compatibility with
assistive technology
Hardware
Use inexpensive 3D
objects that users
can manipulate and
see such as Play-
Doh, Styrofoam,
blocks, and 3D prints
Voice
Leverage free text-
to-speech
technologies to build
out your idea
13

14. Prototypes – Getting close to
real
Web & Application
Start creating “throw
away” code
Be careful of
“prototyping” tools;
they won’t yield
accessible products
Hardware
Use the closest
equivalent, e.g.,
GoPro camera, iPad,
computer monitor
Voice
Leverage devices
that create realistic
voices
Use existing
accessible
telecommunication
devices
14

15. Wizard of Oz
“Fake it until you make it”
testing method
Users think the system is
real, researchers create the
illusion
15

16. WOZ Example 1:
Telephone IVR
User dials a desk phone;
designer acts out script
Provides interface
validation before
production
16

17. WOZ Example 2:
Alexa Skill
User talks to speaker;
developer plays the files
Can determine user
grammar and keep study
under control
17

18. Product – The real, full artifact
Web & Application
WCAG or Section
508 compliant
Hardware
ADA compliant
Voice
CVAA compliant
Caveat: If your system is not accessible, work with other
methods and an audit until you can get there; be cautious
of bringing in users to test broken systems
18

19. What the user is testing
needs to be accessible
Take the extra steps to seek out (or advocate
for) accessible prototyping tools
19

20. 3.
ACCESSIBLE
TESTING

21. Research
Methods
Special considerations
for certain methods

22. Methods: Biometric & Card
Sort
Biometric Testing
Ensure equally thorough
alternative collection for
participants who can’t use
biometrics
Ensure equal
representation and
triangulation of data from
biometrics and other
sources
Card Sort & Tree Test
Use an accessible (WCAG-
compliant) tool or leverage
accessible drag-and-drop
examples to build a system
22

23. Methods: Diary & First Click
Diary Study
Use multiple collection
methods – Word doc,
email, voicemail, video, etc.
Note, blind people can and
do take photos
First Click Test
Needs accessible (WCAG-
compliant) webpage, not
image of page
Must detect keyboard
events and mouse clicks
from assistive technology
such as voice dictation
software
23

24. Methods: Focus Group &
Observations
Focus Group
Ensure this is appropriate
for the participant group
and everyone can
contribute
Consider alternatives such
as live chat or group forum
for those with speech
impairments, delays, or
anxieties
Observations
If following along in public,
note you will be mistaken
for an aide
Consider instrumenting
participant with video rather
than being present
24

25. Methods: Participatory Design
& Online Surveys
Participatory Design
If creating artifact on behalf
of participant, get explicit
confirmation idea was
properly conveyed
Consider leaving artifact
with participant then
interviewing them after time
to reflect
Online Surveys
Use accessible (WCAG-
compliant) survey tool
Minimize use of complex
question types
Carefully word
demographic questions;
not everyone identifies as
having a “disability”
25

26. Methods: Usability Testing
Usability Testing
Be mindful that participants
don’t want to look helpless
Unmoderated testing is
feasible but leaves out
valuable data that can be
educational for teams, e.g.,
recording troubleshooting
techniques and general
assistive technology use
Time on task may not be
applicable for a diverse
population, but may apply if
using an exclusive
population
SUS and Ease of Use
measures may need to be
separate; for instance, a
system may be difficult for
blind users but not others
26
Synopsis of Research Methods from User Zoom

27. A Few Notes About Communication
Use Interpreters if
Needed
Determine if sign language
interpreters are needed; if
so, ask for
recommendations and offer
to pay
Determine if a caregiver
can relay messages from
someone with a speech
impairment
Be Prepared to Type
Determine if typed versus
spoken interactions will be
easier or necessary
27

28. Recruiting
Techniques for
finding the right
participants

29. Two ways to recruit
Relationship Building
Use same resources for
learning accessibility (Slide
6) to get to know the
communities they serve
Downside: May limit pool of
participants
Recruiting Firm
Contract with professionals
or disability organizations
that have a database of
users, e.g., Knowbility’s
Access Works
Downside: May not have a
specific demographic
29

30. Reality of recruiting
Minority Group
Implications
Ideal participants may not
be involved in
organizations or signed up
for research; try reaching
out to key stakeholders,
though this will take more
time
People are still breaking
barriers in some areas
(e.g., Haben Girma)
Use Proxies
Consider using older adults
as proxies for your studies
Use people who meet the
disability requirements and
can sit in for other
requirements; example,
participants who are not
currently enrolled in school
to test college-level course
material
30

31. Logistics
Consider details
from start to finish

32. Paperwork & Payments
Consent Form
Use email or verbal
consent, do not require a
signature
Ensure the form is
readable with assistive
technology
Ensure the wording is not
too legal or hard to
understand
Ensure participants can
give their own consent
Payment System
Ensure compensation
systems (such as gift card
payment systems) are
accessible
Attempt to compensate
logically (that is, no
payments to services
participants can’t use)
32

33. Technical Considerations
Tech Check
For remote testing, have
participants join the remote
system days prior to testing
and simulate computer use
to work out interruptions
Recommend using Zoom
as many user testing tools
will not be accessible
Equipment Needs
Determine if participants
will bring and use their own
equipment or will you
supply it (recommend they
use their own if possible)
33

34. Physical Spaces
Meeting in a Lab
Provide street-to-door
directions
Ensure space is ADA-
compliant & transit-friendly
(or provide compensation
for transportation)
Communicate exact
meeting point
Ensure space will not
cause participant anxiety
Meeting in Public or
Home
Ensure and agree the
public space is conducive
to research
Determine and
communicate boundaries if
conducting study in-home
Compromise: Meet at
common space such as
public library where go for
other meetings
34

35. Analysis &
Reporting
Getting the
message
out

36. Conduct Appropriate Analysis
Interpret Findings
Need to understand how
participants intend to use
the system, especially if
using assistive devices or
technology (AT)
E.g., web issues can be
due to the design, novice
experience with AT,
incorrect code markup, an
AT defect, a browser bug,
or operating system issue
Situate Findings
Participants with disabilities
will likely be a smaller
group in inclusive studies,
especially when
segmented by disability
Determine the weight and
significance to give to
findings and validate them
with additional studies or
related work as appropriate
36

37. Provide Mindful Reporting
Add Disability Details
Include background on
accessibility to level-set
reviewers, including videos
Add applicable standards
to reports (e.g., WCAG)
Ensure the report is
accessible, use it as an
example for others
Report to All
Stakeholders
Findings will relate to
designers, developers,
engineers, testers, product
managers, project
managers, everyone!
The entire team must be
involved in making an
accessible system and thus
share your findings and
experiences with everyone
37

38. Use Research to Help the Team
Research is Education
Use research to educate
the entire team/company
on accessibility through
first-person interactions
Create powerful videos
from the research to help
points resonate with
stakeholders
Directly Connect
Stakeholders
Bring users in for large group
testing (“user nights” concept)
or take teams on field trips to
connect them to users with
disabilities directly
Invest in training to ensure
stakeholders know how to act
on what you’re sharing
38

39. Accessibility user
testing needs
accessible tests
The protocol and facilitation details will
make or break the data collection
39

40. Closing
Thoughts

41. In Summary
▸ Inclusive testing takes an ecosystem:
▹ Designers and Developers must know how to build
accessible systems (e.g., WCAG- compliant)
▹ Researchers must know how to facilitate research
and interpret findings
▹ Testers must know how to test for accessibility
and how users use assistive technology
▹ Product Managers need to prioritize accessibility
▸ Ultimately to be successful, everything the participant
interacts with needs to be accessible: consent form,
location, tools for study, artifacts being tested, payment
41

42. Related Work

43. Additional Resources
▸ Prior Research found under “Publications” on LinkedIn
▹ "Pray before you step out": Describing personal and
situational blind navigation behaviors
▹ Reading, writing arithmetic: Experiences of US students
with disabilities
▹ The question still remains: How do I know you know
accessibility?
▹ What not to wearable: Using participatory workshops to
explore wearable device form factors for blind users
▹ Collaboratively designing assistive technology
▸ User Research with People with Disabilities: What you Need to
Know (Peter McNally; UXPA Boston)
▸ A large user pool for accessibility research with representative
users (Marianne Dee, Vicki Hanson; ACM ASSETS)
▸ Knowbility Access Works participants
43

44. Presentation Template
“Arviragus” from SlidesCarnival
44