Data Con LA 2020 Description A in-depth discussion of vA11y, VMware's open-source effort to make machine learning-based accessibility testing available to everyone for free. 30 % of accessibility tests are automated. But of the 70 % that are manual, almost half of them could potentially be automated through NLP, image comparison and pattern recognition. Here is what VMware is doing to make the accessibility testing split 50/50 by the end of 2021, and releasing that work into open source. - review of existing accessibility automation - review of what is manual - highlighting 26 currently manual tests and how VMware is planning on automating them - open source This talk will be captioned Speaker Sheri Byrne-Haber, VMware, Head Of Accessibility

1. ©2020 VMware, Inc.
Automated Accessibility
Machine Learning
SHERI BYRNE-HABER
Accessibility Architect, Staff II

2. ©2020 VMware, Inc.
Agenda
2
Introduction
AGENDA
Next StepsvA11yCurrent State

3. 3
©2020 VMware, Inc.
Introduction

4. ©2020 VMware, Inc. 4
Machine Learning - Definition
Definition by Tom Mitchell (1998):
Machine Learning is the study of algorithms that
• improve their performance P
• at some task T
• with experience E.

5. ©2020 VMware, Inc. 5
Natural Language Processing
Dog Breeds
Dachshund Pass
Great Dane Pass
Samoyed Pass
Petunia Fail
NLP is the automatic manipulation like speech and text by software.

6. ©2020 VMware, Inc. 6
Image Recognition
The ability of
software
to identify
objects, places,
people, writing
and actions in
images
Source: Imgur
Dog
Bagel
Dog
Bagel

7. ©2020 VMware, Inc. 7
Pattern Recognition
The automated
recognition
of patterns and
regularities
in data
Bagels don’t have eyes, a
tail, or noses
Dogs don’t have sesame
seeds

8. ©2020 VMware, Inc.
In the beginning, all the data is unstructured …
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Supervised Machine Learning – You start with unstructured data

9. ©2020 VMware, Inc.
Data Data DataData
Good Data Model Bad Data Model
Data Data
A human manually sorts the data, which builds a model

10. ©2020 VMware, Inc.
Pass
Then you test unknown data against the models you have built
Fail
Good Model Bad Model
Unk.
Data
Unk.
Data
Unk.
Data
Unk.
Data
Unk.
Data
Unk.
Data

11. ©2020 VMware, Inc. 11
Pivoting to Disability
Disability is the largest minority group impacting more than
1.3 billion people globally.
• Mobility & dexterity
• Hearing
• Other
• Vision
• Neurological & cognitive

12. ©2020 VMware, Inc. 12
25% of disabilities are visible.
75% of disabilities are visible.
• ADHD
• Asthma
• Cancer
• Chronic Fatigue Syndrome
• Chronic Pain
• Cystic Fibrosis
• Diabetes
• Epilepsy
• HIV/AIDS
• Learning Disabilities
• Mental Health
• Traumatic Brain Injury
• Autism
• Down Syndrome
• Fetal Alcohol Syndrome
• Tourette Syndrome
• Amputations
• Paralysis
• Cerebral Palsy
• Muscular Dystrophy (MD)
• Multiple Sclerosis (MS)
Resource: World Health Organization

13. 13
©2020 VMware, Inc.
Automated Accessibility
Testing – Current State

14. ©2020 VMware, Inc. 14
Automated Accessibility Testing – Current State
1. Code Analysis
2. Framework Dependent
3. Open Source for unlimited commercial use – unavailable
• No live data
• Only 30 % of tests can be executed automatically

15. 15
©2020 VMware, Inc.
vA11y

16. ©2020 VMware, Inc. 16
How vA11y came into being
• Started as an idea when I realized I had way more testing to do than I had
time for
• Went into a contest at work where it was chosen as the winning idea
• Was accepted into our experimental development program as a proof of
concept
• When debugging the code we went from 1 site to 30 sites to 300 sites – and
found a serious bug in the home page of a computer company known for its
public commitment to accessibility.
• VMware partners are clamoring for it
• Has been promoted to experimental labs full track – 3 updates scheduled in
2021

17. ©2020 VMware, Inc. 17
What’s in vA11y Proof of Concept
• Automatic detection of captions and subtitles in videos
• Automatic detection of transcripts for podcasts
• Automatic detection of the presence of keyboard focus indicators
• Automatic measurement if a keyboard focus indicator meets the thickness
and contrast requirements

18. ©2020 VMware, Inc. 18
Planned 2021 vA11y additions
1. 2.4.4 - Link consider ARIA overrides
2. 4.1.2 - name role value look for announcements associated with state changes
3. 1.3.1 - Required fields look for legend and marker
4. 2.4.1 - Check skip link behavior not just for presence
5. 1.4.4 / 1.4.10 - Is the web page responsive
6. 1.3.5 - If the page contains a form, is autocomplete available?
7. 1.4.13 - hover cancelation
8. 2.1.5 - Form and session timeout
9. 2.1.1 - are there mouse only events or ARIA hidden objects

19. ©2020 VMware, Inc. 19
Planned 2021 vA11y additions (Continued)
10. 2.4.5 - multiple ways
11. 3.2.1 / 3.2.2 - Is there a live region when an action triggers changes?
12. 4.1.3 - is there a live region for message announcements when message doesn't
get focus?
13. 1.4.5 - warning when there is text embedded in picture
14. 1.4.11 - KFI approach except as applied to image buttons,
15. 2.2.2 - automatically moving carousel
16. 3.2.3 - Take inventory of menus, compare to menu organization on subsequent
pages, focus on horizonal and vertical nav

20. ©2020 VMware, Inc. 20
Planned 2021 vA11y additions (Continued)
17. If a keyboard shortcut is used, test to make sure only one non-alpha key is
used
18. 1.4.12 - Check to see if works are cut off or obscured after spacing
changes
19. 1.1.1 - Everything that is not “Google I am not a robot v3” is a fail
20. 1.3.2 / 2.4.3 - focus order
21. 1.2.3/1.2.5 DA/AD button

21. ©2020 VMware, Inc. 21
How did VMware Test vA11y
Supervised learning at every level
UK.gov
31 Sites, 8 industries
Top 300 “safe for work” sites
Branch into building custom data models

22. 22
©2020 VMware, Inc.
Next Steps

23. ©2020 VMware, Inc. 23
vA11y Demo

24. ©2020 VMware, Inc. 24
How do you use vA11y
• Go to Git, download the package
• Install the package
• Use the default model or build your own
• Make vA11y API calls
• Review results
• Feedback corrections into your model (optional)

25. ©2020 VMware, Inc. 25
Interested in contributing to vA11y?
We need help with the following
• Early testers
• Documentation translators
• Larger, more evolved, generic data models for other vA11y users to work
from
• People to monitor slack channel and answer basic questions
• Participate in coding one of the 26 planned additions in 2021

26. Confidential │ ©2020 VMware, Inc.
Thank You
THANK YOU
SHERI BYRNE-HABER
sbyrnehaber@vmware.com
https://medium.com/@sheribyrnehaber
https://www.linkedin.com/in/sheribyrnehaber/