Description of a series of challenges and benefits related to AI and Inclusion for those with disabilities. Discussion around automating web accessibility checks and supporting augmentative and alternative communication symbol searches with better classification using linked data, image recognition and machine learning.

1. AI and Inclusion
Seeking to understand the design and
deployment of AI and inclusion to
benefit those with disabilities, which will
also provide digital accessibility for all
members of society
Professor Mike Wald
E.A. Draffan
Dr Chaohai Ding

2. Challenges…
AI and
Inclusion

3. Make algorithmic systems fair,
transparent, and ethical
Of the nine protected characteristics
identified by the Equality Act 2010,
disability is the least homogeneous and
so techniques need to be developed to
ensure algorithms work fairly for these
‘edge cases/outliers’

4. Health and Inclusion
• Social vs Medical model of
disability
• Large data sets for diagnosis
• Lack of data to overcome barriers

5. Some Principles
• Assistive/Augmentative AI
• Include disabled people in design
• Design for ‘edge cases/outliers’

6. Some Examples
• Speech recognition
• Captioning speech and
sounds
• Sign Language Synthesis
& Recognition
• Intelligent personalised
interfaces
• Artificial Assistance
• Symbol
communication
• Simplification of text
• Describing images
• Recommendations of
AT solutions
• Autonomous mobility
guides

7. Speech Recognition more accurate
than Human Transcribers
• Professionals had 5.9% and 11.3%
error rates
• Speech recognition had 5.9% and
11.1% error rates
Achieving Human Parity in Conversational Speech Recognition : W.
Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G.
Zweig: Microsoft Research Technical Report MSR-TR-2016-71 February
2017

8. A proof-of-concept demonstration of
SpeechBubbles. The user (right) equipped with a
Microsoft HoloLens views speech bubbles
adjacent to two speakers (left and middle).
Peng et al 2018 CHI Conference.

9. AI can identify speakers with 92%
accuracy
https://ai.googleblog.com/2018/11/accurate-online-speaker-diarization.html

10. AI can caption some sounds
(Applause, music, laughter)
https://youtube-eng.googleblog.com/2017/03/visualizing-sound-effects.html

11. AI can be used to improve
lecture capture

12. https://s3-us-west-
1.amazonaws.com/vocals22/VENEC/plot_sounds_auto_expdimem.html
Speech emotion detection

13. Paralletdots
https://www.paralleldots.com/facial-emotion

14. Using % of colours to represent
emotions in a caption

15. Lip reading

16. https://www.youtube.com/watch?v=fFScL-7vWHM
Average experienced lipreader
performance is only 52%
LipNet goes up to 93%

17. https://www.youtube.com/watch?v=103CXDFhpcc
Lip Reading Sentences in the Wild

18. 50% Word accuracy on the held-out
test set.
Many of the errors are tenses, plurals, etc.

19. Also improves Speech Recognition
Performance

20. https://www.apple.com/uk/iphone-xs/cameras/
3D Cameras provide even better
Lipreading and Speech Recognition
accuracy

21. https://www.youtube.com/watch?v=ohmajJTcpNk
Could also replace face with a
more lipreadable one

22. Reenactment Pipeline

23. Captioning
https://ai.googleblog.com/2016/09/show-and-tell-image-captioning-open.html

24. Tags

25. https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision/
Analyse video in near real-time
• Use computer vision APIs with your video
files by extracting frames of the video from
your device
• Send those frames to the API calls of your
choice.
• Get results from your videos more quickly

26. Challenge…
• Automatic audio description of videos
requires reasoning and understanding subtle
meanings and context to identify what visual
information is important.
e.g. If a person leaves a room is it important
to know they did not hear what was said after
they left?

27. Real time sign language
translation
https://isigner.com/

28. Avatar
http://asl.cs.depaul.edu/demo.html

29. Challenge…
• AI can provide automatic sign language
translation of captions using human video
clips or avatars
• But the quality of translation for a visual
language is not as good as translations
between written languages which have vast
amounts of data available for training the AI
systems.

30. Mobility guides
CMU and Pittsburgh Airport launch smart suitcase
for blind passengers

31. Recommendations of AT solutions
• Inclusive Employment
• Workplace Reasonable Adjustment
• What we have?
• 20,000 + assessments reports over 5
years
• Word document/PDF
• Multiple-version of documents
layouts
• Tasks
• Extract useful content to free-text
• Data anonymisation
• NLP + Deep Learning

32. Challenges…
• AI can help with workplace assessment
• Speed up the process
• Reduce the cost
• Remote or self assessment
• AT recommendation
• But there is still a need for
• More data (new ATs)
• Explainable algorithms
• End-to-End solutions

33. Web Accessibility checking
• Image recognition providing alternative text for images
• Automatic context sensitive captions for images
• Detecting contextual hyperlink text and flagging for
correction
• Form completion support

34. Machine learning – Intelligent
personalised interfaces, text
simplification, summarisation and
writing support
• Chatbots and Conversational AI as digital
assistants
• Complex text clarified for those with cognitive
impairments
• Summarisation for confusing convoluted
sentences
• Blog writing made easier with targeted support

35. Challenges…
• Need to augment human abilities allowing
for empathy and other human creativity
• Increase data representing diversity
• Develop understanding of the human
condition at the time of use to be truly
assistive
• Solve misrecognition - simplification,
summarisation and support can be too
general and fail to respond to context or miss
main facts.

36. Complex Communication Needs
• Machine learning to adapt to Augmentative and
Alternative Communication user’s situation and
skills
Livox https://youtu.be/m_oWJqa1vtY (2mins)

37. Global
Symbols
• Using machine
learning to classify
the symbol's
category based on
ConceptNet.
• Image recognition
for concept
detection
• Deep learning for
symbol style
transfer and
generation

38. Final Thoughts …
“Some global institutions are beginning to examine
how AI can impact and contribute to the social good,
but there is much work to be done. This emerging “AI
Divide” - if allowed to continue - could jeopardize equal
treatment of people within and among nations. This
asymmetry is a critical issue that must be addressed
locally and globally.” Berkman Klein Center for Internet & Society
"The top priority is creating a diverse, inclusive,
substantial pipeline of ethical, competent and talented
MSc graduates highly skilled in Machine Learning and
Artificial Intelligence.." (British Computer Society)

39. Thank You
Discussion
AI and Inclusion Team