Ai and inclusion - Challenges and Benefits for those with disabilities.

1. AI and Inclusion Seeking to understand the design and deployment of AI and inclusion to benefit those with disabilities, which will also provide digital accessibility for all members of society Professor Mike Wald E.A. Draffan Dr Chaohai Ding

2. Challenges… AI and Inclusion

3. Make algorithmic systems fair, transparent, and ethical Of the nine protected characteristics identified by the Equality Act 2010, disability is the least homogeneous and so techniques need to be developed to ensure algorithms work fairly for these ‘edge cases/outliers’

4. Health and Inclusion • Social vs Medical model of disability • Large data sets for diagnosis • Lack of data to overcome barriers

5. Some Principles • Assistive/Augmentative AI • Include disabled people in design • Design for ‘edge cases/outliers’

6. Some Examples • Speech recognition • Captioning speech and sounds • Sign Language Synthesis & Recognition • Intelligent personalised interfaces • Artificial Assistance • Symbol communication • Simplification of text • Describing images • Recommendations of AT solutions • Autonomous mobility guides

7. Speech Recognition more accurate than Human Transcribers • Professionals had 5.9% and 11.3% error rates • Speech recognition had 5.9% and 11.1% error rates Achieving Human Parity in Conversational Speech Recognition : W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig: Microsoft Research Technical Report MSR-TR-2016-71 February 2017

8. A proof-of-concept demonstration of SpeechBubbles. The user (right) equipped with a Microsoft HoloLens views speech bubbles adjacent to two speakers (left and middle). Peng et al 2018 CHI Conference.

9. AI can identify speakers with 92% accuracy https://ai.googleblog.com/2018/11/accurate-online-speaker-diarization.html

10. AI can caption some sounds (Applause, music, laughter) https://youtube-eng.googleblog.com/2017/03/visualizing-sound-effects.html

11. AI can be used to improve lecture capture

12. https://s3-us-west- 1.amazonaws.com/vocals22/VENEC/plot_sounds_auto_expdimem.html Speech emotion detection

13. Paralletdots https://www.paralleldots.com/facial-emotion

14. Using % of colours to represent emotions in a caption

15. Lip reading

16. https://www.youtube.com/watch?v=fFScL-7vWHM Average experienced lipreader performance is only 52% LipNet goes up to 93%

17. https://www.youtube.com/watch?v=103CXDFhpcc Lip Reading Sentences in the Wild

18. 50% Word accuracy on the held-out test set. Many of the errors are tenses, plurals, etc.

19. Also improves Speech Recognition Performance

20. https://www.apple.com/uk/iphone-xs/cameras/ 3D Cameras provide even better Lipreading and Speech Recognition accuracy

21. https://www.youtube.com/watch?v=ohmajJTcpNk Could also replace face with a more lipreadable one

22. Reenactment Pipeline

23. Captioning https://ai.googleblog.com/2016/09/show-and-tell-image-captioning-open.html

24. Tags

25. https://azure.microsoft.com/en-gb/services/cognitive-services/computer-vision/ Analyse video in near real-time • Use computer vision APIs with your video files by extracting frames of the video from your device • Send those frames to the API calls of your choice. • Get results from your videos more quickly

26. Challenge… • Automatic audio description of videos requires reasoning and understanding subtle meanings and context to identify what visual information is important. e.g. If a person leaves a room is it important to know they did not hear what was said after they left?

27. Real time sign language translation https://isigner.com/

28. Avatar http://asl.cs.depaul.edu/demo.html

29. Challenge… • AI can provide automatic sign language translation of captions using human video clips or avatars • But the quality of translation for a visual language is not as good as translations between written languages which have vast amounts of data available for training the AI systems.

30. Mobility guides CMU and Pittsburgh Airport launch smart suitcase for blind passengers

31. Recommendations of AT solutions • Inclusive Employment • Workplace Reasonable Adjustment • What we have? • 20,000 + assessments reports over 5 years • Word document/PDF • Multiple-version of documents layouts • Tasks • Extract useful content to free-text • Data anonymisation • NLP + Deep Learning

32. Challenges… • AI can help with workplace assessment • Speed up the process • Reduce the cost • Remote or self assessment • AT recommendation • But there is still a need for • More data (new ATs) • Explainable algorithms • End-to-End solutions

33. Web Accessibility checking • Image recognition providing alternative text for images • Automatic context sensitive captions for images • Detecting contextual hyperlink text and flagging for correction • Form completion support

34. Machine learning – Intelligent personalised interfaces, text simplification, summarisation and writing support • Chatbots and Conversational AI as digital assistants • Complex text clarified for those with cognitive impairments • Summarisation for confusing convoluted sentences • Blog writing made easier with targeted support

35. Challenges… • Need to augment human abilities allowing for empathy and other human creativity • Increase data representing diversity • Develop understanding of the human condition at the time of use to be truly assistive • Solve misrecognition - simplification, summarisation and support can be too general and fail to respond to context or miss main facts.

36. Complex Communication Needs • Machine learning to adapt to Augmentative and Alternative Communication user’s situation and skills Livox https://youtu.be/m_oWJqa1vtY (2mins)

37. Global Symbols • Using machine learning to classify the symbol's category based on ConceptNet. • Image recognition for concept detection • Deep learning for symbol style transfer and generation

38. Final Thoughts … “Some global institutions are beginning to examine how AI can impact and contribute to the social good, but there is much work to be done. This emerging “AI Divide” - if allowed to continue - could jeopardize equal treatment of people within and among nations. This asymmetry is a critical issue that must be addressed locally and globally.” Berkman Klein Center for Internet & Society "The top priority is creating a diverse, inclusive, substantial pipeline of ethical, competent and talented MSc graduates highly skilled in Machine Learning and Artificial Intelligence.." (British Computer Society)

39. Thank You Discussion AI and Inclusion Team