This document summarizes a case study examining whether automatic transcription software produces transcripts accurate enough to serve accessibility needs. It tested several transcription tools on recordings from experts in property management, construction management, and related fields. Analysis found wide variability in accuracy across tools and recordings, with average errors ranging from 73-263 words per 1000. Feedback from students found transcripts generally useful, especially for non-native English speakers, though not always accurate enough to replace manual transcripts. The study concluded current tools may provide a starting point for accessibility but do not achieve the high accuracy required, particularly in technical disciplines.

1. ©UCEM
Are Automatic Transcriptions Good
Enough for Accessibility?
EDEN NAP webinar and #EDENChat on 27 November 2019
http://www.eden-online.org/eden-nap-webinar-and-edenchat-on-27-november-2019/

2. ©UCEM
Automatic transcription software: Good enough
for accessibility?
Dr Tharindu Liyanagunawardena
A case study from built environment education

3. ©UCEM
Plan
• Background
• Accessibility
• University College of Estate Management
• The Study
• Data Collection
• Analysis
• Findings
• Conclusion

4. ©UCEM
Begin programming: Build your first mobile game

5. ©UCEM
Background
• Dramatic increase in video use
• Over 500 million watch video on Facebook every day1
• One billion hours of video watched on YouTube!
• 70% YouTube watched on mobile devices
• The State of Video in Education 2018 report from Kaltura
• Over 1500 surveyed
• Use of lecture capture, ↑ 21% to 79%
• Use of video by students for assignments 69%
• Video feedback on student assignments in 35% of institutions
• Closed captions are in use at 52% of institutions

6. ©UCEM
Accessibility
• The quality of being easily reached, entered, or
used by people who have a disability
- Oxford Living Dictionary
• Captions and Transcripts
• Manual Captioning
• Resource Intensive
• Rule of Thumb
allow four hours of transcribing for an hour of recording
(Punch & Oancea, 2015)
• Automatic Speech Recognition
• Google Duplex
• Apple Siri, Amazon Alexa
Punch, K. & Oancea, A. (2015). Introduction to Research Methods in Education (2nd eds). London: SAGE.

7. ©UCEM
Accessibility Continued
• EU Accessibility Directive
• New UK Regulations on Accessibility for Public Sector Bodies
• 23rd September 2018
• Accessible VLEs – Making the most of the new regulations
• The right thing to do
• What stops us from doing the right thing?
• Case of University of California Berkeley

8. ©UCEM
University College of Estate Management
• Online Distance Education Materials and Accessibility: Case Study of
University College of Estate Management (2016)
• Captioning videos for accessibility: A case study of University College of
Estate Management (2017)
• ALT South webinar: Online Learning Materials and Accessibility (2018)
• Established 1919
• Postal  Online
• College  University College (2015)
• Supported online learning
• The leading provider of HE for
construction, surveying and real estate
globally (HESA statistics 2015/16)

9. ©UCEM
The study
• Selected software
• Descript
• IBM Watson Speech to Text (Watson)
• Sonix
• Synote
• Trint
• Zoom

10. ©UCEM
9

11. ©UCEM
Data Collection
• 7 recordings
• Recording length  from 5:57 to 8:21minutes
Participant Gender
Native English
Speaker
English Accent
(as identified by participant)
1 Male Yes Generic British
2 Female No South American
3 Male Yes Generic Scottish
4 Male Yes Generic British
5 Female No South Asian
6 Male No Greek
7 Male No African

12. ©UCEM
Analysis
• Recordings transcribed
• Check with recording
• Microsoft Word Compare
• Manual analysis
• Word Error Rate (WER)
WER = (Substitution + Deletion + Insertions) / N
where N is the total number of words in the reference transcript (Apone, Botkin,
Brooks, & Goldberg, 2011).
Apone, T., Botkin, B., Brooks, M., & Goldberg, L. (2011). Caption Accuracy Metrics Project: Research into
Automated Error Ranking of Real-time Captions in Live Television News Programs. Retrieved from
http://ncam.wgbh.org/file_download/136

13. ©UCEM
Word Error Rate
• WER = (Substitution + Deletion + Insertions) / N
• Example:
(actual) this process will be quick
(caption) this proswilling quick
(actual) this process will be quick
(caption) this proswilling **** ** quick
S D D
Apone, T., Botkin, B., Brooks, M., & Goldberg, L. (2011). Caption Accuracy Metrics Project: Research into
Automated Error Ranking of Real-time Captions in Live Television News Programs. Retrieved from
http://ncam.wgbh.org/file_download/136

14. ©UCEM
Results
0
100
200
300
400
500
600
700
800
2 3 4 5 6 7
Transcript number
Transcription Errors (per 1000 words)
Trint - Sonix - Descript - Watson - Zoom - Synonte -

15. ©UCEM
Findings
Property
Management
Construction
Management
Building Pathology
Property and
Contract Law
Expert 1 Expert 2 Expert 3 Expert 4 Expert 5 Expert 6 Expert 7
2 x x x x x x x
3 x x x x x x x
4 x
Good
enough
Good
enough
x
Good
enough
Good
enough
x
5 x
Good
enough
x
Almost
good
enough
x x x
6 x x x x x x x
7 x x x x x x x

16. ©UCEM
Conclusions
• Quality of recording matters
• In a technical discipline WER may not be sufficient predictor of
quality
• Good starting point for creating a transcript as an accessibility aid
• At present, off-the-shelf automatic transcription software does not
produce a high enough level of accuracy for the creation of
accessibility aids for the built environment sector

17. ©UCEM
Use of automatic transcripts
• Webinars (about 10 in each module)
• Introduced Zoom – Spring 2019
• Zoom provides automatic captions
• Not 100% accurate
• Average errors per 1000 words  263
• According to the study  about 73.7% accurate
• User feedback survey  283 student responses

18. ©UCEM
Feedback: Are automatic transcripts useful?
80%
14%
6%
Yes No Other

19. ©UCEM
Feedback: non-native English speakers
Yes
95%
No
5%
Yes No

20. ©UCEM
Use of automatic transcripts: different views
So useful! Amazing. The search functionality is revolutionary. If I vaguely
remember something being said in the webinar but can't quite remember
where or when I used to have to watch almost the entire webinar again -
and it took so long to load. Now, I just type a word and everything comes
up. It's amazing. Ok, the audio transcript is not always spot on, but frankly
it's amazing how much it does pick up especially with different accents and
pronunciation of words and acronyms. Really love it.
As a non-native (English) speaker, it's great to understand the audio-
transcript.
No, poor quality transcription. Could confuse some students, especially those
for whom English isn't their first language.
yes, i am dyslexic and found it very useful when i needed to make notes and
couldnt remember

21. ©UCEM
How do students use automatic transcript?
• A search mechanism!
• Notes
• Instances where audio is not usable
• Aid for learners with disabilities
• Aid for learners with English as an additional language

22. ©UCEM
Automatic Transcription Software: Good Enough for Accessibility? A Case
Study from Built Environment Education (PDF)
tharindu@ucem.ac.uk
@Tharindu__