This presentation discusses measuring subtitle quality and regulation. It argues that measuring accuracy, reading speed, and synchronization makes sense but focusing only on metrics ignores viewer comprehension. Different countries have different subtitle rules but all agree captions should be accurate and synchronized. Live subtitles are prone to errors so should be a last resort. As a regulator, it is better to focus on ensuring quality standards rather than punishing one-off errors, and ensuring the market supports high quality captions for viewers.

1. Presentation by
Alex Varley
Does measuring subtitle quality
really make any difference?

2. Welcome to the Future

3. SubtitHAL 9000

4. Automatic scanning result:
Reading speed = 182 wpm
91.87% accuracy

5. This is your second offence.
Your broadcast license will
be suspended for 20 hours.
You will be automatically
fined €40,000.

6. Metrics are much better.
Easy to
Clear rules
measure
Easy to Easy to
regulate understand
Easy to see
errors

7. This type of approach works best for
everyone but the viewer
Viewer
Regulator,
supplier,
broadcaster

8. Different countries = different rules

9. Everybody agrees on some rules
Subtitles should be accurate
Non-live subtitles should be
synchronised
Subtitles must not cover on-screen
information

10. Accuracy
100%-75%
What does accuracy mean?

11. Delay
Time lag for live subtitles:
3-7 seconds
Or delay the broadcast

12. A live program is
1 hour
? 4 hours
Before broadcast
96 hours 24 hours
72 hours

13. Reading speed
120 wpm – no limits
180 wpm used to be popular

14. If you make everything
measurable then people
focus only on the metrics

15. In Australia quality will be measured by:
• Readability
• Synchronisation
• Accuracy
The key outcome is can the viewer
comprehend the program?

16. In Australia we recognise that:
Live subtitling is where all of the errors and
problems usually occur
So live subtitles should be a last resort,
Even for “Live” programs

17. Live sport will have live scrolling subtitles

18. News should have a mix of block
and scrolling subtitles.

19. These all impact on quality of live subtitles
Market
Training
pricing
Preparation
Regulation
time
Complexity
of output

20. As a regulator it is better to focus on:
• Do viewers know what to expect?
• Are quality errors repeated?
• Is every supplier doing the same thing?
• Are you acting consistently?
• Do you work with suppliers to fix issues?
• Is the market working?

21. Punishing one-off errors is a waste of time

22. But letting cowboys into the market means
that everybody has to be a cowboy.

23. Contact:
alex.varley@mediaaccess.org.au
www.mediaaccess.org.au
Thanks to Chris Mikul who sits
through the hours of regulator
meetings and helped research this
presentation.