How we use Serverless & Step Functions to quickly adapt our workflows disguised with a buzzwordy title with examples of Machine Learning and cloud architecture Presented at MetEXPO (SMPTE) Sydney 18th July 2019

1. Quick Wins
three times ML
enhanced video
workflows for 7plus

2. I’m Jeremy Brown
Head of Video Architecture
Seven West Media Australia
Hello!

3. Organiser syd<video>
2,250 members
With speakers from
Also

4. “
Machine Learning; the ability to learn
without being explicitly programmed
- Arthur Samuel

5. “
“Quick wins with ML”
is not a neural network AI talk

6. We do a lot of live
53 channels 24 x 7 x 365
The biggest TV events in sport, shows & news

7. “
40 additional channels for Olympics
Lucky enough to stream
some of the largest events

8. “Live streaming uses both
on prem and cloud architecture

9. “
We do a lot of VOD

10. “
We do a lot of VOD
Hundreds of thousands of clips
across Seven West Media

11. “
With publishing VOD and
media preparation. We use
serverless architecture

12. “Serverless infrastructure is code
templates, instantly deployable

13. “Serverless event-driven
architecture mirrors traditional
video workflows

14. 1. Managed services
2. Auto scaling highly resilient
3. Cheap, per transaction model
4. Continuous Integration & Deployment
Serverless Benefits

15. “Step Functions chain serverless
functions to video workflows

16. Captions
Post-production
Editing
Image
Processing
Quick Wins

17. Automating
Post-production Editing

18. “
We work in constant envy of
new features on Netflix

19. “This binge feature requires
editorial markers for credits

20. “
Computer Vision models can
detect objects, faces, emotion,
travel paths, text, celebrities...

21. “Can process video or images
for significantly cheaper

22. “Images are significantly
cheaper than video

23. “7plus scrub feature generates
easily indexable images

24. “Many shows lead credits with
easily detectable black
Black detected
21:47 21:48 21:49
21:50 21:51 21:52

25. “We also find credits with
detected dominant black
Black detected
Black detected Black detected
Black detected Black detected
Black detected

26. “
But credits aren’t always black

27. “Credits lead with several
detectable text events
Director Producer .
Producer .
Producer . Producer .
Editor .

28. “
Easily configured

29. “

30. “

31. “

32. “
Black Detected .

33. “

34. “

35. “
“Producer” Detected .

36. “
"♫ you know we belong together♫"
Tests to detect the intro
using audio would work

37. Automated
Image Processing

38. “
Images grabbed automatically can
be unflattering have little context

39. “
Studios provide optimised
images per episode

40. “
Studio images aren’t 16:9
Still camera shots in
portrait, 4:3 & 3:2

41. “
Centre cropping images
were unbalanced

42. “
By avoiding image scaling
created ugly borders

43. “
Original ResizeEnergy map Seams
Energy map edge detection
Algorithm finds important parts
Generates individual seams (top to down, or left to right)
Compute the cumulative minimum energy connected seams
Content aware image resizing

44. “
Amazing!
Content aware image resizing

45. “
Results were mixed

46. “
Results were mixed

47. “
Results were mixed

48. “
7plus built a face detection tool:
● Face recognition algorithm
● Train with dozens of cast images (from IMDB)
● Find safest crop point
DIY Machine Learning Face Detection Tool

49. “
Model needed more training
lots of work per new show

50. “
● Used by Google, Yahoo, Microsoft, Intel, IBM, Sony, Honda, Toyota
● Trained and deployed in many use cases:
● Stitching street view images
● Detecting surveillance video
● Helping robots navigate
● Inspecting labels on products
● Rapid face detection ← Cool!
Open Source Computer Vision Library

51. “
Results were mixed

52. “

53. “

54. “
Worked even with portrait

55. Automated
Closed Captioning

56. “
A significant portion of content is
supplied uncaptioned

57. “
Extracting captions from video
Speech to text can
generate captions

58. “Cloud providers have
working solutions

59. We trialled different types of
content for speech-to-text

60. News & Talk Shows
Clearly enunciated
No background sound
Regular presenters

61. News & Talk Shows
Clearly enunciated
No background sound
Regular presenters
Premium Drama
Various countries
Range of accents
Different actors

62. News & Talk Shows
Clearly enunciated
No background sound
Regular presenters
Premium Drama
Various countries
Range of accents
Different actors
Fishing Shows
Presented outside
Strong accents
Slang & species

63. “
Results were mixed
With serverless and a speech API
we had built a POC in hours

64. “
Results were mixed
The accuracy wasn’t perfect

65. “
Results were mixed
“Windsor”
Location names

66. “
“Quay”
Homophones

67. “
“how the fight”
Australian accent?

68. “Windsor” - Tow (0.4940) wins (0.9490) a (0.9801) bridge (0.6395)
“Quay” - Circular (1.0000) key (0.3497)
“How the fight” - About (0.9996) hell (0.8830) the (0.9879) f--k (0.8509)
“Turnbull” - Alleged (0.9986) malcolm (0.9986) terrible (0.4550) also (1.0000)
Each translation is scored, these
indications used for review

69. Why 7plus reviews output?
1. As a broadcaster we’ve committed to highest
accuracy on our content
2. When publishing news, we could be liable
when an error implicates others

70. Developers praise the technology;
but considered user reaction
"Are they illiterate?, this is embarrassing!"

71. Speech-to-text continually
improves, is ready many use cases

72. “
Humans are uniquely adaptable
Humans have so much other contextual knowledge not available to a machine transcription algorithm
Humans are much better transcribers in difficult situations, poor quality audio, speakers are far from recording devices, or
speakers are familiar with one another.
Machine translation has similar human-level performance. So essentially it is a propagation of error problem.
Doing machine translation on a machine transcription will have the same performance as if you used human translation on the
text from a human transcription

73. On The Fly
Multi-Lingual
Captioning

74. “
Like the video, captions are manifests
directing players to text segments
VTT caption text
file segments

75. “
Broadcast supply captions in English

76. “We can translate text captions
to many languages cheaply

77. “

78. “
Group rows into sentence
1. The company already employs
2. many Australians
3. in its four other
4. overseas offices.
VTT language is segmented into a
broken format, we group logically first
Results
La compañía ya emplea a muchos
australianos en sus otras cuatro
oficinas en el extranjero.
Das Unternehmen beschäftigt
bereits viele Australier in seinen
vier anderen Auslandsbüros.

79. Captions
Post-production
Editing
Image
Processing
Quick Wins

80. This is the
80th slide!

81. Jeremy Brown
Head of Video Architecture
Seven West Media, Australia
Thanks

82. Jeremy Brown
Head of Video Architecture
Seven West Media, Australia
Thanks