De presentatie van Joost de Wit van Media Distillery tijdens de Technology Update - Watskeburt?! 6 oktober 2016

1. Watskeburt?!
Onder de motorkap
Technology Update
6 oktober, 2016

2. Founded by Geert Vos and Joost de Wit in 2014

3. We still can’t search in video content!

4. Adding metadata used to be manual labour

5. We use advanced data mining to analyse video content
Photo courtesy of Google

6. Both broadcast & on-line sources
Teletext
Speech
Subtitles
Logos
Faces
Filmstrips

7. Ingest
Analysis
Storage
Presentation /
interaction
API

8. caskstillwell
foundation

9. Large Scale
Video Storage
Online Video
Service
Storage
Service
RSS feed
Video scraper
RabbitMQ
Speech
Recognizer
Subtitle
Extractor
Filmstrip Creator …
individual frames
chunked video
annotations
well still caskfoundation

10. Speech
Recognizer
Subtitle
Extractor
Filmstrip Creator …
RabbitMQ Indexer Elasticsearch
Content
Service
Notification
Service
Search
Service
Authentication
Service
Feed
Service
well still caskfoundation
annotations Large Scale
Video Storage

11. Large Vocabulary Automatic Speech Recognition (ASR)
Photo courtesy of IBM

12. r eh k ao g n ay z s p iy ch
"recognize speech"
Speech / non-speech
Speaker diarisation
Phonemes (acoustic model)
Vocabulary (language model)

13. Dialect de NS > Dennis

14. SjoemelsoftwarePhone calls

15. Subtitle extraction

16. ROI selection
Text extraction
OCR
Utterance detection
Garbage detection

18. Logo recognition

20. Voor het kijken
Tijdens het kijken
While browsingWhile watching

21. Research questions
• How to “visually summarise” a video clip as a filmstrip?
• #frames to show (fixed / variable)?
• Which frames to show?
• Size of the frames to show (fixed / variable)?
• Part of the frame to show?
• How to present the filmstrip?
• How should users interact with it?

22. Frame sampling
Shot detection
Frame selection
Merge
[Bar clipping]
Videoclip
Filmstrip & manifest

23. Features for frame/bar selection
• Gezichten (en de beweging ervan over
het scherm)
• Gezichtsuitdrukking
• Open ogen
• Tekst
• Infographics
• Scene-overgangen
• Scherpte in het beeld
• Rule-of-thirds
• Aanwezigheid van muziek / spraak
• Ondertiteling
• Visuele eigenschappen frame
(sharpness, saturation,
kleurhistogram)
• Programma specifiek (studio,
naambordjes, …)

24. Filmstrip as (eventually) tested
• Variable number of frames (one per shot)
• Selected frame just after shot changed
• Fixed width, no clipping
• Focused on the presentation & interaction

25. Examples
• Short clip
• One long shot taken from a helicopter
• No voice-over or text present

26. Examples
• The subtitling tells the story
• Selected images don’t contain much information

27. Examples
• Great example
• Reads like a comic book
• Lucky shot

28. Alternative filmstrip
• Variable number of frames
• “bar” width based on length of shot
• “bar” cut with respect to frame’s center

29. Storing the filmstrip
• We store parts of the filmstrip up to a maximum length separately
• Parts are stored in Cassandra
• Efficient retrieval based on program ID and start time of the strip
• Scalability and redundancy are build-in in Cassandra
Part 1
Part 2

30. The API
• REST API (created using Jersey, running in Apache Tomcat)
• JSON & JPG as output

31. {
"height" : 480, // The height of the filmstrip
"strips" : [
{
"program_id" : "b214753f-5e17-46ec-afeb-c1b3d9ee6565", // The Id of the program from which
// the filmstrip was created
"start" : 0, // Start time of the strip (in milliseconds)
"stop" : 12000, // Stop time of the strip (in milliseconds)
"url" : "<base>/strip_001.jpg", // The URL of this strip's image
"width" : 6700, // The width of this strip
"segments" : [ // Segments are the tiles in the strip
{
"start_offset" : 0, // The start offset of this segment (in pixels)
"stop_offset" : 100, // The stop offset of this segment (in pixels)
"start_timestamp" : 0, // The start time of this segment (in milliseconds)
"stop_timestamp" : 1234 // The stop time of this segment (in milliseconds)
}, {
"start_offset" : 101,
"stop_offset" : 305,
"start_timestamp" : 1235,
"stop_timestamp" : 2345
}
]
}
],
"previous" : false, // A reference to the previous filmstrip file (when present)
"next" : "<base>/next_strip.json" // A reference to the next filmstrip file (when present)
}
/api/{version}/filmstrip/{program_id}

32. Flexibility of the API (for other applications)
Parameter Required Description Default
program_id yes The id of the program for which the filmstrip should
be returned
-
start no The start time (in milliseconds) in the video for
which the filmstrip is constructed
0
max_duration no The maximum duration (from {start}) of the video
covered by the filmstrip
-1
(complete clip)
max_width no The maximum width (in pixels) of one single
filmstrip
15000
size no The size of the filmstrip, either small, medium or
large
small

33. /api/{version}/filmstrip/{program_id}/strip

34. UI challenges
• Cross platform & responsive (is a hassle)
• We had to ‘fold’ the frames to shorten the strip
• Interaction between the strip and the player
• Move strip while watching?
• Interact with strip while watching?
• Equally sized frames represent different shot
durations
• Implementing ‘the wiggle’ was challenging, but
required to stimulate swiping
• IOS only plays video in fullscreen mode

35. The filmstrip UI
• Runs in the (mobile) browser
• Responsive design (using Bootstrap)
• Heavily using jQuery + libraries
• jQuery UI
• jQuery Mobile (Events)
• Kinetic
• SmoothDivScroll
• Handlebars
• Thin backend that’s basically a proxy to the Media Distillery platform

36. Any questions?

37. Media Distillery
John M. Keynesplein 12-46
1066 EP Amsterdam
info@mediadistillery.tv
+31 (0)6 50 983 893