Keynote talk given at the SALMM Workshop (https://salmm-workshop.org/program/) on Monday 21 October 2019, at ACM Multimedia 2019, Nice (France)

1. HyperTED: Exploring Video
Lectures at the Fragment Levels
for Enhancing Learning
Raphaël Troncy <raphael.troncy@eurecom.fr>
Data Science, EURECOM
@rtroncy

2. Credits
Julien Plu
Pasquale Lisena
Giuseppe Rizzo
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 2
Mariella Sabatino José Luis Redondo Garcia

3. 21/10/2019 - Workshop on Search as Learning with Multimedia Information

4. Once upon a time …
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 4

5. … leading to sharing Media Fragments
 Publishing status message containing
a Media Fragment URI
 Use a ‘#’ !
 Highlight a
video
sequence
 Highlight a
region
to pay
attention to
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 5

6. Making video a "first class citizen"
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 6

7. t0 20 35
temporal media fragment
spatial media fragment
track media fragment
named media fragment“Scared Scene”
What are Media Fragments?
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 7

8. Media Fragments (temporal)
Fragment beginning Fragment endPlayback progress
Original resource
length
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 8

9. Media Fragments (spatial)
semi-opaque
overlay
highlighted
fragment
http://ninsuna.elis.ugent.be/MFPlayer/html5
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 9

10. Media Fragments URIs
 Bookmark / Share parts (fragments) of audio/video
content
 Annotate media fragments
 Search for media fragments
 Develop Mash-ups/Collage
 Conserve bandwidth
http://www.w3.org/TR/media-frags-reqs/
http://www.w3.org/TR/media-frags/
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 10

11. 1984
.com
2006
CHAPTERS
2014 HOT SPOTS
ENTITIES
RELATED TED’S
CHAPTERS
THE MYSTERIOUS
FIELD OF
ENGINEERING
SYSTEMS
UNDERSTANDING
ENVIRONMENT:
A SYSTEM
APPROACH
SYSTEMS
PRACTICE:
MANAGING
SUSTAINABILITY
COURSES

12. New Consuming Paradigm
Users
overwhelmed
with audio-visual
content
What are the
potentially
relevant
fragments ?
How can users easily find
related documents which
complement the video
Can the video be
divided into
meaningful
fragments?
How can those
fragments be
properly
described?

13. Media Fragment support
 Chapters
 Hot Spots
Media Fragment annotations
 Named Entity Extraction
 Topic Detection
Hyperlinking
 With TED talks chapters
 With other educational online resources
HyperTED

14. http://www.w3.org/TR/media-frags/
A Media Fragment is a portion of a multimedia resource
Temporal Fragments
sections along the time dimension of the media
resource with a start and an end point
Media Fragments

15. TED Talks have paragraphs:
a human-made subdivision of subtitles
MF: Chapters

16. MF: Chapters

17. “This is Nikita, a security guard from one of the bars in St. Petersburg.”
“This is Nikita, a security guard from one of the bars in St. Petersburg.”
NER
Example taken from the transcript of
https://www.ted.com/talks/2089
PERSON
FUNCTION
LOCATION
Category:
type in the NER task
Natural Language Processing (NPL) Task 
disambiguating URL in a knowledge base
e.g. https://www.wikidata.org/wiki/Q656 or
http://dbpedia.org/resource/Saint_Petersburg
Annotations: Named Entities

18. NER Extractors
• Integrates different NER tools available on Web
• Unify NER extractors in a common output

19. Annotations: Named Entities

20. Mobile computers
Annotations: Topics
“I'm wearing a camera, just a simple webcam, a portable, battery-powered
projection system with a little mirror. These components communicate to my
cell phone in my pocket which acts as the communication and computation
device. And in the video here we see my student Pranav Mistry, who's really the
genius who's been implementing and designing this whole system...”
Battery (electricity)
Consumer electronics
Example taken from the transcript of
https://www.ted.com/talks/pattie_maes_demos_the_sixth_sense
Chapter 3

21. 1. Clustering of consecutive chapters which talk
about similar topics and entities
2. Ordering of those fragments based on
annotation relevance (TF-IDF)
3. Filtering: Hot Spots are fragments whose
relative relevance falls under the first quarter of
the final score distribution
MF: Hot Spots
Hot Spot 1
Chapters
Hot Spot 2
Hot Spots

22. 1. Clustering of consecutive chapters which talk about similar topics and entities
2. Ordering of those fragments based on annotation relevance (TF-IDF)
3. Filtering: fragments whose relative relevance falls under the first quarter
MF: Hot Spots

23. MF: Hot Spots
https://github.com/jluisred/HotSpots

24. • Topics
• Entities
• time code references (startNPT and endNPT)
• extractor confidence
• Resource identifier
• Full text transcript
Granularity level
• Chapter
Features indexed
Hyperlink: Indexing TED Talks

25. Datasets
• Open Courseware
• Open University
Anchors used in search
• Entities Too specific
• Topics Courses about the same thematic
Attributes used in search
• Title
• Description
• Subject, thematic …
Hyperlink: Finding related courses

26. Hyperlink: Finding related courses

27. Architecture

28. HyperTED Re-Born
http://linkedtv.eurecom.fr/HyperTED
3194 TED Talks
69033 chapters

29. HyperTED Demo
https://www.ted.com/talks/tim_flannery_can_seaweed_
help_curb_global_warming
Chapter 15: recommendation about the sea
https://www.ted.com/talks/clay_shirky_how_cellphones
_twitter_facebook_can_make_history
Chapter 2: recommendation about democracy
Chapter 5: recommendation about collaboration

30. MediaMixer Demonstrator (CERTH-ITI)
 Video lectures shot segmentation
 Concept detection adapted to
video lectures (37 concepts)
http://multimedia.iti.gr/mediamixer
/demonstrator.html
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 30

31. MediaMixer Mashup (JSI)
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 31

32. Let's go even back in time [AIED, 1999]
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 32
Sassine Abou-Jaoude, Claude Frasson, Olivier Charra and Raphaël Troncy.
On the Application of a Believable Layer in ITS. In (AIED'99) Workshop on Synthetic
Agents, Le Mans, France, July 19, 1999

33. Learning by Gaming
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 33

34. Summary / Take Away
 Trove of learning content buried in videos
 need tools to segmentate / annotate / discover this content
 We developed HyperTED in 2014!
 the concept is still original
 Natural Language Processing (information extraction)
 deep learning-based named entity extractors
 word and entity embeddings (multilingual, multimodal)
 watch ADEL: https://github.com/jplu/ADEL
 watch entity2vec: https://github.com/D2KLab/entity2vec/
 Recommender systems … at the fragment level
 deep learning architecture, using KG embeddings
21/10/2019 - Workshop on Search as Learning with Multimedia Information - 34