Event-based Summarization for
Media Hyperlinking
Mathilde Sahuguet, Xueliang Liu, Benoit Huet
EURECOM, Sophia Antipolis, F...
Hyperlinking and Second Screen
 Nielsen 12/2012: 85 Percent Of Tablet And
Smartphone Owners Use Devices As “Second
Screen...
Hyperlinking and Second Screen

09/10/2013

Benoit HUET - Event-based Summarization for
Media Hyperlinking
Audiovisual Explosion
 EU alone hosts 500+ online video platforms
 42.7m hrs of footage in online archives of
broadcaste...
Hyperlinking at the fragment level
 MediaMixer:
 “set up and sustain a community of video producers,
hosters and redistr...
Hyperlinking and Second Screen
 LinkedTV: linking television to
Web content

 Second screen scenario for
enriching telev...
Enrichment framework

Input video

Media Annotation

Segment annotation

Annotated
set of media
items

Media
items

Web Me...
Topic Enrichment
 Need to enrich given topics:
 based on terms/topics appearing in video (passive or
active)
 based on ...
Event-based socially aware enrichment

Publish content
09/10/2013

Benoit HUET - Event-based Summarization for
Media Hyper...
Event-based socially aware enrichment

Search term

Google
Trends

09/10/2013

Query content

Benoit HUET - Event-based Su...
Outline
 Related work
 Framework
 Enrichments for Passive Second Screen
 Query Formulation
 Static Event visualisatio...
Related work
 Chieu and al. (ACM CIGIR 2004)
 Query based event extraction along a timeline

 Yan et al. (CIKM2012)
 V...
Our Contributions
 Event-based methodology
 4W paradigm (Who, What, Where, When)

 Mining events from user contributed ...
Passive Second Screen Framework
 Aim:
Enrich & Illustrate the Event on the TV Screen
 Step1: parse the semantic from que...
Event Querying
 Events are queried from Digg.com
 a news aggregator with an editorially driven front page

Two advantage...
Textual Summarization
 Source: Full textual search on Twitter;
 query words = event title
 filter by time = event occur...
Visual Summarization
 Source: Full textual search on Google Images
 Query words = event title
 Sorted by the similarity...
Event Summary

09/10/2013

Benoit HUET - Event-based Summarization for
Media Hyperlinking
Event Summary

09/10/2013

Benoit HUET - Event-based Summarization for
Media Hyperlinking
Active Second Screen Framework
 Aim:
Propose Enrichments which Illustrate the Event
on the TV Screen
 Interaction with t...
Active Second Screen Framework
 Goal = automatically create and illustrate a
timeline of events on a given topic
 Event:...
Trends mining
 Bursts = peaks: sudden increase in the search
volume => unexpected event, massive search
behavior

 Long-...
Trends mining
 Granularity:
 Trend value: Week (imposed by Google trends)
 Trend keywords: Monthly

 Feature:
 Burst ...
Focused search on time segments
 Goal: search for items relevant to the event on
the extracted dates
 Extracting keyword...
Video Selection
 Goal: represent each event with a media item
 Features:
 Using text features (from media title + descr...
Example query: Snowden

1. Get trends
10/21/2013

Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13

- ...
Example query: Snowden

23rd to 29th June 2013
9th to 15th June 2013

2. Mine time segments
10/21/2013

Socially Motivated...
Example query: Snowden

23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlf...
Example query: Snowden

23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlf...
Example query: Snowden

23rd to 29th June 2013
9th to 15th June 2013
ed snowden
edward
edward snowden
edward snowden girlf...
Example query: Snowden
Snowden identity was
revealed on the 9th

Snowden flied to Russia on
the 23rd
23rd to 29th June 201...
Experiments
 Focus on people: automatic event-based
timeline generation of 5 celebrities
 “Ground truth” extracted from ...
Event extraction Evaluation
 Performance of our approach to detect
important events


TP: true positive, TP: false posit...
Event discovery: Beyoncé
 Burst week: 22nd to 28th July, 2013
 Nothing in the ground truth
 Associated keywords:
 Beyo...
Video representative evaluation
 Challenge: choice of an illustrative video
 Event Relatedness ER (mean over all events)...
Example query: Pistorius
 Week: 10th to 16th February, 2013

09/10/2013

Benoit HUET - Event-based Summarization for
Medi...
Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
oscar pistorius girlfriend
oscar pisto...
Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 DataSet tag cloud:

09/10/2013

Beno...
Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:

oscar pistorius ...
Example query: Pistorius
 Week: 10th to 16th February, 2013
 Associated keywords:
 Dataset tag cloud:

oscar pistorius ...
Conclusion
 Socially Aware Hyperlinking Approaches
 Automated Event-based Illustration of topics
 Creation of an Event-...
References
 http://www.linkedtv.eu
 http://www.mediamixer.eu
 M. Sahuguet and B. Huet, “Mining the Web for Multimedia-b...
09/10/2013

Benoit HUET - Event-based Summarization for Media Hyperlinking

43
Thank you for your attention

 Questions?

Benoit Huet
EURECOM
tel: +33 (0)493008179
Campus SophiaTech, 450 route des Cha...
Upcoming SlideShare
Loading in...5
×

Event-based Summarization for Media Hyperlinking @ EBMIP 2013 Workshop on Event-based Media Integration and Processing co-located with ACM Multimedia 2013 – October 21-22, Barcelona, Spain

565

Published on

The exponential growth of media sharing and demand on the Web comes with a need for effective methods to explore them. Hence, media hyperlinking, which consists in linking together videos based on their content, uncovering the relation between them, is becoming an important functionality for providing users with a way to navigate between video entities and satisfy their information needs. Thanks to such technology, multimedia search can often be replaced by recommendation. A particular usage of hyperlinking is to provide, through a second screen application, extra information or content about the video watched on a main screen (TV). In this talk, we will focus on media hyperlinking from the news: the task at hand consists in locating and identifying relevant media items, and display them on the second screen. The related material is selected based on underlying events that will be detected in the news: events are seen as structuring elements, defined in terms of date, location, intent and attendance. Two approaches for event-based mining of such additional and related information will be presented. Each of them satisfying a different user information need.

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
565
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Two first reference tackle the problem of timeline summarization from a topic-oriented collection of documents => summarizationThe 2 next references deal with hot topics discovery and the evolution of a topic over time => more trends analysis orientedLast work is the most similar to ours. 2 main differences: our trend analysis is adaptive to the dataset (they use a fixed threshold for any query) ; timeline illustration is made from news (“official”) while we are people-oriented.
  • Event-based Summarization for Media Hyperlinking @ EBMIP 2013 Workshop on Event-based Media Integration and Processing co-located with ACM Multimedia 2013 – October 21-22, Barcelona, Spain

    1. 1. Event-based Summarization for Media Hyperlinking Mathilde Sahuguet, Xueliang Liu, Benoit Huet EURECOM, Sophia Antipolis, France
    2. 2. Hyperlinking and Second Screen  Nielsen 12/2012: 85 Percent Of Tablet And Smartphone Owners Use Devices As “Second Screen” Monthly, 40 Percent Do So Daily 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    3. 3. Hyperlinking and Second Screen 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    4. 4. Audiovisual Explosion  EU alone hosts 500+ online video platforms  42.7m hrs of footage in online archives of broadcasters and producers (61% of archive footage is online)  UGC on the advance:  YouTube receives 60 hrs of video/minute  Internet video is now 40 percent of consumer Internet traffic, and will reach 62 percent by the end of 2015 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    5. 5. Hyperlinking at the fragment level  MediaMixer:  “set up and sustain a community of video producers, hosters and redistributors…”  “who will be supported in the adoption of semantic multimedia technology…”  “to build an European market for media fragment repurposing and re-selling.” http://community.mediamixer.eu 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    6. 6. Hyperlinking and Second Screen  LinkedTV: linking television to Web content  Second screen scenario for enriching television content and achieving interaction between user and content 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    7. 7. Enrichment framework Input video Media Annotation Segment annotation Annotated set of media items Media items Web Media Search Multimedia Mining Set of relevant media with cues Personalization, Display, etc 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    8. 8. Topic Enrichment  Need to enrich given topics:  based on terms/topics appearing in video (passive or active)  based on user query (active)  Select and organize media items  Different approaches for searching for information  Diverse information rendering  What are people interested to see ?  In this work, we propose and investigate, Event-based socially aware enrichment 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    9. 9. Event-based socially aware enrichment Publish content 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    10. 10. Event-based socially aware enrichment Search term Google Trends 09/10/2013 Query content Benoit HUET - Event-based Summarization for Media Hyperlinking
    11. 11. Outline  Related work  Framework  Enrichments for Passive Second Screen  Query Formulation  Static Event visualisation  Enrichments for Active Second Screen Trends mining Focused search Video selection Evaluation and Results  Conclusion 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    12. 12. Related work  Chieu and al. (ACM CIGIR 2004)  Query based event extraction along a timeline  Yan et al. (CIKM2012)  Visualizing timelines: evolutionary summarization via iterative reinforcement between text and image streams.  Christiansen et al. (WIDM '12)  Modeling topic trends on the social web using temporal signatures  Chen et al (Knowledge and Data Engineering 2007)  Hot Topic Extraction Based on Timeline Analysis and Multidimensional Sentence Modeling  Tan et al. (ACM MM’10)  Topical summarization of web videos by visual-text time-dependent alignment. 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    13. 13. Our Contributions  Event-based methodology  4W paradigm (Who, What, Where, When)  Mining events from user contributed data and search trends  People-based point of view (not editorial choice)  Event discovery  Event illustration  Mining from a pool of potential media items  Choice of illustrative media item(s) 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    14. 14. Passive Second Screen Framework  Aim: Enrich & Illustrate the Event on the TV Screen  Step1: parse the semantic from query string  Step2: Extract events from Social News site  Step3: Illustrating with tag cloud and image collage Step3 Step1 query Step2 time location events topic 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    15. 15. Event Querying  Events are queried from Digg.com  a news aggregator with an editorially driven front page Two advantages: 1. Save storage and computation 2. Leverage from collective intelligence 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    16. 16. Textual Summarization  Source: Full textual search on Twitter;  query words = event title  filter by time = event occurrence date  Representation: Tag Cloud 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    17. 17. Visual Summarization  Source: Full textual search on Google Images  Query words = event title  Sorted by the similarity of image and event title  Representation: Visual Collage 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    18. 18. Event Summary 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    19. 19. Event Summary 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    20. 20. Active Second Screen Framework  Aim: Propose Enrichments which Illustrate the Event on the TV Screen  Interaction with the viewer  Multiple choices  Dynamic rendering TV / Main screen might need to be paused  Leverage on user generated content and collective intelligence 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    21. 21. Active Second Screen Framework  Goal = automatically create and illustrate a timeline of events on a given topic  Event: occurrence of abnormal activity on a limited time segment, that captured a lot of interest and triggered massive web search 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    22. 22. Trends mining  Bursts = peaks: sudden increase in the search volume => unexpected event, massive search behavior  Long-term events: abnormal high interest spread over several time units 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    23. 23. Trends mining  Granularity:  Trend value: Week (imposed by Google trends)  Trend keywords: Monthly  Feature:  Burst value= left derivative of the trends values  Threshold to select bursts:  Adaptive threshold  mean + standard deviation  Output:  Set of week dates  Objective:  link trendy time-segment to some event in the real world 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    24. 24. Focused search on time segments  Goal: search for items relevant to the event on the extracted dates  Extracting keywords for a focused search:  Input: initial search term + dates (time segment)  Query to Google trends  Output: list of associated rising search terms  Focused media search:  Private / Public Media repositories  Social Media platform 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    25. 25. Video Selection  Goal: represent each event with a media item  Features:  Using text features (from media title + description)  Discard non-English content  TF-IDF with cosine distance  Media Item Selection  Item with average similarity to all other media items 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    26. 26. Example query: Snowden 1. Get trends 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 26
    27. 27. Example query: Snowden 23rd to 29th June 2013 9th to 15th June 2013 2. Mine time segments 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 27
    28. 28. Example query: Snowden 23rd to 29th June 2013 9th to 15th June 2013 ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa 3. Extract associated keywords (month granularity) 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 28
    29. 29. Example query: Snowden 23rd to 29th June 2013 9th to 15th June 2013 ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa 3. Focused media search: Tag cloud of associated text 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 29
    30. 30. Example query: Snowden 23rd to 29th June 2013 9th to 15th June 2013 ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa 4. Illustrative video selection 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 30
    31. 31. Example query: Snowden Snowden identity was revealed on the 9th Snowden flied to Russia on the 23rd 23rd to 29th June 2013 9th to 15th June 2013 ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa ed snowden edward edward snowden edward snowden girlfriend edward snowden news edward snowden nsa 5. Ground truth 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 31
    32. 32. Experiments  Focus on people: automatic event-based timeline generation of 5 celebrities  “Ground truth” extracted from expert biographies and Wikipedia (from 2004) Beyoncé Knowles 28 events 10/21/2013 Mark Zuckerberg 27 events Oscar Pistorius 9 events Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 Batman 9 events - p 32 Edward Snowden 43 events
    33. 33. Event extraction Evaluation  Performance of our approach to detect important events  TP: true positive, TP: false positive, FN: false negative, DE: discovered event query # bursts TP FP FN DE Beyoncé K. 9 7 1 21 1 M.Zuckerberg 6 2 4 25 0 O.Pistorius 1 1 0 8 0 Batman 6 2 2 7 2 E. Snowden 2 2 0 41 0 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 33
    34. 34. Event discovery: Beyoncé  Burst week: 22nd to 28th July, 2013  Nothing in the ground truth  Associated keywords:  Beyonce fall video  Beyonce falling  Beyonce falls  Beyonce orlando  Beyonce stairs  Caida beyonce 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 34
    35. 35. Video representative evaluation  Challenge: choice of an illustrative video  Event Relatedness ER (mean over all events):  evaluation of the content of the video (not the quality) Query # events Event Relatedness Beyoncé K. 8 0,81 M. Zuckerberg 2 1 O. Pistorius 1 0 Batman 4 1 E. Snowden 2 1  Based on user generated textual features 10/21/2013 Socially Motivated Multimedia Topic Timeline Summarization - IWSAM13 - p 35
    36. 36. Example query: Pistorius  Week: 10th to 16th February, 2013 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    37. 37. Example query: Pistorius  Week: 10th to 16th February, 2013  Associated keywords: oscar pistorius girlfriend oscar pistorius news pistorius fidanzata pistorius girlfriend pistorius news reeva reeva pistorius reeva steenkamp 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    38. 38. Example query: Pistorius  Week: 10th to 16th February, 2013  Associated keywords:  DataSet tag cloud: 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking oscar pistorius girlfriend oscar pistorius news pistorius fidanzata pistorius girlfriend pistorius news reeva reeva pistorius reeva steenkamp
    39. 39. Example query: Pistorius  Week: 10th to 16th February, 2013  Associated keywords:  Dataset tag cloud: oscar pistorius girlfriend oscar pistorius news pistorius fidanzata pistorius girlfriend pistorius news reeva reeva pistorius reeva steenkamp  Illustrative video:  Nothing about Pistorius himself or the murder case  One of his opponents 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    40. 40. Example query: Pistorius  Week: 10th to 16th February, 2013  Associated keywords:  Dataset tag cloud: oscar pistorius girlfriend oscar pistorius news pistorius fidanzata pistorius girlfriend pistorius news reeva reeva pistorius reeva steenkamp  Illustrative video:  Why: associated text BBC News -Paralympic Star Oscar Pistorius 'shoots girlfriend' 2013 Inscreva-se em nosso canal. Subscribe yourself to our channel. Suscríbase a nuestro canal, News for Oscar Pistorius girlfriend shot dead Oscar Pistorius charged with murder after girlfriend shot dead The Guardian ?- 17 hours ago South African athlete will appear in court on Friday after Reeva Steenkamp was found dead at his home in Pretoria. Pistorius' girlfriend killed on Valentine's Day she was looking forward to CNN? - 15 hours ago South African sprinter Oscar Pistorius charged with murder after girlfriend, Reeva Steenkamp, shot dead in his home New York Daily News? - 6 hours ago Pistorius 'shot, killed' girlfriend - The Courier-Mail www.couriermail.com.au/.../ 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    41. 41. Conclusion  Socially Aware Hyperlinking Approaches  Automated Event-based Illustration of topics  Creation of an Event-based media-illustrated timeline related to a topic  “new” Events Discovery = non-official but triggered massive attention/searches  Future work:  Work with visual features, ASR.  Adding facial information  Work on long-term events 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    42. 42. References  http://www.linkedtv.eu  http://www.mediamixer.eu  M. Sahuguet and B. Huet, “Mining the Web for Multimedia-based Enriching”, Multimedia Modeling 2014, Dublin, Ireland  X. Liu and B. Huet “Event Representation and Visualization from Social Media”, Pacific Rim Conference on Multimedia 2013, Dec 13-16, Nanjing, China  M. Sahuguet and B. Huet, “Socially motivated multimedia topic timeline summarization”, 2nd ACM International Workshop on Socially-Aware Multimedia, In conjunction with ACM Multimedia 2013, 21 October 2013, Barcelona, Spain  X. Liu and B. Huet “On the automatic online collection of training data for visual event modeling” Multimedia Tools and Applications, February 2013, ISSN: 1380-7501 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking
    43. 43. 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking 43
    44. 44. Thank you for your attention  Questions? Benoit Huet EURECOM tel: +33 (0)493008179 Campus SophiaTech, 450 route des Chappes fax: +33 (0)493008200 06410 Biot Sophia Antipolis email: Benoit.Huet@eurecom.fr France http://www.eurecom.fr/en/people/huet-benoit 09/10/2013 Benoit HUET - Event-based Summarization for Media Hyperlinking

    ×