SlideShare a Scribd company logo
1 of 16
Maria Eskevich, Quoc-Minh Bui, Hoang-An Le,
Benoit Huet
Multimedia Department
EURECOM
Sophia-Antipolis, France
Exploring Video Hyperlinking
in Broadcast Media
Motivation
 CURRENTLY: Constantly growing quantity of
multimedia content produced by both
professionals and individual users.
 NEED: navigation systems that allow access to this
data on different levels of granularity in order to
contribute to further discovery of a topic of interest
for the user or to facilitate individual user browsing
within a collection.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 2
Task and Challenges
 We envisage the users to be interested in creating their
own path within the multimedia collection based on
their level of awareness/knowledge of it, and tasks.
 How do we create a hyperlinked collection to allow
each individual user to follow their own path of interest
within?
 Why is it challenging?
 Lack of interpretability of the content:
Overall textual description on the video level which leads to:
Retrieval process is based on text features, while the rich multimodality of
videos is not exploited;
Not possible to partially retrieve a video, a specific fragment without having
to watch the entire video.
 Archives are not static:
Need for dynamic creation of links
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 3
Insights in Hyperlinking
 Hyperlinking
Creating “links” between media
 Video Hyperlinking
video to video
video fragment to video fragment
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 4
Related experiments: Search and
Hyperlinking @ MediaEval/TRECVid
 Different techniques based on:
 Video segmentation method:
 Fixed-length units (same length, sentences, speech segments) with
further adjustment to match full sentences, using speech segment
boundaries and pauses. they build a probabilistic framework to
model the importance of words and refine segment boundaries
accordingly;
 Meaningful segments, based on topics derived from transcripts:
classification trees to define the starting and ending times of these
segments; lexical cohesion within segment.
 Features used for retrieval:
 Text similarity (vector-based models and TF-IDF weightings);
 Named entities or synonyms;
 Visual information: visual concepts or SURF and SIFT features,
detected at the shot level.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 5
Search and Hyperlinking @
MediaEval/TRECVid
 Our solution:
 Scene segmentation technique based on visual
and temporal coherence of the video segments
that constitute the scenes of the video.
 Experiment with hybrid segmentation
Visual, Topic and Temporal Coherence
 Use visual features to improve the ranking of
results of the hyperlinking including visual
analysis during the search process.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 6
System overview
Broadcast Media: BBC content
Manually transcribed
subtitles
Metadata:
title, cast, description, broadcast time
Shot segmentation,
keyframes
Visual content analysis:
Scene segmentation
Concept detection
on shot level
Lucene/Solr:
Indexing/Retrieval on shot level
Media database:
76 214 fragments
Webservice:
(HTML5/AJAX/PHP)
Videos:
2323 items/1697 hours
User Interface
Shot segmentation,
keyframe extraction
Result list:
Media fragments level
151 Visual Concepts (TrecVid 2012)
 3_Or_More_People
 Actor
 Adult
 Adult_Female_Human
 Adult_Male_Human
 Airplane
 Airplane_Flying
 Airport_Or_Airfield
 Anchorperson
 Animal
 Animation_Cartoon
 Armed_Person
 Athlete
 Baby
 Baseball
 Basketball
 Beach
 Bicycles
 Bicycling
 Birds
 Boat_Ship
 Boy
• Building
• Bus
• Car
• Car_Racing
• Cats
• Cattle
• Chair
• Charts
• Child
• Church
• City
• Cityscape
• Classroom
• Clouds
• Construction_Vehicles
• Court
• Crowd
• Dancing
• Daytime_Outdoor
• Demonstration_Or_Protest
• Desert
• Dogs
• Emergency_Vehicles
• Explosion_Fire
• Face
• Factory
• Female-Human-Face-Closeup
• Female_Anchor
• Female_Human_Face
• Female_Person
• Female_Reporter
• Fields
• Flags
• Flowers
• Football
• Forest
• Girl
• Golf
• Graphic
• Greeting
• Ground_Combat
• Gun
• Handshaking
• Harbors
• Helicopter_Hovering
• Helicopters
• Highway
• Hill
• Hockey
• Horse
• Hospital
• Human_Young_Adult
• Indoor
• Insect
• Kitchen
• Laboratory
• Landscape
• Machine_Guns
• Male-Human-Face-Closeup
• Male_Anchor
• Male_Human_Face
• Male_Person
• Male_Reporter
• Man_Wearing_A_Suit
• Maps
• Meeting
• …
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 8
Scene segmentation
 Scene : group of shots based on content similarity
and temporal consistency among shots;
 Content similarity : visual similarity between HSV
histograms extracted from the keyframes of
different shots;
 Grouping is performed using two extensions of the
Scene Transition Graph (STG):
 reduces the computational cost of STG-based shot grouping
by considering shot linking transitivity,
 builds on the former to construct a probabilistic framework that
alleviates the need for manual STG parameter selection.
[Apostolidis et al. “Automatic fine-grained hyperlinking of videos within a closed
collection using scene segmentation.” ACM MM 2014]
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 9
Handling Visual Concepts in Solr
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 10
http://localhost:8983/solr/collection_mediaEval/select?q=text:(Children
out on poetry trip Exploration of poetry by school children Poem writing)
Animal:[0.2 TO 1] Building:[0.2 TO 1]
 Schema = structure of document using fields of
different types
 Query
<doc>
<field name="id"> 20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of#t=399,402</field>
<field name="begin">00:06:39.644</field>
<field name="end">00:06:42.285</field>
<field name="videoId">20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of</field>
<field name="subtitle">'It was very, very successful.'</field>
<field name="Actor">0.143</field>
<field name="Adult">0.239</field>
<field name="Animal">0.0572</field>
</doc>
Results (S&H 2014)
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 11
Results discussion
 Text (Audio) performs best on the 2014
MediaEval Hyperlinking task
 Using visual features isn’t straightforward
 The ‘MoreLikeThis’ approach is outperformed
by the Text only search
Possibly due to query formulation differences
Sentences vs Keywords
 Visual Scenes outperform other fragmentation
levels (Topic, Sentences)
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 12
[Safadi et al. “When textual and visual information join forces
for multimedia retrieval”, ICMR 2014]
DEMO
 Hyper Video Browser
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 13
Search/Browse Hyperlinking
Conclusion and Future work
 We proposed and evaluated an approach to include
visual properties in the search of video segments for
hyperlinking.
 Experimental results show that :
 Visual properties provide meaningful cues for segmenting the video
 Mapping text-based queries to visual concepts is not easy
 Automatic selection of relevant concepts is required (human
intervention is impractical and does not necessarily lead to perfect
results)
 Operating at the keyword level rather than full text offers
improvements
 Current/Future work: incorporate query semantics
when identifying key visual semantic concepts
 based on named entity recognition approaches and keyword/visual
concepts co-occurrences
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 14
References
 E. Apostolidis, V. Mezaris, M. Sahuguet, B. Huet, B. Cervenkova, D. Stein, S. Eickeler, J.
L. Redondo Garcia,R. Troncy, and L. Pikora. “Automatic ne-grained hyperlinking of videos
within a closed collection using scene segmentation”. In ACMMM 2014, 22nd ACM
International Conference on Multimedia, Orlando, Florida, USA, 11 2014.
 H. Le, Q. Bui, B. Huet, B. Cervenkova, J. Bouchner, E. Apostolidis, F. Markatopoulou, A.
Pournaras, V. Mezaris, D. Stein, S. Eickeler, and M. Stadtschnitzer,.“LinkedTV at
MediaEval 2014 Search and Hyperlinking Task”. In Proceedings of MediaEval 2014
Workshop, 2014.
 R. J. F. Ordelman, M. Eskevich, R. Aly, B. Huet, and G. J. F. Jones. “Dening and
evaluating video hyperlinking for navigating multimedia archives”. In Proceedings of the
24th International Conference on World Wide Web Companion, WWW 2015, Florence,
Italy, May 18-22, 2015 - Companion Volume, pages 727-732, 2015.
 M. Sahuguet, B. Huet, B. Cervenkova, E. Apostolidis, V. Mezaris, D. Stein, S. Eickeler, J.
L. Redondo Garcia, and L. Pikora. “LinkedTV at MediaEval 2013 search and hyperlinking
task”. In MediaEval 2013 Workshop, Barcelona, Spain, 10 2013.
 B. Safadi, M. Sahuguet, and B. Huet. “When textual and visual information join forces for
multimedia retrieval”. In Proceedings of International Conference on Multimedia
Retrieval (ICMR '14). ACM, New York, NY, USA, , Pages 265 , 8 pages, 2014.
10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 15
Questions?
Thank you for your
attention!

More Related Content

Recently uploaded

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKJago de Vreede
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusZilliz
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024The Digital Insurer
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfOverkill Security
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 

Recently uploaded (20)

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdf
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 

Featured

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by HubspotMarius Sescu
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTExpeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 

Featured (20)

2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot2024 State of Marketing Report – by Hubspot
2024 State of Marketing Report – by Hubspot
 
Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 

Exploring Video Hyperlinking in Broadcast Media

  • 1. Maria Eskevich, Quoc-Minh Bui, Hoang-An Le, Benoit Huet Multimedia Department EURECOM Sophia-Antipolis, France Exploring Video Hyperlinking in Broadcast Media
  • 2. Motivation  CURRENTLY: Constantly growing quantity of multimedia content produced by both professionals and individual users.  NEED: navigation systems that allow access to this data on different levels of granularity in order to contribute to further discovery of a topic of interest for the user or to facilitate individual user browsing within a collection. 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 2
  • 3. Task and Challenges  We envisage the users to be interested in creating their own path within the multimedia collection based on their level of awareness/knowledge of it, and tasks.  How do we create a hyperlinked collection to allow each individual user to follow their own path of interest within?  Why is it challenging?  Lack of interpretability of the content: Overall textual description on the video level which leads to: Retrieval process is based on text features, while the rich multimodality of videos is not exploited; Not possible to partially retrieve a video, a specific fragment without having to watch the entire video.  Archives are not static: Need for dynamic creation of links 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 3
  • 4. Insights in Hyperlinking  Hyperlinking Creating “links” between media  Video Hyperlinking video to video video fragment to video fragment 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 4
  • 5. Related experiments: Search and Hyperlinking @ MediaEval/TRECVid  Different techniques based on:  Video segmentation method:  Fixed-length units (same length, sentences, speech segments) with further adjustment to match full sentences, using speech segment boundaries and pauses. they build a probabilistic framework to model the importance of words and refine segment boundaries accordingly;  Meaningful segments, based on topics derived from transcripts: classification trees to define the starting and ending times of these segments; lexical cohesion within segment.  Features used for retrieval:  Text similarity (vector-based models and TF-IDF weightings);  Named entities or synonyms;  Visual information: visual concepts or SURF and SIFT features, detected at the shot level. 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 5
  • 6. Search and Hyperlinking @ MediaEval/TRECVid  Our solution:  Scene segmentation technique based on visual and temporal coherence of the video segments that constitute the scenes of the video.  Experiment with hybrid segmentation Visual, Topic and Temporal Coherence  Use visual features to improve the ranking of results of the hyperlinking including visual analysis during the search process. 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 6
  • 7. System overview Broadcast Media: BBC content Manually transcribed subtitles Metadata: title, cast, description, broadcast time Shot segmentation, keyframes Visual content analysis: Scene segmentation Concept detection on shot level Lucene/Solr: Indexing/Retrieval on shot level Media database: 76 214 fragments Webservice: (HTML5/AJAX/PHP) Videos: 2323 items/1697 hours User Interface Shot segmentation, keyframe extraction Result list: Media fragments level
  • 8. 151 Visual Concepts (TrecVid 2012)  3_Or_More_People  Actor  Adult  Adult_Female_Human  Adult_Male_Human  Airplane  Airplane_Flying  Airport_Or_Airfield  Anchorperson  Animal  Animation_Cartoon  Armed_Person  Athlete  Baby  Baseball  Basketball  Beach  Bicycles  Bicycling  Birds  Boat_Ship  Boy • Building • Bus • Car • Car_Racing • Cats • Cattle • Chair • Charts • Child • Church • City • Cityscape • Classroom • Clouds • Construction_Vehicles • Court • Crowd • Dancing • Daytime_Outdoor • Demonstration_Or_Protest • Desert • Dogs • Emergency_Vehicles • Explosion_Fire • Face • Factory • Female-Human-Face-Closeup • Female_Anchor • Female_Human_Face • Female_Person • Female_Reporter • Fields • Flags • Flowers • Football • Forest • Girl • Golf • Graphic • Greeting • Ground_Combat • Gun • Handshaking • Harbors • Helicopter_Hovering • Helicopters • Highway • Hill • Hockey • Horse • Hospital • Human_Young_Adult • Indoor • Insect • Kitchen • Laboratory • Landscape • Machine_Guns • Male-Human-Face-Closeup • Male_Anchor • Male_Human_Face • Male_Person • Male_Reporter • Man_Wearing_A_Suit • Maps • Meeting • … 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 8
  • 9. Scene segmentation  Scene : group of shots based on content similarity and temporal consistency among shots;  Content similarity : visual similarity between HSV histograms extracted from the keyframes of different shots;  Grouping is performed using two extensions of the Scene Transition Graph (STG):  reduces the computational cost of STG-based shot grouping by considering shot linking transitivity,  builds on the former to construct a probabilistic framework that alleviates the need for manual STG parameter selection. [Apostolidis et al. “Automatic fine-grained hyperlinking of videos within a closed collection using scene segmentation.” ACM MM 2014] 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 9
  • 10. Handling Visual Concepts in Solr 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 10 http://localhost:8983/solr/collection_mediaEval/select?q=text:(Children out on poetry trip Exploration of poetry by school children Poem writing) Animal:[0.2 TO 1] Building:[0.2 TO 1]  Schema = structure of document using fields of different types  Query <doc> <field name="id"> 20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of#t=399,402</field> <field name="begin">00:06:39.644</field> <field name="end">00:06:42.285</field> <field name="videoId">20080401_013000_bbcfour_legends_marty_feldman_six_degrees_of</field> <field name="subtitle">'It was very, very successful.'</field> <field name="Actor">0.143</field> <field name="Adult">0.239</field> <field name="Animal">0.0572</field> </doc>
  • 11. Results (S&H 2014) 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 11
  • 12. Results discussion  Text (Audio) performs best on the 2014 MediaEval Hyperlinking task  Using visual features isn’t straightforward  The ‘MoreLikeThis’ approach is outperformed by the Text only search Possibly due to query formulation differences Sentences vs Keywords  Visual Scenes outperform other fragmentation levels (Topic, Sentences) 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 12 [Safadi et al. “When textual and visual information join forces for multimedia retrieval”, ICMR 2014]
  • 13. DEMO  Hyper Video Browser 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 13 Search/Browse Hyperlinking
  • 14. Conclusion and Future work  We proposed and evaluated an approach to include visual properties in the search of video segments for hyperlinking.  Experimental results show that :  Visual properties provide meaningful cues for segmenting the video  Mapping text-based queries to visual concepts is not easy  Automatic selection of relevant concepts is required (human intervention is impractical and does not necessarily lead to perfect results)  Operating at the keyword level rather than full text offers improvements  Current/Future work: incorporate query semantics when identifying key visual semantic concepts  based on named entity recognition approaches and keyword/visual concepts co-occurrences 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 14
  • 15. References  E. Apostolidis, V. Mezaris, M. Sahuguet, B. Huet, B. Cervenkova, D. Stein, S. Eickeler, J. L. Redondo Garcia,R. Troncy, and L. Pikora. “Automatic ne-grained hyperlinking of videos within a closed collection using scene segmentation”. In ACMMM 2014, 22nd ACM International Conference on Multimedia, Orlando, Florida, USA, 11 2014.  H. Le, Q. Bui, B. Huet, B. Cervenkova, J. Bouchner, E. Apostolidis, F. Markatopoulou, A. Pournaras, V. Mezaris, D. Stein, S. Eickeler, and M. Stadtschnitzer,.“LinkedTV at MediaEval 2014 Search and Hyperlinking Task”. In Proceedings of MediaEval 2014 Workshop, 2014.  R. J. F. Ordelman, M. Eskevich, R. Aly, B. Huet, and G. J. F. Jones. “Dening and evaluating video hyperlinking for navigating multimedia archives”. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume, pages 727-732, 2015.  M. Sahuguet, B. Huet, B. Cervenkova, E. Apostolidis, V. Mezaris, D. Stein, S. Eickeler, J. L. Redondo Garcia, and L. Pikora. “LinkedTV at MediaEval 2013 search and hyperlinking task”. In MediaEval 2013 Workshop, Barcelona, Spain, 10 2013.  B. Safadi, M. Sahuguet, and B. Huet. “When textual and visual information join forces for multimedia retrieval”. In Proceedings of International Conference on Multimedia Retrieval (ICMR '14). ACM, New York, NY, USA, , Pages 265 , 8 pages, 2014. 10/30/15 SLAM Workshop @ ACM MM 2015, Brisbane, Australia 15
  • 16. Questions? Thank you for your attention!

Editor's Notes

  1. MLT1 -> text only MLT2 -> text + annotations