SlideShare a Scribd company logo
1 of 20
Download to read offline
The MIRROR project has received funding from the European Union’s Horizon 2020 research and innovation action program under grant agreement № 832921.
Migration-Related Semantic
Concepts for the Retrieval of
Relevant Video Content
Erick Elejalde, Damianos Galanopoulos, Claudia Niederee, Vasileios Mezaris
Int. Workshop on Artificial Intelligence and Robotics for Law Enforcement
Agencies (AIRLEAs) @ 3rd Int. Conf. on Intelligent Technologies and
Applications (INTAP 2020), Gjovik, Norway, Sept. 2020.
2
Introduction
Problem
● Migration is a complex process and a critical issue
● A plethora of factors lead to migration decisions
● Social media may be used to manipulate perception and lead to misperceptions
Solutions
● A better understanding of these decisions is critical
● Automatic analysis of migration related media items
● A novel approach to bridge the gap between the migration driven factors and
their expressions in a video
3
Approach
Top-down and bottom-up approach combination
● Top-down approach
○ Theoretical understanding of migration factors and decisions
○ Domain conceptualization
○ A set of Migration-Related Semantic Concepts (MRSCs) is defined
● Bottom-up approach
○ Visual content interpretation and analysis
○ Video analysis for retrieving related video or images
○ How are MRSCs expressed in videos and images?
4
Migration-Related Semantic Concepts (MRSCs)
What are MRSCs?
● Semantic concepts relevant in the context of migration
How are MRSCs defined?
● In-depth study of migration theories
● Discussions with domain experts
● Semantic concepts collection expresses the migration aspects
● Based on three popular theoretical approaches
○ Νeo-classical economic equilibrium
○ Historical-structural approach
○ Migration systems theory
5
MRSC - Migration Theories
Νeo-classical economic equilibrium
● Focuses on imbalance conditions between origin country and destination
● People try to maximize their benefits take into consideration any constraints
● Criticism: ignores the historical antecedents of movements and ignoring the
role of the state
6
MRSC - Migration Theories
Historical-structural approach
● Based on the Marxist view of political economy
● Stresses the unequal distribution of the economy
● Global scaled recruitment of cheap labor from the capital
● Uneven economy development maintenance
● Criticism: no attention to personal motivations
7
MRSC - Migration Theories
Migration systems theory
● Response to other theories criticism
● More holistic analysis of the migration factors
● The migration process is the result of interacting macro-, meso- and
microstructures
8
MRSC - Factors Classification
● Semantic concepts are combined to form meaningful templates
○ For example, “family” and “war” combined as “Families in war”
● Based on such patterns, a hierarchical structure is constructed
● 106 MRSCs are grouped into five categories
○ Economic
○ Social
○ Demographic
○ Environmental
○ Political
● 20 on the first level and 86 under them
9
MRSC - Factor Classification
106 MRSCs organized on five categories and two levels
10
MRSC-based Video Retrieval
● Ad-Hoc Video Search (AVS) is a similar cross-modal retrieval problem
● Based on a state-of-the-art method for the AVS problem
● Video shots and free text encoding into a joint feature space
● Retrieve the most relevant video shots by inputting an MRSC
● Attention-based dual encoding network
● Trained with video-caption pairs
● MRSCs augmentation with a small set of complex sentences
11
MRSC-based Video Retrieval
Method overview
12
MRSC-based Video Retrieval
Attention-based dual encoding network adjustment
13
Experiments and Results
Experimental set-up
● Training datasets
○ TGIF & MSR-VTT
● Keyframe representation
○ ResNet 152 trained on Imagenet 11K
● Word embeddings
○ Word2Vec
○ BERT
● Evaluation datasets
○ TRECVID SIN 2013 & 2015
● Evaluation metric
○ Mean extended inferred average precision (MXinfAP)
14
Experiments and Results
Why TRECVID SIN as an evaluation dataset?
● Excessively specific problem
● No domain-specific video retrieval datasets are available
● SIN task is very similar to ours
● Abstract concepts to be correlated with video shots
● Well known state-of-the-art methods to be compared
15
Experiments and Results
16
Experiments and Results
SIN concept augmentation improvements
● “Telephones” is described as “speaking on a telephone” and “talking on a
telephone”
○ SIN’13: XinfAP improved from 0.0 to 0.3151
○ SIN’15: XinfAP improved from 0.0 to 0.308
● “Bicycling” is described as “a man riding a bike”, “people riding bicycles”
and “a woman on a bike”
○ SIN’15: XinfAP improved from 0.0569 to 0.3730
17
Experiments and Results
● Conventional concept retrieval methods are used as the baseline
● Baselines use predefined sets of visual concepts and positive exemplars for
every concept
● Competitive results even with the absence of training exemplars
18
Experiments and Results
● Visual examples
19
Conclusion and Future work
● A novel approach for understanding migration and migration decisions
● Theoretically defined MRSCs provides
○ A better view of the migration topic
○ A common language for analysis
● Video analysis to bridge the gap between MRSCs and video
Future work
● Fully automatic pipeline for automatic MRSCs augmentation
● Better encoding and improved visual and text representations
● Domain-specific dataset
20
Contact details
Dr. Vasileios Mezaris
Information Technologies Institute-CERTH
bmezaris@iti.gr
www.iti.gr/~bmezaris

More Related Content

Similar to Migration-related video retrieval

QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018
QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018
QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018Nacho Foche Pérez
 
The millennium project overview
The millennium project overviewThe millennium project overview
The millennium project overviewJerome Glenn
 
Keynote, Oman Geospatial Expo, Dec 2013
Keynote, Oman Geospatial Expo, Dec 2013Keynote, Oman Geospatial Expo, Dec 2013
Keynote, Oman Geospatial Expo, Dec 2013Steven Ramage
 
Enhancing transparency lessons learned from capacity development, Rebecca Carman
Enhancing transparency lessons learned from capacity development, Rebecca CarmanEnhancing transparency lessons learned from capacity development, Rebecca Carman
Enhancing transparency lessons learned from capacity development, Rebecca CarmanOECD Environment
 
2015-04-01 research seminar
2015-04-01 research seminar2015-04-01 research seminar
2015-04-01 research seminarifi8106tlu
 
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...Hugo Bruneliere
 
How to Deliver Data Insights by Fmr Google Sr Analytical Lead
How to Deliver Data Insights by Fmr Google Sr Analytical LeadHow to Deliver Data Insights by Fmr Google Sr Analytical Lead
How to Deliver Data Insights by Fmr Google Sr Analytical LeadProduct School
 
Better Living Through Analytics - Louis Cialdella Product School
Better Living Through Analytics - Louis Cialdella Product SchoolBetter Living Through Analytics - Louis Cialdella Product School
Better Living Through Analytics - Louis Cialdella Product SchoolLouis Cialdella
 
Perspectives on ISGAN KTP in Mexico, Genk and Bengaluru
Perspectives on ISGAN KTP in Mexico, Genk and BengaluruPerspectives on ISGAN KTP in Mexico, Genk and Bengaluru
Perspectives on ISGAN KTP in Mexico, Genk and BengaluruLeonardo ENERGY
 
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...Joshua Engel-Yan
 
Foresight for SD GS - Saritas
Foresight for SD GS - SaritasForesight for SD GS - Saritas
Foresight for SD GS - SaritasSTS FORUM 2016
 
National cybersecurity capacity building framework for countries in a transit...
National cybersecurity capacity building framework for countries in a transit...National cybersecurity capacity building framework for countries in a transit...
National cybersecurity capacity building framework for countries in a transit...Mohamed Ben Naseir
 
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slides
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slidesSTDM-Online_The-Social-Tenure-Domain-Model-Tool_slides
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slidesArnulf Christl
 
Iconference 2018 stiller trkulja-digital literacy session-27-03
Iconference 2018 stiller trkulja-digital literacy session-27-03Iconference 2018 stiller trkulja-digital literacy session-27-03
Iconference 2018 stiller trkulja-digital literacy session-27-03Juliane Stiller
 
Introduction to Global Software Engineering TDT4140
Introduction to Global Software Engineering TDT4140Introduction to Global Software Engineering TDT4140
Introduction to Global Software Engineering TDT4140Anh Nguyen Duc
 

Similar to Migration-related video retrieval (20)

QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018
QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018
QuestMark Framework for Dhis2 Android Apps - Dhis2 symposium 2018
 
The millennium project overview
The millennium project overviewThe millennium project overview
The millennium project overview
 
Technology Acceptance of Virtual Reality for Travel Planning
Technology Acceptance of Virtual Reality for Travel PlanningTechnology Acceptance of Virtual Reality for Travel Planning
Technology Acceptance of Virtual Reality for Travel Planning
 
GFCE 2019 Presentation (Updated Nov 2019)
GFCE 2019 Presentation (Updated Nov 2019)GFCE 2019 Presentation (Updated Nov 2019)
GFCE 2019 Presentation (Updated Nov 2019)
 
Keynote, Oman Geospatial Expo, Dec 2013
Keynote, Oman Geospatial Expo, Dec 2013Keynote, Oman Geospatial Expo, Dec 2013
Keynote, Oman Geospatial Expo, Dec 2013
 
Enhancing transparency lessons learned from capacity development, Rebecca Carman
Enhancing transparency lessons learned from capacity development, Rebecca CarmanEnhancing transparency lessons learned from capacity development, Rebecca Carman
Enhancing transparency lessons learned from capacity development, Rebecca Carman
 
2015-04-01 research seminar
2015-04-01 research seminar2015-04-01 research seminar
2015-04-01 research seminar
 
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...
Model Driven Engineering for Design-Runtime Interaction in Complex Systems: S...
 
Public defence_v3
Public defence_v3Public defence_v3
Public defence_v3
 
How to Deliver Data Insights by Fmr Google Sr Analytical Lead
How to Deliver Data Insights by Fmr Google Sr Analytical LeadHow to Deliver Data Insights by Fmr Google Sr Analytical Lead
How to Deliver Data Insights by Fmr Google Sr Analytical Lead
 
Better Living Through Analytics - Louis Cialdella Product School
Better Living Through Analytics - Louis Cialdella Product SchoolBetter Living Through Analytics - Louis Cialdella Product School
Better Living Through Analytics - Louis Cialdella Product School
 
Perspectives on ISGAN KTP in Mexico, Genk and Bengaluru
Perspectives on ISGAN KTP in Mexico, Genk and BengaluruPerspectives on ISGAN KTP in Mexico, Genk and Bengaluru
Perspectives on ISGAN KTP in Mexico, Genk and Bengaluru
 
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...
Telling the Story: Transforming Data & Evidence into a Compelling Narrative t...
 
Ml conference slides
Ml conference slidesMl conference slides
Ml conference slides
 
Foresight for SD GS - Saritas
Foresight for SD GS - SaritasForesight for SD GS - Saritas
Foresight for SD GS - Saritas
 
National cybersecurity capacity building framework for countries in a transit...
National cybersecurity capacity building framework for countries in a transit...National cybersecurity capacity building framework for countries in a transit...
National cybersecurity capacity building framework for countries in a transit...
 
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slides
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slidesSTDM-Online_The-Social-Tenure-Domain-Model-Tool_slides
STDM-Online_The-Social-Tenure-Domain-Model-Tool_slides
 
Toward supporting decision-making under uncertainty in digital humanities wit...
Toward supporting decision-making under uncertainty in digital humanities wit...Toward supporting decision-making under uncertainty in digital humanities wit...
Toward supporting decision-making under uncertainty in digital humanities wit...
 
Iconference 2018 stiller trkulja-digital literacy session-27-03
Iconference 2018 stiller trkulja-digital literacy session-27-03Iconference 2018 stiller trkulja-digital literacy session-27-03
Iconference 2018 stiller trkulja-digital literacy session-27-03
 
Introduction to Global Software Engineering TDT4140
Introduction to Global Software Engineering TDT4140Introduction to Global Software Engineering TDT4140
Introduction to Global Software Engineering TDT4140
 

More from VasileiosMezaris

Multi-Modal Fusion for Image Manipulation Detection and Localization
Multi-Modal Fusion for Image Manipulation Detection and LocalizationMulti-Modal Fusion for Image Manipulation Detection and Localization
Multi-Modal Fusion for Image Manipulation Detection and LocalizationVasileiosMezaris
 
CERTH-ITI at MediaEval 2023 NewsImages Task
CERTH-ITI at MediaEval 2023 NewsImages TaskCERTH-ITI at MediaEval 2023 NewsImages Task
CERTH-ITI at MediaEval 2023 NewsImages TaskVasileiosMezaris
 
Spatio-Temporal Summarization of 360-degrees Videos
Spatio-Temporal Summarization of 360-degrees VideosSpatio-Temporal Summarization of 360-degrees Videos
Spatio-Temporal Summarization of 360-degrees VideosVasileiosMezaris
 
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...VasileiosMezaris
 
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022VasileiosMezaris
 
TAME: Trainable Attention Mechanism for Explanations
TAME: Trainable Attention Mechanism for ExplanationsTAME: Trainable Attention Mechanism for Explanations
TAME: Trainable Attention Mechanism for ExplanationsVasileiosMezaris
 
Explaining video summarization based on the focus of attention
Explaining video summarization based on the focus of attentionExplaining video summarization based on the focus of attention
Explaining video summarization based on the focus of attentionVasileiosMezaris
 
Combining textual and visual features for Ad-hoc Video Search
Combining textual and visual features for Ad-hoc Video SearchCombining textual and visual features for Ad-hoc Video Search
Combining textual and visual features for Ad-hoc Video SearchVasileiosMezaris
 
Explaining the decisions of image/video classifiers
Explaining the decisions of image/video classifiersExplaining the decisions of image/video classifiers
Explaining the decisions of image/video classifiersVasileiosMezaris
 
Learning visual explanations for DCNN-based image classifiers using an attent...
Learning visual explanations for DCNN-based image classifiers using an attent...Learning visual explanations for DCNN-based image classifiers using an attent...
Learning visual explanations for DCNN-based image classifiers using an attent...VasileiosMezaris
 
Are all combinations equal? Combining textual and visual features with multi...
Are all combinations equal?  Combining textual and visual features with multi...Are all combinations equal?  Combining textual and visual features with multi...
Are all combinations equal? Combining textual and visual features with multi...VasileiosMezaris
 
CA-SUM Video Summarization
CA-SUM Video SummarizationCA-SUM Video Summarization
CA-SUM Video SummarizationVasileiosMezaris
 
Video smart cropping web application
Video smart cropping web applicationVideo smart cropping web application
Video smart cropping web applicationVasileiosMezaris
 
PGL SUM Video Summarization
PGL SUM Video SummarizationPGL SUM Video Summarization
PGL SUM Video SummarizationVasileiosMezaris
 
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal RetrievalHard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal RetrievalVasileiosMezaris
 
Misinformation on the internet: Video and AI
Misinformation on the internet: Video and AIMisinformation on the internet: Video and AI
Misinformation on the internet: Video and AIVasileiosMezaris
 
PoR_evaluation_measure_acm_mm_2020
PoR_evaluation_measure_acm_mm_2020PoR_evaluation_measure_acm_mm_2020
PoR_evaluation_measure_acm_mm_2020VasileiosMezaris
 

More from VasileiosMezaris (20)

Multi-Modal Fusion for Image Manipulation Detection and Localization
Multi-Modal Fusion for Image Manipulation Detection and LocalizationMulti-Modal Fusion for Image Manipulation Detection and Localization
Multi-Modal Fusion for Image Manipulation Detection and Localization
 
CERTH-ITI at MediaEval 2023 NewsImages Task
CERTH-ITI at MediaEval 2023 NewsImages TaskCERTH-ITI at MediaEval 2023 NewsImages Task
CERTH-ITI at MediaEval 2023 NewsImages Task
 
Spatio-Temporal Summarization of 360-degrees Videos
Spatio-Temporal Summarization of 360-degrees VideosSpatio-Temporal Summarization of 360-degrees Videos
Spatio-Temporal Summarization of 360-degrees Videos
 
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...
Masked Feature Modelling for the unsupervised pre-training of a Graph Attenti...
 
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022
Cross-modal Networks and Dual Softmax Operation for MediaEval NewsImages 2022
 
TAME: Trainable Attention Mechanism for Explanations
TAME: Trainable Attention Mechanism for ExplanationsTAME: Trainable Attention Mechanism for Explanations
TAME: Trainable Attention Mechanism for Explanations
 
Gated-ViGAT
Gated-ViGATGated-ViGAT
Gated-ViGAT
 
Explaining video summarization based on the focus of attention
Explaining video summarization based on the focus of attentionExplaining video summarization based on the focus of attention
Explaining video summarization based on the focus of attention
 
Combining textual and visual features for Ad-hoc Video Search
Combining textual and visual features for Ad-hoc Video SearchCombining textual and visual features for Ad-hoc Video Search
Combining textual and visual features for Ad-hoc Video Search
 
Explaining the decisions of image/video classifiers
Explaining the decisions of image/video classifiersExplaining the decisions of image/video classifiers
Explaining the decisions of image/video classifiers
 
Learning visual explanations for DCNN-based image classifiers using an attent...
Learning visual explanations for DCNN-based image classifiers using an attent...Learning visual explanations for DCNN-based image classifiers using an attent...
Learning visual explanations for DCNN-based image classifiers using an attent...
 
Are all combinations equal? Combining textual and visual features with multi...
Are all combinations equal?  Combining textual and visual features with multi...Are all combinations equal?  Combining textual and visual features with multi...
Are all combinations equal? Combining textual and visual features with multi...
 
CA-SUM Video Summarization
CA-SUM Video SummarizationCA-SUM Video Summarization
CA-SUM Video Summarization
 
Video smart cropping web application
Video smart cropping web applicationVideo smart cropping web application
Video smart cropping web application
 
PGL SUM Video Summarization
PGL SUM Video SummarizationPGL SUM Video Summarization
PGL SUM Video Summarization
 
Video Thumbnail Selector
Video Thumbnail SelectorVideo Thumbnail Selector
Video Thumbnail Selector
 
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal RetrievalHard-Negatives Selection Strategy for Cross-Modal Retrieval
Hard-Negatives Selection Strategy for Cross-Modal Retrieval
 
Misinformation on the internet: Video and AI
Misinformation on the internet: Video and AIMisinformation on the internet: Video and AI
Misinformation on the internet: Video and AI
 
LSTM Structured Pruning
LSTM Structured PruningLSTM Structured Pruning
LSTM Structured Pruning
 
PoR_evaluation_measure_acm_mm_2020
PoR_evaluation_measure_acm_mm_2020PoR_evaluation_measure_acm_mm_2020
PoR_evaluation_measure_acm_mm_2020
 

Recently uploaded

Speed Breeding in Vegetable Crops- innovative approach for present era of cro...
Speed Breeding in Vegetable Crops- innovative approach for present era of cro...Speed Breeding in Vegetable Crops- innovative approach for present era of cro...
Speed Breeding in Vegetable Crops- innovative approach for present era of cro...jana861314
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPRPirithiRaju
 
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPRPirithiRaju
 
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxEnvironmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxpriyankatabhane
 
Science (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsScience (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsDobusch Leonhard
 
Food_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyFood_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyHemantThakare8
 
Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTAlexander F. Mayer
 
Presentation about adversarial image attacks
Presentation about adversarial image attacksPresentation about adversarial image attacks
Presentation about adversarial image attacksKoshinKhodiyar
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxPayal Shrivastava
 
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaSTELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaAMiracle3
 
Think Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinThink Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinNathan Cone
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPirithiRaju
 
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀Tokyo Chemicals Industry (TCI)
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasChayanika Das
 
Production technology of Brinjal -Solanum melongena
Production technology of Brinjal -Solanum melongenaProduction technology of Brinjal -Solanum melongena
Production technology of Brinjal -Solanum melongenajana861314
 
Advances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerAdvances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerLuis Miguel Chong Chong
 
Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxpriyankatabhane
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Understanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfUnderstanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfHabibouKarbo
 

Recently uploaded (20)

Speed Breeding in Vegetable Crops- innovative approach for present era of cro...
Speed Breeding in Vegetable Crops- innovative approach for present era of cro...Speed Breeding in Vegetable Crops- innovative approach for present era of cro...
Speed Breeding in Vegetable Crops- innovative approach for present era of cro...
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
 
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
6.2 Pests of Sesame_Identification_Binomics_Dr.UPR
 
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxEnvironmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
 
Battery Reasearch Reagents from TCI Chemicals
Battery Reasearch Reagents from TCI ChemicalsBattery Reasearch Reagents from TCI Chemicals
Battery Reasearch Reagents from TCI Chemicals
 
Science (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and PitfallsScience (Communication) and Wikipedia - Potentials and Pitfalls
Science (Communication) and Wikipedia - Potentials and Pitfalls
 
Food_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyFood_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiology
 
Interpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWSTInterpreting SDSS extragalactic data in the era of JWST
Interpreting SDSS extragalactic data in the era of JWST
 
Presentation about adversarial image attacks
Presentation about adversarial image attacksPresentation about adversarial image attacks
Presentation about adversarial image attacks
 
FBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptxFBI Profiling - Forensic Psychology.pptx
FBI Profiling - Forensic Psychology.pptx
 
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By KarishmaSTELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
STELLAR SYSTEM IN PTERIDOPHYTE Seminar 2023- By Karishma
 
Think Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig BobchinThink Science: What Are Eclipses (101), by Craig Bobchin
Think Science: What Are Eclipses (101), by Craig Bobchin
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPR
 
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
𝗧𝗖𝗢 (𝙩𝙧𝙖𝙣𝙨-𝗰𝘆𝗰𝗹𝗼𝗼𝗰𝘁𝗲𝗻𝗲) 𝗗𝗲𝗿𝗶𝘃𝗮𝘁𝗶𝘃𝗲𝘀: 𝗧𝗵𝗲 𝗙𝗮𝘀𝘁𝗲𝘀𝘁 𝗖𝗹𝗶𝗰𝗸 𝗥𝗲𝗮𝗰𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗴𝗲𝗻𝘁𝘀
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
 
Production technology of Brinjal -Solanum melongena
Production technology of Brinjal -Solanum melongenaProduction technology of Brinjal -Solanum melongena
Production technology of Brinjal -Solanum melongena
 
Advances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of CancerAdvances in AI-driven Image Recognition for Early Detection of Cancer
Advances in AI-driven Image Recognition for Early Detection of Cancer
 
Loudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptxLoudspeaker- direct radiating type and horn type.pptx
Loudspeaker- direct radiating type and horn type.pptx
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Understanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfUnderstanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdf
 

Migration-related video retrieval

  • 1. The MIRROR project has received funding from the European Union’s Horizon 2020 research and innovation action program under grant agreement № 832921. Migration-Related Semantic Concepts for the Retrieval of Relevant Video Content Erick Elejalde, Damianos Galanopoulos, Claudia Niederee, Vasileios Mezaris Int. Workshop on Artificial Intelligence and Robotics for Law Enforcement Agencies (AIRLEAs) @ 3rd Int. Conf. on Intelligent Technologies and Applications (INTAP 2020), Gjovik, Norway, Sept. 2020.
  • 2. 2 Introduction Problem ● Migration is a complex process and a critical issue ● A plethora of factors lead to migration decisions ● Social media may be used to manipulate perception and lead to misperceptions Solutions ● A better understanding of these decisions is critical ● Automatic analysis of migration related media items ● A novel approach to bridge the gap between the migration driven factors and their expressions in a video
  • 3. 3 Approach Top-down and bottom-up approach combination ● Top-down approach ○ Theoretical understanding of migration factors and decisions ○ Domain conceptualization ○ A set of Migration-Related Semantic Concepts (MRSCs) is defined ● Bottom-up approach ○ Visual content interpretation and analysis ○ Video analysis for retrieving related video or images ○ How are MRSCs expressed in videos and images?
  • 4. 4 Migration-Related Semantic Concepts (MRSCs) What are MRSCs? ● Semantic concepts relevant in the context of migration How are MRSCs defined? ● In-depth study of migration theories ● Discussions with domain experts ● Semantic concepts collection expresses the migration aspects ● Based on three popular theoretical approaches ○ Νeo-classical economic equilibrium ○ Historical-structural approach ○ Migration systems theory
  • 5. 5 MRSC - Migration Theories Νeo-classical economic equilibrium ● Focuses on imbalance conditions between origin country and destination ● People try to maximize their benefits take into consideration any constraints ● Criticism: ignores the historical antecedents of movements and ignoring the role of the state
  • 6. 6 MRSC - Migration Theories Historical-structural approach ● Based on the Marxist view of political economy ● Stresses the unequal distribution of the economy ● Global scaled recruitment of cheap labor from the capital ● Uneven economy development maintenance ● Criticism: no attention to personal motivations
  • 7. 7 MRSC - Migration Theories Migration systems theory ● Response to other theories criticism ● More holistic analysis of the migration factors ● The migration process is the result of interacting macro-, meso- and microstructures
  • 8. 8 MRSC - Factors Classification ● Semantic concepts are combined to form meaningful templates ○ For example, “family” and “war” combined as “Families in war” ● Based on such patterns, a hierarchical structure is constructed ● 106 MRSCs are grouped into five categories ○ Economic ○ Social ○ Demographic ○ Environmental ○ Political ● 20 on the first level and 86 under them
  • 9. 9 MRSC - Factor Classification 106 MRSCs organized on five categories and two levels
  • 10. 10 MRSC-based Video Retrieval ● Ad-Hoc Video Search (AVS) is a similar cross-modal retrieval problem ● Based on a state-of-the-art method for the AVS problem ● Video shots and free text encoding into a joint feature space ● Retrieve the most relevant video shots by inputting an MRSC ● Attention-based dual encoding network ● Trained with video-caption pairs ● MRSCs augmentation with a small set of complex sentences
  • 12. 12 MRSC-based Video Retrieval Attention-based dual encoding network adjustment
  • 13. 13 Experiments and Results Experimental set-up ● Training datasets ○ TGIF & MSR-VTT ● Keyframe representation ○ ResNet 152 trained on Imagenet 11K ● Word embeddings ○ Word2Vec ○ BERT ● Evaluation datasets ○ TRECVID SIN 2013 & 2015 ● Evaluation metric ○ Mean extended inferred average precision (MXinfAP)
  • 14. 14 Experiments and Results Why TRECVID SIN as an evaluation dataset? ● Excessively specific problem ● No domain-specific video retrieval datasets are available ● SIN task is very similar to ours ● Abstract concepts to be correlated with video shots ● Well known state-of-the-art methods to be compared
  • 16. 16 Experiments and Results SIN concept augmentation improvements ● “Telephones” is described as “speaking on a telephone” and “talking on a telephone” ○ SIN’13: XinfAP improved from 0.0 to 0.3151 ○ SIN’15: XinfAP improved from 0.0 to 0.308 ● “Bicycling” is described as “a man riding a bike”, “people riding bicycles” and “a woman on a bike” ○ SIN’15: XinfAP improved from 0.0569 to 0.3730
  • 17. 17 Experiments and Results ● Conventional concept retrieval methods are used as the baseline ● Baselines use predefined sets of visual concepts and positive exemplars for every concept ● Competitive results even with the absence of training exemplars
  • 19. 19 Conclusion and Future work ● A novel approach for understanding migration and migration decisions ● Theoretically defined MRSCs provides ○ A better view of the migration topic ○ A common language for analysis ● Video analysis to bridge the gap between MRSCs and video Future work ● Fully automatic pipeline for automatic MRSCs augmentation ● Better encoding and improved visual and text representations ● Domain-specific dataset
  • 20. 20 Contact details Dr. Vasileios Mezaris Information Technologies Institute-CERTH bmezaris@iti.gr www.iti.gr/~bmezaris