SlideShare a Scribd company logo
1 of 27
Download to read offline
Retrieving Diverse Social Images Task
- task overview -
2017
University Politehnica
of Bucharest
Maia Zaharieva (TUW, Austria)
Bogdan Ionescu (UPB, Romania)
Alexandru Lucian Gînscǎ (CEA LIST, France)
Rodrygo L.T. Santos (UFMG, Brazil)
Henning Müller (HES-SO in Sierre, Switzerland)
Bogdan Boteanu (UPB, Romania)
September 13-15, Dublin, Irelandce
Universidade Federal de
Minas Gerais, Brazil
 The Retrieving Diverse Social Images Task
 Dataset and Evaluation
 Participants
 Results
 Discussion and Perspectives
2
Outline
3
Diversity Task: Objective & Motivation
Objective: image search result diversification in the context of
social photo retrieval.
Why diversifying search results?
- to respond to the needs of different users;
- as a method of tackling queries with unclear information needs;
- to widen the pool of possible results (increase performance);
- to reduce the number/redundancy of the returned items;
…
3
Diversity Task: Objective & Motivation #2
4
Diversity Task: Objective & Motivation #2
5
Diversity Task: Objective & Motivation #3
7
Diversity Task: Definition
For each query, participants receive a ranked list of photos retrieved
from Flickr using its default “relevance” algorithm.
Query = general-purpose, multi-topic term
e.g.: autumn colors, bee on a flower, home office, snow in
the city, holding hands, ...
Goal of the task: refine the results by providing a ranked list of up
to 50 photos (summary) that are considered to be both relevant and
diverse representations of the query.
relevant: a common photo representation of the query topics (all at once);
bad quality photos (e.g., severely blurred, out of focus) are not considered
relevant in this scenario
diverse: depicting different visual characteristics of the query topics and
subtopics with a certain degree of complementarity, i.e., most of the
perceived visual information is different from one photo to another.
8
Dataset: General Information & Resources
Provided information:
 query text formulation;
 ranked list of Creative Commons photos from Flickr*
(up to 300 photos per query);
 metadata from Flickr (e.g., tags, description, views,
comments, date-time photo was taken, username, userid, etc);
 visual, text & user annotation credibility descriptors;
 semantic vectors for general English terms computed on top of
the English Wikipedia(wikiset);
 relevance and diversity ground truth.
Photos:
Development: 110 queries 32,340 photos
Test: 84 queries 24,986 photos
9
Dataset: Provided Descriptors
General purpose visual descriptors:
 e.g., Auto Color Correlogram, Color and Edge Directivity
Descriptor, Pyramid of Histograms of Orientation Gradients, etc;
Convolutional Neural Network based descriptors:
 Caffe framework based;
General purpose text descriptors:
 e.g., term frequency information, document frequency
information and their ratio, i.e., TF-IDF;
User annotation credibility descriptors (give an automatic
estimation of the quality of users' tag-image content relationships):
 e.g., measure of user image relevance, total number of images a
user shared, the percentage of images with faces.
10
Dataset: Basic Statistics
devset
(design the methods)
testset
(final benchmarking)
#queries 110 84
#images 32,340 24,986
#img. per query
(min-average-max )
141 - 295 - 300 299 - 300 - 300
% relevant img. 53 57.4
avg. #clusters per query 17 14
avg. #img. per cluster 9 14
11
Dataset: Ground Truth - annotations
Relevance and diversity annotations were carried out by
expert annotators:
 devset:
relevance: 8 annotators + 1 master (3 annotations/query)
diversity: 1 annotation/query
 testset:
relevance: 8 annotators + 1 master (3 annotations/query)
diversity: 12 annotators (3 annotations/query)
 Lenient majority voting for relevance
12
Evaluation: Run Specification
Participants are required to submit up to 5 runs:
 required runs:
run 1: automated using visual information only;
run 2: automated using textual information only;
run 3: automated using textual-visual fused without other
resources than provided by the organizers;
 general runs:
run 4: everything allowed, e.g. human-based or hybrid human-
machine approaches, including using data from external
sources, (e.g., Internet) or pre-trained models obtained from
external datasets related to this task;
run 5: everything allowed.
13
Evaluation: Official Metrics
 Cluster Recall* @ X = Nc/N (CR@X)
where X is the cutoff point, N is the total number of clusters for the
current query (from ground truth, N<=25) and Nc is the number of
different clusters represented in the X ranked images;
*cluster recall is computed only for the relevant images.
 Precision @ X = R/X (P@X)
where R is the number of relevant images;
 F1-measure @ X = harmonic mean of CR and P (F1@X)
Metrics are reported for different values of X (5, 10, 20, 30, 40 & 50)
on per topic as well as overall (average).
official ranking F1@20
14
Participants: Basic Statistics
 Survey:
- 22 respondents were interested in the task;
 Registration:
- 14 teams registered (1 team is organizer related);
 Run submission:
- 6 teams finished the task, including 1 organizer related team;
- 29 runs were submitted;
 Workshop participation:
- 5 teams are represented at the workshop.
15
Participants: Submitted Runs (29)
*organizer related team
Team Country
Required Runs General Runs Results (best)
1 (visual) 2 (text) 3 (vis-text) 4 5 P@20 CR@20 F1@20
NLE France ✓ ✓ ✓ visual-text visual-text 0.793 0.679 0.705
MultiBrazil Brazil ✓ ✓ ✓ visual-text-cred. visual-text-cred. 0.7208 0.6524 0.6634
UMONS Belgium ✓ ✓ ✓ visual-text-cred. visual-cred. 0.8071 0.5856 0.6554
CFM China ✓ ✓ ✓ text-cred. text-cred. 0.6881 0.6671 0.6533
tud-mmc Netherlands ✓ ✓ ✓ text-intent ✗ 0.7262 0.6142 0.6462
Flickr 0.6595 0.5831 0.5922
LAPI* Romania ✓ ✓ ✓ visual cred. 0.633 0.6045 0.5777
16
Results: P vs. CR @20 (all runs - testset)
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.5 0.55 0.6 0.65 0.7
P@20
CR@20
Flickr Initial
CFM
LAPI
MultiBrazil
NLE
tud-mmc
UMONS
Flickr
initial
NLE
UMONS
17
Results: Best Team Runs (F1 @)
0.3
0.4
0.5
0.6
0.7
0.8
@5 @10 @20 @30 @40 @50
F1@X
Flickr Initial
CFM_run5_text_cred.txt
LAPI_HC_PSRF_Run5.txt
run3VisualTextual_MultiBrasil.txt
NLE_run3_CMRF_MMR.txt
tudmmc_run4_tudmmc_intent.txt
UMONS_run5_visual_user_G.txt
18
Results: Best Team Runs (Cluster Recall @)
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
@5 @10 @20 @30 @40 @50
CR@X
Flickr Initial
CFM_run5_text_cred.txt
LAPI_HC_PSRF_Run5.txt
run3VisualTextual_MultiBrasil.txt
NLE_run3_CMRF_MMR.txt
tudmmc_run4_tudmmc_intent.txt
UMONS_run5_visual_user_G.txt
Results: Visual Results – Flickr Initial Results
Truck Camper
19
Results: Visual Results – Flickr Initial Results
Truck Camper CR@20=0.35, P@20=0.3, F1@20=0.32
19
Results: Visual Results #2 – Best run (F1@20)
20
Truck Camper
Results: Visual Results #2 – Best run (F1@20)
20
Truck Camper CR@20=0.68, P@20=0.8, F1@20=0.74
Results: Visual Results #3 – Lowest run
21
Truck Camper
Results: Visual Results #3 – Lowest run
21
Truck Camper CR@20=0.5, P@20=0.5, F1@20=0.5
22
Brief Discussion
Methods:
 this year mainly classification/clustering (& fusion), re-ranking,
relevance feedback, & neural-network based;
 best run F1@20: improving relevancy (text) + neural network-based
clustering; use of visual-text information (team NLE).
Dataset:
 getting very complex (read diverse);
 still low resources for Creative Commons on Flickr;
 descriptors were very well received (employed by all of the
participants as provided).
23
Acknowledgements
Task auxiliaries:
Bogdan Boteanu, UPB, Romania & Mihai Lupu, Vienna University of
Technology, Austria
Task supporters:
Alberto Ueda, Bruno Laporais, Felipe Moraes, Lucas Chaves, Jordan
Silva, Marlon Dias, Rafael Glater
Catalin Mitrea, Mihai Dogariu, Liviu Stefan, Gabriel Petrescu, Alexandru
Toma, Alina Banica, Andreea Roxana, Mihaela Radu, Bogdan Guliman,
Sebastian Moraru
24
Questions & Answers
Thank you!

More Related Content

Similar to MediaEval 2017 Retrieving Diverse Social Images Task (Overview)

Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Managementtechkrish
 
Building and Measuring Privacy-Preserving Mobility Analytics
Building and Measuring Privacy-Preserving Mobility AnalyticsBuilding and Measuring Privacy-Preserving Mobility Analytics
Building and Measuring Privacy-Preserving Mobility AnalyticsEmiliano De Cristofaro
 
Cikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueCikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueXavier Amatriain
 
IRJET- Foster Hashtag from Image and Text
IRJET-  	  Foster Hashtag from Image and TextIRJET-  	  Foster Hashtag from Image and Text
IRJET- Foster Hashtag from Image and TextIRJET Journal
 
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...Anirudh Prabhu
 
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...Lippo Group Digital
 
Intelligent Career Guidance System.pptx
Intelligent Career Guidance System.pptxIntelligent Career Guidance System.pptx
Intelligent Career Guidance System.pptxAnonymous366406
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
 
Bayesian Network 을 활용한 예측 분석
Bayesian Network 을 활용한 예측 분석Bayesian Network 을 활용한 예측 분석
Bayesian Network 을 활용한 예측 분석datasciencekorea
 
Deep Recommender Systems - PAPIs.io LATAM 2018
Deep Recommender Systems - PAPIs.io LATAM 2018Deep Recommender Systems - PAPIs.io LATAM 2018
Deep Recommender Systems - PAPIs.io LATAM 2018Gabriel Moreira
 
Sample CS Senior Capstone Projects
Sample CS Senior Capstone ProjectsSample CS Senior Capstone Projects
Sample CS Senior Capstone ProjectsFred Annexstein
 
Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Gabriel Moreira
 
Seven Degrees Presentation for 2015 ICEAA
Seven Degrees Presentation for 2015 ICEAASeven Degrees Presentation for 2015 ICEAA
Seven Degrees Presentation for 2015 ICEAAJames Lawlor
 
The state of the art in integrating machine learning into visual analytics
The state of the art in integrating machine learning into visual analyticsThe state of the art in integrating machine learning into visual analytics
The state of the art in integrating machine learning into visual analyticsCagatay Turkay
 
IRJET- A Survey on Image Retrieval using Machine Learning
IRJET- A Survey on Image Retrieval using Machine LearningIRJET- A Survey on Image Retrieval using Machine Learning
IRJET- A Survey on Image Retrieval using Machine LearningIRJET Journal
 
Combining a co-occurrence-based and a semantic measure for entity linking
Combining a co-occurrence-based and a semantic measure for entity linkingCombining a co-occurrence-based and a semantic measure for entity linking
Combining a co-occurrence-based and a semantic measure for entity linkingBesnik Fetahu
 

Similar to MediaEval 2017 Retrieving Diverse Social Images Task (Overview) (20)

Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Management
 
Building and Measuring Privacy-Preserving Mobility Analytics
Building and Measuring Privacy-Preserving Mobility AnalyticsBuilding and Measuring Privacy-Preserving Mobility Analytics
Building and Measuring Privacy-Preserving Mobility Analytics
 
Poster (1)
Poster (1)Poster (1)
Poster (1)
 
Cikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business ValueCikm 2013 - Beyond Data From User Information to Business Value
Cikm 2013 - Beyond Data From User Information to Business Value
 
IRJET- Foster Hashtag from Image and Text
IRJET-  	  Foster Hashtag from Image and TextIRJET-  	  Foster Hashtag from Image and Text
IRJET- Foster Hashtag from Image and Text
 
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...
A Rules-Based Service for Suggesting Visualizations to Analyze Earth Science ...
 
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...
Profiler for Smartphone Users Interests Using Modified Hierarchical Agglomera...
 
Intelligent Career Guidance System.pptx
Intelligent Career Guidance System.pptxIntelligent Career Guidance System.pptx
Intelligent Career Guidance System.pptx
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data Science
 
Data Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data ScienceData Science as a Service: Intersection of Cloud Computing and Data Science
Data Science as a Service: Intersection of Cloud Computing and Data Science
 
Bayesian Network 을 활용한 예측 분석
Bayesian Network 을 활용한 예측 분석Bayesian Network 을 활용한 예측 분석
Bayesian Network 을 활용한 예측 분석
 
Paper 153
Paper 153Paper 153
Paper 153
 
MultiModal Retrieval Image
MultiModal Retrieval ImageMultiModal Retrieval Image
MultiModal Retrieval Image
 
Deep Recommender Systems - PAPIs.io LATAM 2018
Deep Recommender Systems - PAPIs.io LATAM 2018Deep Recommender Systems - PAPIs.io LATAM 2018
Deep Recommender Systems - PAPIs.io LATAM 2018
 
Sample CS Senior Capstone Projects
Sample CS Senior Capstone ProjectsSample CS Senior Capstone Projects
Sample CS Senior Capstone Projects
 
Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação Sistemas de Recomendação sem Enrolação
Sistemas de Recomendação sem Enrolação
 
Seven Degrees Presentation for 2015 ICEAA
Seven Degrees Presentation for 2015 ICEAASeven Degrees Presentation for 2015 ICEAA
Seven Degrees Presentation for 2015 ICEAA
 
The state of the art in integrating machine learning into visual analytics
The state of the art in integrating machine learning into visual analyticsThe state of the art in integrating machine learning into visual analytics
The state of the art in integrating machine learning into visual analytics
 
IRJET- A Survey on Image Retrieval using Machine Learning
IRJET- A Survey on Image Retrieval using Machine LearningIRJET- A Survey on Image Retrieval using Machine Learning
IRJET- A Survey on Image Retrieval using Machine Learning
 
Combining a co-occurrence-based and a semantic measure for entity linking
Combining a co-occurrence-based and a semantic measure for entity linkingCombining a co-occurrence-based and a semantic measure for entity linking
Combining a co-occurrence-based and a semantic measure for entity linking
 

More from multimediaeval

Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...
Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...
Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...multimediaeval
 
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...multimediaeval
 
Sports Video Classification: Classification of Strokes in Table Tennis for Me...
Sports Video Classification: Classification of Strokes in Table Tennis for Me...Sports Video Classification: Classification of Strokes in Table Tennis for Me...
Sports Video Classification: Classification of Strokes in Table Tennis for Me...multimediaeval
 
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...multimediaeval
 
Essex-NLIP at MediaEval Predicting Media Memorability 2020 Task
Essex-NLIP at MediaEval Predicting Media Memorability 2020 TaskEssex-NLIP at MediaEval Predicting Media Memorability 2020 Task
Essex-NLIP at MediaEval Predicting Media Memorability 2020 Taskmultimediaeval
 
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...multimediaeval
 
Fooling an Automatic Image Quality Estimator
Fooling an Automatic Image Quality EstimatorFooling an Automatic Image Quality Estimator
Fooling an Automatic Image Quality Estimatormultimediaeval
 
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...multimediaeval
 
Pixel Privacy: Quality Camouflage for Social Images
Pixel Privacy: Quality Camouflage for Social ImagesPixel Privacy: Quality Camouflage for Social Images
Pixel Privacy: Quality Camouflage for Social Imagesmultimediaeval
 
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matching
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-MatchingHCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matching
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matchingmultimediaeval
 
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...multimediaeval
 
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...multimediaeval
 
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...multimediaeval
 
Deep Conditional Adversarial learning for polyp Segmentation
Deep Conditional Adversarial learning for polyp SegmentationDeep Conditional Adversarial learning for polyp Segmentation
Deep Conditional Adversarial learning for polyp Segmentationmultimediaeval
 
A Temporal-Spatial Attention Model for Medical Image Detection
A Temporal-Spatial Attention Model for Medical Image DetectionA Temporal-Spatial Attention Model for Medical Image Detection
A Temporal-Spatial Attention Model for Medical Image Detectionmultimediaeval
 
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...multimediaeval
 
Fine-tuning for Polyp Segmentation with Attention
Fine-tuning for Polyp Segmentation with AttentionFine-tuning for Polyp Segmentation with Attention
Fine-tuning for Polyp Segmentation with Attentionmultimediaeval
 
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...multimediaeval
 
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...multimediaeval
 
Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...
 Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ... Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...
Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...multimediaeval
 

More from multimediaeval (20)

Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...
Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...
Classification of Strokes in Table Tennis with a Three Stream Spatio-Temporal...
 
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...
HCMUS at MediaEval 2020: Ensembles of Temporal Deep Neural Networks for Table...
 
Sports Video Classification: Classification of Strokes in Table Tennis for Me...
Sports Video Classification: Classification of Strokes in Table Tennis for Me...Sports Video Classification: Classification of Strokes in Table Tennis for Me...
Sports Video Classification: Classification of Strokes in Table Tennis for Me...
 
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...
Predicting Media Memorability from a Multimodal Late Fusion of Self-Attention...
 
Essex-NLIP at MediaEval Predicting Media Memorability 2020 Task
Essex-NLIP at MediaEval Predicting Media Memorability 2020 TaskEssex-NLIP at MediaEval Predicting Media Memorability 2020 Task
Essex-NLIP at MediaEval Predicting Media Memorability 2020 Task
 
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...
Overview of MediaEval 2020 Predicting Media Memorability task: What Makes a V...
 
Fooling an Automatic Image Quality Estimator
Fooling an Automatic Image Quality EstimatorFooling an Automatic Image Quality Estimator
Fooling an Automatic Image Quality Estimator
 
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable C...
 
Pixel Privacy: Quality Camouflage for Social Images
Pixel Privacy: Quality Camouflage for Social ImagesPixel Privacy: Quality Camouflage for Social Images
Pixel Privacy: Quality Camouflage for Social Images
 
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matching
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-MatchingHCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matching
HCMUS at MediaEval 2020:Image-Text Fusion for Automatic News-Images Re-Matching
 
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...
Efficient Supervision Net: Polyp Segmentation using EfficientNet and Attentio...
 
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...
HCMUS at Medico Automatic Polyp Segmentation Task 2020: PraNet and ResUnet++ ...
 
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...
Depth-wise Separable Atrous Convolution for Polyps Segmentation in Gastro-Int...
 
Deep Conditional Adversarial learning for polyp Segmentation
Deep Conditional Adversarial learning for polyp SegmentationDeep Conditional Adversarial learning for polyp Segmentation
Deep Conditional Adversarial learning for polyp Segmentation
 
A Temporal-Spatial Attention Model for Medical Image Detection
A Temporal-Spatial Attention Model for Medical Image DetectionA Temporal-Spatial Attention Model for Medical Image Detection
A Temporal-Spatial Attention Model for Medical Image Detection
 
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...
HCMUS-Juniors 2020 at Medico Task in MediaEval 2020: Refined Deep Neural Netw...
 
Fine-tuning for Polyp Segmentation with Attention
Fine-tuning for Polyp Segmentation with AttentionFine-tuning for Polyp Segmentation with Attention
Fine-tuning for Polyp Segmentation with Attention
 
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...
Bigger Networks are not Always Better: Deep Convolutional Neural Networks for...
 
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...
Insights for wellbeing: Predicting Personal Air Quality Index using Regressio...
 
Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...
 Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ... Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...
Use Visual Features From Surrounding Scenes to Improve Personal Air Quality ...
 

Recently uploaded

Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsSérgio Sacani
 
Understanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfUnderstanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfHabibouKarbo
 
HEMATOPOIESIS - formation of blood cells
HEMATOPOIESIS - formation of blood cellsHEMATOPOIESIS - formation of blood cells
HEMATOPOIESIS - formation of blood cellsSachinSuresh44
 
3.-Acknowledgment-Dedication-Abstract.docx
3.-Acknowledgment-Dedication-Abstract.docx3.-Acknowledgment-Dedication-Abstract.docx
3.-Acknowledgment-Dedication-Abstract.docxUlahVanessaBasa
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxzeus70441
 
Food_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyFood_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyHemantThakare8
 
complex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfcomplex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfSubhamKumar3239
 
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxEnvironmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxpriyankatabhane
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...Chayanika Das
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsDanielBaumann11
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPRPirithiRaju
 
Oxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxOxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxfarhanvvdk
 
AICTE activity on Water Conservation spreading awareness
AICTE activity on Water Conservation spreading awarenessAICTE activity on Water Conservation spreading awareness
AICTE activity on Water Conservation spreading awareness1hk20is002
 
bonjourmadame.tumblr.com bhaskar's girls
bonjourmadame.tumblr.com bhaskar's girlsbonjourmadame.tumblr.com bhaskar's girls
bonjourmadame.tumblr.com bhaskar's girlshansessene
 
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfKDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfGABYFIORELAMALPARTID1
 
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyLAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyChayanika Das
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learningvschiavoni
 

Recently uploaded (20)

Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive stars
 
Understanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdfUnderstanding Nutrition, 16th Edition pdf
Understanding Nutrition, 16th Edition pdf
 
HEMATOPOIESIS - formation of blood cells
HEMATOPOIESIS - formation of blood cellsHEMATOPOIESIS - formation of blood cells
HEMATOPOIESIS - formation of blood cells
 
3.-Acknowledgment-Dedication-Abstract.docx
3.-Acknowledgment-Dedication-Abstract.docx3.-Acknowledgment-Dedication-Abstract.docx
3.-Acknowledgment-Dedication-Abstract.docx
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptx
 
PLASMODIUM. PPTX
PLASMODIUM. PPTXPLASMODIUM. PPTX
PLASMODIUM. PPTX
 
Food_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiologyFood_safety_Management_pptx.pptx in microbiology
Food_safety_Management_pptx.pptx in microbiology
 
complex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdfcomplex analysis best book for solving questions.pdf
complex analysis best book for solving questions.pdf
 
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptxEnvironmental Acoustics- Speech interference level, acoustics calibrator.pptx
Environmental Acoustics- Speech interference level, acoustics calibrator.pptx
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
 
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological CorrelationsTimeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
Timeless Cosmology: Towards a Geometric Origin of Cosmological Correlations
 
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
6.1 Pests of Groundnut_Binomics_Identification_Dr.UPR
 
Oxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxOxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptx
 
AICTE activity on Water Conservation spreading awareness
AICTE activity on Water Conservation spreading awarenessAICTE activity on Water Conservation spreading awareness
AICTE activity on Water Conservation spreading awareness
 
Introduction Classification Of Alkaloids
Introduction Classification Of AlkaloidsIntroduction Classification Of Alkaloids
Introduction Classification Of Alkaloids
 
bonjourmadame.tumblr.com bhaskar's girls
bonjourmadame.tumblr.com bhaskar's girlsbonjourmadame.tumblr.com bhaskar's girls
bonjourmadame.tumblr.com bhaskar's girls
 
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdfKDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
KDIGO-2023-CKD-Guideline-Public-Review-Draft_5-July-2023.pdf
 
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyLAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
 
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep LearningCombining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
Combining Asynchronous Task Parallelism and Intel SGX for Secure Deep Learning
 

MediaEval 2017 Retrieving Diverse Social Images Task (Overview)

  • 1. Retrieving Diverse Social Images Task - task overview - 2017 University Politehnica of Bucharest Maia Zaharieva (TUW, Austria) Bogdan Ionescu (UPB, Romania) Alexandru Lucian Gînscǎ (CEA LIST, France) Rodrygo L.T. Santos (UFMG, Brazil) Henning Müller (HES-SO in Sierre, Switzerland) Bogdan Boteanu (UPB, Romania) September 13-15, Dublin, Irelandce Universidade Federal de Minas Gerais, Brazil
  • 2.  The Retrieving Diverse Social Images Task  Dataset and Evaluation  Participants  Results  Discussion and Perspectives 2 Outline
  • 3. 3 Diversity Task: Objective & Motivation Objective: image search result diversification in the context of social photo retrieval. Why diversifying search results? - to respond to the needs of different users; - as a method of tackling queries with unclear information needs; - to widen the pool of possible results (increase performance); - to reduce the number/redundancy of the returned items; …
  • 4. 3 Diversity Task: Objective & Motivation #2
  • 5. 4 Diversity Task: Objective & Motivation #2
  • 6. 5 Diversity Task: Objective & Motivation #3
  • 7. 7 Diversity Task: Definition For each query, participants receive a ranked list of photos retrieved from Flickr using its default “relevance” algorithm. Query = general-purpose, multi-topic term e.g.: autumn colors, bee on a flower, home office, snow in the city, holding hands, ... Goal of the task: refine the results by providing a ranked list of up to 50 photos (summary) that are considered to be both relevant and diverse representations of the query. relevant: a common photo representation of the query topics (all at once); bad quality photos (e.g., severely blurred, out of focus) are not considered relevant in this scenario diverse: depicting different visual characteristics of the query topics and subtopics with a certain degree of complementarity, i.e., most of the perceived visual information is different from one photo to another.
  • 8. 8 Dataset: General Information & Resources Provided information:  query text formulation;  ranked list of Creative Commons photos from Flickr* (up to 300 photos per query);  metadata from Flickr (e.g., tags, description, views, comments, date-time photo was taken, username, userid, etc);  visual, text & user annotation credibility descriptors;  semantic vectors for general English terms computed on top of the English Wikipedia(wikiset);  relevance and diversity ground truth. Photos: Development: 110 queries 32,340 photos Test: 84 queries 24,986 photos
  • 9. 9 Dataset: Provided Descriptors General purpose visual descriptors:  e.g., Auto Color Correlogram, Color and Edge Directivity Descriptor, Pyramid of Histograms of Orientation Gradients, etc; Convolutional Neural Network based descriptors:  Caffe framework based; General purpose text descriptors:  e.g., term frequency information, document frequency information and their ratio, i.e., TF-IDF; User annotation credibility descriptors (give an automatic estimation of the quality of users' tag-image content relationships):  e.g., measure of user image relevance, total number of images a user shared, the percentage of images with faces.
  • 10. 10 Dataset: Basic Statistics devset (design the methods) testset (final benchmarking) #queries 110 84 #images 32,340 24,986 #img. per query (min-average-max ) 141 - 295 - 300 299 - 300 - 300 % relevant img. 53 57.4 avg. #clusters per query 17 14 avg. #img. per cluster 9 14
  • 11. 11 Dataset: Ground Truth - annotations Relevance and diversity annotations were carried out by expert annotators:  devset: relevance: 8 annotators + 1 master (3 annotations/query) diversity: 1 annotation/query  testset: relevance: 8 annotators + 1 master (3 annotations/query) diversity: 12 annotators (3 annotations/query)  Lenient majority voting for relevance
  • 12. 12 Evaluation: Run Specification Participants are required to submit up to 5 runs:  required runs: run 1: automated using visual information only; run 2: automated using textual information only; run 3: automated using textual-visual fused without other resources than provided by the organizers;  general runs: run 4: everything allowed, e.g. human-based or hybrid human- machine approaches, including using data from external sources, (e.g., Internet) or pre-trained models obtained from external datasets related to this task; run 5: everything allowed.
  • 13. 13 Evaluation: Official Metrics  Cluster Recall* @ X = Nc/N (CR@X) where X is the cutoff point, N is the total number of clusters for the current query (from ground truth, N<=25) and Nc is the number of different clusters represented in the X ranked images; *cluster recall is computed only for the relevant images.  Precision @ X = R/X (P@X) where R is the number of relevant images;  F1-measure @ X = harmonic mean of CR and P (F1@X) Metrics are reported for different values of X (5, 10, 20, 30, 40 & 50) on per topic as well as overall (average). official ranking F1@20
  • 14. 14 Participants: Basic Statistics  Survey: - 22 respondents were interested in the task;  Registration: - 14 teams registered (1 team is organizer related);  Run submission: - 6 teams finished the task, including 1 organizer related team; - 29 runs were submitted;  Workshop participation: - 5 teams are represented at the workshop.
  • 15. 15 Participants: Submitted Runs (29) *organizer related team Team Country Required Runs General Runs Results (best) 1 (visual) 2 (text) 3 (vis-text) 4 5 P@20 CR@20 F1@20 NLE France ✓ ✓ ✓ visual-text visual-text 0.793 0.679 0.705 MultiBrazil Brazil ✓ ✓ ✓ visual-text-cred. visual-text-cred. 0.7208 0.6524 0.6634 UMONS Belgium ✓ ✓ ✓ visual-text-cred. visual-cred. 0.8071 0.5856 0.6554 CFM China ✓ ✓ ✓ text-cred. text-cred. 0.6881 0.6671 0.6533 tud-mmc Netherlands ✓ ✓ ✓ text-intent ✗ 0.7262 0.6142 0.6462 Flickr 0.6595 0.5831 0.5922 LAPI* Romania ✓ ✓ ✓ visual cred. 0.633 0.6045 0.5777
  • 16. 16 Results: P vs. CR @20 (all runs - testset) 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.5 0.55 0.6 0.65 0.7 P@20 CR@20 Flickr Initial CFM LAPI MultiBrazil NLE tud-mmc UMONS Flickr initial NLE UMONS
  • 17. 17 Results: Best Team Runs (F1 @) 0.3 0.4 0.5 0.6 0.7 0.8 @5 @10 @20 @30 @40 @50 F1@X Flickr Initial CFM_run5_text_cred.txt LAPI_HC_PSRF_Run5.txt run3VisualTextual_MultiBrasil.txt NLE_run3_CMRF_MMR.txt tudmmc_run4_tudmmc_intent.txt UMONS_run5_visual_user_G.txt
  • 18. 18 Results: Best Team Runs (Cluster Recall @) 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 @5 @10 @20 @30 @40 @50 CR@X Flickr Initial CFM_run5_text_cred.txt LAPI_HC_PSRF_Run5.txt run3VisualTextual_MultiBrasil.txt NLE_run3_CMRF_MMR.txt tudmmc_run4_tudmmc_intent.txt UMONS_run5_visual_user_G.txt
  • 19. Results: Visual Results – Flickr Initial Results Truck Camper 19
  • 20. Results: Visual Results – Flickr Initial Results Truck Camper CR@20=0.35, P@20=0.3, F1@20=0.32 19
  • 21. Results: Visual Results #2 – Best run (F1@20) 20 Truck Camper
  • 22. Results: Visual Results #2 – Best run (F1@20) 20 Truck Camper CR@20=0.68, P@20=0.8, F1@20=0.74
  • 23. Results: Visual Results #3 – Lowest run 21 Truck Camper
  • 24. Results: Visual Results #3 – Lowest run 21 Truck Camper CR@20=0.5, P@20=0.5, F1@20=0.5
  • 25. 22 Brief Discussion Methods:  this year mainly classification/clustering (& fusion), re-ranking, relevance feedback, & neural-network based;  best run F1@20: improving relevancy (text) + neural network-based clustering; use of visual-text information (team NLE). Dataset:  getting very complex (read diverse);  still low resources for Creative Commons on Flickr;  descriptors were very well received (employed by all of the participants as provided).
  • 26. 23 Acknowledgements Task auxiliaries: Bogdan Boteanu, UPB, Romania & Mihai Lupu, Vienna University of Technology, Austria Task supporters: Alberto Ueda, Bruno Laporais, Felipe Moraes, Lucas Chaves, Jordan Silva, Marlon Dias, Rafael Glater Catalin Mitrea, Mihai Dogariu, Liviu Stefan, Gabriel Petrescu, Alexandru Toma, Alina Banica, Andreea Roxana, Mihaela Radu, Bogdan Guliman, Sebastian Moraru