SlideShare a Scribd company logo
1 of 25
Graph-based multimodal clustering for social event
detection in large collections of images
Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas,
Yiannis Kompatsiaris
Information Technologies Institute (ITI)
Centre for Research & Technologies Hellas (CERTH)
MMM 2014
Overview
• The problem of social event detection
• Existing approaches
• Proposed approach
• Evaluation
• Summary & future work

MMM 2014

#2

Georgios Petkos et al.
the problem

MMM 2014

Georgios Petkos et al.
Social events?
Attended by people and represented by multimedia content shared online
news
demonstration /
riot / speech

personal
wedding /
birthday / drinks

entertainment
concert / play /
sports

MMM 2014

#4

Georgios Petkos et al.
Pope Benedict
2007: iPhone release
2008: Android release
2010: iPad release

Pope Francis

http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/

MMM 2014

#5

Georgios Petkos et al.
Social event detection
Social event detection involves the automatic
organization of a multimedia collection C into groups
of items, each (group) of which corresponds to a
distinct event.
Can be treated as a multimodal clustering problem
COLLECTION

EVENT SET
E1
EVENT DETECTION

E2

EN
MMM 2014

#6

Georgios Petkos et al.
existing approaches

MMM 2014

Georgios Petkos et al.
Supervised event detection
• Rationale: use a large number of “known” event assignments
to “learn” how to identify “same event” / “same cluster”
relationships
Two variants:
• Item-to-item: learn whether two items belong to the same
event cluster or not.
– Model Input: the set of per modality distances between two images.

• Item-to-cluster: learn whether a new item belongs to a given
event cluster or not.
– Model input: the set of per modality distances between an image and
a prototype representation of the event.
MMM 2014

#8

Georgios Petkos et al.
Utilizing the “same event” model for clustering
• Item-to-item:
– (Incremental). For each incoming image, average all item-to-item SE
scores for all items in each cluster. Assign to best-matching cluster if
average above threshold or create new cluster (Becker et. al.).
– (Batch). Compute all item-item SE scores between each image and all
other images and form an indicator vector. Cluster indicator vectors
(Petkos et. al.).

• Item-to-cluster:
– (Incremental). For each cluster maintain a multimodal representation.
Compute SE score between each incoming item and the existing
prototype event representations. Assign to best-matching cluster if
above threshold or create new cluster (Becker et. al). Alternatively use
a second model for deciding if a new cluster should be added or not
(Reuter et. al.).
MMM 2014

#9

Georgios Petkos et al.
proposed approach

MMM 2014

Georgios Petkos et al.
Overview of proposed approach

• Item-to-item SE model utilized.
• Candidate neighbours selection step (first appears in (Reuter et. al)) using a set of per
modality indexes.
• Graph representation.
• Community detection on graph. Two variants of the algorithm:
• Batch: SCAN
• Incremental: QCA
MMM 2014

#11

Georgios Petkos et al.
Proposed approach: advantages

• Item-to-cluster methods may suffer from incorrect prototype representations (due to
averaging).
• Candidate neighbours selection step makes the application of the method much more
scalable.
• Graph representation: in order to introduce a scalable item-to-item approach without
averaging.
MMM 2014

#12

Georgios Petkos et al.
evaluation

MMM 2014

Georgios Petkos et al.
Evaluation setup
• Used the dataset of the 2012 SED task of MediaEval
• Ground truth: 7,779 photos clustered around 149
events (18 technical, 79 soccer, 52 Indignados)
• Assess the following aspects:
– accuracy of same-event classification
– compare clustering quality between item-to-cluster and
the two versions of item-to-item (batch & incremental)
– measure contributions of different features
– study generalization abilities of same event model

MMM 2014

Georgios Petkos et al.
Evaluation setup
Features:
• Uploader identity.
• Actual image content:
– GIST
– SURF, aggregated using the VLAD scheme

• Textual features: title, description and tags. Either a TF-IDF or
a BM25 weighting scheme is utilized.
• Time of media creation.
• Location, when available (geodesic distance).
Appropriate indices are utilized in order to rapidly fetch the
candidate neighbours for each modality.
MMM 2014

Georgios Petkos et al.
Evaluation: SE accuracy & clustering quality
• Same event classification accuracy 98.58% (SVM)
– 10K pos/neg training, 10K pos/neg testing (random)

• Clustering quality (NMI): 30/119 training/testing events [10 random splits]
– Incremental same or better than batch
– Item-to-item better than item-to-cluster (significant at 0.95 confidence)
BATCH

INCREMENTAL

ITEM-TO-CLUSTER

AVG

0.924

0.934

0.898

STD

0.019

0.021

0.027

• When non-event photos enter the dataset, NMI degrades quickly
NON-EVENT

BATCH

INCREMENTAL

ITEM-TO-CLUSTER

5%

0.4824

0.5164

0.3954

10%

0.3421

0.3683

0.2899

*

* In the second table, results were obtained using sed2011 for training and sed2012 for testing.
MMM 2014

Georgios Petkos et al.
Evaluation: contribution of features
• Same experiments using limited sets of features
FEATUERS

BATCH

INCREMENTAL

VISUAL

0.8020 ∓ 0.0193

0.8179 ∓ 0.0151

TEXTUAL

0.7925 ∓ 0.0255

0.7792 ∓ 0.0310

VISUAL+TIME

0.9244 ∓ 0.0195

0.9360 ∓ 0.0183

TEXTUAL+TIME

0.9016 ∓ 0.0173

0.9049 ∓ 0.0209

• Repeating the same experiments without the use of
blocking led to significantly worse results
– e.g. 0.030 for visual, 0.7148 for textual

• Time is an extremely important feature
MMM 2014

Georgios Petkos et al.
Evaluation: generalizing same event model
• Train using one event type > test on a different one
• In most cases negative impact
• In few cases, performance is very high!
BATCH
soccer

technical

Indignados

soccer

-

0.8658

0.8494

technical

0.7967

-

0.8977

Indignados

0.9645

0.8456

-

INCREMENTAL
soccer

Indignados

soccer

-

0.8892

0.8667

technical

0.7661

-

0.7735

Indignados
MMM 2014

technical

0.9845

0.8482

Georgios Petkos et al.
summary & future work

MMM 2014

Georgios Petkos et al.
Summary
• Scalable item-to-item multimodal clustering approach for SED
• Key characteristics:
– Item-to-item “same event” model
– Candidate neighbor selection
– Organization of “same event” relationships to a graph
– Efficient graph clustering algorithms: SCAN (batch) / QCA
(incremental)
• In general though, item-to-item approaches are less scalable
than item-to-cluster approaches
MMM 2014

#20

Georgios Petkos et al.
Future work
• Extend method so that non-event images are properly
handled
• Multiple sources of multimedia
• The MediaEval datasets are somewhat limited. Investigate
the effect of crawling / image collection to the quality of
results

MMM 2014

#21

Georgios Petkos et al.
thank you!
questions?
Acknowledgements

MMM 2014

Georgios Petkos et al.
online clustering of same-event graph
QCA maintains community structure incrementally following
graph change operations: node & edge addition (removal
operations not applicable in same event graph): based on the
concept of community attraction forces
Cz
new edge

new node

force from Cu to Cz

A

D

X

force from Cz to Cu

C

Cw

B

Cu

• Depending on a test (computed based on local
graph structure), community structure could
remain the same, X assigned to Cu or A to Cz.
• If A is assigned to Cu, all its neighbours will be
checked for potential reassignment.

MMM 2014

#23

Georgios Petkos et al.
graph clustering :: SCAN

hub
(μ,ε)- core

structural similarity

outlier

• resilient to spurious links (e.g. visual links that connect
unrelated images)
• very fast (scales linearly to the number of edges)
• leaves less-/ and over-connected items out of the clustering
MMM 2014

#24

Georgios Petkos et al.
References
•
•
•

•

•

Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media
streams. In Proceedings of the 2nd ACM International Conference on Multimedia
Retrieval (p. 22). ACM.
Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection
using multimodal clustering and integrating supervisory signals. In Proceedings of
the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM.
Becker, H., Naaman, M. & Gravano, L.. Learning similarity metrics for event
identification in social media. In Proceedings of the third ACM International
Conference on Web search and Data Mining, WSDM ’10, pages 291–300, New
York.
Nguyen, N., Dinh, T., Xuan, Y., & Thai, M.. Adaptive algorithms for detecting
community structure in dynamic social networks. In INFOCOM 2011. 30th IEEE
International Conference on Computer Communications, Joint Conference of the
IEEE Computer and Communications Societies, 10-15 April 2011, Shanghai, China,
pages 2282–2290. IEEE, 2011.
Xu, X., Yuruk, N., Feng, Z. & Schweiger, T.. SCAN: a structural clustering algorithm
for networks. In Proceedings of the 13th ACM SIGKDD, KDD ’07, pages 824–833,
NY, USA, 2007. ACM

MMM 2014

#25

Georgios Petkos et al.

More Related Content

Similar to Graph-based multimodal clustering for social event detection in large collections of images

Event Mining in Social Multimedia
Event Mining in Social MultimediaEvent Mining in Social Multimedia
Event Mining in Social MultimediaSymeon Papadopoulos
 
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...multimediaeval
 
Feature based similarity search in 3 d object databases
Feature based similarity search in 3 d object databasesFeature based similarity search in 3 d object databases
Feature based similarity search in 3 d object databasesunyil96
 
CERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskCERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskMediaEval2012
 
CERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskCERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskSymeon Papadopoulos
 
Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Managementtechkrish
 
Knowledge Graph Embeddings for Recommender Systems
Knowledge Graph Embeddings for Recommender SystemsKnowledge Graph Embeddings for Recommender Systems
Knowledge Graph Embeddings for Recommender SystemsEnrico Palumbo
 
Presentation of the InVID tools for image forensics analysis
Presentation of the InVID tools for image forensics analysisPresentation of the InVID tools for image forensics analysis
Presentation of the InVID tools for image forensics analysisInVID Project
 
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...REVEAL - Social Media Verification
 
IRJET- Comparative Analysis of Video Processing Object Detection
IRJET- Comparative Analysis of Video Processing Object DetectionIRJET- Comparative Analysis of Video Processing Object Detection
IRJET- Comparative Analysis of Video Processing Object DetectionIRJET Journal
 
Predictive analytics in mobility
Predictive analytics in mobilityPredictive analytics in mobility
Predictive analytics in mobilityEktimo
 
Media REVEALr: A social multimedia monitoring and intelligence system for Web...
Media REVEALr: A social multimedia monitoring and intelligence system for Web...Media REVEALr: A social multimedia monitoring and intelligence system for Web...
Media REVEALr: A social multimedia monitoring and intelligence system for Web...Symeon Papadopoulos
 
A Web-based Service for Image Tampering Detection
A Web-based Service for Image Tampering DetectionA Web-based Service for Image Tampering Detection
A Web-based Service for Image Tampering DetectionSymeon Papadopoulos
 
Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
Photo Clustering of Social Events by Extending PhotoTOC to a Rich ContextPhoto Clustering of Social Events by Extending PhotoTOC to a Rich Context
Photo Clustering of Social Events by Extending PhotoTOC to a Rich ContextUniversitat Politècnica de Catalunya
 
Data fusion for city live event detection
Data fusion for city live event detectionData fusion for city live event detection
Data fusion for city live event detectionAlket Cecaj
 
Object tracking presentation
Object tracking  presentationObject tracking  presentation
Object tracking presentationMrsShwetaBanait1
 

Similar to Graph-based multimodal clustering for social event detection in large collections of images (20)

Event Mining in Social Multimedia
Event Mining in Social MultimediaEvent Mining in Social Multimedia
Event Mining in Social Multimedia
 
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...
MediaEval 2015 - Retrieving Diverse Social Images at MediaEval 2015: Challeng...
 
Feature based similarity search in 3 d object databases
Feature based similarity search in 3 d object databasesFeature based similarity search in 3 d object databases
Feature based similarity search in 3 d object databases
 
CERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskCERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection Task
 
CERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection TaskCERTH @ MediaEval 2012 Social Event Detection Task
CERTH @ MediaEval 2012 Social Event Detection Task
 
Big-Data Analytics for Media Management
Big-Data Analytics for Media ManagementBig-Data Analytics for Media Management
Big-Data Analytics for Media Management
 
Knowledge Graph Embeddings for Recommender Systems
Knowledge Graph Embeddings for Recommender SystemsKnowledge Graph Embeddings for Recommender Systems
Knowledge Graph Embeddings for Recommender Systems
 
Presentation of the InVID tools for image forensics analysis
Presentation of the InVID tools for image forensics analysisPresentation of the InVID tools for image forensics analysis
Presentation of the InVID tools for image forensics analysis
 
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
 
Exploration – A Serious Game
Exploration – A Serious GameExploration – A Serious Game
Exploration – A Serious Game
 
IRJET- Comparative Analysis of Video Processing Object Detection
IRJET- Comparative Analysis of Video Processing Object DetectionIRJET- Comparative Analysis of Video Processing Object Detection
IRJET- Comparative Analysis of Video Processing Object Detection
 
Predictive analytics in mobility
Predictive analytics in mobilityPredictive analytics in mobility
Predictive analytics in mobility
 
Media REVEALr: A social multimedia monitoring and intelligence system for Web...
Media REVEALr: A social multimedia monitoring and intelligence system for Web...Media REVEALr: A social multimedia monitoring and intelligence system for Web...
Media REVEALr: A social multimedia monitoring and intelligence system for Web...
 
Brownie v1.0
Brownie v1.0Brownie v1.0
Brownie v1.0
 
A Web-based Service for Image Tampering Detection
A Web-based Service for Image Tampering DetectionA Web-based Service for Image Tampering Detection
A Web-based Service for Image Tampering Detection
 
Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
Photo Clustering of Social Events by Extending PhotoTOC to a Rich ContextPhoto Clustering of Social Events by Extending PhotoTOC to a Rich Context
Photo Clustering of Social Events by Extending PhotoTOC to a Rich Context
 
Data fusion for city live event detection
Data fusion for city live event detectionData fusion for city live event detection
Data fusion for city live event detection
 
Technology Acceptance of Virtual Reality for Travel Planning
Technology Acceptance of Virtual Reality for Travel PlanningTechnology Acceptance of Virtual Reality for Travel Planning
Technology Acceptance of Virtual Reality for Travel Planning
 
Object tracking final
Object tracking finalObject tracking final
Object tracking final
 
Object tracking presentation
Object tracking  presentationObject tracking  presentation
Object tracking presentation
 

More from Symeon Papadopoulos

DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...Symeon Papadopoulos
 
Deepfakes: An Emerging Internet Threat and their Detection
Deepfakes: An Emerging Internet Threat and their DetectionDeepfakes: An Emerging Internet Threat and their Detection
Deepfakes: An Emerging Internet Threat and their DetectionSymeon Papadopoulos
 
Knowledge-based Fusion for Image Tampering Localization
Knowledge-based Fusion for Image Tampering LocalizationKnowledge-based Fusion for Image Tampering Localization
Knowledge-based Fusion for Image Tampering LocalizationSymeon Papadopoulos
 
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...Deepfake Detection: The Importance of Training Data Preprocessing and Practic...
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...Symeon Papadopoulos
 
COVID-19 Infodemic vs Contact Tracing
COVID-19 Infodemic vs Contact TracingCOVID-19 Infodemic vs Contact Tracing
COVID-19 Infodemic vs Contact TracingSymeon Papadopoulos
 
Similarity-based retrieval of multimedia content
Similarity-based retrieval of multimedia contentSimilarity-based retrieval of multimedia content
Similarity-based retrieval of multimedia contentSymeon Papadopoulos
 
Twitter-based Sensing of City-level Air Quality
Twitter-based Sensing of City-level Air QualityTwitter-based Sensing of City-level Air Quality
Twitter-based Sensing of City-level Air QualitySymeon Papadopoulos
 
Aggregating and Analyzing the Context of Social Media Content
Aggregating and Analyzing the Context of Social Media ContentAggregating and Analyzing the Context of Social Media Content
Aggregating and Analyzing the Context of Social Media ContentSymeon Papadopoulos
 
Verifying Multimedia Content on the Internet
Verifying Multimedia Content on the InternetVerifying Multimedia Content on the Internet
Verifying Multimedia Content on the InternetSymeon Papadopoulos
 
Learning to detect Misleading Content on Twitter
Learning to detect Misleading Content on TwitterLearning to detect Misleading Content on Twitter
Learning to detect Misleading Content on TwitterSymeon Papadopoulos
 
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN LayersNear-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN LayersSymeon Papadopoulos
 
Verifying Multimedia Use at MediaEval 2016
Verifying Multimedia Use at MediaEval 2016Verifying Multimedia Use at MediaEval 2016
Verifying Multimedia Use at MediaEval 2016Symeon Papadopoulos
 
Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...Symeon Papadopoulos
 
In-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging PerformanceIn-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging PerformanceSymeon Papadopoulos
 
Perceived versus Actual Predictability of Personal Information in Social Netw...
Perceived versus Actual Predictability of Personal Information in Social Netw...Perceived versus Actual Predictability of Personal Information in Social Netw...
Perceived versus Actual Predictability of Personal Information in Social Netw...Symeon Papadopoulos
 
Web and Social Media Image Forensics for News Professionals
Web and Social Media Image Forensics for News ProfessionalsWeb and Social Media Image Forensics for News Professionals
Web and Social Media Image Forensics for News ProfessionalsSymeon Papadopoulos
 
Predicting News Popularity by Mining Online Discussions
Predicting News Popularity by Mining Online DiscussionsPredicting News Popularity by Mining Online Discussions
Predicting News Popularity by Mining Online DiscussionsSymeon Papadopoulos
 
Finding Diverse Social Images at MediaEval 2015
Finding Diverse Social Images at MediaEval 2015Finding Diverse Social Images at MediaEval 2015
Finding Diverse Social Images at MediaEval 2015Symeon Papadopoulos
 
CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015Symeon Papadopoulos
 

More from Symeon Papadopoulos (20)

DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...
DeepFake Detection: Challenges, Progress and Hands-on Demonstration of Techno...
 
Deepfakes: An Emerging Internet Threat and their Detection
Deepfakes: An Emerging Internet Threat and their DetectionDeepfakes: An Emerging Internet Threat and their Detection
Deepfakes: An Emerging Internet Threat and their Detection
 
Knowledge-based Fusion for Image Tampering Localization
Knowledge-based Fusion for Image Tampering LocalizationKnowledge-based Fusion for Image Tampering Localization
Knowledge-based Fusion for Image Tampering Localization
 
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...Deepfake Detection: The Importance of Training Data Preprocessing and Practic...
Deepfake Detection: The Importance of Training Data Preprocessing and Practic...
 
COVID-19 Infodemic vs Contact Tracing
COVID-19 Infodemic vs Contact TracingCOVID-19 Infodemic vs Contact Tracing
COVID-19 Infodemic vs Contact Tracing
 
Similarity-based retrieval of multimedia content
Similarity-based retrieval of multimedia contentSimilarity-based retrieval of multimedia content
Similarity-based retrieval of multimedia content
 
Twitter-based Sensing of City-level Air Quality
Twitter-based Sensing of City-level Air QualityTwitter-based Sensing of City-level Air Quality
Twitter-based Sensing of City-level Air Quality
 
Aggregating and Analyzing the Context of Social Media Content
Aggregating and Analyzing the Context of Social Media ContentAggregating and Analyzing the Context of Social Media Content
Aggregating and Analyzing the Context of Social Media Content
 
Verifying Multimedia Content on the Internet
Verifying Multimedia Content on the InternetVerifying Multimedia Content on the Internet
Verifying Multimedia Content on the Internet
 
Learning to detect Misleading Content on Twitter
Learning to detect Misleading Content on TwitterLearning to detect Misleading Content on Twitter
Learning to detect Misleading Content on Twitter
 
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN LayersNear-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
Near-Duplicate Video Retrieval by Aggregating Intermediate CNN Layers
 
Verifying Multimedia Use at MediaEval 2016
Verifying Multimedia Use at MediaEval 2016Verifying Multimedia Use at MediaEval 2016
Verifying Multimedia Use at MediaEval 2016
 
Multimedia Privacy
Multimedia PrivacyMultimedia Privacy
Multimedia Privacy
 
Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...
 
In-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging PerformanceIn-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging Performance
 
Perceived versus Actual Predictability of Personal Information in Social Netw...
Perceived versus Actual Predictability of Personal Information in Social Netw...Perceived versus Actual Predictability of Personal Information in Social Netw...
Perceived versus Actual Predictability of Personal Information in Social Netw...
 
Web and Social Media Image Forensics for News Professionals
Web and Social Media Image Forensics for News ProfessionalsWeb and Social Media Image Forensics for News Professionals
Web and Social Media Image Forensics for News Professionals
 
Predicting News Popularity by Mining Online Discussions
Predicting News Popularity by Mining Online DiscussionsPredicting News Popularity by Mining Online Discussions
Predicting News Popularity by Mining Online Discussions
 
Finding Diverse Social Images at MediaEval 2015
Finding Diverse Social Images at MediaEval 2015Finding Diverse Social Images at MediaEval 2015
Finding Diverse Social Images at MediaEval 2015
 
CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015
 

Recently uploaded

What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesSanjay Willie
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your QueriesExploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
Exploring ChatGPT Prompt Hacks To Maximally Optimise Your Queries
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 

Graph-based multimodal clustering for social event detection in large collections of images

  • 1. Graph-based multimodal clustering for social event detection in large collections of images Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas, Yiannis Kompatsiaris Information Technologies Institute (ITI) Centre for Research & Technologies Hellas (CERTH) MMM 2014
  • 2. Overview • The problem of social event detection • Existing approaches • Proposed approach • Evaluation • Summary & future work MMM 2014 #2 Georgios Petkos et al.
  • 4. Social events? Attended by people and represented by multimedia content shared online news demonstration / riot / speech personal wedding / birthday / drinks entertainment concert / play / sports MMM 2014 #4 Georgios Petkos et al.
  • 5. Pope Benedict 2007: iPhone release 2008: Android release 2010: iPad release Pope Francis http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/ MMM 2014 #5 Georgios Petkos et al.
  • 6. Social event detection Social event detection involves the automatic organization of a multimedia collection C into groups of items, each (group) of which corresponds to a distinct event. Can be treated as a multimodal clustering problem COLLECTION EVENT SET E1 EVENT DETECTION E2 EN MMM 2014 #6 Georgios Petkos et al.
  • 8. Supervised event detection • Rationale: use a large number of “known” event assignments to “learn” how to identify “same event” / “same cluster” relationships Two variants: • Item-to-item: learn whether two items belong to the same event cluster or not. – Model Input: the set of per modality distances between two images. • Item-to-cluster: learn whether a new item belongs to a given event cluster or not. – Model input: the set of per modality distances between an image and a prototype representation of the event. MMM 2014 #8 Georgios Petkos et al.
  • 9. Utilizing the “same event” model for clustering • Item-to-item: – (Incremental). For each incoming image, average all item-to-item SE scores for all items in each cluster. Assign to best-matching cluster if average above threshold or create new cluster (Becker et. al.). – (Batch). Compute all item-item SE scores between each image and all other images and form an indicator vector. Cluster indicator vectors (Petkos et. al.). • Item-to-cluster: – (Incremental). For each cluster maintain a multimodal representation. Compute SE score between each incoming item and the existing prototype event representations. Assign to best-matching cluster if above threshold or create new cluster (Becker et. al). Alternatively use a second model for deciding if a new cluster should be added or not (Reuter et. al.). MMM 2014 #9 Georgios Petkos et al.
  • 11. Overview of proposed approach • Item-to-item SE model utilized. • Candidate neighbours selection step (first appears in (Reuter et. al)) using a set of per modality indexes. • Graph representation. • Community detection on graph. Two variants of the algorithm: • Batch: SCAN • Incremental: QCA MMM 2014 #11 Georgios Petkos et al.
  • 12. Proposed approach: advantages • Item-to-cluster methods may suffer from incorrect prototype representations (due to averaging). • Candidate neighbours selection step makes the application of the method much more scalable. • Graph representation: in order to introduce a scalable item-to-item approach without averaging. MMM 2014 #12 Georgios Petkos et al.
  • 14. Evaluation setup • Used the dataset of the 2012 SED task of MediaEval • Ground truth: 7,779 photos clustered around 149 events (18 technical, 79 soccer, 52 Indignados) • Assess the following aspects: – accuracy of same-event classification – compare clustering quality between item-to-cluster and the two versions of item-to-item (batch & incremental) – measure contributions of different features – study generalization abilities of same event model MMM 2014 Georgios Petkos et al.
  • 15. Evaluation setup Features: • Uploader identity. • Actual image content: – GIST – SURF, aggregated using the VLAD scheme • Textual features: title, description and tags. Either a TF-IDF or a BM25 weighting scheme is utilized. • Time of media creation. • Location, when available (geodesic distance). Appropriate indices are utilized in order to rapidly fetch the candidate neighbours for each modality. MMM 2014 Georgios Petkos et al.
  • 16. Evaluation: SE accuracy & clustering quality • Same event classification accuracy 98.58% (SVM) – 10K pos/neg training, 10K pos/neg testing (random) • Clustering quality (NMI): 30/119 training/testing events [10 random splits] – Incremental same or better than batch – Item-to-item better than item-to-cluster (significant at 0.95 confidence) BATCH INCREMENTAL ITEM-TO-CLUSTER AVG 0.924 0.934 0.898 STD 0.019 0.021 0.027 • When non-event photos enter the dataset, NMI degrades quickly NON-EVENT BATCH INCREMENTAL ITEM-TO-CLUSTER 5% 0.4824 0.5164 0.3954 10% 0.3421 0.3683 0.2899 * * In the second table, results were obtained using sed2011 for training and sed2012 for testing. MMM 2014 Georgios Petkos et al.
  • 17. Evaluation: contribution of features • Same experiments using limited sets of features FEATUERS BATCH INCREMENTAL VISUAL 0.8020 ∓ 0.0193 0.8179 ∓ 0.0151 TEXTUAL 0.7925 ∓ 0.0255 0.7792 ∓ 0.0310 VISUAL+TIME 0.9244 ∓ 0.0195 0.9360 ∓ 0.0183 TEXTUAL+TIME 0.9016 ∓ 0.0173 0.9049 ∓ 0.0209 • Repeating the same experiments without the use of blocking led to significantly worse results – e.g. 0.030 for visual, 0.7148 for textual • Time is an extremely important feature MMM 2014 Georgios Petkos et al.
  • 18. Evaluation: generalizing same event model • Train using one event type > test on a different one • In most cases negative impact • In few cases, performance is very high! BATCH soccer technical Indignados soccer - 0.8658 0.8494 technical 0.7967 - 0.8977 Indignados 0.9645 0.8456 - INCREMENTAL soccer Indignados soccer - 0.8892 0.8667 technical 0.7661 - 0.7735 Indignados MMM 2014 technical 0.9845 0.8482 Georgios Petkos et al.
  • 19. summary & future work MMM 2014 Georgios Petkos et al.
  • 20. Summary • Scalable item-to-item multimodal clustering approach for SED • Key characteristics: – Item-to-item “same event” model – Candidate neighbor selection – Organization of “same event” relationships to a graph – Efficient graph clustering algorithms: SCAN (batch) / QCA (incremental) • In general though, item-to-item approaches are less scalable than item-to-cluster approaches MMM 2014 #20 Georgios Petkos et al.
  • 21. Future work • Extend method so that non-event images are properly handled • Multiple sources of multimedia • The MediaEval datasets are somewhat limited. Investigate the effect of crawling / image collection to the quality of results MMM 2014 #21 Georgios Petkos et al.
  • 23. online clustering of same-event graph QCA maintains community structure incrementally following graph change operations: node & edge addition (removal operations not applicable in same event graph): based on the concept of community attraction forces Cz new edge new node force from Cu to Cz A D X force from Cz to Cu C Cw B Cu • Depending on a test (computed based on local graph structure), community structure could remain the same, X assigned to Cu or A to Cz. • If A is assigned to Cu, all its neighbours will be checked for potential reassignment. MMM 2014 #23 Georgios Petkos et al.
  • 24. graph clustering :: SCAN hub (μ,ε)- core structural similarity outlier • resilient to spurious links (e.g. visual links that connect unrelated images) • very fast (scales linearly to the number of edges) • leaves less-/ and over-connected items out of the clustering MMM 2014 #24 Georgios Petkos et al.
  • 25. References • • • • • Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media streams. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 22). ACM. Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection using multimodal clustering and integrating supervisory signals. In Proceedings of the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM. Becker, H., Naaman, M. & Gravano, L.. Learning similarity metrics for event identification in social media. In Proceedings of the third ACM International Conference on Web search and Data Mining, WSDM ’10, pages 291–300, New York. Nguyen, N., Dinh, T., Xuan, Y., & Thai, M.. Adaptive algorithms for detecting community structure in dynamic social networks. In INFOCOM 2011. 30th IEEE International Conference on Computer Communications, Joint Conference of the IEEE Computer and Communications Societies, 10-15 April 2011, Shanghai, China, pages 2282–2290. IEEE, 2011. Xu, X., Yuruk, N., Feng, Z. & Schweiger, T.. SCAN: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD, KDD ’07, pages 824–833, NY, USA, 2007. ACM MMM 2014 #25 Georgios Petkos et al.