Graph-based multimodal clustering for social event
detection in large collections of images
Georgios Petkos, Symeon Papadopoulos, Emmanouil Schinas,
Yiannis Kompatsiaris
Information Technologies Institute (ITI)
Centre for Research & Technologies Hellas (CERTH)
MMM 2014

Overview
• The problem of social event detection
• Existing approaches
• Proposed approach
• Evaluation
• Summary & future work
MMM 2014
#2
Georgios Petkos et al.

the problem
MMM 2014
Georgios Petkos et al.

Social events?
Attended by people and represented by multimedia content shared online
news
demonstration /
riot / speech
personal
wedding /
birthday / drinks
entertainment
concert / play /
sports
MMM 2014
#4
Georgios Petkos et al.

Pope Benedict
2007: iPhone release
2008: Android release
2010: iPad release
Pope Francis
http://petapixel.com/2013/03/14/a-starry-sea-of-cameras-at-the-unveiling-of-pope-francis/
MMM 2014
#5
Georgios Petkos et al.

Social event detection
Social event detection involves the automatic
organization of a multimedia collection C into groups
of items, each (group) of which corresponds to a
distinct event.
Can be treated as a multimodal clustering problem
COLLECTION
EVENT SET
E1
EVENT DETECTION
E2
EN
MMM 2014
#6
Georgios Petkos et al.

existing approaches
MMM 2014
Georgios Petkos et al.

Supervised event detection
• Rationale: use a large number of “known” event assignments
to “learn” how to identify “same event” / “same cluster”
relationships
Two variants:
• Item-to-item: learn whether two items belong to the same
event cluster or not.
– Model Input: the set of per modality distances between two images.
• Item-to-cluster: learn whether a new item belongs to a given
event cluster or not.
– Model input: the set of per modality distances between an image and
a prototype representation of the event.
MMM 2014
#8
Georgios Petkos et al.

Utilizing the “same event” model for clustering
• Item-to-item:
– (Incremental). For each incoming image, average all item-to-item SE
scores for all items in each cluster. Assign to best-matching cluster if
average above threshold or create new cluster (Becker et. al.).
– (Batch). Compute all item-item SE scores between each image and all
other images and form an indicator vector. Cluster indicator vectors
(Petkos et. al.).
• Item-to-cluster:
– (Incremental). For each cluster maintain a multimodal representation.
Compute SE score between each incoming item and the existing
prototype event representations. Assign to best-matching cluster if
above threshold or create new cluster (Becker et. al). Alternatively use
a second model for deciding if a new cluster should be added or not
(Reuter et. al.).
MMM 2014
#9
Georgios Petkos et al.

proposed approach
MMM 2014
Georgios Petkos et al.

Overview of proposed approach
• Item-to-item SE model utilized.
• Candidate neighbours selection step (first appears in (Reuter et. al)) using a set of per
modality indexes.
• Graph representation.
• Community detection on graph. Two variants of the algorithm:
• Batch: SCAN
• Incremental: QCA
MMM 2014
#11
Georgios Petkos et al.

Proposed approach: advantages
• Item-to-cluster methods may suffer from incorrect prototype representations (due to
averaging).
• Candidate neighbours selection step makes the application of the method much more
scalable.
• Graph representation: in order to introduce a scalable item-to-item approach without
averaging.
MMM 2014
#12
Georgios Petkos et al.

evaluation
MMM 2014
Georgios Petkos et al.

Evaluation setup
• Used the dataset of the 2012 SED task of MediaEval
• Ground truth: 7,779 photos clustered around 149
events (18 technical, 79 soccer, 52 Indignados)
• Assess the following aspects:
– accuracy of same-event classification
– compare clustering quality between item-to-cluster and
the two versions of item-to-item (batch & incremental)
– measure contributions of different features
– study generalization abilities of same event model
MMM 2014
Georgios Petkos et al.

Evaluation setup
Features:
• Uploader identity.
• Actual image content:
– GIST
– SURF, aggregated using the VLAD scheme
• Textual features: title, description and tags. Either a TF-IDF or
a BM25 weighting scheme is utilized.
• Time of media creation.
• Location, when available (geodesic distance).
Appropriate indices are utilized in order to rapidly fetch the
candidate neighbours for each modality.
MMM 2014
Georgios Petkos et al.

Evaluation: SE accuracy & clustering quality
• Same event classification accuracy 98.58% (SVM)
– 10K pos/neg training, 10K pos/neg testing (random)
• Clustering quality (NMI): 30/119 training/testing events [10 random splits]
– Incremental same or better than batch
– Item-to-item better than item-to-cluster (significant at 0.95 confidence)
BATCH
INCREMENTAL
ITEM-TO-CLUSTER
AVG
0.924
0.934
0.898
STD
0.019
0.021
0.027
• When non-event photos enter the dataset, NMI degrades quickly
NON-EVENT
BATCH
INCREMENTAL
ITEM-TO-CLUSTER
5%
0.4824
0.5164
0.3954
10%
0.3421
0.3683
0.2899
*
* In the second table, results were obtained using sed2011 for training and sed2012 for testing.
MMM 2014
Georgios Petkos et al.

Evaluation: contribution of features
• Same experiments using limited sets of features
FEATUERS
BATCH
INCREMENTAL
VISUAL
0.8020 ∓ 0.0193
0.8179 ∓ 0.0151
TEXTUAL
0.7925 ∓ 0.0255
0.7792 ∓ 0.0310
VISUAL+TIME
0.9244 ∓ 0.0195
0.9360 ∓ 0.0183
TEXTUAL+TIME
0.9016 ∓ 0.0173
0.9049 ∓ 0.0209
• Repeating the same experiments without the use of
blocking led to significantly worse results
– e.g. 0.030 for visual, 0.7148 for textual
• Time is an extremely important feature
MMM 2014
Georgios Petkos et al.

Evaluation: generalizing same event model
• Train using one event type > test on a different one
• In most cases negative impact
• In few cases, performance is very high!
BATCH
soccer
technical
Indignados
soccer
-
0.8658
0.8494
technical
0.7967
-
0.8977
Indignados
0.9645
0.8456
-
INCREMENTAL
soccer
Indignados
soccer
-
0.8892
0.8667
technical
0.7661
-
0.7735
Indignados
MMM 2014
technical
0.9845
0.8482
Georgios Petkos et al.

summary & future work
MMM 2014
Georgios Petkos et al.

Summary
• Scalable item-to-item multimodal clustering approach for SED
• Key characteristics:
– Item-to-item “same event” model
– Candidate neighbor selection
– Organization of “same event” relationships to a graph
– Efficient graph clustering algorithms: SCAN (batch) / QCA
(incremental)
• In general though, item-to-item approaches are less scalable
than item-to-cluster approaches
MMM 2014
#20
Georgios Petkos et al.

Future work
• Extend method so that non-event images are properly
handled
• Multiple sources of multimedia
• The MediaEval datasets are somewhat limited. Investigate
the effect of crawling / image collection to the quality of
results
MMM 2014
#21
Georgios Petkos et al.

thank you!
questions?
Acknowledgements
MMM 2014
Georgios Petkos et al.

online clustering of same-event graph
QCA maintains community structure incrementally following
graph change operations: node & edge addition (removal
operations not applicable in same event graph): based on the
concept of community attraction forces
Cz
new edge
new node
force from Cu to Cz
A
D
X
force from Cz to Cu
C
Cw
B
Cu
• Depending on a test (computed based on local
graph structure), community structure could
remain the same, X assigned to Cu or A to Cz.
• If A is assigned to Cu, all its neighbours will be
checked for potential reassignment.
MMM 2014
#23
Georgios Petkos et al.

graph clustering :: SCAN
hub
(μ,ε)- core
structural similarity
outlier
• resilient to spurious links (e.g. visual links that connect
unrelated images)
• very fast (scales linearly to the number of edges)
• leaves less-/ and over-connected items out of the clustering
MMM 2014
#24
Georgios Petkos et al.

References
•
•
•
•
•
Reuter, T., & Cimiano, P. (2012, June). Event-based classification of social media
streams. In Proceedings of the 2nd ACM International Conference on Multimedia
Retrieval (p. 22). ACM.
Petkos, G., Papadopoulos, S., & Kompatsiaris, Y. (2012). Social event detection
using multimodal clustering and integrating supervisory signals. In Proceedings of
the 2nd ACM International Conference on Multimedia Retrieval (p. 23). ACM.
Becker, H., Naaman, M. & Gravano, L.. Learning similarity metrics for event
identification in social media. In Proceedings of the third ACM International
Conference on Web search and Data Mining, WSDM ’10, pages 291–300, New
York.
Nguyen, N., Dinh, T., Xuan, Y., & Thai, M.. Adaptive algorithms for detecting
community structure in dynamic social networks. In INFOCOM 2011. 30th IEEE
International Conference on Computer Communications, Joint Conference of the
IEEE Computer and Communications Societies, 10-15 April 2011, Shanghai, China,
pages 2282–2290. IEEE, 2011.
Xu, X., Yuruk, N., Feng, Z. & Schweiger, T.. SCAN: a structural clustering algorithm
for networks. In Proceedings of the 13th ACM SIGKDD, KDD ’07, pages 824–833,
NY, USA, 2007. ACM
MMM 2014
#25
Georgios Petkos et al.