SlideShare a Scribd company logo
1 of 33
Geotagging Social Media Content with a
Refined Language Modelling Approach
Georgios Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris
Centre for Research and Technology Hellas (CERTH) – Information Technologies Institute (ITI)
PAISI 2015, May 19, 2015, Ho Chi Minh City, Vietnam
Where is it?
#2
Depicted landmark
Eiffel Tower
Location
Paris, Tennessee
Keyword “Tennesee” is very important
to correctly place the photo.
Source (Wikipedia):
http://en.wikipedia.org/wiki/Eiffel_Tow
er_(Paris,_Tennessee)
The Problem
• A lot of multimedia content is associated with
geographic information
• Being able to collect and analyze large amounts of
geotagged content could be very useful for several
applications, e.g., situational awareness in incidents such
as natural disasters, verification, geographic trends, etc.
• Yet, only a very small percentage of Web media content
carries explicit information (i.e. GPS coordinates), for
instance ~1% of tweets are geotagged
• To this end, methods that can infer the geographic
location of Web multimedia content are of interest.
#3
A Refined Language Model for Geotagging
• Extend and improve the widely used Language
Model for the problem of location estimation from
text metadata
• The proposed improvements include:
– Feature selection based on a cross-validation approach
– Feature weighting based on spatial entropy
– Multiple resolution grids
• Extensive evaluation on a public benchmark shows
highly competitive performance and reveals new
insights and challenges
#4
Related Work: Gazetteer-based
Methods that use large dictionaries and Volunteered
Geographic Information (VGI), e.g., Geonames, Yahoo!
GeoPlanet, OpenStreetMap, etc.
• Semantics-based IR approach for integrating
gazetteers and VGI (Kessler et al., 2009)
• Similarity matching mediating multiple gazetteers in
a meta-gazetteer service (Smart et al., 2010)
• Comma groups extracted with heuristic methods
from lists of toponyms (Lieberman et al., 2010)
#5
Related Work: Language Models
Language Models: large corpora of geotagged text to
create location-specific language model, i.e. what are
the most frequent keywords for a given location
• Base approach (Serdyukov et al., 2009)
• Disjoint dynamically sized cells (Hauff et al., 2012)
• User frequency instead of term frequency (O’Hare &
Murdock, 2012)
• Clustering, use of χ2 for feature selection and
similarity search (Van Laere et al., 2011)
#6
Related Work: Multimodal Approaches
Multimodal approaches do not only use text, but also
leverage the visual content and other social metadata
of geotagged multimedia.
• Combination of text metadata and visual content at
two levels of granularity, city- (100km) and
landmark-level (100m) (Crandall et al., 2009)
• Build user models leveraging user’s upload history,
SN data and hometown (Trevisiol et al., 2013)
• Hierarchical approach using both text-based and
visual similarity (Kelm et al., 2011)
#7
Related Work: MediaEval Placing Task
• Yearly benchmarking task where different approaches
compete
– Each participant can submit up to 5 runs with different
instances/configurations of their method
• Dataset for Placing Task 2014
– Flickr CC-licensed images & videos, subset of YFCC 100M
– Training: 5M, Testing: 510K (multiple subsets of increasing size
are used for reporting)
• Evaluation
– Estimated location of test image/video is compared against the
known one, and it is checked whether it belongs to a circle of
radius of 10m, 100m, 1km, 10km, 100km and 1000km
– Then, the percentage of images/videos that were correctly
placed within each radius are reported, e.g., P@1km
• Competing approaches: both gazetteer- and LM-based
#8
Overview of Approach
#9
Geographic Language Model (1/2)
• Training data: Corpus Dtr of images and videos
• Test data: Corpus Dts
• For each item (either in training or test data), we
have: user id, title, tags, description
• Title and tags of training images used for building the
model. For testing, description is used only if the
item has neither title nor tags associated with it.
• Pre-processing: punctuation and symbol removal,
lowercasing, numeric tags removed, composite
phrases (e.g. “new+york”  “new”, “york”) are split
into their components
#10
Geographic Language Model (2/2)
• Generate rectangular grid C of areas (cells) of size 0.01⁰ x
0.01⁰ (~1km x 1km near equator)
• For each cell and each tag in the training corpus,
compute the tag-cell probability:
𝑝 𝑡 𝑐 =
𝑁 𝑢
𝑁𝑡
– Nu: number of users in Dtr that used tag t inside the borders of c
– Nt: total count of users that used tag t in any cell
• For a new text T with N tags, compute the Most Likely
Cell (MLC) cj based on the following:
𝑐𝑗 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑖
𝑘=1
𝑁
𝑝(𝑡 𝑘|𝑐𝑖)
#11
Geographic Language Model: Example
#12
new: 0.15
york: 0.27
manhattan: 0.45
liberty: 0.33
…
nyc: 0.52
Feature Selection
• Retaining all possible tags of the training set might result in
noise and overfitting and makes the resulting model very
memory-demanding  select only those tags that are really
discriminative
• Use a variant of Cross-Validation on the training set:
– split the training set in p folds (=10 in our tests)
– use the p-1 folds for creating the LM and one for computing its
accuracy (P@r, where r is the radius we are interested to optimize)
– compute tag geographicity: 𝑡𝑔𝑒𝑜 𝑡 =
𝑁 𝑟
𝑁𝑡
, where Nr is the number of
correctly classified items where tag t appears and Nt is the total
number of items where tag t appears
– select only tags that exceed threshold θtgeo and that have been used
by a minimum number of users θu
#13
Feature Weighting using Spatial Entropy
• We want to penalize tags that appear in many different places and to give more
importance to tags that appear only for specific places.
• To this end, we define a measure of the stochasticity of the tag’s appearance, i.e.
its spatial entropy:
𝑒 𝑡 = −
𝑖=1
𝑀
𝑝 𝑡 𝑐𝑖 log 𝑝(𝑡|𝑐𝑖)
where M is the total number of cells.
• Once the entropy values are computed, we apply Gaussian normalization to
suppress the weight of tags that take extreme values:
𝑁 𝑒 𝑡 , 𝜇, 𝜎 =
1
𝜎 2𝜋
𝑒−(
𝑒 𝑡 −𝜇
2𝜎 )2
where N is the Gaussian function and μ, σ are the mean value and variance
of the distribution and are estimated on Dtr.
• Consequently, the MLC is computed based on:
𝑐𝑗 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑖
𝑘=1
𝑁
𝑝(𝑡 𝑘|𝑐𝑖) ∙ 𝑁 𝑒 𝑡 𝑘 , 𝜇, 𝜎
#14
Entropy Histogram & Gaussian Weighting
#15
+ --
Similarity Search
• Having assigned an item to the Most Likely Cell (MLC), we refine the
location estimate by searching for the most similar item in the cell.
• The k most similar items are retrieved (from Dtr) using Jaccard similarity
on the respective sets of tags:
𝐽 𝑥, 𝑦 =
|𝑇𝑥 ∩ 𝑇𝑦|
|𝑇𝑥 ∪ 𝑇𝑦|
• The final estimation is the centre of gravity of the k most similar images:
𝑙𝑜𝑐 𝑥 =
1
𝑘
𝑖=1
𝑘
𝐽 𝑥, 𝑦𝑖
𝛼 𝑙𝑜𝑐(𝑦𝑖)
where α determines how strongly the result is influenced by the
most similar items.
• If less than k images are retrieved, then only those are used in the above
calculation. If no similar images are retrieved, then the centre of MLC is
provided as output.
#16
Multiple Resolution Grids
• To increase the granularity of the prediction and at
the same time its reliability, we devised the following
dual grid scheme:
– we build two LMs: one of size 0.01° x 0.01° (coarse
granularity) and one of size 0.001° x 0.001° (fine
granularity)
– conduct location estimations based on both
– if the fine granularity estimations falls within the cell of
the estimation based on the coarse granularity, then we
select the fine granularity
– otherwise, we select the coarse (since we consider it by
default more reliable)
#17
Evaluation
• Benchmark dataset: MediaEval 2014
• Training set: 5M, Test set: 510K
• All experiments conducted on the full test set (510K)
• Two stages of evaluation:
– participation in contest (with a limited version of the
proposed approach)
– post-contest performance exploration
#18
Evaluation: MediaEval 2014 Contest (1/4)
• Out of the five runs, three were based on variations
of the presented approach
– run1: LM + feature weighting with spatial entropy +
similarity search + multiple resolution grid
– run4: LM only
– run5: LM + similarity search
(similarity search parameters: α=1, k=4)
• Performance was measured with P@r, where r =
10m, 100m, 1km, 10km, 100km and 1000km
#19
Evaluation: MediaEval 2014 Contest (2/4)
#20
• Proposed improvements (run1) outperform base
approach (run4) and base approach + similarity
search (run5)
• The improvement is more pronounced in the small
ranges (10m, 100m, 1km)
Evaluation: MediaEval 2014 Contest (3/4)
#21
Proposed
Proposed
Evaluation: MediaEval 2014 Contest (4/4)
#22
Number of image tags
Post-Contest Evaluation
Explore the role of different factors:
• Big training set (YFCC100M): ~48M geotagged items
• Feature Selection (FS)
• Feature Weighting with Spatial Entropy (SE)
• Multiple Resolution Grid (MG)
• Similarity Search (SS)
Two settings:
• FAIR: All users from the training set are completely
removed from the test set
• OVERFIT: Users are not removed from the test set even
when some of their media items are included in the
training set.
#23
Post-Contest Evaluation
#24
• Clear improvement with the addition of MG and SS
• The proposed improvements together with the use
of the bigger dataset make the approach perform
better than all other methods in MediaEval 2014
Geographic Error Analysis
• More data leads to
lower error across
the globe.
• Several small US
cities suffer from low
accuracy due to
having names of
large European
cities.
#25
ALL + YFCC100M
run 4
Big Data vs. Complex Algorithms
#26
Using 10x more
data for training led
to equivalent
performance with
using more
complex algorithm
(LM + extensions)
with less data!
Placeability of Media Items
• Sum of cell-tag probabilities is a good indicator of
how confident we are in the decision of the classifier.
#27
Complicated Cases
#28
Statue of Liberty
Security Incident Heatmaps
#29
earthquake
riot
Conclusion
• Key contributions
– Improved geotagging approach, extending the widely used
language model in three ways: feature selection, weighting,
multiple resolution grids
– Thorough analysis of geotagging accuracy offering new insights
and highlighting new challenges
• Future Work
– Exploit visual features to improve (currently visual-only
approaches perform very poorly)
– Integrate gazetteer and structured location data sources (e.g.
Foursquare venues, OpenStreetMap, etc.)
– Evaluate in more challenging settings and datasets (e.g. Twitter,
Instagram)
#30
References (1/2)
• C. Kessler, K. Janowicz, and M. Bishr. An Agenda for the Next Generation
Gazetteer: Geographic Information Contribution and Retrieval. In Proceedings of
the 17th ACM SIGSPATIAL International Conference on Advances in Geographic
Information Systems, pages 91100. ACM, 2009
• P. Smart, C. Jones, and F. Twaroch. Multi-source Toponym Data Integration and
Mediation for a Meta-gazetteer Service. In Proceedings of the 6th international
conference on Geographic information science. GIScience10. Springer-Verlag,
Berlin, Heidelberg, 234248, 2010
• M.D. Lieberman, H. Samet, and J. Sankaranayananan. Geotagging: using Proximity,
Sibling, and Prominence Clues to Understand Comma Groups. In Proceedings of
the 6th Workshop on Geographic Information Retrieval, 2010
• P. Serdyukov, V. Murdock, and R. Van Zwol. Placing Flickr Photos on a Map. In
SIGIR09, pages 484-491, New York, NY, USA, 2009. ACM
• C. Hauff and G. Houben. Geo-location Estimation of Flickr images: Social Web
based Enrichment. ECIR 2012, p. 85-96. Springer LNCS 7224, April 1-5 2012
• N. O'Hare, and V. Murdock. Modeling Locations with Social Media. Information
Retrieval, pp. 133, 2012
• O. Van Laere, S. Schockaert, and B. Dhoedt. Finding Locations of Flickr Resources
using Language Models and Similarity Search. ICMR 11, New York, USA, 2011. ACM
#31
References (2/2)
• D.J. Crandall, L. Backstrom, D. Huttenlocher, and J.
Kleinberg. Mapping the World's Photos. In Proceedings
of the 18th international conference on World wide web,
WWW 09, pages 761770, New York, NY, USA, 2009. ACM
• M. Trevisiol, H. Jegou, J. Delhumeau, and S. Gravier.
Retrieving Geo-Location of Videos with a Divide and
Conquer Hierarchical Multimodal Approach. ICMR13,
Dallas, United States, April 2013. ACM
• P. Kelm, S. Schmiedeke, and T. Sikora. A Hierarchical,
Multi-modal Approach for Placing Videos on the Map
using Millions of Flickr Photographs. In Proceedings of
the 2011 ACM Workshop on Social and Behavioural
Networked Media Access, SBNMA 11, pages 1520, New
York, NY, USA, 2011. ACM
#32
Thank you!
• Resources:
Slides: http://www.slideshare.net/sympapadopoulos/reveal-geotagging
Code: https://github.com/socialsensor/multimedia-geotagging
Benchmark:
http://www.multimediaeval.org/mediaeval2014/placing2014/
• Get in touch:
@sympapadopoulos / papadop@iti.gr
George Kordopatis / georgekordopatis@iti.gr
#33

More Related Content

What's hot

Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
zukun
 
Tracking and counting human in visual surveillance system
Tracking and counting human in visual surveillance systemTracking and counting human in visual surveillance system
Tracking and counting human in visual surveillance system
iaemedu
 

What's hot (18)

A scalable collaborative filtering framework based on co clustering
A scalable collaborative filtering framework based on co clusteringA scalable collaborative filtering framework based on co clustering
A scalable collaborative filtering framework based on co clustering
 
Detection focal loss 딥러닝 논문읽기 모임 발표자료
Detection focal loss 딥러닝 논문읽기 모임 발표자료Detection focal loss 딥러닝 논문읽기 모임 발표자료
Detection focal loss 딥러닝 논문읽기 모임 발표자료
 
Dear - 딥러닝 논문읽기 모임 김창연님
Dear - 딥러닝 논문읽기 모임 김창연님Dear - 딥러닝 논문읽기 모임 김창연님
Dear - 딥러닝 논문읽기 모임 김창연님
 
Attentive semantic alignment with offset aware correlation kernels
Attentive semantic alignment with offset aware correlation kernelsAttentive semantic alignment with offset aware correlation kernels
Attentive semantic alignment with offset aware correlation kernels
 
Application of Image Retrieval Techniques to Understand Evolving Weather
Application of Image Retrieval Techniques to Understand Evolving WeatherApplication of Image Retrieval Techniques to Understand Evolving Weather
Application of Image Retrieval Techniques to Understand Evolving Weather
 
Incremental collaborative filtering via evolutionary co clustering
Incremental collaborative filtering via evolutionary co clusteringIncremental collaborative filtering via evolutionary co clustering
Incremental collaborative filtering via evolutionary co clustering
 
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...Iccv2009 recognition and learning object categories   p2 c01 - recognizing a ...
Iccv2009 recognition and learning object categories p2 c01 - recognizing a ...
 
Yuki Oyama - Incorporating context-dependent energy into the pedestrian dynam...
Yuki Oyama - Incorporating context-dependent energy into the pedestrian dynam...Yuki Oyama - Incorporating context-dependent energy into the pedestrian dynam...
Yuki Oyama - Incorporating context-dependent energy into the pedestrian dynam...
 
Co-clustering of multi-view datasets: a parallelizable approach
Co-clustering of multi-view datasets: a parallelizable approachCo-clustering of multi-view datasets: a parallelizable approach
Co-clustering of multi-view datasets: a parallelizable approach
 
Review : PolarMask: Single Shot Instance Segmentation with Polar Representati...
Review : PolarMask: Single Shot Instance Segmentation with Polar Representati...Review : PolarMask: Single Shot Instance Segmentation with Polar Representati...
Review : PolarMask: Single Shot Instance Segmentation with Polar Representati...
 
CSTalks - Object detection and tracking - 25th May
CSTalks - Object detection and tracking - 25th MayCSTalks - Object detection and tracking - 25th May
CSTalks - Object detection and tracking - 25th May
 
Interaction Networks for Learning about Objects, Relations and Physics
Interaction Networks for Learning about Objects, Relations and PhysicsInteraction Networks for Learning about Objects, Relations and Physics
Interaction Networks for Learning about Objects, Relations and Physics
 
Seed net automatic seed generation with deep reinforcement learning for robus...
Seed net automatic seed generation with deep reinforcement learning for robus...Seed net automatic seed generation with deep reinforcement learning for robus...
Seed net automatic seed generation with deep reinforcement learning for robus...
 
Tracking and counting human in visual surveillance system
Tracking and counting human in visual surveillance systemTracking and counting human in visual surveillance system
Tracking and counting human in visual surveillance system
 
Shai Avidan's Support vector tracking and ensemble tracking
Shai Avidan's Support vector tracking and ensemble trackingShai Avidan's Support vector tracking and ensemble tracking
Shai Avidan's Support vector tracking and ensemble tracking
 
CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps
CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of MapsCPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps
CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps
 
Class Weighted Convolutional Features for Image Retrieval
Class Weighted Convolutional Features for Image Retrieval Class Weighted Convolutional Features for Image Retrieval
Class Weighted Convolutional Features for Image Retrieval
 
Deep image retrieval - learning global representations for image search - ub ...
Deep image retrieval - learning global representations for image search - ub ...Deep image retrieval - learning global representations for image search - ub ...
Deep image retrieval - learning global representations for image search - ub ...
 

Viewers also liked

A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
SlideShare
 

Viewers also liked (15)

Verification of UGC/Eyewitness Media: Challenges and Approaches
Verification of UGC/Eyewitness Media: Challenges and Approaches Verification of UGC/Eyewitness Media: Challenges and Approaches
Verification of UGC/Eyewitness Media: Challenges and Approaches
 
REVEAL Project - Trust and Credibility Analysis
REVEAL Project - Trust and Credibility AnalysisREVEAL Project - Trust and Credibility Analysis
REVEAL Project - Trust and Credibility Analysis
 
"Extracting Attributed Verification and Debunking Reports from Social Media: ...
"Extracting Attributed Verification and Debunking Reports from Social Media: ..."Extracting Attributed Verification and Debunking Reports from Social Media: ...
"Extracting Attributed Verification and Debunking Reports from Social Media: ...
 
News Impact Summit - Verification, Investigation and Digital Ethics – Hamburg...
News Impact Summit - Verification, Investigation and Digital Ethics – Hamburg...News Impact Summit - Verification, Investigation and Digital Ethics – Hamburg...
News Impact Summit - Verification, Investigation and Digital Ethics – Hamburg...
 
WWW2015 - RDSM2015 Workshop - Trust and Credibility Analysis
WWW2015 - RDSM2015 Workshop - Trust and Credibility AnalysisWWW2015 - RDSM2015 Workshop - Trust and Credibility Analysis
WWW2015 - RDSM2015 Workshop - Trust and Credibility Analysis
 
Web image size prediction for efficient focused image crawling
Web image size prediction for efficient focused image crawlingWeb image size prediction for efficient focused image crawling
Web image size prediction for efficient focused image crawling
 
Prix Italia 2015 - Verification in Social Newsgathering
Prix Italia 2015 - Verification in Social NewsgatheringPrix Italia 2015 - Verification in Social Newsgathering
Prix Italia 2015 - Verification in Social Newsgathering
 
Veracity & Velocity of Social Media Content during Breaking News
Veracity & Velocity of Social Media Content during Breaking NewsVeracity & Velocity of Social Media Content during Breaking News
Veracity & Velocity of Social Media Content during Breaking News
 
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
Mediarevealr: A social multimedia monitoring and intelligence system for Web ...
 
News-oriented multimedia search over multiple social networks
News-oriented multimedia search over multiple social networksNews-oriented multimedia search over multiple social networks
News-oriented multimedia search over multiple social networks
 
Cross-Media Konferenz "Think Cross - Change Media" in Magdeburg, Germany
 Cross-Media Konferenz "Think Cross - Change Media" in Magdeburg, Germany Cross-Media Konferenz "Think Cross - Change Media" in Magdeburg, Germany
Cross-Media Konferenz "Think Cross - Change Media" in Magdeburg, Germany
 
Geoparsing and Real-time Social Media Analytics - technical and social challe...
Geoparsing and Real-time Social Media Analytics - technical and social challe...Geoparsing and Real-time Social Media Analytics - technical and social challe...
Geoparsing and Real-time Social Media Analytics - technical and social challe...
 
A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
A Guide to SlideShare Analytics - Excerpts from Hubspot's Step by Step Guide ...
 
How to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & TricksHow to Make Awesome SlideShares: Tips & Tricks
How to Make Awesome SlideShares: Tips & Tricks
 
Getting Started With SlideShare
Getting Started With SlideShareGetting Started With SlideShare
Getting Started With SlideShare
 

Similar to Geotagging Social Media Content with a Refined Language Modelling Approach

“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
Edge AI and Vision Alliance
 
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
Konstantinos Demertzis
 

Similar to Geotagging Social Media Content with a Refined Language Modelling Approach (20)

In-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging PerformanceIn-depth Exploration of Geotagging Performance
In-depth Exploration of Geotagging Performance
 
Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...Placing Images with Refined Language Models and Similarity Search with PCA-re...
Placing Images with Refined Language Models and Similarity Search with PCA-re...
 
MediaEval 2016 - Placing Images with Refined Language Models and Similarity S...
MediaEval 2016 - Placing Images with Refined Language Models and Similarity S...MediaEval 2016 - Placing Images with Refined Language Models and Similarity S...
MediaEval 2016 - Placing Images with Refined Language Models and Similarity S...
 
CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015CERTH/CEA LIST at MediaEval Placing Task 2015
CERTH/CEA LIST at MediaEval Placing Task 2015
 
Where Next
Where NextWhere Next
Where Next
 
Carpita metulini 111220_dssr_bari_version2
Carpita metulini 111220_dssr_bari_version2Carpita metulini 111220_dssr_bari_version2
Carpita metulini 111220_dssr_bari_version2
 
Medical diagnosis classification
Medical diagnosis classificationMedical diagnosis classification
Medical diagnosis classification
 
MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...
MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...
MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...
 
ACL2015 Poster: Twitter User Geolocation Using a Unified Text and Network Pre...
ACL2015 Poster: Twitter User Geolocation Using a Unified Text and Network Pre...ACL2015 Poster: Twitter User Geolocation Using a Unified Text and Network Pre...
ACL2015 Poster: Twitter User Geolocation Using a Unified Text and Network Pre...
 
Lidar Point Cloud Classification Using Expectation Maximization Algorithm
Lidar Point Cloud Classification Using Expectation Maximization AlgorithmLidar Point Cloud Classification Using Expectation Maximization Algorithm
Lidar Point Cloud Classification Using Expectation Maximization Algorithm
 
LIDAR POINT CLOUD CLASSIFICATION USING EXPECTATION MAXIMIZATION ALGORITHM
LIDAR POINT CLOUD CLASSIFICATION USING EXPECTATION MAXIMIZATION ALGORITHMLIDAR POINT CLOUD CLASSIFICATION USING EXPECTATION MAXIMIZATION ALGORITHM
LIDAR POINT CLOUD CLASSIFICATION USING EXPECTATION MAXIMIZATION ALGORITHM
 
“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
“Person Re-Identification and Tracking at the Edge: Challenges and Techniques...
 
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
GeoAI: A Model-Agnostic Meta-Ensemble Zero-Shot Learning Method for Hyperspec...
 
MediaEval 2015 - UNED-UV @ Retrieving Diverse Social Images Task - Poster
MediaEval 2015 - UNED-UV @ Retrieving Diverse Social Images Task - PosterMediaEval 2015 - UNED-UV @ Retrieving Diverse Social Images Task - Poster
MediaEval 2015 - UNED-UV @ Retrieving Diverse Social Images Task - Poster
 
MediaEval 2015 - CERTH at MediaEval 2015 Synchronization of Multi-User Event ...
MediaEval 2015 - CERTH at MediaEval 2015 Synchronization of Multi-User Event ...MediaEval 2015 - CERTH at MediaEval 2015 Synchronization of Multi-User Event ...
MediaEval 2015 - CERTH at MediaEval 2015 Synchronization of Multi-User Event ...
 
geolocation twitter network text geotagging
geolocation twitter network text geotagginggeolocation twitter network text geotagging
geolocation twitter network text geotagging
 
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTIONHOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
 
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTIONHOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
HOL, GDCT AND LDCT FOR PEDESTRIAN DETECTION
 
Combinatorial optimization and deep reinforcement learning
Combinatorial optimization and deep reinforcement learningCombinatorial optimization and deep reinforcement learning
Combinatorial optimization and deep reinforcement learning
 
Analysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approachAnalysis of Educational Robotics activities using a machine learning approach
Analysis of Educational Robotics activities using a machine learning approach
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 

Geotagging Social Media Content with a Refined Language Modelling Approach

  • 1. Geotagging Social Media Content with a Refined Language Modelling Approach Georgios Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris Centre for Research and Technology Hellas (CERTH) – Information Technologies Institute (ITI) PAISI 2015, May 19, 2015, Ho Chi Minh City, Vietnam
  • 2. Where is it? #2 Depicted landmark Eiffel Tower Location Paris, Tennessee Keyword “Tennesee” is very important to correctly place the photo. Source (Wikipedia): http://en.wikipedia.org/wiki/Eiffel_Tow er_(Paris,_Tennessee)
  • 3. The Problem • A lot of multimedia content is associated with geographic information • Being able to collect and analyze large amounts of geotagged content could be very useful for several applications, e.g., situational awareness in incidents such as natural disasters, verification, geographic trends, etc. • Yet, only a very small percentage of Web media content carries explicit information (i.e. GPS coordinates), for instance ~1% of tweets are geotagged • To this end, methods that can infer the geographic location of Web multimedia content are of interest. #3
  • 4. A Refined Language Model for Geotagging • Extend and improve the widely used Language Model for the problem of location estimation from text metadata • The proposed improvements include: – Feature selection based on a cross-validation approach – Feature weighting based on spatial entropy – Multiple resolution grids • Extensive evaluation on a public benchmark shows highly competitive performance and reveals new insights and challenges #4
  • 5. Related Work: Gazetteer-based Methods that use large dictionaries and Volunteered Geographic Information (VGI), e.g., Geonames, Yahoo! GeoPlanet, OpenStreetMap, etc. • Semantics-based IR approach for integrating gazetteers and VGI (Kessler et al., 2009) • Similarity matching mediating multiple gazetteers in a meta-gazetteer service (Smart et al., 2010) • Comma groups extracted with heuristic methods from lists of toponyms (Lieberman et al., 2010) #5
  • 6. Related Work: Language Models Language Models: large corpora of geotagged text to create location-specific language model, i.e. what are the most frequent keywords for a given location • Base approach (Serdyukov et al., 2009) • Disjoint dynamically sized cells (Hauff et al., 2012) • User frequency instead of term frequency (O’Hare & Murdock, 2012) • Clustering, use of χ2 for feature selection and similarity search (Van Laere et al., 2011) #6
  • 7. Related Work: Multimodal Approaches Multimodal approaches do not only use text, but also leverage the visual content and other social metadata of geotagged multimedia. • Combination of text metadata and visual content at two levels of granularity, city- (100km) and landmark-level (100m) (Crandall et al., 2009) • Build user models leveraging user’s upload history, SN data and hometown (Trevisiol et al., 2013) • Hierarchical approach using both text-based and visual similarity (Kelm et al., 2011) #7
  • 8. Related Work: MediaEval Placing Task • Yearly benchmarking task where different approaches compete – Each participant can submit up to 5 runs with different instances/configurations of their method • Dataset for Placing Task 2014 – Flickr CC-licensed images & videos, subset of YFCC 100M – Training: 5M, Testing: 510K (multiple subsets of increasing size are used for reporting) • Evaluation – Estimated location of test image/video is compared against the known one, and it is checked whether it belongs to a circle of radius of 10m, 100m, 1km, 10km, 100km and 1000km – Then, the percentage of images/videos that were correctly placed within each radius are reported, e.g., P@1km • Competing approaches: both gazetteer- and LM-based #8
  • 10. Geographic Language Model (1/2) • Training data: Corpus Dtr of images and videos • Test data: Corpus Dts • For each item (either in training or test data), we have: user id, title, tags, description • Title and tags of training images used for building the model. For testing, description is used only if the item has neither title nor tags associated with it. • Pre-processing: punctuation and symbol removal, lowercasing, numeric tags removed, composite phrases (e.g. “new+york”  “new”, “york”) are split into their components #10
  • 11. Geographic Language Model (2/2) • Generate rectangular grid C of areas (cells) of size 0.01⁰ x 0.01⁰ (~1km x 1km near equator) • For each cell and each tag in the training corpus, compute the tag-cell probability: 𝑝 𝑡 𝑐 = 𝑁 𝑢 𝑁𝑡 – Nu: number of users in Dtr that used tag t inside the borders of c – Nt: total count of users that used tag t in any cell • For a new text T with N tags, compute the Most Likely Cell (MLC) cj based on the following: 𝑐𝑗 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑖 𝑘=1 𝑁 𝑝(𝑡 𝑘|𝑐𝑖) #11
  • 12. Geographic Language Model: Example #12 new: 0.15 york: 0.27 manhattan: 0.45 liberty: 0.33 … nyc: 0.52
  • 13. Feature Selection • Retaining all possible tags of the training set might result in noise and overfitting and makes the resulting model very memory-demanding  select only those tags that are really discriminative • Use a variant of Cross-Validation on the training set: – split the training set in p folds (=10 in our tests) – use the p-1 folds for creating the LM and one for computing its accuracy (P@r, where r is the radius we are interested to optimize) – compute tag geographicity: 𝑡𝑔𝑒𝑜 𝑡 = 𝑁 𝑟 𝑁𝑡 , where Nr is the number of correctly classified items where tag t appears and Nt is the total number of items where tag t appears – select only tags that exceed threshold θtgeo and that have been used by a minimum number of users θu #13
  • 14. Feature Weighting using Spatial Entropy • We want to penalize tags that appear in many different places and to give more importance to tags that appear only for specific places. • To this end, we define a measure of the stochasticity of the tag’s appearance, i.e. its spatial entropy: 𝑒 𝑡 = − 𝑖=1 𝑀 𝑝 𝑡 𝑐𝑖 log 𝑝(𝑡|𝑐𝑖) where M is the total number of cells. • Once the entropy values are computed, we apply Gaussian normalization to suppress the weight of tags that take extreme values: 𝑁 𝑒 𝑡 , 𝜇, 𝜎 = 1 𝜎 2𝜋 𝑒−( 𝑒 𝑡 −𝜇 2𝜎 )2 where N is the Gaussian function and μ, σ are the mean value and variance of the distribution and are estimated on Dtr. • Consequently, the MLC is computed based on: 𝑐𝑗 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑖 𝑘=1 𝑁 𝑝(𝑡 𝑘|𝑐𝑖) ∙ 𝑁 𝑒 𝑡 𝑘 , 𝜇, 𝜎 #14
  • 15. Entropy Histogram & Gaussian Weighting #15 + --
  • 16. Similarity Search • Having assigned an item to the Most Likely Cell (MLC), we refine the location estimate by searching for the most similar item in the cell. • The k most similar items are retrieved (from Dtr) using Jaccard similarity on the respective sets of tags: 𝐽 𝑥, 𝑦 = |𝑇𝑥 ∩ 𝑇𝑦| |𝑇𝑥 ∪ 𝑇𝑦| • The final estimation is the centre of gravity of the k most similar images: 𝑙𝑜𝑐 𝑥 = 1 𝑘 𝑖=1 𝑘 𝐽 𝑥, 𝑦𝑖 𝛼 𝑙𝑜𝑐(𝑦𝑖) where α determines how strongly the result is influenced by the most similar items. • If less than k images are retrieved, then only those are used in the above calculation. If no similar images are retrieved, then the centre of MLC is provided as output. #16
  • 17. Multiple Resolution Grids • To increase the granularity of the prediction and at the same time its reliability, we devised the following dual grid scheme: – we build two LMs: one of size 0.01° x 0.01° (coarse granularity) and one of size 0.001° x 0.001° (fine granularity) – conduct location estimations based on both – if the fine granularity estimations falls within the cell of the estimation based on the coarse granularity, then we select the fine granularity – otherwise, we select the coarse (since we consider it by default more reliable) #17
  • 18. Evaluation • Benchmark dataset: MediaEval 2014 • Training set: 5M, Test set: 510K • All experiments conducted on the full test set (510K) • Two stages of evaluation: – participation in contest (with a limited version of the proposed approach) – post-contest performance exploration #18
  • 19. Evaluation: MediaEval 2014 Contest (1/4) • Out of the five runs, three were based on variations of the presented approach – run1: LM + feature weighting with spatial entropy + similarity search + multiple resolution grid – run4: LM only – run5: LM + similarity search (similarity search parameters: α=1, k=4) • Performance was measured with P@r, where r = 10m, 100m, 1km, 10km, 100km and 1000km #19
  • 20. Evaluation: MediaEval 2014 Contest (2/4) #20 • Proposed improvements (run1) outperform base approach (run4) and base approach + similarity search (run5) • The improvement is more pronounced in the small ranges (10m, 100m, 1km)
  • 21. Evaluation: MediaEval 2014 Contest (3/4) #21 Proposed Proposed
  • 22. Evaluation: MediaEval 2014 Contest (4/4) #22 Number of image tags
  • 23. Post-Contest Evaluation Explore the role of different factors: • Big training set (YFCC100M): ~48M geotagged items • Feature Selection (FS) • Feature Weighting with Spatial Entropy (SE) • Multiple Resolution Grid (MG) • Similarity Search (SS) Two settings: • FAIR: All users from the training set are completely removed from the test set • OVERFIT: Users are not removed from the test set even when some of their media items are included in the training set. #23
  • 24. Post-Contest Evaluation #24 • Clear improvement with the addition of MG and SS • The proposed improvements together with the use of the bigger dataset make the approach perform better than all other methods in MediaEval 2014
  • 25. Geographic Error Analysis • More data leads to lower error across the globe. • Several small US cities suffer from low accuracy due to having names of large European cities. #25 ALL + YFCC100M run 4
  • 26. Big Data vs. Complex Algorithms #26 Using 10x more data for training led to equivalent performance with using more complex algorithm (LM + extensions) with less data!
  • 27. Placeability of Media Items • Sum of cell-tag probabilities is a good indicator of how confident we are in the decision of the classifier. #27
  • 30. Conclusion • Key contributions – Improved geotagging approach, extending the widely used language model in three ways: feature selection, weighting, multiple resolution grids – Thorough analysis of geotagging accuracy offering new insights and highlighting new challenges • Future Work – Exploit visual features to improve (currently visual-only approaches perform very poorly) – Integrate gazetteer and structured location data sources (e.g. Foursquare venues, OpenStreetMap, etc.) – Evaluate in more challenging settings and datasets (e.g. Twitter, Instagram) #30
  • 31. References (1/2) • C. Kessler, K. Janowicz, and M. Bishr. An Agenda for the Next Generation Gazetteer: Geographic Information Contribution and Retrieval. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 91100. ACM, 2009 • P. Smart, C. Jones, and F. Twaroch. Multi-source Toponym Data Integration and Mediation for a Meta-gazetteer Service. In Proceedings of the 6th international conference on Geographic information science. GIScience10. Springer-Verlag, Berlin, Heidelberg, 234248, 2010 • M.D. Lieberman, H. Samet, and J. Sankaranayananan. Geotagging: using Proximity, Sibling, and Prominence Clues to Understand Comma Groups. In Proceedings of the 6th Workshop on Geographic Information Retrieval, 2010 • P. Serdyukov, V. Murdock, and R. Van Zwol. Placing Flickr Photos on a Map. In SIGIR09, pages 484-491, New York, NY, USA, 2009. ACM • C. Hauff and G. Houben. Geo-location Estimation of Flickr images: Social Web based Enrichment. ECIR 2012, p. 85-96. Springer LNCS 7224, April 1-5 2012 • N. O'Hare, and V. Murdock. Modeling Locations with Social Media. Information Retrieval, pp. 133, 2012 • O. Van Laere, S. Schockaert, and B. Dhoedt. Finding Locations of Flickr Resources using Language Models and Similarity Search. ICMR 11, New York, USA, 2011. ACM #31
  • 32. References (2/2) • D.J. Crandall, L. Backstrom, D. Huttenlocher, and J. Kleinberg. Mapping the World's Photos. In Proceedings of the 18th international conference on World wide web, WWW 09, pages 761770, New York, NY, USA, 2009. ACM • M. Trevisiol, H. Jegou, J. Delhumeau, and S. Gravier. Retrieving Geo-Location of Videos with a Divide and Conquer Hierarchical Multimodal Approach. ICMR13, Dallas, United States, April 2013. ACM • P. Kelm, S. Schmiedeke, and T. Sikora. A Hierarchical, Multi-modal Approach for Placing Videos on the Map using Millions of Flickr Photographs. In Proceedings of the 2011 ACM Workshop on Social and Behavioural Networked Media Access, SBNMA 11, pages 1520, New York, NY, USA, 2011. ACM #32
  • 33. Thank you! • Resources: Slides: http://www.slideshare.net/sympapadopoulos/reveal-geotagging Code: https://github.com/socialsensor/multimedia-geotagging Benchmark: http://www.multimediaeval.org/mediaeval2014/placing2014/ • Get in touch: @sympapadopoulos / papadop@iti.gr George Kordopatis / georgekordopatis@iti.gr #33

Editor's Notes

  1. http://irevolution.net/2014/04/03/using-aidr-to-collect-and-analyze-tweets-from-chile-earthquake/