In this work, we describe the approach proposed by the RECOD team for the Placing Task, Locale-based sub-task, at MediaEval 2015. Our approach is based on the use of as much evidence as possible (textual, visual, and/or audio descriptors) to automatically assign geographical locations to images and videos.
http://ceur-ws.org/Vol-1436/
http://www.multimediaeval.org
Web & Social Media Analytics Previous Year Question Paper.pdf
MediaEval 2015 - RECOD@Placing Task of MediaEval 2015
1. RECOD @ Placing Task of
MediaEval 2015
L. T. Li1
, J. A. V. Muñoz1
, J. Almeida1,3
,
R. T. Calumby1,4
, O. A. B. Penatti1,2
, I. C. Dourado1
,
K. Nogueira6
, P. R. Mendes Júnior1
,
D. C. G. Pedronette1,5
, J. A. dos Santos6
,
M. A. Gonçalves6
, and R. S. Torres1
Daniel Moreira
On behalf of the authors.
1. UNICAMP – 2. UNIFESP – 3. SAMSUNG – 4. UEFS – 5. UNESP – 6. UFMG
– BRAZIL –
2. 2015 Participation
● We focused on the Localization subtask;
● Innovations concerning the last year (2014)
– Rank agreggation based on Genetic
Programming;
– Geocoding improvement with Ranked List Density
Analysis (RLDA).
3. Submitted Runs
● Run 1
Textual Features with Genetic Programming
– Image and Video
Descriptors
BM25*
TF-IDF*
IBS*
LMD*
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
Single ranked list
(GP-Agg - based)
* All from Lucene package (http://lucene.apache.org/core/)
4. GP-Agg Framework
Parameter Value
Number of
generations
30
Genetics
operators
Reproduction, Mutation,
Crossover
Fitness
functions
FFP1, WAS, MAP, NDCG
Rank Agg.
methods
CombMAX, CombMIN,
CombSUM, CombMED,
CombANZ, CombMNZ,
RLSim, BordaCount, RRF,
MRA
5. Submitted Runs
● Run 1
Textual Features with Genetic Programming
– Image and Video
Descriptors Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
Single ranked list
(GP-Agg - based)
BM25*
TF-IDF*
IBS*
LMD*
* All from Lucene package (http://lucene.apache.org/core/)
6. Submitted Runs
● Run 2
Visual Features with RLDA
– Image
Descriptor
BIC
Rank
BIC
Ranked list
Geocoding
Improvement
Improved geocode
(top 100 RLDA-based)
7. Ranked List Density Analysis
Ranked list
1~N
ID1 LAT1 LONG1
ID2 LAT2 LONG2
… … ...
IDX LATX LONGX
Top X items'
lat/long
defined as
points of a
OPF cluster
Node v:
(lat1,long1)
edge (v, u):
connect k-nn d(v, u)
d(v, u): geo-
distance between v
and u
k=3
u: (lat2,long2)
8. Submitted Runs
● Run 2
Visual Features with RLDA
– Image
Descriptor
BIC
Rank
BIC
Ranked list
Improved geocode
(top 100 RLDA-based)
Geocoding
Improvement
9. Submitted Runs
● Run 2
Visual Features with RLDA
– Video
Descriptors
LIRE 1
...
LIRE n
HMP
Ranks
Ranked list 1
...
Ranked list n
Ranked list n+1
Rank Aggregation
Single ranked list
(GP-Agg - based)
Improved geocode
(top 100 RLDA-based)
Geocoding
Improvement
10. Submitted Runs
● Run 3
Multimodal Solution with RLDA
– Image
Descriptors
BM25
TF-IDF
BS
LMD
BIC
...
SCD
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Ranked list 5
…
Ranked list 8
Rank Aggregation
Single ranked list
(GP-Agg - based)
11. Submitted Runs
● Run 3
Multimodal Solution with RLDA
– Video
Descriptors
BM25
TF-IDF
BS
LMD
HMP
...
MFCC
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Ranked list 5
…
Ranked list n
Rank Aggregation
Single ranked list
(GP-Agg - based)
12. Submitted Runs
● Run 4
Textual with RLDA
– Image and Video
Descriptors
BM25
TF-IDF
BS
LMD
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
and Improvement
Single ranked list
(top 5 RLDA-based)
15. Conclusions
● GP-Agg automatically combines lists and
aggregation functions.
● Top-N RLDA improves even GP-Agg results.
● RLDA was better (see Run 4) than using GP-
Agg alone.
● There is room for improving the GP-Agg
approach.
16. Future Work
● Develop other fitness fuctions in the GP-Agg
approach.
● Use more visual descriptors.
● Evaluate different clustering strategies.
24. Visual Features (HMP): Extracting
●
Histograms of Motion Patterns
●
Keyframes: Not used
●
Applying an algorithm to compare video
sequence
(1)partial decoding;
(2)feature extraction;
(3)signature generation.
“Comparison of video sequences with histograms of
motion patterns”, J. Almeida et al. ICIP, 2011.
25. Visual Features (HMP): overview
[Almeida et al., Comparison of video sequences with histograms of motion patterns. ICIP 2011]