In this work, we describe the approach proposed by the RECOD team for the Placing Task, Locale-based sub-task, at MediaEval 2015. Our approach is based on the use of as much evidence as possible (textual, visual, and/or audio descriptors) to automatically assign geographical locations to images and videos. http://ceur-ws.org/Vol-1436/ http://www.multimediaeval.org

1. RECOD @ Placing Task of
MediaEval 2015
L. T. Li1
, J. A. V. Muñoz1
, J. Almeida1,3
,
R. T. Calumby1,4
, O. A. B. Penatti1,2
, I. C. Dourado1
,
K. Nogueira6
, P. R. Mendes Júnior1
,
D. C. G. Pedronette1,5
, J. A. dos Santos6
,
M. A. Gonçalves6
, and R. S. Torres1
Daniel Moreira
On behalf of the authors.
1. UNICAMP – 2. UNIFESP – 3. SAMSUNG – 4. UEFS – 5. UNESP – 6. UFMG
– BRAZIL –

2. 2015 Participation
● We focused on the Localization subtask;
● Innovations concerning the last year (2014)
– Rank agreggation based on Genetic
Programming;
– Geocoding improvement with Ranked List Density
Analysis (RLDA).

3. Submitted Runs
● Run 1
Textual Features with Genetic Programming
– Image and Video
Descriptors
BM25*
TF-IDF*
IBS*
LMD*
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
Single ranked list
(GP-Agg - based)
* All from Lucene package (http://lucene.apache.org/core/)

4. GP-Agg Framework
Parameter Value
Number of
generations
30
Genetics
operators
Reproduction, Mutation,
Crossover
Fitness
functions
FFP1, WAS, MAP, NDCG
Rank Agg.
methods
CombMAX, CombMIN,
CombSUM, CombMED,
CombANZ, CombMNZ,
RLSim, BordaCount, RRF,
MRA

5. Submitted Runs
● Run 1
Textual Features with Genetic Programming
– Image and Video
Descriptors Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
Single ranked list
(GP-Agg - based)
BM25*
TF-IDF*
IBS*
LMD*
* All from Lucene package (http://lucene.apache.org/core/)

6. Submitted Runs
● Run 2
Visual Features with RLDA
– Image
Descriptor
BIC
Rank
BIC
Ranked list
Geocoding
Improvement
Improved geocode
(top 100 RLDA-based)

7. Ranked List Density Analysis
Ranked list
1~N
ID1 LAT1 LONG1
ID2 LAT2 LONG2
… … ...
IDX LATX LONGX
Top X items'
lat/long
defined as
points of a
OPF cluster
Node v:
(lat1,long1)
edge (v, u):
connect k-nn d(v, u)
d(v, u): geo-
distance between v
and u
k=3
u: (lat2,long2)

8. Submitted Runs
● Run 2
Visual Features with RLDA
– Image
Descriptor
BIC
Rank
BIC
Ranked list
Improved geocode
(top 100 RLDA-based)
Geocoding
Improvement

9. Submitted Runs
● Run 2
Visual Features with RLDA
– Video
Descriptors
LIRE 1
...
LIRE n
HMP
Ranks
Ranked list 1
...
Ranked list n
Ranked list n+1
Rank Aggregation
Single ranked list
(GP-Agg - based)
Improved geocode
(top 100 RLDA-based)
Geocoding
Improvement

10. Submitted Runs
● Run 3
Multimodal Solution with RLDA
– Image
Descriptors
BM25
TF-IDF
BS
LMD
BIC
...
SCD
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Ranked list 5
…
Ranked list 8
Rank Aggregation
Single ranked list
(GP-Agg - based)

11. Submitted Runs
● Run 3
Multimodal Solution with RLDA
– Video
Descriptors
BM25
TF-IDF
BS
LMD
HMP
...
MFCC
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Ranked list 5
…
Ranked list n
Rank Aggregation
Single ranked list
(GP-Agg - based)

12. Submitted Runs
● Run 4
Textual with RLDA
– Image and Video
Descriptors
BM25
TF-IDF
BS
LMD
Ranks
Ranked list 1
Ranked list 2
Ranked list 3
Ranked list 4
Rank Aggregation
and Improvement
Single ranked list
(top 5 RLDA-based)

13. Results – 2015 Global
Run 1
Run 2
Run 3
Run 4
0 5 10 15 20 25 30 35 40 45 50
0,15
0
0,14
0,12
0,54
0,01
0,53
0,62
5,49
0,09
5,35
6,44
19,75
0,44
19,11
21,74
36,6
1,99
35,31
38,38
44,89
3,57
43,26
46,91
58,97
20,38
57,67
63,22
1m
10m
100m
1km
10km
100km
1,000km
GP-Agg Combined Textual
GP-Agg Combined Non-textual
GP-Agg Multimodal
Textual RLDL

14. Results – Median Distance
With Metadata Runs
Run4
Run3
Run1
50 150 250 350 450 550 650 750
309.86
394.89
196.01

15. Conclusions
● GP-Agg automatically combines lists and
aggregation functions.
● Top-N RLDA improves even GP-Agg results.
● RLDA was better (see Run 4) than using GP-
Agg alone.
● There is room for improving the GP-Agg
approach.

16. Future Work
● Develop other fitness fuctions in the GP-Agg
approach.
● Use more visual descriptors.
● Evaluate different clustering strategies.

17. Acknowledgments
● MediaEval 2015
● FAPESP
● CNPq
● CAPES
● Samsung

18. Thank You!
{lintzyli,pedro.mendes,luis.pereira,rtorres}@ic.unicamp.br,
jurandy.almeida@unifesp.br,rtcalumby@ecomp.uefs.br,
o.penatti@samsung.com,jalvarm.acm@gmail.com,
icaro.dourado@students.ic.unicamp.br,daniel@rc.unesp.br,
{keiller.nogueira,jefersson, mgoncalv}@dcc.ufmg.br
L. T. Li, J. A. V. Muñoz, J. Almeida,
R. T. Calumby, O. A. B. Penatti, I. C. Dourado,
K. Nogueira, P. R. Mendes Júnior,
D. C. G. Pedronette, J. A. dos Santos,
M. A. Gonçalves, and R. S. Torres

19. Support Slides

20. Run1 Run3 Run4
Min. : 0.000 Min. : 0.000 Min. : 0.000
1st Qu.: 1.897 1st Qu.: 2.124 1st Qu.: 1.482
Median : 309.865 Median : 394.889 Median : 196.008
Mean : 2913.598 Mean : 2976.632 Mean : 2483.614
3rd Qu.: 5573.930 3rd Qu.: 5766.894 3rd Qu.: 3798.622
Max. :19959.808 Max. :19959.808 Max. :19954.130
Basic analysis of distances (km) in Test set:
from Predicted to Expected lat/long
Run2
Min. : 0
1st Qu.: 1240
Median : 5883
Mean : 5597
3rd Qu.: 8637
Max. :19972

21. Videos-only Test Results (%)
Run 1: GP-Agg Textual
Run 2: GP-Agg only visual (HMP+ all Lire) + RLDA (top100)
Run 3: GP-Agg multimodal (text, visual, audio)
Run 4: BM25_RLDA (top 5)
Run 1 Run 2 Run 3 Run 4
1m 0.08 0 0.08 0.06
10m 0.4 0 0.37 0.41
100m 5.46 0.01 5.13 5.79
1km 17.62 0.02 16.74 17.89
10km 32.44 0.11 32.1 32.24
100km 40.27 3.8 39.69 39.92
1,000km 54.13 20.39 53.67 55.68
10,000km 90.57 91.97 91.5 93.16

22. Video-only Summary
Run1 Run2 Run3
Min. : 0.000 Min. : 0.05 Min. : 0.000
1st Qu.: 2.914 1st Qu.: 1304.13 1st Qu.: 3.138
Median : 619.747 Median : 6351.71 Median : 660.864
Mean : 3191.163 Mean : 5709.17 Mean : 3158.724
3rd Qu.: 6035.939 3rd Qu.: 8463.49 3rd Qu.: 6154.147
Max. :19656.624 Max. :19596.38 Max. :19524.628
Run4
Min. : 0.000
1st Qu.: 2.992
Median : 547.648
Mean : 2879.899
3rd Qu.: 5548.780
Max. :19656.624

23. GP-Agg Individual example – Run 1
Input: bm25 (description, fusion, tags, title), tf-
idf(description, fusion, tags, title), lmd_fusion, ibs_fusion
Individual:
CombSUM( MRA( CombMNZ( RRF(ibs_fusion,lmd_fusion),
CombMNZ(td-idf_fusion, bm25_fusion), RLSim(lmd_fusion, bm25_tags,
tf-idf_fusion)), CombMNZ(CombSUM(tf-idf_tags, tf-idf_fusion, tf-
idf_tags), CombMIN(tf-idf_description, bm25_description, tf-idf_tags),
RLSim(bm25_title, tf-idf_title, tf-idf_title)), RRF(CombMAX(bm25_fusion,
bm25_tags), RRF(tf-idf_fusion, ibs_fusion, tf-idf_fusion), BordaCount(tf-
idf_fusion, bm25_fusion, tf-idf_tags))), RLSim(CombSUM(CombSUM(tf-
idf_tags, tf-idf_tags, lmd_fusion), bm25_fusion), bm25_fusion))

24. Visual Features (HMP): Extracting
●
Histograms of Motion Patterns
●
Keyframes: Not used
●
Applying an algorithm to compare video
sequence
(1)partial decoding;
(2)feature extraction;
(3)signature generation.
“Comparison of video sequences with histograms of
motion patterns”, J. Almeida et al. ICIP, 2011.

25. Visual Features (HMP): overview
[Almeida et al., Comparison of video sequences with histograms of motion patterns. ICIP 2011]

26. HMP: Comparing Video

27. OPF Density formula
● d(s,t) distânce from s to t (used haversine dist.)
● A(s): list of adjacency of s.
● directed graph