The sixth edition of the Placing Task at MediaEval introduces two new sub-tasks: (1) locale-based placing, which emphasizes the need to move away from an evaluation purely based on latitude and longitude towards an entity-centered evaluation, and (2) mobility-based placing, which addresses predicting missing locations within a sequence of movements; the latter is a specific real-world use case that so far has received little attention within the research community. Two additional changes over the previous years are the introduction of open source organizer baselines for both sub-tasks shortly after the official data release, and the implementation of a live leaderboard, which allows the participants to gain insights into the effectiveness of their approaches compared to the official baselines and in relation to each other at an early stage, before the actual run submissions are due. http://ceur-ws.org/Vol-1436/ http://www.multimediaeval.org

1. PLACING TASK 2015
Jaeyoung Choi (ICSI, Berkeley / TU Delft, Netherlands)
Claudia Hauff (TU Delft, Netherlands)
Olivier Van Laere (Blueshift Labs, San Francisco)
Bart Thomee (Yahoo Labs, San Francisco)
Sept 15th, 2015
Wurzen, Germany

2. TASK CHANGES
• two new sub-tasks
• locale-based sub-task
• mobility-based sub-task
• organizer baselines
• live leaderboard

3. LOCALE-BASED SUBTASK
• given a video or a photo, place it as accurately as
possible on a map

4. LOCALE-BASED SUBTASK
• participants were given a geographic hierarchy of
locales across the world
• principally based on the GADM administrative boundaries
• supplemented with neighborhood data from ClickThatHood
• participants choose the locale in the hierarchy where
they most confidently believe a photo/video was taken
• or, if sufficiently confident, they pick a longitude and
latitude coordinate

5. GADM
DATABASE OF GLOBAL ADMINISTRATIVE AREAS
http://www.gadm.org/

6. CLICKTHAT ‘HOOD!
click-that-hood.com

7. HIERARCHY DATA EXAMPLE
• United States@CA|Calif.|California@San
Francisco@South of Market
• ‘@‘ separates each layer in the hierarchy
• ‘|’ separates any alternative name spellings

8. MOBILITY-BASED SUBTASK
• participants were given a sequence of photos captured
by a user within a certain city
• not all of the photos within the sequence were geo-tagged
• participants were asked to estimate the location of all
photos with missing geo-coordinates

9. TASK DATASET
• drawn from theYahoo Flickr Creative Commons 100
Million (YFCC100M) dataset
training testing
#photos #videos #photos #videos
locale-based 4,672,382 22,767 931,573 18,316
mobility-based 148,349 0 33,026 0

10. PRECOMPUTED FEATURES
• textual metadata
• as included inYFCC100M
• visual features
• LIRE, GIST, SIFT
• audio features
• MFCC, Pitch (Kaldi, SAcC)
paul bica

11. ORGANIZER BASELINES
• two open-source baselines were provided, one for
each subtask
• locale-based baseline - LM-based using textual metadata [Van
Laere et al. 2013]
• mobility-based baseline - extrapolates location from temporal
neighbors
[Hauff et al. SIGIR 2012]
https://github.com/ovlaere/placing-text/blob/mediaeval2015/README_placing2015.md
https://github.com/chauff/ImageLocationEstimation

12. LIVE LEADERBOARD
• participants could submit runs and view their relative
standing toward others
• evaluated on a development set, approx. 30% of test set
• considered to be useful by participants

13. TASK EVALUATION
• locale-based subtask
• hierarchical distance between ground truth locale and
predicted locale in the place hierarchy
• geographic distance between ground truth coordinate and
predicted coordinate
• mobility-based subtask
• geographic distance-based metric
• custom formula was used for the hierarchy-based metric
• Karney’s formula was used for the distance-based metric

14. TASK EVALUATION

15. RUNS
• run 1
• only provided textual metadata may be used
• run 2
• only provided visual & aural features may be used
• run 3
• only provided textual metadata, visual features and the visual &
aural features may be used
• run 4–5
• everything is allowed, except for crawling the exact items
contained in the test set, or any items by a test user taken
within 24 hours before the first and after the last timestamp
of a photo sequence in the mobility test set

16. PARTICIPANTS STATISTICS
locale mobility
run1 run2 run3 run4/5 run1 run2 run3 run4/5
CERTH/
CEA LIST
O O O O
RECOD O O O O
ImCube O O O O O O O O
JKU_
Satellite
O O O
Geo_ML O
TUDelft O (late)

17. LOCALE RESULTS - RUN 1
cumulativepercentage
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Geo_ML
Baseline
cumulativepercentage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Geo_ML
Baseline

18. LOCALE RESULTS - RUN 2
cumulativepercentage
0
8
16
24
32
40
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD TUD*
cumulativepercentage
0
8
16
24
32
40
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD TUD*

19. LOCALE RESULTS - RUN 3
cumulativepercentage
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Baseline
cumulativepercentage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Baseline

20. LOCALE RESULTS - RUN 4/5
cumulativepercentage
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Baseline
cumulativepercentage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCube
RECOD Baseline

21. MOBILITY RESULTS - RUN 1+2
cumulativepercentage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite
cumulativepercentage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite

22. MOBILITY RESULTS - RUN 3+4
cumulativepercentage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite
cumulativepercentage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube

23. CONCLUSIONS
• no visible trend; many different approaches
• pre-processing, language model, clustering, GP, etc.
• limited time available for participants
• leaderboard, features were helpful
• local semantic features from visual cue to be added