Paper: http://ceur-ws.org/Vol-2283/MediaEval_18_paper_10.pdf Youtube: https://youtu.be/yq1nIPc6dWw Benjamin Bischke, Patrick Helber, Zhengyu Zhao, Jens de Bruijn, Damian Borth, The Multimedia Satellite Task at MediaEval 2018. Proc. of MediaEval 2018, 29-31 October 2018, Sophia Antipolis, France. Abstract: This paper provides a description of the MediaEval 2018 Multimedia Satellite Task. The primary goal of the task is to extract and fuse content of events which are present in Satellite Imagery and Social Media. Establishing a link from Satellite Imagery to Social Multimedia can yield to a comprehensive event representation which is vital for numerous applications. Focusing on natural disaster events, the main objective of the task is to leverage the combined event representation within the context of emergency response and environmental monitoring. In particular, our task focuses on flooding events and consists of two subtasks. The first Image Classification from Social Media subtask requires participants to retrieve images from Social Media which show a direct evidence for road passability during flooding events. The second task Flood Detection from Satellite Images aims to extract potentially flooded road sections from satellite images. The task seeks to go beyond state-of-the-art flooding map generation by focusing on information about the road passability and accessibility to urban infrastructure. Such information shows a clear potential to complement information from social images with satellite imagery is of vital importance for emergency management. Presented by Benjamin Bischke

1. DFKI – KM - DLCC
Multimedia Satellite Task 2018
ALL RIGHTS RESERVED. No part of this work may be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage
and retrieval system without expressed written permission from the authors.
MediaEval Benchmark 2018
Benjamin Bischke, Patrick Helber, Zhengyu Zhao,
Jens de Brujin, Damian Borth
Emergency Response for Flooding Events
Sophia Antipolis, 30.10.2018

2. DFKI – SDS
2
Natural Disasters - Emergency Response
2
• 2016 & 2017 were one of the most impact-full hurricane seasons
• Hurricane Matthew, Harvey, Irma and Maria caused a high economic
and influence on private life
• The number of natural disasters is increasing

3. DFKI – SDS
3
Current Natural Disasters (October 2018)
3

4. DFKI – SDS
4
Current Natural Disasters (October 2018)
4
Floods
Hurricane
Floods

5. DFKI – SDS
5
Flooding Events
5

6. DFKI – SDS
6
6
Natural Disasters – Rappid Mapping
• Creating maps for
emergency response
are semi-automatic
• Fast Access to
information is
prioritized over
accurracy

7. DFKI – SDS7
Is Satellite Imagery enough?
Washington Post

8. DFKI – SDS
Is Satellite Imagery enough? (Idea)
8
Washington Post

9. DFKI – KM - DLCC
Multimedia Satellite Task 2017

10. DFKI – SDS
• Multimedia Satellite Task 2017 with two subtasks:
– Disaster Image Retrieval from Social Media (DIRSM)
– Flood Detection in Satellite Imagery (FDSI)
• 11 Teams actively participated from all the world (Brasil, Australia,
Greece, Brunei, Italy, UK, Germany, Netherlands, Norway…)
– More than 60 submission on two subtasks
DigitalGlobe, October 2017
Emergency Response
MediaEval Workshop 2017
Multimedia Satellite Task 2017 - Overview
10

11. DFKI – SDS
Multimedia Satellite Task 2017 - Overview
11
Disaster Image Retrieval from Social Media (DIRSM)
• Based on more than 6000 Flickr Images
• Fusion of multimodal information very important (visual & text)
• Features of CNNs trained on Places and VSO achieved high results

12. DFKI – SDS
Disaster Image Retrieval from Social Media (DIRSM)
• Based on more than 6000 Flickr Images
• Fusion of multimodal information very important (visual & text)
• Features of CNNs trained on Places and VSO achieved high results
Flood Detection in Satellite Imagery (FDSI)
• Based on Planet Images for 7 events (4 meter spatial resolution)
• Incorporation of NIR-(Band) in Segmentation Approaches
Multimedia Satellite Task 2017 - Overview
12
Input: PlanetLab Images Output: Segmentation masks

13. DFKI – KM - DLCC
MMSat 2018 - Multimedia Satellite Task 2018

14. DFKI – SDS
14
MMSat 2018 – Road Passability during Flooding Events
14
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability

15. DFKI – SDS
15
MMSat 2018 – Road Passability during Flooding Events
15
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability

16. DFKI – SDS
16
MMSat 2018 – Road Passability during Flooding Events
16
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability

17. DFKI – SDS
17
MMSat 2018 – Road Passability during Flooding Events
17
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability

18. DFKI – SDS
18
MMSat 2018 – Road Passability during Flooding Events
18
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability

19. DFKI – SDS
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability
19
MMSat 2018 – Road Passability during Flooding Events
19

20. DFKI – SDS
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability
20
MMSat 2018 – Road Passability during Flooding Events
20
Flood Detection in Satellite Imagery

21. DFKI – SDS
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability
21
MMSat 2018 – Road Passability during Flooding Events
21
Flood Detection in Satellite Imagery
Not Passable /
Blocked Road

22. DFKI – SDS
Flood Classification from Social Media
• Classification for Evidence of Road Passability
• Classification for Road Passability
22
MMSat 2018 – Road Passability during Flooding Events
22
Flood Detection in Satellite Imagery
Not Passable /
Blocked Road
Passable
Road

23. DFKI – SDS
23
Dataset Overview - Hurricane events in 2017
23
Three Hurricane events:
• Harvey
• Irma
• Maria

24. DFKI – SDS
Task Dataset – Twitter and Satellite Imagery
24
Flood Classification from Social Multimedia
• Collected >100.000 Tweets for the three Hurricane events
• 7.512 Tweet-Ids with accompanying images
• Metadata and traditional visual features
• Development/Test-set split with 80:20 ratio (location wise sampling)
• Slightly imblanced (36/64 for evidence and 45/55 for passability)
• Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim

25. DFKI – SDS
25
Task Dataset – Twitter and Satellite Imagery
25
Flood Detection in Satellite Imagery
• Pre-processed and high resolution satellite imagery
from DgitialGlobe (30 cm pixel resolution)
• 1700 Image-Patches with two coordinates and a label
for the passability of roads
Flood Classification from Social Multimedia
• Collected >100.000 Tweets for the three Hurricane events
• 7.512 Tweet-Ids with accompanying images
• Metadata and traditional visual features
• Development/Test-set split with 80:20 ratio (location wise sampling)
• Slightly imblanced (36/64 for evidence and 45/55 for passability)
• Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim

26. DFKI – SDS
Run Submissions and Eval-Metrics
26
Evidence
No
Evidence
Passable C1 C2
Non
Passable
C3 C4
Flood Classification from Social Multimedia
• Run Submissions:
• One required run with the provided dev set
(visual modality only), four general runs
• Evaluation Metric:
• 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)

27. DFKI – SDS
27
Run Submissions and Eval-Metrics
27
Evidence
No
Evidence
Passable C1 C2
Non
Passable
C3 C4
Flood Detection in Satellite Imagery
• Run Submissions:
• Two required runs with the provided satellite
imagery of the dev. set only, three general runs
• Evaluation Metrics:
• 𝑆𝑐𝑜𝑟𝑒 = 𝐹1 𝑛𝑜𝑛 𝑝𝑎𝑠𝑠𝑎𝑏𝑙𝑒
Flood Classification from Social Multimedia
• Run Submissions:
• One required run with the provided dev set
(visual modality only), four general runs
• Evaluation Metric:
• 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)

28. DFKI – SDS
Ground Truth Annotations
28
Flood Classification from Social Multimedia
• Crowd-sourcing experiment on FigureEight
• Questions about the evidence, road passability,
surrounding context
• 38.127 judgements from 1.513 different persons
• At least three annotations per image

29. DFKI – SDS
29
Ground Truth Annotations
29
Flood Detection in Satellite Imagery
• We built an interactive web based tool to
annotate satellite images
• Manually selected and annotated patches by
> 20 Stundents & PhDs
Flood Classification from Social Multimedia
• Crowd-sourcing experiment on FigureEight
• Questions about the evidence, road passability,
surrounding context
• 38.127 judgements from 1.513 different persons
• At least three annotations per image

30. DFKI – SDS
30
Task Participation & Approaches
30
Task Participation
• 18 teams registered, 9 submitted runs
• 9 teams submitted for the first subtask
• 4 teams submitted for the second subtask
• In total 51 submission runs
Approaches of Participants
• Features:
• Visual Features (CNN Features, local features, CV features)
• Metadata (word embeddings of text, tags)
• Classifiers:
• Convolutional Neural Networks, LSTMs
• Ensemble Models, SVMs
• Spectral Regression based Kernel Discriminant Analysis
MediaEval Workshop 2017

31. DFKI – SDS
31
Results on Social Media Dataset – Average F1-Score
31
Visual Metadata
Visual +
Metadata
Open run Open run
B-CVC 24.09 19.30 28.34 67.13 68.38
CERTH-ITI 66.65 30.17 66.43 55.12 54.48
ParanoMundo 64.81 - 60.92 62.93 62.91
UTAOS 65.03 - - 60.59 63.58
NUCES-KHI 45.04 31.15 45.56 - -
RU-iCIS 63.13 12.86 - 63.89 63.88
EVUS-ikg 64.35 32.81 59.49 52.16 51.59
MC-FHSTP 20.39 23.88 - 17.24 35.39
DFKI 65.21 - - 66.48 64.96

32. DFKI – SDS
32
Results on Social Media Dataset – Average F1-Score
32
Required Required Open run Open run Open run
ParanoMundo 71.71 56.81 68.62 62.68 73.26
CERTH-ITI 56.45 - - - -
UTAOS 62.29 61.01 - - -
MC-FHSTP 56.80 32.39 38.92 55.62 57.30

33. DFKI – SDS
33
Insights and Conclusion
33
• Road Passability from Social Media is a challenging problem (Subjectivity)
• Satellite Image Analysis without the segmentation of Road Network
• Metadata/Text modality is not helpful for road passability classification
• Multimodal fusion often worse than visual modality only
• CNN-Features of ImageNet pre-trained models are not good for this task
• Places, Visual Sentiment Ontology
• Local Features can be leveraged for road passability classification
• No or no big improvement compared to Image level CNN-Features
• Future analysis with participants (sensitvity analsysis and agreement
with annotator reasons for the choices)
• Continue on Flooding Events next year with more detailed analysis
(Water level, Water stream flow, Impact)

34. DFKI – SDS34
Acknowledgements
Thank you for your attention!
Don‘t hesitate to contact me: Benjamin.bischke@dfki.de