Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MediaEval 2018: Multimedia Satellite Task: Emergency Response for Flooding Events

48 views

Published on

Paper: http://ceur-ws.org/Vol-2283/MediaEval_18_paper_10.pdf
Youtube: https://youtu.be/yq1nIPc6dWw

Benjamin Bischke, Patrick Helber, Zhengyu Zhao, Jens de Bruijn, Damian Borth, The Multimedia Satellite Task at MediaEval 2018. Proc. of MediaEval 2018, 29-31 October 2018, Sophia Antipolis, France.

Abstract: This paper provides a description of the MediaEval 2018 Multimedia Satellite Task. The primary goal of the task is to extract and fuse content of events which are present in Satellite Imagery and Social Media. Establishing a link from Satellite Imagery to Social Multimedia can yield to a comprehensive event representation which is vital for numerous applications. Focusing on natural disaster events, the main objective of the task is to leverage the combined event representation within the context of emergency response and environmental monitoring. In particular, our task focuses on flooding events and consists of two subtasks. The first Image Classification from Social Media subtask requires participants to retrieve images from Social Media which show a direct evidence for road passability during flooding events. The second task Flood Detection from Satellite Images aims to extract potentially flooded road sections from satellite images. The task seeks to go beyond state-of-the-art flooding map generation by focusing on information about the road passability and accessibility to urban infrastructure. Such information shows a clear potential to complement information from social images with satellite imagery is of vital importance for emergency management.

Presented by Benjamin Bischke

Published in: Science
  • Be the first to comment

  • Be the first to like this

MediaEval 2018: Multimedia Satellite Task: Emergency Response for Flooding Events

  1. 1. DFKI – KM - DLCC Multimedia Satellite Task 2018 ALL RIGHTS RESERVED. No part of this work may be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without expressed written permission from the authors. MediaEval Benchmark 2018 Benjamin Bischke, Patrick Helber, Zhengyu Zhao, Jens de Brujin, Damian Borth Emergency Response for Flooding Events Sophia Antipolis, 30.10.2018
  2. 2. DFKI – SDS 2 Natural Disasters - Emergency Response 2 • 2016 & 2017 were one of the most impact-full hurricane seasons • Hurricane Matthew, Harvey, Irma and Maria caused a high economic and influence on private life • The number of natural disasters is increasing
  3. 3. DFKI – SDS 3 Current Natural Disasters (October 2018) 3
  4. 4. DFKI – SDS 4 Current Natural Disasters (October 2018) 4 Floods Hurricane Floods
  5. 5. DFKI – SDS 5 Flooding Events 5
  6. 6. DFKI – SDS 6 6 Natural Disasters – Rappid Mapping • Creating maps for emergency response are semi-automatic • Fast Access to information is prioritized over accurracy
  7. 7. DFKI – SDS7 Is Satellite Imagery enough? Washington Post
  8. 8. DFKI – SDS Is Satellite Imagery enough? (Idea) 8 Washington Post
  9. 9. DFKI – KM - DLCC Multimedia Satellite Task 2017
  10. 10. DFKI – SDS • Multimedia Satellite Task 2017 with two subtasks: – Disaster Image Retrieval from Social Media (DIRSM) – Flood Detection in Satellite Imagery (FDSI) • 11 Teams actively participated from all the world (Brasil, Australia, Greece, Brunei, Italy, UK, Germany, Netherlands, Norway…) – More than 60 submission on two subtasks DigitalGlobe, October 2017 Emergency Response MediaEval Workshop 2017 Multimedia Satellite Task 2017 - Overview 10
  11. 11. DFKI – SDS Multimedia Satellite Task 2017 - Overview 11 Disaster Image Retrieval from Social Media (DIRSM) • Based on more than 6000 Flickr Images • Fusion of multimodal information very important (visual & text) • Features of CNNs trained on Places and VSO achieved high results
  12. 12. DFKI – SDS Disaster Image Retrieval from Social Media (DIRSM) • Based on more than 6000 Flickr Images • Fusion of multimodal information very important (visual & text) • Features of CNNs trained on Places and VSO achieved high results Flood Detection in Satellite Imagery (FDSI) • Based on Planet Images for 7 events (4 meter spatial resolution) • Incorporation of NIR-(Band) in Segmentation Approaches Multimedia Satellite Task 2017 - Overview 12 Input: PlanetLab Images Output: Segmentation masks
  13. 13. DFKI – KM - DLCC MMSat 2018 - Multimedia Satellite Task 2018
  14. 14. DFKI – SDS 14 MMSat 2018 – Road Passability during Flooding Events 14 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
  15. 15. DFKI – SDS 15 MMSat 2018 – Road Passability during Flooding Events 15 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
  16. 16. DFKI – SDS 16 MMSat 2018 – Road Passability during Flooding Events 16 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
  17. 17. DFKI – SDS 17 MMSat 2018 – Road Passability during Flooding Events 17 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
  18. 18. DFKI – SDS 18 MMSat 2018 – Road Passability during Flooding Events 18 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
  19. 19. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 19 MMSat 2018 – Road Passability during Flooding Events 19
  20. 20. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 20 MMSat 2018 – Road Passability during Flooding Events 20 Flood Detection in Satellite Imagery
  21. 21. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 21 MMSat 2018 – Road Passability during Flooding Events 21 Flood Detection in Satellite Imagery Not Passable / Blocked Road
  22. 22. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 22 MMSat 2018 – Road Passability during Flooding Events 22 Flood Detection in Satellite Imagery Not Passable / Blocked Road Passable Road
  23. 23. DFKI – SDS 23 Dataset Overview - Hurricane events in 2017 23 Three Hurricane events: • Harvey • Irma • Maria
  24. 24. DFKI – SDS Task Dataset – Twitter and Satellite Imagery 24 Flood Classification from Social Multimedia • Collected >100.000 Tweets for the three Hurricane events • 7.512 Tweet-Ids with accompanying images • Metadata and traditional visual features • Development/Test-set split with 80:20 ratio (location wise sampling) • Slightly imblanced (36/64 for evidence and 45/55 for passability) • Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim
  25. 25. DFKI – SDS 25 Task Dataset – Twitter and Satellite Imagery 25 Flood Detection in Satellite Imagery • Pre-processed and high resolution satellite imagery from DgitialGlobe (30 cm pixel resolution) • 1700 Image-Patches with two coordinates and a label for the passability of roads Flood Classification from Social Multimedia • Collected >100.000 Tweets for the three Hurricane events • 7.512 Tweet-Ids with accompanying images • Metadata and traditional visual features • Development/Test-set split with 80:20 ratio (location wise sampling) • Slightly imblanced (36/64 for evidence and 45/55 for passability) • Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim
  26. 26. DFKI – SDS Run Submissions and Eval-Metrics 26 Evidence No Evidence Passable C1 C2 Non Passable C3 C4 Flood Classification from Social Multimedia • Run Submissions: • One required run with the provided dev set (visual modality only), four general runs • Evaluation Metric: • 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)
  27. 27. DFKI – SDS 27 Run Submissions and Eval-Metrics 27 Evidence No Evidence Passable C1 C2 Non Passable C3 C4 Flood Detection in Satellite Imagery • Run Submissions: • Two required runs with the provided satellite imagery of the dev. set only, three general runs • Evaluation Metrics: • 𝑆𝑐𝑜𝑟𝑒 = 𝐹1 𝑛𝑜𝑛 𝑝𝑎𝑠𝑠𝑎𝑏𝑙𝑒 Flood Classification from Social Multimedia • Run Submissions: • One required run with the provided dev set (visual modality only), four general runs • Evaluation Metric: • 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)
  28. 28. DFKI – SDS Ground Truth Annotations 28 Flood Classification from Social Multimedia • Crowd-sourcing experiment on FigureEight • Questions about the evidence, road passability, surrounding context • 38.127 judgements from 1.513 different persons • At least three annotations per image
  29. 29. DFKI – SDS 29 Ground Truth Annotations 29 Flood Detection in Satellite Imagery • We built an interactive web based tool to annotate satellite images • Manually selected and annotated patches by > 20 Stundents & PhDs Flood Classification from Social Multimedia • Crowd-sourcing experiment on FigureEight • Questions about the evidence, road passability, surrounding context • 38.127 judgements from 1.513 different persons • At least three annotations per image
  30. 30. DFKI – SDS 30 Task Participation & Approaches 30 Task Participation • 18 teams registered, 9 submitted runs • 9 teams submitted for the first subtask • 4 teams submitted for the second subtask • In total 51 submission runs Approaches of Participants • Features: • Visual Features (CNN Features, local features, CV features) • Metadata (word embeddings of text, tags) • Classifiers: • Convolutional Neural Networks, LSTMs • Ensemble Models, SVMs • Spectral Regression based Kernel Discriminant Analysis MediaEval Workshop 2017
  31. 31. DFKI – SDS 31 Results on Social Media Dataset – Average F1-Score 31 Visual Metadata Visual + Metadata Open run Open run B-CVC 24.09 19.30 28.34 67.13 68.38 CERTH-ITI 66.65 30.17 66.43 55.12 54.48 ParanoMundo 64.81 - 60.92 62.93 62.91 UTAOS 65.03 - - 60.59 63.58 NUCES-KHI 45.04 31.15 45.56 - - RU-iCIS 63.13 12.86 - 63.89 63.88 EVUS-ikg 64.35 32.81 59.49 52.16 51.59 MC-FHSTP 20.39 23.88 - 17.24 35.39 DFKI 65.21 - - 66.48 64.96
  32. 32. DFKI – SDS 32 Results on Social Media Dataset – Average F1-Score 32 Required Required Open run Open run Open run ParanoMundo 71.71 56.81 68.62 62.68 73.26 CERTH-ITI 56.45 - - - - UTAOS 62.29 61.01 - - - MC-FHSTP 56.80 32.39 38.92 55.62 57.30
  33. 33. DFKI – SDS 33 Insights and Conclusion 33 • Road Passability from Social Media is a challenging problem (Subjectivity) • Satellite Image Analysis without the segmentation of Road Network • Metadata/Text modality is not helpful for road passability classification • Multimodal fusion often worse than visual modality only • CNN-Features of ImageNet pre-trained models are not good for this task • Places, Visual Sentiment Ontology • Local Features can be leveraged for road passability classification • No or no big improvement compared to Image level CNN-Features • Future analysis with participants (sensitvity analsysis and agreement with annotator reasons for the choices) • Continue on Flooding Events next year with more detailed analysis (Water level, Water stream flow, Impact)
  34. 34. DFKI – SDS34 Acknowledgements Thank you for your attention! Don‘t hesitate to contact me: Benjamin.bischke@dfki.de

×