Recommended
More Related Content
Similar to MediaEval 2018: Multimedia Satellite Task: Emergency Response for Flooding Events
Similar to MediaEval 2018: Multimedia Satellite Task: Emergency Response for Flooding Events (20)
More from multimediaeval
More from multimediaeval (20)
Recently uploaded
Recently uploaded (20)
MediaEval 2018: Multimedia Satellite Task: Emergency Response for Flooding Events
- 1. DFKI – KM - DLCC Multimedia Satellite Task 2018 ALL RIGHTS RESERVED. No part of this work may be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without expressed written permission from the authors. MediaEval Benchmark 2018 Benjamin Bischke, Patrick Helber, Zhengyu Zhao, Jens de Brujin, Damian Borth Emergency Response for Flooding Events Sophia Antipolis, 30.10.2018
- 2. DFKI – SDS 2 Natural Disasters - Emergency Response 2 • 2016 & 2017 were one of the most impact-full hurricane seasons • Hurricane Matthew, Harvey, Irma and Maria caused a high economic and influence on private life • The number of natural disasters is increasing
- 3. DFKI – SDS 3 Current Natural Disasters (October 2018) 3
- 4. DFKI – SDS 4 Current Natural Disasters (October 2018) 4 Floods Hurricane Floods
- 5. DFKI – SDS 5 Flooding Events 5
- 6. DFKI – SDS 6 6 Natural Disasters – Rappid Mapping • Creating maps for emergency response are semi-automatic • Fast Access to information is prioritized over accurracy
- 7. DFKI – SDS7 Is Satellite Imagery enough? Washington Post
- 8. DFKI – SDS Is Satellite Imagery enough? (Idea) 8 Washington Post
- 9. DFKI – KM - DLCC Multimedia Satellite Task 2017
- 10. DFKI – SDS • Multimedia Satellite Task 2017 with two subtasks: – Disaster Image Retrieval from Social Media (DIRSM) – Flood Detection in Satellite Imagery (FDSI) • 11 Teams actively participated from all the world (Brasil, Australia, Greece, Brunei, Italy, UK, Germany, Netherlands, Norway…) – More than 60 submission on two subtasks DigitalGlobe, October 2017 Emergency Response MediaEval Workshop 2017 Multimedia Satellite Task 2017 - Overview 10
- 11. DFKI – SDS Multimedia Satellite Task 2017 - Overview 11 Disaster Image Retrieval from Social Media (DIRSM) • Based on more than 6000 Flickr Images • Fusion of multimodal information very important (visual & text) • Features of CNNs trained on Places and VSO achieved high results
- 12. DFKI – SDS Disaster Image Retrieval from Social Media (DIRSM) • Based on more than 6000 Flickr Images • Fusion of multimodal information very important (visual & text) • Features of CNNs trained on Places and VSO achieved high results Flood Detection in Satellite Imagery (FDSI) • Based on Planet Images for 7 events (4 meter spatial resolution) • Incorporation of NIR-(Band) in Segmentation Approaches Multimedia Satellite Task 2017 - Overview 12 Input: PlanetLab Images Output: Segmentation masks
- 13. DFKI – KM - DLCC MMSat 2018 - Multimedia Satellite Task 2018
- 14. DFKI – SDS 14 MMSat 2018 – Road Passability during Flooding Events 14 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
- 15. DFKI – SDS 15 MMSat 2018 – Road Passability during Flooding Events 15 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
- 16. DFKI – SDS 16 MMSat 2018 – Road Passability during Flooding Events 16 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
- 17. DFKI – SDS 17 MMSat 2018 – Road Passability during Flooding Events 17 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
- 18. DFKI – SDS 18 MMSat 2018 – Road Passability during Flooding Events 18 Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability
- 19. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 19 MMSat 2018 – Road Passability during Flooding Events 19
- 20. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 20 MMSat 2018 – Road Passability during Flooding Events 20 Flood Detection in Satellite Imagery
- 21. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 21 MMSat 2018 – Road Passability during Flooding Events 21 Flood Detection in Satellite Imagery Not Passable / Blocked Road
- 22. DFKI – SDS Flood Classification from Social Media • Classification for Evidence of Road Passability • Classification for Road Passability 22 MMSat 2018 – Road Passability during Flooding Events 22 Flood Detection in Satellite Imagery Not Passable / Blocked Road Passable Road
- 23. DFKI – SDS 23 Dataset Overview - Hurricane events in 2017 23 Three Hurricane events: • Harvey • Irma • Maria
- 24. DFKI – SDS Task Dataset – Twitter and Satellite Imagery 24 Flood Classification from Social Multimedia • Collected >100.000 Tweets for the three Hurricane events • 7.512 Tweet-Ids with accompanying images • Metadata and traditional visual features • Development/Test-set split with 80:20 ratio (location wise sampling) • Slightly imblanced (36/64 for evidence and 45/55 for passability) • Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim
- 25. DFKI – SDS 25 Task Dataset – Twitter and Satellite Imagery 25 Flood Detection in Satellite Imagery • Pre-processed and high resolution satellite imagery from DgitialGlobe (30 cm pixel resolution) • 1700 Image-Patches with two coordinates and a label for the passability of roads Flood Classification from Social Multimedia • Collected >100.000 Tweets for the three Hurricane events • 7.512 Tweet-Ids with accompanying images • Metadata and traditional visual features • Development/Test-set split with 80:20 ratio (location wise sampling) • Slightly imblanced (36/64 for evidence and 45/55 for passability) • Duplicate Filtering: pHash + CNN-Features (ResNet101) with Cos. Sim
- 26. DFKI – SDS Run Submissions and Eval-Metrics 26 Evidence No Evidence Passable C1 C2 Non Passable C3 C4 Flood Classification from Social Multimedia • Run Submissions: • One required run with the provided dev set (visual modality only), four general runs • Evaluation Metric: • 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)
- 27. DFKI – SDS 27 Run Submissions and Eval-Metrics 27 Evidence No Evidence Passable C1 C2 Non Passable C3 C4 Flood Detection in Satellite Imagery • Run Submissions: • Two required runs with the provided satellite imagery of the dev. set only, three general runs • Evaluation Metrics: • 𝑆𝑐𝑜𝑟𝑒 = 𝐹1 𝑛𝑜𝑛 𝑝𝑎𝑠𝑠𝑎𝑏𝑙𝑒 Flood Classification from Social Multimedia • Run Submissions: • One required run with the provided dev set (visual modality only), four general runs • Evaluation Metric: • 𝑆𝑐𝑜𝑟𝑒 = 0.5 ∗ 𝐹1 𝐶1 + 0.5 ∗ 𝐹1(𝐶3)
- 28. DFKI – SDS Ground Truth Annotations 28 Flood Classification from Social Multimedia • Crowd-sourcing experiment on FigureEight • Questions about the evidence, road passability, surrounding context • 38.127 judgements from 1.513 different persons • At least three annotations per image
- 29. DFKI – SDS 29 Ground Truth Annotations 29 Flood Detection in Satellite Imagery • We built an interactive web based tool to annotate satellite images • Manually selected and annotated patches by > 20 Stundents & PhDs Flood Classification from Social Multimedia • Crowd-sourcing experiment on FigureEight • Questions about the evidence, road passability, surrounding context • 38.127 judgements from 1.513 different persons • At least three annotations per image
- 30. DFKI – SDS 30 Task Participation & Approaches 30 Task Participation • 18 teams registered, 9 submitted runs • 9 teams submitted for the first subtask • 4 teams submitted for the second subtask • In total 51 submission runs Approaches of Participants • Features: • Visual Features (CNN Features, local features, CV features) • Metadata (word embeddings of text, tags) • Classifiers: • Convolutional Neural Networks, LSTMs • Ensemble Models, SVMs • Spectral Regression based Kernel Discriminant Analysis MediaEval Workshop 2017
- 31. DFKI – SDS 31 Results on Social Media Dataset – Average F1-Score 31 Visual Metadata Visual + Metadata Open run Open run B-CVC 24.09 19.30 28.34 67.13 68.38 CERTH-ITI 66.65 30.17 66.43 55.12 54.48 ParanoMundo 64.81 - 60.92 62.93 62.91 UTAOS 65.03 - - 60.59 63.58 NUCES-KHI 45.04 31.15 45.56 - - RU-iCIS 63.13 12.86 - 63.89 63.88 EVUS-ikg 64.35 32.81 59.49 52.16 51.59 MC-FHSTP 20.39 23.88 - 17.24 35.39 DFKI 65.21 - - 66.48 64.96
- 32. DFKI – SDS 32 Results on Social Media Dataset – Average F1-Score 32 Required Required Open run Open run Open run ParanoMundo 71.71 56.81 68.62 62.68 73.26 CERTH-ITI 56.45 - - - - UTAOS 62.29 61.01 - - - MC-FHSTP 56.80 32.39 38.92 55.62 57.30
- 33. DFKI – SDS 33 Insights and Conclusion 33 • Road Passability from Social Media is a challenging problem (Subjectivity) • Satellite Image Analysis without the segmentation of Road Network • Metadata/Text modality is not helpful for road passability classification • Multimodal fusion often worse than visual modality only • CNN-Features of ImageNet pre-trained models are not good for this task • Places, Visual Sentiment Ontology • Local Features can be leveraged for road passability classification • No or no big improvement compared to Image level CNN-Features • Future analysis with participants (sensitvity analsysis and agreement with annotator reasons for the choices) • Continue on Flooding Events next year with more detailed analysis (Water level, Water stream flow, Impact)
- 34. DFKI – SDS34 Acknowledgements Thank you for your attention! Don‘t hesitate to contact me: Benjamin.bischke@dfki.de