Visual Information Analysis for Crisis and Natural Disasters Management and Response

1. 1Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Visual Information Analysis for Crisis and Natural Disasters Management and Response Ioannis (Yiannis) Kompatsiaris Information Technologies Institute Centre for Research and Technology Hellas Researcher Director at CERTH-ITI Head of Multimedia Knowledge and Social Media Analytics Laboratory and Deputy Director of the Institute International Conference on Image Processing Theory, Tools and Applications IPTA 2019 November 6-9, 2019 Istanbul, Turkey

2. 2Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Centre for Research and Technology Hellas (CERTH) • Largest research centre in Northern Greece • Ministry of Education, R&D Secretariat • 600 people • five institutes • ICT, Chemical Engineering, Transport, Health and Bio-technology, Agriculture • www.certh.gr • ITI: Information • Technologies Institute • > 350 people • 6 labs in 2 Units • www.iti.gr 1st in Greece in H2020 projects In the list of TOP-10 E.U. institutions in competitive funding

3. 3Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Multimedia Knowledge and Social Media Analytics Lab (MKLab) Research areas • Computer Vision • Semantic technologies • Content Indexing and Search Engines • Social media and big data analytics • IoT (sensors interfaces, fusion, event detection) • Novel interfaces: Brain Computer Interfaces, VR/AR Applications: • Media, eHealth, Culture, Smart Cities, Environmental, Physical & Digital Security applications Personnel • 2 senior researchers, 26 post-doc res., 35 res. assistants, 3 PhD candidates Projects and Publications: • 19 Η2020 projects, 21 active projects, co-ordinating 6 • Journals: 140, Patents: 10, Proceedings - Books: 12, Book Chapters: 47, International Conferences: 476

4. 4Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Security & Safety Applications (IoT) • Emergency response • Natural and man-made disasters • Terrorism and cybercrime • Detection, prediction, prevention, and investigation services • Border security • Enhancing workers’ health, wellness, and productivity • Reduce accidents and chronic diseases • Secure and trusted networked environment

5. 5Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Natural Disasters and Visual Content • Natural disaster management in the previous decades used to involve mainly phone calls, physical maps and other methods to respond during a crisis event • Nowadays, ubiquitous visual content that becomes available to civil protection agencies and authorities from: • Social media • Traffic and Cameras mounted on Unamnned Aerial Vehicles (UAV) • Satellite images from the Copernicus EU programme • These visual data sources are very large and highly heterogeneous • Analysis of all the available data sources is needed to assist in efficient management of the event before, during and after its occurrence • Additional modalities provide richer information but require multi-modal approaches

6. 6Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Social Media

7. 7Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/7 http://blog.tyronesystems.com/how-much-data-is-created-every-minute-by-the-social-media

8. 8Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Social Media as Real-Life Sensors • Social Networks is a big data source with an extremely dynamic nature that reflects events and user’s interests • Huge smartphones and mobile devices penetration provides real-time and location-based user feedback • Transform individually rare but collectively frequent media to meaningful topics, events, points of interest, emotional states and social connections • Present in an efficient way for a variety of applications (natural disaster, news, marketing, science, health, entertainment)

9. 9Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/9 “…if you're more than 100 km away from the epicenter [of an earthquake] you can read about the quake on twitter before it hits you…”

10. 10Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Social Media Data in Natural Disasters • Use of social media on natural disasters management: • Pre-crisis: monitoring, situation awareness and early warning • During crisis: providing real-time information about incidents • Post-crisis: identifying where stress management is most needed • Challenges faced when crawling from Twitter: • The public Streaming API provides 1% of the current posting volume • Less than 2% of tweets are geo-located • Retrieved information is ambiguous and not always relevant (e.g. flooding à flooding my timeline) • Misleading information (aka “Fake News”)

11. 11Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Crawling Social Media Data • Focusing on Twitter posts, collected with Twitter Streaming API https://developer.twitter.com/en/docs/tweets/filter-realtime/overview • Various analysis techniques to obtain further knowledge on the tweets • The complete flow: new tweet Search terms: • Keywords • Accounts • Bounding Boxes Keys & Tokens Twitter Streaming API Client receives tweets Fake tweets detection Text classification Image classification Get tweet in JSON format & find matching use case Nudity detection Tweets localisation Concept extraction tweet has image tweet has no image Inputs:

12. 12Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Crawling Social Media Data • Some of our collections of tweets that concern natural disasters in different countries, posted since June 2017 10 m. about fires in Spain 75 k. about floods in Italy 74 k. about heatwave in Greece 42 k. about snow in Finland

13. 13Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Fake Tweets Classification Model Boididou, C., Papadopoulos, S., Zampoglou, M., Apostolidis, L., Papadopoulou, O., & Kompatsiaris, Y. (2018). Detection and visualization of misleading content on Twitter. International Journal of Multimedia Information Retrieval, 7(1), 71-86. Fake tweets detection

14. 14Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Credibility signals (aka features)

15. 15Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Relevancy estimation of Social Media Text • Preprocessing: • removing stop words/punctuation/hyperlinks • stemming • mapping to concepts • Text representation to vectors • Text classification • Word2vec representation: comprises two-layer neural networks trained to reconstruct linguistic contexts of words and produce eventually word embeddings • Similar words and meanings become “close” (e.g. flooding, disaster, emergency,..) Moumtzidou, A., Andreadis, S., Gialampoukidis, I., Karakostas, A., Vrochidis, S. and Kompatsiaris, I., 2018, April. Flood relevance estimation from visual and textual content in social media streams. In Companion Proceedings of the The Web Conference 2018 (pp. 1621-1627). International World Wide Web Conferences Steering Committee. Text classification

16. 16Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Relevancy estimation of Social Media Text - Experiments • Parameters: • Vector dimension = number of features {100, 200, 300, 400, 500} • Words windows = max. distance between current and predicted word within sentence • Training algorithm = skip-gram (0) or CBOW (1) • Corpora: • mediaEvalFloods_corpus (MediaEval 2017 dataset: https://multimediaeval.github.io/2017-Multimedia-Satellite-Task/) • twitterFloods_corpus (our collection of tweets in English about floods)

17. 17Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Concept Detection in Social Media Images • Extracts high-level concepts from visual low-level information • Fine-tune pre-trained 22-layer GoogleNet DCNN network to recognize the 345 TRECVID INS concepts and thresholding to keep concepts with higher probability • Concept examples: animal, boat_ship, clouds, waterscape_waterfront

18. 18Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Relevancy estimation of Social Media Images • Classification: • use of output of last pooling layer with dimension 1024 for global image representation • SVM classifier per concept • use annotated dataset for training and validating of classifier • tuning of SVM classifier for different t (kernel type) and g (gamma in the kernel function) to achieve maximum performance Moumtzidou, A., Andreadis, S., Gialampoukidis, I., Karakostas, A., Vrochidis, S. and Kompatsiaris, I., 2018, April. Flood relevance estimation from visual and textual content in social media streams. In Companion Proceedings of the The Web Conference 2018 (pp. 1621-1627). International World Wide Web Conferences Steering Committee. Image classification

19. 19Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Relevancy estimation of Social Media Images - Experiments • Evaluation of several visual descriptors and DCNN-based features • Tuning of SVM classifier for different t kernel function types (i.e. linear, polynomial, radial and sigmoid) and g gamma parameter in kernel function • Dataset from MediaEval 2017 (DIRSM task) https://multimediaeval.github .io/2017-Multimedia-Satellite- Task/

20. 20Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ CERTH-ITI participation in MediaEval 2018 First in the social media image classification (Average F1-score) https://www.youtube.com/watch?v=yq1nIPc6dWw&list=PLOPRp1vN OG9ahE5viJmF6Gx8XDk8hG9MP&index=2&t=0s

21. 21Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Similarity Fusion from Multiple Sources • Fusion at features level • Social media posts indexing: • Textual representation using word2vec • Visual features using DCNN-based feature • Visual concepts

22. 22Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Clustering of Social Media Text • Grouping tweets according to the relevancy of their text • Density-based clustering with LDA topic model • discovering the abstract “topics” that occur in a collection of documents • Most frequent words of a textual cluster are visualized as word clouds • Each word cloud comprises the tweets that are grouped into that cluster I. Gialampoukidis, S. Vrochidis, I. Kompatsiaris, A Hybrid framework for news clustering based on the DBSCAN-Martingale and LDA. In proceedings of the 12th international conference on Machine Learning and Data Mining (MLDM2016), New York, July 16-21, 2016. pp. 170-184, Springer International Publishing.

23. 23Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Community detection in Social Media Accounts • Detect and visualize user communities through their interaction • Create a network of social media accounts that are linked when one mentions the other • Modularity maximization (Louvain) community detection • Find the key-players in these communities which are considered “authorities”

24. 24Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Estimation of the location mentioned in a tweet • Long short-term memory-based (LSTM) named-entity recognition (NER) • Semantically linked named entities to external Knowledge Bases (e.g. OpenStreetMaps) English How quick-thinking mother saved family from Grenfell fire by flooding her flat Emergency declared in #Paraguay after flooding from torrential rains. https://t.co/Z........ Italian Presentazione il sistema di #allertameteo della #ProtezioneCivile della città di #Gorizia Ponte Milvio fa acqua: ancora un allagamento in via Prati della Farnesina... #news #Roma Finnish Lumi riittää jo meidän pihaan! #Joensuu #lumi #sää

25. 25Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Estimation of the location mentioned in a tweet • Results of the NER task for English Dataset (CoNLL2003) Precision Recall F1-score Our system (ELMo embeddings) 91.63 93.01 92.32 Best-scoring CoNLL2003 system: Florian et al., 2003 88.99 88.54 88.76 Baevski, A. et al. 2019 (not reported) (not reported) 93.5 Localisation steps after NER has been performed on available tweets: Dataset (EVALITA2009) Precision Recall F1-score Our system (GloVe embeddings) 75.49 75.60 75.37 Best-scoring shared task system: FBK_ZanoliPianta 84.07 80.02 82.00 Nguyen and Moschitti, 2012 85.99 82.73 84.33 • Finnish NER still work in progress: 1. Dataset enhancement 2. Embeddings optimisation • Results of the NER task for Italian

26. 26Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Position of tweets with estimated location and visual content on a GIS view • Two-way navigation • From the Twitter streams to the map • From the map to the Twitter streams

27. 27Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Cameras on UAVs

28. 28Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Identify objects of interest from aerial images • Refers to the visual identification of specific objects • Architectures of Deep Convolutional Neural Networks (DCNN) • Bounding Box: R-CNN, Fast R-CNN, Faster R-CNN, YOLO algorithms • Semantic segmentation: U-Net, Feature Pyramid Network (FPN), Pyramid Scene Parsing Network (PSPNet), Mask R-CNN, DeepLab, Path Aggregation Network (PANet) • Datasets: 2012 PASCAL VOC, PASCAL-Context, DOTA, senseFly, Cityscapes, DJI Mavic Pro Footage, VisDrone2018

29. 29Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Tracking objects of interest • Vision based autonomous control • Affects the navigation of the UAV • Approaches • Hierarchical Particle Filter (HPF), Kernelised Correlation Filter (KCF), Multi-kernel Correlation Filter, Multi-Domain Convolutional Neural Networks • Single object tracking: VOT2016 • Multi-object tracking: MOTChallenge2015, UA-DETRAC

30. 30Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Combination of detection with tracking • Tracker can (follow) track objects of interest. • Fast and efficient but have trouble in difficult situations to keep track of object • Combination of object detector and tracker delivers merits from both worlds: • Efficiency (tracker) • Effectiveness (object detector) • Object detection every N frames and apply tracking in between. Object detection results Class Average precision Class Average precision UAV 0.75330 Boat 0.70251 Car 0.75726 Helicopter - Plane 0.71638 Person 0.82152 Motorcycle- Bicycle 0.73409 Truck 0.53351 Weapon 0.54025 Bus 0.57315 Military vehicle 0.68435 Ship 0.83586 mAP 0.69565

31. 31Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Satellite images

32. 32Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Water body detection in satellite images • Detect water bodies in flooded areas: • Train a Deep Neural Network model • Use Sentinel-1 VV and VH bands • Use Elevation information (DEM) • Increased performance: • Minimizes holes (false negatives) in extended water surfaces • Filters out steep areas (false positives)

33. 33Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Water body detection in satellite images • Overall best results obtained by Adam optimizer and 0.001 learning rate Conclusions: • Accuracy improves for all cases comparing the Deep neural network model to the histogram thresholding method • Maximum F-score of 94.10% for Garda lake with the DNN model Lakes Precision Recall Accuracy F-score Settings Maggiore 98.29 87.84 93.15 92.77 Adam, 0.001 Maggiore 96.23 61.89 79.73 75.33 -22.0 dB (vh) Garda 94.57 93.63 94.13 94.10 Adam, 0.001 Garda 95.55 70.45 83.58 81.10 -21.7 dB (vh) Trasimeno 93.67 84.61 89.45 88.91 Adam, 0.001 Trasimeno 88.07 66.13 78.58 75.54 -13.9 dB (vv)

34. 34Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Change Detection in Satellite Image Time Series • Detects flood incidents within a time-series of optical satellite imagery • Two approaches: • Remote sensing (Image Processing) • Deep Convolutional Neural Network (DCNN) • Evaluated in Multimedia Task of the MediaEval 2019 • Test set: 68 timeseries • Location: African cities

35. 35Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Change Detection in Satellite Image Time Series Remote sensing: Outlier detection on image differencing of MNDWI indices. • Input: Time-series of Green and Swir bands of the events • Method: • Calculate the MNDWI index for all days: MNDWI = • Compute the differences of consecutive days • Detect outliers in the differences images à Generating change masks • Output: Change is determined by considering the sum of outlier pixels • if sum of outlier pixels > 5% of total pixels à flood • Results: MNDWI, γ =2.1, ratio=0.05: 76.47% F-Score SWIRGreen SWIRGreen + -

36. 36Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Change Detection in Satellite Image Time Series • Deep Convolutional Neural Network (DCNN) • Input: Time-series of R-G-B or R-Swir-Nir bands of the events • Method: • Create JPEG of the differences of all days • Fine-tune a VGG-16 model using training set of 367 events, and by considering all combinations of the differences of RGB images • Output: Predicted change in consecutive days denotes a flood event • Results: Red-Green-Blue: 70.58% F-Score

37. 37Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Road passability in satellite images Flood detection in satellite images*: • Given two coordinate points on a satellite image the task is to select the corresponding label (passable/non passable). • Participants receive data and are required to train classifiers. • Fusion of satellite and social multimedia information is encouraged. • The task moves forward the state of the art by concentrating on passibility, whether or not it is possible for a vehicle to pass a road. *http://www.multimediaeval.org/mediaeval2018/multimediasatellite/index.html A. Moumtzidou, P. Giannakeris, S. Andreadis, A. Mavropoulos, G. Meditskos, I. Gialampoukidis, K. Avgerinakis, S. Vrochidis and I. Kompatsiaris, “A multimodal approach in estimating road passability through a flooded area using social media and satellite images”, @ multimedia satellite task MediaEval 2018. In Proceedings of the Working Notes Proceeding MediaEval Workshop, Sophia Antipolis, France, 29-31 October, 2018.

38. 38Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Road passability in satellite images • Infers if a road is passable or non-passable due to water • Deep learning network for determining road passability (i.e Fine-tuned pre-trained on ImageNet VGG-19) • Splits initial image in segments and predicts the passability for each tile (flooded/non-flooded) • Fusion with social media images where the concept “vehicle” has been extracted from items containing “flood” concept

39. 39Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Road passability in satellite images • Approach • Fine-tuning of pre-trained DCNN networks (e.g. ResNet101, VGG-16, Inception, Inception_ResNet_v2) (published in “Big Data from Space Conference 2019”) • Settings: • learning rate values = 0.001, 0.01, 0.1 • batch size values = 32, 64, 128, 256 • optimizer functions = Adam, Stochastic Gradient Descent (SGD)

40. 40Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Road passability in satellite images • Conclusions • Accuracy improves for the lower values of the learning rate • Increase of the batch size generally improves the accuracy Batch size 32 Batch size 64 Batch size 128 DCNN Learning rate Optimizer Validation Set Accuracy Validation Set Accuracy Validation Set Accuracy VGG-19 0,01 Adam 0,7666 0,7666 0,7666 VGG-19 0,001 SGD 0,7071 0,7117 0,7162 Inception_v3 0,01 Adam 0,6247 0,5789 0,5629 Inception_v3 0,001 SGD 0,5950 0,6224 0,5973 VGG-16 0,001 Adam 0,7437 0,7574 0,7551 VGG-16 0,001 SGD 0,7277 0,7208 0,7231 ResNet101 0,1 Adam 0,5492 0,5126 0,5126 ResNet101 0,001 SGD 0,5835 0,5995 0,5881 Inception_ResNet_V2 0,001 Adam 0,6384 0,6178 0,6156 Inception_ResNet_V2 0,01 SGD 0,7002 0,6979 0,6819

41. 41Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ EOPEN project concept

42. Enhancing decision support and management services in extreme weather climate events • Collection of heterogeneous data from several resources such as environmental, social media, input from first responders and/or people in danger • Semantically integration of data to provide decision support services to the crisis management center

43. 43Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Conclusions • Existing visual information processing approaches can contribute to security domain • With adaptations: features, re-training • Not always: lack of data, specific applications • Additional modalities can be very useful • Text, sensor information • Fusion approaches • Need for multi-modal Deep Learning approaches to detect events from joint combinations of text, visual, sound and machine-generated information for multi-faceted multi-variable event detection • Challenges include: overall architecture, big data management, visualizations and interfaces, user acceptance, privacy and ethics

44. 44Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Contributors Stefanos Vrochidis, Senior Researcher – stefanos@iti.gr Ilias Gialampoukidis, Postdoctoral Researcher – heliasgj@iti.gr Konstantinos Ioannidis, Postdoctoral Researcher - kioannid@iti.gr Anastasia Moumtzidou, Research Associate - moumtzid@iti.gr Stelios Andreadis, Research Associate - andreadisst@iti.gr

45. 45Multimedia Knowledge & Social Media Analytics Laboratory http://mklab.iti.gr/ Thank you! Email: ikom@iti.gr Lab: mklab.iti.gr Supported by the projects EOPEN (H2020-776019) and beAWARE (H2020-700475) funded by the European Commission

Visual Information Analysis for Crisis and Natural Disasters Management and Response

You are reading a preview.

Check these out next

Visual Information Analysis for Crisis and Natural Disasters Management and Response

More Related Content

Slideshows for you (7)

Similar to Visual Information Analysis for Crisis and Natural Disasters Management and Response (20)

More from Yiannis Kompatsiaris (16)

Recently uploaded (20)