1.
1Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Visual Information Analysis for Crisis and Natural
Disasters Management and Response
Ioannis (Yiannis) Kompatsiaris
Information Technologies Institute
Centre for Research and Technology Hellas
Researcher Director at CERTH-ITI
Head of Multimedia Knowledge and Social Media Analytics Laboratory and Deputy Director of the Institute
International Conference on Image Processing Theory, Tools and Applications IPTA 2019
November 6-9, 2019
Istanbul, Turkey

2.
2Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Centre for Research and Technology Hellas
(CERTH)
• Largest research centre in Northern Greece
• Ministry of Education, R&D Secretariat
• 600 people
• five institutes
• ICT, Chemical Engineering,
Transport, Health and Bio-technology,
Agriculture
• www.certh.gr
• ITI: Information
• Technologies Institute
• > 350 people
• 6 labs in 2 Units
• www.iti.gr
1st in Greece in H2020
projects
In the list of TOP-10 E.U.
institutions in competitive
funding

3.
3Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Multimedia Knowledge and Social Media
Analytics Lab (MKLab)
Research areas
• Computer Vision
• Semantic technologies
• Content Indexing and Search Engines
• Social media and big data analytics
• IoT (sensors interfaces, fusion, event detection)
• Novel interfaces: Brain Computer Interfaces, VR/AR
Applications:
• Media, eHealth, Culture, Smart Cities, Environmental, Physical & Digital
Security applications
Personnel
• 2 senior researchers, 26 post-doc res., 35 res. assistants, 3 PhD candidates
Projects and Publications:
• 19 Η2020 projects, 21 active projects, co-ordinating 6
• Journals: 140, Patents: 10, Proceedings - Books: 12, Book Chapters: 47,
International Conferences: 476

4.
4Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Security & Safety Applications (IoT)
• Emergency response
• Natural and man-made disasters
• Terrorism and cybercrime
• Detection, prediction, prevention, and
investigation services
• Border security
• Enhancing workers’ health, wellness,
and productivity
• Reduce accidents and chronic diseases
• Secure and trusted networked
environment

5.
5Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Natural Disasters and Visual Content
• Natural disaster management in the previous decades used to involve mainly
phone calls, physical maps and other methods to respond during a crisis event
• Nowadays, ubiquitous visual content that becomes available to civil protection
agencies and authorities from:
• Social media
• Traffic and Cameras mounted on Unamnned Aerial Vehicles (UAV)
• Satellite images from the Copernicus EU programme
• These visual data sources are very large and highly heterogeneous
• Analysis of all the available data sources is needed to assist in efficient
management of the event before, during and after its occurrence
• Additional modalities provide richer information but require multi-modal
approaches

6.
6Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Social Media

7.
7Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/7
http://blog.tyronesystems.com/how-much-data-is-created-every-minute-by-the-social-media

8.
8Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Social Media as Real-Life Sensors
• Social Networks is a big data source with an
extremely dynamic nature that reflects
events and user’s interests
• Huge smartphones and mobile devices
penetration provides real-time and
location-based user feedback
• Transform individually rare but collectively
frequent media to meaningful topics,
events, points of interest, emotional states
and social connections
• Present in an efficient way for a variety of
applications (natural disaster, news,
marketing, science, health, entertainment)

9.
9Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/9
“…if you're more than 100 km away from the epicenter [of an earthquake] you
can read about the quake on twitter before it hits you…”

10.
10Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Social Media Data in Natural Disasters
• Use of social media on natural disasters management:
• Pre-crisis: monitoring, situation awareness and early warning
• During crisis: providing real-time information about incidents
• Post-crisis: identifying where stress management is most needed
• Challenges faced when crawling from Twitter:
• The public Streaming API provides 1% of the current posting volume
• Less than 2% of tweets are geo-located
• Retrieved information is ambiguous and not always relevant (e.g. flooding
à flooding my timeline)
• Misleading information (aka “Fake News”)

11.
11Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Crawling Social Media Data
• Focusing on Twitter posts, collected with Twitter Streaming API
https://developer.twitter.com/en/docs/tweets/filter-realtime/overview
• Various analysis techniques to obtain further knowledge on the tweets
• The complete flow:
new
tweet
Search terms:
• Keywords
• Accounts
• Bounding
Boxes
Keys & Tokens
Twitter
Streaming API
Client
receives
tweets
Fake tweets
detection
Text
classification
Image
classification
Get tweet in
JSON format &
find matching
use case
Nudity
detection
Tweets
localisation
Concept
extraction
tweet
has
image
tweet
has no
image
Inputs:

12.
12Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Crawling Social Media Data
• Some of our collections of tweets that concern natural disasters in different
countries, posted since June 2017
10 m. about
fires in Spain 75 k.
about
floods in
Italy
74 k.
about
heatwave
in Greece
42 k.
about
snow in
Finland

13.
13Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Fake Tweets Classification Model
Boididou, C., Papadopoulos, S., Zampoglou, M., Apostolidis, L., Papadopoulou, O., & Kompatsiaris, Y.
(2018). Detection and visualization of misleading content on Twitter. International Journal of
Multimedia Information Retrieval, 7(1), 71-86.
Fake tweets
detection

14.
14Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Credibility signals (aka features)

15.
15Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Relevancy estimation of Social Media Text
• Preprocessing:
• removing stop words/punctuation/hyperlinks
• stemming
• mapping to concepts
• Text representation to vectors
• Text classification
• Word2vec representation: comprises two-layer neural networks trained to
reconstruct linguistic contexts of words and produce eventually word
embeddings
• Similar words and meanings become “close” (e.g. flooding, disaster, emergency,..)
Moumtzidou, A., Andreadis, S., Gialampoukidis, I., Karakostas, A., Vrochidis, S. and Kompatsiaris, I., 2018, April. Flood
relevance estimation from visual and textual content in social media streams. In Companion Proceedings of the The
Web Conference 2018 (pp. 1621-1627). International World Wide Web Conferences Steering Committee.
Text
classification

16.
16Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Relevancy estimation of Social Media Text -
Experiments
• Parameters:
• Vector dimension = number of features {100, 200, 300, 400, 500}
• Words windows = max. distance between current and predicted word within sentence
• Training algorithm = skip-gram (0) or CBOW (1)
• Corpora:
• mediaEvalFloods_corpus (MediaEval 2017 dataset:
https://multimediaeval.github.io/2017-Multimedia-Satellite-Task/)
• twitterFloods_corpus (our collection of tweets in English about floods)

17.
17Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Concept Detection in Social Media Images
• Extracts high-level concepts from visual low-level information
• Fine-tune pre-trained 22-layer GoogleNet DCNN network to recognize the 345
TRECVID INS concepts and thresholding to keep concepts with higher probability
• Concept examples: animal, boat_ship, clouds, waterscape_waterfront

18.
18Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Relevancy estimation of Social Media
Images
• Classification:
• use of output of last pooling layer with dimension 1024 for global
image representation
• SVM classifier per concept
• use annotated dataset for training and validating of classifier
• tuning of SVM classifier for different t (kernel type) and g (gamma
in the kernel function) to achieve maximum performance
Moumtzidou, A., Andreadis, S., Gialampoukidis, I., Karakostas, A., Vrochidis, S. and Kompatsiaris, I., 2018, April. Flood
relevance estimation from visual and textual content in social media streams. In Companion Proceedings of the The
Web Conference 2018 (pp. 1621-1627). International World Wide Web Conferences Steering Committee.
Image
classification

19.
19Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Relevancy estimation of Social Media Images -
Experiments
• Evaluation of several visual
descriptors and DCNN-based
features
• Tuning of SVM classifier for
different t kernel function
types (i.e. linear, polynomial,
radial and sigmoid) and g
gamma parameter in kernel
function
• Dataset from MediaEval 2017
(DIRSM task)
https://multimediaeval.github
.io/2017-Multimedia-Satellite-
Task/

20.
20Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
CERTH-ITI participation in MediaEval 2018
First in the social media image classification (Average F1-score)
https://www.youtube.com/watch?v=yq1nIPc6dWw&list=PLOPRp1vN
OG9ahE5viJmF6Gx8XDk8hG9MP&index=2&t=0s

21.
21Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Similarity Fusion from Multiple Sources
• Fusion at features level
• Social media posts indexing:
• Textual representation using word2vec
• Visual features using DCNN-based feature
• Visual concepts

22.
22Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Clustering of Social Media Text
• Grouping tweets according to the
relevancy of their text
• Density-based clustering with
LDA topic model
• discovering the abstract “topics”
that occur in a collection of
documents
• Most frequent words of a textual
cluster are visualized as word
clouds
• Each word cloud comprises the
tweets that are grouped into that
cluster
I. Gialampoukidis, S. Vrochidis, I. Kompatsiaris, A Hybrid framework
for news clustering based on the DBSCAN-Martingale and LDA. In
proceedings of the 12th international conference on Machine
Learning and Data Mining (MLDM2016), New York, July 16-21, 2016.
pp. 170-184, Springer International Publishing.

23.
23Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Community detection in Social Media Accounts
• Detect and visualize user
communities through their
interaction
• Create a network of social
media accounts that are linked
when one mentions the other
• Modularity maximization
(Louvain) community detection
• Find the key-players in these
communities which are
considered “authorities”

24.
24Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Estimation of the location mentioned in a tweet
• Long short-term memory-based (LSTM) named-entity recognition (NER)
• Semantically linked named entities to external Knowledge Bases (e.g.
OpenStreetMaps)
English
How quick-thinking mother saved family from Grenfell fire by flooding her flat
Emergency declared in #Paraguay after flooding from torrential rains. https://t.co/Z........
Italian
Presentazione il sistema di #allertameteo della #ProtezioneCivile della città di #Gorizia
Ponte Milvio fa acqua: ancora un allagamento in via Prati della Farnesina... #news #Roma
Finnish
Lumi riittää jo meidän pihaan! #Joensuu #lumi #sää

25.
25Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Estimation of the location mentioned in a tweet
• Results of the NER task for English
Dataset (CoNLL2003) Precision Recall F1-score
Our system (ELMo
embeddings)
91.63 93.01 92.32
Best-scoring CoNLL2003
system: Florian et al., 2003
88.99 88.54 88.76
Baevski, A. et al. 2019 (not
reported)
(not
reported)
93.5
Localisation steps after NER has been
performed on available tweets:
Dataset (EVALITA2009) Precision Recall F1-score
Our system (GloVe
embeddings)
75.49 75.60 75.37
Best-scoring shared task
system: FBK_ZanoliPianta
84.07 80.02 82.00
Nguyen and Moschitti, 2012 85.99 82.73 84.33
• Finnish NER still work in progress:
1. Dataset enhancement
2. Embeddings optimisation
• Results of the NER task for Italian

26.
26Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Position of tweets with estimated location and
visual content on a GIS view
• Two-way navigation
• From the Twitter streams to the map
• From the map to the Twitter streams

27.
27Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Cameras on UAVs

28.
28Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Identify objects of interest from aerial images
• Refers to the visual identification of specific objects
• Architectures of Deep Convolutional Neural Networks (DCNN)
• Bounding Box: R-CNN, Fast R-CNN, Faster R-CNN, YOLO algorithms
• Semantic segmentation: U-Net, Feature Pyramid Network (FPN), Pyramid Scene
Parsing Network (PSPNet), Mask R-CNN, DeepLab, Path Aggregation Network (PANet)
• Datasets: 2012 PASCAL VOC, PASCAL-Context, DOTA, senseFly, Cityscapes, DJI
Mavic Pro Footage, VisDrone2018

29.
29Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Tracking objects of interest
• Vision based autonomous control
• Affects the navigation of the UAV
• Approaches
• Hierarchical Particle Filter (HPF), Kernelised Correlation Filter (KCF), Multi-kernel
Correlation Filter, Multi-Domain Convolutional Neural Networks
• Single object tracking: VOT2016
• Multi-object tracking: MOTChallenge2015, UA-DETRAC

30.
30Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Combination of detection with tracking
• Tracker can (follow) track objects of
interest.
• Fast and efficient but have trouble in
difficult situations to keep track of object
• Combination of object detector and tracker
delivers merits from both worlds:
• Efficiency (tracker)
• Effectiveness (object detector)
• Object detection every N frames and apply
tracking in between.
Object detection results
Class
Average
precision
Class
Average
precision
UAV 0.75330 Boat 0.70251
Car 0.75726
Helicopter -
Plane
0.71638
Person 0.82152
Motorcycle-
Bicycle
0.73409
Truck 0.53351 Weapon 0.54025
Bus 0.57315
Military
vehicle
0.68435
Ship 0.83586 mAP 0.69565

31.
31Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Satellite images

32.
32Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Water body detection in satellite images
• Detect water bodies in flooded areas:
• Train a Deep Neural Network model
• Use Sentinel-1 VV and VH bands
• Use Elevation information (DEM)
• Increased performance:
• Minimizes holes (false negatives) in extended water surfaces
• Filters out steep areas (false positives)

33.
33Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Water body detection in satellite images
• Overall best results obtained by Adam optimizer and 0.001 learning rate
Conclusions:
• Accuracy improves for all cases comparing the Deep neural network model to the
histogram thresholding method
• Maximum F-score of 94.10% for Garda lake with the DNN model
Lakes Precision Recall Accuracy F-score Settings
Maggiore 98.29 87.84 93.15 92.77 Adam, 0.001
Maggiore 96.23 61.89 79.73 75.33 -22.0 dB (vh)
Garda 94.57 93.63 94.13 94.10 Adam, 0.001
Garda 95.55 70.45 83.58 81.10 -21.7 dB (vh)
Trasimeno 93.67 84.61 89.45 88.91 Adam, 0.001
Trasimeno 88.07 66.13 78.58 75.54 -13.9 dB (vv)

34.
34Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Change Detection in Satellite Image Time Series
• Detects flood incidents within a time-series of optical satellite imagery
• Two approaches:
• Remote sensing (Image Processing)
• Deep Convolutional Neural Network (DCNN)
• Evaluated in Multimedia Task of the MediaEval 2019
• Test set: 68 timeseries
• Location: African cities

35.
35Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Change Detection in Satellite Image Time Series
Remote sensing: Outlier detection on image differencing of MNDWI indices.
• Input: Time-series of Green and Swir bands of the events
• Method:
• Calculate the MNDWI index for all days: MNDWI =
• Compute the differences of consecutive days
• Detect outliers in the differences images à Generating change masks
• Output: Change is determined by considering the sum of outlier pixels
• if sum of outlier pixels > 5% of total pixels à flood
• Results: MNDWI, γ =2.1, ratio=0.05: 76.47% F-Score
SWIRGreen
SWIRGreen
+
-

36.
36Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Change Detection in Satellite Image Time Series
• Deep Convolutional Neural Network (DCNN)
• Input: Time-series of R-G-B or R-Swir-Nir bands of the events
• Method:
• Create JPEG of the differences of all days
• Fine-tune a VGG-16 model using training set of 367 events, and by considering all
combinations of the differences of RGB images
• Output: Predicted change in consecutive days denotes a flood event
• Results: Red-Green-Blue: 70.58% F-Score

37.
37Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Road passability in satellite images
Flood detection in satellite images*:
• Given two coordinate points on a satellite image the task is to select
the corresponding label (passable/non passable).
• Participants receive data and are required to train classifiers.
• Fusion of satellite and social multimedia information is encouraged.
• The task moves forward the state of the art by concentrating on
passibility, whether or not it is possible for a vehicle to pass a road.
*http://www.multimediaeval.org/mediaeval2018/multimediasatellite/index.html
A. Moumtzidou, P. Giannakeris, S. Andreadis, A. Mavropoulos, G. Meditskos, I.
Gialampoukidis, K. Avgerinakis, S. Vrochidis and I. Kompatsiaris, “A multimodal approach
in estimating road passability through a flooded area using social media and satellite
images”, @ multimedia satellite task MediaEval 2018. In Proceedings of the Working
Notes Proceeding MediaEval Workshop, Sophia Antipolis, France, 29-31 October, 2018.

38.
38Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Road passability in satellite images
• Infers if a road is passable or non-passable due to water
• Deep learning network for determining road passability (i.e Fine-tuned pre-trained on
ImageNet VGG-19)
• Splits initial image in segments and predicts the passability for each tile (flooded/non-flooded)
• Fusion with social media images
where the concept “vehicle” has been extracted
from items containing “flood” concept

39.
39Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Road passability in satellite images
• Approach
• Fine-tuning of pre-trained DCNN networks (e.g. ResNet101, VGG-16,
Inception, Inception_ResNet_v2) (published in “Big Data from Space
Conference 2019”)
• Settings:
• learning rate values = 0.001, 0.01, 0.1
• batch size values = 32, 64, 128, 256
• optimizer functions = Adam, Stochastic Gradient Descent (SGD)

40.
40Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Road passability in satellite images
• Conclusions
• Accuracy improves for the lower values of the learning rate
• Increase of the batch size generally improves the accuracy
Batch size 32 Batch size 64 Batch size 128
DCNN Learning rate Optimizer
Validation Set
Accuracy
Validation Set
Accuracy
Validation Set
Accuracy
VGG-19 0,01 Adam 0,7666 0,7666 0,7666
VGG-19 0,001 SGD 0,7071 0,7117 0,7162
Inception_v3 0,01 Adam 0,6247 0,5789 0,5629
Inception_v3 0,001 SGD 0,5950 0,6224 0,5973
VGG-16 0,001 Adam 0,7437 0,7574 0,7551
VGG-16 0,001 SGD 0,7277 0,7208 0,7231
ResNet101 0,1 Adam 0,5492 0,5126 0,5126
ResNet101 0,001 SGD 0,5835 0,5995 0,5881
Inception_ResNet_V2 0,001 Adam 0,6384 0,6178 0,6156
Inception_ResNet_V2 0,01 SGD 0,7002 0,6979 0,6819

41.
41Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
EOPEN project concept

42.
Enhancing decision support and management services in
extreme weather climate events
• Collection of heterogeneous data from several resources such as
environmental, social media, input from first responders and/or people in
danger
• Semantically integration of data to provide decision support services to the
crisis management center

43.
43Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Conclusions
• Existing visual information processing approaches can contribute to security
domain
• With adaptations: features, re-training
• Not always: lack of data, specific applications
• Additional modalities can be very useful
• Text, sensor information
• Fusion approaches
• Need for multi-modal Deep Learning approaches to detect events from
joint combinations of text, visual, sound and machine-generated
information for multi-faceted multi-variable event detection
• Challenges include: overall architecture, big data management,
visualizations and interfaces, user acceptance, privacy and ethics

44.
44Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Contributors
Stefanos Vrochidis, Senior Researcher – stefanos@iti.gr
Ilias Gialampoukidis, Postdoctoral Researcher – heliasgj@iti.gr
Konstantinos Ioannidis, Postdoctoral Researcher - kioannid@iti.gr
Anastasia Moumtzidou, Research Associate - moumtzid@iti.gr
Stelios Andreadis, Research Associate - andreadisst@iti.gr

45.
45Multimedia Knowledge & Social Media Analytics Laboratory
http://mklab.iti.gr/
Thank you!
Email: ikom@iti.gr
Lab: mklab.iti.gr
Supported by the projects
EOPEN (H2020-776019) and beAWARE (H2020-700475)
funded by the European Commission