This document provides an overview of the Flood-related Multimedia Task, which aimed to analyze social media posts from Twitter to determine their relevance to actual flooding incidents in northeast Italy. The task involved classifying tweets containing Italian keywords related to floods as either relevant or not relevant to flooding in the target area. 5 teams participated and submitted runs using text, image, or combined features. The best performance was from undersampling and combining three artificial neural networks, achieving a maximum F1-score of 0.5405 for classifying the tweets, though on average performance was low, indicating this was a challenging task.

1. Overview of the Flood-related Multimedia Task
Presenter: Ilias Gialampoukidis (ITI - CERTH)

2. Motivation
● Floods are a natural disaster that affects most
places on Earth & causes a vast number of
deaths and damages
● Social media posts are valuable in all stages
of managing a disaster
○ pre-emergency phase: notify about a
possible disaster
○ during the disaster: provide insights &
detect areas in danger
○ post-emergency phase: assist in the
damage control
2
Image: UNISDR / CRED

3. Motivation
● Problem: the large streams of published posts carry
lots of noise
○ metaphorical use of flood-related words
○ incidents outside the area of interest
○ past events
● Makes it difficult to collect high-quality information
● Automatic estimation of a tweet’s relevance could
address this challenge
→ separate relevant and not relevant tweets
→ filter out unrelated posts
→ support first responders & civil protection
3
Images: Screenshots from Twitter

4. Task description
4
● Tackles the analysis of social multimedia from Twitter
for flooding events
● Participants receive a set of tweets and their
associated images
○ text contains keywords related to floods in a
specific area of interest, i.e. the Eastern Alps
partition in NE Italy
● The relevance of the tweets to actual flooding
incidents in that area is ambiguous!
Image: Screenshot from http://www.alpiorientali.it

5. Task description
5
● Objective: build an information retrieval system /
classifier to distinguish whether or not a tweet is
relevant
● Selection of Italian-language tweets → encourage
researchers to move away from a focus on English
● Participants can tackle the task using text features,
image features, or a combination of both
Images: Screenshots from Twitter

6. Data set
6
● Social media posts were collected from Twitter
between 2017 and 2019, by searching for Italian
flood-related keywords
● Dev set: 5,419; Test set: 2,279
● All tweets contain an attached image
● All tweets were still online when releasing
● To be compliant with the Twitter Developer Policy,
only the IDs of the tweets were distributed
● Ground truth was collected with human annotation
by the Alto Adriatico Water Authority, experts on
flood risk management in the examined area
Dev set
Test set

7. Runs & Evaluation
7
● Participants were allowed to submit up to 5 runs:
1) Required run: automated using textual-visual fused
2) Optional run: automated using textual information only
3) Optional run: automated using visual information only
4) General run: everything automated allowed, incl. external data
5) Same as 4
● The correctness of retrieved tweets for the two classes relevant and not relevant
were evaluated with the F1-Score metric on the test set

8. Participation
8
● 10 teams registered
● 5 teams submitted their results
● 5 teams submitted the working notes paper
● 1st run (textual & visual fused): 4 submissions
● 2nd run (textual): 5 submissions
● 3rd run (visual): 4 submissions
● 4th run (general): 3 submissions
● 5th run (general): 2 submissions

9. Results
9
1st run (textual & visual fused)
2nd run (textual)

10. Results
10
3rd run (visual)
4th run (general)
5th run (general)

11. Results
11

12. Remarks
12
● Methods used:
○ BOW, TF-IDF, BERT for text representation
○ MLP, RNN, CNN, SVM, RF, MNB for classification
○ SMOTE, Augmentor for oversampling; Random for undersampling
● Proved to be a hard task (max F-Score = 0.5405, avg = 0.2183)
● Subjectivity of the annotator
● Better performance with textual features
● Undersampling & combination of three ANNs achieved the best results

13. Information Technologies Institute
Centre for Research and Technology Hellas
Greece
Stelios Andreadis, Ilias Gialampoukidis,
Anastasios Karakostas, Stefanos Vrochidis,
Ioannis Kompatsiaris
13
Alto Adriatico Water Authority
Italy
Roberto Fiorin, Daniele Norbiato,
Michele Ferri
Supported by the EU’s Horizon 2020 research and innovation
programme under grant agreements H2020-700475 beAWARE,
H2020-776019 EOPEN and H2020-832876 aqua3S.
Organization & Acknowledgements

14. Thank you
for your attention!