Paper: http://ceur-ws.org/Vol-2670/MediaEval_19_paper_48.pdf Youtube: https://www.youtube.com/watch?v=bnp0arMsdYc Pierrick Bruneau and Thomas Tamisier, Transfer Learning and Mixed Input Deep Neural Networks for Estimating Flood Severity in News Content. Proc. of MediaEval 2019, 27-29 October 2019, Sophia Antipolis, France. Abstract: This paper describes deep learning approaches which use textual and visual features for flood severity detection in news content. In the context of the MediaEval 2019 Multimedia Satellite task, we test the value of transferring models pre-trained on large related corpora, as well as the improvement brought by dual branch models that combine embeddings output from mixed textual and visual inputs. Presented by Pierrick Bruneau

1. Transfer Learning and Mixed Input Deep
Neural Networks for Estimating Flood Severity
in News Content
Pierrick Bruneau, Thomas Tamisier
MediaEval 2019
10th Anniversary Workshop
27-29 October 2019
EURECOM, Sophia Antipolis, France

2. Motivation
• Addressing NITD and MFLE subtasks
• Not using specialized models (e.g. pose estimation
[Bulat & Tzimiropoulos 16])
• Standard binary classification problems
• Value of transfer learning (= external information)
• Value of dual branch image+text models (≠ Siamese
networks for similarity learning [Huang et al. 17])

3. General Protocol
• 50 epochs, Adam (LR of 1e-4 for fine tuning)
• Validation metric: F1 on positive class
• Among epochs, select model with best val. metric
• Batches of size 32 (training set size vs. imbalance)
• Stratified 5-fold CV, combined as model ensembles
• Instance weighting to balance class importance
• SMOTE [Chawla et al. 2002]: under-sampling majority
class, over-sampling minority class + augmentation

4. Textual Features
• Tried out several architectures
• LSTM, BiLSTM, Multi-head Attention, Hierarchical
Attention, Char-level ConvNet, using Glove vectors
• Evaluation with (balanced) AG-News and IMDB corpora
• No option significantly better than LSTM + random init.
• Classification tasks « simple enough » (e.g. w.r.t.
machine translation)
• Augmentation for text: random offset to fixed size
sequence extraction

5. Textual Features (contd.)
• Hyper-parameters for MFLE: grid search
• Fixed text length (+cut or padding): 50
• Hidden vector size: 100
• Word embedding size: 32
• Hidden vector size used as embedding

6. Image Features
• InceptionV3 (ImageNet), VGG16 (PlacesNet),
MobileNetV2 (random initialization)
• Default layer specifications, 224x224 resizing
• Classic data augmentation techniques (zoom, shift,
shear, horizontal flip)
• Last Fully Connected layer used as embedding

7. Mixed Input Models

8. Results

9. Takeaway
• Fine-tuned VGG16 is the best for NITD,
LSTM+InceptionV3 best for MFLE (!...?)
• Random initialization performs decently at NITD,
largely outperformed at MFLE
• Positive text+image combination effect
• For heavy imbalance (MFLE), also try outlier
detection

10. A Bit of Insight
Class Activation Maps
(InceptionV3)
[Zhou et al. CVPR’16]

11. Questions?
MediaEval 2019
10th Anniversary Workshop
27-29 October 2019
EURECOM, Sophia Antipolis, France