Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises
Report
•6 likes•930 views
Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises
•6 likes•930 views
Download to read offline
Report
The extensive use of social media platforms, especially during disasters, creates unique opportunities for humanitarian organizations to gain situational awareness and launch relief operations accordingly. In addition to the textual content, people post overwhelming amounts of imagery data on social networks within minutes of a disaster hit. Studies point to the importance of this online imagery content for emergency response. Despite recent advances in the computer vision field, automatic processing of the crisis-related social media imagery data remains a challenging task. It is because a majority of which consists of redundant and irrelevant content. In this paper, we present an image processing pipeline that comprises de-duplication and relevancy filtering mechanisms to collect and filter social media image content in real-time during a crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.
Recommended
More Related Content
Similar to Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises
Similar to Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises(20)
Recently uploaded
Recently uploaded(20)
Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises
- 7. Challenges of Social Media Image Processing • High velocity and high volume – Crowd cannot keep up • Sandy Hurricane – 20 millions tweets in 5 days – 16 thousands tweets per minute Background
- 8. Challenges of Social Media Image Processing • High velocity and high volume – Crowd cannot keep up • Sandy Hurricane – 20 millions tweets in 5 days – 16 thousands tweets per minute • Large por>on of redundant/duplicate and irrelevant content – We should not waste crowd’s >me and effort Background
- 13. Related Works • Very few: – Images with on-topic tweets are more relevant (Peters and Joao, 2015) – Visually relevant and irrelevant tweets (F1 70.5% in binary classifica>on task) (Chen et al., 2013) – Damage level from aerial Images (Turker et al., 2004; Fernandez et al. 2015) • Mostly used bag-of-visual-word features Background
- 16. Data: Damage Assessment Corpus Annota'on: • Severe damage: – Substan>al destruc>on of an infrastructure – Non-usable buildings, roads, bridges for example. • Mild damage: – Minor damages with up to 50% • LiQle-to-no damage Number of labeled images for each dataset and each damage category
- 17. Data: Relevancy Detec>on Corpus Annota'on Mapping: • Images has been selected from the data we annotated for damage assessment – {severe, mild} à relevant (randomly sampled 3518 images) • Resulted 7036 Images {relevant, irrelevant} Image annotated as LiQle-to-None ImageNet Object Detec'on Model Object category website, suit, lab coat, envelope, dust jacket, candle, menu, vestment, monitor, street sign, puzzle, television, cash machine, screen irrelevant Most frequently occurring Human Supervision
- 23. Relevancy Filtering (Cont.) Image Processing Pipeline • Deep Learning Model: Build a binary classifier – Transfer learning – Extracted features from VGG16, a pre-trained convolu>onal neural network, e.g., VGG16 (Simonyan et al., 2014 ) – Fine tuned the parameters 1x2 Image credit: hIps://blog.heuritech.com
- 24. Relevancy Filtering (Cont.) * Simonyan, K. and Zisserman, A. (2014). “Very deep convolu>onal networks for large-scale image recogni>on”. In: arXiv preprint arXiv:1409.1556 Image Processing Pipeline • Data Split for Experiments: – Training 60% – Valida>on 20% – Test 20% • Performance
- 27. Duplicate Filtering (Cont.) Image Processing Pipeline • Selected 1100 images for the experiment • Approach: – Compute perceptual hash for each image – Compute hamming distance between a pair of images Two images are duplicate if: distance(Vi, Vj) 10 Vi = pHash(Imagei) Vi ∈ Rn distance(Vi, Vj) = n k=0 |(Vi,k − Vj,k)| Vi = pHash(Imagei) Vi ∈ Rn distance(Vi, Vj) = n k=0 |(Vi,k − Vj,k)| hamming distance(Vi, Vj) = n k=1 (Vi,k ̸= Vj,k) (Lei et al. 2011)
- 33. Damage Assessment • Three-class classifica>on – Categories: severe, mild & liIle-to-none • Dis>nc>on between categories is ambiguous. Image Processing Pipeline Descrip'on Severe Mild None All Number of Images 1765 483 3751 6000
- 36. Summary • We propose a real >me image processing pipeline, which filters – duplicate, near duplicate, and – irrelevant images and categories – images into severe, mild and liIle-to-no damage • Relevancy filter model: AUC: 0.98; F1: 0.98 • Damage assessment model: AUC: 0.72; F1: 0.67 • We can save 62% of the whole budget Summary and Future Works 62% images are irrelevant and duplicates D. T. Nguyen, F. Alam, F. Ofli, M. Imran, “Automa>c image filtering on social networks using deep learning and perceptual hashing during crises”, ISCRAM 2017.
- 37. Future Work • Image processing pipeline – Please check: hIp://aidr.qcri.org • Fine-grained object analysis – Severity of injury, quality of aid/shelter, etc. • Mul>modal analysis – text + image + metadata Summary and Future Works