Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Image4Act: Online Social Media Image
Processing for Disaster Response
Firoj Alam, Muhammad Imran, Ferda Ofli
Qatar Computi...
Time-Critical Events and Information Gaps
Info. Info. Info.
Disaster event (earthquake, flood) Destruction, Damage
Informa...
2013 Pakistan Earthquake
September 28 at 07:34 UTC
2010 Haiti Earthquake
January 12 at 21:53 UTC
Social Media Data and Opp...
Social Media Images During Disasters
Damage Severity Assessment from Images
Social Media is Noisy
(Irrelevant & Duplicate Content)
Examples of irrelevant images showing cartoons, banners, advertisem...
Automatic Image Processing Pipeline
Detailed Architecture
Image URLs
DB
Tweet
Collector
Image
Collector
Image
Filtering
Relevancy
filtering model
De-duplicatio...
Labeled Datasets
NE: Nepal earthquake -- EE: Ecuador earthquake – TR: Typhoon Ruby – HM: Hurricane Matthew
Relevancy Filtering
Examples of irrelevant images showing cartoons, banners, advertisements, celebrities, etc.
Performance...
Duplicate Filtering
Examples of near-duplicate images
Task: Compute similarity between a pair of images
Approach: Perceptu...
Before/After Image Filtering
Number of images that remain in our dataset after each image filtering operation
~ 2 %
~ 2 %
...
Infrastructure Damage Assessment
• Three-class classification
– Categories: severe, mild & little-to-none
• Distinction be...
Deployment and Evaluation during
Cyclone Debbie Event
Randomly selected 500 images
Manually labeled irrelevant images
Rele...
Thanks – Q & A
Follow this project: @aidr_qcri
We are looking for a PostDoc
(Computer vision, natural language processing,...
Upcoming SlideShare
Loading in …5
×

Image4Act: Online Social Media Image Processing for Disaster Response

358 views

Published on

We present an end-to-end social media image processing system called Image4Act. The system aims at collecting, denoising, and classifying imagery content posted on social media platforms to help humanitarian organizations in gaining situational awareness and launching relief operations. The system combines human computation and machine learning techniques to process high-volume social media imagery content in real time during natural and human-made disasters. To cope with the noisy nature of the social media imagery data, we use a deep neural network and perceptual hashing techniques to filter out irrelevant and duplicate images. Furthermore, we present a specific use case to assess the severity of infrastructure damage incurred by a disaster. The evaluations of the system on existing disaster datasets as well as a real-world deployment during a recent cyclone prove the effectiveness of the system.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Image4Act: Online Social Media Image Processing for Disaster Response

  1. 1. Image4Act: Online Social Media Image Processing for Disaster Response Firoj Alam, Muhammad Imran, Ferda Ofli Qatar Computing Research Institute Hamad Bin Khalifa University, Qatar
  2. 2. Time-Critical Events and Information Gaps Info. Info. Info. Disaster event (earthquake, flood) Destruction, Damage Information gathering Humanitarian organizations and local administration Need information to help and launch response Information gathering, especially in real-time, is the most challenging part Relief operations Disaster
  3. 3. 2013 Pakistan Earthquake September 28 at 07:34 UTC 2010 Haiti Earthquake January 12 at 21:53 UTC Social Media Data and Opportunities Social Media Platforms Availability of Immense Data: Around 16 thousands tweets per minute were posted during the hurricane Sandy in the US. Opportunities: - Early warning and event detection - Situational awareness - Actionable information - Rapid crisis response - Post-disaster analysis Disease outbreaks
  4. 4. Social Media Images During Disasters
  5. 5. Damage Severity Assessment from Images
  6. 6. Social Media is Noisy (Irrelevant & Duplicate Content) Examples of irrelevant images showing cartoons, banners, advertisements, celebrities, etc. Posted during the 2015 Nepal earthquake Examples of near-duplicate images posted during the 2015 Nepal Earthquake
  7. 7. Automatic Image Processing Pipeline
  8. 8. Detailed Architecture Image URLs DB Tweet Collector Image Collector Image Filtering Relevancy filtering model De-duplication model Web Crowd Task Manager Image Classifier(s) Persist In-memory DB Crowd tasks & answers Image downloading Tweets Images Images Images Is relevant? Is duplicate? Classified Images (filesystem) Damage Images Injured People Rescue efforts Image Hash DB Database In-memory DB Is URL duplicate? Persister Classified images paths Postgres DB Crowd Images Labels
  9. 9. Labeled Datasets NE: Nepal earthquake -- EE: Ecuador earthquake – TR: Typhoon Ruby – HM: Hurricane Matthew
  10. 10. Relevancy Filtering Examples of irrelevant images showing cartoons, banners, advertisements, celebrities, etc. Performance of the relevancy filtering Task: Build a binary classifier to identify irrelevant images Approach: Transfer learning (fine-tune a pre-trained convolutional neural network, e.g., VGG16)
  11. 11. Duplicate Filtering Examples of near-duplicate images Task: Compute similarity between a pair of images Approach: Perceptual Hash + Hamming Distance (w/ threshold)
  12. 12. Before/After Image Filtering Number of images that remain in our dataset after each image filtering operation ~ 2 % ~ 2 % ~ 50 % ~ 58 % ~ 50 % ~ 30 % Assume tagging an image costs $1, we could have gotten the same job done by paying $17k less, almost saving 2/3s of the budget!!!
  13. 13. Infrastructure Damage Assessment • Three-class classification – Categories: severe, mild & little-to-none • Distinction between categories is ambiguous. • Agreement among human annotators is low. – in particular for mild category • Fine-tuning a pre-trained CNN (e.g., VGG16)
  14. 14. Deployment and Evaluation during Cyclone Debbie Event Randomly selected 500 images Manually labeled irrelevant images Relevancy Filtering - Precision: 0.67 Duplicate Images - Precision: 0.92
  15. 15. Thanks – Q & A Follow this project: @aidr_qcri We are looking for a PostDoc (Computer vision, natural language processing, system development) Contact us: mimran@hbku.edu.qa

×