Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multimedia rescue 161018


Published on

Disasters Happen. We need to manage them to minimize the loss to life and property. Disaster management has been received much attention, but has not been touched much by the latest technology. This paper presents an approach to manage disasters using latest and popular technology. We are interested in building a community of researchers who are interested in developing such tools.

Published in: Technology
  • Be the first to comment

Multimedia rescue 161018

  1. 1. Mengfan Tang, Siripen Pongpaichet, and Ramesh Jain Department of Computer Science University of California, Irvine ACMMM 2016, BNI session Tue 18 October 2016 @2PM MULTIMEDIA FOR EMERGENCY SITUATIONS
  2. 2. Society exists only as a mental concept; in the real world there are only individuals. -- Oscar Wilde
  3. 3. Multimedia: Machines/Devices 3 1993 2016
  4. 4. Technology: From Features to Machine Learning 1993 2016
  5. 5. Going From Advertisement and Entertainment
  6. 6. Going From Advertisement and Entertainment to Social Good
  7. 7. Shit Happens! 7 Hurricanes - Typhoons Forest Fires Earthquakes - Tsunamis Disaster Happens! Infectious diseasesDisease Epidemics -- Zika Riots
  8. 8. Life Cycle of Disaster Management 8 Focus of discussion today.
  9. 9. Community Computing: Merging 3 Big Trends Community-based Apps Real-time Coordination Power to Create and Share Ideas Waze Uber, Airbnb Twitter, Instagram
  10. 10. Micro-events from M reportersSituation at Macro Level Hurricane Matthew
  11. 11. Fundamental Problemduringa Disaster 11 Providing Resources and Information Effectively, Efficiently, and Promptly In evolving Emergency Situations.
  12. 12. Disaster Response 12 Physical World And Informatio n Systems Environment, Resources, Information Personal Situation, Needs, Information Matching Action Signals
  13. 13. Environment and Resources Disaster Related Evolving Situations • Traffic • Precipitation • Meteorology • Power outage • Flood level • Food • Diseases Data Sources  Webcams,  Traffic systems (Google)   NOAA  Government: Division of Emergency Management  Social Media
  14. 14. Personal Situation and Needs Personal Needs • Shelter • Food • Water • Family and Friends • Information Data Sources  Volunteer <Org>  The-Emergency-Food-Assistance- Program  Drinking-water, flooded-wells  Citizen reports  Government announcements Shelter locations Food assistance locations Boil Water Notice
  15. 15. Timely Alerts and Response Evacuation Zone Evacuation Route Communication Methods (pull) - News - Radio Alert Methods (push) - Targeted messages - Mobile apps
  16. 16. Disaster Report and Response App Personal Situation Recognition Situation Recognition and Prediction Data Stream Ingestion and Aggregation Building Emergency Situation ModelsDatabase Real-Time Situation Recognition Offline Process REPORT RESPONSE Matching Resources Determination Identify People Needs Emergency Situations Actionable Decisions
  17. 17. Current Explorations: Clear Potential of Emerging Multimedia • EventShop – Started about 8 years ago – Select and combine diverse heterogeneous geo-spatial data streams to detect situations – Demonstrated for several applications – Inspired by Photoshop • Multimedia Micro-Reports (MMR) – Started about 2 years ago. – Create multimedia platform for citizen sensing – Demonstrated using YFCC100M photo release – Inspired by Krumbs
  18. 18. Prototype: Thai Flood Emergency Response
  19. 19. Flood level - Shelter Flood Level Shelter Twitter Classify (Flood level - Shelter)
  20. 20. FILTER LOC=CA AGG FILTER LOC=CA Asthma Risk Area without Interpolation Sparse PM 2.5 Sensors Pollen Ozone AQI Asthma Risk Area with interpolation AGG Interpolated PM2.5 Physical model outputs Satellite Images Traffic Assimilating Data Sources = Better Situation Recognition correlation with asthma hospitalizations increased from 0.17 to 0.45
  21. 21. Is a Smartphone camera a camera? Smartphone collects all metadata related to the Event. • Exposure Time • Aperture Diameter • Flash • Metering Mode • ISO Ratings • Focal Length • Time • Location • Face and emotions • Activity parameters • Contextual reasoning • Myriad sensors through wearables Camera captures photos. Smartphone camera captures events.
  22. 22. Media Data is about W5H {"micro_reports":[{ "where":{ "geo_location":{ "latitude":32.90233332316081, "longitude":-117.2441166718801}, "when":{ "start_time":"Jun 14, 2009 11:25:19 AM", "end_time":"Jun 14, 2009 11:25:19 AM", "time_zone":"America/Los_Angeles"}, "what":[{ "concept_name":"people", "confidence":0.9836078882217407, "visual_concept_provider":"CLARIFAI"}, … { "concept_name":"food", "confidence":0.8526291847229004, "visual_concept_provider":"CLARIFAI"}], "tag":”#niceday #summer", "source":{"default_src":"https://….jpg"}}, "sub_event":[], "why":[]}, …]} Photo, video What Where When Who Why Sound How
  23. 23. Using Photo Concepts for Creating Worldwide Situation Map! We can detect situations using ”bag of concepts”. 5+ Billion phone photos = Situation Map of the World New frontier for Machine learning.
  24. 24. Resource Multimedia Micro-Reports Situation Needs Rescue Project Rescue ProjectRescue Project Rescue Project
  25. 25. Rescue Project
  26. 26. Research Challenges Human Reporting Data Processing Real-time Recognition Predictive Analytics Decision Support What-If Scenario Situation Recognition Communica- tion Tech Data Identification Data Quality
  27. 27. Big 4 29 Multimedia Participatory Reporting Multimedia Participatory Sensing. Real-time Situation Recognition Situation Recognition: Situation Models from past data Real time recognition Predictive Analytics Predictive analytics Multimedia Semantics Multimedia Semantics
  28. 28. Data Streams Multimedia Semantics: From Different Data Streams to Abstracted Event Streams Real World Event Streams 30
  29. 29. Multimedia Micro-Reports Twitter is a good concept for opinions. However, it is too noisy and too subjective for solving real problems.
  30. 30. Designing Multimedia Micro-Reports Objective Spontaneous Compelling Universal (Language Independent) Requirements of reports Opportunity Photos and Videos Text Contextual data Personal data Subjective and Objective Universal
  31. 31. Recognition Challenge • Yesterday: Feature Based Recognition • Current: Machine Learning for Objects • Tomorrow: Situation Recognition
  32. 32. Real Time Situation Recognition • Use appropriate data sources. • Build Situation Models – New Frontier for Machine Learning. • Great Opportunity: Real Time Situation Recognition algorithms classify world wide situations.
  33. 33. Predictive Analytics: Spatial Interpolation • Observations from sensors are sparse. • We need spatial interpolation. Accurate interpolation usually requires combining multiple data sources.
  34. 34. Predictive Analytics: Cyber Space to Cybernetic Systems • Prediction of evolving situation at every point gives us better chances to control situations. • Can we extend Wiener filtering and Kalman filtering into prediction methods for streaming data? Kalman Filetering
  35. 35. 4 1 3 1 1 1 Our Partners using EventShop.
  36. 36. Our Dream • Build a vibrant Multimedia Rescue community – Research: Ready Research Problems – Implementation – Experience • Deploy and Learn • Multimedia is natural for humans. Let’s make multimedia technology also natural in important situations. Join us, please!