Situation Recognition from Multimodal Data Tutorial (ICME2016)
•Download as PPTX, PDF•
1 like•646 views
This document provides an overview of situation recognition from multimodal data. It discusses situation modeling which involves defining components like operands and operators. Common situation modeling approaches include situation calculus and situation algebra. The document presents a framework for situation recognition involving situation modeling, recognition, and visualization/alerts. It also provides an example of modeling an epidemic situation using social media data streams and aggregation operators.
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Situation Recognition from Multimodal Data Tutorial (ICME2016)
Similar to Situation Recognition from Multimodal Data Tutorial (ICME2016) (20)
Recently uploaded
Recently uploaded (20)
Situation Recognition from Multimodal Data Tutorial (ICME2016)
- 1. /125 SITUATION RECOGNITION FROM MULTIMODAL DATA Vivek K. Singh1, Siripen Pongpaichet2, and Ramesh Jain2 1Rutgers University, 2University of California, Irvine
- 2. /125 Today’s slides 2 http://www.springer.com/us/book/9783319305356 Or email us for a softcopy. http://bit.ly/29JL30M
- 3. /125 Course Outline 1) Concept recognition from Multimedia data (20 mins, Ramesh Jain) • Trends • Why situation recognition is different from object, event, scene recognition etc. 2) Situation recognition across multiple research domains (20 mins, Vivek Singh) • Situation Algebra, Situation Calculus, Robotics, … 3) Situation recognition (45 mins, Vivek Singh) • Situation modeling • Situation operators 4) Designing situation based applications (30 mins, Siripen Pongpaichet) • Motivation and essential requirements. • Application scenarios: Thailand flood, hurricane Sandy, city sensing, asthma relief 5) Future trends and open problems (20 mins, Ramesh Jain) • Future trends • Open problems for Multimedia research 6) Question and Answers (15 mins) 3
- 4. /125 CONCEPT RECOGNITION FROM MULTIMEDIA DATA (20 mins, Ramesh Jain) 4
- 5. /125 Introduction • Object, event, scene recognition • Trends • Situation recognition ( is different from object, event, scene recognition etc.) 5
- 6. /125 Data, Information, Knowledge, Wisdom 6 Data is Essential. But, we are really interested in products: Information, Knowledge, and Wisdom.
- 7. /125 What is Important in ‘Big Data’? Multimedia Realtime Uncertainty 7
- 8. /125 The Grand Challenge Sense making from multimodal massive geo-social data- streams. 8
- 9. /125 Fundamental Problem Connecting People to Resources effectively, efficiently, and promptly in given situations.
- 10. /125 What is Cyber Space? Who invented it? Animals Machines Societies Published first in 1942 10
- 11. /125 •Desired state (Goal) •System model and Control Signal (Actions) •Current State (using multimedia data) 11
- 12. /125 Input Computed using System Model Feedback Output compared with desired goal Actual System Output Observed Continuously 12
- 14. /125 Connecting People And Resources Social Life Networks Aggregation and Composition Situation Detection Alerts Queries Information 7/21/2016 14
- 15. /125 Traditional Social Systems •Models of Systems were difficult to form. •Current State of the system could not be determined. •Real time ‘actions’ could not be implemented. 15
- 16. /125 •Social models can be determined using warehouses of Big Data. •Social observations are now possible with little latency. •Actions could be targeted to precise sources. 16
- 17. /125 EventShop : Global Situation Detection Predictive Situation Recognitio n Evolving Global Situation Predictive Personal Situation Recognitio n Personal EventShop Evolving Personal Situation Need- Resource Matcher Recommendation Engine PersonaDatabas e Resources Needs Data Ingestio n Wearable Sensors Calendar Location…. DataSources …. Data Ingestion and aggregatio n Database Systems Satellite Environmental Sensor Devices Social Network Internet of Things Actionable Information 17
- 18. /125 Concept Recognition: Last Century 18 Environ ments Real world Objects Situations Activities SingleMedia SPACE TIME ScenesLocation aware Visual Objects Trajectories Visual Events Location unaware Static Dynamic Location aware Location unaware Static Dynamic Data = Text or Images or Video
- 19. /125 Visual Concept Recognition: First research papers • 1963: Object Recognition [Lawrence + Roberts] • 1967: Scene Analysis [Guzman] • 1984: Trajectory detection [Ed Chang+ Kurz] • 1986: Event Recognition [Haynes + Jain] • 1988: Situation Recognition [Dickmanns] 1960 1970 1980 1990 2000 2010 Object Scene Trajectory Event Situation
- 20. /125 Concept Recognition: This Century 20 Environ ments Real world Objects Situations Activities SPACE TIME Location aware Location unaware Static Dynamic HeterogeneousMedia Location aware Location unaware Static Dynamic Data is just Data. Medium and sources do not matter.
- 21. /125 Concept recognition from multimedia data 21 SPACE TIME ScenesLocation aware Visual Objects Situations Visual Events Location unaware Static Dynamic HeterogeneousMedia SingleMedia 360 K 11.4K 3.4 K Location aware Location unaware Static Dynamic
- 22. /125 Situation Recognition: Next Frontier • Data Abundance. • Big Opportunity for Multimedia • Time • Space • Multiple diverse data streams • Need new frameworks. 22
- 23. /125 SITUATION RECOGNITION ACROSS MULTIPLE RESEARCH DOMAINS (20 mins, Vivek Singh) 23
- 24. /125 Related Work: Data to Situations Area Combine hetero data Human sensors Data analytics Define situations Location aware Real-time streams Toolkits Situation Awareness X X o o X Situation Calculus X Web data mining o X X o X Social media mining o X X o X Multimedia Event detection X o o o Complex event processing/ Active DB X X o X GIS X o X X o Mashup toolkits (Y! pipes, ifttt) X X o X X X X X X X X X X X X X X This work X X X X X X X XX 24 o = partial support
- 25. /125 Defining Situations: Situation Calculus • A situation s is the complete state of the universe at an instant of time. –McCarthy, 1968 • Snapshot of the world at a given time.- Reiter, 1991 • The set of necessary and sufficient world state descriptors for undertaking control decision. – Singh & Jain, 2009 25
- 26. /125 Situation Calculus: Quick overview • enter(P1), startWork(P1) • enter(P1), exit(P1), enter(P1), startWork(P1), stopWork(P1), startWork(P1) - isInRoom(P1, s(k)) - isWorking(P1, s(k)) isInRoom(P1, s) ˄ ~isWorking(P1, s) → IncreaseMusicVolume() isInRoom(P1, s) 0 isWorking(P1, s) 01 1 Situation = Not events , nor sequence of events, but their assimilated descriptor
- 27. /125 Problems with this approach • Scalability: • Listing all the rules • Frame problem - Specifying the non-effects • Assumes 100% confidence in events detected • Space is not a first class citizen. • Does not deal well with heterogeneous data 27
- 28. /125 Situations • Multiple definitions • Situation awareness • Situation modeling • Situation detection • Situation calculus • Context based computing “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future (Endsley, 1988)”. “knowing what is going on so you can figure out what to do” (Adam, 1993)”. “the complete state of the universe at an instant of time” (McCarthy, 1969) “a set of past contexts and/or actions of individual devices relevant to future device actions” ” (Wang,2004)”. “…extensive information about the environment to be collected from all sensors independent of their interface technology. Data is transformed into abstract symbols. A combination of symbols leads to representation of current situations…which can be detected”(Dietrich, 2003) “A situation is a set of contexts in the application over a period of time that affects future system behavior” (Yau, 2006)
- 29. /125 Situations: commonalities 29 • Goal Based • Space-Time • Future Actions • Abstraction • Computationally Grounded Work Goal Based Space-Time Future Actions Abstraction Computationally Grounded McCarthy, 1968 X Barwise, 1971 X X Endsley, 1988 X X X X Sarter, 1991 o X Adam, 1993 X X Dominguez,1994 X X X X Smith, 1995 X o X X Steinberg, 1999 X X X o Jeannot, 2003 X Moray, 2004 o X Dietrich, 2004 X X Yau, 2006 X X X Dostal, 2007 o X Singh, 2009 X X X Merriam-Webster (accessed 2012) o This work (aim) X X X X X o = Partial support
- 30. /125 Situation: Definition • Situation: An actionable abstraction of observed spatio-temporal characteristics. • e.g. flu epidemic, severe asthma threat, road congestion, wildfire, flash-mob 30 Goal Based Space-Time Future Actions Abstraction Computationally Grounded
- 31. /125 Overall Framework: Motivating example 31 STT data Tweet: ‘Urrgh… sinus’ Loc: NYC, Date: 3rd Jun, 2011 Theme: Allergy Situation Detection User-Feedback ‘Please visit nearest CDC center at 4th St immediately’ Date: 3rd Jun, 2011 Aggregation, 1) Classification 2) Control action Operations Alert level = High
- 32. /125 Applications • Healthcare • Alert me if there is a flu epidemic in my area • Telepresence: • Which camera feed to send out? • Business analysis: • Where is the most suitable place to open a new ‘iphone’ store ? • Weather • Alert me when the fall colors blossom in New England? • Daily living: • Which place (and at what time) is conducive for exercising? • Weather, climate, politics, traffic, … 32
- 33. /125 SITUATION RECOGNITION FRAMEWORK (45 mins, Vivek Singh) 33
- 34. /125 A) Situation Modeling B) Situation Recognition C) Visualization, Personalization, and Alerts … STT Stream Emage Situation C1 v2 v3 v5 v6 @ ∏ Δ @ i) Visualization ii) Personalization + + Available resources iii) Alerts Personal context Personal ized situation Overall framework 34
- 35. /125 A) Situation Modeling • Help domain experts externalize their internal models of situations of interest e.g. epidemic. • Building blocks: • Operators • Operands • Wizard: • A prescriptive approach for modeling situations using the operators and operands 35 Singh, Gao, Jain: Situation recognition: An evolving problem for heterogeneous dynamic big multimedia data, ACM Multimedia ‘12.
- 36. /125 Growth rate (Flu reports) Feature Thresholds (0, 50) Data source Meta-data -Emage (#Reports) Representation level Twitter-Flu Building Blocks: Operands 36 • Knowledge or data driven building blocks
- 37. /125 Building Blocks: Operators Δ Transform … Spatio-temporal window 37 Aggregate + Classification Classification method @ Characterization Growth Rate = 125% Property required Pattern Matching 72% + Pattern ∏ Select + Mask Φ Learn Learning method {Features} {Situation} f f 1) Data into right representation 2) Analyze data to derive features 3) Use features to evaluate situations Supporting parameter(s) Data OutputOperator Type
- 38. /125 Situation Modeling Get_components (Situation v){ 1) Identify output state space 2) Identify S-T bounds 3) Define component features: v=f(v1, …, vk) • If (type = imprecise) • identify learning data source, method 4) ForEach (feature vi) { If (atomic) • Identify Data source. • Type, URL, ST bounds • Identify highest Rep. level reqd. • Identify operations Else Get_components(vi) } } 38 v f1 v4 v2 v3 @ D1 Emage Δ D2 ∏ Emage Δ D3 Δ @ Emage D2 ∏ Emage Δ f2 v5 v6 <USA, 5 mins, 0.01x 0.01> ϵ { Low, Mid, High}
- 39. /125 Epidemic Outbreaks Unusual Activity? Growth Rate Current activity level Historical activity level Emage (#reports ILI) Δ Twitter-Flu Twitter.com <USA, 5 mins, 0.01x 0.01> Emage (Historical avg) Δ Twitter-Avg DB, <USA, 5 mins, 0.01x 0.01> Δ Twitter-Flu Emage (#reports ILI) Twitter.com <USA, 5 mins, 0.01x 0.01> ϵ {Low, mid, high}, <USA, 5 mins, 0.01x 0.01> Growing Unusual activity 1)Model Emage (#reports ILI) Δ Twitter-Flu Emage (population) Δ CSV- Population π Twitter.com <USA, 5 mins, 0.01x 0.01> Census.gov, <USA, 5 mins, 0.01x 0.01> 2) Revise Subtract Subtract Multiply Classification: Thresh (30,70) Normalize [0,100] 3) Instantiate 39
- 40. /125 Level 1: Unified representation (STT Data) Level 3: Symbolic rep. (Situations) Properties Properties Properties Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, … Level 2: Aggregation (Emage) … STT Stream Emage Situation B) Situation evaluation: Workflow 40 Operations
- 41. /125 Data Representation • E-mage • Visualization • Spatio temporal data representation • Data analysis using media processing operators (e.g. segmentation, background subtraction, convolution) 41
- 42. /125 Data Model • Spatio-temporal element • stel = [s-t-coord, theme(s), value(s), pointer(s)] • E-mage • g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N) • Temporal E-mage Set • TES= {(t1, g1), ..., (tn, gn)}, • Temporal Pixel Set • TPS = {(t1, p1), ..., (tn, pn)},
- 43. /125 Situation Recognition Algebra Singh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10. 43 S. No Operator Input Output 1 Filter ∏ Temporal E-mage Stream Temporal E-mage Stream 2 Aggregation K*Temporal E-mage Stream Temporal E-mage Stream 3 Classification Temporal E-mage Stream Temporal E-mage Stream 4 Characterization : @ Spatial Temporal Temporal E-mage Stream Temporal Pixel Stream Temporal Pixel Stream Temporal Pixel Stream 5 Pattern Matching Spatial Temporal Temporal E-mage Stream Temporal Pixel Stream Temporal Pixel Stream Temporal Pixel Stream
- 45. /125 Implementation and results • Twitter feeds • Geo-coding user home location • Loops of location based queries for different terms • Over 100 million tweets using ‘Spritzer’ stream (since Jun 2009), and the higher rate ‘Gardenhose’ stream since Nov, 2009. • Flickr feeds • API • Tags, RGB values from >800K images
- 46. /125 Testing Data Representation + Algebra • Applications • Business analytics • Political event analytics • Seasonal characteristics • Data • Twitter feeds archive • Loops of location based queries for different terms • Over 100 million tweets using ‘Spritzer’/ ‘Gardenhose’ APIs • Flickr feeds • API: Tags, RGB values from >800K images • Implementation • Matlab + Java + Python 46
- 47. /125 Sample Queries • Select E-mages of USA for theme ‘Obama’. • ∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama) • Identify three clusters for each E-mage above. • kmeans(3) (∏spatial(region=[24,-125],[24,-65])(TEStheme=Obama)) • Show me the cluster with most interest in ‘Obama’. • ∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama))) • Show me the speed for high interest cluster in ‘Katrina’ emages • @speed(@epicenter(∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))) • How similar is pattern above to ‘exponential increase’? • exp-increase(@speed(@epicenter (∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina)))) 47
- 48. /12548 AT&T retail locations AT&T total catchment area iPhone theme based e-mage, Jun 2 Aggregate interest Under-served interest areas - Subtract Decision Best Location is at Geocode [39, - 122] , just north of Bay Area, CA @Spatial.Max <geoname> <name>College City</name> <lat>39.0057303</lat> <lng>-122.0094129</lng> <geonameId>5338600</geonameId> <countryCode>US</countryCode> <countryName>United States</countryName> <fcl>P</fcl> <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> <distance>1.0332</distance> </geoname> + Add to Jun 15, 2009 Convolution . * Store catchment area Convolution. * Store catchment area
- 49. /125 Flickr Social Emages • Jan – Dec 2009
- 50. /125 Seasonal characteristics analysis • Fall colors in New England • Show me the difference between red and green colors for New England region, as it varies throughout the year. • subtract(@spatial(sum)(πspatial(R=[(40,-76), (44,-71)]) (TEStheme=Red)), @spatial(sum)(πspatial(R=[(40,-76), (44,-71)])(TEStheme=Green))) 50 Jan 0 Dec
- 51. /125 Year average Peak of green At [35, -84], at the junction of Chattahoochee National Forest, Nantahala National Forest, Cherokee National Forest and Great Smoky Mountains National Park
- 53. /125 FraPPE: a vocabulary to represent heterogeneous spatio-temporal data to support visual analytics Marco Balduini, Emanuele Della Valle ISWC 2015 – Data Sets and Ontologies Slides adapted from: http://www.slideshare.net/MarcoBalduini/frappe-a-vocabulary-to-represent-heterogeneous-spatiotemporal-data-to-support-visual-analytics
- 54. /125 Proposed Approach 54 21/07/2016 Re-use consolidated concepts from • geo-spatial vocabularies • time related vocabularies • provenance vocabularies Model visual analytics concepts * • pixels • frame Fill the gap between heterogeneous geo-spatial time-varying data and visual analytics concepts to create actionable information and ease the decision making processes of final users * Singh, V.K., Gao, M., Jain, R.: Social pixels: genesis and evaluation (ACM MM 2010)
- 60. /125 FraPPE, visually 60 21/07/2016 Event B Place B Frame 2 Frame 1
- 61. /125 City Sensing listens to the pulse of Milano Design Week on April 9th, 2014 61
- 62. /125 Tweeting Cameras 62 Slides courtesy: Yuhui Wang, Francesco Gelli, and Mohan Kankanhalli Adapted from “Tweeting Cameras for Event Detection” in Proc. WWW Conference 2013.
- 63. /125 Physical & Social Sensors Fusion For Situation Awareness Physical Sensors Social Sensors
- 64. /125 Real-world Events Hispanic ParadeCBGB Musical Festival Columbus Day Parade twtw tw tw tw tw twtw tw tw tw tw twtw tw tw tw tw twtw tw tw tw tw Historic TweetsRecent Tweets Event time and location Retrieve recent tweets Retrieve historic tweets
- 65. /125 Probabilistic Spatio-Temporal Data • Definition (PST: Probabilistic Spatio-Temporal Data) The fundamental building block for low-level concept representation is the probabilistic spatio-temporal element “pst”. • pst = [loc, temp, label, prob, pointer] (1) where • loc = [lat, lon] represents the geo-location – latitude and longitude – of the camera location. • temp stores the time information of captured data. • label represents semantic concept such as car, human,crowd, parade, etc., detected in the stream. Generally, these concepts express low-level abstraction of information which could be semi-reliably detected by existing detectors or classifiers. • prob is the confidence value in [0,1] representing the output of a concept detector as a probability value. • pointer points to actual raw data stream. 65
- 66. /125 Physical Sensors (Concept: “Crowd”)
- 67. /125 Social Sensors (#MillionMarchNYC, #BlackLivesMatter)
- 69. /125 CMage 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Gaussian Process based Prediction Sensor Image Patch
- 70. /125 Fusing Sensor Cmage with Social Cmage Sensor Cmage (concept: “people marching”) Social Cmage (concept: “MillionsMarchNYC”) Fused Cmage
- 71. /125 DESIGNING SITUATION BASED APPLICATIONS (30 mins, Siripen Pongpaichet) 71
- 72. /125 Outline • EventShop System Requirements • EventShop System Architecture • Demo • Building Applications using EventShop • Conclusion 72
- 73. /125 EventShop Requirement 73 Granulari- ties Heterogen eous Model Prediction Users Open- Source Storage Generic Streams Support fast data flow Handle heterogeneous types of data streams Efficiently aggregate data at different granularities Provide storage system to archive both data input and system output. Create situation model and provide actionable information Generic computational platform for situation recognition Open-source software User friendly and interactive interface Contain predictive component
- 74. /125 EventShop Architecture 74 Alert/OutputData Ingestor Data Source Parser Data Adapter Emage Generator (+resolution mapper) Query Processing EventShop Storage Query Parser Query Rewriter Emage Stream Processing Action Parser Situation Emage Visualization (Dashboard) Actuator Communication Event Property & Other Information (e.g., spatio-temporal pattern) ᴨ ᴨ µ Data Access Manager Live Stream Archived Stream Situation Stream Physical Data Source (e.g., sensor streams, geo-image streams) Logical Data Source (e.g., preprocessing data streams, social media streams) Raw Event REST API Services Data Source / Query / Alerts / STT-Emage EventShop UI External AppsVisual Analytics InterfaceAPIProcessingLayerStorageLayer
- 75. /125 EventShop UI 75 Save Query Reset Query Create Query Pollen Tweets_ Asthma Available Available 350 AQI Available357 361 Grouping Stopped35 Asthma_ Risk Stopped36 Asthma_ Interpolate Stopped37 Asthma_ Interpolate Stopped37 Asthma_ Stopped37 Query Graph Filter Group Aggregation Spatial Pattern Temporal Pattern Spatial Char Temporal Char Redraw ds361 ds350 ds357 F1 Q1 F2 F3 Q2 Q3 A1 Q4 G1 Q5 http://eventshop.ics.uci.edu:8080/eventshoplinux/
- 78. /125 Building applications using EventShop S.No Application Data Used Application deployed? Scale Data modalities Operators used 1 Wildfire detection in California Real Yes Macro Satellite data, Google insights F, A, Ch 2 Hurricane monitoring Simulated No Macro n/a F, A, Ch, P 3 Flu epidemic surveillance Real No Macro Twitter, Census F, A, C 4 Allergy recommendation Real Yes Macro Twitter, Air Quality, Pollen Count F, A, C 5 Asthma management Real Yes Macro, Personalized alerts In situ sensors, Satellite data, Asthma Tracking F, I, Pr 6. Thailand flood mitigation Real Yes Macro, Personalized alerts KML F, A, C 7. Photos as Micro- Reports Real Yes Macro Flickr F, Cl 8. Trash management Real & Simulated In progress Macro Trash sensors, micro-reports F, A, Pr, 78
- 80. /125 Asthma Management Application 80 (1) Macro Situation Macro Data Streams (3) Situation- Action Rules Sensor streams Social media Geo-temporal data Personal Data Streams (2) Personal Situation Behavioral streams Profile + Preferences
- 81. /125 Asthma Risk Estimation 81 Traffic Flow Aerosol Concentration PM2.5 CMAQ Model PM2.5 Concentration *visualize data on Feb 12th, 2008 Mengfan Tang, Pranav Agrawal, Siripen Pongpaichet, Ramesh Jain: Geospatial interpolation analytics for data streams in EventShop. ICME 2015 Spectral Spatial Gaussian Process (SSGP)
- 82. /125 Experiment Results 82 Data Model PMSE MAPE Single Data Source CMAQ - 1.0619 27.2873 CMAQ LR 0.9586 27.1077 Stations Kriging 0.9077 22.9672 CMAQ SSGP 0.3468 14.2727 Multiple Data Sources ALL SGP 0.3006 13.5109 ALL SSGP 0.2858 13.1087 CMAQ Kriging SSGP
- 83. /125 Asthma Risk Estimator Model and Result 83 Asthma Hospitalization Ground Truth FILTER LOC=CA FILTER LOC=CA AGG FUNC=AVG GROUP THRESHOLD Asthma Risk Area without Interpolation GROUP THRESHOLD Asthma Risk Area with interpolation AGG FUNC=AVG PM2.5 Concentration From Stations Interpolated PM2.5 using SSGP Pollen Ozone AQI
- 84. /125 A GRAPH BASED MULTIMODAL GEOPATIAL INTERPOLATION FRAMEWORK Mengfan Tang1, Pranav Agrawal1, Feiping Nie2, Siripen Pongpaichet1, Ramesh Jain1 1University of California, Irvine, USA 2Northwestern Polytechnical University, China Tuesday July 12th, 2016 at 5PM Room: Cascade I, Special Session: Multimedia Cloud Computing and Big Data
- 86. /125 PHOTOS as Kodak Moments
- 87. /125 Disruption: PHOTOS as Information Smartphone camera captures EVENTS
- 88. /125 • Truthfulness, • Accuracy, • Objectivity, • Fairness and Public accountability Reports of Events from Journalists Seek Truth and Report it as Fully as Possible
- 89. /125 Reports of Events from Citizens
- 90. /125 FASTSubjective EASY Noisy LATES T Ambiguous were SO yesterday!Micro-Blogs Multimedia Micro-Reports
- 91. /125 Compelling Universal Objective Spontaneous Multimedia Micro-Reports (MMRs) are now and future
- 92. /125 Capturing and Reporting events using multimedia such as photos, videos, sensors, and texts Converting multimedia data to multimedia micro-reports using MediaJSON Integrating multimedia micro-reports with other data sources for situation recognition, trend analysis, and culture analytics. Emerging opportunities for numerous apps: smart city, public health, emergency rescue What are the challenges?
- 93. /125 Capturing and Reporting Events with Krumbs SDK https://krumbs.net/ What: Objects Who: People When: Events Where: Location Why: Intent/Emotions How: Photo and audio
- 94. /125 Capturing and Reporting Events with Krumbs SDK https://krumbs.net/
- 96. /125 Converting Multimedia Data into MMR {"micro_reports":[{ "where":{ "geo_location":{ "latitude":32.90233332316081, "longitude":-117.2441166718801}, "when":{ "start_time":"Jun 14, 2009 11:25:19 AM", "end_time":"Jun 14, 2009 11:25:19 AM", "time_zone":"America/Los_Angeles"}, "what":[{ "concept_name":"people", "confidence":0.9836078882217407, "visual_concept_provider":"CLARIFAI"}, … { "concept_name":"food", "confidence":0.8526291847229004, "visual_concept_provider":"CLARIFAI"}] , "tag":”#niceday #summer", "source":{"default_src":"https://….jpg"}}, "sub_event":[], "why":[]}, …]} Photo What Where When Who Why Sound MediaJSON
- 97. /125 MediaJSON Data Wrapper Data Wrapper Data Wrapper Data Wrapper IoT (Event-driven operation) Converting Multimedia Data into MMR MediaJSON Data Wrapper
- 98. /125 Number of photos in London per day
- 99. /125 Evolving Photo Concepts in London
- 100. /125 • Year of interest: year 2008 and 2012 • Training locations: Beijing (China) • Testing location: London (UK) • Create “Olympic Games” Event Model from “BAG of Visual Concepts” Detecting Olympic Games Olympic Games = {basketball, court game, gymnastics, people, sport, stadium, swim, tennis}
- 101. /125 Evolving Photo Concepts in Beijing Event model of “Olympic Game”
- 102. /125 Detecting London Olympic Games Summer Olympic Game in July and August Paralympic Games in September
- 103. /125 • Temporal range: 1 year from July 2011 to June 2012 • Location: Thailand Detecting Emergency Situations City flood = {outdoor, water, road, car}
- 104. /125 Photos from City Flood Cluster November 5, 2011
- 105. /125 Smart City Project in DC 105 U.S. Presidential Inauguration in DC Earth Day Concert Cherry Blossom Festival
- 106. /125 Integrating MMR with other data sources for Situation Recognition (In progress) h t t p : / / s m a r t c i t i e s i n n o v a t i o n . c o m /
- 107. /125 Trash Fill Level Situation in DC 107
- 108. /125 0 35 50 90 0 20 40 60 80 100 7:30 8:00 8:30 9:00 9:30 10:00 10:30 Real-Time Fill Level Situations at a given location of an event Prediction based on Events History Events Data Real-Time Trash Fill Level Situation 20 42 Now Predicted Trash Fill Level in 30 minutes at a given location 78 99 30 minutes 10 30 40 70 90 100 20 0 20 40 60 80 100 120 7:30 8:00 8:30 9:00 9:30 10:00 10:30 Projected Trash Fill Level at a given location based on Event History
- 109. /125 Conclusion • EventShop Architecture • Situation-based Applications using EventShop • Online Service: http://sln.ics.uci.edu:8085/eventshoplinux • Open Source: http://dabuntu.github.io/es/ • For more information about EventShop including tutorial videos, presentations and publications, please visit my home page http://www.ics.uci.edu/~spongpai Siripen Pongpaichet (spongpai@uci.edu) 109
- 110. /125 FUTURE TRENDS AND OPEN PROBLEMS (20 mins, Ramesh Jain) 110
- 111. /125 Future Trends • Future trends • Open problems for Multimedia research 111
- 112. /125 This century is different from the last. Should we think differently???
- 113. /125In 20th century, we tolerated photos in our textual documents. In 21st century, you create visual documents that tolerate text.
- 114. /125 Major Disruption in Photos: From Memories to Information Sources. Photos are the most compelling source of information.
- 115. /125115
- 116. /125 We are immersed in Big Data. Multimedia Realtime Uncertainty 116
- 117. /125117
- 118. /125 Data as a Platform. • Multi-modal • Multimedia has to become multimodal • Data Streams • Important things – Situation recognition • Real time action for 118 Connecting People to Resources effectively, efficiently, and promptly in given situations.
- 119. /125 Contact Information • Vivek Singh, Rutgers University • Vivek.k.singh@Rutgers.edu • • Siripen Pongpaichet • spongpai@ics.uci.edu • Ramesh Jain • jain@ics.uci.edu 119
- 120. /125 Useful links • Copies of publications • http://wp.comminfo.rutgers.edu/vsingh/publication/ • Today’s Slides: • https://dl.dropboxusercontent.com/u/5887580/Tutorial_SituationRecog. pdf • EventShop • Online Service: http://sln.ics.uci.edu:8085/eventshoplinux • Open Source: http://dabuntu.github.io/es/ • For more information about EventShop including tutorial videos, presentations and publications: http://www.ics.uci.edu/~spongpai • Related Projects • Tweeting Cameras: https://sites.google.com/site/fredyuhuiwang/ • Frappe: http://www.streamreasoning.org/live/festivalcomunicazione2014/ 120
Editor's Notes
- Bring Predictive here.
- A real part of the world is represented by the map in the figure. The data geo-spatial time-varying is too complex to be used for a visualization.
- A GRID, composed by 4 CELLS, is placed between the map and the film that will capture the actual reflection of the reality. The GRID is needed to aggregate the date over space
- A film capture a FRAME, made by 4 PIXEL, that represents the temporal representation of the reality Using the PIXEL and the FRAME FraPPE aggregates the data over time.
- Once something happen in the PLACE A (geo-laceted place in a CELL and in a GRID) the reality, for example the pick-up event of a taxi ride, the related EVENT is impressed over the film in the corresponding PIXEL and FRAME.
- The reality is mediated again with the 4X4 GRID
- A new FRAME is impressed on the film, representing a new reflection of the reality with a new EVENT that is happening in PLACE B and I represented on the FRAME by the EVENT b
- We are trying to fuse physical sensors (e.g. CCTV cameras with social sensors (Twitter tweets ) together to have a better understanding of ongoing situations
- Red dots are fixed camera at different locations in Manhattan, tweeting about different concept (expressed by color intensity, redder means high confidence of the concept) world clouds are generated by mining tweets content using tf-idf (we put geo tagged human tweets round a camera location during event time of event day into one document, and tweets of same location, same time span but past days into other documents; words with high tfidf mean they are discussed more frequently during event hours of event days then same hours but past days.)
- Red dots are fixed camera at different locations in Manhattan, tweeting about crowdedness (expressed by color intensity, redder means more crowded)
- Green dots are human tweets containing hashtags (#MillionMarchNYC, #BlackLivesMatter) at different locations in Manhattan. Left top is the their marching route ground truth.
- Red dots are fixed camera at different locations in Manhattan, tweeting about crowdedness (expressed by color intensity, redder means more crowded) Green dots are human tweets containing hashtags (#MillionMarchNYC, #BlackLivesMatter) at different locations in Manhattan. Left top is the their marching route ground truth. Combining camera tweets with human tweets, world clouds are generated by mining tweets content using tf-idf (we put geo tagged human tweets round a camera location during event time of event day into one document, and tweets of same location, same time span but past days into other documents; words with high tfidf mean they are discussed more frequently during event hours of event days then same hours but past days.) As can be seen, though cameras at different places tweet about “crowdedness”, they mean different events (mid-Manhattan is “Santacon Event” while lower Manhattan is “MillionMarchNYC” protest event )
- From camera feeds to concept image (Cmage) : Gaussian Process could be applied on such data representation to infer situations in place without sensors
- Fusing sensor cmage with social cmage for “MillionsMarchNYC” protest events Sensor cmage intensity represents confidence values of “people marching” concept extracted from CCTV camera images Social cmage intensity is tf-idf values of term “MillionMarchNYC” calculated from Twitter tweets In fused image, noise has been removed and saliency of events has been enhanced
- Support fast data flow Handle heterogeneous types of data streams Efficiently aggregate data at different granularities Provide storage system to archive both data input and system output. Create situation model and provide actionable information Contain predictive component Generic computational platform for situation recognition Open-source software User friendly and interactive interface
- Traffic: loop detectors & traffic monitoring sparse in space, dense in time – from caltrans performance measurement system Aerosol Concentratation: satellite data, sparse in space and time – from MODIS Aerosol optical depth PM2.5 CMAQ: Chemical transportation model, sparse in space and time – from community modeling and analysis system PM2.5 Concentration station: air measurement stations, limited # of stations – US EPA (environmental projecting agency) air quality system
- multimedia research on object recognition, scene recognition, event recognition - on photos summarization, Browse through photos on different dimension. Space-time-people-event..
- We Capture and Share Moments -- Really Events What are these natures of photos lease to new opportunities. Trillion photos have been uploaded last year. 7 billions people in the world. 150 photos have been uploaded by individual person.
- Now. Let’s talk about REPORTS! For century, in journalism philosophy, the reports MUST have these characteristics. Truthfulness, accuracy, objectivity, and fairness. All in all, the goal is to “seek the truth and report it as fully as possible” without reporters’ own opinion.
- How people report the events? Through all these social media; twitter, facebook, snapchat, instagram. For example, during ucla shooting last week, people tweeted about it, took some pictures and videos as they were witnessing that situations.
- Pro Became very powerful very fast Easy to use – like SMS. Aggregated for trend analysis, analytics, visualization of evolving situations. Con Missing/misleading context – ambiguous – poor context – lots of research on that sentiment analysis Noise – limited number of character, all topics are mixed together, Hashtag # for addressing a theme, however, this doesn’t solve the problem. Subjective These disadvantages violate all journalism philosophy !!! Great concept but has limitations. New technology can overcome these limitations. We need a new mechanism to handle this new type of report – micro-report What is the successful story of focused micro-reports- WAZE This multimedia micro-reports concept is inspired by Focused MicroBlogs first introduced by Ramesh Jain. http://ngs.ics.uci.edu/focused-microblogs-fmbs-going-beyond-twitter/
- Compelling (context around the photos) Universal (independent of Language) Objective (not opinion) Spontaneous (share immediately) ---- Compelling: Micro-reports should effortlessly capture the moment with rich context along with compelling photo, audio, video and/or other signals. As are well known photos, videos, and audio which make a report of an experience more compelling and more informative. Universal: Micro-reports need to have a universal language, that is beyond literacy and geographical boundaries. Photos are universal language. Objective: Micro-reports should emphasize on facts and not subjective opinions of people and their personal atti- tude toward the facts. Opinions are important, but should appear explicitly as opinions. Objective and subjective components in a report should be clear. Spontaneous: Micro-reports should be shareable immediately. If creating a report requires thinking and typing a message, fewer people will use it during the event. A report submitted later relies on memory and may not have correct contextual information.
- Event-driven operation!!! Reports sent by IoTs
- For each day, the probabilities for the same concept is aggregated. Figure 6 gives an example of how four concepts, people, running, sport and swim evolve during the time of interest. Let’s first look at the dynamic change in the ‘people’ concept, we observe that London is less crowded in winter season and attracts more people in summer and autumn. London usually hosts many kinds of sport events, such as swimming, and running, which also can been detected in this figure. However, single data stream (‘people’ theme) finds itself hard to detect situations. Social events and trends are usually detected by the concurrences of photo’s visual concepts.
- bounding box of London (-0.489,51.28 0.236,51.686)
- Figure 5 shows the evolving concepts in Beijing in the year of 2008. As we know, Beijing Olympic Games was held in Beijing in the summer of 2008. This event is clearly reflected in Figure 5 where the number of these ten concepts ‘basketball’,‘court game’,‘gymnastics’,‘people’,‘sport’,‘stadium’,‘swim’ and ‘tennis’ reaches peak at the same time of Olympic Games.
- Summer Olympics in July and August and Paralympic Games in September in
- From the historical data and human knowledge, these concepts, ‘outdoor’, ‘wa- ter’, ‘road’, and ‘car’, are highly occurred together during a city flood
- Photos from one of the cluster in Thailand on November 5, 2011. The length of edges in the graph represents a similarity between a flood event model (rectangle) and a photo (circle) considering its visual concepts. The shorter length means the higher similarity. Most of the photos related to flood event can be captured by our framework.
- Prob/Opportunity Slide
- Prob/Opportunity Slide