Vivek Singh, profile picture
Uploaded byVivek Singh
PPTX, PDF1,723 views

Thesis personalized situation recognition

The document discusses the framework for personalized situation recognition, focusing on the computational definition of situations from multimedia data and the development of a generic process for situation modeling and evaluation. It highlights contributions such as the creation of web-based systems for situation detection and personalized alerts, emphasizing the use of social media as human sensors to gather real-time data. The work aims to facilitate sense-making from geo-social data streams through advanced algorithms and event recognition techniques.

Embed presentation

Downloaded 32 times
/ 46 PERSONALIZED SITUATION RECOGNITION Vivek K. Singh Information Systems Group, University of California, Irvine Advised by: Professor Ramesh Jain
2 / 46 Trends SOCIAL GEO- SOCIAL
3 / 46 Trends PLANETARY SCALE
4 / 46 Trends SENSE MAKING
5 / 46 The Big Challenge Sense making from planetary scale geo-social data-streams Situation recognition
6 / 46 Concept recognition from multimedia data Heterogeneous Media Heterogeneous Media Single Media Location Scenes Environ Trajectories Situations mentsK 3.4 Single Media aware Location Visual Real world Visual 360 K 11.4K Activities unaware Objects Objects Events Static Dynamic SPACE TIME
7 / 46 Contributions 1. Computationally define situations 2. Define a generic process for Situation recognition a) Situation Modeling b) Situation Evaluation: • E-mage + Situation Recognition Algebra c) Personalized Alerts 3. EventShop: Web-based system for situation evaluation
8 / 46 Situations: Other definitions • Endsley, 1988: “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” • Merriam-Webster dictionary: “relative position or combination of circumstances at a certain moment” • McCarthy, 1969: “A situation is a finite sequence of actions.” • Yau, 2006: “A situation is a set of contexts in the application over a period of time that affects future system behavior” • Dietrich, 2003: “…extensive information about the environment to be collected from all sensors independent of their interface technology. Data is transformed into abstract symbols. A combination of symbols leads to representation of current situations…which can be detected”
9 / 46 Situations: commonalities • Goal Based • Abstraction • Space-Time • Computationally • Future Actions Grounded Future Computationally Work Goal Based Space-Time Abstraction Actions Grounded McCarthy, 1968 X Barwise, 1971 X X Endsley, 1988 X X X X Sarter, 1991 o X Adam, 1993 X X Dominguez,1994 X X X X Smith, 1995 X o X X Steinberg, 1999 X X X o Jeannot, 2003 X Moray, 2004 o X Dietrich, 2004 X X Yau, 2006 X X X Dostal, 2007 o X Singh, 2009 X X X Merriam-Webster (accessed 2012) o This work (aim) X X X X X o = Partial support
10 / 46 Situation: Definition • Situation: An actionable abstraction of observed spatio-temporal characteristics. • e.g. flu epidemic, severe asthma threat, road congestion, wildfire, flash-mob Future Computationally Goal Based Space-Time Abstraction Actions Grounded
11 / 46 Overall Framework: Motivating example Aggregation, Operations Alert level = High Date: 3rd Jun, 2011 STT data Situation Detection User-Feedback Tweet: 1) Classification ‘Please visit nearest CDC ‘Urrgh… sinus’ 2) Control action center at 4th St immediately’ Loc: NYC, Date: 3rd Jun, 2011 Theme: Allergy
12 / 46 Eco-system: Situation based applications Human Sensor/ Wisdom source App logic Analyst Analysis & insights Spatio- Macro Device Situation Temporal situation Situation Sensors detection aggregation based operators controller Personal situation Archives Control decisions Event processing engine Human Sensor/ Alerts Actuator Singh, Jain: Situation based control. (Best Student Paper) IEEE Situation Management Workshop’09 Singh, Kankanhalli, Jain: Motivating contributors. (Best Paper) ACM Workshop on Social Media ’09
Overall framework 13 / 46 A) Situation B) Situation C) Visualization, Modeling Recognition Personalization, and Alerts i) Visualization C1 …  v2 v3 Personal @  context ii) Personalization Personal v5 v6 + ized STT situation ∏ @ Δ Stream Available resources + Emage iii) Alerts Situation
14 / 46 Design principles • Humans as sensors • Space + Time as fundamental axes • Real time situation evaluation (E-mage Streams) (a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream) d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)
15 / 46 A) Situation Modeling • Help domain experts externalize their internal models of situations of interest e.g. epidemic. • Building blocks: • Operators • Operands • Wizard: • A prescriptive approach for modeling situations using the operators and operands Singh, Gao, Jain: Situation recognition: An evolving problem for heterogeneous dynamic big multimedia data, ACM Multimedia ‘12.
16 / 46 Building Blocks: Operands • Knowledge or data driven building blocks Growth rate (Flu reports) Feature Twitter-Flu Data source -Emage (#Reports) Representation level Thresholds (0, 50) Meta-data
17 / 46 Building Blocks: Operators Supporting Operator Type Data parameter(s) Output 1) Data into right representation Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate + 2) Analyze data to  Classification derive features Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + Pattern 72% {Features} 3) Use features to Φ Learn f Learning method f evaluate situations {Situation}
18 / 46 Situation Modeling v v ϵ { Low, Mid, High} f1  <USA, 5 mins, 0.01x 0.01> v2 v3 v4 v=f(v1, …, vk) @ f2  ∏ • If (type = imprecise) • identify learning data source, method Emage v5 v6 Emage vi ∏ @ Δ Δ If (atomic) Emage • Identify Data source. D1 Emage D2 Δ Δ • Type, URL, ST bounds • Identify highest Rep. level reqd. D2 D3 • Identify operations Else Get_components(vi) } }
19 / 46 3) Instantiate 2) Revise 1)Model Epidemic ϵ {Low, mid, high}, Outbreaks <USA, 5 mins, 0.01x Classification:  0.01> Thresh (30,70) Growing Unusual activity  Multiply Unusual Activity? Growth Rate  Subtract Historical Current activity Emage activity level level (#reports ILI)  Subtract Δ Normalize Emage π [0,100] (Historical avg) Emage Twitter-Flu Δ Emage Emage (#reports ILI) ILI) (#reports (population) Twitter.com Δ Δ <USA, 5 mins, Twitter-Avg Δ 0.01x 0.01> DB, Twitter.com Twitter-Flu CSV- Census.gov, Twitter.com Twitter-Flu <USA, 5 mins, <USA, 5 mins, Population <USA, 5 mins, <USA, 5 mins, 0.01x 0.01> 0.01x 0.01> 0.01x 0.01> 0.01x 0.01>
20 / 46 B) Situation evaluation: Workflow Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, … … Level 1: Unified representation Properties (STT Data) STT Stream Level 2: Aggregation Properties Emage (Emage) Operations Level 3: Symbolic rep. Properties Situation (Situations)
21 / 46 Data Representation • E-mage • Visualization • Spatio temporal data representation • Data analysis using media processing operators (e.g. segmentation, background subtraction, convolution)
22 / 46 Data Representation • Spatio-temporal element • STTPoint = {s-t-coord, theme, value, pointer} • E-mage • g = (theme, x, v(x) | x ϵ X = R2 , and v(x) ϵ V = N) • Temporal E-mage Stream • TES=((t0, g0), ..., (tk, gk), …) • Temporal Pixel Stream • TPS = ((t0, p0), ..., (tk, pk), …)
23 / 46 Situation Recognition Algebra Supporting Operator Type Data parameter(s) Output ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
24 / 46 Situation Recognition Algebra S. No Operator Input Output 1 Filter ∏ Temporal E-mage Stream Temporal E-mage Stream 2 Aggregation  K*Temporal E-mage Stream Temporal E-mage Stream 3 Classification  Temporal E-mage Stream Temporal E-mage Stream 4 Characterization : @  Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel Stream 5 Pattern Matching   Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel Stream Singh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10.
25 / 46 Sample Queries • Select E-mages of USA for theme ‘Obama’. • ∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama) • Identify three clusters for each E-mage above. • kmeans(3) (∏spatial(region=[24,-125],[24,-65])(TEStheme=Obama)) • Show me the cluster with most interest in ‘Obama’. • ∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama))) • Show me the speed for high interest cluster in ‘Katrina’ emages • @speed(@epicenter(∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))) • How similar is pattern above to ‘exponential increase’? • exp-increase(@speed(@epicenter (∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))
26 / 46 C) Personalization and Alerts Personalized situation: An actionable integration of a user's personal context with surrounding spatiotemporal situation. 1) Macro situation Macro 2) Personal data-sources Context Personalized situation Profile + 3) Preferences Personalized alerts User Available data resources Resource data IF user Ui <is-in> (PSj) THEN <connect-to> Rk
27 / 46 Personalized Situation Recognition: Operators Supporting Operator Type Data parameter(s) Output ∏ Filter … + User location  Aggregate … + … …  Classification Classification … method … @ Characterization … Property Growth Rate required = 125%  Pattern Matching … + Match= 42% Pattern
28 / 46 Situation Action Rules •
29 / 46 EVENTSHOP: Recognizing situations from web streams
30 / 46 EventShop: System Implementation • Front end: • Javascript (JSLinb library) • Front-Back end Interaction • Java servlets, Apache • Back End • Java • C++ (OpenCV classes) • Ingestion wrappers available for: • Twitter streams, Flickr stream, CSV data, KML data, Geo-images, MySQL data archives, Funf (mobile phone sensors) Gao, Singh, Jain: EventShop: From Heterogeneous Web Streams to Personalized Situation Detection and Control, ACM WebScience ‘12.
31 / 46 S.No Query Language Operator Media processing Media processing Operator Details Translation into Operator Media processing 1. Filter -Spatial Arithmetic AND with the spatial mask operators -Temporal Arithmetic AND with the temporal mark -Thematic Arithmetic = -Value Arithmetic AND, >, <, = 2. Aggregation -Max, Min, +,-,%,* Arithmetic Max, Min, +,-,%,* - NOT, OR, AND, Logical NOT, OR, AND -Convolution Convolution Convolution 3. Classification - Predefined segments count Segmentation K-means - Predefined segment boundaries Segmentation thresholds 4. Characterization i) Spatial - Count, Min, Max, Sum, Average, Variation Statistical Count, Min, Max, Sum, Average - Coverage Arithmetic Count - Epicenter Arithmetic Weighted average - Circularity Convolution Scale free convolution with known circular kernel - Growth rate Arithmetic +, -, % ii) Temporal - Displacement, Distance, Velocity, Arithmetic +, -, %, * Acceleration, Growth rate - Future estimation Arithmetic Multiplication with Kernels based on users choice e.g. linear, progression exponential growth - Periodicity Convolution Auto correlation i.e. Self convolution with time-lagged variant. 5. Pattern Matching - Scaled Matching Convolution Convolution with user defined or pre-defined Kernels - Scale free Matching Convolution, Statistical Maxima from Loops of Convolution with different image sizes.
32 / 46 Evaluations 1. Design principles • Humans as sensors to detect real world events 2. Data representation and Situation recognition algebra • Expressive, computable and explicit • Real world results 3. Framework for situation recognition • modeling, • situation evaluation, • personalized alerts
33 / 46 Humans as sensors • Can social media be used to detect real world events? Observed Observed S.No Category Event Physical Date Physical Location Temporal Peak Spatial Peak 38.89, -77.03 1 Politics Health Care Bill passed 2010-03-21 2010-03-21 41, -74 (Washington) 37.77, -122.41 2 Politics California Prop 8, Trial Day 1 2010-01-11 2010-01-11 38,-122 (San Francisco) 31.13, -97.78 3 Society Fort Hood Shootings 2009-11-05 2009-11-05 33,-97 (Fort Hood, TX) 28.54, -81.38 4 Society SeaWorld Whale Accident 2010-02-12 2010-02-12 29,-81 (Orlando, FL) Winter Olympics Opening 49.24, -123.11 5 Sports 2010-02-12 2010-02-12 44,-79 ceremony (Vancouver) 40.71, -74.00 6 Sports Baseball World Series final 2009-11-04 2009-11-04 41, -74 (New York) 34.05, -118.24 7 Entertainment Oscars 2010-03-07 2010-03-07 34, -118 (Los Angeles) 2010-03-12 to 30.26, -97.74 8 Entertainment South by Southwest festival 2010-03-15 30, -98 2010-03-21 (Austin, TX) 2010-01-05 to 36.17, -115.13 9 Tech. Conv. CES 2010 2010-01-06 34,-118 2010-01-07 (Las Vegas) 2010-02-10 to 33.76, -118.19 10 Tech. Conv. TED 2010 2010-01-10 34, -118 2010-02-13 (Long Beach,CA)
34 / 46 Data representation + Algebra • Applications • Business analytics • Political event analytics • Seasonal characteristics • Data • Twitter feeds archive • Loops of location based queries for different terms • Over 100 million tweets using ‘Spritzer’/ ‘Gardenhose’ APIs • Flickr feeds • API: Tags, RGB values from >800K images • Implementation • Matlab + Java + Python
35 / 46 iPhone theme AT&T based e-mage, retail Jun 2 to Jun 15, 2009 locations . Convolution Store + Add * catchment area Aggregate Subtract AT&T total interest - catchment area <geoname> Convolution . @Spatial.Max Decision <name>College City</name> <lat>39.0057303</lat> <lng>-122.0094129</lng> Best Location is at <geonameId>5338600</geonameId> * <countryCode>US</countryCode> Geocode [39, - <countryName>United States</countryName> 122] , just north of <fcl>P</fcl> Bay Area, CA <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> Under-served <distance>1.0332</distance> interest areas Store catchment </geoname> area
36 / 46 Seasonal characteristics analysis • Fall colors in New England • Show me the difference between red and green colors for New England region, as it varies throughout the year. • subtract(@spatial(sum)(πspatial(R=[(40,-76), (44,-71)]) (TEStheme=Red)), @spatial(sum)(πspatial(R=[(40,-76), (44,-71)])(TEStheme=Green))) 0 Jan Dec
37 / 46 Building applications using the framework Application Data Operators S.No Application Data Used Scale deployed? modalities used Wildfire detection in Satellite data, 1 Real Yes Macro F, A, Ch California Google insights 2 Hurricane monitoring Simulated No Macro n/a F, A, Ch, P Flu epidemic 3 Real No Macro Twitter, Census F, A, C surveillance Macro, Twitter, Air Allergy/ Asthma 4 Real In-progress Personalized Quality, Pollen F, A, C recommendation alerts Count Macro, Thailand flood 5 Real Yes Personalized KML F, A, C mitigation alerts Legend: F = Filter, A = Aggregate, C = Classification, Ch = Characterization, P = Pattern Matching
38 / 46 Wildfire recognition model (Satellite data) Fire detector ϵ {fire, non-fire}, (Satellite driven) <California, 24hrs, 0.01x 0.01>  AND Significant band Unclouded? Hot enough? variation? Thresh Thresh  AND =392 =310 Emage (12 µm band Emage (Mid IR Absolute value Spatial Neighbor temp.) surface temp.) variation variation Δ Δ Thresh= 30 ∏ ∏ Thresh= 5 Satellite Satellite Band 4 Spatial Neighborhood Band 12 Difference value Difference LAADS.com,   LAADS.com, <California, 24hrs, Subtract <California, 24hrs, Subtract 0.01x 0.01> 0.01x 0.01> Difference Neighborhood Emage (4 µm Emage (11µm value Mean value temperature) temperature)  Convolve (7X7) Satellite Satellite Band 12 Band 12 Difference value LAADS.com, LAADS.com, <California, 24hrs, <California, 24hrs, 0.01x 0.01> 0.01x 0.01>
39 / 46 Wildfire recognition model (Social data) Fire detector ϵ {fire, non-fire}, (Social) <California, 24hrs,  0.01x 0.01> And Spatially anomalous Temporally anomalous ∏ ∏ Thresh=7 Thresh= 5 Difference with other Difference with Historical areas today average   Subtract Subtract Emage (Google Spatial Avg. of Emage (Google Emage (Google Insights- Fire) Interest Insights- Fire) Insights- Historical Avg) Δ  Δ Δ Average Google Emage (Google Google Google Insights-Fire Insights- Fire) Insights-Fire Insights-Fire Δ Google.com/insights, Google.com/insights, Google.com/insights, <California, 24hrs, <California, 24hrs, <California, 24hrs, Metros> Google Metros> Metros> Insights-Fire Google.com/insights, <California, 24hrs, Metros>
40 / 46 Wildfire recognition Fire detector ϵ {fire, non-fire}, Situation <California, 24hrs, 0.01x 0.01>  OR Modeling Fire detector Fire detector (Social) (Satellite) Situation Evaluation 50 45 40 35 30 Social detector Results 25 20 Satellite detector 15 Combined 10 Ground truth 5 Number of 0 Wildfires detected 2010 2011 Total
41 / 46 Demo: Asthma Recommendation Application
42 / 46 Thailand Flood mitigation
43 / 46 Social Life Networks Connecting People and Resources Situation aware routing Information Aggregation Situation and Detection Alerts Composition Queries Jain, Singh, Gao: Social Life Networks for the Middle of the Pyramid, ACM 43 Workshop on Social Media Engagement ‘11.
44 / 46 Related Work: Snapshot Area Combine Human Data Define Location Real-time Toolkits hetero sensors analytics situations aware streams data Situation X X o o X Awareness X Situation X Calculus X Web data o X X o X mining X X Social media o X X o X mining X X X Multimedia X o o o Event detection X Complex event X X o X processing/ X X Active DB GIS XX o X X XX o Mashup toolkits X X o X X (Y! pipes, ifttt) X This work X X X X X X X o = partial support
45 / 46 Future work • EventShop: • Personalization • Scalability • Prediction • Using such tools to nudge people into taking desired actions • Supporting Grids and Graphs for analysis • Social Life Networks
46 / 46 Summary • Personalized Actionable Situations • 1st Systematic approach • Situation Modeling • EventShop: Web based system for Situation Evaluation • Apps: Democratize data and action taking • Eco-system for data-to-action
/ 46 THANKS !
48 / 46 BACKUP SLIDES
49 / 46 Analyzing Big Data Field/ Approach Databases Networks Spatio-temporal Data structure Tables Graphs Grids Apps Business records, Internet traffic, Healthcare, Disaster Banking Social network, relief, Business, Roads Security Problems Querying, Searching Shortest path, Situation detection influence, anomaly Operators Select, Project, Join Diameter, influence Select, Aggregate, detection, connected ST characterization, components ST pattern matching, Classification Modeling ER modeling, Network diagrams, Situation models Query plan PetriNets Tools SQL server, Oracle NS2, NetworkX EventShop DBMS
50 / 46 Geo-Social Power Laws • Studied 5.6 Million Tweets for a month • There is a fixed relative ratio for the occurrence of events of different magnitude across space or time. Across Space Across Time Whole world Only USA 1 month 1 week Around New York 1 day 3 weeks city 30 mins 2 weeks Log(Rank) Log(Rank) Log(Magnitude) Log(Magnitude) Singh, Jain: Structural Analysis of Emerging Event-Web, (Short Paper) World Wide Web Conference‘10.
51 / 46 Situation Modeling • A conceptual step before physically implementing situation detection filters • Analogy: E/R modeling, UML • Helps domains experts externalize concepts e.g. ‘Epidemic’
52 / 46 Building Blocks: Operators Supporting Operator Type Data parameter(s) Output {Features} Φ Learn w Learning w = {0.3, 0.6, 0.1} method {Classification} Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
53 / 46 Queries • Seasonal characteristics • Show me the segments based on average greenery, as they vary over the year. • kmeans(n=3)(∏temporal(t>1293840)(TEStheme=‘green’)) • Political event analytics • Show me the difference of interests in Personalities (p1, p2) in places where H is an issue. • mult(diff(TEStheme=p1,TEStheme=p2), thresholds(30)(TEStheme=H)) p1=Obama, p2=Romney, H=Guns, Aug 9, 2012, via EventShop
54 / 46 Modeling personalized situations Personal c ϵ {Low, mid, threat level high}  Classification: Thresh(30,70)  And Physical Asthma threat exertion level Normalize ∏ ∏ Normalize (0, 100) (0, 100) TPS (Asthma) TPS (Funf) ∏ Δ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
55 / 46 Asthma Recommendation Application Macro situation model Asthma Threat c ϵ {Low, mid, high}, level [USA, 6 hrs, 0.1x 0.1]  Air Quality Pollen Count Allergy reports Emage Emage (Pollen Emage (Number (AQI.) Level) of reports) Δ Δ Δ Visual- Visual- Twitter-Allergy Air quality Pollen level Weather.com, Twitter API, [USA, 6 hrs, Pollen.com, [USA, 6 hrs, [USA, 6 hrs, 0.1x 0.1] 0.1x 0.1] 0.1x 0.1]
56 / 46 Asthma threat: personalized situation Personal threat level c ϵ {Low, mid, high}  Classification: Thresh(30,70)  And Physical Asthma exertion threat level Normalize ∏ ∏ Normalize (0, 100) (0, 100) TPS TPS (Funf) (Asthma) Δ ∏ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
9/26/2012 Proprietary and Confidential, Not For Distribution 57 / 46 iPhone: Interest over 12 days.
58 / 46 S4) Situation detection operators

More Related Content

PDF
AAAI08 tutorial: visual object recognition
byzukun
 
PPTX
Explanation & learning slides (talk @ pittsburgh science of learning center)
byJoseph Jay Williams
 
PDF
Week 3 descriptive research design
byuniv
 
PPTX
E Cognition User Summit2009 S Lang Zgis Object Validity
byTrimble Geospatial Munich
 
PDF
Abu Dhabi Island: Analysis of Development and Vegetation Change Using Remote ...
byThe Geospatial Technologist
 
PDF
Where is my mind?
byNicolas Rougier
 
PDF
A Cognitive Heuristic model for Local Community Recognition
byFET AWARE project - Self Awareness in Autonomic Systems
 
KEY
Visual Fixation and Image Quality
byEric Larson
 
AAAI08 tutorial: visual object recognition
byzukun
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
byJoseph Jay Williams
 
Week 3 descriptive research design
byuniv
 
E Cognition User Summit2009 S Lang Zgis Object Validity
byTrimble Geospatial Munich
 
Abu Dhabi Island: Analysis of Development and Vegetation Change Using Remote ...
byThe Geospatial Technologist
 
Where is my mind?
byNicolas Rougier
 
A Cognitive Heuristic model for Local Community Recognition
byFET AWARE project - Self Awareness in Autonomic Systems
 
Visual Fixation and Image Quality
byEric Larson
 

Viewers also liked

PPTX
UMAP 2011: Analyzing User Modeling on Twitter for Personalized News Recommend...
byWeb Information Systems, TU Delft
 
PPT
Social media analysis in R using twitter API
byMohd Shadab Alam
 
PDF
How to Improve Your Sales & Marketing Alignment
byHubSpot
 
PPTX
Sentiment analysis of twitter data
byBhagyashree Deokar
 
PPTX
Natural Language Processing in R (rNLP)
byfridolin.wild
 
PPT
Twitter sentiment-analysis Jiit2013-14
byRachit Goel
 
KEY
R by example: mining Twitter for consumer attitudes towards airlines
byJeffrey Breen
 
PPTX
TextMining with R
byAleksei Beloshytski
 
DOCX
Twitter analysis by Kaify Rais
byAjay Ohri
 
PPTX
4 Essential Lessons for Adopting Predictive Analytics in Healthcare
byHealth Catalyst
 
PPTX
Sentiment analysis of tweets
byVasu Jain
 
PPTX
Sentiment Analysis in Twitter
byAyushi Dalmia
 
PDF
Text Mining with R -- an Analysis of Twitter Data
byYanchang Zhao
 
PPTX
Big Data
bySubhavinolin Raja
 
UMAP 2011: Analyzing User Modeling on Twitter for Personalized News Recommend...
byWeb Information Systems, TU Delft
 
Social media analysis in R using twitter API
byMohd Shadab Alam
 
How to Improve Your Sales & Marketing Alignment
byHubSpot
 
Sentiment analysis of twitter data
byBhagyashree Deokar
 
Natural Language Processing in R (rNLP)
byfridolin.wild
 
Twitter sentiment-analysis Jiit2013-14
byRachit Goel
 
R by example: mining Twitter for consumer attitudes towards airlines
byJeffrey Breen
 
TextMining with R
byAleksei Beloshytski
 
Twitter analysis by Kaify Rais
byAjay Ohri
 
4 Essential Lessons for Adopting Predictive Analytics in Healthcare
byHealth Catalyst
 
Sentiment analysis of tweets
byVasu Jain
 
Sentiment Analysis in Twitter
byAyushi Dalmia
 
Text Mining with R -- an Analysis of Twitter Data
byYanchang Zhao
 
Big Data
bySubhavinolin Raja
 

Similar to Thesis personalized situation recognition

PPTX
Topic defense- Situation modeling and detection
byVivek Singh
 
PPTX
Situation based analysis and control for supporting Event-web applications
byVivek Singh
 
PDF
Situation recognition acm mm 121029
byRamesh Jain
 
PPTX
Situation Recognition from Multimodal Data Tutorial (ICME2016)
bySiripen Pongpaichet
 
PPTX
From Events to Situations: An Event-web perspective
byVivek Singh
 
PPTX
Real-Time Analysis of Streaming Sensor Data
byArtificial Intelligence Institute at UofSC
 
PDF
Eventshop 120721
byRamesh Jain
 
PPTX
Data Tactics Unified Dataspace Architecture and Description
byDataTactics
 
PPT
Albert Simard - Mobilizing Knowledge: Acquisition, Analysis, and Action
byInstitute for Knowledge Mobilization
 
PPT
Knowledge mobilization
byIntegrated Knowledge Services
 
PDF
Net-Enabled Test Environment: a Federated Approach to Establishing an Inter-a...
byWen Zhu
 
PDF
Mobile social search
byRamesh Jain
 
PPTX
Geographic Information Systems and Social Learning in Participatory Spatial P...
byRobert Goodspeed
 
PDF
Designing intelligent social systems 121205
byRamesh Jain
 
KEY
Evolution: It's a process
byChristine Connors
 
PDF
From Awareness to Adaptation
byFET AWARE project - Self Awareness in Autonomic Systems
 
PDF
Claudio Sapateiro ISCRAM 2009 Poster Session
byClaudioSapateiro
 
PPTX
MuMe Slide M. Wolpers 18 Nov
byMartin Wolpers
 
PDF
Integration of temporal and semantic components into the Geographic Informati...
byBeniamino Murgante
 
PDF
Building Social Life Networks 130818
byRamesh Jain
 
Topic defense- Situation modeling and detection
byVivek Singh
 
Situation based analysis and control for supporting Event-web applications
byVivek Singh
 
Situation recognition acm mm 121029
byRamesh Jain
 
Situation Recognition from Multimodal Data Tutorial (ICME2016)
bySiripen Pongpaichet
 
From Events to Situations: An Event-web perspective
byVivek Singh
 
Real-Time Analysis of Streaming Sensor Data
byArtificial Intelligence Institute at UofSC
 
Eventshop 120721
byRamesh Jain
 
Data Tactics Unified Dataspace Architecture and Description
byDataTactics
 
Albert Simard - Mobilizing Knowledge: Acquisition, Analysis, and Action
byInstitute for Knowledge Mobilization
 
Knowledge mobilization
byIntegrated Knowledge Services
 
Net-Enabled Test Environment: a Federated Approach to Establishing an Inter-a...
byWen Zhu
 
Mobile social search
byRamesh Jain
 
Geographic Information Systems and Social Learning in Participatory Spatial P...
byRobert Goodspeed
 
Designing intelligent social systems 121205
byRamesh Jain
 
Evolution: It's a process
byChristine Connors
 
From Awareness to Adaptation
byFET AWARE project - Self Awareness in Autonomic Systems
 
Claudio Sapateiro ISCRAM 2009 Poster Session
byClaudioSapateiro
 
MuMe Slide M. Wolpers 18 Nov
byMartin Wolpers
 
Integration of temporal and semantic components into the Geographic Informati...
byBeniamino Murgante
 
Building Social Life Networks 130818
byRamesh Jain
 

Recently uploaded

PDF
Security Forum Sessions from Houston 2025 Event
byMark Simos
 
PDF
Data Virtualization in Action: Scaling APIs and Apps with FME
bySafe Software
 
PDF
How to Evaluate a High Performance Database
byScyllaDB
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PPTX
communication-skills-with-technology tools
byJaleto Sunkemo
 
PDF
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
PDF
100 Insights After the 200th Issue of NewMind AI Journal
byNewMind AI
 
PPTX
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
PDF
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
PDF
Fundamentals and Frontiers of Post Quantum Cryptography
byGokul Alex
 
PDF
API-First Architecture in Financial Systems
bycyy111112
 
PDF
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
PPTX
Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]
byUiPathCommunity
 
PDF
10 Things AI-First Apps Do Differently by iProgrammer Solutions
byiProgrammer Solutions Private Limited
 
PPTX
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
PPTX
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 
PDF
Access Control 2025: From Security Silo to Software-Defined Ecosystem
byMemoori
 
PPTX
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
PDF
What Is a Private LLM and Why Enterprises Need It
byAivada
 
Security Forum Sessions from Houston 2025 Event
byMark Simos
 
Data Virtualization in Action: Scaling APIs and Apps with FME
bySafe Software
 
How to Evaluate a High Performance Database
byScyllaDB
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
communication-skills-with-technology tools
byJaleto Sunkemo
 
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
100 Insights After the 200th Issue of NewMind AI Journal
byNewMind AI
 
DYNAMICALLY.pptx good for the teachers or students to do seminars and for tea...
bympoorvika031
 
DevFest El Jadida 2025 - Product Thinking
byElmehdi AMLOU
 
Fundamentals and Frontiers of Post Quantum Cryptography
byGokul Alex
 
API-First Architecture in Financial Systems
bycyy111112
 
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]
byUiPathCommunity
 
10 Things AI-First Apps Do Differently by iProgrammer Solutions
byiProgrammer Solutions Private Limited
 
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 
Access Control 2025: From Security Silo to Software-Defined Ecosystem
byMemoori
 
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
What Is a Private LLM and Why Enterprises Need It
byAivada
 

Thesis personalized situation recognition

  • 1.
    / 46 PERSONALIZED SITUATION RECOGNITION VivekK. Singh Information Systems Group, University of California, Irvine Advised by: Professor Ramesh Jain
  • 2.
    2 / 46 Trends SOCIAL GEO- SOCIAL
  • 3.
    3 / 46 Trends PLANETARY SCALE
  • 4.
    4 / 46 Trends SENSE MAKING
  • 5.
    5 / 46 The Big Challenge Sense making from planetary scale geo-social data-streams Situation recognition
  • 6.
    6 / 46 Concept recognition from multimedia data Heterogeneous Media Heterogeneous Media Single Media Location Scenes Environ Trajectories Situations mentsK 3.4 Single Media aware Location Visual Real world Visual 360 K 11.4K Activities unaware Objects Objects Events Static Dynamic SPACE TIME
  • 7.
    7 / 46 Contributions 1. Computationally define situations 2. Define a generic process for Situation recognition a) Situation Modeling b) Situation Evaluation: • E-mage + Situation Recognition Algebra c) Personalized Alerts 3. EventShop: Web-based system for situation evaluation
  • 8.
    8 / 46 Situations: Other definitions • Endsley, 1988: “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” • Merriam-Webster dictionary: “relative position or combination of circumstances at a certain moment” • McCarthy, 1969: “A situation is a finite sequence of actions.” • Yau, 2006: “A situation is a set of contexts in the application over a period of time that affects future system behavior” • Dietrich, 2003: “…extensive information about the environment to be collected from all sensors independent of their interface technology. Data is transformed into abstract symbols. A combination of symbols leads to representation of current situations…which can be detected”
  • 9.
    9 / 46 Situations: commonalities • Goal Based • Abstraction • Space-Time • Computationally • Future Actions Grounded Future Computationally Work Goal Based Space-Time Abstraction Actions Grounded McCarthy, 1968 X Barwise, 1971 X X Endsley, 1988 X X X X Sarter, 1991 o X Adam, 1993 X X Dominguez,1994 X X X X Smith, 1995 X o X X Steinberg, 1999 X X X o Jeannot, 2003 X Moray, 2004 o X Dietrich, 2004 X X Yau, 2006 X X X Dostal, 2007 o X Singh, 2009 X X X Merriam-Webster (accessed 2012) o This work (aim) X X X X X o = Partial support
  • 10.
    10 / 46 Situation:Definition • Situation: An actionable abstraction of observed spatio-temporal characteristics. • e.g. flu epidemic, severe asthma threat, road congestion, wildfire, flash-mob Future Computationally Goal Based Space-Time Abstraction Actions Grounded
  • 11.
    11 / 46 Overall Framework: Motivating example Aggregation, Operations Alert level = High Date: 3rd Jun, 2011 STT data Situation Detection User-Feedback Tweet: 1) Classification ‘Please visit nearest CDC ‘Urrgh… sinus’ 2) Control action center at 4th St immediately’ Loc: NYC, Date: 3rd Jun, 2011 Theme: Allergy
  • 12.
    12 / 46 Eco-system: Situation based applications Human Sensor/ Wisdom source App logic Analyst Analysis & insights Spatio- Macro Device Situation Temporal situation Situation Sensors detection aggregation based operators controller Personal situation Archives Control decisions Event processing engine Human Sensor/ Alerts Actuator Singh, Jain: Situation based control. (Best Student Paper) IEEE Situation Management Workshop’09 Singh, Kankanhalli, Jain: Motivating contributors. (Best Paper) ACM Workshop on Social Media ’09
  • 13.
    Overall framework 13 / 46 A) Situation B) Situation C) Visualization, Modeling Recognition Personalization, and Alerts i) Visualization C1 …  v2 v3 Personal @  context ii) Personalization Personal v5 v6 + ized STT situation ∏ @ Δ Stream Available resources + Emage iii) Alerts Situation
  • 14.
    14 / 46 Design principles • Humans as sensors • Space + Time as fundamental axes • Real time situation evaluation (E-mage Streams) (a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream) d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)
  • 15.
    15 / 46 A)Situation Modeling • Help domain experts externalize their internal models of situations of interest e.g. epidemic. • Building blocks: • Operators • Operands • Wizard: • A prescriptive approach for modeling situations using the operators and operands Singh, Gao, Jain: Situation recognition: An evolving problem for heterogeneous dynamic big multimedia data, ACM Multimedia ‘12.
  • 16.
    16 / 46 BuildingBlocks: Operands • Knowledge or data driven building blocks Growth rate (Flu reports) Feature Twitter-Flu Data source -Emage (#Reports) Representation level Thresholds (0, 50) Meta-data
  • 17.
    17 / 46 Building Blocks: Operators Supporting Operator Type Data parameter(s) Output 1) Data into right representation Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate + 2) Analyze data to  Classification derive features Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + Pattern 72% {Features} 3) Use features to Φ Learn f Learning method f evaluate situations {Situation}
  • 18.
    18 / 46 Situation Modeling v v ϵ { Low, Mid, High} f1  <USA, 5 mins, 0.01x 0.01> v2 v3 v4 v=f(v1, …, vk) @ f2  ∏ • If (type = imprecise) • identify learning data source, method Emage v5 v6 Emage vi ∏ @ Δ Δ If (atomic) Emage • Identify Data source. D1 Emage D2 Δ Δ • Type, URL, ST bounds • Identify highest Rep. level reqd. D2 D3 • Identify operations Else Get_components(vi) } }
  • 19.
    19 / 46 3)Instantiate 2) Revise 1)Model Epidemic ϵ {Low, mid, high}, Outbreaks <USA, 5 mins, 0.01x Classification:  0.01> Thresh (30,70) Growing Unusual activity  Multiply Unusual Activity? Growth Rate  Subtract Historical Current activity Emage activity level level (#reports ILI)  Subtract Δ Normalize Emage π [0,100] (Historical avg) Emage Twitter-Flu Δ Emage Emage (#reports ILI) ILI) (#reports (population) Twitter.com Δ Δ <USA, 5 mins, Twitter-Avg Δ 0.01x 0.01> DB, Twitter.com Twitter-Flu CSV- Census.gov, Twitter.com Twitter-Flu <USA, 5 mins, <USA, 5 mins, Population <USA, 5 mins, <USA, 5 mins, 0.01x 0.01> 0.01x 0.01> 0.01x 0.01> 0.01x 0.01>
  • 20.
    20 / 46 B) Situation evaluation: Workflow Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, … … Level 1: Unified representation Properties (STT Data) STT Stream Level 2: Aggregation Properties Emage (Emage) Operations Level 3: Symbolic rep. Properties Situation (Situations)
  • 21.
    21 / 46 DataRepresentation • E-mage • Visualization • Spatio temporal data representation • Data analysis using media processing operators (e.g. segmentation, background subtraction, convolution)
  • 22.
    22 / 46 DataRepresentation • Spatio-temporal element • STTPoint = {s-t-coord, theme, value, pointer} • E-mage • g = (theme, x, v(x) | x ϵ X = R2 , and v(x) ϵ V = N) • Temporal E-mage Stream • TES=((t0, g0), ..., (tk, gk), …) • Temporal Pixel Stream • TPS = ((t0, p0), ..., (tk, pk), …)
  • 23.
    23 / 46 SituationRecognition Algebra Supporting Operator Type Data parameter(s) Output ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
  • 24.
    24 / 46 Situation Recognition Algebra S. No Operator Input Output 1 Filter ∏ Temporal E-mage Stream Temporal E-mage Stream 2 Aggregation  K*Temporal E-mage Stream Temporal E-mage Stream 3 Classification  Temporal E-mage Stream Temporal E-mage Stream 4 Characterization : @  Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel Stream 5 Pattern Matching   Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel Stream Singh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10.
  • 25.
    25 / 46 SampleQueries • Select E-mages of USA for theme ‘Obama’. • ∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama) • Identify three clusters for each E-mage above. • kmeans(3) (∏spatial(region=[24,-125],[24,-65])(TEStheme=Obama)) • Show me the cluster with most interest in ‘Obama’. • ∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama))) • Show me the speed for high interest cluster in ‘Katrina’ emages • @speed(@epicenter(∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))) • How similar is pattern above to ‘exponential increase’? • exp-increase(@speed(@epicenter (∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))
  • 26.
    26 / 46 C) Personalization and Alerts Personalized situation: An actionable integration of a user's personal context with surrounding spatiotemporal situation. 1) Macro situation Macro 2) Personal data-sources Context Personalized situation Profile + 3) Preferences Personalized alerts User Available data resources Resource data IF user Ui <is-in> (PSj) THEN <connect-to> Rk
  • 27.
    27 / 46 PersonalizedSituation Recognition: Operators Supporting Operator Type Data parameter(s) Output ∏ Filter … + User location  Aggregate … + … …  Classification Classification … method … @ Characterization … Property Growth Rate required = 125%  Pattern Matching … + Match= 42% Pattern
  • 28.
    28 / 46 SituationAction Rules •
  • 29.
    29 / 46 EVENTSHOP: Recognizing situations from web streams
  • 30.
    30 / 46 EventShop:System Implementation • Front end: • Javascript (JSLinb library) • Front-Back end Interaction • Java servlets, Apache • Back End • Java • C++ (OpenCV classes) • Ingestion wrappers available for: • Twitter streams, Flickr stream, CSV data, KML data, Geo-images, MySQL data archives, Funf (mobile phone sensors) Gao, Singh, Jain: EventShop: From Heterogeneous Web Streams to Personalized Situation Detection and Control, ACM WebScience ‘12.
  • 31.
    31 / 46 S.No Query Language Operator Media processing Media processing Operator Details Translation into Operator Media processing 1. Filter -Spatial Arithmetic AND with the spatial mask operators -Temporal Arithmetic AND with the temporal mark -Thematic Arithmetic = -Value Arithmetic AND, >, <, = 2. Aggregation -Max, Min, +,-,%,* Arithmetic Max, Min, +,-,%,* - NOT, OR, AND, Logical NOT, OR, AND -Convolution Convolution Convolution 3. Classification - Predefined segments count Segmentation K-means - Predefined segment boundaries Segmentation thresholds 4. Characterization i) Spatial - Count, Min, Max, Sum, Average, Variation Statistical Count, Min, Max, Sum, Average - Coverage Arithmetic Count - Epicenter Arithmetic Weighted average - Circularity Convolution Scale free convolution with known circular kernel - Growth rate Arithmetic +, -, % ii) Temporal - Displacement, Distance, Velocity, Arithmetic +, -, %, * Acceleration, Growth rate - Future estimation Arithmetic Multiplication with Kernels based on users choice e.g. linear, progression exponential growth - Periodicity Convolution Auto correlation i.e. Self convolution with time-lagged variant. 5. Pattern Matching - Scaled Matching Convolution Convolution with user defined or pre-defined Kernels - Scale free Matching Convolution, Statistical Maxima from Loops of Convolution with different image sizes.
  • 32.
    32 / 46 Evaluations 1.Design principles • Humans as sensors to detect real world events 2. Data representation and Situation recognition algebra • Expressive, computable and explicit • Real world results 3. Framework for situation recognition • modeling, • situation evaluation, • personalized alerts
  • 33.
    33 / 46 Humans as sensors • Can social media be used to detect real world events? Observed Observed S.No Category Event Physical Date Physical Location Temporal Peak Spatial Peak 38.89, -77.03 1 Politics Health Care Bill passed 2010-03-21 2010-03-21 41, -74 (Washington) 37.77, -122.41 2 Politics California Prop 8, Trial Day 1 2010-01-11 2010-01-11 38,-122 (San Francisco) 31.13, -97.78 3 Society Fort Hood Shootings 2009-11-05 2009-11-05 33,-97 (Fort Hood, TX) 28.54, -81.38 4 Society SeaWorld Whale Accident 2010-02-12 2010-02-12 29,-81 (Orlando, FL) Winter Olympics Opening 49.24, -123.11 5 Sports 2010-02-12 2010-02-12 44,-79 ceremony (Vancouver) 40.71, -74.00 6 Sports Baseball World Series final 2009-11-04 2009-11-04 41, -74 (New York) 34.05, -118.24 7 Entertainment Oscars 2010-03-07 2010-03-07 34, -118 (Los Angeles) 2010-03-12 to 30.26, -97.74 8 Entertainment South by Southwest festival 2010-03-15 30, -98 2010-03-21 (Austin, TX) 2010-01-05 to 36.17, -115.13 9 Tech. Conv. CES 2010 2010-01-06 34,-118 2010-01-07 (Las Vegas) 2010-02-10 to 33.76, -118.19 10 Tech. Conv. TED 2010 2010-01-10 34, -118 2010-02-13 (Long Beach,CA)
  • 34.
    34 / 46 Datarepresentation + Algebra • Applications • Business analytics • Political event analytics • Seasonal characteristics • Data • Twitter feeds archive • Loops of location based queries for different terms • Over 100 million tweets using ‘Spritzer’/ ‘Gardenhose’ APIs • Flickr feeds • API: Tags, RGB values from >800K images • Implementation • Matlab + Java + Python
  • 35.
    35 / 46 iPhone theme AT&T based e-mage, retail Jun 2 to Jun 15, 2009 locations . Convolution Store + Add * catchment area Aggregate Subtract AT&T total interest - catchment area <geoname> Convolution . @Spatial.Max Decision <name>College City</name> <lat>39.0057303</lat> <lng>-122.0094129</lng> Best Location is at <geonameId>5338600</geonameId> * <countryCode>US</countryCode> Geocode [39, - <countryName>United States</countryName> 122] , just north of <fcl>P</fcl> Bay Area, CA <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> Under-served <distance>1.0332</distance> interest areas Store catchment </geoname> area
  • 36.
    36 / 46 Seasonalcharacteristics analysis • Fall colors in New England • Show me the difference between red and green colors for New England region, as it varies throughout the year. • subtract(@spatial(sum)(πspatial(R=[(40,-76), (44,-71)]) (TEStheme=Red)), @spatial(sum)(πspatial(R=[(40,-76), (44,-71)])(TEStheme=Green))) 0 Jan Dec
  • 37.
    37 / 46 Buildingapplications using the framework Application Data Operators S.No Application Data Used Scale deployed? modalities used Wildfire detection in Satellite data, 1 Real Yes Macro F, A, Ch California Google insights 2 Hurricane monitoring Simulated No Macro n/a F, A, Ch, P Flu epidemic 3 Real No Macro Twitter, Census F, A, C surveillance Macro, Twitter, Air Allergy/ Asthma 4 Real In-progress Personalized Quality, Pollen F, A, C recommendation alerts Count Macro, Thailand flood 5 Real Yes Personalized KML F, A, C mitigation alerts Legend: F = Filter, A = Aggregate, C = Classification, Ch = Characterization, P = Pattern Matching
  • 38.
    38 / 46 Wildfire recognition model (Satellite data) Fire detector ϵ {fire, non-fire}, (Satellite driven) <California, 24hrs, 0.01x 0.01>  AND Significant band Unclouded? Hot enough? variation? Thresh Thresh  AND =392 =310 Emage (12 µm band Emage (Mid IR Absolute value Spatial Neighbor temp.) surface temp.) variation variation Δ Δ Thresh= 30 ∏ ∏ Thresh= 5 Satellite Satellite Band 4 Spatial Neighborhood Band 12 Difference value Difference LAADS.com,   LAADS.com, <California, 24hrs, Subtract <California, 24hrs, Subtract 0.01x 0.01> 0.01x 0.01> Difference Neighborhood Emage (4 µm Emage (11µm value Mean value temperature) temperature)  Convolve (7X7) Satellite Satellite Band 12 Band 12 Difference value LAADS.com, LAADS.com, <California, 24hrs, <California, 24hrs, 0.01x 0.01> 0.01x 0.01>
  • 39.
    39 / 46 Wildfirerecognition model (Social data) Fire detector ϵ {fire, non-fire}, (Social) <California, 24hrs,  0.01x 0.01> And Spatially anomalous Temporally anomalous ∏ ∏ Thresh=7 Thresh= 5 Difference with other Difference with Historical areas today average   Subtract Subtract Emage (Google Spatial Avg. of Emage (Google Emage (Google Insights- Fire) Interest Insights- Fire) Insights- Historical Avg) Δ  Δ Δ Average Google Emage (Google Google Google Insights-Fire Insights- Fire) Insights-Fire Insights-Fire Δ Google.com/insights, Google.com/insights, Google.com/insights, <California, 24hrs, <California, 24hrs, <California, 24hrs, Metros> Google Metros> Metros> Insights-Fire Google.com/insights, <California, 24hrs, Metros>
  • 40.
    40 / 46 Wildfire recognition Fire detector ϵ {fire, non-fire}, Situation <California, 24hrs, 0.01x 0.01>  OR Modeling Fire detector Fire detector (Social) (Satellite) Situation Evaluation 50 45 40 35 30 Social detector Results 25 20 Satellite detector 15 Combined 10 Ground truth 5 Number of 0 Wildfires detected 2010 2011 Total
  • 41.
    41 / 46 Demo:Asthma Recommendation Application
  • 42.
    42 / 46 ThailandFlood mitigation
  • 43.
    43 / 46 Social Life Networks Connecting People and Resources Situation aware routing Information Aggregation Situation and Detection Alerts Composition Queries Jain, Singh, Gao: Social Life Networks for the Middle of the Pyramid, ACM 43 Workshop on Social Media Engagement ‘11.
  • 44.
    44 / 46 Related Work: Snapshot Area Combine Human Data Define Location Real-time Toolkits hetero sensors analytics situations aware streams data Situation X X o o X Awareness X Situation X Calculus X Web data o X X o X mining X X Social media o X X o X mining X X X Multimedia X o o o Event detection X Complex event X X o X processing/ X X Active DB GIS XX o X X XX o Mashup toolkits X X o X X (Y! pipes, ifttt) X This work X X X X X X X o = partial support
  • 45.
    45 / 46 Futurework • EventShop: • Personalization • Scalability • Prediction • Using such tools to nudge people into taking desired actions • Supporting Grids and Graphs for analysis • Social Life Networks
  • 46.
    46 / 46 Summary •Personalized Actionable Situations • 1st Systematic approach • Situation Modeling • EventShop: Web based system for Situation Evaluation • Apps: Democratize data and action taking • Eco-system for data-to-action
  • 47.
  • 48.
  • 49.
    49 / 46 AnalyzingBig Data Field/ Approach Databases Networks Spatio-temporal Data structure Tables Graphs Grids Apps Business records, Internet traffic, Healthcare, Disaster Banking Social network, relief, Business, Roads Security Problems Querying, Searching Shortest path, Situation detection influence, anomaly Operators Select, Project, Join Diameter, influence Select, Aggregate, detection, connected ST characterization, components ST pattern matching, Classification Modeling ER modeling, Network diagrams, Situation models Query plan PetriNets Tools SQL server, Oracle NS2, NetworkX EventShop DBMS
  • 50.
    50 / 46 Geo-Social Power Laws • Studied 5.6 Million Tweets for a month • There is a fixed relative ratio for the occurrence of events of different magnitude across space or time. Across Space Across Time Whole world Only USA 1 month 1 week Around New York 1 day 3 weeks city 30 mins 2 weeks Log(Rank) Log(Rank) Log(Magnitude) Log(Magnitude) Singh, Jain: Structural Analysis of Emerging Event-Web, (Short Paper) World Wide Web Conference‘10.
  • 51.
    51 / 46 SituationModeling • A conceptual step before physically implementing situation detection filters • Analogy: E/R modeling, UML • Helps domains experts externalize concepts e.g. ‘Epidemic’
  • 52.
    52 / 46 BuildingBlocks: Operators Supporting Operator Type Data parameter(s) Output {Features} Φ Learn w Learning w = {0.3, 0.6, 0.1} method {Classification} Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
  • 53.
    53 / 46 Queries •Seasonal characteristics • Show me the segments based on average greenery, as they vary over the year. • kmeans(n=3)(∏temporal(t>1293840)(TEStheme=‘green’)) • Political event analytics • Show me the difference of interests in Personalities (p1, p2) in places where H is an issue. • mult(diff(TEStheme=p1,TEStheme=p2), thresholds(30)(TEStheme=H)) p1=Obama, p2=Romney, H=Guns, Aug 9, 2012, via EventShop
  • 54.
    54 / 46 Modelingpersonalized situations Personal c ϵ {Low, mid, threat level high}  Classification: Thresh(30,70)  And Physical Asthma threat exertion level Normalize ∏ ∏ Normalize (0, 100) (0, 100) TPS (Asthma) TPS (Funf) ∏ Δ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
  • 55.
    55 / 46 AsthmaRecommendation Application Macro situation model Asthma Threat c ϵ {Low, mid, high}, level [USA, 6 hrs, 0.1x 0.1]  Air Quality Pollen Count Allergy reports Emage Emage (Pollen Emage (Number (AQI.) Level) of reports) Δ Δ Δ Visual- Visual- Twitter-Allergy Air quality Pollen level Weather.com, Twitter API, [USA, 6 hrs, Pollen.com, [USA, 6 hrs, [USA, 6 hrs, 0.1x 0.1] 0.1x 0.1] 0.1x 0.1]
  • 56.
    56 / 46 Asthmathreat: personalized situation Personal threat level c ϵ {Low, mid, high}  Classification: Thresh(30,70)  And Physical Asthma exertion threat level Normalize ∏ ∏ Normalize (0, 100) (0, 100) TPS TPS (Funf) (Asthma) Δ ∏ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
  • 57.
    9/26/2012 Proprietary and Confidential, Not For Distribution 57 / 46 iPhone: Interest over 12 days.
  • 58.
    58 / 46 S4)Situation detection operators

Editor's Notes

  • #4 Some comments
  • #8 This work required combination of efforts coming from stream data processing perspective and situation recognition from a media processing perspective. Hence parts of this work were done in collaboration with Mingyan. She looked at the problem from Stream data processing perspective, while I focused on defining situations as a concept, and their recognition. Specific focus of joint work was on 2b) Situation evaluation, and 3) EventShop implementation.
  • #44 We need to discuss directionality of arrows.