Thesis personalized situation recognition

1,686 views

Published on

Thesis defense slides: Personalized Situation Recognition.
Vivek Singh, University of California, Irvine.
( Advisor: Professor Ramesh Jain )

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,686
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
26
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Some comments
  • This work required combination of efforts coming from stream data processing perspective and situation recognition from a media processing perspective. Hence parts of this work were done in collaboration with Mingyan. She looked at the problem from Stream data processing perspective, while I focused on defining situations as a concept, and their recognition. Specific focus of joint work was on 2b) Situation evaluation, and 3) EventShop implementation.
  • We need to discuss directionality of arrows.
  • Thesis personalized situation recognition

    1. 1. / 46PERSONALIZED SITUATIONRECOGNITIONVivek K. SinghInformation Systems Group,University of California, IrvineAdvised by: Professor Ramesh Jain
    2. 2. 2 / 46 TrendsSOCIALGEO-SOCIAL
    3. 3. 3 / 46TrendsPLANETARY SCALE
    4. 4. 4 / 46TrendsSENSE MAKING
    5. 5. 5 / 46 The Big ChallengeSense making from planetaryscale geo-social data-streams Situation recognition
    6. 6. 6 / 46Concept recognition from multimedia data Heterogeneous Media Heterogeneous Media Single Media Location Scenes Environ Trajectories Situations mentsK 3.4 Single Media aware Location Visual Real world Visual 360 K 11.4K Activities unaware Objects Objects Events Static Dynamic SPACE TIME
    7. 7. 7 / 46Contributions1. Computationally define situations2. Define a generic process for Situation recognition a) Situation Modeling b) Situation Evaluation: • E-mage + Situation Recognition Algebra c) Personalized Alerts3. EventShop: Web-based system for situation evaluation
    8. 8. 8 / 46Situations: Other definitions• Endsley, 1988: “the perception of elements in the environment within avolume of time and space, the comprehension of their meaning, and theprojection of their status in the near future”• Merriam-Webster dictionary: “relative position or combination ofcircumstances at a certain moment”• McCarthy, 1969: “A situation is a finite sequence of actions.”• Yau, 2006: “A situation is a set of contexts in the application over a periodof time that affects future system behavior”• Dietrich, 2003: “…extensive information about the environment to becollected from all sensors independent of their interface technology. Data istransformed into abstract symbols. A combination of symbols leads torepresentation of current situations…which can be detected”
    9. 9. 9 / 46Situations: commonalities• Goal Based • Abstraction• Space-Time • Computationally• Future Actions Grounded Future ComputationallyWork Goal Based Space-Time Abstraction Actions GroundedMcCarthy, 1968 XBarwise, 1971 X XEndsley, 1988 X X X XSarter, 1991 o XAdam, 1993 X XDominguez,1994 X X X XSmith, 1995 X o X XSteinberg, 1999 X X X oJeannot, 2003 XMoray, 2004 o XDietrich, 2004 X XYau, 2006 X X XDostal, 2007 o XSingh, 2009 X X XMerriam-Webster(accessed 2012) oThis work (aim) X X X X X o = Partial support
    10. 10. 10 / 46Situation: Definition• Situation: An actionable abstraction of observed spatio-temporal characteristics. • e.g. flu epidemic, severe asthma threat, road congestion, wildfire, flash-mob Future Computationally Goal Based Space-Time Abstraction Actions Grounded
    11. 11. 11 / 46Overall Framework: Motivating example Aggregation, Operations Alert level = High Date: 3rd Jun, 2011 STT data Situation Detection User-Feedback Tweet: 1) Classification ‘Please visit nearest CDC ‘Urrgh… sinus’ 2) Control action center at 4th St immediately’ Loc: NYC,Date: 3rd Jun, 2011 Theme: Allergy
    12. 12. 12 / 46 Eco-system: Situation based applications Human Sensor/ Wisdom source App logic Analyst Analysis & insights Spatio- MacroDevice Situation Temporal situation SituationSensors detection aggregation based operators controller Personal situation Archives Control decisions Event processing engine Human Sensor/ Alerts Actuator Singh, Jain: Situation based control. (Best Student Paper) IEEE Situation Management Workshop’09 Singh, Kankanhalli, Jain: Motivating contributors. (Best Paper) ACM Workshop on Social Media ’09
    13. 13. Overall framework 13 / 46A) Situation B) Situation C) Visualization, Modeling Recognition Personalization, and Alerts i) Visualization C1 … v2 v3 Personal@  context ii) Personalization Personal v5 v6 + ized STT situation ∏ @ Δ Stream Available resources + Emage iii) Alerts Situation
    14. 14. 14 / 46 Design principles • Humans as sensors • Space + Time as fundamental axes • Real time situation evaluation (E-mage Streams)(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)
    15. 15. 15 / 46A) Situation Modeling• Help domain experts externalize their internal models of situations of interest e.g. epidemic.• Building blocks: • Operators • Operands• Wizard: • A prescriptive approach for modeling situations using the operators and operandsSingh, Gao, Jain: Situation recognition: An evolving problem for heterogeneous dynamic big multimedia data, ACM Multimedia ‘12.
    16. 16. 16 / 46Building Blocks: Operands• Knowledge or data driven building blocks Growth rate (Flu reports) Feature Twitter-Flu Data source -Emage (#Reports) Representation level Thresholds (0, 50) Meta-data
    17. 17. 17 / 46 Building Blocks: Operators Supporting Operator Type Data parameter(s) Output1) Data into right representation Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate +2) Analyze data to  Classification derive features Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + Pattern 72% {Features}3) Use features to Φ Learn f Learning method fevaluate situations {Situation}
    18. 18. 18 / 46 Situation Modeling v v ϵ { Low, Mid, High} f1  <USA, 5 mins, 0.01x 0.01> v2 v3 v4 v=f(v1, …, vk) @ f2  ∏ • If (type = imprecise) • identify learning data source, method Emage v5 v6 Emage vi ∏ @ Δ Δ If (atomic) Emage • Identify Data source. D1 Emage D2 Δ Δ • Type, URL, ST bounds • Identify highest Rep. level reqd. D2 D3 • Identify operations Else Get_components(vi) }}
    19. 19. 19 / 463) Instantiate2) Revise1)Model Epidemic ϵ {Low, mid, high}, Outbreaks <USA, 5 mins, 0.01x Classification:  0.01> Thresh (30,70) Growing Unusual activity  Multiply Unusual Activity? Growth Rate  Subtract Historical Current activity Emage activity level level (#reports ILI)  Subtract Δ Normalize Emage π [0,100] (Historical avg) Emage Twitter-Flu Δ Emage Emage (#reports ILI) ILI) (#reports (population) Twitter.com Δ Δ <USA, 5 mins, Twitter-Avg Δ 0.01x 0.01> DB, Twitter.com Twitter-Flu CSV- Census.gov, Twitter.com Twitter-Flu<USA, 5 mins, <USA, 5 mins, Population <USA, 5 mins, <USA, 5 mins, 0.01x 0.01> 0.01x 0.01> 0.01x 0.01> 0.01x 0.01>
    20. 20. 20 / 46 B) Situation evaluation: Workflow Level 0: Raw data streams e.g. tweets, cameras, traffic, weather, … … Level 1: Unified representation Properties (STT Data) STT Stream Level 2: Aggregation Properties Emage (Emage)Operations Level 3: Symbolic rep. Properties Situation (Situations)
    21. 21. 21 / 46Data Representation• E-mage • Visualization • Spatio temporal data representation • Data analysis using media processing operators (e.g. segmentation, background subtraction, convolution)
    22. 22. 22 / 46Data Representation• Spatio-temporal element • STTPoint = {s-t-coord, theme, value, pointer}• E-mage • g = (theme, x, v(x) | x ϵ X = R2 , and v(x) ϵ V = N)• Temporal E-mage Stream • TES=((t0, g0), ..., (tk, gk), …)• Temporal Pixel Stream • TPS = ((t0, p0), ..., (tk, pk), …)
    23. 23. 23 / 46Situation Recognition Algebra Supporting Operator Type Data parameter(s) Output ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
    24. 24. 24 / 46 Situation Recognition AlgebraS. No Operator Input Output1 Filter ∏ Temporal E-mage Stream Temporal E-mage Stream2 Aggregation  K*Temporal E-mage Stream Temporal E-mage Stream3 Classification  Temporal E-mage Stream Temporal E-mage Stream4 Characterization : @  Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel Stream5 Pattern Matching   Spatial  Temporal E-mage Stream  Temporal Pixel Stream  Temporal  Temporal Pixel Stream  Temporal Pixel StreamSingh, Gao, Jain: Social Pixels: Genesis and Evaluation, ACM Multimedia ‘10.
    25. 25. 25 / 46Sample Queries• Select E-mages of USA for theme ‘Obama’. • ∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama)• Identify three clusters for each E-mage above. • kmeans(3) (∏spatial(region=[24,-125],[24,-65])(TEStheme=Obama))• Show me the cluster with most interest in ‘Obama’. • ∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Obama)))• Show me the speed for high interest cluster in ‘Katrina’ emages • @speed(@epicenter(∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina)))))• How similar is pattern above to ‘exponential increase’? • exp-increase(@speed(@epicenter (∏value(v=1) (kmeans(n=3) (∏spatial(region=[24,-125],[24,-65]) (TEStheme=Katrina))))
    26. 26. 26 / 46 C) Personalization and AlertsPersonalized situation: An actionable integration of a users personalcontext with surrounding spatiotemporal situation. 1) Macro situation Macro 2) Personaldata-sources Context Personalized situation Profile + 3) Preferences Personalized alerts User Available data resources Resource data IF user Ui <is-in> (PSj) THEN <connect-to> Rk
    27. 27. 27 / 46Personalized Situation Recognition:Operators Supporting Operator Type Data parameter(s) Output ∏ Filter … + User location  Aggregate … + … …  Classification Classification … method … @ Characterization … Property Growth Rate required = 125%  Pattern Matching … + Match= 42% Pattern
    28. 28. 28 / 46Situation Action Rules•
    29. 29. 29 / 46 EVENTSHOP:Recognizing situations from web streams
    30. 30. 30 / 46EventShop: System Implementation• Front end: • Javascript (JSLinb library)• Front-Back end Interaction • Java servlets, Apache• Back End • Java • C++ (OpenCV classes)• Ingestion wrappers available for: • Twitter streams, Flickr stream, CSV data, KML data, Geo-images, MySQL data archives, Funf (mobile phone sensors) Gao, Singh, Jain: EventShop: From Heterogeneous Web Streams to Personalized Situation Detection and Control, ACM WebScience ‘12.
    31. 31. 31 / 46 S.No Query Language Operator Media processing Media processing Operator DetailsTranslation into OperatorMedia processing 1. Filter -Spatial Arithmetic AND with the spatial maskoperators -Temporal Arithmetic AND with the temporal mark -Thematic Arithmetic = -Value Arithmetic AND, >, <, = 2. Aggregation -Max, Min, +,-,%,* Arithmetic Max, Min, +,-,%,* - NOT, OR, AND, Logical NOT, OR, AND -Convolution Convolution Convolution 3. Classification - Predefined segments count Segmentation K-means - Predefined segment boundaries Segmentation thresholds 4. Characterization i) Spatial - Count, Min, Max, Sum, Average, Variation Statistical Count, Min, Max, Sum, Average - Coverage Arithmetic Count - Epicenter Arithmetic Weighted average - Circularity Convolution Scale free convolution with known circular kernel - Growth rate Arithmetic +, -, % ii) Temporal - Displacement, Distance, Velocity, Arithmetic +, -, %, * Acceleration, Growth rate - Future estimation Arithmetic Multiplication with Kernels based on users choice e.g. linear, progression exponential growth - Periodicity Convolution Auto correlation i.e. Self convolution with time-lagged variant. 5. Pattern Matching - Scaled Matching Convolution Convolution with user defined or pre-defined Kernels - Scale free Matching Convolution, Statistical Maxima from Loops of Convolution with different image sizes.
    32. 32. 32 / 46Evaluations1. Design principles • Humans as sensors to detect real world events2. Data representation and Situation recognition algebra • Expressive, computable and explicit • Real world results3. Framework for situation recognition • modeling, • situation evaluation, • personalized alerts
    33. 33. 33 / 46 Humans as sensors • Can social media be used to detect real world events? Observed ObservedS.No Category Event Physical Date Physical Location Temporal Peak Spatial Peak 38.89, -77.03 1 Politics Health Care Bill passed 2010-03-21 2010-03-21 41, -74 (Washington) 37.77, -122.41 2 Politics California Prop 8, Trial Day 1 2010-01-11 2010-01-11 38,-122 (San Francisco) 31.13, -97.78 3 Society Fort Hood Shootings 2009-11-05 2009-11-05 33,-97 (Fort Hood, TX) 28.54, -81.38 4 Society SeaWorld Whale Accident 2010-02-12 2010-02-12 29,-81 (Orlando, FL) Winter Olympics Opening 49.24, -123.11 5 Sports 2010-02-12 2010-02-12 44,-79 ceremony (Vancouver) 40.71, -74.00 6 Sports Baseball World Series final 2009-11-04 2009-11-04 41, -74 (New York) 34.05, -118.24 7 Entertainment Oscars 2010-03-07 2010-03-07 34, -118 (Los Angeles) 2010-03-12 to 30.26, -97.74 8 Entertainment South by Southwest festival 2010-03-15 30, -98 2010-03-21 (Austin, TX) 2010-01-05 to 36.17, -115.13 9 Tech. Conv. CES 2010 2010-01-06 34,-118 2010-01-07 (Las Vegas) 2010-02-10 to 33.76, -118.19 10 Tech. Conv. TED 2010 2010-01-10 34, -118 2010-02-13 (Long Beach,CA)
    34. 34. 34 / 46Data representation + Algebra• Applications • Business analytics • Political event analytics • Seasonal characteristics• Data • Twitter feeds archive • Loops of location based queries for different terms • Over 100 million tweets using ‘Spritzer’/ ‘Gardenhose’ APIs • Flickr feeds • API: Tags, RGB values from >800K images• Implementation • Matlab + Java + Python
    35. 35. 35 / 46 iPhone theme AT&T based e-mage, retail Jun 2 to Jun 15, 2009 locations . Convolution Store + Add * catchment areaAggregate Subtract AT&T totalinterest - catchment area <geoname> Convolution . @Spatial.Max Decision <name>College City</name> <lat>39.0057303</lat> <lng>-122.0094129</lng> Best Location is at <geonameId>5338600</geonameId> * <countryCode>US</countryCode> Geocode [39, - <countryName>United States</countryName> 122] , just north of <fcl>P</fcl> Bay Area, CA <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> Under-served <distance>1.0332</distance> interest areas Store catchment </geoname> area
    36. 36. 36 / 46Seasonal characteristics analysis• Fall colors in New England • Show me the difference between red and green colors for New England region, as it varies throughout the year. • subtract(@spatial(sum)(πspatial(R=[(40,-76), (44,-71)]) (TEStheme=Red)), @spatial(sum)(πspatial(R=[(40,-76), (44,-71)])(TEStheme=Green))) 0 Jan Dec
    37. 37. 37 / 46Building applications using the framework Application Data OperatorsS.No Application Data Used Scale deployed? modalities used Wildfire detection in Satellite data, 1 Real Yes Macro F, A, Ch California Google insights 2 Hurricane monitoring Simulated No Macro n/a F, A, Ch, P Flu epidemic 3 Real No Macro Twitter, Census F, A, C surveillance Macro, Twitter, Air Allergy/ Asthma 4 Real In-progress Personalized Quality, Pollen F, A, C recommendation alerts Count Macro, Thailand flood 5 Real Yes Personalized KML F, A, C mitigation alerts Legend: F = Filter, A = Aggregate, C = Classification, Ch = Characterization, P = Pattern Matching
    38. 38. 38 / 46 Wildfire recognition model (Satellite data) Fire detector ϵ {fire, non-fire}, (Satellite driven) <California, 24hrs, 0.01x 0.01>  AND Significant band Unclouded? Hot enough? variation?Thresh Thresh  AND =392 =310 Emage (12 µm band Emage (Mid IR Absolute value Spatial Neighbor temp.) surface temp.) variation variation Δ Δ Thresh= 30 ∏ ∏ Thresh= 5 Satellite Satellite Band 4 Spatial Neighborhood Band 12 Difference value Difference LAADS.com,   LAADS.com, <California, 24hrs, Subtract <California, 24hrs, Subtract 0.01x 0.01> 0.01x 0.01> Difference Neighborhood Emage (4 µm Emage (11µm value Mean value temperature) temperature)  Convolve (7X7) Satellite Satellite Band 12 Band 12 Difference value LAADS.com, LAADS.com, <California, 24hrs, <California, 24hrs, 0.01x 0.01> 0.01x 0.01>
    39. 39. 39 / 46Wildfire recognition model (Social data) Fire detector ϵ {fire, non-fire}, (Social) <California, 24hrs,  0.01x 0.01> And Spatially anomalous Temporally anomalous ∏ ∏ Thresh=7 Thresh= 5 Difference with other Difference with Historical areas today average   Subtract SubtractEmage (Google Spatial Avg. of Emage (Google Emage (Google Insights- Fire) Interest Insights- Fire) Insights- Historical Avg) Δ  Δ Δ Average Google Emage (Google Google Google Insights-Fire Insights- Fire) Insights-Fire Insights-Fire Δ Google.com/insights, Google.com/insights, Google.com/insights, <California, 24hrs, <California, 24hrs, <California, 24hrs, Metros> Google Metros> Metros> Insights-Fire Google.com/insights, <California, 24hrs, Metros>
    40. 40. 40 / 46 Wildfire recognition Fire detector ϵ {fire, non-fire},Situation <California, 24hrs, 0.01x 0.01>  ORModeling Fire detector Fire detector (Social) (Satellite)SituationEvaluation 50 45 40 35 30 Social detectorResults 25 20 Satellite detector 15 Combined 10 Ground truth 5 Number of 0 Wildfires detected 2010 2011 Total
    41. 41. 41 / 46Demo: Asthma RecommendationApplication
    42. 42. 42 / 46Thailand Flood mitigation
    43. 43. 43 / 46 Social Life Networks Connecting People and Resources Situation aware routing Information Aggregation Situation and Detection Alerts Composition QueriesJain, Singh, Gao: Social Life Networks for the Middle of the Pyramid, ACM 43 Workshop on Social Media Engagement ‘11.
    44. 44. 44 / 46 Related Work: Snapshot Area Combine Human Data Define Location Real-time Toolkits hetero sensors analytics situations aware streams dataSituation X X o o XAwareness XSituation XCalculus XWeb data o X X o Xmining X XSocial media o X X o Xmining X X XMultimedia X o o oEvent detection XComplex event X X o Xprocessing/ X XActive DBGIS XX o X X XX oMashup toolkits X X o X X(Y! pipes, ifttt) XThis work X X X X X X X o = partial support
    45. 45. 45 / 46Future work• EventShop: • Personalization • Scalability • Prediction• Using such tools to nudge people into taking desired actions• Supporting Grids and Graphs for analysis• Social Life Networks
    46. 46. 46 / 46Summary• Personalized Actionable Situations• 1st Systematic approach• Situation Modeling• EventShop: Web based system for Situation Evaluation• Apps: Democratize data and action taking• Eco-system for data-to-action
    47. 47. / 46THANKS !
    48. 48. 48 / 46BACKUP SLIDES
    49. 49. 49 / 46Analyzing Big DataField/ Approach Databases Networks Spatio-temporalData structure Tables Graphs GridsApps Business records, Internet traffic, Healthcare, Disaster Banking Social network, relief, Business, Roads SecurityProblems Querying, Searching Shortest path, Situation detection influence, anomalyOperators Select, Project, Join Diameter, influence Select, Aggregate, detection, connected ST characterization, components ST pattern matching, ClassificationModeling ER modeling, Network diagrams, Situation models Query plan PetriNetsTools SQL server, Oracle NS2, NetworkX EventShop DBMS
    50. 50. 50 / 46 Geo-Social Power Laws • Studied 5.6 Million Tweets for a month • There is a fixed relative ratio for the occurrence of events of different magnitude across space or time. Across Space Across Time Whole worldOnly USA 1 month 1 week Around New York 1 day 3 weeks city 30 mins 2 weeks Log(Rank) Log(Rank) Log(Magnitude) Log(Magnitude) Singh, Jain: Structural Analysis of Emerging Event-Web, (Short Paper) World Wide Web Conference‘10.
    51. 51. 51 / 46Situation Modeling• A conceptual step before physically implementing situation detection filters • Analogy: E/R modeling, UML• Helps domains experts externalize concepts e.g. ‘Epidemic’
    52. 52. 52 / 46Building Blocks: Operators Supporting Operator Type Data parameter(s) Output {Features} Φ Learn w Learning w = {0.3, 0.6, 0.1} method {Classification} Δ Transform … Spatio-temporal window ∏ Filter + Mask  Aggregate +  Classification Classification method @ Characterization Property Growth Rate required = 125%  Pattern Matching + 72% Pattern
    53. 53. 53 / 46Queries• Seasonal characteristics • Show me the segments based on average greenery, as they vary over the year. • kmeans(n=3)(∏temporal(t>1293840)(TEStheme=‘green’))• Political event analytics • Show me the difference of interests in Personalities (p1, p2) in places where H is an issue. • mult(diff(TEStheme=p1,TEStheme=p2), thresholds(30)(TEStheme=H)) p1=Obama, p2=Romney, H=Guns, Aug 9, 2012, via EventShop
    54. 54. 54 / 46Modeling personalized situations Personal c ϵ {Low, mid, threat level high}  Classification: Thresh(30,70)  And Physical Asthma threat exertion level Normalize ∏ ∏ Normalize (0, 100) (0, 100) TPS (Asthma) TPS (Funf) ∏ Δ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
    55. 55. 55 / 46Asthma Recommendation Application Macro situation model Asthma Threat c ϵ {Low, mid, high}, level [USA, 6 hrs, 0.1x 0.1]  Air Quality Pollen Count Allergy reports Emage Emage (Pollen Emage (Number (AQI.) Level) of reports) Δ Δ Δ Visual- Visual- Twitter-Allergy Air quality Pollen level Weather.com, Twitter API, [USA, 6 hrs, Pollen.com, [USA, 6 hrs, [USA, 6 hrs, 0.1x 0.1] 0.1x 0.1] 0.1x 0.1]
    56. 56. 56 / 46Asthma threat: personalized situation Personal threat level c ϵ {Low, mid, high}  Classification: Thresh(30,70)  And Physical Asthma exertion threat levelNormalize ∏ ∏ Normalize (0, 100) (0, 100) TPS TPS (Funf) (Asthma) Δ ∏ UserLoc Funf-activity EventShop [USA, 6 hrs, Phone sensors, 0.1x 0.1] (relaxMinder app), [USA, 6 hrs, 0.1x 0.1]
    57. 57. 9/26/2012 Proprietary and Confidential, Not For Distribution 57 / 46iPhone: Interest over 12 days.
    58. 58. 58 / 46S4) Situation detection operators

    ×