Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Social pixels acm_mm


Published on

Using social media to understand the situations occurring in the real world.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Social pixels acm_mm

  1. 1. SOCIAL PIXELS: GENESIS & EVALUATIONVivek Singh, Mingyan Gao, and Ramesh Jain University of California, Irvine
  2. 2. Outline Concept Approach Applications Challenges
  3. 3. Motivation People are sharing massive amounts of information on the web (Twitter, Flickr, Facebook, …) How to do effective data consumption, not just data creation  Geo-spatial situation awareness  Real time updates of the world state  From data to actionable knowledge
  4. 4. Concept Understanding evolving world situations by combining spatio-temporal-thematic data coming from social media (e.g. Twitter/Flickr). „Iphone‟ social image for mainland USA. Jun 11, 2009
  5. 5. Social Pixels Traditional Pixels  Photons aggregating at locations on CCD Social Pixels  User interest aggregating at geo-locations Create social Image, social Video… Image/Media Processing operators Situation Detection operators (e.g. convolution, filtering, background subtraction)
  6. 6. Design principles Humans as sensors Social pixel approach  Visualization  Intuitive query and mental model  Common spatio-temporal data representation  Data analysis using media processing Combining media processing with declarative query algebra
  7. 7. Overall Approach1. Micro-event detection2. Spatio-temporal aggregation using social pixel approach3. Media processing engine4. Query engine
  8. 8. Micro-event detection Simple bag-of-words approach for detecting what event is the user talking about.  e.g. „Sore throat‟, „Flu‟, „H1N1‟, … Tweet: „caught sore-throat today…arrrgh !‟ Micro-event detected for user X. Spatial Temporal Thematic
  9. 9. Spatio-temporal aggregation using social pixels Higher level abstractions have trade-offs with lower level details Percolate up what is necessary for the application Can be:  Count of tweets with the term  Average green channel value of images  Mean audio energy  Average monthly income, rainfall, population etc.
  10. 10. Data Model Spatio-temporal element  stel = [s-t-coord, theme(s), value(s), pointer(s)] E-mage g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N) Temporal E-mage Set  TES= {(t1, g1), ..., (tn, gn)}, Temporal Pixel Set  TPS = {(t1, p1), ..., (tn, pn)},
  11. 11. Operations1. Selection Operation2. Arithmetic and Logical Operation3. Aggregation Operation α4. Grouping Operation5. Characterization Operation  Spatial  Temporal6. Pattern Matching Operation  Spatial  Temporal
  12. 12. 1. Selection Operation Select part of E-mage based on predicate P Input: Temporal E-mage Set TES = {(t1, g1), …, (tn , gn)} Output: Temporal E-mage Set TES‟ Spatial or Value predicate Pi on Emage  Pi(TES) = {(t1, Pi(g1)), …, (tn, Pi(gn))}, where Pi(g) = {(x, y) | y=g(x), if Pi(x,y) is true; y=0, otherwise} Boolean predicate Pt on time  Pt(TES) = {(t1‟ g1‟), …, (tm‟, gm‟)}, where P(ti‟) is true, e.g. date = „2010-03-10‟
  13. 13. Selection Examples Show last one week‟s E-mages of California for topic „Obama‟  R=cal t <= 1wk theme= Obama(TES)
  14. 14. 2. Arithmetic Operation Binary operations between two (or more) E- mage Sets (g1, g2) = g3(x, (v1(x), v2(x))), where {+, -, *, /, max, min, convolution}, g1 and g2 are the same size. Example:  TES1=Temporal E-mage Set for „Unemployment rate‟  TES2=Temporal E-mage Set for „normalized Gas prices‟  TES3= (TES1, TES2)
  15. 15. 3. Aggregation Operation α Aggregates multiple E-mages in TES based on function . (g1, g2) = g3(x, (v1(x), v2(x))), where {+, *, mean, max, min}, g1 and g2 are the same size. Example:  Show the average emage of last one week‟s emages from California for Obama.  α mean ( R=cal t <= 1wk theme= Obama(TES))
  16. 16. 4. Grouping Operation Group stels in an E-mage g based on certain function f Input: Temporal E-mage Set TES = {(t1, g1), …, (tn , gn)} Output: Temporal E-mage Set TES‟ Function f essentially splits g, into multiple sub-e- mages. f(TES) = f((t1, g1)) … f((tn,gn)), where f((ti, gi)) = {(ti , gi1‟), …, (ti , gik‟)}, and each gij‟ is a sub-E-mage of g based on f f {segmentation, clustering, blob-detection, etc.}
  17. 17. Grouping Example Identify 3 clusters for each E-mage in the TES set having last one week‟s E-mages of California.  clustering, n=3( R=cal t <= 1wk(TES))
  18. 18. 5a. Characterization Op. (Spatial) Represent each E-mage g based on a characteristic C, and store result as a stel. Input: Temporal E-mage Set TES = {(t1, g1), …, (tn, gn)} Output: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)} C(TES) = {(t1, (g1)), …, (tn, (gn))}, where (gi) is a pixel characterizing gi C {count, max, min, sum, average, coverage, epi center, density, shape, growth_rate, periodicity }
  19. 19. Characterization Examples(Spatial) Find the epicenter of each cluster E-mage in the last one week‟s E-mages of USA from TES  epicenter ( clustering, n=3( R=USA t <= 1wk theme=Obama(TES))
  20. 20. 5b. Characterization Op.(Temporal) Characterize a temporal pixel set, which is the result of E-mage characterization Input: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)} Output: Temporal Pixel Set TPS‟ (TPS) = {(tk , ((t1, p1), …, (tk, pk))) | k [2, n]}, where {displacement, distance, velocity, speed, accel eration, linear extrapolation, exponential growth, exponential decay, etc.}
  21. 21. Temporal CharacterizationExamples Find the velocity of epicenter of each cluster E- mage over the last one week‟s E-mages of California from TES for theme Katrina  velocity ( epicenter ( clustering, n=3( R=Cal t <= 1wk theme = Katrina (TES))))
  22. 22. 5. Pattern Matching Pattern Matching (Spatial)  Compare the similarity between each E-mage and a given pattern P  Input: Temporal E-mage Set TES = {(t1, g1), …, (tn, gn)}, and pattern P  Output: Temporal Pixel Set TPS  P(TES) = {(t1, p1), …, (tn, pn)}, where each value in pi represents the similarity between the E-mage and the given pattern  Patterns (i.e. Kernels) can be loaded from a library or be historical data samples.
  23. 23. Pattern Matching Temporal Pattern matching:  Compare the similarity of the temporal value changing with a given pattern, e.g. „increasing‟, „decreasing‟, or „Enron‟s stock in 1999‟, … Input: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)}, and a pattern P Output: Temporal Pixel Set TPS‟ P(TPS) = {(tn , p)}, where v(x) in p is the similarity value
  24. 24. Pattern Matching Examples Compare the similarity between each E-mage in the last one week‟s E-mages of California from TES with radial decay  radial_decay( R=cal t <= 1wk theme = Obama (TES)) How close is the similarity above to pattern of “Enron‟s stock price in 1999”?  Enron‟s stock( radial_decay( R=cal t <= 1wk(TES)))
  25. 25. Situation detection operatorsS. No Operator Input Output1 Selection Temporal Temporal E-mage Set E-mage Set2 Arithmetic & K*Temporal E-mage Temporal E-mage Set Logical Set3 Aggregation α Temporal E-mage set Temporal E-mage Set4 Grouping Temporal E-mage Set Temporal E-mage Set5 Characterization : •Spatial •Temporal E-mage Set •Temporal Pixel Set •Temporal •Temporal Pixel Set •Temporal Pixel Set6 Pattern Matching •Spatial •Temporal E-mage Set •Temporal Pixel Set •Temporal •Temporal Pixel Set •Temporal Pixel Set
  26. 26. Mediaprocessing engine
  27. 27. Implementation and results Twitter feeds  Geo-coding user home location  Loops of location based queries for different terms  Over 100 million tweets using „Spritzer‟ stream (since Jun 2009), and the higher rate „Gardenhose‟ stream since Nov, 2009. Flickr feeds  API  Tags, RGB values from >800K images
  28. 28. Correlation with real worldevents
  29. 29. Applications Business decision making Political event analytics Seasonal characteristics analysis
  30. 30. Situation awareness: iPhonelaunch
  31. 31. Spatio temporal variation:Visualization
  32. 32. Business intelligence: Queries
  33. 33. iPhone theme AT&T based e-mage, retail Jun 2 to Jun 11 locations . Convolution Store + Aggregation * catchment area DifferenceAggregateinterest Combination of operators - AT&T total catchmen t area <geoname> Convolution . MAXIMA <name>College City</name> Decision <lat>39.0057303</lat> <lng>-122.0094129</lng> Best Location is at <geonameId>5338600</geonameId> * <countryCode>US</countryCode> Geocode [39, - <countryName>United States</countryName> 122] , just north of <fcl>P</fcl> Bay Area, CA <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> Under-served <distance>1.0332</distance> </geoname> interest areas Store catchment
  34. 34. Political event analytics:Queries
  35. 35. Snapshot
  36. 36. Flickr Social Emages Jan – Dec 2009
  37. 37. Seasonal characteristicsanalysis
  38. 38. Year average Peak of green At [35, -84], at the junction of Chattahoochee National Forest, Nantahala National Forest, Cherokee National Forest and Great Smoky Mountains National Park
  39. 39. Variations throughout the year Total Energy Jan DecFall colors of New England  [R-G] channel data 0 Jan Dec
  40. 40. Conclusions Combining spatio-temporal event data for visualization, and analytics. An e-mage representation of spatio-temporal thematic data coming in real-time. Defined operators for real-time situation analysis Applications in multiple domains
  41. 41. Challenges: Future work Defining a (visual) query language using operators Scalability  Realtime data management for all possible topics which user might be interested in Automatic tweets from sensors A reverse-911 like control/recommendation mechanism Creating an event web by connecting all event related data