Social pixels acm_mm
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Social pixels acm_mm

Uploaded on

Using social media to understand the situations occurring in the real world.

Using social media to understand the situations occurring in the real world.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. SOCIAL PIXELS: GENESIS & EVALUATIONVivek Singh, Mingyan Gao, and Ramesh Jain University of California, Irvine
  • 2. Outline Concept Approach Applications Challenges
  • 3. Motivation People are sharing massive amounts of information on the web (Twitter, Flickr, Facebook, …) How to do effective data consumption, not just data creation  Geo-spatial situation awareness  Real time updates of the world state  From data to actionable knowledge
  • 4. Concept Understanding evolving world situations by combining spatio-temporal-thematic data coming from social media (e.g. Twitter/Flickr). „Iphone‟ social image for mainland USA. Jun 11, 2009
  • 5. Social Pixels Traditional Pixels  Photons aggregating at locations on CCD Social Pixels  User interest aggregating at geo-locations Create social Image, social Video… Image/Media Processing operators Situation Detection operators (e.g. convolution, filtering, background subtraction)
  • 6. Design principles Humans as sensors Social pixel approach  Visualization  Intuitive query and mental model  Common spatio-temporal data representation  Data analysis using media processing Combining media processing with declarative query algebra
  • 7. Overall Approach1. Micro-event detection2. Spatio-temporal aggregation using social pixel approach3. Media processing engine4. Query engine
  • 8. Micro-event detection Simple bag-of-words approach for detecting what event is the user talking about.  e.g. „Sore throat‟, „Flu‟, „H1N1‟, … Tweet: „caught sore-throat today…arrrgh !‟ Micro-event detected for user X. Spatial Temporal Thematic
  • 9. Spatio-temporal aggregation using social pixels Higher level abstractions have trade-offs with lower level details Percolate up what is necessary for the application Can be:  Count of tweets with the term  Average green channel value of images  Mean audio energy  Average monthly income, rainfall, population etc.
  • 10. Data Model Spatio-temporal element  stel = [s-t-coord, theme(s), value(s), pointer(s)] E-mage g = (x, {(tm, v(x))}|xϵ X = R2 , tm ϵ θ, and v(x) ϵ V = N) Temporal E-mage Set  TES= {(t1, g1), ..., (tn, gn)}, Temporal Pixel Set  TPS = {(t1, p1), ..., (tn, pn)},
  • 11. Operations1. Selection Operation2. Arithmetic and Logical Operation3. Aggregation Operation α4. Grouping Operation5. Characterization Operation  Spatial  Temporal6. Pattern Matching Operation  Spatial  Temporal
  • 12. 1. Selection Operation Select part of E-mage based on predicate P Input: Temporal E-mage Set TES = {(t1, g1), …, (tn , gn)} Output: Temporal E-mage Set TES‟ Spatial or Value predicate Pi on Emage  Pi(TES) = {(t1, Pi(g1)), …, (tn, Pi(gn))}, where Pi(g) = {(x, y) | y=g(x), if Pi(x,y) is true; y=0, otherwise} Boolean predicate Pt on time  Pt(TES) = {(t1‟ g1‟), …, (tm‟, gm‟)}, where P(ti‟) is true, e.g. date = „2010-03-10‟
  • 13. Selection Examples Show last one week‟s E-mages of California for topic „Obama‟  R=cal t <= 1wk theme= Obama(TES)
  • 14. 2. Arithmetic Operation Binary operations between two (or more) E- mage Sets (g1, g2) = g3(x, (v1(x), v2(x))), where {+, -, *, /, max, min, convolution}, g1 and g2 are the same size. Example:  TES1=Temporal E-mage Set for „Unemployment rate‟  TES2=Temporal E-mage Set for „normalized Gas prices‟  TES3= (TES1, TES2)
  • 15. 3. Aggregation Operation α Aggregates multiple E-mages in TES based on function . (g1, g2) = g3(x, (v1(x), v2(x))), where {+, *, mean, max, min}, g1 and g2 are the same size. Example:  Show the average emage of last one week‟s emages from California for Obama.  α mean ( R=cal t <= 1wk theme= Obama(TES))
  • 16. 4. Grouping Operation Group stels in an E-mage g based on certain function f Input: Temporal E-mage Set TES = {(t1, g1), …, (tn , gn)} Output: Temporal E-mage Set TES‟ Function f essentially splits g, into multiple sub-e- mages. f(TES) = f((t1, g1)) … f((tn,gn)), where f((ti, gi)) = {(ti , gi1‟), …, (ti , gik‟)}, and each gij‟ is a sub-E-mage of g based on f f {segmentation, clustering, blob-detection, etc.}
  • 17. Grouping Example Identify 3 clusters for each E-mage in the TES set having last one week‟s E-mages of California.  clustering, n=3( R=cal t <= 1wk(TES))
  • 18. 5a. Characterization Op. (Spatial) Represent each E-mage g based on a characteristic C, and store result as a stel. Input: Temporal E-mage Set TES = {(t1, g1), …, (tn, gn)} Output: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)} C(TES) = {(t1, (g1)), …, (tn, (gn))}, where (gi) is a pixel characterizing gi C {count, max, min, sum, average, coverage, epi center, density, shape, growth_rate, periodicity }
  • 19. Characterization Examples(Spatial) Find the epicenter of each cluster E-mage in the last one week‟s E-mages of USA from TES  epicenter ( clustering, n=3( R=USA t <= 1wk theme=Obama(TES))
  • 20. 5b. Characterization Op.(Temporal) Characterize a temporal pixel set, which is the result of E-mage characterization Input: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)} Output: Temporal Pixel Set TPS‟ (TPS) = {(tk , ((t1, p1), …, (tk, pk))) | k [2, n]}, where {displacement, distance, velocity, speed, accel eration, linear extrapolation, exponential growth, exponential decay, etc.}
  • 21. Temporal CharacterizationExamples Find the velocity of epicenter of each cluster E- mage over the last one week‟s E-mages of California from TES for theme Katrina  velocity ( epicenter ( clustering, n=3( R=Cal t <= 1wk theme = Katrina (TES))))
  • 22. 5. Pattern Matching Pattern Matching (Spatial)  Compare the similarity between each E-mage and a given pattern P  Input: Temporal E-mage Set TES = {(t1, g1), …, (tn, gn)}, and pattern P  Output: Temporal Pixel Set TPS  P(TES) = {(t1, p1), …, (tn, pn)}, where each value in pi represents the similarity between the E-mage and the given pattern  Patterns (i.e. Kernels) can be loaded from a library or be historical data samples.
  • 23. Pattern Matching Temporal Pattern matching:  Compare the similarity of the temporal value changing with a given pattern, e.g. „increasing‟, „decreasing‟, or „Enron‟s stock in 1999‟, … Input: Temporal Pixel Set TPS = {(t1, p1), …, (tn, pn)}, and a pattern P Output: Temporal Pixel Set TPS‟ P(TPS) = {(tn , p)}, where v(x) in p is the similarity value
  • 24. Pattern Matching Examples Compare the similarity between each E-mage in the last one week‟s E-mages of California from TES with radial decay  radial_decay( R=cal t <= 1wk theme = Obama (TES)) How close is the similarity above to pattern of “Enron‟s stock price in 1999”?  Enron‟s stock( radial_decay( R=cal t <= 1wk(TES)))
  • 25. Situation detection operatorsS. No Operator Input Output1 Selection Temporal Temporal E-mage Set E-mage Set2 Arithmetic & K*Temporal E-mage Temporal E-mage Set Logical Set3 Aggregation α Temporal E-mage set Temporal E-mage Set4 Grouping Temporal E-mage Set Temporal E-mage Set5 Characterization : •Spatial •Temporal E-mage Set •Temporal Pixel Set •Temporal •Temporal Pixel Set •Temporal Pixel Set6 Pattern Matching •Spatial •Temporal E-mage Set •Temporal Pixel Set •Temporal •Temporal Pixel Set •Temporal Pixel Set
  • 26. Mediaprocessing engine
  • 27. Implementation and results Twitter feeds  Geo-coding user home location  Loops of location based queries for different terms  Over 100 million tweets using „Spritzer‟ stream (since Jun 2009), and the higher rate „Gardenhose‟ stream since Nov, 2009. Flickr feeds  API  Tags, RGB values from >800K images
  • 28. Correlation with real worldevents
  • 29. Applications Business decision making Political event analytics Seasonal characteristics analysis
  • 30. Situation awareness: iPhonelaunch
  • 31. Spatio temporal variation:Visualization
  • 32. Business intelligence: Queries
  • 33. iPhone theme AT&T based e-mage, retail Jun 2 to Jun 11 locations . Convolution Store + Aggregation * catchment area DifferenceAggregateinterest Combination of operators - AT&T total catchmen t area <geoname> Convolution . MAXIMA <name>College City</name> Decision <lat>39.0057303</lat> <lng>-122.0094129</lng> Best Location is at <geonameId>5338600</geonameId> * <countryCode>US</countryCode> Geocode [39, - <countryName>United States</countryName> 122] , just north of <fcl>P</fcl> Bay Area, CA <fcode>PPL</fcode> <fclName>city, village,...</fclName> <fcodeName>populated place</fcodeName> <population/> Under-served <distance>1.0332</distance> </geoname> interest areas Store catchment
  • 34. Political event analytics:Queries
  • 35. Snapshot
  • 36. Flickr Social Emages Jan – Dec 2009
  • 37. Seasonal characteristicsanalysis
  • 38. Year average Peak of green At [35, -84], at the junction of Chattahoochee National Forest, Nantahala National Forest, Cherokee National Forest and Great Smoky Mountains National Park
  • 39. Variations throughout the year Total Energy Jan DecFall colors of New England  [R-G] channel data 0 Jan Dec
  • 40. Conclusions Combining spatio-temporal event data for visualization, and analytics. An e-mage representation of spatio-temporal thematic data coming in real-time. Defined operators for real-time situation analysis Applications in multiple domains
  • 41. Challenges: Future work Defining a (visual) query language using operators Scalability  Realtime data management for all possible topics which user might be interested in Automatic tweets from sensors A reverse-911 like control/recommendation mechanism Creating an event web by connecting all event related data