Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

People as sensors - mining social media for meaningful information


Published on

The video of this talk is available at

More and more we are all broadcasting information. Geolocation data, “this x sucks” data, weather data, etc.

More and more that data is being parsed and analysed in realtime, such that we have now become sensors.

How does this work, what does this mean, and what risks/benefits will it bring?

Published in: Technology, Business

People as sensors - mining social media for meaningful information

  1. 1. People as Sensors Tom Raftery ThingMonk, London Dec 2013. 1
  2. 2. Tom Raftery • Lead analyst, energy and sustainability practice, RedMonk • • • • +34 677 695 468 • 2
  3. 3. Mobile data 3 Mobile phones with GPS, accelerometers, compass, M7, barometers, moisture, etc
  4. 4. Mobile data 4 Mobile phones with GPS and data retention laws Green party sued Deutsche Telecom for data Put it online Source of concern
  5. 5. Social Media 5 Mobile is not the only data we’re broadcasting
  6. 6. Facebook 6 1.19 billion monthly active users (MAU) as of September 30, 2013 727 million daily active users (DAU) in Sept 2013 874 million mobile MAUs as of September 30 2013
  7. 7. Facebook 7 Most recent usage data is from May 2013 - so well out-of-date 4.75 billion content items shared per day (status updates + wall posts + photos + videos + comments) 4.5 billion Likes per day
  8. 8. Twitter 8 Between 4,000-8,500 tweets per second per day Avg over 6,000 tweets per sec, all stored in perpetuity Tweets drop off at midnight ET, start picking up again at 6am ET!
  9. 9. Twitter 9 Increasing tweets per sec over the last year (max value per day used) Two peaks on the right-hand-side
  10. 10. Twitter 10 Increasing tweets per day over the last year
  11. 11. Twitter 11 Most recent numbers from Twitter: Twitter avg - >500m TPD Twitter avg - 5,700 TPS Aug 2nd had a 143k TPS record (>28x the average) - no blip to service
  12. 12. Data for sale… 12 Twitter in its IPO filings disclosed it is making $47.5m from selling access to its data
  13. 13. Google+ 13 Google+ 540m MAU 300m active in stream 1.5bn photos per month
  14. 14. Sina Weibo 14 China’s Sina Weibo is growing with 74% year on year user growth ( Has 220m ‘active users’ while Twitter has 170m ‘active users’
  15. 15. Waze 15 Waze had an estimated 50m users in June 2013
  16. 16. Waze 16 Waze had an estimated 50m users in June 2013
  17. 17. Use Cases 17 Some positive use cases of the data
  18. 18. Crowd-Sourcing 18 Academic study on feasibility of using Twitter to crowdsourced data
  19. 19. Meteorology 19 UK Snow Map by Ben Marsh Tweet #uksnow, postcode and x/10 rating
  20. 20. Utilities 20 GE’s Grid IQ Insight can mine social media for mentions of outages Gives early notifications of an outage in an area If geotagged and/or includes images/video can confirm cause of outage and speed up time to resolution
  21. 21. Utilities 21 Utilities are aware of reason for outage, speeds up time to resolution (reduces need for investigatory truck roll)
  22. 22. Risk Analysis 22 United Nations Development Program & their Recorded Future project Using publicly sourced data looking for signs of disruption or unrest
  23. 23. Risk Analysis 23 The same graph turned into media sources- who is talking about Georgia in this period of time
  24. 24. Risk Analysis 24 Mentions turned into social network analysis- who is talking to whom, who is meeting whom “next phase will focus on conducting a regional political risk analysis and forecasting for South Eastern Europe and Central Asia” UNDP slides courtesy of Milica Begovich (aka @elim????)
  25. 25. Automotive 25 Social Media monitoring tool developed by Pamplin College of Business Initial version worked from automotive fora and blogs, now expanding to take in Twitter and Facebook “Robust” way to discover and classify vehicle defects from social media posts across multiple automotive brands Faster than reporting back up through the dealer chain
  26. 26. Finance 26 Academic paper from University of Manchester and Indiana University shows that Twitter can predict the Dow Jones Industrial Average with 87.6% accuracy
  27. 27. Finance 27 UK Firm Derwent Capital Markets signed an exclusive deal with the authors to create a hedge fund - became Cayman Atlantic
  28. 28. Law Enforcement 28 SAS produced an interesting white paper on this space and bought UK firm Memex - definitely chasing this space Citing use cases like - finding individuals - analysing their social graph to find accomplices/gang structure Also identify precursor activity to events like riots
  29. 29. Law Enforcement 29 “Social media is a huge network of informants—and one you don't have to pay for.” Law Enforcement use cases (information distribution, fake profile creation, etc.) Helps first responders gain situational awareness prior to having feet on the ground Helps Emergency Operations Centres gain information in the event of natural disasters, for example Sunday’s train crash in NY, for example
  30. 30. Law Enforcement X Other vendors in this space outlined in this article 3i-Mind - HMS Technologies - Visible Technologies - Attensity - CrowdControlHQ - and As well as Law Enforcement use cases (information distribution, fake profile creation, etc.)
  31. 31. Law Enforcement 30 Good infographic on Law Enforcement use of social media - based on a LexisNexis Risk Solutions survey of 1,200 law enforcement professionals Full report is available at
  32. 32. Law Enforcement 31 Needs to be approached sensitively - the way some of this is reported often prompts visions of ‘pre-crime’ and Minority Report
  33. 33. Smart Cities 32 Graffiti, Pothole, & LA school district apps
  34. 34. Healthcare 33 Google use frequency of certain search terms as a way to estimate flu activity Also have one for Dengue Fever Search data is increasingly mobile
  35. 35. Healthcare 34 Google wrote this up as an academic paper and it was published in Nature
  36. 36. Healthcare 35 A group led from Harvard Medical School studied viability of using social media for predicting cholera outbreak Found that the data from Twitter closely corresponded with government data, was available up to two weeks earlier The paper concludes informal media could be used to study the activity of other disease outbreaks around the world Financial support was provided by
  37. 37. Healthcare 36 Formerly asthmapolis - wireless asthma puff data - where/when Can map where asthma outbreaks occur - people sensitive can avoid triggers
  38. 38. Healthcare 37 Rolled out in conjunction with city of Louisville Ky Residents experience as much as a 13-year gap in life expectancy depending upon where they live Findings eagerly anticipated - only rolled out in Oct
  39. 39. CRM 38 T-Mobile in US analysed its 33m customer data records, web logs, billing data and social media information to predict customer defections It halved customer defections in 3 months
  40. 40. Brand Management 39 Nestle were Greenpeace’d because palm oil in Kit Kat came from Sinar Mas - company involved in deforestation
  41. 41. Brand Management 40 In the social media storm which followed Nestlé made every mistake in the book Nestlé received over 200,000 protest emails and their share price was negatively affected So they decided to work with Greenpeace to fix their supply chain and To initiate a social media strategy for the organisation
  42. 42. Brand Management 41 Set up a social media command centre, staffed by their Digital Acceleration Team
  43. 43. Brand Management 42 In 2013 Nestle entered the Reputation Institute’s Global top 10 for the first time.
  44. 44. Transportation 43 Waze data now being incorporated in Google Maps
  45. 45. Transportation 44 2.5bn anonymised call records from 5m Orange phone users in Ivory Coast Looked at patterns of people’s movements in Abidjan - capital city of Ivory Coast Realised they could reduce travel times of ppl by 10%
  46. 46. Looking Ahead 45 Google Glass
  47. 47. Looking Ahead 46 Instabeat gives swimmers stats in their goggles as they swim And subsequent download
  48. 48. Looking ahead 47 Fitbit force, Nike+ Fuelband, Jawbone Up Can see a situation where sports players are broadcasting vital stats similar to F1
  49. 49. Conclusion Data and data sources are increasing exponentially - go hack that data for good. 48
  50. 50. Thanks! Contact information: Tom Raftery Principal Analyst, Energy & Sustainability, RedMonk,, +34 677 695 468 No tweets were hurt in the making of this presentation 49