People as sensors - mining social media for meaningful information

14,761 views
14,415 views

Published on

The video of this talk is available at https://www.youtube.com/watch?v=4ZdknOPY_jQ

More and more we are all broadcasting information. Geolocation data, “this x sucks” data, weather data, etc.

More and more that data is being parsed and analysed in realtime, such that we have now become sensors.

How does this work, what does this mean, and what risks/benefits will it bring?

Published in: Technology, Business
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
14,761
On SlideShare
0
From Embeds
0
Number of Embeds
93
Actions
Shares
0
Downloads
43
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

People as sensors - mining social media for meaningful information

  1. 1. People as Sensors Tom Raftery ThingMonk, London Dec 2013. 1
  2. 2. Tom Raftery • Lead analyst, energy and sustainability practice, RedMonk • GreenMonk.net • twitter.com/tomraftery • tom@redmonk.com • +34 677 695 468 • SlideShare.net/TomRaftery 2
  3. 3. Mobile data http://www.flickr.com/photos/traftery/8551389911/ 3 Mobile phones with GPS, accelerometers, compass, M7, barometers, moisture, etc
  4. 4. Mobile data http://www.zeit.de/datenschutz/malte-spitz-data-retention 4 Mobile phones with GPS and data retention laws Green party sued Deutsche Telecom for data Put it online Source of concern
  5. 5. Social Media 5 Mobile is not the only data we’re broadcasting
  6. 6. Facebook http://newsroom.fb.com/Key-Facts 6 1.19 billion monthly active users (MAU) as of September 30, 2013 727 million daily active users (DAU) in Sept 2013 874 million mobile MAUs as of September 30 2013
  7. 7. Facebook http://techcrunch.com/2013/05/17/facebook-growth/ 7 Most recent usage data is from May 2013 - so well out-of-date 4.75 billion content items shared per day (status updates + wall posts + photos + videos + comments) 4.5 billion Likes per day
  8. 8. Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 8 Between 4,000-8,500 tweets per second per day Avg over 6,000 tweets per sec, all stored in perpetuity Tweets drop off at midnight ET, start picking up again at 6am ET!
  9. 9. Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 9 Increasing tweets per sec over the last year (max value per day used) Two peaks on the right-hand-side
  10. 10. Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 10 Increasing tweets per day over the last year
  11. 11. Twitter https://blog.twitter.com/2013/new-tweets-per-second-record-and-how 11 Most recent numbers from Twitter: Twitter avg - >500m TPD Twitter avg - 5,700 TPS Aug 2nd had a 143k TPS record (>28x the average) - no blip to service
  12. 12. Data for sale… http://online.wsj.com/news/articles/SB10001424052702304441404579118531954483974 12 Twitter in its IPO filings disclosed it is making $47.5m from selling access to its data
  13. 13. Google+ http://googleblog.blogspot.com.es/2013/10/google-hangouts-and-photos-save-some.html 13 Google+ 540m MAU 300m active in stream 1.5bn photos per month
  14. 14. Sina Weibo http://sg.finance.yahoo.com/news/sina-weibo-passes-500-million-151054944.html 14 China’s Sina Weibo is growing with 74% year on year user growth (http://www.chinadaily.com.cn/bizchina/2013-02/21/content_16243933.htm) Has 220m ‘active users’ while Twitter has 170m ‘active users’ http://blogs.wsj.com/chinarealtime/2013/03/12/how-many-people-really-usesina-weibo/
  15. 15. Waze https://www.waze.com 15 Waze had an estimated 50m users in June 2013
  16. 16. Waze https://www.waze.com 16 Waze had an estimated 50m users in June 2013
  17. 17. Use Cases 17 Some positive use cases of the data
  18. 18. Crowd-Sourcing http://csce.uark.edu/~tingxiny/courses/5013spring13/readingList/crowdsource.pdf 18 Academic study on feasibility of using Twitter to crowdsourced data
  19. 19. Meteorology http://uksnowmap.com/#/ 19 UK Snow Map by Ben Marsh Tweet #uksnow, postcode and x/10 rating
  20. 20. Utilities http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/ 20 GE’s Grid IQ Insight can mine social media for mentions of outages Gives early notifications of an outage in an area If geotagged and/or includes images/video can confirm cause of outage and speed up time to resolution
  21. 21. Utilities http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/ 21 Utilities are aware of reason for outage, speeds up time to resolution (reduces need for investigatory truck roll)
  22. 22. Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 22 United Nations Development Program & their Recorded Future project Using publicly sourced data looking for signs of disruption or unrest
  23. 23. Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 23 The same graph turned into media sources- who is talking about Georgia in this period of time
  24. 24. Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 24 Mentions turned into social network analysis- who is talking to whom, who is meeting whom “next phase will focus on conducting a regional political risk analysis and forecasting for South Eastern Europe and Central Asia” UNDP slides courtesy of Milica Begovich (aka @elim????)
  25. 25. Automotive http://www.magazine.pamplin.vt.edu/fall12/vehicledefects.html 25 Social Media monitoring tool developed by Pamplin College of Business Initial version worked from automotive fora and blogs, now expanding to take in Twitter and Facebook “Robust” way to discover and classify vehicle defects from social media posts across multiple automotive brands Faster than reporting back up through the dealer chain
  26. 26. Finance http://arxiv.org/PS_cache/arxiv/pdf/1010/1010.3003v1.pdf 26 Academic paper from University of Manchester and Indiana University shows that Twitter can predict the Dow Jones Industrial Average with 87.6% accuracy
  27. 27. Finance http://www.caymanatlantic.com/investment-management/4574471088 27 UK Firm Derwent Capital Markets signed an exclusive deal with the authors to create a hedge fund - became Cayman Atlantic
  28. 28. Law Enforcement http://support.sas.com/resources/papers/proceedings12/309-2012.pdf 28 SAS produced an interesting white paper on this space and bought UK firm Memex - definitely chasing this space Citing use cases like - finding individuals - analysing their social graph to find accomplices/gang structure Also identify precursor activity to events like riots
  29. 29. Law Enforcement http://www.policemag.com/blog/technology/story/2012/09/social-media-analytics-in-law-enforcement.aspx 29 “Social media is a huge network of informants—and one you don't have to pay for.” Law Enforcement use cases (information distribution, fake profile creation, etc.) Helps first responders gain situational awareness prior to having feet on the ground Helps Emergency Operations Centres gain information in the event of natural disasters, for example Sunday’s train crash in NY, for example
  30. 30. Law Enforcement http://www.huffingtonpost.com/2012/09/04/web-surveillance-social-media_n_1854750.html X Other vendors in this space outlined in this article 3i-Mind - http://www.3i-mind.com/ HMS Technologies - http://www.hmstech.com/ Visible Technologies - http://www.visibletechnologies.com/ Attensity - http://www.attensity.com/home/ CrowdControlHQ - http://www.crowdcontrolhq.com/index.php and As well as Law Enforcement use cases (information distribution, fake profile creation, etc.)
  31. 31. Law Enforcement http://www.lexisnexis.com/government/investigations/ 30 Good infographic on Law Enforcement use of social media - based on a LexisNexis Risk Solutions survey of 1,200 law enforcement professionals Full report is available at http://solutions.lexisnexis.com/forms/GV12LEOMPSoMeSurveyforLE9677
  32. 32. Law Enforcement 31 Needs to be approached sensitively - the way some of this is reported often prompts visions of ‘pre-crime’ and Minority Report
  33. 33. Smart Cities https://itunes.apple.com/us/app/boston-citizens-connect/id330894558 32 Graffiti, Pothole, & LA school district apps
  34. 34. Healthcare http://www.google.org/flutrends/intl/en_us/ 33 Google use frequency of certain search terms as a way to estimate flu activity Also have one for Dengue Fever Search data is increasingly mobile
  35. 35. Healthcare http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html 34 Google wrote this up as an academic paper and it was published in Nature
  36. 36. Healthcare http://www.ajtmh.org/content/86/1/39.abstract 35 A group led from Harvard Medical School studied viability of using social media for predicting cholera outbreak Found that the data from Twitter closely corresponded with government data, was available up to two weeks earlier The paper concludes informal media could be used to study the activity of other disease outbreaks around the world Financial support was provided by Google.org
  37. 37. Healthcare http://propellerhealth.com 36 Formerly asthmapolis - wireless asthma puff data - where/when Can map where asthma outbreaks occur - people sensitive can avoid triggers
  38. 38. Healthcare http://www.huffingtonpost.com/keith-runyon/louisville-chooses-asthma_b_4086297.html 37 Rolled out in conjunction with city of Louisville Ky Residents experience as much as a 13-year gap in life expectancy depending upon where they live Findings eagerly anticipated - only rolled out in Oct
  39. 39. CRM http://www.ft.com/intl/cms/s/0/bd5a5ce2-aa57-11e1-899d-00144feabdc0.html#axzz2OAlD9lav 38 T-Mobile in US analysed its 33m customer data records, web logs, billing data and social media information to predict customer defections It halved customer defections in 3 months
  40. 40. Brand Management http://www.youtube.com/watch?v=VaJjPRwExO8 39 Nestle were Greenpeace’d because palm oil in Kit Kat came from Sinar Mas - company involved in deforestation
  41. 41. Brand Management http://greenmonk.net/2010/03/19/can-corporate-social-responsibility-affect-your-companys-bottom-line/ 40 In the social media storm which followed Nestlé made every mistake in the book Nestlé received over 200,000 protest emails and their share price was negatively affected So they decided to work with Greenpeace to fix their supply chain and To initiate a social media strategy for the organisation
  42. 42. Brand Management http://uk.reuters.com/article/2012/10/26/uk-nestle-online-water-idUKBRE89P07Q20121026 41 Set up a social media command centre, staffed by their Digital Acceleration Team
  43. 43. Brand Management http://www.reputationinstitute.com/thought-leadership/global-reptrak-100 42 In 2013 Nestle entered the Reputation Institute’s Global top 10 for the first time.
  44. 44. Transportation https://twitter.com/ehn/status/396307684661530624 43 Waze data now being incorporated in Google Maps
  45. 45. Transportation http://www.bbc.co.uk/news/technology-22357748 44 2.5bn anonymised call records from 5m Orange phone users in Ivory Coast Looked at patterns of people’s movements in Abidjan - capital city of Ivory Coast Realised they could reduce travel times of ppl by 10%
  46. 46. Looking Ahead http://www.flickr.com/photos/35468133931@N01/8699901706 45 Google Glass
  47. 47. Looking Ahead http://www.instabeat.me 46 Instabeat gives swimmers stats in their goggles as they swim And subsequent download
  48. 48. Looking ahead http://www.fitbit.com/force 47 Fitbit force, Nike+ Fuelband, Jawbone Up Can see a situation where sports players are broadcasting vital stats similar to F1
  49. 49. Conclusion Data and data sources are increasing exponentially - go hack that data for good. 48
  50. 50. Thanks! Contact information: Tom Raftery Principal Analyst, Energy & Sustainability, RedMonk Tom@redmonk.com, GreenMonk.net, Twitter.com/tomraftery +34 677 695 468 No tweets were hurt in the making of this presentation 49

×