• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
People as sensors - mining social media for meaningful information

People as sensors - mining social media for meaningful information



The video of this talk is available at https://www.youtube.com/watch?v=4ZdknOPY_jQ ...

The video of this talk is available at https://www.youtube.com/watch?v=4ZdknOPY_jQ

More and more we are all broadcasting information. Geolocation data, “this x sucks” data, weather data, etc.

More and more that data is being parsed and analysed in realtime, such that we have now become sensors.

How does this work, what does this mean, and what risks/benefits will it bring?



Total Views
Views on SlideShare
Embed Views



8 Embeds 70

https://twitter.com 36
http://lanyrd.com 21
http://ec2-54-243-189-159.compute-1.amazonaws.com 6
http://www.feedspot.com 3
http://eventifier.co 1
http://www.linkedin.com 1
http://www.eventifier.co 1
http://eventifier2013.agilealliance.org 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    People as sensors - mining social media for meaningful information People as sensors - mining social media for meaningful information Presentation Transcript

    • People as Sensors Tom Raftery ThingMonk, London Dec 2013. 1
    • Tom Raftery • Lead analyst, energy and sustainability practice, RedMonk • GreenMonk.net • twitter.com/tomraftery • tom@redmonk.com • +34 677 695 468 • SlideShare.net/TomRaftery 2
    • Mobile data http://www.flickr.com/photos/traftery/8551389911/ 3 Mobile phones with GPS, accelerometers, compass, M7, barometers, moisture, etc
    • Mobile data http://www.zeit.de/datenschutz/malte-spitz-data-retention 4 Mobile phones with GPS and data retention laws Green party sued Deutsche Telecom for data Put it online Source of concern
    • Social Media 5 Mobile is not the only data we’re broadcasting
    • Facebook http://newsroom.fb.com/Key-Facts 6 1.19 billion monthly active users (MAU) as of September 30, 2013 727 million daily active users (DAU) in Sept 2013 874 million mobile MAUs as of September 30 2013
    • Facebook http://techcrunch.com/2013/05/17/facebook-growth/ 7 Most recent usage data is from May 2013 - so well out-of-date 4.75 billion content items shared per day (status updates + wall posts + photos + videos + comments) 4.5 billion Likes per day
    • Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 8 Between 4,000-8,500 tweets per second per day Avg over 6,000 tweets per sec, all stored in perpetuity Tweets drop off at midnight ET, start picking up again at 6am ET!
    • Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 9 Increasing tweets per sec over the last year (max value per day used) Two peaks on the right-hand-side
    • Twitter http://hide.dyndns.info/tweetcounter/index-en.cgi 10 Increasing tweets per day over the last year
    • Twitter https://blog.twitter.com/2013/new-tweets-per-second-record-and-how 11 Most recent numbers from Twitter: Twitter avg - >500m TPD Twitter avg - 5,700 TPS Aug 2nd had a 143k TPS record (>28x the average) - no blip to service
    • Data for sale… http://online.wsj.com/news/articles/SB10001424052702304441404579118531954483974 12 Twitter in its IPO filings disclosed it is making $47.5m from selling access to its data
    • Google+ http://googleblog.blogspot.com.es/2013/10/google-hangouts-and-photos-save-some.html 13 Google+ 540m MAU 300m active in stream 1.5bn photos per month
    • Sina Weibo http://sg.finance.yahoo.com/news/sina-weibo-passes-500-million-151054944.html 14 China’s Sina Weibo is growing with 74% year on year user growth (http://www.chinadaily.com.cn/bizchina/2013-02/21/content_16243933.htm) Has 220m ‘active users’ while Twitter has 170m ‘active users’ http://blogs.wsj.com/chinarealtime/2013/03/12/how-many-people-really-usesina-weibo/
    • Waze https://www.waze.com 15 Waze had an estimated 50m users in June 2013
    • Waze https://www.waze.com 16 Waze had an estimated 50m users in June 2013
    • Use Cases 17 Some positive use cases of the data
    • Crowd-Sourcing http://csce.uark.edu/~tingxiny/courses/5013spring13/readingList/crowdsource.pdf 18 Academic study on feasibility of using Twitter to crowdsourced data
    • Meteorology http://uksnowmap.com/#/ 19 UK Snow Map by Ben Marsh Tweet #uksnow, postcode and x/10 rating
    • Utilities http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/ 20 GE’s Grid IQ Insight can mine social media for mentions of outages Gives early notifications of an outage in an area If geotagged and/or includes images/video can confirm cause of outage and speed up time to resolution
    • Utilities http://greenmonk.net/2012/11/01/sustainability-social-media-and-big-data/ 21 Utilities are aware of reason for outage, speeds up time to resolution (reduces need for investigatory truck roll)
    • Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 22 United Nations Development Program & their Recorded Future project Using publicly sourced data looking for signs of disruption or unrest
    • Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 23 The same graph turned into media sources- who is talking about Georgia in this period of time
    • Risk Analysis http://europeandcis.undp.org/blog/2012/11/16/social-media-and-political-risk-analysis/ 24 Mentions turned into social network analysis- who is talking to whom, who is meeting whom “next phase will focus on conducting a regional political risk analysis and forecasting for South Eastern Europe and Central Asia” UNDP slides courtesy of Milica Begovich (aka @elim????)
    • Automotive http://www.magazine.pamplin.vt.edu/fall12/vehicledefects.html 25 Social Media monitoring tool developed by Pamplin College of Business Initial version worked from automotive fora and blogs, now expanding to take in Twitter and Facebook “Robust” way to discover and classify vehicle defects from social media posts across multiple automotive brands Faster than reporting back up through the dealer chain
    • Finance http://arxiv.org/PS_cache/arxiv/pdf/1010/1010.3003v1.pdf 26 Academic paper from University of Manchester and Indiana University shows that Twitter can predict the Dow Jones Industrial Average with 87.6% accuracy
    • Finance http://www.caymanatlantic.com/investment-management/4574471088 27 UK Firm Derwent Capital Markets signed an exclusive deal with the authors to create a hedge fund - became Cayman Atlantic
    • Law Enforcement http://support.sas.com/resources/papers/proceedings12/309-2012.pdf 28 SAS produced an interesting white paper on this space and bought UK firm Memex - definitely chasing this space Citing use cases like - finding individuals - analysing their social graph to find accomplices/gang structure Also identify precursor activity to events like riots
    • Law Enforcement http://www.policemag.com/blog/technology/story/2012/09/social-media-analytics-in-law-enforcement.aspx 29 “Social media is a huge network of informants—and one you don't have to pay for.” Law Enforcement use cases (information distribution, fake profile creation, etc.) Helps first responders gain situational awareness prior to having feet on the ground Helps Emergency Operations Centres gain information in the event of natural disasters, for example Sunday’s train crash in NY, for example
    • Law Enforcement http://www.huffingtonpost.com/2012/09/04/web-surveillance-social-media_n_1854750.html X Other vendors in this space outlined in this article 3i-Mind - http://www.3i-mind.com/ HMS Technologies - http://www.hmstech.com/ Visible Technologies - http://www.visibletechnologies.com/ Attensity - http://www.attensity.com/home/ CrowdControlHQ - http://www.crowdcontrolhq.com/index.php and As well as Law Enforcement use cases (information distribution, fake profile creation, etc.)
    • Law Enforcement http://www.lexisnexis.com/government/investigations/ 30 Good infographic on Law Enforcement use of social media - based on a LexisNexis Risk Solutions survey of 1,200 law enforcement professionals Full report is available at http://solutions.lexisnexis.com/forms/GV12LEOMPSoMeSurveyforLE9677
    • Law Enforcement 31 Needs to be approached sensitively - the way some of this is reported often prompts visions of ‘pre-crime’ and Minority Report
    • Smart Cities https://itunes.apple.com/us/app/boston-citizens-connect/id330894558 32 Graffiti, Pothole, & LA school district apps
    • Healthcare http://www.google.org/flutrends/intl/en_us/ 33 Google use frequency of certain search terms as a way to estimate flu activity Also have one for Dengue Fever Search data is increasingly mobile
    • Healthcare http://www.nature.com/nature/journal/v457/n7232/full/nature07634.html 34 Google wrote this up as an academic paper and it was published in Nature
    • Healthcare http://www.ajtmh.org/content/86/1/39.abstract 35 A group led from Harvard Medical School studied viability of using social media for predicting cholera outbreak Found that the data from Twitter closely corresponded with government data, was available up to two weeks earlier The paper concludes informal media could be used to study the activity of other disease outbreaks around the world Financial support was provided by Google.org
    • Healthcare http://propellerhealth.com 36 Formerly asthmapolis - wireless asthma puff data - where/when Can map where asthma outbreaks occur - people sensitive can avoid triggers
    • Healthcare http://www.huffingtonpost.com/keith-runyon/louisville-chooses-asthma_b_4086297.html 37 Rolled out in conjunction with city of Louisville Ky Residents experience as much as a 13-year gap in life expectancy depending upon where they live Findings eagerly anticipated - only rolled out in Oct
    • CRM http://www.ft.com/intl/cms/s/0/bd5a5ce2-aa57-11e1-899d-00144feabdc0.html#axzz2OAlD9lav 38 T-Mobile in US analysed its 33m customer data records, web logs, billing data and social media information to predict customer defections It halved customer defections in 3 months
    • Brand Management http://www.youtube.com/watch?v=VaJjPRwExO8 39 Nestle were Greenpeace’d because palm oil in Kit Kat came from Sinar Mas - company involved in deforestation
    • Brand Management http://greenmonk.net/2010/03/19/can-corporate-social-responsibility-affect-your-companys-bottom-line/ 40 In the social media storm which followed Nestlé made every mistake in the book Nestlé received over 200,000 protest emails and their share price was negatively affected So they decided to work with Greenpeace to fix their supply chain and To initiate a social media strategy for the organisation
    • Brand Management http://uk.reuters.com/article/2012/10/26/uk-nestle-online-water-idUKBRE89P07Q20121026 41 Set up a social media command centre, staffed by their Digital Acceleration Team
    • Brand Management http://www.reputationinstitute.com/thought-leadership/global-reptrak-100 42 In 2013 Nestle entered the Reputation Institute’s Global top 10 for the first time.
    • Transportation https://twitter.com/ehn/status/396307684661530624 43 Waze data now being incorporated in Google Maps
    • Transportation http://www.bbc.co.uk/news/technology-22357748 44 2.5bn anonymised call records from 5m Orange phone users in Ivory Coast Looked at patterns of people’s movements in Abidjan - capital city of Ivory Coast Realised they could reduce travel times of ppl by 10%
    • Looking Ahead http://www.flickr.com/photos/35468133931@N01/8699901706 45 Google Glass
    • Looking Ahead http://www.instabeat.me 46 Instabeat gives swimmers stats in their goggles as they swim And subsequent download
    • Looking ahead http://www.fitbit.com/force 47 Fitbit force, Nike+ Fuelband, Jawbone Up Can see a situation where sports players are broadcasting vital stats similar to F1
    • Conclusion Data and data sources are increasing exponentially - go hack that data for good. 48
    • Thanks! Contact information: Tom Raftery Principal Analyst, Energy & Sustainability, RedMonk Tom@redmonk.com, GreenMonk.net, Twitter.com/tomraftery +34 677 695 468 No tweets were hurt in the making of this presentation 49