Your SlideShare is downloading. ×
UN Global Pulse: Big Data for a Better World (Strata Conf NYC)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

UN Global Pulse: Big Data for a Better World (Strata Conf NYC)

4,132
views

Published on

Presentation by UN Global Pulse at the Strata Big Data conference in New York, October 2012. http://strataconf.com/stratany2012/public/schedule/detail/24956

Presentation by UN Global Pulse at the Strata Big Data conference in New York, October 2012. http://strataconf.com/stratany2012/public/schedule/detail/24956


0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,132
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
137
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. “BEYOND TARGETED ADSBIG DATA FOR A BETTER WORLD” Robert Kirkpatrick Director, UN Global Pulse O’Reilly Strata Conference | New York | October 2012 www.unglobalpulse.org @unglobalpulse
  • 2. 25  October  2012  |  www.unglobalpulse.org  
  • 3. Kenyan Farm Workers Photo Credit: John Oyuke25  October  2012  |  www.unglobalpulse.org  
  • 4. Hyperconnectivity Airplane Flights Telephone Calls Internet Traffic Social Networking25  October  2012  |  www.unglobalpulse.org  
  • 5. 20th-Century tools…25  October  2012  |  www.unglobalpulse.org  
  • 6. PRIVATE SECTOR Monitor operations…in real time Track market trends…in real timeGet customer feedback…in real time GLOBAL DEVELOPMENT Unemployment? Food Security? Public Health? Education? Migration? Disaster Relief?
  • 7. BIG DATA IN REAL TIME: 3 OPPORTUNITIES 1.  Better early warning: Earlier detection of anomalies, trends and events allows earlier response. 2.  Real-time awareness: A more accurate and up-to- date picture of population needs supports more effective planning and implementation 3.  Real-time feedback: Understanding sooner where needs are changing -- or are not being met -- allows for rapid, adaptive course correction25  October  2012  |  www.unglobalpulse.org  
  • 8. BIG DATA IS A HUMAN RIGHTS ISSUE •  Never analyze personally identifiable information •  Never analyze confidential data •  Never seek to re-identify individuals25  October  2012  |  www.unglobalpulse.org  
  • 9. BACKGROUND: A GROWING BODY OF EVIDENCERESEARCH
  • 10. How Mobile Phone Carriers See the World Call Detail Records (CDRs) •  Caller ID (hashed phone #) •  Caller Tower Location •  Receiver ID (hashed phone #) •  Receiver Tower Location •  Call Start Time •  Call Duration Airtime Expense Records •  Caller ID (hashed phone #) •  Caller Tower Location •  Amount of Purchase •  Time of Purchase •  Balance at Time of Purchase25  October  2012  |  www.unglobalpulse.org  
  • 11. Modeling Behaviors in Mobile Data Consumption variables • Number of calls, call duration, SMS/MMS/voice • Size, frequency, total number of airtime purchases • Handset Type and Features Social variables • Degree of the social network • Weight of the contacts, frequency of communication Mobility variables • Diameter of mobility and social network • Radius of gyration • Mobility Patterns Source:  Telefonica  Research,  2011  25  October  2012  |  www.unglobalpulse.org  
  • 12. EXAMPLE: AIRTIME CREDIT PURCHASE DATA25  October  2012  |  www.unglobalpulse.org  
  • 13. SIZE AND FREQUENCY PREDICT HOUSEHOLD INCOME Higher household income Average size of purchase Lower household income Average # of purchases / month25  October  2012  |  www.unglobalpulse.org  
  • 14. CALLING PATTERNS AND ECONOMIC OPPORTUNITY Lower Higher socioeconomic level socioeconomic level25  October  2012  |  www.unglobalpulse.org  
  • 15. MEN AND WOMEN USE THEIR PHONES DIFFERENTLY Men: Women: •  Fewer calls •  More calls •  Shorter calls •  Longer calls •  Smaller social network •  Larger social network •  More work-related calls •  More personal calls25  October  2012  |  www.unglobalpulse.org  
  • 16. Tracking population movement to predict cholera Source: Linus Bengtsson et. al., PLoS Medicine, 201125  October  2012  |  www.unglobalpulse.org  
  • 17. A mobility index to evaluate H1N1 response in Mexico City Source: Telefonica Research, 2011 See: http://www.unglobalpulse.org/publicpolicyandcellphonedata25  October  2012  |  www.unglobalpulse.org  
  • 18. TWITTER PREDICTS SPREAD OF INFLUENZA r2 = .958 “You Are What You Tweet: Analyzing Twitter for Public Health. M. J. Paul and M. Dredze, 2011.” http://www.cs.jhu.edu/%7Empaul/files/2011.icwsm.twitter_health.pdf25  October  2012  |  www.unglobalpulse.org  
  • 19. Rumi Chunara et. al., American Journal of Tropical Medicine and Hygiene, 2012 86:39-4525  October  2012  |  www.unglobalpulse.org  
  • 20. GOOGLE SEARCHES FOR SYMPTOMS PREDICT DENGUE25  October  2012  |  www.unglobalpulse.org  
  • 21. 2010 VS. 2011: INDONESIAN TWEETS ABOUT HIV See: http://www.unglobalpulse.org/WorldAIDSDay-Part225  October  2012  |  www.unglobalpulse.org  
  • 22. GLOBAL PULSE RESEARCH 2011PROOF OF CONCEPTSTUDIESOnline at: http://www.unglobalpulse.org/applyingbigdatatodevelopment
  • 23. http://www.unglobalpulse.org/projects/can-social-media-mining-add-depth-unemployment-statistics
  • 24. 25  October  2012  |  www.unglobalpulse.org  
  • 25. Online Discussions & Unemployment Ireland25  October  2012  |  www.unglobalpulse.org  
  • 26. Online Discussions & Unemployment United States25  October  2012  |  www.unglobalpulse.org  
  • 27. http://www.unglobalpulse.org/projects/twitter-and-perceptions-crisis-related-stress
  • 28. Jakarta: nine million tweets per day Map  of  Twi*er  usage  in  Jakarta  –  by  Eric  Fischer    25  October  2012  |  www.unglobalpulse.org  
  • 29. Tweets per day about food, during Ramadan in Indonesia Start of Ramadan End of Ramadan25  October  2012  |  www.unglobalpulse.org  
  • 30. Tweets predict food basket inflation (rice, chilies, fish, sugar, corn, cooking oil) Tweets about the price of rice (per month) Official Food Price Inflation (monthly from 25 cities)25  October  2012  |  www.unglobalpulse.org  
  • 31. ABOUTGLOBAL PULSE
  • 32. DIGITAL SERVICES AS HUMAN SENSOR NETWORKS: Observing fluctuations in well-being…in real-time COPING STRATEGIES DIGITAL “SMOKE SIGNALS” •  Buy cheaper foods •  Depletion of airtime credit •  Work longer hours •  Smaller mobile airtime •  Reduce energy use purchases •  Draw down savings •  Failure to repay microloans via •  Sell assets mobile financial services •  Borrow from relatives •  Changes in calling patterns •  Inbound money transfers •  Web searches for jobs, health •  Sales of livestock via mobile trading network •  “Venting” on social media25  October  2012  |  www.unglobalpulse.org  
  • 33. Photo: Ministry of Foreign Affairs, Iceland25  October  2012  |  www.unglobalpulse.org  
  • 34. AGILE GLOBAL DEVELOPMENT?25  October  2012  |  www.unglobalpulse.org  
  • 35. Integrating real-time data into an institution •  This data may be less accurate that official sources. •  But it’s faster. •  And it’s cheaper to collect. •  How can we leverage USGS Twitter Earthquake Detector the speed to change (TED) the outcome?25  October  2012  |  www.unglobalpulse.org  
  • 36. THE PROBLEM WITH TELESCOPES… …AND MACROSCOPES There’s a universe of data that we can’t see.25  October  2012  |  www.unglobalpulse.org  
  • 37. 25  October  2012  |  www.unglobalpulse.org  
  • 38. Data Philanthropy?
  • 39. A global real-time public/privatedata commons?
  • 40. EXAMPLE R&D PROJECT: Mobile Networks as Drought Sensors in the Sahel Proposal •  Obtain 2011-2012 mobile CDRs and airtime purchases. •  Derive mobility, consumption, and social variables. •  Correlate variables with precipitation levels, survey data. •  Identify signatures of drought impacts in 2011. •  Identify signatures of aid impact in 2012. •  Develop and evaluate prototype during next drought. •  Release open source “appliance” through GSMA.25  October  2012  |  www.unglobalpulse.org  
  • 41. PULSE LABS
  • 42. Joint Research | Rapid Prototyping | Capacity Building25  October  2012  |  www.unglobalpulse.org  
  • 43. Pulse Lab Network Pulse Lab Jakarta…………October 2012 Pulse Lab Kampala……….January 2013 Other locations…………...???25  October  2012  |  www.unglobalpulse.org  
  • 44. PULSE LABS R&D INNOVATION STRATEGY 1.  Partner with governments to establish Pulse Labs 2.  Build world-class teams of data scientists, engineers, and policy experts 3.  Partner with private sector for real-time data and cutting edge technology 4.  Work with UN agencies and academia to conduct research around challenges in 5.  Build open source prototypes of tools to automatic real-time monitoring 6.  Support broad adoption of useful innovations 7.  Share everything we learn and build25  October  2012  |  www.unglobalpulse.org  
  • 45. SO HOW DO I GET INVOLVED? ARE YOU.. •  A company with powerful data you think could make the world a better place? •  A technology provider with screaming fast computing or killer analytics? •  A whiz data scientist interested in hard problems, positive impact, and global scale? •  A big data privacy expert who understands that we cannot help unless we also protect?25  October  2012  |  www.unglobalpulse.org  
  • 46. 25  October  2012  |  www.unglobalpulse.org  
  • 47. Research Tool 1 Crimson Hexagon: ForSight  25  October  2012  |  www.unglobalpulse.org  
  • 48. Food Prices: What a real crisis looks like 23  July  -­‐  2  Aug   ‘tempeh’  and  ‘tofu’  hot  debate   14  -­‐  21  Aug     Ramadhan  /  Idul  Fitri  25  October  2012  |  www.unglobalpulse.org  
  • 49. Comparing Crises Tweets  about  food   18  Mar  -­‐  7  Apr   23  July  -­‐  2  Aug  Fuel  subsidy  cut  plan  and   ‘tempeh’  and  ‘tofu’  hot  debate   protests  against  it   during  soybean  shortage  
  • 50. Research Tool 2 SAS Social Media Analytics and SAS Text Miner  25  October  2012  |  www.unglobalpulse.org  
  • 51. Analytic Workflow1)  Over  200,000  new     5)  Explore  results  and  correlate   3)  Capture  senKment   with  official  staKsKcs  to  official  Indonesian  language   and  mood  for  Bahasa     BPS  staKsKcs  :  Consumer  Price  documents  per  day   Index  (CPI)  for  12  common  foods   Global Pulse Sentiment, Topic & Internet Relevance Mood & Geography Interactive Conversation Filter Influence Categories Dashboard 2)  Extract  conversaKons   4)  Detect  locaKon,  price,   about  rice,  cooking  oil,   availability,  specific   fuel,  employment,  etc.   govt.  programs,  etc.    anxious,      confident,    confused,      hosKle,      sad,      happy   (-”-) ;-) ((+_+)) :@ :( :)
  • 52. What’s the deal with Indonesians and eggs?   For every 5000 more tweets about eggs… …we see a 2-3% decrease in food CPI?
  • 53. The Signals Are Getting Stronger   è Big increase in volume of relevant conversations over 18 months 40000   35000   minyak  (oil)   ketahanan  pangan  (food  security)   30000   budidaya  (culKvaKon)   25000   telur  (eggs)   20000   15000   10000   5000   0   Indonesians are increasingly using social media to discuss basic needs25  October  2012  |  www.unglobalpulse.org  
  • 54. So Are the Temporal Correlations   è Listening to social conversations provides insight on official data 2.5   2   1.5   1   0.5   0   -­‐0.5   -­‐1   -­‐1.5   Social  Media  Food  Index   -­‐2   BAPPENAS  Food  Price  Index   -­‐2.5  25  October  2012  |  www.unglobalpulse.org  
  • 55. Next up for Pulse Lab Jakarta research: 1 year of anonymized Indonesian CDRs •  4 largest carriers •  170 million subscribers •  200 billion call records •  80 terabytes of data25  October  2012  |  www.unglobalpulse.org  
  • 56. 25  October  2012  |  www.unglobalpulse.org  
  • 57. ROBERT KIRKPATRICKDirectorUN Global Pulsewww.unglobalpulse.orgkirkpatrick@un.org+1 (650) 796-5709 Image credit: Aaron Koblin 24 hours of AT&T phone calls and Internet traffic flowing through New York City