Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data & Smart City Applications


Published on

A talk that was given to share experience in building Smart City applications to the city administrators of Dunedin, NZ.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Big Data & Smart City Applications

  1. 1. Big Data & Smart City Applications: A CityPulse Perspective Put Knoesis Banner Presentation to the Dunedin City Council, Dunedin, NZ. 29 April 2015 Payam Barnaghi Institute for Communication Systems (ICS) University of Surrey Guildford, United Kingdom Amit Sheth Ohio Center of Excellence in Knowledge-enabled Computing (Kno.e.sis) Wright State University, Dayton, OH USA Thanks: Pramod Anantharam
  2. 2. Top organization in WWW: 10-yr Field Rating (MAS) 2
  3. 3. • Social Media Big Data – Twitris, eDrugTrends • Sensor/IoT Big Data – CityPulse, kHealth • Healthcare Big Data – kHealth, EMR, Prediction • Biomedical Big Data – Biomarker from NextGen Sequencing and Proteomics, SCOONER • Big and Smart Data Certificate Kno.e.sis private cloud: 864 CPU cores, 18TB RAM, 17TB SSD, 435TB disk 3
  4. 4. 4 Source LAT Times, Future cities: a view from 1998 Thanks to Dr. Payam Barnaghi for sharing the slide
  5. 5. 5 Image courtesy:[default]/0/ Source: wikipedia Back to the Future: 2013
  6. 6. 6
  7. 7. 7
  8. 8. Enablers of Economic Developments Image credit: Image credit: Economic development on trade routesCivilizations on river banks Economic development now increasingly rely on digital infrastructure 8
  9. 9. Unprecedented Digital Data Growth • Every thing is becoming data driven • Many types of data: Physical, Cyber, and Social • Effective collection and use of this Big Data has to be a core part of designing Smart Cities 9
  10. 10. • Increased citizen participation (Social) • Increase monitoring using sensors (Physical) • Increase Digital Government (eGov) data (Cyber) Understanding wealth of data Let’s not develop future applications with constraints of the past India ranks 8th in civic engagement! 10
  11. 11. Data Lifecycle 11 Source: The IET Technical Report, Digital Technology Adoption in the Smart Built Environment: Challenges and opportunities of data driven systems for building, community and city-scale applications,
  12. 12. Smart City Data Analysis • Analysis of thousands of traffic, pollution, weather, congestion, public transport, waste and event sensory data to provide better transport and city management. • Converting smart meter readings to information that can help prediction and balance of power consumption in a city. • Monitoring elderly homes, personal and public healthcare applications. • Event and incident analysis and prediction using (near) real-time data collected by citizen and device sensors. • Turning social media data (e.g. Tweets) related to city issues into event and sentiment analysis. • Any many more… 12 CityPulse/Payam Barnaghi
  13. 13. Smart City Data • Data is multi-modal and heterogeneous • Noisy and incomplete • Time and location dependent • Dynamic and varies in quality • Crowed sourced data can be unreliable • Requires (near-) real-time analysis • Privacy and security are important issues • Data can be biased- we need to know our data! • Data alone may not give a clear picture -we need contextual information, background knowledge, multi-source information and obviously better data analytics solutions… 13CityPulse/Payam Barnaghi
  14. 14. 14 CityPulse: Large-scale data analytics for smart cities Industrial SIE, ERIC SME AI, Higher Education UNIS, NUIG, UASO, WSU City BR, AA Partners: Duration: 36 months
  15. 15. Designing for City Problems
  16. 16. 16 Data/Event Visualisation
  17. 17. Reference Datasets 17
  18. 18. Importance of Complementary Data 18
  19. 19. Users in control or losing control? 19 Image source: Julian Walker, Flicker
  20. 20. One aspect of characterizing a City: All its functions Image credit: 20
  21. 21. Use cases 21
  22. 22. Scenario ranking 22
  23. 23. 101 Scenarios 23
  24. 24. 101 Smart City Use-case Scenarios 24
  25. 25. 25 Use-case Scenarios
  26. 26. Big (IoT) Data Analytics . . . Real World (Live) Data Smart City Framework Smart City Scenarios
  27. 27. 27 What do we need for developing Smart City Applications? Amit Sheth, Pramod Anantharam, Cory Henson, 'Physical-Cyber-Social Computing: An Early 21st Century Approach,' IEEE Intelligent Systems, vol. 28, no. 1, pp. 78-82, Jan.-Feb., 2013. Physical Cyber Social* Developers need to Consider observations from Physical-Cyber-Social systems in Building Smart City applications *
  28. 28. 28 Physical: Sensors monitoring physical world - Programmable devices - Off-the-shelf gadgets/tools Thanks to Dr. Payam Barnaghi for sharing the slide
  29. 29. 29 Cyber: Observations pushed to the cyber world Thanks to Dr. Payam Barnaghi for sharing the slide
  30. 30. 30 Motion sensor Motion sensor Motion sensor ECG sensor World Wide Web Road block, A3 Road block, A3 Social: People interacting with the physical world Thanks to Dr. Payam Barnaghi for sharing the slide
  31. 31. CityPulse 31
  32. 32. 32 Analytics Toolbox Context-aware Decision Support, Visualisation Knowledge- based Stream Processing Real-Time Monitoring & Testing Accuracy & Trust Modelling Semantic Integration On Demand Data Federation Open Reference Data Sets Real-Time IoT Information Extraction IoT Stream Processing Federation of Heterogenous Data Streams Design-Time Run-Time Testing Exposure APIs
  33. 33. In summary 33 Data:
  34. 34. Public parking space availability prediction • Finding parking space in a city can be challenging • Predicting the probability of parking given various input variables such as scheduled events, time of day & location. • Reduced emission and frustration for citizens 34
  35. 35. Extracting City Events from Social Streams Toward a Citizen Centered Smart City 35
  36. 36. Image credit: Public Safety Urban planning Gov. & agency admin. Energy & water Environmental Transportation Social Programs Healthcare Education Pulse of a City (CityPulse) 36
  37. 37. • Are people talking about city infrastructure on twitter? • Can we extract city infrastructure related events from twitter? • How can we leverage event and location knowledge bases for event extraction? • How well can we extract city events? Research Questions 37
  38. 38. Are People Talking About City Infrastructure on Twitter? 38
  39. 39. Some Challenges in Extracting Events from Tweets • No well accepted definition of ‘events related to a city’ • Tweets are short (140 characters) and its informal nature make it hard to analyze – Entity, location, time, and type of the event • Multiple reports of the same event and sparse report of some events (biased sample) – Numbers don’t necessarily indicate intensity • Validation of the solution is hard due to the open domain nature of the problem 39
  40. 40. Formal Text Informal Text Closed Domain Open Domain [Roitman et al. 2012][Kumaran and Allan 2004] [Lampos and Cristianini 2012] [Becker et al. 2011] [Wang et al. 2012] [Ritter et al. 2012] Related Work on Event Extraction 40
  41. 41. City Infrastructure Tweets from a city POS Tagging Hybrid NER+ Event term extraction Geohashing Temporal Estimation Impact Assessment Event Aggregation OSM Locations SCRIBE ontology hierarchy City Event Extraction City Event Extraction Solution Architecture City Event Annotation 41
  42. 42. • City Event Annotation – Automated creation of training data – Annotation task (our CRF model vs. baseline CRF model) • City Event Extraction – Use aggregation algorithm for event extraction – Extracted events AND ground truth • Dataset (Aug – Nov 2013) ~ 8 GB of data on disk – Over 8 million tweets – Over 162 million sensor data points – 311 active events and 170 scheduled events Evaluation 42
  43. 43. 43 Understanding traffic flow variations
  44. 44. Vehicular traffic data from San Francisco Bay Area aggregated from on-road sensors (numerical) and incident reports (textual) 44 Every minute update of speed, volume, travel time, and occupancy resulting in 178 million link status observations, 738 active events, and 146 scheduled events with many unevenly sampled observations collected over 3 months. Variety Volume VeracityVelocity Value Can we detect the onset of traffic congestion? Can we characterize traffic congestion based on events? Can we provide actionable information to decision makers? semantics Representing prior knowledge of traffic lead to a focused exploration of this massive dataset Big Data to Smart Data: Traffic Management example
  45. 45. Semantic Annotation using Background Knowledge Image Credit: slow-moving-traffic Domain knowledge in the form of traffic vocabulary Domain knowledge of traffic flow synthesized from sensor data 45 Explained-by Horizontal operator: relating/mapping data from different modality to a concept (theme) within a spatio-temporal context; Spatial context even include what it means to have a slow traffic for the type of road
  46. 46. Ground Truth Data (only incident reports) -- City Event Extraction We have around 162 million data records from sensors monitoring over 3,700 links in San Franciso Bay Area <link_id, link_speed, link_volume, link_travel_time,time_stamp>  a data record GREEN – Active Events YELLOW – Scheduled Events 311 active events and 170 scheduled events 46
  47. 47. Evaluation – Extracted Events AND Ground Truth 47
  48. 48. Coordination using Disaters 48
  49. 49. 49 Image: voices-in-a-crowd/ BIG QUESTION: Can these needles be identified in the haystack of massive datasets? Me and @CeceVancePR are coordinating a clothing/food drive for families affected by Hurricane Sandy. If you would like to donate, DM us Does anyone know how to donate clothes to hurricane #Sandy victims? [REQUEST/DEMAND] [OFFER/SUPPLY] Coordination teams want to hear! [BIG] Ad-hoc Community with Varying but [FEW] Important Intents
  50. 50. • May lead to second disaster to be managed: – Under-supply of required demands – Over-supply of not required resources • Hurricane Sandy example, “Thanks, but no thanks”, NPR, Jan 12 2013 Story link: nks-but-no-thanks-when-post-disaster-donations- overwhelm Uncoordinated Engagement
  51. 51. 51 How to volunteer, donate to Hurricane Sandy: <URL> If you have clothes to donate to those who are victims of Hurricane Sandy … Red Cross is urging blood donations to support those affected <URL> I have TONS of cute shoes & purses I want to donate to hurricane victims … Does anyone know how to donate clothes to hurricane #Sandy victims? Does anyone know of community service organizations to volunteer to help out? Needs to get something, suggests scarcity: REQUEST (demand) Offers or wants to give, suggests abundance: OFFER (supply) Matching requests with offers
  52. 52. Want to help animals in #Oklahoma? @ASPCA tells how you can help: x RESPONSE TEAMS (including humanitarian org. and ‘pseudo’ responders) VICTIM SITE Where do I go to help out for volunteer work around Moore? Anyone know? Anyone know where to donate to help the animals from the Oklahoma disaster? #oklaho ma #dogs Matchable Matchable If you would like to volunteer today, help is desperately needed in Shawnee. Call 273-5331 for more info 52 CITIZEN SENSORS DEMAND SUPPLY Match-making: Assisting Coordination Image:
  53. 53. Smart Cities: Opportunities • empower citizens • provide more business opportunities for companies (and SMEs) and private sector services • create better governance of our cities and better public services • provide smarter monitoring and control • improve energy efficiency, create greener environments… • create better healthcare, elderly-care… Thanks to Dr. Payam Barnaghi for sharing the slide 53
  54. 54. Smart Cities: Challenges • We need to know our data and its context (density, quality, reliability, …) • Open Data (adherence by all city departments, more real-time data), Complementary data • Citizens in participation in prioritization and data collection (reliability and quality of citizen reporting of city events) • Transparency and data management issues (privacy, security, trust, …) • Reliability and dependability of the systems 54
  55. 55. Thank you., @amit_p, @pbarnaghi, Acknowledgement: CityPulse Consortium Annual Report: Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing Wright State University, Dayton, Ohio, USA
  56. 56. 56