Advertisement

More Related Content

Slideshows for you(20)

Similar to Big Data, open data, IOT(20)

Advertisement

More from ConfindustriaFirenze(20)

Recently uploaded(20)

Advertisement

Big Data, open data, IOT

  1. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Big Data & Analitycs e CyberSecurity, July 2018 http://www.disit.org/ Paolo Nesi, paolo.nesi@unifi.it https://www.Km4City.orghttps://www.snap4City.org Big Data, Open data, IOT 1
  2. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 The data! 2
  3. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Real time private Real time public (open data) Static Public (open data)Private and static statistics: accidents, census, votations • Fiscal Code, SSN • Non shared pictures • Level aspects • Patient health record • .. Smart City Course, October 2018 3
  4. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Km4City in Tuscany Area Road Graph (Tuscany region) 132,923 Roads , 389,711 Road Elements 318,160 Road Nodes, 1,508,207 Street Numbers Info on: points, paths, areas, etc. Services (20 cat, 512 cat.) 16 Public Transport Operators 21.280 Bus stops & 1081 bus lines Dynamic/real-time in Tuscany Region • Real time bus lines: 144 updates X day X line • 1081 Transport Pub Lines: 1-2 up per day, time-path • >210 parking lots status: 76 updates X day X sensor • >796 traffic Sensors: 288 updates X day X sensor • 285 weather area: 2 updates X day X area • >12 hospital Triage status: 96 updates X day X FA • 22 Environmental data: 20 updates X day X sensor • 39 Bike Sharing data: Pisa and Siena • 12 Pollination data • 140 recharging stations • Smart benches, waste mng, irrigators, lighting,… • Florence ent.events: about 60 new events X day • Different kinds of Florence traffic events, • [1600 Fuel stations: 1 update X day X station] • Wi-Fi: > 400.000 measures X day • App mobiles: >50.000 measures X day • more than 40.000 distinct users X day • From 600.000 to 4.5 M Tweets X day • many IOT sensors ……http://servicemap.km4city.org 4
  5. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 5V of Big Data Variety Volume VariabilityVelocity Value 5V’s of Big Data When data are BIG data? The excel file size? 5
  6. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Cloud Infrastructure Km4CitySmartCityAPI Knowledge Base ETL Processes, Data Analytic, R; IOT App; etc. Data Processing Tools Development and Management Tools ETL Processes Resource Manager DataGate/ CKAN Km4City Ontology Phoenix, Hbase + indexing Big Data Storage Knowledge IoT/IoE Applications AMMA Linked Open Graph ServiceMap Data Flow Analysis DevDash Elastic Management of Containers Mobile and Web Apps Final Users’ Tools Dashboards Social Media IoT/IoE Open Data Personal Data Industry 4.0 GIS + Map Data IOT / IOE Apps IOT Directory Management Authentication, Authorization, GDPR, Security Assessment Powered by Smart City Course, October 2018 6
  7. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 Services from Data via Smart City API 7
  8. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Km4City in Tuscany Area What is enabling and providing smart services • Smart Parking, in Tuscany • Smart First Aid in Tuscany • Smart Fuel pricing in Tuscany • Smart search for POI and public transport srv. • Public Transportation in Tuscany • Routing in Tuscany • Social Media Monitoring and acting • Traffic events and Resilience in Florence • Bike Sharing in Pisa and Siena • Recharge stations for e-vehicles • Entertainment Events in Florence • Traffic Sensors in Tuscany • Weather forecast/condition in Tuscany • Pollution and Pollination in Tuscany • People Monitoring Assessment in the City, in Florence via WiFi • People Monitoring, in Tuscany via App All Point of Interests, cultural activities, IOT, … Over than 1.2 Million of complex events per day!http://servicemap.km4city.org 8
  9. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Scenarious vs SmartCity API Smart City Course, October 2018 9 • Search data: by text, near, along, etc... – Resolving text to GPS and formal city nodes model • Empowering the city users • Access to event information • Supporting City Users in using Public Mobility • Supporting City Users in using Private Mobility • New Experience to access at Cultural and Touristic info • New way to access at health services • Access at Environmental information • Profiled Suggestions to City Users • Personal Assistant • Sharing knowledge among cities
  10. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org • Areas, Bus lines, bike lanes, tram, RTZ, etc. Smart City Course, October 2018 Km4City in Firenze 10
  11. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Along a Line Smart City Course, October 2018 Into an Area 11
  12. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Supporting City Users in using Public Mobility Smart City Course, October 2018 • Public Transport, PT – Getting tickets – Getting bus stops, lines, and timelines for bus, train and tramline (GTFS, ETL, ..) – Searching Services along a Pub. Transport line or closer to a stop – Searching the closest bus stops – searching for BUS stops via name – real time delays of busses – modal routing for Pub. Transport 12
  13. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Supporting City Users using Private Mobility Smart City Course, October 2018 • Private Transport – Parking status (DATEX II, …) – Getting closer parking – Getting parking forecast – Getting closer free space on parking – Getting fuel stations location and fuel product prices – Getting bike sharing rack status – Searching Services along a path or closer to a point or Service as Hotel, Restaurants, square, etc. – Getting closer cycling paths – Recharging stations: location and status – Getting traffic information 13 13
  14. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Private Mobility: routing and navigation paths Smart City Course, October 2018 • To get the path from two points/POIs: –Shortest for pedestrian –Quietest for pedestrian –Shortest for private vehicles • Search for POIs along the identified Path! • http://www.disit.org/ServiceMap 14
  15. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org New way to access at health services Smart City Course, October 2018 • Searching for pharmacies and hospitals • Getting the closest hospital first aid locations and status • Getting real time updated information about the first aid status of major hospitals (triage) 15
  16. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Access at Environmental information Smart City Course, October 2018 • Getting weather forecast for the next hours and days • Getting alert information from Civil protection • Getting air quality status • Getting pollination status • getting actual weather status: temperature, humidity, pressure, rain level, • etc. 16
  17. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org IOT Applications and IOT Smart City Course, October 2018 17
  18. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Level 4 users: dashboard with intelligence App • Dashboards with IOT Applications for enforcing smart and intelligence into them. DashboardsIOT and City data World IOT Applications My IOT Devices Dashboard-IOT App Smart City Course, October 2018 18 Applications
  19. IOT Edge on the field IOT Devices IOT Edge With IOT App distributed Sensors/ Actuators Sensors/ Actuators Sensors/ Actuators Sensors/ Actuators Sensors/ Actuators Sensors/Actuators Raspberry pi -- PC: Win, Linux Android IOT Directory (1) Registration (2) Production of IOT App on IOT edge (3) Production of Dashboard user interface (4) IOT App and its Dashboard are executed (0) Sensors & Actuators Internet IOT App can be executed on IOT Edge and/or on Cloud. MicroServices calling Dashboards, Storage and Analytics are executed on Cloud. On Cloud Other Connected IOT App Smart City Course, October 2018 19
  20. Node.js Blocks on NodeRed SotA Smart City Course, October 2018 20
  21. Snap4City MicroServices Smart City Course, October 2018 21
  22. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Personal Data vs Open Smart City Course, October 2018 22
  23. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org • Manage Profile and MyPersonalData • For each Data Type: – Start as private → making them public (anonymous) and revoke – The Owner is the only one that can: (1) modify values; (2) change the ownership – Define/revoke Delegation to Access – Delete/forget per Data Type and “me all!”. – Auditing GDPR Compliant Smart City Course, October 2018 23
  24. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Managing MyPersonalData in secure manner Smart City Course, October 2018 24 Examples: • 1) Social IOT: A group of friends share some data with other according to GDPR: GPS position, Medical parameters as Glucose, etc. • 2) saving and retrieve personal sensitive information. The users manage their Personal data via personal mobile Dash and IOT App, and configuration on the portal and/or Mobile App
  25. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Managing MyPersonalData in secure manner Smart City Course, October 2018 25 Example: • Piero shares some data with selected friends according to GDPR: GPS position • He managed the data via personal mobile Dashboard and IOT Application Encrypted Data Storage Smart City Services and IOT/IOE
  26. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Big Data Analytics Smart City Course, October 2018 26
  27. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking space trends Smart City Course, October 2018 12 parking areas in Florence 28
  28. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking PREDICTIONS Smart City Course, October 2018 • Active on Apps – «Firenze dove cosa» – «Toscana dove cosa» Careggi car park Model features BRNN model results R-squared RMSE MASE Baseline 0.974 24 1.87 Baseline + Weather 0.975 24 1.75 Baseline + Traffic sensors 0.975 24 2.04 Baseline + Weather + Traffic sensors 0.975 24 1.87 29
  29. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org User Behaviour Analysis Smart City Course, October 2018 • Monitoring movements by traffic flow sensors – Spires and virtual spires • Monitoring movements from Mobile Cells – Unsuitable for precise tracking and OD production • Monitoring movements from Wi-Fi • Monitoring movements and much more from mobile Apps 30
  30. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Predicting Models for Admin. & City Users Smart City Course, October 2018 • Aiming at improving – quality of service, distributing workload – early warning • Predictions: Short (15 min, 30 Min) and mid Term (1 week) • Data Analytics: ML, NLP/SA, Clust… – Traffic Flows → multiflow reconstruction – Parking Status → free slots – People Flows (WiFi, Twitter) → crowd , #number of people Predicting at EXPO2015 Early Warning Water Bomb Early Warning Hot in Tuscany Predicting City Users on Areas 31
  31. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Characterizing City Areas Smart City Course, October 2018 Predicting City Areas Crowd level characterizing Users’ BehaviorsWi-Fi based 32
  32. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 Prediction and identification of anomalies Cluster confidence AP average and confidence Actual AP trend for today AP prediction for the next time slot in the day on the basis of past weeks Guessing number of users of Wi-Fi Access Points 33
  33. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Origin Destination Matrix Estimation Wi-Fi based DISIT lab overview, January 2017, Florence
  34. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 Traffic Flow Tools • Spire and Virtual Spires (cameras), Bluetooth, .. • Specifically located: along, around, .. • Traffic Tuscany 35
  35. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 Traffic Flow reconstruction, real time http://firenzetraffic.km4city.org 36
  36. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Routing and Multimodal Routing • Pedonal • Veichles • Public Multimodal • Delivering Smart City Course, October 2018 37
  37. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Routing • Pedonal – Normal – Quite Smart City Course, October 2018 38
  38. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org App as data collection and User Engagement Smart City Course, October 2018 39
  39. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Km4City APP • Smart Parking, in Tuscany • Smart First Aid in Tuscany • Smart Public Transportation in Tuscany • Smart Fuel pricing in Tuscany • Bike Sharing in Pisa • Weather condition in Tuscany • Pollution and Pollination in Tuscany • Traffic Sensors in Tuscany • Smart Routing in Tuscany • Smart Transportation in Florence • Events, traffic, … • Entertainment Events in Florence Smart City Course, October 2018 40
  40. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 41
  41. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org App as data collection and Engagement Smart City Course, October 2018 42
  42. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance • http://www.disit.org/tv • http://www.disit.org/rttv • Citizens as sensors to – Assess sentiment on services, events, … – Response of consumers wrt… – Early detection of critical conditions – Information channel – Opinion leaders – Communities – Formation – Predicting volume of visitors for tuning the services Smart City Course, October 2018 43
  43. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Prediction/Assessment • Football game results as related to the volume of Tweets • Number of votes on political elections, via sentiment analysis, SA • Size and inception of contagious diseases • marketability of consumer goods • public health seasonal flu • box-office revenues for movies • places to be visited, most visited • number of people in locations like airports • audience of TV programmes, political TV shows • weather forecast information • Appreciation of services Smart City Course, October 2018 44
  44. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance Smart City Course, October 2018 Early Warning Predictive models Hot flows Attendance at long lasting events: EXPO2015 Attendance at recurrent events: TV, footbal 45
  45. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Visual Analytics and Dashboards Smart City Course, October 2018 46
  46. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 47
  47. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 48
  48. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 49
  49. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org The Living Lab Approach Smart City Course, October 2018 50
  50. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Connect IOT/IOE Upload context Open Data Connect external Services Advanced Smart City API Run Applications & Dashboard Monitor City Platform Life Cycle experiments workshops tutorials networking agreements events Start-ups Research groups City Users City Operators Large Industries collaborations Licensing, Gold services personal services Case Studies Inhouse companies Resource Operators Tech providers partnerships documentation Help desk Category Associations Corporations Advertisers Community Building subscription to applications Produce City Applications & Dashboard Promote Applications & Dashboards Produce Applications for City Users Set Up: ETL & Data Analytic algorithms Collaborative Platform Early Adopters Snap4City Smart City Course, October 2018 51
  51. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Mobile e Web Apps Tools for Final Users Km4City Smart City Engine Transport systems Mobility, parking Km4CitySmartCityAPI Public Services Govern, events, … Sensors, IOT Cameras, .. Environment, Water, energy Social Media WiFi, network DISCES--DistributedandparallelarchitectureonCloud Shops, services, operators Km4City Big Data Analytics Smartening Tools Development Tools Recommender Personal Assistant Http://www.km4city.org Smart Decision Support Twitter VigilanceServiceMap browser Analyzers of City User Behavior Dashboards City Operators and Decision Makers Smart City Course, October 2018 52
  52. IOT Directory Back Office Processes IOT Broker IOT Broker IOT Broker IOT Broker ETL Process Data Analytics ETL Process ETL Process ETL Process Data Analytics Data Analytics Data Analytics Knowledge Base, Km4City Smart City API from Knowledge Base and other tools Ontology SPARQL, FLINT LOG.disit.org ServiceMap ServiceMap3D Swagger MicroServices IOT ApplicationsWeb and Mobile AppsDISCES and back office management tools Smart City Course, October 2018 53 MicroApplications Resource Manager
  53. Smart City Course, October 2018 54
  54. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 55
  55. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Course, October 2018 56
  56. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence
  57. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Predicting Models for Admin. & City Users • Aiming at improving – quality of service, – distributing workload – early warning • Traffic Flows & People Flows → crowd , #number of people • Parking Status → free slots • Weather Forecast (LAMMA) DISIT lab overview, January 2017, Florence Predicting at EXPO2015 Early Warning Water Bomb Early Warning Hot in Tuscany Predicting City Users on Areas
  58. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Predicting City users movements • Issue: – How they move: vehicles, pedestrian, bike, ferry, metro, – Where they go…. • Impact: – Tuning the services: cleaning, police, control, security • Several metrics related to – Knowledge of the city – Monitoring traffic and people flow – ……. DISIT lab overview, January 2017, Florence .000 50.000 100.000 150.000 200.000 0:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 • Daily trends • OD matrices • Trajectories • Prediction models
  59. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org User Behaviour Analysis DISIT lab overview, January 2017, Florence • Monitoring movements by – traffic flow sensors – Spires and virtual spires • Monitoring users’ movements – from Mobile Cells – Unsuitable for precise tracking and OD production • Monitoring movements via Wi-Fi • Monitoring movements and much more from mobile Apps
  60. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Social Media 61DISIT lab, Sii-Mobility, Km4City, 1st June 2017
  61. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  62. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Twitter Vigilance Analysis Global View (private data)
  63. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org in Numbers• Used by several users: – UnivFirenze, LAMMA, IBIMET, ARPAT, Master on Big Data, … • Active since Aprile 2015 • 3 platforms for automated: – Daily collection: statistical direct analysis and sentiment analysis – Real time collection and statiscal, sentiment analysis – Full faceted indexing: thus enabling search on collected tweets • All: precomputation of basic metric opening the activities of deep analysis • More than 24 million of tweets in the storage: ready on Hadoop cluster • More than 150 channels • More than 450 search activities daily • From 400.000 to 4 Million of tweets per day. DISIT lab overview, January 2017, Florence
  64. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance • http://www.disit.org/tv • http://www.disit.org/rttv • Citizens as sensors to – Assess sentiment on services, events, … – Response of consumers wrt… – Early detection of critical conditions – Information channel – Opinion leaders – Communities – Formation – Predicting volume of visitors for tuning the services DISIT lab overview, January 2017, Florence
  65. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Sentiment Analysis Real Time Twitter Vigilance, Early Warning
  66. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgPrediction/Assessment • Football game results as related to the volume of Tweets • Number of votes on political elections, via sentiment analysis, SA • Size and inception of contagious diseases • marketability of consumer goods • public health seasonal flu • box-office revenues for movies • places to be visited, most visited • number of people in locations like airports • audience of TV programmes, political TV shows • weather forecast information • Appreciation of services DISIT lab overview, January 2017, Florence
  67. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Predicting Audience on Social intensive TV show • Issue: – How to predict the number of people following a TV reality show in life • Impact: – Making Advertising, promotion – Valorizing advertising – Adjusting the show • Several metrics related to – Structure of volume of TW, RTW – Features of the tweet authors – Relationships …. DISIT lab overview, January 2017, Florence • Periodic events • Specific rules • Strong influence and user engagment • Audience can vote • Audience espress appreciation and rejects • .. Similar to the presence at large and log terms event, such as EXPO2015
  68. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgPredicting Audience: X-Factor, PechinoExpress… • Trend of TW and RTW for X-Factor 9 – Several searches • Similar model for other Social Intensive TV shows – See below Pechino Express DISIT lab overview, January 2017, Florence 𝑥𝑡 = 𝛽1 𝑧1,𝑡 + 𝛽2 𝑧2,𝑡 + 𝛽3 𝑧3,𝑡 + ⋯ + 𝛽 𝑘 𝑧 𝑘,𝑡 + 𝑛
  69. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Details of Predictive Models Validities Metrics collected over the 5 days before the event. X-Factor 9 - Model Pechino Express - Model Coeff Std Err t-val p-val Coeff Std Err t-val p-val Total number of tweets + retweets on main hashtag 𝛽1 -73.48 58.49 -1.256 0.2494 -954.3 64.69 -14.750 0.0045 Total number of tweets on main hashtag, 𝛽2 122.7 70.27 1.745 0.1244 4144 284 14.590 0.0046 Ratio between: number of RTW/TW on main hashtag, 𝛽3 135885 1 462704 2.937 0.0218 937920 80946 11.590 0.0073 UnqURetweet 𝛽4 264.3 153 1.728 0.1277 2175 345.6 6.293 0.0243 FUnqUsers 𝛽5 -214.9 132.5 -1.622 0.1488 -1640 270.6 -6.061 0.0261 Intercept 𝑛 -762730 627238 -1.216 0.2634 -2560461 401675 -6.374 0.0237 R squared 0.727 0,995 RMSE 66467 8851 MAE 55589 6805 AIC 340 182 TV broadcasting company Sky RAI Weeks 13 9 millions of registered tweets on Twitter Vigilance 1.625 0.455
  70. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Predicting Confidence DISIT lab overview, January 2017, Florence
  71. Predicting EXPO2015 DISIT lab overview, January 2017, Florence Twitter Vigilance on EXPO2015 channel
  72. Twitter Metrics • TW: Number of Tweets per Search/Channel (as called Volume) , per day, per hour • RTW: Number of ReTweets per Search/Channel, per day, per hour • NRT/TW: ratio from ReTweets and Tweets per Search/Channel, per day, per hour • NumSearch: number of Tweets including the Search per Channel, per day, per hour • Sentiment Analysis Score per Search/Channel, per day, per hour • Num of xxxxx DISIT lab overview, January 2017, Florence
  73. Twitter Vigilance monitoring and predictions DISIT lab overview, January 2017, Florence Twitter Vigilance on EXPO2015 channel Predizioni al 90% Predicting volume of visitors for tuning the services
  74. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Predicting the reTweet Proneness • Issue: – How to understand if a tweet has a good probability of being retweeted? • Impact: – Advertising, promotion, training • Several metrics related to – Structure of the tweet – Features of the tweetting author – Relationships …. DISIT lab overview, January 2017, Florence Twitter Analytics
  75. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Tweet proneness Metrics Tweet metrics URLs Count # of URLs in the tweet Mentions Count # of mentions/citation of Twitter users in the tweet Hashtags Count # of hashtags included in the tweet Favorites Count # of favorite obtained by the tweet Publication Time Local hour H24 in which the tweet has been published in the day according to the author’ local time. Author of Tweet metrics Days Count # of days since the tweet’s author created its Twitter account Statuses Count # of tweets made by the tweet’s author since the creation of its own account Author Network metrics Followers Count # of followers the author of the tweet Followees Count # of friends the tweet’s author is following Listed Count # of people added the tweet’s author to a list DISIT lab overview, January 2017, Florence Data sets: • 100 Million of Tweet • 500 K • 100 K
  76. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it reTweet proneness: assessment • PCA DISIT lab overview, January 2017, Florence Metrics PC1 PC2 PC3 PC4 PC5 Retweet Count -0.1623 0.4346 0.1635 -0.0026 -0.1009 Favorites Count -0.6294 0.3908 0.1922 -0.1128 -0.1880 Followers Count -0.7599 0.2736 0.0522 -0.0983 -0.0857 Followees Count -0.1336 -0.0907 -0.4627 -0.2494 0.1182 Listed Count -0.8431 -0.1549 -0.0498 0.1500 0.1871 Statuses Count -0.4256 -0.5016 -0.3781 0.2795 0.2410 Hashtags Count -0.1585 -0.5661 0.4377 -0.0517 0.0309 Mentions Count 0.0394 0.2194 0.0786 -0.1607 0.7697 URLs Count -0.1288 -0.5483 0.2539 -0.3388 -0.3248 Publication Time 0.0076 -0.0728 0.3639 -0.5186 0.3707 Days Count -0.0370 0.0070 -0.5072 -0.6604 -0.1691
  77. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it reTweet proneness: Classification methods • Statistic classifications vs machine-learning methods • 80% of training data set, 20% of testing data sets; 500K data set • → Recursive partitioning procedure models (RPART), good compromise for Big data problems DISIT lab overview, January 2017, Florence Classifier Models Accuracy Precision Recall 𝐅𝟏 score Processing Time in sec. Recursive Partitioning (Stat) 0.6807 0.8512 0.7767 0.8122 180 Random Forests (ML) 0.6884 0.8601 0.7866 0.8217 198968 Gradient boosting (ML) 0.6796 0.8534 0.7731 0.8113 64448 Multinomial Model (Stat) 0.6411 0.8367 0.7245 0.7765 31576
  78. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it reTweet proneness (RPART), 100M Assessment drivers Degree of Retweeting Classes 0 1-100 101-1000 1001-10000 Over 10000 Sensitivity 0.7737 0.8105 0.3142 0.0208 0.0136 Specificity 0.9132 0.6694 0.9199 0.9996 1.0000 Positive Predictive Value 0.8564 0.6256 0.3752 0.7345 0.8488 Negative Predictive Value 0.8579 0.8382 0.8975 0.9485 0.9915 Prevalence 0.4007 0.4053 0.1328 0.0526 0.0086 Detection Rate 0.3100 0.3285 0.0417 0.0011 0.0001 Detection Prevalence 0.3620 0.5251 0.1112 0.0015 0.0001 Balanced Accuracy 0.8435 0.7399 0.6170 0.5102 0.5068 DISIT lab overview, January 2017, Florence Accuracy 0.6815 Accuracy 95% Confidential Interval (min, max) (0.6813, 0.6817) Recall 0.7737 Precision 0.8564 Kappa 0.4922
  79. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Predictive models VS metrics relevance DISIT lab overview, January 2017, Florence 00% 10% 20% 30% 40% 50% 60% 70% 80% Variable Importance between Models Random Forests Gradient Boosting Multinomial Recursive Partitioning
  80. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Early warning, detection • Issue: – Detection of critical condition – Not easily detected with other means • Impact: – Early warning, faster reaction – Increased resilience • Several metrics related to – Volume of retweets – Sentiment analysis DISIT lab overview, January 2017, Florence City Resilience damage Prepare Absorb Recover Adapt
  81. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence City Resilience ERMG 30 years probability Arno flooding200 years probability Arno flooding Water bomb (down burst) in South FlorenceArno Flood Impact on Tram Line & Traffic
  82. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Twitter Vigilance and Water Bomb DISIT lab overview, January 2017, Florence Early Warning
  83. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring via Mobile App 84DISIT lab, Sii-Mobility, Km4City, 1st June 2017 http://www.km4city.org/?controlRoom http://www.km4city.org/?devTools http://www.km4city.org/?infoDocs http://www.km4city.org/?app
  84. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  85. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Mobile Computing • Smart City Problems: – Reaching the users – Understanding the user preferences and behavior – Understating how they move, where they go, etc.. • Solutions: – Monitoring the activities on the mobile device – Monitoring the activities of user in the environment • Technologies for Solutions: – Assessing the usage of Smart city and services – Integrated Indoor/outdoor navigation • Routing, multimodal routing – Content distribution: e-learning – User networking and collaboration – OS: iOS, Android, Windows Phone, etc. – Tech: IOT, iBeacoms, NFC, QR, …. DISIT lab overview, January 2017, Florence
  86. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Km4City APP, features 1/3 • 5 languages: IT, EN, SP, DE, FR • city users: citizens, commuter, students, Tourists, etc.. • Profiled menu for POI, different for different City users • Personalized main menu • Search Textual • Search for POI, POI kind, etc.. – Close to you, close to a point • Direct searches – Events, green areas, public transport, – Cycling paths, Parking (NEW: triage, fuel station) – Etc. • POI sharing and contributing – Preferred, Social icon connection – Ranking, Comments, images DISIT lab overview, January 2017, Florence
  87. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Km4City APP, features 2/3 • Mobility – Lines, bus stops, schedule – Parking status – Tickets for Busses – Cycling paths – Fuel stations • Personal Assistant – Info and help – Engagement – Civil protection, alerts – Hospital triage status • Suggestions: – Personalized and adaptive: banned & profiles for each users. – POI, Twitter hints, Events, – Weather forecast – …… DISIT lab overview, January 2017, Florence
  88. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgKm4City APP, features 3/3 • Navigation 3D (BETA) • App as a tool for city user behavior analysis – Measuring Wifi status: power distribution of Free Wi-Fi AP – Detection and measure of Beacon – Computing User Behavior • Fluxes of people via APP, GPS: • OD matrix • Fluxes out of Tuscany and more • Producing Engagements • Producing Multimodal Routing paths DISIT lab overview, January 2017, Florence
  89. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Heat Map from Mobile: users as sensors
  90. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org User Behavior Analyzer for Collective profiling DISIT lab overview, January 2017, Florence Who When What Where? Why? How move Where they go ahead
  91. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Anonymous User Behavior Analysis How city users are moving Mobile App based DISIT lab overview, January 2017, Florence
  92. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Problems of Trajectories from Apps • From mobile app: – Resolving GPS location: GPS, cells, wifi-network, ..mixt – Noisy, different kind of devices, .. – Smart algorithm on devices for location acquisition – Anonymized data, terms of use on mobile • Issues and Filtering – Gps Accuracy, kind of measure (GPS, mixt) – Jump in time, space, velocity – General noise (diff. devices) – Knowledge of precision map • Clustering: time, space, user kind, etc. DISIT lab overview, January 2017, Florence • X area, x user type • Velocity,, Direction • Time, acceleration • Vehicle kind ?? • Record and Replay, …
  93. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Cluster di Trajectories
  94. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Heat Map from Mobile: users as sensors DISIT lab overview, January 2017, Florence
  95. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Heat Map from Mobile: users as sensors DISIT lab overview, January 2017, Florence
  96. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org User Behavior Analyzer Mobile App based DISIT lab overview, January 2017, Florence
  97. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence OD Matrix scalabile
  98. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring Traffic Flow 99DISIT lab, Sii-Mobility, Km4City, 1st June 2017 http://www.km4city.org/?controlRoom
  99. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  100. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgTraffic Flow Tools • Spire and Virtual Spires (cameras), Bluetooth, .. • Specifically located: along, around, .. • Traffic Tuscany Pisa DISIT lab overview, January 2017, Florence
  101. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, Sii-Mobility, Km4City, 1st June 2017 RealTime Values 3D http://www.disit.org/servicemap3d Real time Showing: - traffic flow - People flow - Free Parking slots - Water level, rain, etc. - Sensors values…. 102
  102. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, Sii-Mobility, Km4City, 1st June 2017 103 Traffic Flow data
  103. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab, Sii-Mobility, Km4City, 1st June 2017 104 Traffic Flow Reconstruction http://www.disit.org/siimobilitytrafficflow2/
  104. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring Parking status 105DISIT lab, Sii-Mobility, Km4City, 1st June 2017 http://www.km4city.org/?controlRoom
  105. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  106. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org 104 Parking in Tuscany DISIT lab overview, January 2017, Florence
  107. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking space trends DISIT lab overview, January 2017, Florence Pieraccini Meyer, Careggi Beccaria S. Lorenzo 12 parking areas in Florence
  108. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking space trends DISIT lab overview, January 2017, Florence Stazione Fortezza Fiera
  109. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking space trends DISIT lab overview, January 2017, Florence Careggi S. Lorenzo
  110. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Parking PREDICTIONS • Active on • «Firenze dove cosa» • «Toscana dove cosa» DISIT lab overview, January 2017, Florence Careggi car park Model features BRNN model results R-squared RMSE MASE Baseline 0.974 24 1.87 Baseline + Weather 0.975 24 1.75 Baseline + Traffic sensors 0.975 24 2.04 Baseline + Weather + Traffic sensors 0.975 24 1.87
  111. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Free Space on Parking lots DISIT lab overview, January 2017, Florence
  112. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring City users Via Wi-Fi 113DISIT lab, Sii-Mobility, Km4City, 1st June 2017 http://www.km4city.org/?controlRoom http://www.km4city.org/?devTools http://www.km4city.org/?infoDocs http://www.km4city.org/?app
  113. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  114. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Monitoring City usage via Wi-Fi http://wifimap.km4city.org/
  115. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org • Instrumenting Wi-Fi – 1500 AP → 345 instrumented – Stream from AP to DISIT Lab – Real time monitoring → dashboard • Data Analytics – heat maps – Analysis of user behavior – Clustering user behavior – Predictive models about user behavior – Identification of critical conditions, anomalies DISIT lab overview, January 2017, Florence Monitoring City usage via Wi-Fi
  116. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Wi-Fi Monitor tool Recency and frequency
  117. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Real Time Monitoring of Wi-Fi network DISIT lab overview, January 2017, Florence
  118. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgUser Behavior Analysis DISIT lab overview, January 2017, Florence Distinct APs: 343 Distinct APs (last 24 hours): 311 Distinct Users (last 180 days): 1102098 Distinct Excursionists (last 180 days, < 24 h): 687025 Recency Where Excursionists New City Users VS Returning
  119. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgDistribution in the first 24 hours DISIT lab overview, January 2017, Florence 0 50000 100000 150000 200000 250000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 4 means: permanence for more than 3 hours and less than 4 hours
  120. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Origin Destination Matrix Estimation Wi-Fi based DISIT lab overview, January 2017, Florence
  121. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Characterizing City Areas Predicting City Areas Crowd level characterizing Users’ BehaviorsWi-Fi based DISIT lab overview, January 2017, Florence
  122. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Lunedi-Venerdi Sabato Domenica Clustering e Modelli Predittivi
  123. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Predizione e identificazione anomalie DISIT lab overview, January 2017, Florence Cluster confidence AP average and confidence Actual AP trend for today AP prediction for the next time slot in the day on the basis of past weeks Guessing number of users of Wi-Fi Access Points
  124. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.orgClustering of City Usage of Wi-Fi • Approaches: K-Means, K-Medoids (PAM), ….. • Assessment Methods: ELBOW, GAP • Identification of the most suitable K DISIT lab overview, January 2017, Florence 800 1300 1800 2300 2800 1 3 5 7 9 11 13 15 17 19 ELBOW K-means ELBOW PAM 0.75 0.85 0.95 1.05 1.15 1 3 5 7 9 11 13 15 17 19 GAP K-… GAP PAM 0 20000 40000 60000 80000 100000 120000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 EII VII EEI VEI VEE VVE
  125. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring via Cellular Data 127DISIT lab, Sii-Mobility, Km4City, 1st June 2017
  126. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Overview• Analisi Dati raccolti via Social Media – Predicting presences • Analisi Dati raccolti via Mobile App – Tracce, matrici OD, heatmap – Regency e frequency – Impatto in uscita da Firenze • Analisi Dati raccolti dai Flussi di traffico – Ricostruzione del traffico in punti non misurati • Analisi Dati raccolti dai Parcheggi – Predizione dei posti liberi • Analisi Dati raccolti via Firenze WiFi – Tracce, matrici OD, heatmap – Predicting presences • Analisi Dati Dati raccolti via Cellular – Valutazione comparativa TIM-VODA – Valutazione comparativa FirenzeWiFi-Tim-VodaDISIT lab overview, January 2017, Florence
  127. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Letture possibili • Andamenti giornalieri • Andamenti settimanali • Matrici OD (Voda) • Heatmap di TIM (250 metri) non paiono precise • Differenze sulle fasce orarie • Differenze sul calcolo del numero delle presenze, mediate • ….. DISIT lab overview, January 2017, Florence
  128. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence
  129. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence
  130. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Aree ACE
  131. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Confronto TIM-Vodafone Correlazione "00-06" "06-12" "12-18" "18-24" media 0,6350 0,8948 0,8965 0,8576 var 0,0720 0,0271 0,0227 0,0220 mediana 0,7214 0,9471 0,9515 0,9182 DISIT lab overview, January 2017, Florence -Su tutte le aree ACE - Probabili differenze su residenti !
  132. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org • Andamento della correlazione nel tempo • Varie fasce orarie DISIT lab overview, January 2017, Florence -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 august augustaugustaugustaugust august august august august august august august august august august august august august august august august august july july july july july july july july july july july july july july july july july julyjulyjulyjuly july junejunejunejune june june june june june june june june june june june june june june june june june june may may may may may may may may may may may may may may may may maymaymaymay "00-06" "06-12" "12-18" "18-24" Confronto TIM-Vodafone
  133. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Confronto TIM-Vodafone • Volumi del numero delle persone per ogni fascia oraria sovrapponibile e per ogni mese – Maggio, Giugno, Luglio, Agosto – Settembre non e‘ completo DISIT lab overview, January 2017, Florence Vodafone tim Voda/Tim Media, Numero utenti (non distinti) mese 8.183.017 15.036.991 0,582 varianza 2,27549E+12 4,83645E+12 0,009
  134. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi DISIT lab overview, January 2017, Florence 23 21 20 22
  135. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze - Sunday June 4 2017 23:43:57 • Distinct APs: 368 • Distinct APs (last 24 hours): 288 • Distinct Users (last 180 days): 1009929 • Distinct Users (last 180 days, < 24 h): 563200 DISIT lab overview, January 2017, Florence
  136. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi DISIT lab overview, January 2017, Florence
  137. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Mobile App, Km4City DISIT lab overview, January 2017, Florence
  138. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi number of events per day DISIT lab overview, January 2017, Florence 000 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 450,000 500,000 5/d/yyyy 0:00 6/d/yyyy 0:00 7/d/yyyy 0:00 8/d/yyyy 0:00 9/d/yyyy 0:00 10/d/yyyy 0:00 11/d/yyyy 0:00 12/d/yyyy 0:00 1/d/yyyy 0:00 2/d/yyyy 0:00 3/d/yyyy 0:00 4/d/yyyy 0:00 5/d/yyyy 0:00 Events
  139. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi number of unique users per day DISIT lab overview, January 2017, Florence 000 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 50,000 5/d/yyyy 0:00 6/d/yyyy 0:00 7/d/yyyy 0:00 8/d/yyyy 0:00 9/d/yyyy 0:00 10/d/yyyy 0:00 11/d/yyyy 0:00 12/d/yyyy 0:00 1/d/yyyy 0:00 2/d/yyyy 0:00 3/d/yyyy 0:00 4/d/yyyy 0:00 5/d/yyyy 0:00
  140. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Firenze Wi-Fi permanenze da 1 a xxx - Valutazione a 180 gg - 50% circa sta meno di 24 ore
  141. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi nuovi arrivi a Firenze DISIT lab overview, January 2017, Florence
  142. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence Firenze WiFi Cosa accade nelle prime 24 ore - Valutazione a 180 gg - Quanto sta il 50% circa che sta meno di 24 ore - Cosa fa ? → si puo’ vedere dalle App
  143. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi • AP → heatmap sparsa • Inflow/outflow • New/Old users • per fascia oraria DISIT lab overview, January 2017, Florence
  144. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Firenze Wi-Fi DISIT lab overview, January 2017, Florence • AP → matrice OD sparsa • Inflow/outflow • New/Old users • Flussi per fascia oraria
  145. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org 0 0.2 0.4 0.6 0.8 1 1.2 20 20 20 21 21 21 22 22 22 23 23 23 luglio agosto settembre luglio agosto settembre luglio agosto settembre luglio agosto settembre "00-06" 06-12 12-18 18-24 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 20 20 20 21 21 21 22 22 22 23 23 23 luglio agosto settembre luglio agosto settembre luglio agosto settembre luglio agosto settembre "00-06" 06-12 12-18 18-24 Firenze Wi-Fi vs cell • Correlazione nella fascia 6-12 • Scarsa correlazione se vi sono pochi AP e pochi dati, oltre 350.000 eventi si ha una correlazione elevata – Settembre • In 22 vi sono pochi AP • Scarsa correlazione per fasce 18-24, 0-6 dove l’incidenza di residenti e’ elevata sui cellulari • Vodafone presenta una maggior correlazione per la presenza di un mino numero di residenti DISIT lab overview, January 2017, Florence Vodafone TIM R-squared R-squared
  146. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org considerazioni • Dati telefonici: – Catturano sia residenti che turisti senza una forte distinzione – Cons: Non permettano di tracciare matrici origine destinazione – Cons: Sono ad una risoluzione troppo bassa: 15 minuti → 6 ore – Pros: valutano anche fuori dall’area urbana • Dati Firenze WiFi: – Catturano principalmente i turisti e movimenti in strada – Permettono di fare matrici OD solo se la rete e’ ben instrumentata – Forte correlazione con dati delle reti cellulari negli orari centrali – Cons: non lavorano fuori dall’area ubana DISIT lab overview, January 2017, Florence
  147. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org END DISIT lab overview, January 2017, Florence
  148. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Early warning, detection • Issue: – Detection of critical condition – Not easily detected with other means • Impact: – Early warning, faster reaction – Increased resilience • Several metrics related to – Volume of retweets – Sentiment analysis DISIT lab overview, January 2017, Florence City Resilience damage Prepare Absorb Recover Adapt
  149. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DISIT lab overview, January 2017, Florence City Resilience ERMG 30 years probability Arno flooding200 years probability Arno flooding Water bomb (down burst) in South FlorenceArno Flood Impact on Tram Line & Traffic
  150. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Overview • Decision Support System • Predicting User Movements • Early Warning, Diagnosi precoce • Mobile User Behavior Analysis • Altri Campi di Applicazione DISIT lab overview, January 2017, Florence
  151. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Inform You have parked out of your residential parking zone The Road cleaning is this night The waste in S.Andreas Road is full Engage Provide a comment, a score, etc.. Stimulate / recommend Events in the city, services your may be interested, etc.. Provide Bonus Since you have parked here you we can get 1 Bonus We suggest you to leave the car out of the city, this bonus can be used to buy a bus ticket Any Mobile and Web App City & City Operators Strategy Editor DISIT lab overview, January 2017, Florence
  152. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Overview • Decision Support System • Predicting User Movements • Early Warning, Diagnosi precoce • Mobile User Behavior Analysis • Altri Campi di Applicazione DISIT lab overview, January 2017, Florence
  153. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Smart Factory, Factory 4.0 • Frontman (Novicrom) – Improving efficiency into the production process via a set of heterogeneous numerical control machines • Green Capacity (ALTAIR) – Optimizing chemical plant, automating maintenance and control in large chemical plant, dashboarding • Indoor/outdoor navigation system for maintenance • → → costs reduction, increase efficiency DISIT lab overview, January 2017, Florence
  154. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Smart Retail • Feedback Project, from Feb 2017 – Flexible Advanced Engagement Exploiting User Profiles and Product/Production Knowledge – VAR, PatriziaPepe (Tessilform), DISIT, Effective Knowledge, SICE – Keywords: retail, GDO, … • Goals and drivers: – adaptive user engagement, customer experience – Advanced user profiling, user behavior analysis – Predictive models for engagement – IOT and instrumentation – Integrated incity customer experience DISIT lab overview, January 2017, Florence
Advertisement