• Snap4City Architecture
• Snap4City: Smart City IOT as a Service
• Snap4City Living Lab For Collaborative Work
• Smart City Development Life Cycle
• Analysis and Design for Innovation (Co-Creation and Co-Working)
• Development Tools
• How to Add Functions that are not present in the Platform
• Snap4City vs Fi-Ware
• Snap4City vs State of the Art Solutions
• Snap4City Services: Consulting and Developing
• Snap4City vs Snap4Industry 4.0
• Installing Snap4City
• The view of the Administrator
• Monitoring Resource Consumption and Traffic
• Managing and Monitoring Data Traffic in the BackOffice
• Auditing Activities
• Managing Back Office processes via Containers
• Acknowledgement
Snap4City November 2019 Course: Smart City IOT development of IOT ApplicationsPaolo Nesi
• IOT Applications, Devices and Dashboards
– Managing IOT Applications
– Authoring IOT Applications
– IOT App vs Data Analytic
– IOT App vs Web Scraping
– IOT Apps Examples
• From Simple to Data Processing IOT Applications
– Create a Simple IOT Application (Demo)
– Production of IOT Application (Exercitation)
– Data Processing with IOT Application (Demo)
– Processing Data with IOT Applications (Exercitation)
• IOT Network Support
– Proprietary IOT Devices as Well as Open hardware / Open Software
• IOT end-2-end Secure Stack
OSGi for European and Japanese smart cities - experiences and lessons learnt ...mfrancis
OSGi Community Event 2016 Presentation by Levent Gurgen (CEA)
Internet of Things (IoT) is the digital skin of the physical world. It has specific requirements such as dynamicity to self-adapt to the continuously changing physical context. The world is heterogeneous and the objects should interoperate to collaborate, thus interoperability is essential. Openness and short learning curve are other requirements so that innovators (e.g. startups) can rapidly build applications with reduced time-to-market and avoid vendor lock-in. Reuse of software and hardware is also particularly important since billion of devices are expected to be deployed in the coming decades and those devices should be multi-purpose and reusable by applications from different domains and not be specific to a given domain.
OSGi has – since 15 years ago – the answers to those requirements of today’s IoT. Its powerful run-time environment for the dynamicity, its service-oriented approach hiding heterogeneity, its modularity making the reuse extremely simple, and last but not least, its open approach giving the potential to democratize the IoT.
The talk will illustrate the benefits of OSGi for IoT with concrete deployed examples, in particular in smart city domain in Europe and Japan via the collaborative projects such as ClouT and FESTIVAL.
FIWARE Global Summit - Factory Shop Floor Digitalization using FogFlowFIWARE
Presentation by Deepak Kumar Gupta (General Manager, NEC Technologies India Pvt Ltd.) and Anand Sahu (Senior Technical Manager, NEC Technologies India Pvt Ltd.)
FIWARE Global Summit
23-24 October 2019 - Berlin, Germany
CONTENTS
1. Why OASC? Martin Brynskov, Aarhus University,
Chair OASC
2. OASC mechanisms, Juanjo Hierro, Telefonica, Chief
Architect of FIWARE, OASC task force.
3. City of Antwerp, Prof. Pieter Ballon, Director Living
Labs, iMinds, OASC task force
4. City of Tampere, Seppo Haataja, Director
InnovaPon programmes, OASC Director
5. Hostabee, Vincent DemorPer, Hostabee, FI-C3 A16
6. The Porto FIWARE Ecosystem, Rui Costa, Ubiwhere
7. Q&A
Snap4City November 2019 Course: Smart City IOT development of IOT ApplicationsPaolo Nesi
• IOT Applications, Devices and Dashboards
– Managing IOT Applications
– Authoring IOT Applications
– IOT App vs Data Analytic
– IOT App vs Web Scraping
– IOT Apps Examples
• From Simple to Data Processing IOT Applications
– Create a Simple IOT Application (Demo)
– Production of IOT Application (Exercitation)
– Data Processing with IOT Application (Demo)
– Processing Data with IOT Applications (Exercitation)
• IOT Network Support
– Proprietary IOT Devices as Well as Open hardware / Open Software
• IOT end-2-end Secure Stack
OSGi for European and Japanese smart cities - experiences and lessons learnt ...mfrancis
OSGi Community Event 2016 Presentation by Levent Gurgen (CEA)
Internet of Things (IoT) is the digital skin of the physical world. It has specific requirements such as dynamicity to self-adapt to the continuously changing physical context. The world is heterogeneous and the objects should interoperate to collaborate, thus interoperability is essential. Openness and short learning curve are other requirements so that innovators (e.g. startups) can rapidly build applications with reduced time-to-market and avoid vendor lock-in. Reuse of software and hardware is also particularly important since billion of devices are expected to be deployed in the coming decades and those devices should be multi-purpose and reusable by applications from different domains and not be specific to a given domain.
OSGi has – since 15 years ago – the answers to those requirements of today’s IoT. Its powerful run-time environment for the dynamicity, its service-oriented approach hiding heterogeneity, its modularity making the reuse extremely simple, and last but not least, its open approach giving the potential to democratize the IoT.
The talk will illustrate the benefits of OSGi for IoT with concrete deployed examples, in particular in smart city domain in Europe and Japan via the collaborative projects such as ClouT and FESTIVAL.
FIWARE Global Summit - Factory Shop Floor Digitalization using FogFlowFIWARE
Presentation by Deepak Kumar Gupta (General Manager, NEC Technologies India Pvt Ltd.) and Anand Sahu (Senior Technical Manager, NEC Technologies India Pvt Ltd.)
FIWARE Global Summit
23-24 October 2019 - Berlin, Germany
CONTENTS
1. Why OASC? Martin Brynskov, Aarhus University,
Chair OASC
2. OASC mechanisms, Juanjo Hierro, Telefonica, Chief
Architect of FIWARE, OASC task force.
3. City of Antwerp, Prof. Pieter Ballon, Director Living
Labs, iMinds, OASC task force
4. City of Tampere, Seppo Haataja, Director
InnovaPon programmes, OASC Director
5. Hostabee, Vincent DemorPer, Hostabee, FI-C3 A16
6. The Porto FIWARE Ecosystem, Rui Costa, Ubiwhere
7. Q&A
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...Paolo Nesi
Open Source and inter-operable tools to: (i) keep city under control via personalized dashboards - monitoring services status of city operators; - monitoring and understanding the city users behaviour; - collecting moods, contributions and data from the city users; - monitoring social media for city services and events, event predictions; (ii) improve city resilience, reducing risks and decision support by: - assessing city resilience level; - improving city resilience, providing objective hints; - improving city users awareness with personal city assistants and participatory tools; (iii) transform data in value for the city: - enabling commercial and business applications; - aggregating multi-domain data and services for SMEs and city operators; - enabling integrated city services into third party web portal for all; - providing suggestion on demand services for SMEs and city operators; - accelerating and simplifying the implementation of business and service oriented Apps. Follow the Km4City City Smartener Process
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...FIWARE
Presentation by Antonio Jara (CEO, HOP Ubiquitous S.L.) and Iris Cuevas (Head of Laboratory, HOP Ubiquitous S.L.)
FIWARE Global Summit
21-22 May 2019 - Genoa, Italy
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbiasymbiote-h2020
Presentation of the H2020 symbIoTe project during the IoT Week Belgrade (http://iot-week.eu/events/iot-week-belgrade/), Serbia. More info at www.symbiote-h2020.eu.
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...symbiote-h2020
Presentation of the H2020 symbIoTe project during the "Industry of Internet of Things: Experience, challenges and evolution" Workshop at SpliTech 2016 (http://splitech2016.fesb.hr/iot/), Split, Croatia. More info at www.symbiote-h2020.eu.
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece symbiote-h2020
Presentation of the H2020 symbIoTe project during the AIOTI Open Day, at the National Documentation Centre, Athens, Greece. More info at www.symbiote-h2020.eu
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...symbiote-h2020
Presentation of the H2020 symbIoTe project during the 25th Edition of the European Conference on Networks and Communications (EuCNC; http://www.eucnc.eu/), Athens, Greece. More info at www.symbiote-h2020.eu.
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWAREFIWARE
Architecting Your Smart Solution Powered by FIWARE Webinar - 24 June 2020
Corresponding webinar recording: https://youtu.be/pTkZk4VF0gY
Learn how Smart Solutions solving a particular challenge are architected with FIWARE.
Chapter: Cross-Domain
Difficulty: 2
Audience: Any Technical
Presenter: Antonio Jare (CEO, HOPU) and (CTO, FIWARE Foundation)
This infographic presents the partnership between TM Forum and FIWARE, and how it eases the development of innovative digital services for the citizens and municipalities and enables the creation of a Digital Single Market that can reach, engage and profit different urban communities around the world.
Live with IOT (Borys Pratsiuk Technology Stream) IT Arena
Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneur and startuppers. Annually it takes place on 2-4 of October in Lviv at the Arena Lviv stadium. In 2015 conference gathered more than 1400 participants and over 100 speakers from companies like Facebook. FitBit, Mail.ru, HP, Epson and IBM. More details about conference at itarene.lviv.ua.
Palestra IoT - Internet of Things - Internet das CoisasLuiz Oliveira
Internet of Things, Internet das Coisas, Real world internet, RWI, SENSEI, Smartsantander, zigbee, 802.15.4, 6lopan, internet do futuro, future internet, domotic, domotica, automation, automação.
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...Paolo Nesi
• Recall on Snap4City Architecture
• Dashboard Usage and Authoring
– Dashboard usage vs Widgets
– Multi Data Map Widget
– Unified Data and Services Inspection (data inspector)
– Data Inspector vs Data Processes Details
– From Data to Graphic Widgets
– GIS Data Import, Export and Exploitation
• Dashboard Creation & Exercitation
– Dashboard Creation, Exercitation
– Synoptics and Custom Widgets Creation
– Dashboards Intelligence on Web and Mobile Devices
– Dashboard ChatRoom and Notifications
• Data Type Management GDPR Compliant
– Dashboard Listing, Managing, Sharing, Delegation
– Dashboards GDPR Management
– Personal Data Management GDPR Compliant: POI, Data, KPI
• Snap4City and Km4City Projects
• Acknowledgments
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...Paolo Nesi
• Smart City API: Internal and External
• Forging and Managing Flexible Mobile Apps, Web Apps and Micro Apps
– Web and Mobile App with Open Development Kit
– Understanding how City User are using the City Services
– Engaging City Users, towards a participated attitude
• Advanced Smart City API, MicroServices, Snap4City API
• Federated Knowledge Base and Smart City API
• Web and Mobile App Development Kit
• Acknowledgement
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Paolo Nesi
• Overview
• Urban Platform (main concepts vs Living Lab)
• Snap4City Architecture, roadmap, logos, innovations
• Dashboards: from City Dashboards to Applications
• Trajectories and real time tracking
• Dashboards Intelligence and web and mobile devices
• Dashboard chatrooms and notifications
• Smart City Control Room
• Dashboards production
• Data Gathering and City Data Knowledge Management
• Protocol vs Data
• Data Gathering processes
• GIS Data Import, Export and Exploitation
• Semantic Modeling and City Knowledge Base: Km4CIty
• IOT Applications, Devices and Dashboards
• IOT Devices
• Forging & Managing Flexible Mobile Apps, Web App, MicroApplications
• Web and Mobile App with Open Development Kit
• Understanding how city users are using the city services
• Engaging City Users Towards Virtuous Behaviour
• Data Analytic, Big Data Science
• Data Analytics: predictions
• Smart Parking: predictions
• User behaviour Analysis via Wi-Fi, OD Matrices, Trajectories
• Recognition of Used Transportation Means
• Traffic Flow Reconstruction, from traffic sensors data
• Quality of Public Transport
• Origin Destination Matrices
• Demand of Mobility vs Offer of Transportation
• Modal and Multimodal Routing for Navigation and Travel Planning
• Environmental Data Predictions
• Prediction of Qir Quality
• Anomaly Detection
• Environmental data prediction
• Social Media Analysis
• Snap4City Living Lab for Collaborative Work
• Development Life Cycle
• Development tools
• Data protection, personal da vs GDPR
• Snap4City and Km4City Projects
• Acknowledgment
Snap4City has been created in response to Select4Cities PCP (http://www.select4cities.eu/) call as an open, standardized, data-driven, service-oriented, user-centric platform enabling large-scale co-creation IOT/IOE applications and services for Helsinki, Copenhagen and Antwerp.
Snap4City is a fully open source, robust, scalable, easy to use solution, provides tools for co-creation of mixt data driven, stream and batch processing, extending the powerful semantic reasoner of Km4City https://www.km4city.org, with IOT/IOE, GDPR, and city dashboards.
Km4City has been validated in multiple devices (PC, Android, Raspberry, ..), and domains: mobility and transport, tourism, health, welfare, social and cities such as Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and million of data per day.
The innovation is mainly related to semantic reasoning, IOT interoperability, microservices, automated dashboard production, .. thus setting up smart city solutions in a snap
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...Paolo Nesi
Open Source and inter-operable tools to: (i) keep city under control via personalized dashboards - monitoring services status of city operators; - monitoring and understanding the city users behaviour; - collecting moods, contributions and data from the city users; - monitoring social media for city services and events, event predictions; (ii) improve city resilience, reducing risks and decision support by: - assessing city resilience level; - improving city resilience, providing objective hints; - improving city users awareness with personal city assistants and participatory tools; (iii) transform data in value for the city: - enabling commercial and business applications; - aggregating multi-domain data and services for SMEs and city operators; - enabling integrated city services into third party web portal for all; - providing suggestion on demand services for SMEs and city operators; - accelerating and simplifying the implementation of business and service oriented Apps. Follow the Km4City City Smartener Process
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...FIWARE
Presentation by Antonio Jara (CEO, HOP Ubiquitous S.L.) and Iris Cuevas (Head of Laboratory, HOP Ubiquitous S.L.)
FIWARE Global Summit
21-22 May 2019 - Genoa, Italy
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbiasymbiote-h2020
Presentation of the H2020 symbIoTe project during the IoT Week Belgrade (http://iot-week.eu/events/iot-week-belgrade/), Serbia. More info at www.symbiote-h2020.eu.
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...symbiote-h2020
Presentation of the H2020 symbIoTe project during the "Industry of Internet of Things: Experience, challenges and evolution" Workshop at SpliTech 2016 (http://splitech2016.fesb.hr/iot/), Split, Croatia. More info at www.symbiote-h2020.eu.
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece symbiote-h2020
Presentation of the H2020 symbIoTe project during the AIOTI Open Day, at the National Documentation Centre, Athens, Greece. More info at www.symbiote-h2020.eu
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...symbiote-h2020
Presentation of the H2020 symbIoTe project during the 25th Edition of the European Conference on Networks and Communications (EuCNC; http://www.eucnc.eu/), Athens, Greece. More info at www.symbiote-h2020.eu.
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWAREFIWARE
Architecting Your Smart Solution Powered by FIWARE Webinar - 24 June 2020
Corresponding webinar recording: https://youtu.be/pTkZk4VF0gY
Learn how Smart Solutions solving a particular challenge are architected with FIWARE.
Chapter: Cross-Domain
Difficulty: 2
Audience: Any Technical
Presenter: Antonio Jare (CEO, HOPU) and (CTO, FIWARE Foundation)
This infographic presents the partnership between TM Forum and FIWARE, and how it eases the development of innovative digital services for the citizens and municipalities and enables the creation of a Digital Single Market that can reach, engage and profit different urban communities around the world.
Live with IOT (Borys Pratsiuk Technology Stream) IT Arena
Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneur and startuppers. Annually it takes place on 2-4 of October in Lviv at the Arena Lviv stadium. In 2015 conference gathered more than 1400 participants and over 100 speakers from companies like Facebook. FitBit, Mail.ru, HP, Epson and IBM. More details about conference at itarene.lviv.ua.
Palestra IoT - Internet of Things - Internet das CoisasLuiz Oliveira
Internet of Things, Internet das Coisas, Real world internet, RWI, SENSEI, Smartsantander, zigbee, 802.15.4, 6lopan, internet do futuro, future internet, domotic, domotica, automation, automação.
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...Paolo Nesi
• Recall on Snap4City Architecture
• Dashboard Usage and Authoring
– Dashboard usage vs Widgets
– Multi Data Map Widget
– Unified Data and Services Inspection (data inspector)
– Data Inspector vs Data Processes Details
– From Data to Graphic Widgets
– GIS Data Import, Export and Exploitation
• Dashboard Creation & Exercitation
– Dashboard Creation, Exercitation
– Synoptics and Custom Widgets Creation
– Dashboards Intelligence on Web and Mobile Devices
– Dashboard ChatRoom and Notifications
• Data Type Management GDPR Compliant
– Dashboard Listing, Managing, Sharing, Delegation
– Dashboards GDPR Management
– Personal Data Management GDPR Compliant: POI, Data, KPI
• Snap4City and Km4City Projects
• Acknowledgments
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...Paolo Nesi
• Smart City API: Internal and External
• Forging and Managing Flexible Mobile Apps, Web Apps and Micro Apps
– Web and Mobile App with Open Development Kit
– Understanding how City User are using the City Services
– Engaging City Users, towards a participated attitude
• Advanced Smart City API, MicroServices, Snap4City API
• Federated Knowledge Base and Smart City API
• Web and Mobile App Development Kit
• Acknowledgement
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Paolo Nesi
• Overview
• Urban Platform (main concepts vs Living Lab)
• Snap4City Architecture, roadmap, logos, innovations
• Dashboards: from City Dashboards to Applications
• Trajectories and real time tracking
• Dashboards Intelligence and web and mobile devices
• Dashboard chatrooms and notifications
• Smart City Control Room
• Dashboards production
• Data Gathering and City Data Knowledge Management
• Protocol vs Data
• Data Gathering processes
• GIS Data Import, Export and Exploitation
• Semantic Modeling and City Knowledge Base: Km4CIty
• IOT Applications, Devices and Dashboards
• IOT Devices
• Forging & Managing Flexible Mobile Apps, Web App, MicroApplications
• Web and Mobile App with Open Development Kit
• Understanding how city users are using the city services
• Engaging City Users Towards Virtuous Behaviour
• Data Analytic, Big Data Science
• Data Analytics: predictions
• Smart Parking: predictions
• User behaviour Analysis via Wi-Fi, OD Matrices, Trajectories
• Recognition of Used Transportation Means
• Traffic Flow Reconstruction, from traffic sensors data
• Quality of Public Transport
• Origin Destination Matrices
• Demand of Mobility vs Offer of Transportation
• Modal and Multimodal Routing for Navigation and Travel Planning
• Environmental Data Predictions
• Prediction of Qir Quality
• Anomaly Detection
• Environmental data prediction
• Social Media Analysis
• Snap4City Living Lab for Collaborative Work
• Development Life Cycle
• Development tools
• Data protection, personal da vs GDPR
• Snap4City and Km4City Projects
• Acknowledgment
Snap4City has been created in response to Select4Cities PCP (http://www.select4cities.eu/) call as an open, standardized, data-driven, service-oriented, user-centric platform enabling large-scale co-creation IOT/IOE applications and services for Helsinki, Copenhagen and Antwerp.
Snap4City is a fully open source, robust, scalable, easy to use solution, provides tools for co-creation of mixt data driven, stream and batch processing, extending the powerful semantic reasoner of Km4City https://www.km4city.org, with IOT/IOE, GDPR, and city dashboards.
Km4City has been validated in multiple devices (PC, Android, Raspberry, ..), and domains: mobility and transport, tourism, health, welfare, social and cities such as Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and million of data per day.
The innovation is mainly related to semantic reasoning, IOT interoperability, microservices, automated dashboard production, .. thus setting up smart city solutions in a snap
Snap4City has been created in response to Select4Cities PCP (http://www.select4cities.eu/) call as an open, standardized, data-driven, service-oriented, user-centric platform enabling large-scale co-creation IOT/IOE applications and services for Helsinki, Copenhagen and Antwerp.
Snap4City is 100% open source:
robust, scalable, easy to use solution, provides tools for co-creation of mixt data driven, stream and batch processing, GDPR, and city dashboards.
extending with IOT/IOE the semantic reasoner of Km4City https://www.km4city.org
Km4City has been validated in multiple devices (PC, Android, Raspberry, ..), and domains: mobility and transport, tourism, health, welfare, social and cities such as Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and million of data per day.
The innovation is mainly related to semantic reasoning, IOT interoperability, microservices, automated dashboard production, .. thus
setting up smart city solutions in a snap
Serve as a City Dashboard, App User Interface, etc.
Real time and historical data, any device, sensors and actuators
Sensors, KPI, maps, data trends, real time data, charts, etc.
Referral / historical data, and Open Data:
shadow, access (API, storage, any protocol), production of OD, export
Data Driven Real Time communication & processing:
IOT Applications, IOT edge, multiple operating systems, embedded systems, MicroServices
in/out data driven from/to the field into: applications, notifications, etc.
Data Analytics: Machine Learning, statistics, reasoning, …
Serve as Living Lab: open innovation, coworking; collaborative work; sharing: data, processes, dashboard, experiences, solutions, ….
Experimented on large scale cases
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0Paolo Nesi
Snap4 has been created as an open, standardized, data-driven, service-oriented, user-centric platform enabling large-scale co-creation IOT/IOE applications and services for Helsinki, Copenhagen and Antwerp. Snap4 is a fully open source, robust, scalable, easy to use solution, provides tools for co-creation of mixt data driven, stream and batch processing, extending the powerful semantic reasoner of Km4City https://www.km4city.org, with IOT/IOE, GDPR, and city dashboards. Snap4 for Smart city is Snap4City (Https://www.snap4city.org ) is an open platform and solution for setting up Living Labs engaging different all kinds of stakeholders (city operators, researchers, city users, in house, industries) in contributing to the city evolutions, with a platform providing online tools for developing IOT applications, web and mobile Apps, data analytics, micro Applications, external services, KPI, POI, dashboards, IOT edge, etc.
Snap4city has been validated in multiple devices (PC, Android, Raspberry, IOT Button, Arduino, ..), and domains: mobility and transport, tourism, health, welfare, social and cities such as Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and millions of data per day. The innovation is mainly related to semantic reasoning, IOT interoperability, microservices, automated dashboard production, end-2-end encrypted secure communications, GDPR, .. thus setting up in a Snap smart city solutions.
The solution is fully complient with NodeJS with nodex published on JS foundation, is powered by Fi-Ware, compliant with LoraWan, SigFox, Mqtt, AMQP, Coap, NGSI, OMAM2M, WSs, Https, powered by Km4City, TensorFlow NVIDIA, Hadoop, etc. etc.
slides and demos: the platform includes full stack, any format, any protocol, from IOT Device, IOT Edges, Data Analytics, and Dashboards.
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Paolo Nesi
• Data Ingestion Capabilities
• Data Ingestion Strategy
• Setting Up the Road Graph on Knowledge Base
• Data Set Load via Data Gate (plus how to load triples into Knowledge base)
• Data Ingestion and Transformation via ETL Processes
• Data Ingestion via IOT Brokers
• IOT Network: recall of basic concepts
• IOT Directory
• IOT Devices and IOT Brokers Registration
• Data Ingestion via IOT Applications
• Data Ingestion from API, External Services, Custom MicroServices
• Data Ingestion via Web Scraping
• Data Streams from Smart City API, participatory
• Data Streams from Mobile Devices
• Data Streams from Dashboards
• GIS Data Import and Export
• Social Media data collection and exploitation
• Acknowledgements
Snap4City a Solution for highly collaborative Smart Cities Environments Paolo Nesi
Snap4City has been created in response to Select4Cities PCP (http://www.select4cities.eu/) call as an open, standardized, data-driven, service-oriented, user-centric platform enabling large-scale co-creation IOT/IOE applications and services for Helsinki, Copenhagen and Antwerp. Snap4City is a fully open source, robust, scalable, easy to use solution, provides tools for co-creation of mixt data driven, stream and batch processing, extending the powerful semantic reasoner of Km4City https://www.km4city.org, with IOT/IOE, GDPR, and city dashboards. Snap4City (Https://www.snap4city.org ) is a solution for setting up Living Labs engaging different all kinds of stakeholders (city operators, researchers, city users, in house, industries) in contributing to the city evolutions, with a platform providing online tools for developing IOT applications, web and mobile Apps, data analytics, micro Applications, external services, KPI, POI, dashboards, IOT edge, etc.
Snap4City/Km4City has been validated in multiple devices (PC, Android, Raspberry, IOT Button, Arduino, ..), and domains: mobility and transport, tourism, health, welfare, social and cities such as Florence, Pisa, Arezzo, and large area of millions on inhabitants as Tuscany and millions of data per day. The innovation is mainly related to semantic reasoning, IOT interoperability, microservices, automated dashboard production, end-2-end encrypted secure communications, GDPR, .. thus setting up in a Snap smart city solutions.
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...Paolo Nesi
The complexity of smart and sentient applications in smart cities is progressively increasing.
to reach higher precision.
time series prediction artificial intelligent, machine learning
Single data sets multi data sets, and big data: addressing heterogeneity, low quality and discontinuity, etc.
integration of open data, real time data and private IOT / personal data is increasing complexity of cyber-physical-social aspects:
to have the full control on the rights associated to their content
GDPR normative (since May 2018 in force) to regulate the access and control of privacy
I am bringing the experience of addressing
GDPR and Security and into Smart City Solutions with IOT
-----smart city impact----------------
Signed Consent vs Informed Consent
Smart Applications exploit personal data about (for example)
user position and actions for providing geolocated suggestions
home/work position, trajectories,
Payments and traces from bike sharing, from navigators, etc.
body signals/data for monitoring healthiness (glucose, temperature, etc.), for training sport, ..
User actions on applications: choices on menu, queries performed, mobile phones, requested paths, payments, etc.
IOT Devices data: mobile, buttons, trackers, but also temperature in house; position of our dogs, children, cars, bikes, …
etc.
----GDPR: General Data Protection Regulation ---
Users are going to decide to:
provide access to who, for do what, until we consent
accept terms of use by signed consent for each data management service, before was a simple informed consent
from each service, the user has to be capable to
See what the provider collect in terms of its Data Type: traces, logs, paths, profiles, accesses, IOT devices, sensors, maps, etc.
Download, delete, inspect each single Data Type
Auditing and Revoke access or grant access right to each single Data Type
Delete all Data Types in single shot or singularly (forget all about me)
Km4City: Smart City Ontology Building for Effective Erogation of ServicesPaolo Nesi
Provides a unique point of service with integrated and aggregated data and tools for
-- Qualified users: public administrations à developers
-- Operators: mobility, energy, SME, shops, ….. à developers
-- Final users à citizens, students, pendular, tourists
Problems:
--Aggregated Data are not available:
not semantically interoperable, heterogeneous for: format, vocabulary, structure, velocity, volume, ownership/control, access / license, …
---As OD, LD, LOD, private data, ..
---Lack of Services and tools to make the adoption simple
Final Users tools:
--Km4City mobile app with personal assistant is coming…
--Km4City mobile applications: Google Play, Apple Store, …
--Km4City web application: http://www.km4city.org
--Open Source Mobile Application, FODD: an example in open source http://www.disit.org/6595
Public administrator tools:
--Smart decision support system, http://smartds.disit.org
--Developers http://www.disit.org/km4city tools:
--Service Map Server, plus API, http://servicemap.disit.org
--LOG LOD browser: an ultimate visual tool to browse the RDF Store.
--Ontology Documentation: an ultimate tool to understand,
if needed !!
The dirty work of Km4City service
--Data Ingestion Manager, DIM
--RDF Indexer Manager, RIM
--RDF Store Methodology
--RDF store enricher with dbPedia
--Distributed SCE Scheduler, DISCES
--SCE: Smart City Engine
--Doc and info on http://www.disit.org/km4city
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIESPaolo Nesi
• Dashboards: from City Dashboard to Applications
• Data gathering and City Data Knowledge Management
• Forging & Managing Open and Flexible Web and Mobile Apps
• IOT/IOE Devices and Networks
• IOT Applications, the Logic and the Smartness
• IOT Applications vs IOT Edge Devices
• Advanced Smart City API, MicroServices, Snap4City API
• Snap4City Living Lab for Collaborative Work
• Snap4City for Beginners
• Open to developers and stakeholders
• Snap4City Architecture and Ecosystem
• Decision Support System vs Resilience
• Twitter Vigilance: Social Media Analysis
• Snap4City and Km4City Projects
• How to Adopt Snap4City, and our Roadmap
• Snap4City: the view of the Administrators
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...Paolo Nesi
Km4City From Data to Services for the Sentient Cities
Open Source and inter-operable tools to keep city under control via personalized dashboards
- monitoring services’ status of city operators
- monitoring and understanding the city users behaviour
- collecting moods, contributions and data from the city users
- monitoring social media for city services and events, event predictions
improve city resilience, reducing risks and decision support by:
- assessing city resilience level
- improving city resilience, providing objective hints
- improving city users awareness with personal city assistants and participatory tools
transform data in value for the city:
- enabling commercial and business applications
- aggregating multi-domain data and services for SMEs and city operators
- enabling integrated city services into third party web portal for all
- providing suggestion on demand services for SMEs and city operators
- accelerating and simplifying the implementation of business and service oriented Apps
Follow the Km4City City Smartener Process
A Smart City Development kit for designing Web and Mobile AppsPaolo Nesi
Presentation of some of the Km4City development tools: ServiceMap and App Development Kit, ADK.
ServiceMap is focused on providing information to the developers, to help them learning how to access to the data model, to exploit and use the API
ADK is a drafted modular web and mobile application based on HTML5 and JavaScript (apache Cordova) that can be used to exploit Smart City API to develop a large range of applications.
It is modular, flexible, etc. and allow performing users behavior analysis.
The solutions are currently in use on several EC and national Projects such as: Sii-Mobility, RESOLUTE, REPLICATE, Weee, …
Cities aims at providing new Smart Services to city users:
operators, final users, etc.
In most cases via Web and Mobile Apps which exploit data:
Structural data, open data, real time data, etc., private data from companies
to be aggregated and transformed in services (providing: prediction, information, early warning, relations)
at reasonable cost for: developers, operators, and SME to realize new Apps and services.
If cost is not affordable, Services and Apps are not developed, in most cases the Apps are also provided for free, so that high costs are not sustainable Public Private Partnership
Scenarious vs SmartCity API
Search data: by text, near, along, etc...
Resolving text to GPS and formal city nodes model
Empowering the city users
Access to event information
Supporting City Users in using Public Mobility
Supporting City Users in using Private Mobility
New Experience to access at Cultural and Touristic info
New way to access at health services
Access at Environmental information
Profiled Suggestions to City Users
Personal Assistant
Sharing knowledge among cities
ServiceMap tool
with Km4City are substantially a Smart City Expert System, SCES
includes the Smart City API
is a for developers to: search and browse on Smart City Knowledge, also to generate examples of the Smart City API call to be used in the development of Web and Mobile Apps
The IEEE Smart World Congress originated from the 2005 Workshop on Ubiquitous Smart Worlds (USW, Taipei) and the 2005 Symposium on Ubiquitous Intelligence and Smart World (UISW, Nagasaki). SmartWorld 2017 in San Francisco is the next edition after the successful SmartWorld 2016 in Toulouse France and SmartWorld 2015 in Beijing China. SmartWorld 2017 is to provide a high-profile, leading-edge platform for researchers and engineers to exchange and explore state-of-art advances and innovations in graceful integrations of Cyber, Physical, Social, and Thinking Worlds for the theme
http://ieee-smartworld.org/2017/smartworld/
hackathon smart city API, dai dati ai serviziPaolo Nesi
DAI DATI INTELLIGENTI AI SERVIZI
Smart City API Hackathon
Premi per 14.000 euro
Data: 7 e 8 aprile 2017
Luogo: Scuola di Ingegneria, Università degli Studi di Firenze
Il progetto Sii-Mobility, Smart City nazionale (MIUR), organizza il primo hackathon per promuovere lo sviluppo di applicazioni fisse e mobili sulla base delle http://www.disit.org/6991che si basano sul modello http://www.km4city.org .
Scopo dell'evento di hackathon è identificare nuove applicazioni che possano essere sviluppate sulla base di dati ed elaborazioni messi disposizione dalle smart city API di Sii-Mobility. I Dati sono in tutta la toscana e come dagli scenari http://www.disit.org/6995, sono relativi alla mobilità pubblica e privata, alla partecipazione, alle informazioni geolocalizzate dei punti di interesse, della salute, ambiente, e servizi di suggerimento e di coinvolgimento e assistenza.
Le tematiche affrontate dalle App proposte dovranno essere relative ad aspetti di mobilità, e in particolare ai seguenti 5 temi: Trasporto pubblico; Coinvolgimento dei cittadini, mobilità e turismo, mobilità e servizi, giochi in mobilità.
http://www.sii-mobility.org/index.php/eventi/hackathon-sii-mobility/registrati-all-evento-del-7-mattina
Documentazione e informazioni dalla pagina: http://www.sii-mobility.org/
Scadenza sottomissione delle proposte: 31 marzo 2017.
Premi per 14.000 euro, #hackathon #smartcity API, #bigdata #opendata della #Toscanadigitale #firenze #pisa #arezzo #direfare #forumpa
#hackathon #smartcity #bigdata #opendata #Toscanadigitale #firenze #pisa #arezzo #direfare #forumpa
Smart City API, 14.000 euro di premi, Hackathon
Open Urban Platform for Smart City: Technical View Paolo Nesi
Km4City Roadmap
Data and Model
Control Room
Monitoring Traffic Flow and Parking
Monitoring City Users via Wi-Fi
Engaging Users Via Mobile App
Development Tools
Who is using it
City Resilience and DSS
Info and Documents
Similar to Snap4City November 2019 Course: Smart City IOT platform installation, deploy, development life cycle (20)
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Snap4City November 2019 Course: Smart City IOT platform installation, deploy, development life cycle
1. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
1
LIVING LAB
Be smart in a SNAP!
6th Day, System & Deploy, Nov 2019
https://www.snap4city.org/501
2. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
Snap4City (C), November 2019
Paolo Nesi, paolo.nesi@unifi.it
https://www.Km4City.org
https://www.disit.org
https://www.Snap4City.org
scalable Smart aNalytic APplication builder for sentient
Cities: for Living Lab and co-working with Stakeholders
Powered by
2
6th Day, System & Deploy, Nov 2019
https://www.snap4city.org/501
3. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
General Overview of the full Course
• 1st Day: General overview (1 day, 6 hours)
• 2nd Day: Dashboards, how to build and manage them (4 hours)
• 3rd Day: IOT Applications development, IOT Devices, IOT Networks (4 hours)
• 4th Day: Data Analytics, in R Studio, In Python, how to integrate with IOT
Applications (4 hours)
• 5th Day: Data Ingestion, Data Warehouse, ETL Development, Data Gate, IOT
Device Data ingestion, etc.. (5 hours)
• 6th Day: Snap4City Architecture, How To Install Snap4City (3 hours)
• 7th Day: Smart city API (internal and external) Web and Mobile App
development tool kit (4 hours)
A number of the training sections include exercitations
Updated versions on: https://www.snap4city.org/501
Snap4City (C), November 2019 3
4. 6th day Agenda
• Snap4City Architecture
• Snap4City: Smart City IOT as a Service
• Snap4City Living Lab For Collaborative Work
– Smart City Development Life Cycle
– Analysis and Design for Innovation (Co-Creation and Co-Working)
– Development Tools
– How to Add Functions that are not present in the Platform
• Snap4City vs Fi-Ware
• Snap4City vs State of the Art Solutions
• Snap4City Services: Consulting and Developing
• Snap4City vs Snap4Industry 4.0
• Installing Snap4City
• The view of the Administrator
– Monitoring Resource Consumption and Traffic
– Managing and Monitoring Data Traffic in the BackOffice
– Auditing Activities
– Managing Back Office processes via Containers
• Acknowledgement
Snap4City (C), November 2019 4
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
5. Levels of Difficulty
• Easy.
• Moderate.
• Good.
• Golden.
• Professional.
• Excellent.
Snap4City (C), November 2019 5
non programmer level
Some JavaScript rudiment coding
JavaScript programming
Programming in R Studio
Exploiting Smart City API
Developing Full IOT Applications,
Dashboard and Mobile Apps
10. Unique of Snap4City Platform (1)
• Data ingestion and model
– Unified data model (exploited in the Wizard and Knowledge base)
– Semantic Reasoner modelling city entities, supporting sematic
search, expert system, digital Twin, etc.
– IOT Directory abstracting complexity of IOT
Devices, Edge, Brokers, protocols and data formats
• Data Analytics and Data Processes
– Flexible and extensible IOT Applications
– Data Analytic: multiple programming languages,
• Visual Analytics, dashboarding, Apps
– Wizard: expert system for immediate dashboard production
matching data vs graphics representation
– Dashboards specialized multidomain for Smart Cities
– Custom Widgets and Synoptics
– Ready to use Mobile App, instant App, MicroApplication
– Strategies formalization supports
Snap4City (C), November 2019 10
11. Unique of Snap4City Platform (2)
• Openness to any developers
– Living Lab support for coworking, sharing, and delegating
– Advanced Smart City APIs and MicroServices
– 100% Open Source, Open hardware
• Security and Privacy
– End-2-end encrypted communication,
on devices, platform, ... dashboards
– GDPR compliant privacy/security
• Non functional
– on cloud and on premise, your private installation
– Ready to use Appliance Virtual Machines and/or
Containers for a modules and tools.
– Flexible, Modular, Elastic, scalable and robust
Snap4City (C), November 2019 11
13. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
How to adopt Snap4City
Snap4City (C), November 2019 13
Https://www.snap4city.org
Smart City as a Service
• Supporting Org
• 100% Open Source Platform: Github
• Further developments
• Publishing Appliances and Dockers
• Training courses, docs
• Consulting
• Forums
• Etc.
Download and
deploy
Installation on your premise
• Virtual Machines or Dockers
• Different configurations
• From small to scalable
• Exploiting your legacy tools
• Interoperable with any tool
• No vendor lock-in, No tech lock-in
Mixed solutions! For example:
• Start on Cloud as Smart City as a Service
• Migrate on premise on the fly
• Start on Cloud into a sand box
• Pass to install on premise what you need
On your premise
14. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 14
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
15. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
Snap4City (C), November 2019 15
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
17. Smart City IOT as a Service
• Snap4City applications may exploit multiple paradigms as data driven,
stream and batch processing, putting co-creation tools in the hands
of:
– Smart Living Lab users and developers a plethora of solutions to develop
applications without vendor lock-in nor technology lock-in,
– final users customizable / flexible mobile Apps and tools,
– city operators and decision makers specialized / sophisticated city dashboards
and IOT/IOE applications for city status monitoring, control and decision
support.Open to Organizations
• Training and manuals: https://www.snap4city.org/108
• Help Desk: https://www.snap4city.org/3
• SLA: https://www.snap4city.org/497
• Terms of Use: https://www.snap4city.org/drupal/legal
Snap4City (C), November 2019 17
https://marketplace.docker-fid.grid.cyf-
kr.edu.pl/services/snap4city
18. Specialized Offer
Snap4City as Smart City IOT as a Service for
• Who would like to create Living Labs as community exchanging
experience with other cities as well;
• Research Institutions, Departments and Projects which would like to
perform research and experiments in the area of Smart City and IOT,
without the needs of setting up the infrastructure, exploiting open
data, collaborating, accessing to Data Analysis on demands, etc. This is
the spirit of EOSC, European Open Science Cloud Marketplace at
which Snap4City is registered as DISIT Lab, see [EOSC].
• Public Administrations, as small cities that would like to offer smart
services and does not have economic power to manage service on t
heir premise from them self.
Snap4City (C), November 2019 18
19. Help Desk and SLA
• https://www.snap4city.org/drupal/contact
• Bug Reporting
– https://docs.google.com/forms/d/e/1FAIpQLSfD
QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5
jsoXw/viewform
• For Service Level Agreement see:
– Service Level Agreement
• Help Desk and Contact:
– https://www.snap4city.org/drupal/node/3
• Availability rates:
– https://www.snap4city.org/388
Snap4City (C), November 2019 19
21. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 21
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
22. Snap4City (C), November 2019 22
Your Level
SLIDES
News
Flyer
Exercises
Full Search
All Tools
Personalized Suggestions
24. Living Lab Flexibility
• Multiple modalities to perform the
same activities
• Tuned for Beginners and Skilled people
• Visual interface and programming tools
• Resources and artefacts sharing for
learn acceleration and co-working
• Open Living and co-working Portal:
https://www.Snap4City.org
Snap4City (C), November 2019 24
Snap4City Satisfies all
Requirements of ENOLL
Select4Cities and EIP-SCC
25. For the user: different levels of engagement
• Multiple Organizations
• Roles:
– Managers
– AreaManagers (developers):
special access to computing
resources
• Levels for self-assessment
– Suggestions on next steps to
learn on the basis of personal
progresses Snap4City (C), November 2019 25
26. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Level 1 Users: creating dashboards
See how Dashboards can be created using the wizard: dashboards with selectors,
time trends, maps, etc.
• TC1.8. Visual production of Dashboard via Wizard
• TC1.9. Search on Wizard for any kind of data managed into the platform, from POI
to sensors, KPI, social, etc.
• TC1.10. Dashboard delegation to access, and passage of ownership, and/or
cloning
• TC1.11. IOT Discovery, on Dashboard Wizard
• TC1.13. Dashboard Builder External Services and Widgets
Snap4City (C), November 2019 26
27. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 27
All Text on the Portal are
Hypertext with Links for
navigation among major
concepts
28. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
MultiOrganization, Groups and Profiles
Organizations may have their distinct :
• menus and functionalities, GeoArea, Data, Dashboard,
Groups of users, managers, Knowledge Base, repositories,
etc.
Users may:
• Have personal IOT Devices, Data, Dashboards, IOT App,..
• Have access to multiple Groups of Multiple Org.
• Delegate them in usage or access
• Assesses their usage and themselves
Snap4City (C), November 2019 28
29. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Self
Assessment
Snap4City (C), November 2019 29
30. Snap4City (C), November 2019 30
Validation with developers
• Helsinki and Antwerp, plus Florence Training, CINI Challenge, ..
• 65 performed operational activities:
– dashboards, IOT Applications, registering IOT devices, etc.
– More than the 80% created both Dashboards and IOT Applications, thus validating the solution
and the process of engaging them in working on the platform
The 65 users left on
platform
Average per day
over last 90 days
Total activity
90 days
Number of IOT Applications 117 81,6 7341
Number of private IOT devices 27 25,5 2296
Number of public dashboards 11 6,2 562
Number of private dashboards 173 135,1 12159
Number of accesses to dashboards -- 33,9 3048
Number of minutes -- 337,1 30337
32. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Smart City
Development Life Cycle
Snap4City (C), November 2019 32
TOP
33. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 33
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
34. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 34
Formal DesignLegenda phase Tool/Process MicroService/Node Widget
In/Out
Functional
Historical
Data
Visual Interface
Analysis
Design
IOT Application
Implementation
Requirements
Specification
ServiceMap
Data Inspector
Discovery
IOT Directory MS
Functional MS
Analytical
Data Analytics MS
Yes
Data Ingestion&modeling
Data Gate/ETL
IOT Directory for IOT
Device Regis.
External Services
No
Search/Store MS
Ext. Services MS
Python
R-Studio
Java, ……
Develop. Data Analytic
Dashboard MS
IOT App Widgets
Data Flow Development
Dashboard Builder
& Wizard
User Interface
Development
Utilities MS
Private Info MyKPI, Per. MSMyKPI Modeling
Analysis and Design
Development, Deploy, Testing
Deploy,
Testing
Dev. Special Tools
IOT App Widgets
Custom Widgets
Regular Widgets
Web Scraping WebScraping MSWeb Data
Development Life Cycle
Smart City Services
35. Snap4City (C), November 2019 36
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
36. Analysis and Design for Innovation
(Co-Creation and Co-Working)
Snap4City (C), November 2019 37
TOP
37. Development Life Cycle
Smart City Services
Snap4City (C), November 2019 38
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
38. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Analysis & Design for Innovation
• Analysis
– The analysis start with a number of meetings/interviews with stakeholders
– The identification of the target stakeholders/actors/users (target Segments) and their
definition/description
– The meetings are focused on filling the INNOVATRIX approach of IMEC tuned for the Snap4City
platform and solution
– See the schema of the INNOVATRIX reported in the next slide
• Design
– Focused on creating a large number of Use Cases and/or Scenarios for development
– The design start taking into account the Snap4City development life cycles and tools. Thus shortening
all the boring activities and following the typical Snap4City rapid prototyping described in these slides
• Data Discovery
– Data discovery is performed on analysis of the: (i) identified scenarios, (ii) data of the stakeholders,
(iii) international sources, (iv) Snap4City experience, etc.
– Performed by following the Snap4City guidelines on Data Search on web and world.
Snap4City (C), November 2019 39
39. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
INNOVATRIX by IMEC for the Analysis
Snap4City (C), November 2019 40
40. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Use Cases: Design vs Data Discovery
• The analysis may produce possible needs vs current practices
• The new innovative solutions have to be identified by experts working
on these tables, thus producing Use Cases, Scenarios
• Each of them has to be described and assessed taking into account:
– Innovation level, social impact,
– data availability (and thus as output of the Data Discovery)
– User interaction and behavior constraints
– technical feasibility, implementation costs in the context of Snap4City (data
ingestion cost, data costs, development costs, etc. )
– Time and phases to for the development
– Validation approach: feasibility and costs
Snap4City (C), November 2019 44
41. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Taking Decision with all: co-creation
• Once performed the above described approach
– A number of Use Cases, Scenarios have been created and well formalized
• Than they may be
– shortlisted according to the main stakeholder goals (city, operators, etc.)
– voted in a participatory approach in the Living Lab
– selected to be developed according to the next steps described in the
following
• Once developed, a validation with the same stakeholder is needed
Snap4City (C), November 2019 45
42. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Tools
Snap4City (C), November 2019 46
TOP
43. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 47
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
44. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
• Open Data:
– Data gate, federation of Open
Data Portals
– ETL processes (PULL)
– IOT Application processes
• IOT Networks:
– IOT Application processes, data
driven or PULL
– IOT Brokers (Push) → IOT
Shadow
• Web Pages:
– Web scraping, crawling
processes
• Social media: Twitter, Facebook,..
– Twitter Vigilance, IOT App
• Mobile Apps
– Smart City API
• Files upload: CSV, Excel, etc.
– IOT Applications, ETL
• REST API, WS, FTP, etc.
– IOT Applications, ETL
• Data base accesses
– GIS: WFS, WMS
– ETL, IOT Application Snap4City (C), November 2019 48
My Files
IOT App
ETL
IOT Broker
IOT Device
IOT Broker
Web Scraping
DataGate
API, External Services
Rest Call …….MS
GIS data, Maps, …
Sensors/
Actuators IOT Edge
IOT Device
External Data Stores
Any kind of data and flows
45. IOT Network Manager vs Final User
Snap4City (C), November 2019 49
IOT Broker
IOT Broker
IOT Broker
IOT Broker
Network of IOT Brokers
Knowledge Base,
Km4City
Knowledge and Storage
Data from the Field and
City
Discovering
IOT Network
Manager
Registering
Browsing
ServiceMap
Knowledge Base
Discovering
IOT Application
Dashboard Wizard
IOT Directory
Final user
Manager
My IOT Device
Register
50. Web Scraping
Snap4City (C), November 2019 54
IOT App. Editor
Resource Manager
Sharing/saving
reusing Scraping
My Scraping processes
Generating
WEB Scraping
Knowledge Base, Km4City
Web Scraper PORTIA
51. Integrated DataGate/CKAN
Static open data ingestion
Snap4City (C), November 2019 55
Data Set:
- Search
- Loading
- Download
- Share
- Publish
- Also automated
Automated data
regularization
Federated Crawling
Federated Distribution
52. Data Ingestion via Data Gate
Snap4City (C), November 2019 56
Federated Network
Saving /
Sharing
reusing
Knowledge Base, Km4City
Knowledge and Storage Data
from the Field and City
Data Gate
Data Set: sharing,
Harvesting, Loading/
Downloading
Data Set:
- Search
- Loading
- Download
- Share
- Publish
- Also automated
53. Developer in ETL: Penthao Kettle
Snap4City (C), November 2019 57
Batch Processing for dynamic data ingestion
54. Developing ETL, Data Manager
Snap4City (C), November 2019 58
ETL Process
ETL Process
ETL Process
ETL Process
Resource Manager
Saving /
Sharing
reusing
ETL Development EnvironmentDISCES scheduler productionDistributed Back Office
Knowledge Base,
Km4City
Knowledge and Storage Data
from the Field and City
Data Gate
Data Set
Saving /
Sharing
Data Set
Loading/
Downloading
Data
sources
Load data or prepare for data ingestionSchedule Execute
56. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Data Gathering and Knowledge Management
• Data ingestion can be performed by using multiple tools:
– ETL processes, IOT Applications, Data Gate, WebScraping. We suggest:
• ETL for static / periodic data in PULL
• IOT App for real time data and flow, from IOT Brokers/Devices
• DataGate for Static Data, upload them as files, or collected from other CKAN
• WebScraper for scraping data from Web Pages, when authorized!
• See how to test cases:
– HOW TO: add data sources to the Snap4City Platform
– HOW TO: define privacy rules for personal data, produced by the end-users own device
– US6. Developing and using processes for data transformation
– TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share
– TC6.3 - Creating ETL processes for automated data ingestion and data transformation
– TC6.5 - Managing Heterogeneous File Ingestion via ETL processes
– TC6.9 - ETL processes for multiprotocol and format data ingestion, see on GITHUB for library
– TC9.2 - Managing heterogeneous File Ingestion, protocols, formats via IOT applications, and open
standards
Snap4City (C), November 2019 60
57. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Interoperability ESRI
• Snap4City is interoperable with
– ESRI ArcGIS Enterprise, Portal, Pro/MAP, ...
– other GIS tools supporting WFS, WMS,
GeoJSON, GML
• Snap4City is interoperable since:
– Provides info/data in WFS, WMS
– Exploits data/info from WFS, WMS
– Import data/info from WFS/WMS
• The Snap4City platform can be installed
on premise using Snap4City Appliance
https://www.snap4city.org/drupal/node/471
– StartSNAP4CITYVM includes the Dashboard Builder that is capable to work
with WFS WMS protocols for the integration with GIS platforms as ESRI
ArcGIS, QGIS, directly or using Snap4City GIS player.
– KBSSMVM includes the Smart City API and WFS API which can be used to
data harvest from any GIS servers and GIS desktop tool
Snap4City (C), November 2019 61
58. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Linked Open Graph
Snap4City (C), November 2019 62
LOG: https://log.disit.org
Schema: http://www.disit.org/km4city/schema
RDF version: http://www.disit.org/km4city.rdf
Linked Open Data
59. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 63
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
60. Data Analytics Dev. in R Studio and/or Tensor Flow
Snap4City (C), November 2019 64
Knowledge Base,
Km4City
Ontology Schema
SPARQL, FLINT
LOG.disit.org
Swagger
SmartCityAPIfromKnowledgeBaseandothertools
Big Data
Store
Facility
Resource Manager
Creating
MicroServices
Saving /
Sharing
reusing
Using them into
IOT Applications
61. Data Analytics Dev. in Java, Python, ..
Snap4City (C), November 2019 65
Knowledge Base,
Km4City
Ontology Schema
SPARQL, FLINT
LOG.disit.org
Swagger
SmartCityAPIfromKnowledgeBaseandothertools
Big Data
Store
Facility
Resource Manager
Saving /
Sharing
resusing
Process
Process
Process
Process
Distributed Back Office
DISCES scheduler
Monitoring
Coding
Testing
Data
sources
62. Loading new Node-RED
nodes/microservices from external Palet
• Ask to RootAdministrator to have your custom Node-RED
nodes/modules added to your IOT Applications
– https://www.snap4city.org/drupal/contact
• A validation of requested Node-RED nodes will be performed to
avoid violation of security and privacy for all
• Administrators may load custom Node-RED nodes/modules
• This limitation is not present in your on premise installations of
Snap4City
Snap4City (C), November 2019 66
63. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 67
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
64. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard List and Editor
Snap4City (C), November 2019 68
66. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard Wizard
Snap4City (C), November 2019 70
• Select the area of your interest: panning and zooming
• Select the
• graphic aspect of your interest, or
• High Level Type of your interest, or
• Make a search if you a have a precise idea or
• Act on filters: nature, subnature, type, name, value, date, health, owner, …
• Combine them as you like
• Select the lines of your interest
• Then click on Next and get the Dashboard by wizard
67. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard Wizard
Snap4City (C), November 2019 71
The Wizard help you in selecting
only possible combination of data
vs graphic representation
68. Developing Dashboards
Snap4City (C), November 2019 72
Knowledge Base,
Km4City
Create, save, load,
delegate, grant access
Dashboard
Collection
Dashboard Wizard
IOT Applications
MicroApplications
External Services
Dashboard Editor
Widget Collection
Knowledge and Storage
Data from the Field and
City
My Own Dash/App
69. Dashboard Development
Snap4City (C), November 2019 73
Knowledge Base,
Km4City
Create, save, load,
delegate, grant access
Dashboard
Collection
Dashboard Wizard
IOT Applications
MicroApplications
External Services
Dashboard Editor
Widget Collection
Knowledge and Storage
Data from the Field and
City
My Own Dash/App
70. Developing new Dashboard Graphic Widget
• The development of new Widget is feasible for programmers
– in PHP, JavaScript and CSS
• The starting point is the Core Snap4City Virtual
Machine Appliance StartSNAP4CITYVM:
https://www.snap4city.org/471
• While the source code is also accessible on GitHub/DISIT
• see TC3.11 - New graphics widget can be easily created, Dashboard Widget
Creation
– creating a new Graphic widget via SVG and programming: Custom
Synoptics and Widgets for Dashboards https://www.snap4city.org/504
Snap4City (C), November 2019 74
https://www.snap4city.org/drupal/node/504
71. Snap4City (C), November 2019 75
Custom Dashboards and Widgets (interactive,
Animations, etc.)
-- SVG for graphic design
-- MyKPI for collecting data
Custom Widgets
72. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Creating custom Widgets for Dashboards
Snap4City (C), November 2019 76
Designer
JS programmer
Producing
and
Publishing
MyKPI
WS secure
USING
IOT Edge
IOT Devices DEVELOPING
JS Library
73. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboards
• Suitable as: City Dashboard, App interface, and Control Room Dashboards,
Situation Room Dashboard, Operator Dashboard
• Created visually compounding graphic Widgets
– Each widget has an autonomous update
– Each metric/data-source may have associated with an alarm: blinking and sending
events to people and machines in different manners
• Can be: public or private, private dash can be delegated or passed in ownership
• See https://main.snap4city.org/management/dashboards.php?linkId=dashboardsLink&fromSubmenu=false&sorts[title_header]=1
• See the following tutorials
– HOW TO: create a Dashboard in Snap4City
– US1. Using City Dashboards
– US2. Using and Creating Snap4City Applications with Dashboards
– US4. Creating City Dashboards and related Event Monitoring and Actions
Snap4City (C), November 2019 77
74. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Level 1 Users: using dashboards
• TC1.3. Accessing and using Dashboards with any device
• TC1.4. Dashboards Showing Data Real Time and
Historical/trends, comparison
• TC1.5. Dashboards showing a range of different High
Level Types: KPI, POI, IOT dev, MicroApp, Maps using
different kind of Graphics Widget. Monitor city status with
Dashboards
• TC1.6. Dashboards using different kind of graphics
Widgets matching with High Level Types, Monitor city
status with Dashboards
• TC1.12. Dashboard with MicroApplications
• TC4.1. Dashboard and Notificator
• TC4.6. Dashboard with city events, ESB, police, traffic, etc.
Snap4City (C), November 2019 78
75. Snap4City (C), November 2019 79
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
76. Developing Web and Mobile Apps, MicroApps,..
Snap4City (C), November 2019 80
Knowledge Base,
Km4City
Swagger
Advanced Smart City API
Web App HTML5, MicroApplications
Embed into Web pages
Mobile Apps
ServiceMap
DeveloperKm4City Open
Source
development
tool kit
City User
Mobile Application
Monitoring
Administrator
77. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
IOT Directory
Back Office Processes
IOT Broker
IOT Broker
IOT Broker
IOT Broker
ETL Process
Data Analytics
ETL Process
ETL Process
ETL Process
Data Analytics
Data Analytics
Data Analytics
Knowledge Base,
Km4City
Smart City API from Knowledge Base and other tools
Ontology SPARQL, FLINT LOG.disit.org
ServiceMap ServiceMap3D
Swagger MicroServices
IOT ApplicationsWeb and Mobile AppsDISCES and back office management tools
Snap4City (C), November 2019 81
MicroApplications
Resource Manager
78. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
How to Add Functions that are not
present in the Platform
Snap4City (C), November 2019 82
TOP
79. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 83
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
80. Development Life Cycle
Smart City Services
Snap4City (C), November 2019 84
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
81. Adding a Function that is not present on the platform
• Snap4City allows at the AreaManagers to Develop new functions
for:
– Interacting with the field: IOT Sensors/Actuators, IOT Devices
– Data production by using: IOT Devices, IOT Brokers, ..
– Data Ingestion/Transformation by using: ETL, IOT Applications, WebScraper,
DataGate
– Data Analytics by using: R Studio, Java, C++, Python
– Data Rendering on Dashboards: PHP, HTML5/JavaScript, Web App, etc.
– User interaction and App: Mobile App, MicroServices, Dashboards
Snap4City (C), November 2019 85
85. • Is open to the Development of Applications
leaving large space to developers
• Is cantered on the Orion Broker that result
central in the architecture: any Broker or data
source is sending data to Orion
• Security level is not clear, partially demanded
to developers
• Visual Flexible IOT processing is not clearly
provided
• Limited API for IOT data access
• Knowage BI presents several limitations in
showing Smart City Data
• Market place on Open Data
• Support of Developers via Fi-Ware
• Deployed as VM and Dockers
• open source, not the application parts
Snap4City (C), November 2019 90
• Is open to the Development of Applications leaving large
space and providing a large set of ready to use
applicative tools and solutions to build their solutions on
top or aside.
• Is fully distributed, any kind of data source can be
ingested, automatically.
• Orion Broker is only one of the Brokers that can be used.
It can be also protected by Snap4City tech, with Mutual
Authentication
• Visual Flexible IOT processing is provided as Node-RED
and Snap4City MicroServices suites
• Advanced Smart City API are provided on top of
Knowledge Base
• Dashboard Builder has been designed for Smart City Data
and automated dashboards’ production
• Market Place on Open Data, tools, processes, experiences
• Full Support for Living Lab of the city, coworking, tutorials
• Deployed as VM and Dockers
• 100% open Source
vs
86. vs
• Snap4City is an official Fi-Ware Solution via
– NGSI V1, V2 The IOT Orion Broker
– IOT Orion Broker can connect JSON, MQTT, Lightweight M2M,
LoraWAN, OPC, SigFOX, etc. see Fi-Ware Https://www.fiware.org
• Snap4City is compatible with all the above protocols
– via IOT Orion Broker,
– via direct connection on ETL processes on their corresponding IOT
brokers, and/or
– via IOT Applications.
• Snap4City is also compatible with many other protocols, see
the table reported in page: https://www.snap4city.org/65
Snap4City (C), November 2019 91
88. • In Snap4City you can chose to connect your devices at Snap4City
Platform in different manners:
– (a) directly to Snap4City, or
– (b) via an IOT Orion Broker (external IOT Broker or those provided by
Snap4City), or
– (c) via any third party IOT Brokers in any protocol you have.
• Snap4City has
– Improved IOT Orion Broker with the so called Orion Broker Filter (Orion
Broker Filter, NGSI Security Wrapper) which is a secure wrapper for NGSI V1
and V2 protocol for enforcing Mutual Authentication, Security, roles, etc.
– Produced open hardware and open software NGSI Compliant: as
• IOT Devices with mutual authentication and security based
for NGSI on: Android, Arduino and ESP32, IOT Button, etc.
• IOT Edge devices with mutual authentication and security based
for NGSI on: Raspberry PI, Windows, Linux.
Snap4City (C), November 2019 93
vs
90. Market Solutions
Snap4City (C), November 2019 95
IOTDiscoveryAbstraction
Authentication,Authorization
Securityend-2-end,secureon
IOTandDashboards
OpenHWandOpenSW
IntegratedCommunity
management
DataTypes:IOTDevices,IOT
App,Dashboard,Data
DataType:Publish/share,
Delegation,Consentandchange
DataType:Downloadand
Delete
AuditingonDataTypeAccess
OpenSourceend-to-end
ScalabilityIOT
VisualProgrammingend-to-end
applications
AdvancedSmartCityAPI,
MicroServices
MultiDomainSemantic
Platform
StandardbasedModulesand
IOT,OpenDevices
ResourceSharing
DataAnalyticsintegrated
DashboardH24/7,protected
connection
Multi-protocolonIOT
G G G G G
Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y
Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http
IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y
IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT
FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y
ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited
Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP
AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited
Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited
PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y
Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y
CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N
Siemens MindSphere
[60]
Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y
Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT
Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP
Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited
Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
91. Snap4City managed to
provide a maximum of
information, flows, in depth
analysis with the data
provided.
There is no
other platform
that collects
all city actors
together.
The City officials and ICT
officials were impressed
with the performance of
the Platform when loading
the heavy, “resource-”
demanding applications and
dashboard.
The data handling
throughout the Platform
is considered as one of
the strong points in the
Platform and of an
extremely sophisticated
level.
The technical level of the Platform and its
strong points such as the way real-time
data is used, the algorithms, data clean-up
possibilities of the Platform, presented data
and information is state-of-the-art and
impressive.
Snap4City (C), November 2019 97
93. Providing consulting, customization,
training, and developments
• Snap4City solution can be installed on premise and one cloud, private
and public.
• Snap4City, Snap4 SRL, provide support, if needed, for design and/or
Develop:
– Training and tutoring;
– Smart city infrastructure and architecture;
– data analytics, that could be developed as proprietary solution for the
customer;
– data ingestion processes, to enable them to have data into the platform;
– adaptor for some specific protocol or legacy/third part Tool, that we prefer
to release as open source, but if the connection is with some proprietary tool,
the buyer could be interested to keep these solutions as private;
– IOT devices, full solutions, dashboards, specific dashboard widgets, etc.Snap4City (C), November 2019 99
94. Snap4Industry
• Snap4City technology can be exploited on Industy and IOT
solutions:
– Snap4Industry: Snap4City for Industry 4.0 (SLIDES)
– Scenario: 5G Enabled Water Cleaning Control
– Scenario: High Level Control of Industrial Plant
– Custom Synoptics and Widgets for Dashboards
– The integration between data and devices: the Snap4City solution
–
Snap4City (C), November 2019 100
96. Snap4Industry as a Lambda Architecture
Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real
Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
97. Security Architecture at a glance
Data
Visualization
Virtual
Devices
Cloud
Processor
Data
Injection
Device
Management
Edge
Processor
Data
Sources
IoT Devices
(sensors, actuators)
Security and Privacy Management
Context
Brokers
IoT
Broker
IoT App
IoT Context
Brokers
IoT App
IoT Edge IoT
Directory
Registries
and storage
Dashboards
Snap4City (C), November 2019 103
98. Industry 4.0 Application
Snap4City (C), November 2019 104
Sensors/
Actuators
Sensors/
Actuators
Sensors/
Actuators
IOT Broker
IOT Gateway
[IOT Application]
MicroServices
Ext-Services
…….
Executing
permanent
computation
0,00
200,00
400,00
600,00
800,00
1 3 5 7 9 11 13 15 17 19 21 23 25
Serie2
Serie3
Serie4
Serie5
ETL Process
Executing local
computation
Internet
Dashboards
-- predictions
-- suggestions
-- Anomaly
detection
IOT Edge
Sensors/
Actuators
Sensors/
Actuators
Energy
Providers
Data
Analytics
IOT Broker
On the field
IOT Devices
99. GIDA set up
ModBus to
Snap4City
Gateway Edge
5G network
devices
Smart City
data from
many sources
Dashboards and Apps
IOT Applications
Big Data Analytics, Artificial Intelligence
IOT Data
Shadow
Snap4City
Telemonitoring
Telecontrol
103. Green Impact Capacity (GIC)
• Improve productivity of
chemical plant
• Keep GREEN the
environmental impact
• Exploiting innovative
technologies
• Diversify the production
• Monitoring
environmental conditions
Snap4City (C), November 2019 109
105. Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart City API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
Snap4City as a Lambda Architecture
106. Overview of Snap4City platform, for Buyers, for all
Snap4City (C), November 2019 112
Snap4City MAIN
Dashboard, User
Management, etc.
IOT
brokers
HeatMaps and
Maps
Knowledge base,
semantic Store
Other
Services
And
Servers
Mobile Apps, Dashboards
Cluster for Smart
Processes container:
ingestion, analytics,
scraping, etc.
Cluster for big data
storage: static and
real time, Indexing
and search
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Legacy Platform
Back End Front End
107. Overview of Snap4City platform, for Buyers, for all
Snap4City (C), November 2019 113
Snap4City MAIN
Dashboard Builder,
Data Inspector, User Stats,
ExternalSrv, Res Manager,
MyKPI, MyPOI, Synoptic, IOT
Device Manager, Micro
Applications, Authorization/
Authentication, Roles, …
IOTBrokers, secure
IOTOBSFs
GISGeoServer
Heatmap Manager,
WFS, WMS
KBSSM
KB/ServiceMap, ..
SmartCity API, WFS,
ServiceMap3D, LOG/FLINT,
SuperServiceMap, Virtuoso
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
Mobile Apps, Dashboards
Container Cluster
MCLSCont, NCLSCont
Marathon, Mesos, Zookeeper,
Disces-EM, Executing in Container:
IOT App, Web Scraping, R-Studio,
Python, Java, ETL, ...
Data Shadow Cluster
IOTDSES
NIFI, Elastic Search, DevDash, Amma
[DataCluster]
ETL-VM
Phoenix HBASE, HDFS
DISCES, ETL support
IOTBrokers, secure
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Drupal support CRM, ..Legacy Platform
Back End Front End
108. What is included in the Buyers / Full Platform
• Snap4City MAIN
– Dashboard Builder, Wizard
– Dashboard Engine
– Data Inspector, Notificator
– External Srv manager
– Menu Manager
– Ownership Manager
– Authentication and
Authorisation
– WS server
– Resource Manager
– User Stats
– JavaScript Web App in a
Snap, MicroApplications
– Synoptics, custom Widgets
– IOT device manager: IOT
Directory
– Snap4City MicroServices on
IOT Applications: basic and
advanced
– Snap4City GIS Player
– What-IF tools
Snap4City (C), November 2019 114
• IOTOBSF
– IOT Orion Broker
Fi-Ware
– Secure Filter
(Snap4City)
• IOT Edge support
– Linux Ubuntu
– Windows
– Raspberry Pi
– Android
• IOT Devices support
– ESP32
– Arduino
• IOT Application
– Node-RED
– Snap4City Library
of nodes
• KBSSM
– Knowledge Base
Km4City
– ServiceMap
– ServiceMap3D
– SuperServiceMap
– LOG/Flint
– OSM2Km4City
– Smart City API
– WFS API
• GIS GeoServer
– GeoServer
– Heatmap
Manager
• Living Lab Portal
– CRM Drupal
– LDAP Snap4City
– Registration mng
– Etc. etc.
• MCLSCount, NCLSCount
Container[Cluster]
– Containers models
– DISCES-EM
– Monitoring App
• ETL Server
Data[Cluster]
– DISCES
– Phoenix Drivers
– Hbase Model
– ETL processes
• IOTDSES
DataShadow[Cluster]
– NIFI process, Squid
– Elastic Search Model
– Kibana
– AMMA data flow
– DevDash data store
• Other Services
– Routing Server
– DataGate CKAN
– Chat as Rocket
– ETL SDK VM
– Engager ..
– …..
• Data Analytics
– Heatmap
production MS
Rstudio
– OD production
– Predictions MS
RStudio
– Anomaly
detection MS
RStudio
– ETL Collection
– ….
All in source code, and most of components in Appliances/VMs
109. Platform Maintenance, K3.14
• Snap4City modules are released on GITHUB/DISIT and can be updated
from:
– GITHUB/DISIT into VM Appliance or Servers in which they are installed
• https://github.com/disit
– Node-RED tool, using official Library regarding: Snap4City Libraries
– Drupal for the Living-Lab Portal aspects
– Other tools from their corresponding providers
• Maintenance would not be a problem, all users should be capable to
perform the updates autonomously
• Updates on new versions will be provided by Snap4City periodically,
Notification will be provided on NEWS and GitHUB
Snap4City (C), November 2019 115
110. Keeping Platform at the State of The Art
• Updates will be provided by Snap4City periodically and released on
GITHUB, and other portals.
• Snap4City team is involved in contracts since now, so that this will
guarantee that the solution will be evolved to anticipate the state of
the art as we have done in Iteration 3, in which we released a number
of developments.
– See also the activity of dissemination and updated roadmap
– See the presence of Snap4City on EOSC, BeeSmartCity, EO15, etc.
– See the list of concreate developments
– See the list of supporters on the Stand with respect to those of the 2018
– See the planned new developments
Snap4City (C), November 2019 116
111. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Set-up of Open Source version from Scratch
• Guidelines reported on:
– https://www.snap4city.org/drupal/node/471
• How to proceed:
– We have presented 7 Configurations models from A:Alone to F:FullPlatform+LivingLab, but
they can be customized as you like.
– By follow the Configurations the Customers are guided to identify the most suitable
according to their needs;
– Once the most suitable Configuration has been identified, they are supported into the:
i. Download and deploy of the Appliances provided as Virtual Machines or Containers, or
ii. Download and install modules from GITHUB according to the recipes for VM/Container and the
user manual of the single tools, or
iii. Mixt of the above (i) and (ii) approaches.
– IOT Edge can created by (1) installing Node-RED, (2) adding Snap4City library of Nodes from
the Palette Manager of Node-RED, (3) registering on Snap4City:
https://nodered.org/docs/user-guide/editor/palette/manager
Snap4City (C), November 2019 117
Note that some of the material we are presenting has not been published on the portal yet, for the competition.
112. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City Scalable Platform with your business
Snap4City (C), November 2019 118
Configuratio
n Kind
Min#VM
Dockers
IOTBrokerint
IOTBrokerExt
MAP
KBSM
IOTAPP,
MicroServices
Storage&IOT
DataShadow
SSO,Roles
IOTSecurity
Dash+Wizard
GISin
GISout
Heatmap
Mob.App
ASCAPI
LivingLabSup.
Options
supported by
the
configuration
A: Alone 1 6 X X 1 mf X X X X (x) ETL, DG, CM
B: Basic 2 7 X X X 1 mf Small X X X X (x) ETL, DG, CM
C: CityStart 3-4 8-9 X X X X 1 mf Small X X X X X X X X ETL, DG, R, CM, Eng
D: DataCity 5-6 Ask X X X X 70 Mediu
m
X X X X X X X X ETL, DG, R, CM, Eng,
LL
E: ExtensiveCity 6-8+ Ask X X X X 70 Scalable X X X X X X X X ETL, DG, R, CM, Eng,
LL
F: FullPlatform 12-14+ Ask X X X X Scalab
le
Scalable X X X X X X X X ETL, DG, R, RS, CM,
Eng, LL
F: FullPlatform
+ LivingLab
13-15+ Ask X X X X Scalab
le
Scalable X X X X X X X X X ETL, DG, R, RS, CM,
Eng
mf: multiple flows for each IOT App
Read next slide for the other notes and legenda
113. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Legenda and Notes on the previous table
• In all configurations you:
– have: Multiple IOT Brokers; unlimited number of data sources;
interoperability support; KB
– may have: HA, DRS, FT, Balancing, cloning and configuring additional VM;
– may pass at the next configuration without reinstalling the VM;
– may add one or more Optional Services as VM/Containers: ETL, DataGate,
Routing, Engager, Data Analytics, etc. etc.
• From Config. E:ExtensiveCity and upper the number of VMs depends on the
volume of DataStorage and the volume of Data Stream in input.
– These two aspects are managed by two independent clusters of VMs and scale
independently each other. This allows to satisfy any different combination of volumes
in streams and data storage.
• We suggest using Living Lab support only on FullPlatform, while it can be
installed on Smaller Configurations with some limitations in terms of features
Snap4City (C), November 2019 119
Optional Services
• DG: DataGate
CKAN
• R: Routing
• ETL: ETL SDK
VM
• RS: R Studio
Server
• Eng: Engager
• CM: Chat
Manager
• LL: Living Lab
• ..
• ..
114. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Computational and Storage Costs
• The VM is considered (appliances are provided) as 16/24 cores 2.2 Ghz, 16-24 GB Ram,
500 GB HD in thin provisioning, with 25-40 GB HD used at the start, in most cases
Debian.
– to estimate the needed CPU, RAM, Storage for each configuration it is easy from the #of VM in the
configuration.
– Please note that configurations A and B can be executed on 4 cores, 4 GB Ram, ..
• In solutions, with a very large number of users on the Front End: Smart City API (mobile
App users, Dashboard users, a frontend balancer and more FrontEnd servers would be
needed). Please note that the Smart City API are also used by DataAnalytics processes
and by MicroServices in the IOT Applications.
– Typically a thousand of simultaneous users on the front end can be sustained for each VM
– More precise estimations can be performed by knowing the actual workload
Snap4City (C), November 2019 120
115. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
A Alone Configuration
Snap4City (C), November 2019 121
TOP
116. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
A: Alone) For Small Size Business, for example an industry 4.0 application
you can need to install only one VM: Snap4CityMAINVM
• For connecting data sources to Dashboards onpremise!
– Data driven solution, no large data store
– Industry 4.0
– Small Smart City business
– Start with Snap4City platform as seeding element
Snap4City (C), November 2019 122
• Snap4City MAIN VM
– Dashboard Builder, Wizard
– Dashboard Engine
– Data Inspector, Notificator
– External Srv manager
– Menu Manager
– Ownership Manager
– Authentication and
Authorisation
– WS secure server
– Resource Manager
– User Stats
– JavaScript Web App in a
Snap, MicroApplications
– Synoptics, custom Widgets
– IOT device manager: IOT
Directory
– Snap4City MicroServices on
IOT Applications: basic and
advanced
– Snap4City WFS player
– What-IF tools
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, …
IOT Applications, Synoptic,
Snap4City WFS GIS player,
IOT Device Manager….
External
Data Stores
117. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4CityMAIN: Alone Configuration
• As Virtual Machine:
– HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City
– https://www.snap4city.org/drupal/node/487
• As Container Composer:
– HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City
– https://github.com/disit/snap4city-docker/tree/master/Alone
Snap4City (C), November 2019 123
118. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B Basic Configuration
Snap4City (C), November 2019 124
TOP
119. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B: Basic) Addressing a relevant number of IOT devices/ IOT
brokers in your smart city or industry 4.0, a few smart
Applications with multiple flows
Snap4City (C), November 2019 125
For managing
beyond A:Alone
Configuration
+ large number
of IOT devices
and brokers,
any protocol,
format
+ IOT Devices
+ IOT Edge
+ Etc.
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator, WS Server,
Data Inspector,
User Stats,
ExternalServices,
Resource Manager,
MicroApplications
Authorization/Authentic.
MyKPI, MyPOI, …
IOT Applications,
Synoptic, Snap4City GIS
WFS player,
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTOBSFs
Sensors/
ActuatorsIOT Edge
External Data Stores
120. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B: Basic) Addressing a relevant number of IOT
devices/ IOT brokers, a few smart Applications
• IOT Edge: local intelligence, processes and Applications located on the
field, on premise. Those autonomous smart tools can be implemented by
IOT Applications working on IOT Edge:
• Linux Ubuntu Appliance VM: https://www.snap4city.org/drupal/node/298
• Android as IOT Edge https://www.snap4city.org/drupal/node/278 and
• Raspberry Pi as IOT Edge: https://www.snap4city.org/drupal/node/77 You can install
Snap4City library in any Node-RED from the official Node-RED library
• IOT Applications = Node-RED + Snap4City libs of Nodes/MicroServices,
– Available in Snap4CityMAIN VM, so that in any installation
– any installation of Node-RED add Snap4City Library from the official Library of
Node-RED, add new Nodes into the Palette as
• https://nodered.org/docs/user-guide/editor/palette/manager
– Snap4City MicroService collection from the JS foundation with full
documentation inside https://flows.nodered.org/?term=snap4city
• BASIC: suitable for Final User
• ADVANCED: DEVELOPER (you have to install both libraries: basic and advanced)
Snap4City (C), November 2019 126
• IOTOBSF VM
– IOT Orion Broker
Fi-Ware
– Secure Filter
(Snap4City)
• IOT Edge support
– Linux Ubuntu
– Windows
– Raspberry
– Android
– Snap4City Library
of nodes
• IOT Devices support
– ESP32
– Arduino
• IOT Application
– Node-RED
– Snap4City Library
of nodes
121. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C) CityStart Configuration
Snap4City (C), November 2019 127
TOP
122. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C: CityStart) Addressing a relevant number of GeoLocated Points of Interest,
PIN on maps, and eventually heatmaps, a Small Smart City with a few smart
IOT applications on cloud and many on edge
Snap4City (C), November 2019 128
For managing
Beyond B:Basic
Configuration
+ Large number
of GEO
elements on
maps
+ Heatmaps
+ Geo shapes
+ GIS, ….
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
KBSSM
GIS, GeoServer
GIS, GeoServer
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTOBSFs
Sensors/
Actuators IOT Edge
External Data Stores
123. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C: CityStart) Addressing a relevant number of GeoLocated
Points, heatmaps, ...
Snap4City (C), November 2019 129
• KBSSM VM
– Knowledge Base
Km4City
– ServiceMap
– ServiceMap3D
– SuperServiceMap
– LOG/Flint
– OSM2Km4City
– Smart City API
– WFS API
• GIS GeoServer VM
– GeoServer
– Heatmap
Manager
• KMSSM VM:
– knowledge base, KB, positioning
elements on maps, enabling geo-
spatial and temporal reasoning
– Ingestion OSM to load on KB,
ServiceMap and reasoning on it
– GeoReverse data, from street to
data and vice versa
– WFS export and import
• GIS GeoServer VM:
– Producing and providing heatmaps
– Managing Heatmaps colour maps.
– WMS, WFS export, distribution
124. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D) DataCity Configuration
Snap4City (C), November 2019 130
TOP
125. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D: DataCity) A medium/large size Smart City with a number of smart
applications on cloud and a number of IOT brokers,
relevant volume of data entering into the cloud.
Snap4City (C), November 2019 131
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTDSES
KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
126. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D: DataCity) A medium/large size Smart City with a number of smart
applications on cloud and a number of IOT brokers,
relevant volume of data entering into the cloud.
Snap4City (C), November 2019 132
• MCLSCount
Container[Cluster]
– Containers models
– DISCES-EM
– Monitoring App
• ETL Server
Data[Cluster]
– DISCES
– Phoenix Drivers
– Hbase Model
– ETL processes
• IOTDSES
DataShadow[Cluster]
– NIFI process
– Elastic Search Model
– AMMA and DevDash
For Managing
Beyond C:CityStart Configuration
+ A relevant number of IOT Applications
for data transformation and Analytics
+ A small sized Data Shadow for IOT
data management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
127. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E) ExtensiveCity Configuration
Snap4City (C), November 2019 133
TOP
128. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E: ExtensiveCity) A Large size Smart City with smart applications on
cloud and a number of IOT brokers,
large volume of data streams entering in the cloud
Snap4City (C), November 2019 134
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
Balancer
IOTDSES KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
129. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E: ExtensiveCity) A Large size Smart City with smart applications on
cloud and a number of IOT brokers,
large volume of data streams entering in the cloud
Snap4City (C), November 2019 135
• IOTDSES
DataShadow Cluster
– NIFI process
– Squid cache on
SCAPI
– Elastic Search Model
extended API
– Kibana
– AMMA data flow
– DevDash data
monitor
– …
For Managing
Beyond D:DataCity Configuration
+ A relevant number of data streams
entering in the platform
+ A scalable Data Shadow for IOT data
management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
130. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F) FullPlatform Configuration
Snap4City (C), November 2019 136
TOP
131. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F: FullPlatform) For Huge scale Smart City with a Huge number of smart
applications and processes on cloud (thousands), any number of IOT brokers
Snap4City (C), November 2019 137
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
Pool of Containers
NodeRed,
Web Scraping
[Penthao Kettle, Java, Karma]
……..
R-Stutdio
Python, C++, C
…………..
Nodes/VM on Cloud in HA, elastic computing, ..
.
.
.
……
……..
……….
.
Masters in HA: Marathon/Mesos, Zookeeper, ….
Master Master Master
containersDistribSrv.
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
NCLSCont
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
VM on Cloud HA, DRS, [FT]
Balancer
IOTDSES KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
132. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F: FullPlatform) For Huge scale Smart City with a Huge number of smart
applications and processes on cloud (thousands), any number of IOT brokers
Snap4City (C), November 2019 138
• MCLSCount
• NCLSCount
Container[Cluster]
– Marathon, Mesos
Cluster
– Containers models
• IOT App
• Web Scraping
• Data Analytics
• ETL
• Python
• ..
– DISCES-EM
• Elastic management
of containers
– Monitoring App
For Managing
Beyond E:ExtendedCity Configuration
+ A scalable huge number of Containers
including different kind of processes:
IOT, ETL, data analytics, scraping
+ A scalable Data Shadow for IOT data
management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
133. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F+) LivingLab Configuration
Snap4City (C), November 2019 139
TOP
134. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F-LivingLab: FullPlatform + LivingLab) For Huge scale
Smart City as Full Platform plus Living lab Support
Snap4City (C), November 2019 140
Snap4City MAIN
Dashboard Builder,
Data Inspector, User Stats,
ExternalSrv, Res Manager,
MyKPI, MyPOI, Synoptic, IOT
Device Manager, Micro
Applications, Authorization/
Authentication, Roles
IOTBrokers, secure
IOTOBSFs
GISGeoServer
Heatmap Manager,
WFS, WMS
KBSSM
KB/ServiceMap,
SmartCity API, WFS,
ServiceMap3D, LOG/FLINT,
SuperServiceMap, Virtuoso
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
Mobile Apps, Dashboards
Container Cluster
MCLSCont, NCLSCont
Marathon, Mesos, Zookeeper,
Disces-EM, Executing in Container:
IOT App, Web Scraping, R-Studio,
Python, Java, ETL, ...
Data Shadow Cluster
IOTDSES
NIFI, Elastic Search, DevDash, Amma
[DataCluster]
ETL-VM
Phoenix HBASE, HDFS
DISCES, ETL support
IOTBrokers, secure
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Drupal support CRMLegacy Platforms
135. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as
Full Platform plus Living lab Support
Snap4City (C), November 2019 141
• Living Lab Portal
– CRM Drupal
– LDAP Snap4City
– Blogs, Articles,
comments, etc.
– Multiple Organization
– Multiple Groups
– SSO with Snap4City
tools
– Automated
Registration
management
– SOLR indexing on
content
– Monitoring activity
– Reporting and
statistics
For Managing
Beyond F:FullPlatform Configuration
+ Providing Living Lab Support, a full
CRM for stakeholder, co-working,
collaborative work, discussion
environment, chart, forum, etc.
Strongly suggested:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• ETL SDK servers
• User Engagement Server
• Routing Solutions
136. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
THE VIEW OF
THE ADMINISTRATOR
Snap4City (C), November 2019 142
TOP
137. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart City API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
Snap4City as a Lambda Architecture
138. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Management by Organization
• Organizations may have
– name, ID, GPS center, a number of Groups on
Snap4City.org
– users of different kinds and may impose early bounds on
the resourced used by users (IOT Dev, IOT App, Dash)
– on cloud users kinds up to level of Tool Administrator
• ToolAdmin users (requested by Organizations) may
– control processes, consumption of resources, healthiness,
etc.
– manage tools exploited in your configuration
• 24H/7D Help Desk and Assistance
Snap4City (C), November 2019 144
139. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Help Desk and SLA
• https://www.snap4city.org/drupal/contact
• Bug Reporting
– https://docs.google.com/forms/d/e/1FAIpQLSfD
QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5
jsoXw/viewform
• For Service Level Agreement see:
– Service Level Agreement
• Help Desk and Contact:
– https://www.snap4city.org/drupal/node/3
• Availability rates:
– https://www.snap4city.org/388
Snap4City (C), November 2019 145
140. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Monitoring Resource
Consumption and Traffic
Snap4City (C), November 2019 149
TOP
141. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 150
142. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Monitoring Infrastructure
Status
Snap4City (C), November 2019 151
http://www.disit.org/dashboardSmartCity/view/index.php?i
ddasboard=MTQ4
http://www.disit.org/dashboardSmartCity/view/index.ph
p?iddasboard=MTkw
http://www.disit.org/dashboardSmartCity
/view/index.php?iddasboard=MjQ5
143. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 152
144. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Computational Capabilities of Snap4City
• Managing:
– Periodic Processes → ETL, IOT App (Node-RED)
– Asynchronous processes, data driven, real time → Node-RED, IOT
Applications
• Scalability
– Horizontal: Increasing processes performing activities, demand on new
processes for new users, for new applications, for new IOT applications:
VM, Hosts, clusters, Storage SAN
– Vertical: Increasing resources on processes: CPU, MEM, Storage, Network
Snap4City (C), November 2019 153
145. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DevDash Case Study (2)
• Detect potential anomalies or disfunctions by
inspecting the DevDash tool time trend
Snap4City (C), November 2019 154
146. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Managing and Monitoring
Data-Traffic in the BackOffice
Snap4City (C), November 2019 155
TOP
147. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA (1)
Monitor data traffic flows among IoT devices,
services, applications etc. and detect potential
anomalies
Snap4City (C), November 2019 156
Unexpected behaviors can
be revealed by inspecting
the data flow time trend:
a) detecting peaks or
valleys in the trend
b) drill-down on data to
identify single/more
malfunctioning devices
and/or services
quantitatively monitoring
data/message traffic and flows
148. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA (2)
Make drill-down activities on data related to a
single Process ID and check for unexpected
behavior in the Time Trend panel:
c) Filtering data by the Process ID (e.g. for
example those related to a SmartWaste
container)
d) Detect a peak with more data traffic than
expected during its scheduled activity, by
properly filtering on time, the single data
portion corresponding to the unexpected
data flow can be viewed
e) Locate on map the single involved device
or service
Snap4City (C), November 2019 157
149. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA by Elastic Search + Kibana
Snap4City (C), November 2019 158
150. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DevDash by Elastic Search + Kibana
Snap4City (C), November 2019 159
151. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Auditing Activities
Snap4City (C), November 2019 160
TOP
152. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 161
153. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Managing Back Office
Processes via Containers
Snap4City (C), November 2019 162
TOP
154. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Hosts with VM on Cloud HA, DRS, FT
Snap4City (C), November 2019
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Hosts/VM on Cloud in HA, elastic computing, ..
.
.
.
……
……..
……….
.
containersDistribSrv.
slave slave slave slave
Pool of Containers
NodeRed, Java
Penthao Kettle, Java, Karma
……..
R-Stat, R-Parallel
Hadoop/Spark/SparkML
Python, C++, C, java
…………..
Master Master Master
Masters in HA: Marathon/Mesos, Zookeeper, ….
VmWare VSphere
Cnt. Scheduling
Cloud Elastic
Management
Cnt. Scheduling
Cloud Elastic
Management
DISCES
+ Planner +
Elastic Management
Containers
Control API
Cloud Management
System &
Resource Monitoring
Cloud Management
System &
Resource Monitoring
Resource Monitoring
&
Cloud Resource
Management
Monitoring API
VM Control API
163
155. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Container
Snap4City (C), November 2019 164
156. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Elastic Scaling: allocating / deallocating
• Allocation/ deallocation, Rebalancing vs compacting
– Vertical of resources: Docker and/or VM: CPU, Mem
• NodeJS multi-flow for each Docker, the user request data flows and IOT App, Snap4City
allocates them dynamically on demand and perform workload optimization
• VM: management of Mem, CPU; transparent and automatic in DRS VMware
– Horizontal of resources of Dockers and/or VM and/or [Host]:
• Docker: addition of containers, migrations/moving, balancing (per moving) of IOT App
• VM: on/off
• Monitoring resources:
– VM via VMware API, Docker via Marathon and Mesos APIs
• Algorithm in Python for scaling, actions via APIs: VMware, Marathon,..
Snap4City (C), November 2019 165
157. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 166
158. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 167
160. DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Distributed SCE Scheduler
Snap4City (C), November 2019 169
162. Snap4City managed to
provide a maximum of
information, flows, in depth
analysis with the data
provided.
There is no
other platform
that collects
all city actors
together.
The City officials and ICT
officials were impressed
with the performance of
the Platform when loading
the heavy, “resource-”
demanding applications and
dashboard.
The data handling
throughout the Platform
is considered as one of
the strong points in the
Platform and of an
extremely sophisticated
level.
The technical level of the Platform and its
strong points such as the way real-time
data is used, the algorithms, data clean-up
possibilities of the Platform, presented data
and information is state-of-the-art and
impressive.
Snap4City (C), November 2019 171
163. Snap4City (C), November 2019 172
Acknowledgements
• Thanks to the European Commission for founding. All slides reporting logo of Snap4City https://www.snap4city.org of
Select4Cities H2020 are representing tools and research founded by European Commission for the Select4Cities project.
Select4Cities has received funding from the European Research Council (ERC) under the European Union's Horizon 2020
research and innovation Programme (grant agreement n° 688196)
• TRAFAIR is a CEF project. All slides reporting logo of TRAFAIR project are representing tools and research founded by the EC on
CEF programme http://trafair.eu/
• Thanks to the European Commission for founding. All slides reporting logo of REPLICATE H2020 are representing tools and
research founded by European Commission for the REPLICATE project. REPLICATE has received funding from the European
Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n°
691735).
• Thanks to the European Commission for founding. All slides reporting logo of RESOLUTE H2020 are representing tools and
research founded by European Commission for the RESOLUTE project. RESOLUTE has received funding from the European
Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n°
653460).
• Thanks to the MIUR for co-founding and to the University of Florence and companies involved. All slides reporting logo of Sii-
Mobility are representing tools and research founded by MIUR for the Sii-Mobility SCN MIUR project.
• Km4City is an open technology and research line of DISIT Lab exploited by a number of projects. Some of the innovative
solutions and research issues developed into projects are also compliant and contributing to the Km4City approach and thus are
released as open sources and are interoperable, scalable, modular, standard compliant, etc.
INEA CEF-TELECOM Project
funded by European Union
164. Roadmap
- Twitter Vigilance
- Social Media
Analytics, Sentiment
Analysis
- User engagement
- Bike Sharing
- Data Analytics ++
- Social Predictions
- OBD2
- Sardinia Region
Smart City
Strategies and plan
GHOST SIR
(2016-19)
- Smart Energy
- Sustainable Mobility
- Control Room
- Dashboard
- Resilience
Decision Support
- Smart First Aid
- User Behaviour
Analysis, predictions
- Risk Analysis
- Origin-Destination
and trajectories
- Traffic Reconstruction
- Offer Analysis
- OBU, smart devices
- Weather
Forecast
- Real Time Wi-Fi
- Entertainment
- Events
- LOD
- Tuscany
Road Graph
- Mobility
- culture, tourism
- Events
- Parking
- Services
- Linked open graph
2013
2014
2017
2018
2015
2016
Km4City 1.4
- Infomobility
- Mobile App
- Routing
- Multimodality
Km4City
Ontology 1.1 Km4City 1.6.2
- IOT/IOE, IOT App
- Living Lab
- Maker Support
- IOT Edge
- Smart City IOT
- GDPR
- Privacy & Security
CEF
DISIT lab roadmap vs model and tools’ usage
Km4City 1.6.6
5G tech
Energy
Industry 4.0
Synoptics
…2022
IOT/IOE
SII-MOBILITY SCN
(2016-19)
- Smart Waste
(2017-20)
(2015-18)
H2020
(2016-21)
H2020
(2018-21)
- Mobility
Demand / Offer
Analytics and
Strategy
(2018-20)
Km4City 1.6.4
H2020
(2017-19)
GREEN IMPACT
- Industry 4.0
- Critical Plant
- Monitoring
POR FESR 2014-2020
Km4City 1.5
2019
173
- Traffic and Mobility
Impact on Pollution
- NOX predictions(2018-21)
2020
165. Market Solutions
Snap4City (C), November 2019 174
IOTDiscoveryAbstraction
Authentication,Authorization
Securityend-2-end,secureon
IOTandDashboards
OpenHWandOpenSW
IntegratedCommunity
management
DataTypes:IOTDevices,IOT
App,Dashboard,Data
DataType:Publish/share,
Delegation,Consentandchange
DataType:Downloadand
Delete
AuditingonDataTypeAccess
OpenSourceend-to-end
ScalabilityIOT
VisualProgrammingend-to-end
applications
AdvancedSmartCityAPI,
MicroServices
MultiDomainSemantic
Platform
StandardbasedModulesand
IOT,OpenDevices
ResourceSharing
DataAnalyticsintegrated
DashboardH24/7,protected
connection
Multi-protocolonIOT
G G G G G
Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y
Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http
IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y
IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT
FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y
ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited
Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP
AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited
Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited
PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y
Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y
CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N
Siemens MindSphere
[60]
Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y
Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT
Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP
Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited
Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
166. Requirements vs Modules
Snap4City (C), November 2019 175
Requirement.description SecurityandPrivacyManagement
Ownership&Delegation
PersonalData
IoIContextbrokers
IoTDirectory
IoTFirewall
IoTApps,andmanagement
DashboardBui8lderand
management
MicroSerivces
SmartCityAPI
IoTDevices
IoTEdge,IoTApponpremise
DataShadowandStorage
KnowledgeBaseKm4CIty
Snap4CityPlatformSupportLiving
Lab
IV.A
IV.D
IV.C VI IV.C
VI.A
IV.C IV.C IV.A V.B IV.C
V.C
IV.B IV.C V.A
V.C
IV.B
V.B
IV.B
IV.C
IV.B IV.A
IV.B
R1. IOT Brokers X X X
R2. IOT Discovery Abstraction X X X
R3. Authentication, Authorization X X X X
R4. Inform User of Security Level X
R5. Developing Secure Applications X X X X X X
R6. Secure Communications X X
R7. Open HW and Open SW X X X X X
R8. Signed Content Vs Data Types X X X
R9. Managing IoT Data Types X X X X X X X X X X X X X X
R10. Managing Ownership and Delegation X X X X
R11. Support Roles, Org. and Groups X X X X X X X
R12. Encrypted Personal Data X X X
R13. User Profile Management, forgotten X X
R14. Auditing Data Types X X X X X X X X X X X X X X X
R15. Data Breach Detection X X X X X X X X X X X X X X X
R16. Accounting X X X X X X X X X X X X X X X
R17. Data Protection by Design X X X X X X X X X X X X X X X
R18. Evidence of Level of Security X X X
167. GDPR vs Requirements
GDPR Compliance Verification Feature Verif. Reqs.
Signed consent UI R8
User profile management and control UI R13
Data Type private as default UI R8
Rights to access per element UI R9
Rights to transfer per element UI R10
Rights to erase per element and total UI R13
Rights to revoke/change per Data Type UI R10
An interface for Right management for Data Type UI R9
Clear Terms of Use and Privacy Policy UI --
Auditing Tools for Data Type UI R14
Publish as Anonymous UI R9
Encrypt personal users’ data Code R12
Secure Authentication and Authorization Code R3
Data protection by Design Code R17
Secure connection Code R6
Security Control, data breach control, anonymization, etc. PEN
Test
R15,
R16, R18
Snap4City (C), November 2019 176