SlideShare a Scribd company logo
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
1
LIVING LAB
Be smart in a SNAP!
6th Day, System & Deploy, Nov 2019
https://www.snap4city.org/501
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
Snap4City (C), November 2019
Paolo Nesi, paolo.nesi@unifi.it
https://www.Km4City.org
https://www.disit.org
https://www.Snap4City.org
scalable Smart aNalytic APplication builder for sentient
Cities: for Living Lab and co-working with Stakeholders
Powered by
2
6th Day, System & Deploy, Nov 2019
https://www.snap4city.org/501
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
General Overview of the full Course
• 1st Day: General overview (1 day, 6 hours)
• 2nd Day: Dashboards, how to build and manage them (4 hours)
• 3rd Day: IOT Applications development, IOT Devices, IOT Networks (4 hours)
• 4th Day: Data Analytics, in R Studio, In Python, how to integrate with IOT
Applications (4 hours)
• 5th Day: Data Ingestion, Data Warehouse, ETL Development, Data Gate, IOT
Device Data ingestion, etc.. (5 hours)
• 6th Day: Snap4City Architecture, How To Install Snap4City (3 hours)
• 7th Day: Smart city API (internal and external) Web and Mobile App
development tool kit (4 hours)
A number of the training sections include exercitations
Updated versions on: https://www.snap4city.org/501
Snap4City (C), November 2019 3
6th day Agenda
• Snap4City Architecture
• Snap4City: Smart City IOT as a Service
• Snap4City Living Lab For Collaborative Work
– Smart City Development Life Cycle
– Analysis and Design for Innovation (Co-Creation and Co-Working)
– Development Tools
– How to Add Functions that are not present in the Platform
• Snap4City vs Fi-Ware
• Snap4City vs State of the Art Solutions
• Snap4City Services: Consulting and Developing
• Snap4City vs Snap4Industry 4.0
• Installing Snap4City
• The view of the Administrator
– Monitoring Resource Consumption and Traffic
– Managing and Monitoring Data Traffic in the BackOffice
– Auditing Activities
– Managing Back Office processes via Containers
• Acknowledgement
Snap4City (C), November 2019 4
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
GO
Levels of Difficulty
• Easy.
• Moderate.
• Good.
• Golden.
• Professional.
• Excellent.
Snap4City (C), November 2019 5
non programmer level
Some JavaScript rudiment coding
JavaScript programming
Programming in R Studio
Exploiting Smart City API
Developing Full IOT Applications,
Dashboard and Mobile Apps
Snap4City:
Architecture
Snap4City (C), November 2019 6
TOP
7
Standards and Interoperability
Snap4City (C), November 2019 8
Compliant with: AMQP, COAP, MQTT, OneM2M, HTTP, HTTPS, TLS, Rest Call,
SMTP, TCP, UDP, NGSI, LoraWan, TheThingsNetwork, SigFOX, DATEX II, SOAP,
WSDL, Twitter, FaceBook, Telegram, SMS, OLAP, MySQL, Mongo, HBASE, SOLR,
SPARQL, EMAIL, FTP, FTPS, WebSocket, WebSocket Secure, ModBUS, OPC, GML,
RS485, WFS, WMS, ODBC, JDBC, Elastic Search, Phoenix, XML, JSON, CSV, db,
GeoJSON, Enfuser FMI, Android, Raspberry Pi, Local File System, ESP32,
Libelium, IBIMET, OBD2, SVG, XLS, XLSX, TXT, HTML, CSS, etc.
Project and partners, a few
Snap4City (C), November 2019 9
Unique of Snap4City Platform (1)
• Data ingestion and model
– Unified data model (exploited in the Wizard and Knowledge base)
– Semantic Reasoner modelling city entities, supporting sematic
search, expert system, digital Twin, etc.
– IOT Directory abstracting complexity of IOT
Devices, Edge, Brokers, protocols and data formats
• Data Analytics and Data Processes
– Flexible and extensible IOT Applications
– Data Analytic: multiple programming languages,
• Visual Analytics, dashboarding, Apps
– Wizard: expert system for immediate dashboard production
matching data vs graphics representation
– Dashboards specialized multidomain for Smart Cities
– Custom Widgets and Synoptics
– Ready to use Mobile App, instant App, MicroApplication
– Strategies formalization supports
Snap4City (C), November 2019 10
Unique of Snap4City Platform (2)
• Openness to any developers
– Living Lab support for coworking, sharing, and delegating
– Advanced Smart City APIs and MicroServices
– 100% Open Source, Open hardware
• Security and Privacy
– End-2-end encrypted communication,
on devices, platform, ... dashboards
– GDPR compliant privacy/security
• Non functional
– on cloud and on premise, your private installation
– Ready to use Appliance Virtual Machines and/or
Containers for a modules and tools.
– Flexible, Modular, Elastic, scalable and robust
Snap4City (C), November 2019 11
Snap4City (C), November 2019 12
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
How to adopt Snap4City
Snap4City (C), November 2019 13
Https://www.snap4city.org
Smart City as a Service
• Supporting Org
• 100% Open Source Platform: Github
• Further developments
• Publishing Appliances and Dockers
• Training courses, docs
• Consulting
• Forums
• Etc.
Download and
deploy
Installation on your premise
• Virtual Machines or Dockers
• Different configurations
• From small to scalable
• Exploiting your legacy tools
• Interoperable with any tool
• No vendor lock-in, No tech lock-in
Mixed solutions! For example:
• Start on Cloud as Smart City as a Service
• Migrate on premise on the fly
• Start on Cloud into a sand box
• Pass to install on premise what you need
On your premise
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 14
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
Snap4City (C), November 2019 15
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
Snap4City:
Smart City IOT as a Service
Snap4City (C), November 2019 16
TOP
Smart City IOT as a Service
• Snap4City applications may exploit multiple paradigms as data driven,
stream and batch processing, putting co-creation tools in the hands
of:
– Smart Living Lab users and developers a plethora of solutions to develop
applications without vendor lock-in nor technology lock-in,
– final users customizable / flexible mobile Apps and tools,
– city operators and decision makers specialized / sophisticated city dashboards
and IOT/IOE applications for city status monitoring, control and decision
support.Open to Organizations
• Training and manuals: https://www.snap4city.org/108
• Help Desk: https://www.snap4city.org/3
• SLA: https://www.snap4city.org/497
• Terms of Use: https://www.snap4city.org/drupal/legal
Snap4City (C), November 2019 17
https://marketplace.docker-fid.grid.cyf-
kr.edu.pl/services/snap4city
Specialized Offer
Snap4City as Smart City IOT as a Service for
• Who would like to create Living Labs as community exchanging
experience with other cities as well;
• Research Institutions, Departments and Projects which would like to
perform research and experiments in the area of Smart City and IOT,
without the needs of setting up the infrastructure, exploiting open
data, collaborating, accessing to Data Analysis on demands, etc. This is
the spirit of EOSC, European Open Science Cloud Marketplace at
which Snap4City is registered as DISIT Lab, see [EOSC].
• Public Administrations, as small cities that would like to offer smart
services and does not have economic power to manage service on t
heir premise from them self.
Snap4City (C), November 2019 18
Help Desk and SLA
• https://www.snap4city.org/drupal/contact
• Bug Reporting
– https://docs.google.com/forms/d/e/1FAIpQLSfD
QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5
jsoXw/viewform
• For Service Level Agreement see:
– Service Level Agreement
• Help Desk and Contact:
– https://www.snap4city.org/drupal/node/3
• Availability rates:
– https://www.snap4city.org/388
Snap4City (C), November 2019 19
SNAP4CITY LIVING LAB
FOR COLLABORATIVE WORK
Snap4City (C), November 2019 20
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 21
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
Snap4City (C), November 2019 22
Your Level
SLIDES
News
Flyer
Exercises
Full Search
All Tools
Personalized Suggestions
Virtual Assistant
Snap4City (C), November 2019 23
Living Lab Flexibility
• Multiple modalities to perform the
same activities
• Tuned for Beginners and Skilled people
• Visual interface and programming tools
• Resources and artefacts sharing for
learn acceleration and co-working
• Open Living and co-working Portal:
https://www.Snap4City.org
Snap4City (C), November 2019 24
Snap4City Satisfies all
Requirements of ENOLL
Select4Cities and EIP-SCC
For the user: different levels of engagement
• Multiple Organizations
• Roles:
– Managers
– AreaManagers (developers):
special access to computing
resources
• Levels for self-assessment
– Suggestions on next steps to
learn on the basis of personal
progresses Snap4City (C), November 2019 25
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Level 1 Users: creating dashboards
See how Dashboards can be created using the wizard: dashboards with selectors,
time trends, maps, etc.
• TC1.8. Visual production of Dashboard via Wizard
• TC1.9. Search on Wizard for any kind of data managed into the platform, from POI
to sensors, KPI, social, etc.
• TC1.10. Dashboard delegation to access, and passage of ownership, and/or
cloning
• TC1.11. IOT Discovery, on Dashboard Wizard
• TC1.13. Dashboard Builder External Services and Widgets
Snap4City (C), November 2019 26
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 27
All Text on the Portal are
Hypertext with Links for
navigation among major
concepts
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
MultiOrganization, Groups and Profiles
Organizations may have their distinct :
• menus and functionalities, GeoArea, Data, Dashboard,
Groups of users, managers, Knowledge Base, repositories,
etc.
Users may:
• Have personal IOT Devices, Data, Dashboards, IOT App,..
• Have access to multiple Groups of Multiple Org.
• Delegate them in usage or access
• Assesses their usage and themselves
Snap4City (C), November 2019 28
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Self
Assessment
Snap4City (C), November 2019 29
Snap4City (C), November 2019 30
Validation with developers
• Helsinki and Antwerp, plus Florence Training, CINI Challenge, ..
• 65 performed operational activities:
– dashboards, IOT Applications, registering IOT devices, etc.
– More than the 80% created both Dashboards and IOT Applications, thus validating the solution
and the process of engaging them in working on the platform
The 65 users left on
platform
Average per day
over last 90 days
Total activity
90 days
Number of IOT Applications 117 81,6 7341
Number of private IOT devices 27 25,5 2296
Number of public dashboards 11 6,2 562
Number of private dashboards 173 135,1 12159
Number of accesses to dashboards -- 33,9 3048
Number of minutes -- 337,1 30337
Snap4City (C), November 2019 31
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Smart City
Development Life Cycle
Snap4City (C), November 2019 32
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 33
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 34
Formal DesignLegenda phase Tool/Process MicroService/Node Widget
In/Out
Functional
Historical
Data
Visual Interface
Analysis
Design
IOT Application
Implementation
Requirements
Specification
ServiceMap
Data Inspector
Discovery
IOT Directory MS
Functional MS
Analytical
Data Analytics MS
Yes
Data Ingestion&modeling
Data Gate/ETL
IOT Directory for IOT
Device Regis.
External Services
No
Search/Store MS
Ext. Services MS
Python
R-Studio
Java, ……
Develop. Data Analytic
Dashboard MS
IOT App Widgets
Data Flow Development
Dashboard Builder
& Wizard
User Interface
Development
Utilities MS
Private Info MyKPI, Per. MSMyKPI Modeling
Analysis and Design
Development, Deploy, Testing
Deploy,
Testing
Dev. Special Tools
IOT App Widgets
Custom Widgets
Regular Widgets
Web Scraping WebScraping MSWeb Data
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 36
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
Analysis and Design for Innovation
(Co-Creation and Co-Working)
Snap4City (C), November 2019 37
TOP
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 38
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Analysis & Design for Innovation
• Analysis
– The analysis start with a number of meetings/interviews with stakeholders
– The identification of the target stakeholders/actors/users (target Segments) and their
definition/description
– The meetings are focused on filling the INNOVATRIX approach of IMEC tuned for the Snap4City
platform and solution
– See the schema of the INNOVATRIX reported in the next slide
• Design
– Focused on creating a large number of Use Cases and/or Scenarios for development
– The design start taking into account the Snap4City development life cycles and tools. Thus shortening
all the boring activities and following the typical Snap4City rapid prototyping described in these slides
• Data Discovery
– Data discovery is performed on analysis of the: (i) identified scenarios, (ii) data of the stakeholders,
(iii) international sources, (iv) Snap4City experience, etc.
– Performed by following the Snap4City guidelines on Data Search on web and world.
Snap4City (C), November 2019 39
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
INNOVATRIX by IMEC for the Analysis
Snap4City (C), November 2019 40
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Use Cases: Design vs Data Discovery
• The analysis may produce possible needs vs current practices
• The new innovative solutions have to be identified by experts working
on these tables, thus producing Use Cases, Scenarios
• Each of them has to be described and assessed taking into account:
– Innovation level, social impact,
– data availability (and thus as output of the Data Discovery)
– User interaction and behavior constraints
– technical feasibility, implementation costs in the context of Snap4City (data
ingestion cost, data costs, development costs, etc. )
– Time and phases to for the development
– Validation approach: feasibility and costs
Snap4City (C), November 2019 44
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Taking Decision with all: co-creation
• Once performed the above described approach
– A number of Use Cases, Scenarios have been created and well formalized
• Than they may be
– shortlisted according to the main stakeholder goals (city, operators, etc.)
– voted in a participatory approach in the Living Lab
– selected to be developed according to the next steps described in the
following
• Once developed, a validation with the same stakeholder is needed
Snap4City (C), November 2019 45
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Tools
Snap4City (C), November 2019 46
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 47
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
• Open Data:
– Data gate, federation of Open
Data Portals
– ETL processes (PULL)
– IOT Application processes
• IOT Networks:
– IOT Application processes, data
driven or PULL
– IOT Brokers (Push) → IOT
Shadow
• Web Pages:
– Web scraping, crawling
processes
• Social media: Twitter, Facebook,..
– Twitter Vigilance, IOT App
• Mobile Apps
– Smart City API
• Files upload: CSV, Excel, etc.
– IOT Applications, ETL
• REST API, WS, FTP, etc.
– IOT Applications, ETL
• Data base accesses
– GIS: WFS, WMS
– ETL, IOT Application Snap4City (C), November 2019 48
My Files
IOT App
ETL
IOT Broker
IOT Device
IOT Broker
Web Scraping
DataGate
API, External Services
Rest Call …….MS
GIS data, Maps, …
Sensors/
Actuators IOT Edge
IOT Device
External Data Stores
Any kind of data and flows
IOT Network Manager vs Final User
Snap4City (C), November 2019 49
IOT Broker
IOT Broker
IOT Broker
IOT Broker
Network of IOT Brokers
Knowledge Base,
Km4City
Knowledge and Storage
Data from the Field and
City
Discovering
IOT Network
Manager
Registering
Browsing
ServiceMap
Knowledge Base
Discovering
IOT Application
Dashboard Wizard
IOT Directory
Final user
Manager
My IOT Device
Register
Snap4City (C), November 2019
50
Snap4City (C), November 2019 51
Developing IOT Applications
Snap4City (C), November 2019 52
MicroServices collections IOT App. Editor
Resource Manager
Sharing/saving
reusing IOT App
My IOT Applications
ServiceMap Discovery Dashboard Collection,
Editor and Wizard
Generating IOT App
With Dashboard
Knowledge Base, Km4City
IOTDiscovering
Web Scraping
Snap4City (C), November 2019 53
Web Scraping
Snap4City (C), November 2019 54
IOT App. Editor
Resource Manager
Sharing/saving
reusing Scraping
My Scraping processes
Generating
WEB Scraping
Knowledge Base, Km4City
Web Scraper PORTIA
Integrated DataGate/CKAN
Static open data ingestion
Snap4City (C), November 2019 55
Data Set:
- Search
- Loading
- Download
- Share
- Publish
- Also automated
Automated data
regularization
Federated Crawling
Federated Distribution
Data Ingestion via Data Gate
Snap4City (C), November 2019 56
Federated Network
Saving /
Sharing
reusing
Knowledge Base, Km4City
Knowledge and Storage Data
from the Field and City
Data Gate
Data Set: sharing,
Harvesting, Loading/
Downloading
Data Set:
- Search
- Loading
- Download
- Share
- Publish
- Also automated
Developer in ETL: Penthao Kettle
Snap4City (C), November 2019 57
Batch Processing for dynamic data ingestion
Developing ETL, Data Manager
Snap4City (C), November 2019 58
ETL Process
ETL Process
ETL Process
ETL Process
Resource Manager
Saving /
Sharing
reusing
ETL Development EnvironmentDISCES scheduler productionDistributed Back Office
Knowledge Base,
Km4City
Knowledge and Storage Data
from the Field and City
Data Gate
Data Set
Saving /
Sharing
Data Set
Loading/
Downloading
Data
sources
Load data or prepare for data ingestionSchedule Execute
Resource Manager: public and sharing
Snap4City (C), November 2019 59
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Data Gathering and Knowledge Management
• Data ingestion can be performed by using multiple tools:
– ETL processes, IOT Applications, Data Gate, WebScraping. We suggest:
• ETL for static / periodic data in PULL
• IOT App for real time data and flow, from IOT Brokers/Devices
• DataGate for Static Data, upload them as files, or collected from other CKAN
• WebScraper for scraping data from Web Pages, when authorized!
• See how to test cases:
– HOW TO: add data sources to the Snap4City Platform
– HOW TO: define privacy rules for personal data, produced by the end-users own device
– US6. Developing and using processes for data transformation
– TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share
– TC6.3 - Creating ETL processes for automated data ingestion and data transformation
– TC6.5 - Managing Heterogeneous File Ingestion via ETL processes
– TC6.9 - ETL processes for multiprotocol and format data ingestion, see on GITHUB for library
– TC9.2 - Managing heterogeneous File Ingestion, protocols, formats via IOT applications, and open
standards
Snap4City (C), November 2019 60
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Interoperability ESRI
• Snap4City is interoperable with
– ESRI ArcGIS Enterprise, Portal, Pro/MAP, ...
– other GIS tools supporting WFS, WMS,
GeoJSON, GML
• Snap4City is interoperable since:
– Provides info/data in WFS, WMS
– Exploits data/info from WFS, WMS
– Import data/info from WFS/WMS
• The Snap4City platform can be installed
on premise using Snap4City Appliance
https://www.snap4city.org/drupal/node/471
– StartSNAP4CITYVM includes the Dashboard Builder that is capable to work
with WFS WMS protocols for the integration with GIS platforms as ESRI
ArcGIS, QGIS, directly or using Snap4City GIS player.
– KBSSMVM includes the Smart City API and WFS API which can be used to
data harvest from any GIS servers and GIS desktop tool
Snap4City (C), November 2019 61
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Linked Open Graph
Snap4City (C), November 2019 62
LOG: https://log.disit.org
Schema: http://www.disit.org/km4city/schema
RDF version: http://www.disit.org/km4city.rdf
Linked Open Data
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 63
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Data Analytics Dev. in R Studio and/or Tensor Flow
Snap4City (C), November 2019 64
Knowledge Base,
Km4City
Ontology Schema
SPARQL, FLINT
LOG.disit.org
Swagger
SmartCityAPIfromKnowledgeBaseandothertools
Big Data
Store
Facility
Resource Manager
Creating
MicroServices
Saving /
Sharing
reusing
Using them into
IOT Applications
Data Analytics Dev. in Java, Python, ..
Snap4City (C), November 2019 65
Knowledge Base,
Km4City
Ontology Schema
SPARQL, FLINT
LOG.disit.org
Swagger
SmartCityAPIfromKnowledgeBaseandothertools
Big Data
Store
Facility
Resource Manager
Saving /
Sharing
resusing
Process
Process
Process
Process
Distributed Back Office
DISCES scheduler
Monitoring
Coding
Testing
Data
sources
Loading new Node-RED
nodes/microservices from external Palet
• Ask to RootAdministrator to have your custom Node-RED
nodes/modules added to your IOT Applications
– https://www.snap4city.org/drupal/contact
• A validation of requested Node-RED nodes will be performed to
avoid violation of security and privacy for all
• Administrators may load custom Node-RED nodes/modules
• This limitation is not present in your on premise installations of
Snap4City
Snap4City (C), November 2019 66
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 67
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard List and Editor
Snap4City (C), November 2019 68
Dashboard List and Editor
Snap4City (C), November 2019 69
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard Wizard
Snap4City (C), November 2019 70
• Select the area of your interest: panning and zooming
• Select the
• graphic aspect of your interest, or
• High Level Type of your interest, or
• Make a search if you a have a precise idea or
• Act on filters: nature, subnature, type, name, value, date, health, owner, …
• Combine them as you like
• Select the lines of your interest
• Then click on Next and get the Dashboard by wizard
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboard Wizard
Snap4City (C), November 2019 71
The Wizard help you in selecting
only possible combination of data
vs graphic representation
Developing Dashboards
Snap4City (C), November 2019 72
Knowledge Base,
Km4City
Create, save, load,
delegate, grant access
Dashboard
Collection
Dashboard Wizard
IOT Applications
MicroApplications
External Services
Dashboard Editor
Widget Collection
Knowledge and Storage
Data from the Field and
City
My Own Dash/App
Dashboard Development
Snap4City (C), November 2019 73
Knowledge Base,
Km4City
Create, save, load,
delegate, grant access
Dashboard
Collection
Dashboard Wizard
IOT Applications
MicroApplications
External Services
Dashboard Editor
Widget Collection
Knowledge and Storage
Data from the Field and
City
My Own Dash/App
Developing new Dashboard Graphic Widget
• The development of new Widget is feasible for programmers
– in PHP, JavaScript and CSS
• The starting point is the Core Snap4City Virtual
Machine Appliance StartSNAP4CITYVM:
https://www.snap4city.org/471
• While the source code is also accessible on GitHub/DISIT
• see TC3.11 - New graphics widget can be easily created, Dashboard Widget
Creation
– creating a new Graphic widget via SVG and programming: Custom
Synoptics and Widgets for Dashboards https://www.snap4city.org/504
Snap4City (C), November 2019 74
https://www.snap4city.org/drupal/node/504
Snap4City (C), November 2019 75
Custom Dashboards and Widgets (interactive,
Animations, etc.)
-- SVG for graphic design
-- MyKPI for collecting data
Custom Widgets
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Creating custom Widgets for Dashboards
Snap4City (C), November 2019 76
Designer
JS programmer
Producing
and
Publishing
MyKPI
WS secure
USING
IOT Edge
IOT Devices DEVELOPING
JS Library
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Dashboards
• Suitable as: City Dashboard, App interface, and Control Room Dashboards,
Situation Room Dashboard, Operator Dashboard
• Created visually compounding graphic Widgets
– Each widget has an autonomous update
– Each metric/data-source may have associated with an alarm: blinking and sending
events to people and machines in different manners
• Can be: public or private, private dash can be delegated or passed in ownership
• See https://main.snap4city.org/management/dashboards.php?linkId=dashboardsLink&fromSubmenu=false&sorts[title_header]=1
• See the following tutorials
– HOW TO: create a Dashboard in Snap4City
– US1. Using City Dashboards
– US2. Using and Creating Snap4City Applications with Dashboards
– US4. Creating City Dashboards and related Event Monitoring and Actions
Snap4City (C), November 2019 77
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Level 1 Users: using dashboards
• TC1.3. Accessing and using Dashboards with any device
• TC1.4. Dashboards Showing Data Real Time and
Historical/trends, comparison
• TC1.5. Dashboards showing a range of different High
Level Types: KPI, POI, IOT dev, MicroApp, Maps using
different kind of Graphics Widget. Monitor city status with
Dashboards
• TC1.6. Dashboards using different kind of graphics
Widgets matching with High Level Types, Monitor city
status with Dashboards
• TC1.12. Dashboard with MicroApplications
• TC4.1. Dashboard and Notificator
• TC4.6. Dashboard with city events, ESB, police, traffic, etc.
Snap4City (C), November 2019 78
Snap4City (C), November 2019 79
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data
Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Application
Development
Application
Requirements
Analysis
AdvancedSmartCityAPIand
MicroServices
Snap4City Mobile
& Web Apps
Development Kit
Deploy
Testing
Publication
Production
Smart City Services Mobile and Web Apps
Develop Mobile & Web Applications
Exploiting Snap4City Smart City Services
Developing Web and Mobile Apps, MicroApps,..
Snap4City (C), November 2019 80
Knowledge Base,
Km4City
Swagger
Advanced Smart City API
Web App HTML5, MicroApplications
Embed into Web pages
Mobile Apps
ServiceMap
DeveloperKm4City Open
Source
development
tool kit
City User
Mobile Application
Monitoring
Administrator
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
IOT Directory
Back Office Processes
IOT Broker
IOT Broker
IOT Broker
IOT Broker
ETL Process
Data Analytics
ETL Process
ETL Process
ETL Process
Data Analytics
Data Analytics
Data Analytics
Knowledge Base,
Km4City
Smart City API from Knowledge Base and other tools
Ontology SPARQL, FLINT LOG.disit.org
ServiceMap ServiceMap3D
Swagger MicroServices
IOT ApplicationsWeb and Mobile AppsDISCES and back office management tools
Snap4City (C), November 2019 81
MicroApplications
Resource Manager
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
How to Add Functions that are not
present in the Platform
Snap4City (C), November 2019 82
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Connect
IOT/IOE
Upload context
Open Data
Connect external
Services
Advanced Smart
City API,
MicroServices
Manage
Apps & Dashboards,
User Engagement
Monitor
City Platform
experiments
workshops
tutorials
networking
agreements
events
Start-ups
Research
groups
City
Users
Large
Industries
collaborations
Licensing,
Gold services
personal
services
Case
Studies
Inhouse
companies
Resource Operators
Tech
providers
partnerships
documentation
Help
desk
Category
Associations
Corporations
Advertisers
Community
Building
subscription to
applications
Produce City
IOT Applications &
Dashboards
Promote
Applications &
Dashboards
Produce Apps and
Dashboards for City
Users
Data Ingestion and
Analytic algorithms
Early Adopters
Snap4City (C), November 2019 83
Collaborative
Platform
hackathons
GO!
City Operators
Living Lab Accelerating
Development Life Cycle
Smart City Services
Snap4City (C), November 2019 84
Analysis
Design
Data
Discovery
Data
Ingestion
IOT App
Development
Dashboard
Development
Data Analytics
Development
Deploy
TestingPublication
Production
Special Tool
Development
Data Analytics
Development
Deploy
Analysis & Design
Adding a Function that is not present on the platform
• Snap4City allows at the AreaManagers to Develop new functions
for:
– Interacting with the field: IOT Sensors/Actuators, IOT Devices
– Data production by using: IOT Devices, IOT Brokers, ..
– Data Ingestion/Transformation by using: ETL, IOT Applications, WebScraper,
DataGate
– Data Analytics by using: R Studio, Java, C++, Python
– Data Rendering on Dashboards: PHP, HTML5/JavaScript, Web App, etc.
– User interaction and App: Mobile App, MicroServices, Dashboards
Snap4City (C), November 2019 85
Snap4City vs Fi-Ware
Snap4City (C), November 2019 86
TOP
Snap4City (C), November 2019 88
89
• Is open to the Development of Applications
leaving large space to developers
• Is cantered on the Orion Broker that result
central in the architecture: any Broker or data
source is sending data to Orion
• Security level is not clear, partially demanded
to developers
• Visual Flexible IOT processing is not clearly
provided
• Limited API for IOT data access
• Knowage BI presents several limitations in
showing Smart City Data
• Market place on Open Data
• Support of Developers via Fi-Ware
• Deployed as VM and Dockers
• open source, not the application parts
Snap4City (C), November 2019 90
• Is open to the Development of Applications leaving large
space and providing a large set of ready to use
applicative tools and solutions to build their solutions on
top or aside.
• Is fully distributed, any kind of data source can be
ingested, automatically.
• Orion Broker is only one of the Brokers that can be used.
It can be also protected by Snap4City tech, with Mutual
Authentication
• Visual Flexible IOT processing is provided as Node-RED
and Snap4City MicroServices suites
• Advanced Smart City API are provided on top of
Knowledge Base
• Dashboard Builder has been designed for Smart City Data
and automated dashboards’ production
• Market Place on Open Data, tools, processes, experiences
• Full Support for Living Lab of the city, coworking, tutorials
• Deployed as VM and Dockers
• 100% open Source
vs
vs
• Snap4City is an official Fi-Ware Solution via
– NGSI V1, V2 The IOT Orion Broker
– IOT Orion Broker can connect JSON, MQTT, Lightweight M2M,
LoraWAN, OPC, SigFOX, etc. see Fi-Ware Https://www.fiware.org
• Snap4City is compatible with all the above protocols
– via IOT Orion Broker,
– via direct connection on ETL processes on their corresponding IOT
brokers, and/or
– via IOT Applications.
• Snap4City is also compatible with many other protocols, see
the table reported in page: https://www.snap4city.org/65
Snap4City (C), November 2019 91
Standards and Interoperability
Snap4City (C), November 2019 92
Compliant with: AMQP, COAP, MQTT, OneM2M, HTTP, HTTPS, TLS, Rest Call,
SMTP, TCP, UDP, NGSI, LoraWan, TheThingsNetwork, SigFOX, DATEX II, SOAP,
WSDL, Twitter, FaceBook, Telegram, SMS, OLAP, MySQL, Mongo, HBASE, SOLR,
SPARQL, EMAIL, FTP, FTPS, WebSocket, WebSocket Secure, ModBUS, OPC, GML,
RS485, WFS, WMS, ODBC, JDBC, Elastic Search, Phoenix, XML, JSON, CSV, db,
GeoJSON, Enfuser FMI, Android, Raspberry Pi, Local File System, ESP32,
Libelium, IBIMET, OBD2, SVG, XLS, XLSX, TXT, HTML, CSS, etc.
• In Snap4City you can chose to connect your devices at Snap4City
Platform in different manners:
– (a) directly to Snap4City, or
– (b) via an IOT Orion Broker (external IOT Broker or those provided by
Snap4City), or
– (c) via any third party IOT Brokers in any protocol you have.
• Snap4City has
– Improved IOT Orion Broker with the so called Orion Broker Filter (Orion
Broker Filter, NGSI Security Wrapper) which is a secure wrapper for NGSI V1
and V2 protocol for enforcing Mutual Authentication, Security, roles, etc.
– Produced open hardware and open software NGSI Compliant: as
• IOT Devices with mutual authentication and security based
for NGSI on: Android, Arduino and ESP32, IOT Button, etc.
• IOT Edge devices with mutual authentication and security based
for NGSI on: Raspberry PI, Windows, Linux.
Snap4City (C), November 2019 93
vs
Snap4City vs
State of the Art Solutions
Snap4City (C), November 2019 94
TOP
Market Solutions
Snap4City (C), November 2019 95
IOTDiscoveryAbstraction
Authentication,Authorization
Securityend-2-end,secureon
IOTandDashboards
OpenHWandOpenSW
IntegratedCommunity
management
DataTypes:IOTDevices,IOT
App,Dashboard,Data
DataType:Publish/share,
Delegation,Consentandchange
DataType:Downloadand
Delete
AuditingonDataTypeAccess
OpenSourceend-to-end
ScalabilityIOT
VisualProgrammingend-to-end
applications
AdvancedSmartCityAPI,
MicroServices
MultiDomainSemantic
Platform
StandardbasedModulesand
IOT,OpenDevices
ResourceSharing
DataAnalyticsintegrated
DashboardH24/7,protected
connection
Multi-protocolonIOT
G G G G G
Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y
Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http
IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y
IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT
FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y
ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited
Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP
AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited
Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited
PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y
Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y
CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N
Siemens MindSphere
[60]
Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y
Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT
Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP
Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited
Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
Snap4City managed to
provide a maximum of
information, flows, in depth
analysis with the data
provided.
There is no
other platform
that collects
all city actors
together.
The City officials and ICT
officials were impressed
with the performance of
the Platform when loading
the heavy, “resource-”
demanding applications and
dashboard.
The data handling
throughout the Platform
is considered as one of
the strong points in the
Platform and of an
extremely sophisticated
level.
The technical level of the Platform and its
strong points such as the way real-time
data is used, the algorithms, data clean-up
possibilities of the Platform, presented data
and information is state-of-the-art and
impressive.
Snap4City (C), November 2019 97
Snap4City Services:
Consulting & Developing
Snap4City (C), November 2019 98
TOP
Providing consulting, customization,
training, and developments
• Snap4City solution can be installed on premise and one cloud, private
and public.
• Snap4City, Snap4 SRL, provide support, if needed, for design and/or
Develop:
– Training and tutoring;
– Smart city infrastructure and architecture;
– data analytics, that could be developed as proprietary solution for the
customer;
– data ingestion processes, to enable them to have data into the platform;
– adaptor for some specific protocol or legacy/third part Tool, that we prefer
to release as open source, but if the connection is with some proprietary tool,
the buyer could be interested to keep these solutions as private;
– IOT devices, full solutions, dashboards, specific dashboard widgets, etc.Snap4City (C), November 2019 99
Snap4Industry
• Snap4City technology can be exploited on Industy and IOT
solutions:
– Snap4Industry: Snap4City for Industry 4.0 (SLIDES)
– Scenario: 5G Enabled Water Cleaning Control
– Scenario: High Level Control of Industrial Plant
– Custom Synoptics and Widgets for Dashboards
– The integration between data and devices: the Snap4City solution
–
Snap4City (C), November 2019 100
Snap4Industry
Snap4City tech. for Industry 4.0
https://www.snap4city.org
Snap4City (C), November 2019 101
TOP
Snap4Industry as a Lambda Architecture
Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real
Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
Security Architecture at a glance
Data
Visualization
Virtual
Devices
Cloud
Processor
Data
Injection
Device
Management
Edge
Processor
Data
Sources
IoT Devices
(sensors, actuators)
Security and Privacy Management
Context
Brokers
IoT
Broker
IoT App
IoT Context
Brokers
IoT App
IoT Edge IoT
Directory
Registries
and storage
Dashboards
Snap4City (C), November 2019 103
Industry 4.0 Application
Snap4City (C), November 2019 104
Sensors/
Actuators
Sensors/
Actuators
Sensors/
Actuators
IOT Broker
IOT Gateway
[IOT Application]
MicroServices
Ext-Services
…….
Executing
permanent
computation
0,00
200,00
400,00
600,00
800,00
1 3 5 7 9 11 13 15 17 19 21 23 25
Serie2
Serie3
Serie4
Serie5
ETL Process
Executing local
computation
Internet
Dashboards
-- predictions
-- suggestions
-- Anomaly
detection
IOT Edge
Sensors/
Actuators
Sensors/
Actuators
Energy
Providers
Data
Analytics
IOT Broker
On the field
IOT Devices
GIDA set up
ModBus to
Snap4City
Gateway Edge
5G network
devices
Smart City
data from
many sources
Dashboards and Apps
IOT Applications
Big Data Analytics, Artificial Intelligence
IOT Data
Shadow
Snap4City
Telemonitoring
Telecontrol
Demo UC5 GIDA
https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MjIyNg==
Demo UC5 GIDA
https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MjIyNg==
Green Inpact project
Snap4City (C), November 2019 108
Green Impact
Capacity (GIC)
Altair Control room
Green Impact Capacity (GIC)
• Improve productivity of
chemical plant
• Keep GREEN the
environmental impact
• Exploiting innovative
technologies
• Diversify the production
• Monitoring
environmental conditions
Snap4City (C), November 2019 109
Installing Snap4City
Snap4City (C), November 2019 110
TOP
https://www.snap4city.org/471
To get an updated version read it!
Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart City API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
Snap4City as a Lambda Architecture
Overview of Snap4City platform, for Buyers, for all
Snap4City (C), November 2019 112
Snap4City MAIN
Dashboard, User
Management, etc.
IOT
brokers
HeatMaps and
Maps
Knowledge base,
semantic Store
Other
Services
And
Servers
Mobile Apps, Dashboards
Cluster for Smart
Processes container:
ingestion, analytics,
scraping, etc.
Cluster for big data
storage: static and
real time, Indexing
and search
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Legacy Platform
Back End Front End
Overview of Snap4City platform, for Buyers, for all
Snap4City (C), November 2019 113
Snap4City MAIN
Dashboard Builder,
Data Inspector, User Stats,
ExternalSrv, Res Manager,
MyKPI, MyPOI, Synoptic, IOT
Device Manager, Micro
Applications, Authorization/
Authentication, Roles, …
IOTBrokers, secure
IOTOBSFs
GISGeoServer
Heatmap Manager,
WFS, WMS
KBSSM
KB/ServiceMap, ..
SmartCity API, WFS,
ServiceMap3D, LOG/FLINT,
SuperServiceMap, Virtuoso
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
Mobile Apps, Dashboards
Container Cluster
MCLSCont, NCLSCont
Marathon, Mesos, Zookeeper,
Disces-EM, Executing in Container:
IOT App, Web Scraping, R-Studio,
Python, Java, ETL, ...
Data Shadow Cluster
IOTDSES
NIFI, Elastic Search, DevDash, Amma
[DataCluster]
ETL-VM
Phoenix HBASE, HDFS
DISCES, ETL support
IOTBrokers, secure
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Drupal support CRM, ..Legacy Platform
Back End Front End
What is included in the Buyers / Full Platform
• Snap4City MAIN
– Dashboard Builder, Wizard
– Dashboard Engine
– Data Inspector, Notificator
– External Srv manager
– Menu Manager
– Ownership Manager
– Authentication and
Authorisation
– WS server
– Resource Manager
– User Stats
– JavaScript Web App in a
Snap, MicroApplications
– Synoptics, custom Widgets
– IOT device manager: IOT
Directory
– Snap4City MicroServices on
IOT Applications: basic and
advanced
– Snap4City GIS Player
– What-IF tools
Snap4City (C), November 2019 114
• IOTOBSF
– IOT Orion Broker
Fi-Ware
– Secure Filter
(Snap4City)
• IOT Edge support
– Linux Ubuntu
– Windows
– Raspberry Pi
– Android
• IOT Devices support
– ESP32
– Arduino
• IOT Application
– Node-RED
– Snap4City Library
of nodes
• KBSSM
– Knowledge Base
Km4City
– ServiceMap
– ServiceMap3D
– SuperServiceMap
– LOG/Flint
– OSM2Km4City
– Smart City API
– WFS API
• GIS GeoServer
– GeoServer
– Heatmap
Manager
• Living Lab Portal
– CRM Drupal
– LDAP Snap4City
– Registration mng
– Etc. etc.
• MCLSCount, NCLSCount
Container[Cluster]
– Containers models
– DISCES-EM
– Monitoring App
• ETL Server
Data[Cluster]
– DISCES
– Phoenix Drivers
– Hbase Model
– ETL processes
• IOTDSES
DataShadow[Cluster]
– NIFI process, Squid
– Elastic Search Model
– Kibana
– AMMA data flow
– DevDash data store
• Other Services
– Routing Server
– DataGate CKAN
– Chat as Rocket
– ETL SDK VM
– Engager ..
– …..
• Data Analytics
– Heatmap
production MS
Rstudio
– OD production
– Predictions MS
RStudio
– Anomaly
detection MS
RStudio
– ETL Collection
– ….
All in source code, and most of components in Appliances/VMs
Platform Maintenance, K3.14
• Snap4City modules are released on GITHUB/DISIT and can be updated
from:
– GITHUB/DISIT into VM Appliance or Servers in which they are installed
• https://github.com/disit
– Node-RED tool, using official Library regarding: Snap4City Libraries
– Drupal for the Living-Lab Portal aspects
– Other tools from their corresponding providers
• Maintenance would not be a problem, all users should be capable to
perform the updates autonomously
• Updates on new versions will be provided by Snap4City periodically,
Notification will be provided on NEWS and GitHUB
Snap4City (C), November 2019 115
Keeping Platform at the State of The Art
• Updates will be provided by Snap4City periodically and released on
GITHUB, and other portals.
• Snap4City team is involved in contracts since now, so that this will
guarantee that the solution will be evolved to anticipate the state of
the art as we have done in Iteration 3, in which we released a number
of developments.
– See also the activity of dissemination and updated roadmap
– See the presence of Snap4City on EOSC, BeeSmartCity, EO15, etc.
– See the list of concreate developments
– See the list of supporters on the Stand with respect to those of the 2018
– See the planned new developments
Snap4City (C), November 2019 116
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Set-up of Open Source version from Scratch
• Guidelines reported on:
– https://www.snap4city.org/drupal/node/471
• How to proceed:
– We have presented 7 Configurations models from A:Alone to F:FullPlatform+LivingLab, but
they can be customized as you like.
– By follow the Configurations the Customers are guided to identify the most suitable
according to their needs;
– Once the most suitable Configuration has been identified, they are supported into the:
i. Download and deploy of the Appliances provided as Virtual Machines or Containers, or
ii. Download and install modules from GITHUB according to the recipes for VM/Container and the
user manual of the single tools, or
iii. Mixt of the above (i) and (ii) approaches.
– IOT Edge can created by (1) installing Node-RED, (2) adding Snap4City library of Nodes from
the Palette Manager of Node-RED, (3) registering on Snap4City:
https://nodered.org/docs/user-guide/editor/palette/manager
Snap4City (C), November 2019 117
Note that some of the material we are presenting has not been published on the portal yet, for the competition.
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City Scalable Platform with your business
Snap4City (C), November 2019 118
Configuratio
n Kind
Min#VM
Dockers
IOTBrokerint
IOTBrokerExt
MAP
KBSM
IOTAPP,
MicroServices
Storage&IOT
DataShadow
SSO,Roles
IOTSecurity
Dash+Wizard
GISin
GISout
Heatmap
Mob.App
ASCAPI
LivingLabSup.
Options
supported by
the
configuration
A: Alone 1 6 X X 1 mf X X X X (x) ETL, DG, CM
B: Basic 2 7 X X X 1 mf Small X X X X (x) ETL, DG, CM
C: CityStart 3-4 8-9 X X X X 1 mf Small X X X X X X X X ETL, DG, R, CM, Eng
D: DataCity 5-6 Ask X X X X 70 Mediu
m
X X X X X X X X ETL, DG, R, CM, Eng,
LL
E: ExtensiveCity 6-8+ Ask X X X X 70 Scalable X X X X X X X X ETL, DG, R, CM, Eng,
LL
F: FullPlatform 12-14+ Ask X X X X Scalab
le
Scalable X X X X X X X X ETL, DG, R, RS, CM,
Eng, LL
F: FullPlatform
+ LivingLab
13-15+ Ask X X X X Scalab
le
Scalable X X X X X X X X X ETL, DG, R, RS, CM,
Eng
mf: multiple flows for each IOT App
Read next slide for the other notes and legenda
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Legenda and Notes on the previous table
• In all configurations you:
– have: Multiple IOT Brokers; unlimited number of data sources;
interoperability support; KB
– may have: HA, DRS, FT, Balancing, cloning and configuring additional VM;
– may pass at the next configuration without reinstalling the VM;
– may add one or more Optional Services as VM/Containers: ETL, DataGate,
Routing, Engager, Data Analytics, etc. etc.
• From Config. E:ExtensiveCity and upper the number of VMs depends on the
volume of DataStorage and the volume of Data Stream in input.
– These two aspects are managed by two independent clusters of VMs and scale
independently each other. This allows to satisfy any different combination of volumes
in streams and data storage.
• We suggest using Living Lab support only on FullPlatform, while it can be
installed on Smaller Configurations with some limitations in terms of features
Snap4City (C), November 2019 119
Optional Services
• DG: DataGate
CKAN
• R: Routing
• ETL: ETL SDK
VM
• RS: R Studio
Server
• Eng: Engager
• CM: Chat
Manager
• LL: Living Lab
• ..
• ..
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Computational and Storage Costs
• The VM is considered (appliances are provided) as 16/24 cores 2.2 Ghz, 16-24 GB Ram,
500 GB HD in thin provisioning, with 25-40 GB HD used at the start, in most cases
Debian.
– to estimate the needed CPU, RAM, Storage for each configuration it is easy from the #of VM in the
configuration.
– Please note that configurations A and B can be executed on 4 cores, 4 GB Ram, ..
• In solutions, with a very large number of users on the Front End: Smart City API (mobile
App users, Dashboard users, a frontend balancer and more FrontEnd servers would be
needed). Please note that the Smart City API are also used by DataAnalytics processes
and by MicroServices in the IOT Applications.
– Typically a thousand of simultaneous users on the front end can be sustained for each VM
– More precise estimations can be performed by knowing the actual workload
Snap4City (C), November 2019 120
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
A Alone Configuration
Snap4City (C), November 2019 121
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
A: Alone) For Small Size Business, for example an industry 4.0 application
you can need to install only one VM: Snap4CityMAINVM
• For connecting data sources to Dashboards onpremise!
– Data driven solution, no large data store
– Industry 4.0
– Small Smart City business
– Start with Snap4City platform as seeding element
Snap4City (C), November 2019 122
• Snap4City MAIN VM
– Dashboard Builder, Wizard
– Dashboard Engine
– Data Inspector, Notificator
– External Srv manager
– Menu Manager
– Ownership Manager
– Authentication and
Authorisation
– WS secure server
– Resource Manager
– User Stats
– JavaScript Web App in a
Snap, MicroApplications
– Synoptics, custom Widgets
– IOT device manager: IOT
Directory
– Snap4City MicroServices on
IOT Applications: basic and
advanced
– Snap4City WFS player
– What-IF tools
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, …
IOT Applications, Synoptic,
Snap4City WFS GIS player,
IOT Device Manager….
External
Data Stores
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4CityMAIN: Alone Configuration
• As Virtual Machine:
– HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City
– https://www.snap4city.org/drupal/node/487
• As Container Composer:
– HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City
– https://github.com/disit/snap4city-docker/tree/master/Alone
Snap4City (C), November 2019 123
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B Basic Configuration
Snap4City (C), November 2019 124
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B: Basic) Addressing a relevant number of IOT devices/ IOT
brokers in your smart city or industry 4.0, a few smart
Applications with multiple flows
Snap4City (C), November 2019 125
For managing
beyond A:Alone
Configuration
+ large number
of IOT devices
and brokers,
any protocol,
format
+ IOT Devices
+ IOT Edge
+ Etc.
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator, WS Server,
Data Inspector,
User Stats,
ExternalServices,
Resource Manager,
MicroApplications
Authorization/Authentic.
MyKPI, MyPOI, …
IOT Applications,
Synoptic, Snap4City GIS
WFS player,
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTOBSFs
Sensors/
ActuatorsIOT Edge
External Data Stores
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
B: Basic) Addressing a relevant number of IOT
devices/ IOT brokers, a few smart Applications
• IOT Edge: local intelligence, processes and Applications located on the
field, on premise. Those autonomous smart tools can be implemented by
IOT Applications working on IOT Edge:
• Linux Ubuntu Appliance VM: https://www.snap4city.org/drupal/node/298
• Android as IOT Edge https://www.snap4city.org/drupal/node/278 and
• Raspberry Pi as IOT Edge: https://www.snap4city.org/drupal/node/77 You can install
Snap4City library in any Node-RED from the official Node-RED library
• IOT Applications = Node-RED + Snap4City libs of Nodes/MicroServices,
– Available in Snap4CityMAIN VM, so that in any installation
– any installation of Node-RED add Snap4City Library from the official Library of
Node-RED, add new Nodes into the Palette as
• https://nodered.org/docs/user-guide/editor/palette/manager
– Snap4City MicroService collection from the JS foundation with full
documentation inside https://flows.nodered.org/?term=snap4city
• BASIC: suitable for Final User
• ADVANCED: DEVELOPER (you have to install both libraries: basic and advanced)
Snap4City (C), November 2019 126
• IOTOBSF VM
– IOT Orion Broker
Fi-Ware
– Secure Filter
(Snap4City)
• IOT Edge support
– Linux Ubuntu
– Windows
– Raspberry
– Android
– Snap4City Library
of nodes
• IOT Devices support
– ESP32
– Arduino
• IOT Application
– Node-RED
– Snap4City Library
of nodes
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C) CityStart Configuration
Snap4City (C), November 2019 127
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C: CityStart) Addressing a relevant number of GeoLocated Points of Interest,
PIN on maps, and eventually heatmaps, a Small Smart City with a few smart
IOT applications on cloud and many on edge
Snap4City (C), November 2019 128
For managing
Beyond B:Basic
Configuration
+ Large number
of GEO
elements on
maps
+ Heatmaps
+ Geo shapes
+ GIS, ….
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
KBSSM
GIS, GeoServer
GIS, GeoServer
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTOBSFs
Sensors/
Actuators IOT Edge
External Data Stores
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
C: CityStart) Addressing a relevant number of GeoLocated
Points, heatmaps, ...
Snap4City (C), November 2019 129
• KBSSM VM
– Knowledge Base
Km4City
– ServiceMap
– ServiceMap3D
– SuperServiceMap
– LOG/Flint
– OSM2Km4City
– Smart City API
– WFS API
• GIS GeoServer VM
– GeoServer
– Heatmap
Manager
• KMSSM VM:
– knowledge base, KB, positioning
elements on maps, enabling geo-
spatial and temporal reasoning
– Ingestion OSM to load on KB,
ServiceMap and reasoning on it
– GeoReverse data, from street to
data and vice versa
– WFS export and import
• GIS GeoServer VM:
– Producing and providing heatmaps
– Managing Heatmaps colour maps.
– WMS, WFS export, distribution
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D) DataCity Configuration
Snap4City (C), November 2019 130
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D: DataCity) A medium/large size Smart City with a number of smart
applications on cloud and a number of IOT brokers,
relevant volume of data entering into the cloud.
Snap4City (C), November 2019 131
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOTDSES
KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
D: DataCity) A medium/large size Smart City with a number of smart
applications on cloud and a number of IOT brokers,
relevant volume of data entering into the cloud.
Snap4City (C), November 2019 132
• MCLSCount
Container[Cluster]
– Containers models
– DISCES-EM
– Monitoring App
• ETL Server
Data[Cluster]
– DISCES
– Phoenix Drivers
– Hbase Model
– ETL processes
• IOTDSES
DataShadow[Cluster]
– NIFI process
– Elastic Search Model
– AMMA and DevDash
For Managing
Beyond C:CityStart Configuration
+ A relevant number of IOT Applications
for data transformation and Analytics
+ A small sized Data Shadow for IOT
data management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E) ExtensiveCity Configuration
Snap4City (C), November 2019 133
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E: ExtensiveCity) A Large size Smart City with smart applications on
cloud and a number of IOT brokers,
large volume of data streams entering in the cloud
Snap4City (C), November 2019 134
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
Balancer
IOTDSES KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
E: ExtensiveCity) A Large size Smart City with smart applications on
cloud and a number of IOT brokers,
large volume of data streams entering in the cloud
Snap4City (C), November 2019 135
• IOTDSES
DataShadow Cluster
– NIFI process
– Squid cache on
SCAPI
– Elastic Search Model
extended API
– Kibana
– AMMA data flow
– DevDash data
monitor
– …
For Managing
Beyond D:DataCity Configuration
+ A relevant number of data streams
entering in the platform
+ A scalable Data Shadow for IOT data
management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F) FullPlatform Configuration
Snap4City (C), November 2019 136
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F: FullPlatform) For Huge scale Smart City with a Huge number of smart
applications and processes on cloud (thousands), any number of IOT brokers
Snap4City (C), November 2019 137
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
[Phoenix ]
[HDFS, Hbase]
Pool of Containers
NodeRed,
Web Scraping
[Penthao Kettle, Java, Karma]
……..
R-Stutdio
Python, C++, C
…………..
Nodes/VM on Cloud in HA, elastic computing, ..
.
.
.
……
……..
……….
.
Masters in HA: Marathon/Mesos, Zookeeper, ….
Master Master Master
containersDistribSrv.
MCLSCont
Mesos Balancing
Cnt. Scheduling
DISCES-EM: Cloud
Elastic Management
NCLSCont
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets
Notificator
Data Inspector
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI
IOT Device Managner
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
SCAPIBalanc.
SmartCityAPI
KB/ServiceMap
ServiceMap3D
LOG/FLINT
Virtuoso
DeveloperBalancer
Snap4CityMAIN
Dashboard Builder
Wizard, Widgets, IOT App1
Notificator, WS Server,
Data Inspector, User Stats,
ExternalSrv, Resource
Manager, MicroApps
Authorization/Authentic.
MyKPI, MyPOI, Synoptic
IOT Device Manager
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
IOT Broker
IOT Context Broker
ProxyFilter Security
VM on Cloud HA, DRS, [FT]
Balancer
IOTDSES KBSSM
IOTOBSFs
GIS, GeoServer
GIS, GeoServer
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F: FullPlatform) For Huge scale Smart City with a Huge number of smart
applications and processes on cloud (thousands), any number of IOT brokers
Snap4City (C), November 2019 138
• MCLSCount
• NCLSCount
Container[Cluster]
– Marathon, Mesos
Cluster
– Containers models
• IOT App
• Web Scraping
• Data Analytics
• ETL
• Python
• ..
– DISCES-EM
• Elastic management
of containers
– Monitoring App
For Managing
Beyond E:ExtendedCity Configuration
+ A scalable huge number of Containers
including different kind of processes:
IOT, ETL, data analytics, scraping
+ A scalable Data Shadow for IOT data
management
Optionally:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• Routing Solutions
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F+) LivingLab Configuration
Snap4City (C), November 2019 139
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F-LivingLab: FullPlatform + LivingLab) For Huge scale
Smart City as Full Platform plus Living lab Support
Snap4City (C), November 2019 140
Snap4City MAIN
Dashboard Builder,
Data Inspector, User Stats,
ExternalSrv, Res Manager,
MyKPI, MyPOI, Synoptic, IOT
Device Manager, Micro
Applications, Authorization/
Authentication, Roles
IOTBrokers, secure
IOTOBSFs
GISGeoServer
Heatmap Manager,
WFS, WMS
KBSSM
KB/ServiceMap,
SmartCity API, WFS,
ServiceMap3D, LOG/FLINT,
SuperServiceMap, Virtuoso
Services
CKAN DataGate
Routing Server
Chat Manager
R Studio Server
ETLSDK Server
Engager Server
……. Server
Mobile Apps, Dashboards
Container Cluster
MCLSCont, NCLSCont
Marathon, Mesos, Zookeeper,
Disces-EM, Executing in Container:
IOT App, Web Scraping, R-Studio,
Python, Java, ETL, ...
Data Shadow Cluster
IOTDSES
NIFI, Elastic Search, DevDash, Amma
[DataCluster]
ETL-VM
Phoenix HBASE, HDFS
DISCES, ETL support
IOTBrokers, secure
GIS
IOT
Networks
Social
Media
CRM …..
Living Lab Portal
Drupal support CRMLegacy Platforms
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as
Full Platform plus Living lab Support
Snap4City (C), November 2019 141
• Living Lab Portal
– CRM Drupal
– LDAP Snap4City
– Blogs, Articles,
comments, etc.
– Multiple Organization
– Multiple Groups
– SSO with Snap4City
tools
– Automated
Registration
management
– SOLR indexing on
content
– Monitoring activity
– Reporting and
statistics
For Managing
Beyond F:FullPlatform Configuration
+ Providing Living Lab Support, a full
CRM for stakeholder, co-working,
collaborative work, discussion
environment, chart, forum, etc.
Strongly suggested:
• Data Analytics in R Studio, Tensor
Flow (NVIDIA)
• Hbase / Phoenix Big Data Store
• Data Gate CKAN for Open Data
• Chat Manager on Dashboard
• ETL SDK servers
• User Engagement Server
• Routing Solutions
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
THE VIEW OF
THE ADMINISTRATOR
Snap4City (C), November 2019 142
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Knowledge base
Semantic reasoners
Data
Sources
PULL
Data
Data
Sources,
Brokers,
External
Services
Data
Driven,
Real Time
Data Ingestion,
aggregation,
regularization,
reconcile
ETL, NIFI,
DISCES
Big Data Cluster
HDFS, Hbase, Phoenix Search and
Query,
Smart City API
Facet, semantic
searchIndexing and aggregating
Elastic search
Data Analytics
R, Tensor Flow, Python, MapReduce, …
Visual
Analytics
Special
dash/
tools
Rendering
Acting,
Widgets,
MicroApps
User
interface,
Interactive
Dashboard,
Drill down,
maps,
heatmaps
Inform, announce, Act!, warning, alarms, What-IF, ..
IOT Applications
Node-RED + Snap4City MicroSrvs
Snap4City as a Lambda Architecture
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Management by Organization
• Organizations may have
– name, ID, GPS center, a number of Groups on
Snap4City.org
– users of different kinds and may impose early bounds on
the resourced used by users (IOT Dev, IOT App, Dash)
– on cloud users kinds up to level of Tool Administrator
• ToolAdmin users (requested by Organizations) may
– control processes, consumption of resources, healthiness,
etc.
– manage tools exploited in your configuration
• 24H/7D Help Desk and Assistance
Snap4City (C), November 2019 144
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Help Desk and SLA
• https://www.snap4city.org/drupal/contact
• Bug Reporting
– https://docs.google.com/forms/d/e/1FAIpQLSfD
QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5
jsoXw/viewform
• For Service Level Agreement see:
– Service Level Agreement
• Help Desk and Contact:
– https://www.snap4city.org/drupal/node/3
• Availability rates:
– https://www.snap4city.org/388
Snap4City (C), November 2019 145
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Monitoring Resource
Consumption and Traffic
Snap4City (C), November 2019 149
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 150
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Monitoring Infrastructure
Status
Snap4City (C), November 2019 151
http://www.disit.org/dashboardSmartCity/view/index.php?i
ddasboard=MTQ4
http://www.disit.org/dashboardSmartCity/view/index.ph
p?iddasboard=MTkw
http://www.disit.org/dashboardSmartCity
/view/index.php?iddasboard=MjQ5
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 152
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Computational Capabilities of Snap4City
• Managing:
– Periodic Processes → ETL, IOT App (Node-RED)
– Asynchronous processes, data driven, real time → Node-RED, IOT
Applications
• Scalability
– Horizontal: Increasing processes performing activities, demand on new
processes for new users, for new applications, for new IOT applications:
VM, Hosts, clusters, Storage SAN
– Vertical: Increasing resources on processes: CPU, MEM, Storage, Network
Snap4City (C), November 2019 153
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DevDash Case Study (2)
• Detect potential anomalies or disfunctions by
inspecting the DevDash tool time trend
Snap4City (C), November 2019 154
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Managing and Monitoring
Data-Traffic in the BackOffice
Snap4City (C), November 2019 155
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA (1)
Monitor data traffic flows among IoT devices,
services, applications etc. and detect potential
anomalies
Snap4City (C), November 2019 156
Unexpected behaviors can
be revealed by inspecting
the data flow time trend:
a) detecting peaks or
valleys in the trend
b) drill-down on data to
identify single/more
malfunctioning devices
and/or services
quantitatively monitoring
data/message traffic and flows
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA (2)
Make drill-down activities on data related to a
single Process ID and check for unexpected
behavior in the Time Trend panel:
c) Filtering data by the Process ID (e.g. for
example those related to a SmartWaste
container)
d) Detect a peak with more data traffic than
expected during its scheduled activity, by
properly filtering on time, the single data
portion corresponding to the unexpected
data flow can be viewed
e) Locate on map the single involved device
or service
Snap4City (C), November 2019 157
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
AMMA by Elastic Search + Kibana
Snap4City (C), November 2019 158
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
DevDash by Elastic Search + Kibana
Snap4City (C), November 2019 159
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Auditing Activities
Snap4City (C), November 2019 160
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 161
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Managing Back Office
Processes via Containers
Snap4City (C), November 2019 162
TOP
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Hosts with VM on Cloud HA, DRS, FT
Snap4City (C), November 2019
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Event Log
Zeppelin
Phoenix
HDFS, Hbase
debian
Hosts/VM on Cloud in HA, elastic computing, ..
.
.
.
……
……..
……….
.
containersDistribSrv.
slave slave slave slave
Pool of Containers
NodeRed, Java
Penthao Kettle, Java, Karma
……..
R-Stat, R-Parallel
Hadoop/Spark/SparkML
Python, C++, C, java
…………..
Master Master Master
Masters in HA: Marathon/Mesos, Zookeeper, ….
VmWare VSphere
Cnt. Scheduling
Cloud Elastic
Management
Cnt. Scheduling
Cloud Elastic
Management
DISCES
+ Planner +
Elastic Management
Containers
Control API
Cloud Management
System &
Resource Monitoring
Cloud Management
System &
Resource Monitoring
Resource Monitoring
&
Cloud Resource
Management
Monitoring API
VM Control API
163
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Container
Snap4City (C), November 2019 164
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Elastic Scaling: allocating / deallocating
• Allocation/ deallocation, Rebalancing vs compacting
– Vertical of resources: Docker and/or VM: CPU, Mem
• NodeJS multi-flow for each Docker, the user request data flows and IOT App, Snap4City
allocates them dynamically on demand and perform workload optimization
• VM: management of Mem, CPU; transparent and automatic in DRS VMware
– Horizontal of resources of Dockers and/or VM and/or [Host]:
• Docker: addition of containers, migrations/moving, balancing (per moving) of IOT App
• VM: on/off
• Monitoring resources:
– VM via VMware API, Docker via Marathon and Mesos APIs
• Algorithm in Python for scaling, actions via APIs: VMware, Marathon,..
Snap4City (C), November 2019 165
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 166
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Snap4City (C), November 2019 167
Snap4City (C), November 2019 168
Monitoring on Cloud
DISIT Lab, Distributed Data Intelligence and Technologies
Distributed Systems and Internet Technologies
Department of Information Engineering (DINFO)
http://www.disit.dinfo.unifi.it
http://www.disit.org
Distributed SCE Scheduler
Snap4City (C), November 2019 169
Acknowledgements
Snap4City (C), November 2019 170
Snap4City managed to
provide a maximum of
information, flows, in depth
analysis with the data
provided.
There is no
other platform
that collects
all city actors
together.
The City officials and ICT
officials were impressed
with the performance of
the Platform when loading
the heavy, “resource-”
demanding applications and
dashboard.
The data handling
throughout the Platform
is considered as one of
the strong points in the
Platform and of an
extremely sophisticated
level.
The technical level of the Platform and its
strong points such as the way real-time
data is used, the algorithms, data clean-up
possibilities of the Platform, presented data
and information is state-of-the-art and
impressive.
Snap4City (C), November 2019 171
Snap4City (C), November 2019 172
Acknowledgements
• Thanks to the European Commission for founding. All slides reporting logo of Snap4City https://www.snap4city.org of
Select4Cities H2020 are representing tools and research founded by European Commission for the Select4Cities project.
Select4Cities has received funding from the European Research Council (ERC) under the European Union's Horizon 2020
research and innovation Programme (grant agreement n° 688196)
• TRAFAIR is a CEF project. All slides reporting logo of TRAFAIR project are representing tools and research founded by the EC on
CEF programme http://trafair.eu/
• Thanks to the European Commission for founding. All slides reporting logo of REPLICATE H2020 are representing tools and
research founded by European Commission for the REPLICATE project. REPLICATE has received funding from the European
Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n°
691735).
• Thanks to the European Commission for founding. All slides reporting logo of RESOLUTE H2020 are representing tools and
research founded by European Commission for the RESOLUTE project. RESOLUTE has received funding from the European
Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n°
653460).
• Thanks to the MIUR for co-founding and to the University of Florence and companies involved. All slides reporting logo of Sii-
Mobility are representing tools and research founded by MIUR for the Sii-Mobility SCN MIUR project.
• Km4City is an open technology and research line of DISIT Lab exploited by a number of projects. Some of the innovative
solutions and research issues developed into projects are also compliant and contributing to the Km4City approach and thus are
released as open sources and are interoperable, scalable, modular, standard compliant, etc.
INEA CEF-TELECOM Project
funded by European Union
Roadmap
- Twitter Vigilance
- Social Media
Analytics, Sentiment
Analysis
- User engagement
- Bike Sharing
- Data Analytics ++
- Social Predictions
- OBD2
- Sardinia Region
Smart City
Strategies and plan
GHOST SIR
(2016-19)
- Smart Energy
- Sustainable Mobility
- Control Room
- Dashboard
- Resilience
Decision Support
- Smart First Aid
- User Behaviour
Analysis, predictions
- Risk Analysis
- Origin-Destination
and trajectories
- Traffic Reconstruction
- Offer Analysis
- OBU, smart devices
- Weather
Forecast
- Real Time Wi-Fi
- Entertainment
- Events
- LOD
- Tuscany
Road Graph
- Mobility
- culture, tourism
- Events
- Parking
- Services
- Linked open graph
2013
2014
2017
2018
2015
2016
Km4City 1.4
- Infomobility
- Mobile App
- Routing
- Multimodality
Km4City
Ontology 1.1 Km4City 1.6.2
- IOT/IOE, IOT App
- Living Lab
- Maker Support
- IOT Edge
- Smart City IOT
- GDPR
- Privacy & Security
CEF
DISIT lab roadmap vs model and tools’ usage
Km4City 1.6.6
5G tech
Energy
Industry 4.0
Synoptics
…2022
IOT/IOE
SII-MOBILITY SCN
(2016-19)
- Smart Waste
(2017-20)
(2015-18)
H2020
(2016-21)
H2020
(2018-21)
- Mobility
Demand / Offer
Analytics and
Strategy
(2018-20)
Km4City 1.6.4
H2020
(2017-19)
GREEN IMPACT
- Industry 4.0
- Critical Plant
- Monitoring
POR FESR 2014-2020
Km4City 1.5
2019
173
- Traffic and Mobility
Impact on Pollution
- NOX predictions(2018-21)
2020
Market Solutions
Snap4City (C), November 2019 174
IOTDiscoveryAbstraction
Authentication,Authorization
Securityend-2-end,secureon
IOTandDashboards
OpenHWandOpenSW
IntegratedCommunity
management
DataTypes:IOTDevices,IOT
App,Dashboard,Data
DataType:Publish/share,
Delegation,Consentandchange
DataType:Downloadand
Delete
AuditingonDataTypeAccess
OpenSourceend-to-end
ScalabilityIOT
VisualProgrammingend-to-end
applications
AdvancedSmartCityAPI,
MicroServices
MultiDomainSemantic
Platform
StandardbasedModulesand
IOT,OpenDevices
ResourceSharing
DataAnalyticsintegrated
DashboardH24/7,protected
connection
Multi-protocolonIOT
G G G G G
Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y
Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http
IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y
IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT
FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y
ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited
Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP
AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited
Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited
PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y
Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y
CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N
Siemens MindSphere
[60]
Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y
Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT
Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP
Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited
Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
Requirements vs Modules
Snap4City (C), November 2019 175
Requirement.description SecurityandPrivacyManagement
Ownership&Delegation
PersonalData
IoIContextbrokers
IoTDirectory
IoTFirewall
IoTApps,andmanagement
DashboardBui8lderand
management
MicroSerivces
SmartCityAPI
IoTDevices
IoTEdge,IoTApponpremise
DataShadowandStorage
KnowledgeBaseKm4CIty
Snap4CityPlatformSupportLiving
Lab
IV.A
IV.D
IV.C VI IV.C
VI.A
IV.C IV.C IV.A V.B IV.C
V.C
IV.B IV.C V.A
V.C
IV.B
V.B
IV.B
IV.C
IV.B IV.A
IV.B
R1. IOT Brokers X X X
R2. IOT Discovery Abstraction X X X
R3. Authentication, Authorization X X X X
R4. Inform User of Security Level X
R5. Developing Secure Applications X X X X X X
R6. Secure Communications X X
R7. Open HW and Open SW X X X X X
R8. Signed Content Vs Data Types X X X
R9. Managing IoT Data Types X X X X X X X X X X X X X X
R10. Managing Ownership and Delegation X X X X
R11. Support Roles, Org. and Groups X X X X X X X
R12. Encrypted Personal Data X X X
R13. User Profile Management, forgotten X X
R14. Auditing Data Types X X X X X X X X X X X X X X X
R15. Data Breach Detection X X X X X X X X X X X X X X X
R16. Accounting X X X X X X X X X X X X X X X
R17. Data Protection by Design X X X X X X X X X X X X X X X
R18. Evidence of Level of Security X X X
GDPR vs Requirements
GDPR Compliance Verification Feature Verif. Reqs.
Signed consent UI R8
User profile management and control UI R13
Data Type private as default UI R8
Rights to access per element UI R9
Rights to transfer per element UI R10
Rights to erase per element and total UI R13
Rights to revoke/change per Data Type UI R10
An interface for Right management for Data Type UI R9
Clear Terms of Use and Privacy Policy UI --
Auditing Tools for Data Type UI R14
Publish as Anonymous UI R9
Encrypt personal users’ data Code R12
Secure Authentication and Authorization Code R3
Data protection by Design Code R17
Secure connection Code R6
Security Control, data breach control, anonymization, etc. PEN
Test
R15,
R16, R18
Snap4City (C), November 2019 176
Snap4City (C), November 2019 177
Be smart in a SNAP!
TOP

More Related Content

What's hot

FIWARE Agenda for Smart City Expo - Nov 18th-20th
FIWARE Agenda for Smart City Expo  - Nov 18th-20th FIWARE Agenda for Smart City Expo  - Nov 18th-20th
FIWARE Agenda for Smart City Expo - Nov 18th-20th
FIWARE
 
FIWARE Global Summit - AUDP – the Atos Urban Data Platform
FIWARE Global Summit - AUDP – the Atos Urban Data Platform FIWARE Global Summit - AUDP – the Atos Urban Data Platform
FIWARE Global Summit - AUDP – the Atos Urban Data Platform
FIWARE
 
FIWARE Global Summit - Next Steps
FIWARE Global Summit - Next StepsFIWARE Global Summit - Next Steps
FIWARE Global Summit - Next Steps
FIWARE
 
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
Paolo Nesi
 
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
FIWARE
 
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, SerbiasymbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
symbiote-h2020
 
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
FIWARE
 
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
symbiote-h2020
 
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
symbiote-h2020
 
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
symbiote-h2020
 
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWAREFIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
FIWARE
 
FIWARE Global Summit - Smart Cities Showcases Program
FIWARE Global Summit - Smart Cities Showcases ProgramFIWARE Global Summit - Smart Cities Showcases Program
FIWARE Global Summit - Smart Cities Showcases Program
FIWARE
 
FIWARE & TM Forum partnership (Infographic)
FIWARE & TM Forum partnership (Infographic)FIWARE & TM Forum partnership (Infographic)
FIWARE & TM Forum partnership (Infographic)
FIWARE
 
Fiware IoT_intro&scenarios
Fiware IoT_intro&scenariosFiware IoT_intro&scenarios
Fiware IoT_intro&scenarios
FIWARE
 
Borys Pratciuk Augmented reality romania
Borys Pratciuk Augmented reality romaniaBorys Pratciuk Augmented reality romania
Borys Pratciuk Augmented reality romania
Michael Pustovit
 
Live with IOT (Borys Pratsiuk Technology Stream)
Live with IOT (Borys Pratsiuk Technology Stream) Live with IOT (Borys Pratsiuk Technology Stream)
Live with IOT (Borys Pratsiuk Technology Stream)
IT Arena
 
Palestra IoT - Internet of Things - Internet das Coisas
Palestra IoT - Internet of Things - Internet das CoisasPalestra IoT - Internet of Things - Internet das Coisas
Palestra IoT - Internet of Things - Internet das Coisas
Luiz Oliveira
 
FIWARE Global Summit - MD4PROD
FIWARE Global Summit - MD4PRODFIWARE Global Summit - MD4PROD
FIWARE Global Summit - MD4PROD
FIWARE
 

What's hot (18)

FIWARE Agenda for Smart City Expo - Nov 18th-20th
FIWARE Agenda for Smart City Expo  - Nov 18th-20th FIWARE Agenda for Smart City Expo  - Nov 18th-20th
FIWARE Agenda for Smart City Expo - Nov 18th-20th
 
FIWARE Global Summit - AUDP – the Atos Urban Data Platform
FIWARE Global Summit - AUDP – the Atos Urban Data Platform FIWARE Global Summit - AUDP – the Atos Urban Data Platform
FIWARE Global Summit - AUDP – the Atos Urban Data Platform
 
FIWARE Global Summit - Next Steps
FIWARE Global Summit - Next StepsFIWARE Global Summit - Next Steps
FIWARE Global Summit - Next Steps
 
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
Km4City: A reusable example of a Metropolitan-Wide Data Platform, MAJORCITIES...
 
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
FIWARE Global Summit - Moving Towards a Data Economy Business Model: A Real E...
 
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, SerbiasymbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
symbIoTe - IoT Week Belgrade, 02 Jun 2016, Belgrade, Serbia
 
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
FIWARE Global Summit - Smart Data-Contextualization towards NGSI Data Economy...
 
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
Internet of Things Interoperability Challenges - SpliTech, 14 Jul 2016, Split...
 
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
symbIoTe - AIOTI Open Day @ NDC, 08 Feb 2016, Athens, Greece
 
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
Towards the Cross-Domain Interoperability of IoT Platforms - EuCNC, 30 Jun 20...
 
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWAREFIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
FIWARE Wednesday Webinars - Architecting Your Smart Solution Using FIWARE
 
FIWARE Global Summit - Smart Cities Showcases Program
FIWARE Global Summit - Smart Cities Showcases ProgramFIWARE Global Summit - Smart Cities Showcases Program
FIWARE Global Summit - Smart Cities Showcases Program
 
FIWARE & TM Forum partnership (Infographic)
FIWARE & TM Forum partnership (Infographic)FIWARE & TM Forum partnership (Infographic)
FIWARE & TM Forum partnership (Infographic)
 
Fiware IoT_intro&scenarios
Fiware IoT_intro&scenariosFiware IoT_intro&scenarios
Fiware IoT_intro&scenarios
 
Borys Pratciuk Augmented reality romania
Borys Pratciuk Augmented reality romaniaBorys Pratciuk Augmented reality romania
Borys Pratciuk Augmented reality romania
 
Live with IOT (Borys Pratsiuk Technology Stream)
Live with IOT (Borys Pratsiuk Technology Stream) Live with IOT (Borys Pratsiuk Technology Stream)
Live with IOT (Borys Pratsiuk Technology Stream)
 
Palestra IoT - Internet of Things - Internet das Coisas
Palestra IoT - Internet of Things - Internet das CoisasPalestra IoT - Internet of Things - Internet das Coisas
Palestra IoT - Internet of Things - Internet das Coisas
 
FIWARE Global Summit - MD4PROD
FIWARE Global Summit - MD4PRODFIWARE Global Summit - MD4PROD
FIWARE Global Summit - MD4PROD
 

Similar to Snap4City November 2019 Course: Smart City IOT platform installation, deploy, development life cycle

Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
Paolo Nesi
 
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
Paolo Nesi
 
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Paolo Nesi
 
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
Paolo Nesi
 
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Paolo Nesi
 
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
Paolo Nesi
 
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Paolo Nesi
 
Snap4City a Solution for highly collaborative Smart Cities Environments
Snap4City a Solution for highly collaborative Smart Cities Environments Snap4City a Solution for highly collaborative Smart Cities Environments
Snap4City a Solution for highly collaborative Smart Cities Environments
Paolo Nesi
 
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services""Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
diannepatricia
 
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
Paolo Nesi
 
20181012 fiware at_construction_conference
20181012 fiware at_construction_conference20181012 fiware at_construction_conference
20181012 fiware at_construction_conference
stefano de panfilis
 
Webinar on 2nd Open Call - Applications and Trials - slideset
Webinar on 2nd Open Call - Applications and Trials - slidesetWebinar on 2nd Open Call - Applications and Trials - slideset
Webinar on 2nd Open Call - Applications and Trials - slideset
symbiote-h2020
 
Km4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of ServicesKm4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of Services
Paolo Nesi
 
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIESSnap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
Paolo Nesi
 
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
Paolo Nesi
 
IoT and 4IR
IoT and 4IRIoT and 4IR
IoT and 4IR
Dr. Mazlan Abbas
 
Mobile and Social CPS including Augmented Reality
Mobile and Social CPS including Augmented RealityMobile and Social CPS including Augmented Reality
Mobile and Social CPS including Augmented Reality
Bob Marcus
 
A Smart City Development kit for designing Web and Mobile Apps
A Smart City Development kit for designing  Web and Mobile AppsA Smart City Development kit for designing  Web and Mobile Apps
A Smart City Development kit for designing Web and Mobile Apps
Paolo Nesi
 
hackathon smart city API, dai dati ai servizi
hackathon smart city API, dai dati ai servizihackathon smart city API, dai dati ai servizi
hackathon smart city API, dai dati ai servizi
Paolo Nesi
 
Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View
Paolo Nesi
 

Similar to Snap4City November 2019 Course: Smart City IOT platform installation, deploy, development life cycle (20)

Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
Snap4City November 2019 Course: Smart City IOT Dashboard, smart city control ...
 
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
Snap4City November 2019 Course: Smart City API, federation, Web and Mobile Ap...
 
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
Snap4City November 2019 Course: Smart City IOT Geernal overview, from dashboa...
 
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
scalable Smart aNalytic APplication builder for sentient Cities Overview -- S...
 
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
Snap4City: Smart City IOT/IOE Platform scalable Smart aNalytic APplication bu...
 
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
IOT/IOE Elastically Scalable Architecture for Smart City and Industry 4.0
 
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
Snap4City November 2019 Course: Smart City IOT Data Ingestion Interoperabilit...
 
Snap4City a Solution for highly collaborative Smart Cities Environments
Snap4City a Solution for highly collaborative Smart Cities Environments Snap4City a Solution for highly collaborative Smart Cities Environments
Snap4City a Solution for highly collaborative Smart Cities Environments
 
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services""Km4City: Smart City Ontology Building for Effective Erogation of Services"
"Km4City: Smart City Ontology Building for Effective Erogation of Services"
 
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...Complexity of IOT/IOE Architectures for  Smart Service Infrastructures Panel:...
Complexity of IOT/IOE Architectures for Smart Service Infrastructures Panel:...
 
20181012 fiware at_construction_conference
20181012 fiware at_construction_conference20181012 fiware at_construction_conference
20181012 fiware at_construction_conference
 
Webinar on 2nd Open Call - Applications and Trials - slideset
Webinar on 2nd Open Call - Applications and Trials - slidesetWebinar on 2nd Open Call - Applications and Trials - slideset
Webinar on 2nd Open Call - Applications and Trials - slideset
 
Km4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of ServicesKm4City: Smart City Ontology Building for Effective Erogation of Services
Km4City: Smart City Ontology Building for Effective Erogation of Services
 
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIESSnap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
Snap4City: SCALABLE SMART ANALYTIC APPLICATION BUILDER FOR SENTIENT CITIES
 
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
Km4City, Smart City Urban Platform, From Data to Services for the Sentient Ci...
 
IoT and 4IR
IoT and 4IRIoT and 4IR
IoT and 4IR
 
Mobile and Social CPS including Augmented Reality
Mobile and Social CPS including Augmented RealityMobile and Social CPS including Augmented Reality
Mobile and Social CPS including Augmented Reality
 
A Smart City Development kit for designing Web and Mobile Apps
A Smart City Development kit for designing  Web and Mobile AppsA Smart City Development kit for designing  Web and Mobile Apps
A Smart City Development kit for designing Web and Mobile Apps
 
hackathon smart city API, dai dati ai servizi
hackathon smart city API, dai dati ai servizihackathon smart city API, dai dati ai servizi
hackathon smart city API, dai dati ai servizi
 
Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View Open Urban Platform for Smart City: Technical View
Open Urban Platform for Smart City: Technical View
 

Recently uploaded

GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
CatarinaPereira64715
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
Abida Shariff
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Paul Groth
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 

Recently uploaded (20)

GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptxIOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMsTo Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 

Snap4City November 2019 Course: Smart City IOT platform installation, deploy, development life cycle

  • 1. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it 1 LIVING LAB Be smart in a SNAP! 6th Day, System & Deploy, Nov 2019 https://www.snap4city.org/501
  • 2. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it Snap4City (C), November 2019 Paolo Nesi, paolo.nesi@unifi.it https://www.Km4City.org https://www.disit.org https://www.Snap4City.org scalable Smart aNalytic APplication builder for sentient Cities: for Living Lab and co-working with Stakeholders Powered by 2 6th Day, System & Deploy, Nov 2019 https://www.snap4city.org/501
  • 3. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org General Overview of the full Course • 1st Day: General overview (1 day, 6 hours) • 2nd Day: Dashboards, how to build and manage them (4 hours) • 3rd Day: IOT Applications development, IOT Devices, IOT Networks (4 hours) • 4th Day: Data Analytics, in R Studio, In Python, how to integrate with IOT Applications (4 hours) • 5th Day: Data Ingestion, Data Warehouse, ETL Development, Data Gate, IOT Device Data ingestion, etc.. (5 hours) • 6th Day: Snap4City Architecture, How To Install Snap4City (3 hours) • 7th Day: Smart city API (internal and external) Web and Mobile App development tool kit (4 hours) A number of the training sections include exercitations Updated versions on: https://www.snap4city.org/501 Snap4City (C), November 2019 3
  • 4. 6th day Agenda • Snap4City Architecture • Snap4City: Smart City IOT as a Service • Snap4City Living Lab For Collaborative Work – Smart City Development Life Cycle – Analysis and Design for Innovation (Co-Creation and Co-Working) – Development Tools – How to Add Functions that are not present in the Platform • Snap4City vs Fi-Ware • Snap4City vs State of the Art Solutions • Snap4City Services: Consulting and Developing • Snap4City vs Snap4Industry 4.0 • Installing Snap4City • The view of the Administrator – Monitoring Resource Consumption and Traffic – Managing and Monitoring Data Traffic in the BackOffice – Auditing Activities – Managing Back Office processes via Containers • Acknowledgement Snap4City (C), November 2019 4 GO GO GO GO GO GO GO GO GO GO GO GO GO GO
  • 5. Levels of Difficulty • Easy. • Moderate. • Good. • Golden. • Professional. • Excellent. Snap4City (C), November 2019 5 non programmer level Some JavaScript rudiment coding JavaScript programming Programming in R Studio Exploiting Smart City API Developing Full IOT Applications, Dashboard and Mobile Apps
  • 7. 7
  • 8. Standards and Interoperability Snap4City (C), November 2019 8 Compliant with: AMQP, COAP, MQTT, OneM2M, HTTP, HTTPS, TLS, Rest Call, SMTP, TCP, UDP, NGSI, LoraWan, TheThingsNetwork, SigFOX, DATEX II, SOAP, WSDL, Twitter, FaceBook, Telegram, SMS, OLAP, MySQL, Mongo, HBASE, SOLR, SPARQL, EMAIL, FTP, FTPS, WebSocket, WebSocket Secure, ModBUS, OPC, GML, RS485, WFS, WMS, ODBC, JDBC, Elastic Search, Phoenix, XML, JSON, CSV, db, GeoJSON, Enfuser FMI, Android, Raspberry Pi, Local File System, ESP32, Libelium, IBIMET, OBD2, SVG, XLS, XLSX, TXT, HTML, CSS, etc.
  • 9. Project and partners, a few Snap4City (C), November 2019 9
  • 10. Unique of Snap4City Platform (1) • Data ingestion and model – Unified data model (exploited in the Wizard and Knowledge base) – Semantic Reasoner modelling city entities, supporting sematic search, expert system, digital Twin, etc. – IOT Directory abstracting complexity of IOT Devices, Edge, Brokers, protocols and data formats • Data Analytics and Data Processes – Flexible and extensible IOT Applications – Data Analytic: multiple programming languages, • Visual Analytics, dashboarding, Apps – Wizard: expert system for immediate dashboard production matching data vs graphics representation – Dashboards specialized multidomain for Smart Cities – Custom Widgets and Synoptics – Ready to use Mobile App, instant App, MicroApplication – Strategies formalization supports Snap4City (C), November 2019 10
  • 11. Unique of Snap4City Platform (2) • Openness to any developers – Living Lab support for coworking, sharing, and delegating – Advanced Smart City APIs and MicroServices – 100% Open Source, Open hardware • Security and Privacy – End-2-end encrypted communication, on devices, platform, ... dashboards – GDPR compliant privacy/security • Non functional – on cloud and on premise, your private installation – Ready to use Appliance Virtual Machines and/or Containers for a modules and tools. – Flexible, Modular, Elastic, scalable and robust Snap4City (C), November 2019 11
  • 13. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org How to adopt Snap4City Snap4City (C), November 2019 13 Https://www.snap4city.org Smart City as a Service • Supporting Org • 100% Open Source Platform: Github • Further developments • Publishing Appliances and Dockers • Training courses, docs • Consulting • Forums • Etc. Download and deploy Installation on your premise • Virtual Machines or Dockers • Different configurations • From small to scalable • Exploiting your legacy tools • Interoperable with any tool • No vendor lock-in, No tech lock-in Mixed solutions! For example: • Start on Cloud as Smart City as a Service • Migrate on premise on the fly • Start on Cloud into a sand box • Pass to install on premise what you need On your premise
  • 14. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Connect IOT/IOE Upload context Open Data Connect external Services Advanced Smart City API, MicroServices Manage Apps & Dashboards, User Engagement Monitor City Platform experiments workshops tutorials networking agreements events Start-ups Research groups City Users Large Industries collaborations Licensing, Gold services personal services Case Studies Inhouse companies Resource Operators Tech providers partnerships documentation Help desk Category Associations Corporations Advertisers Community Building subscription to applications Produce City IOT Applications & Dashboards Promote Applications & Dashboards Produce Apps and Dashboards for City Users Data Ingestion and Analytic algorithms Early Adopters Snap4City (C), November 2019 14 Collaborative Platform hackathons GO! City Operators Living Lab Accelerating
  • 15. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Develop Mobile & Web Applications Exploiting Snap4City Smart City Services Snap4City (C), November 2019 15 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design Application Development Application Requirements Analysis AdvancedSmartCityAPIand MicroServices Snap4City Mobile & Web Apps Development Kit Deploy Testing Publication Production Smart City Services Mobile and Web Apps
  • 16. Snap4City: Smart City IOT as a Service Snap4City (C), November 2019 16 TOP
  • 17. Smart City IOT as a Service • Snap4City applications may exploit multiple paradigms as data driven, stream and batch processing, putting co-creation tools in the hands of: – Smart Living Lab users and developers a plethora of solutions to develop applications without vendor lock-in nor technology lock-in, – final users customizable / flexible mobile Apps and tools, – city operators and decision makers specialized / sophisticated city dashboards and IOT/IOE applications for city status monitoring, control and decision support.Open to Organizations • Training and manuals: https://www.snap4city.org/108 • Help Desk: https://www.snap4city.org/3 • SLA: https://www.snap4city.org/497 • Terms of Use: https://www.snap4city.org/drupal/legal Snap4City (C), November 2019 17 https://marketplace.docker-fid.grid.cyf- kr.edu.pl/services/snap4city
  • 18. Specialized Offer Snap4City as Smart City IOT as a Service for • Who would like to create Living Labs as community exchanging experience with other cities as well; • Research Institutions, Departments and Projects which would like to perform research and experiments in the area of Smart City and IOT, without the needs of setting up the infrastructure, exploiting open data, collaborating, accessing to Data Analysis on demands, etc. This is the spirit of EOSC, European Open Science Cloud Marketplace at which Snap4City is registered as DISIT Lab, see [EOSC]. • Public Administrations, as small cities that would like to offer smart services and does not have economic power to manage service on t heir premise from them self. Snap4City (C), November 2019 18
  • 19. Help Desk and SLA • https://www.snap4city.org/drupal/contact • Bug Reporting – https://docs.google.com/forms/d/e/1FAIpQLSfD QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5 jsoXw/viewform • For Service Level Agreement see: – Service Level Agreement • Help Desk and Contact: – https://www.snap4city.org/drupal/node/3 • Availability rates: – https://www.snap4city.org/388 Snap4City (C), November 2019 19
  • 20. SNAP4CITY LIVING LAB FOR COLLABORATIVE WORK Snap4City (C), November 2019 20 TOP
  • 21. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Connect IOT/IOE Upload context Open Data Connect external Services Advanced Smart City API, MicroServices Manage Apps & Dashboards, User Engagement Monitor City Platform experiments workshops tutorials networking agreements events Start-ups Research groups City Users Large Industries collaborations Licensing, Gold services personal services Case Studies Inhouse companies Resource Operators Tech providers partnerships documentation Help desk Category Associations Corporations Advertisers Community Building subscription to applications Produce City IOT Applications & Dashboards Promote Applications & Dashboards Produce Apps and Dashboards for City Users Data Ingestion and Analytic algorithms Early Adopters Snap4City (C), November 2019 21 Collaborative Platform hackathons GO! City Operators Living Lab Accelerating
  • 22. Snap4City (C), November 2019 22 Your Level SLIDES News Flyer Exercises Full Search All Tools Personalized Suggestions
  • 24. Living Lab Flexibility • Multiple modalities to perform the same activities • Tuned for Beginners and Skilled people • Visual interface and programming tools • Resources and artefacts sharing for learn acceleration and co-working • Open Living and co-working Portal: https://www.Snap4City.org Snap4City (C), November 2019 24 Snap4City Satisfies all Requirements of ENOLL Select4Cities and EIP-SCC
  • 25. For the user: different levels of engagement • Multiple Organizations • Roles: – Managers – AreaManagers (developers): special access to computing resources • Levels for self-assessment – Suggestions on next steps to learn on the basis of personal progresses Snap4City (C), November 2019 25
  • 26. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Level 1 Users: creating dashboards See how Dashboards can be created using the wizard: dashboards with selectors, time trends, maps, etc. • TC1.8. Visual production of Dashboard via Wizard • TC1.9. Search on Wizard for any kind of data managed into the platform, from POI to sensors, KPI, social, etc. • TC1.10. Dashboard delegation to access, and passage of ownership, and/or cloning • TC1.11. IOT Discovery, on Dashboard Wizard • TC1.13. Dashboard Builder External Services and Widgets Snap4City (C), November 2019 26
  • 27. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 27 All Text on the Portal are Hypertext with Links for navigation among major concepts
  • 28. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org MultiOrganization, Groups and Profiles Organizations may have their distinct : • menus and functionalities, GeoArea, Data, Dashboard, Groups of users, managers, Knowledge Base, repositories, etc. Users may: • Have personal IOT Devices, Data, Dashboards, IOT App,.. • Have access to multiple Groups of Multiple Org. • Delegate them in usage or access • Assesses their usage and themselves Snap4City (C), November 2019 28
  • 29. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Self Assessment Snap4City (C), November 2019 29
  • 30. Snap4City (C), November 2019 30 Validation with developers • Helsinki and Antwerp, plus Florence Training, CINI Challenge, .. • 65 performed operational activities: – dashboards, IOT Applications, registering IOT devices, etc. – More than the 80% created both Dashboards and IOT Applications, thus validating the solution and the process of engaging them in working on the platform The 65 users left on platform Average per day over last 90 days Total activity 90 days Number of IOT Applications 117 81,6 7341 Number of private IOT devices 27 25,5 2296 Number of public dashboards 11 6,2 562 Number of private dashboards 173 135,1 12159 Number of accesses to dashboards -- 33,9 3048 Number of minutes -- 337,1 30337
  • 32. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Smart City Development Life Cycle Snap4City (C), November 2019 32 TOP
  • 33. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Development Life Cycle Smart City Services Snap4City (C), November 2019 33 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 34. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 34 Formal DesignLegenda phase Tool/Process MicroService/Node Widget In/Out Functional Historical Data Visual Interface Analysis Design IOT Application Implementation Requirements Specification ServiceMap Data Inspector Discovery IOT Directory MS Functional MS Analytical Data Analytics MS Yes Data Ingestion&modeling Data Gate/ETL IOT Directory for IOT Device Regis. External Services No Search/Store MS Ext. Services MS Python R-Studio Java, …… Develop. Data Analytic Dashboard MS IOT App Widgets Data Flow Development Dashboard Builder & Wizard User Interface Development Utilities MS Private Info MyKPI, Per. MSMyKPI Modeling Analysis and Design Development, Deploy, Testing Deploy, Testing Dev. Special Tools IOT App Widgets Custom Widgets Regular Widgets Web Scraping WebScraping MSWeb Data Development Life Cycle Smart City Services
  • 35. Snap4City (C), November 2019 36 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design Application Development Application Requirements Analysis AdvancedSmartCityAPIand MicroServices Snap4City Mobile & Web Apps Development Kit Deploy Testing Publication Production Smart City Services Mobile and Web Apps Develop Mobile & Web Applications Exploiting Snap4City Smart City Services
  • 36. Analysis and Design for Innovation (Co-Creation and Co-Working) Snap4City (C), November 2019 37 TOP
  • 37. Development Life Cycle Smart City Services Snap4City (C), November 2019 38 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 38. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Analysis & Design for Innovation • Analysis – The analysis start with a number of meetings/interviews with stakeholders – The identification of the target stakeholders/actors/users (target Segments) and their definition/description – The meetings are focused on filling the INNOVATRIX approach of IMEC tuned for the Snap4City platform and solution – See the schema of the INNOVATRIX reported in the next slide • Design – Focused on creating a large number of Use Cases and/or Scenarios for development – The design start taking into account the Snap4City development life cycles and tools. Thus shortening all the boring activities and following the typical Snap4City rapid prototyping described in these slides • Data Discovery – Data discovery is performed on analysis of the: (i) identified scenarios, (ii) data of the stakeholders, (iii) international sources, (iv) Snap4City experience, etc. – Performed by following the Snap4City guidelines on Data Search on web and world. Snap4City (C), November 2019 39
  • 39. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org INNOVATRIX by IMEC for the Analysis Snap4City (C), November 2019 40
  • 40. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Use Cases: Design vs Data Discovery • The analysis may produce possible needs vs current practices • The new innovative solutions have to be identified by experts working on these tables, thus producing Use Cases, Scenarios • Each of them has to be described and assessed taking into account: – Innovation level, social impact, – data availability (and thus as output of the Data Discovery) – User interaction and behavior constraints – technical feasibility, implementation costs in the context of Snap4City (data ingestion cost, data costs, development costs, etc. ) – Time and phases to for the development – Validation approach: feasibility and costs Snap4City (C), November 2019 44
  • 41. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Taking Decision with all: co-creation • Once performed the above described approach – A number of Use Cases, Scenarios have been created and well formalized • Than they may be – shortlisted according to the main stakeholder goals (city, operators, etc.) – voted in a participatory approach in the Living Lab – selected to be developed according to the next steps described in the following • Once developed, a validation with the same stakeholder is needed Snap4City (C), November 2019 45
  • 42. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Development Tools Snap4City (C), November 2019 46 TOP
  • 43. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Development Life Cycle Smart City Services Snap4City (C), November 2019 47 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 44. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org • Open Data: – Data gate, federation of Open Data Portals – ETL processes (PULL) – IOT Application processes • IOT Networks: – IOT Application processes, data driven or PULL – IOT Brokers (Push) → IOT Shadow • Web Pages: – Web scraping, crawling processes • Social media: Twitter, Facebook,.. – Twitter Vigilance, IOT App • Mobile Apps – Smart City API • Files upload: CSV, Excel, etc. – IOT Applications, ETL • REST API, WS, FTP, etc. – IOT Applications, ETL • Data base accesses – GIS: WFS, WMS – ETL, IOT Application Snap4City (C), November 2019 48 My Files IOT App ETL IOT Broker IOT Device IOT Broker Web Scraping DataGate API, External Services Rest Call …….MS GIS data, Maps, … Sensors/ Actuators IOT Edge IOT Device External Data Stores Any kind of data and flows
  • 45. IOT Network Manager vs Final User Snap4City (C), November 2019 49 IOT Broker IOT Broker IOT Broker IOT Broker Network of IOT Brokers Knowledge Base, Km4City Knowledge and Storage Data from the Field and City Discovering IOT Network Manager Registering Browsing ServiceMap Knowledge Base Discovering IOT Application Dashboard Wizard IOT Directory Final user Manager My IOT Device Register
  • 48. Developing IOT Applications Snap4City (C), November 2019 52 MicroServices collections IOT App. Editor Resource Manager Sharing/saving reusing IOT App My IOT Applications ServiceMap Discovery Dashboard Collection, Editor and Wizard Generating IOT App With Dashboard Knowledge Base, Km4City IOTDiscovering
  • 49. Web Scraping Snap4City (C), November 2019 53
  • 50. Web Scraping Snap4City (C), November 2019 54 IOT App. Editor Resource Manager Sharing/saving reusing Scraping My Scraping processes Generating WEB Scraping Knowledge Base, Km4City Web Scraper PORTIA
  • 51. Integrated DataGate/CKAN Static open data ingestion Snap4City (C), November 2019 55 Data Set: - Search - Loading - Download - Share - Publish - Also automated Automated data regularization Federated Crawling Federated Distribution
  • 52. Data Ingestion via Data Gate Snap4City (C), November 2019 56 Federated Network Saving / Sharing reusing Knowledge Base, Km4City Knowledge and Storage Data from the Field and City Data Gate Data Set: sharing, Harvesting, Loading/ Downloading Data Set: - Search - Loading - Download - Share - Publish - Also automated
  • 53. Developer in ETL: Penthao Kettle Snap4City (C), November 2019 57 Batch Processing for dynamic data ingestion
  • 54. Developing ETL, Data Manager Snap4City (C), November 2019 58 ETL Process ETL Process ETL Process ETL Process Resource Manager Saving / Sharing reusing ETL Development EnvironmentDISCES scheduler productionDistributed Back Office Knowledge Base, Km4City Knowledge and Storage Data from the Field and City Data Gate Data Set Saving / Sharing Data Set Loading/ Downloading Data sources Load data or prepare for data ingestionSchedule Execute
  • 55. Resource Manager: public and sharing Snap4City (C), November 2019 59
  • 56. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Data Gathering and Knowledge Management • Data ingestion can be performed by using multiple tools: – ETL processes, IOT Applications, Data Gate, WebScraping. We suggest: • ETL for static / periodic data in PULL • IOT App for real time data and flow, from IOT Brokers/Devices • DataGate for Static Data, upload them as files, or collected from other CKAN • WebScraper for scraping data from Web Pages, when authorized! • See how to test cases: – HOW TO: add data sources to the Snap4City Platform – HOW TO: define privacy rules for personal data, produced by the end-users own device – US6. Developing and using processes for data transformation – TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share – TC6.3 - Creating ETL processes for automated data ingestion and data transformation – TC6.5 - Managing Heterogeneous File Ingestion via ETL processes – TC6.9 - ETL processes for multiprotocol and format data ingestion, see on GITHUB for library – TC9.2 - Managing heterogeneous File Ingestion, protocols, formats via IOT applications, and open standards Snap4City (C), November 2019 60
  • 57. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Interoperability ESRI • Snap4City is interoperable with – ESRI ArcGIS Enterprise, Portal, Pro/MAP, ... – other GIS tools supporting WFS, WMS, GeoJSON, GML • Snap4City is interoperable since: – Provides info/data in WFS, WMS – Exploits data/info from WFS, WMS – Import data/info from WFS/WMS • The Snap4City platform can be installed on premise using Snap4City Appliance https://www.snap4city.org/drupal/node/471 – StartSNAP4CITYVM includes the Dashboard Builder that is capable to work with WFS WMS protocols for the integration with GIS platforms as ESRI ArcGIS, QGIS, directly or using Snap4City GIS player. – KBSSMVM includes the Smart City API and WFS API which can be used to data harvest from any GIS servers and GIS desktop tool Snap4City (C), November 2019 61
  • 58. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Linked Open Graph Snap4City (C), November 2019 62 LOG: https://log.disit.org Schema: http://www.disit.org/km4city/schema RDF version: http://www.disit.org/km4city.rdf Linked Open Data
  • 59. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Development Life Cycle Smart City Services Snap4City (C), November 2019 63 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 60. Data Analytics Dev. in R Studio and/or Tensor Flow Snap4City (C), November 2019 64 Knowledge Base, Km4City Ontology Schema SPARQL, FLINT LOG.disit.org Swagger SmartCityAPIfromKnowledgeBaseandothertools Big Data Store Facility Resource Manager Creating MicroServices Saving / Sharing reusing Using them into IOT Applications
  • 61. Data Analytics Dev. in Java, Python, .. Snap4City (C), November 2019 65 Knowledge Base, Km4City Ontology Schema SPARQL, FLINT LOG.disit.org Swagger SmartCityAPIfromKnowledgeBaseandothertools Big Data Store Facility Resource Manager Saving / Sharing resusing Process Process Process Process Distributed Back Office DISCES scheduler Monitoring Coding Testing Data sources
  • 62. Loading new Node-RED nodes/microservices from external Palet • Ask to RootAdministrator to have your custom Node-RED nodes/modules added to your IOT Applications – https://www.snap4city.org/drupal/contact • A validation of requested Node-RED nodes will be performed to avoid violation of security and privacy for all • Administrators may load custom Node-RED nodes/modules • This limitation is not present in your on premise installations of Snap4City Snap4City (C), November 2019 66
  • 63. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Development Life Cycle Smart City Services Snap4City (C), November 2019 67 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 64. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Dashboard List and Editor Snap4City (C), November 2019 68
  • 65. Dashboard List and Editor Snap4City (C), November 2019 69
  • 66. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Dashboard Wizard Snap4City (C), November 2019 70 • Select the area of your interest: panning and zooming • Select the • graphic aspect of your interest, or • High Level Type of your interest, or • Make a search if you a have a precise idea or • Act on filters: nature, subnature, type, name, value, date, health, owner, … • Combine them as you like • Select the lines of your interest • Then click on Next and get the Dashboard by wizard
  • 67. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Dashboard Wizard Snap4City (C), November 2019 71 The Wizard help you in selecting only possible combination of data vs graphic representation
  • 68. Developing Dashboards Snap4City (C), November 2019 72 Knowledge Base, Km4City Create, save, load, delegate, grant access Dashboard Collection Dashboard Wizard IOT Applications MicroApplications External Services Dashboard Editor Widget Collection Knowledge and Storage Data from the Field and City My Own Dash/App
  • 69. Dashboard Development Snap4City (C), November 2019 73 Knowledge Base, Km4City Create, save, load, delegate, grant access Dashboard Collection Dashboard Wizard IOT Applications MicroApplications External Services Dashboard Editor Widget Collection Knowledge and Storage Data from the Field and City My Own Dash/App
  • 70. Developing new Dashboard Graphic Widget • The development of new Widget is feasible for programmers – in PHP, JavaScript and CSS • The starting point is the Core Snap4City Virtual Machine Appliance StartSNAP4CITYVM: https://www.snap4city.org/471 • While the source code is also accessible on GitHub/DISIT • see TC3.11 - New graphics widget can be easily created, Dashboard Widget Creation – creating a new Graphic widget via SVG and programming: Custom Synoptics and Widgets for Dashboards https://www.snap4city.org/504 Snap4City (C), November 2019 74 https://www.snap4city.org/drupal/node/504
  • 71. Snap4City (C), November 2019 75 Custom Dashboards and Widgets (interactive, Animations, etc.) -- SVG for graphic design -- MyKPI for collecting data Custom Widgets
  • 72. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Creating custom Widgets for Dashboards Snap4City (C), November 2019 76 Designer JS programmer Producing and Publishing MyKPI WS secure USING IOT Edge IOT Devices DEVELOPING JS Library
  • 73. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Dashboards • Suitable as: City Dashboard, App interface, and Control Room Dashboards, Situation Room Dashboard, Operator Dashboard • Created visually compounding graphic Widgets – Each widget has an autonomous update – Each metric/data-source may have associated with an alarm: blinking and sending events to people and machines in different manners • Can be: public or private, private dash can be delegated or passed in ownership • See https://main.snap4city.org/management/dashboards.php?linkId=dashboardsLink&fromSubmenu=false&sorts[title_header]=1 • See the following tutorials – HOW TO: create a Dashboard in Snap4City – US1. Using City Dashboards – US2. Using and Creating Snap4City Applications with Dashboards – US4. Creating City Dashboards and related Event Monitoring and Actions Snap4City (C), November 2019 77
  • 74. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Level 1 Users: using dashboards • TC1.3. Accessing and using Dashboards with any device • TC1.4. Dashboards Showing Data Real Time and Historical/trends, comparison • TC1.5. Dashboards showing a range of different High Level Types: KPI, POI, IOT dev, MicroApp, Maps using different kind of Graphics Widget. Monitor city status with Dashboards • TC1.6. Dashboards using different kind of graphics Widgets matching with High Level Types, Monitor city status with Dashboards • TC1.12. Dashboard with MicroApplications • TC4.1. Dashboard and Notificator • TC4.6. Dashboard with city events, ESB, police, traffic, etc. Snap4City (C), November 2019 78
  • 75. Snap4City (C), November 2019 79 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design Application Development Application Requirements Analysis AdvancedSmartCityAPIand MicroServices Snap4City Mobile & Web Apps Development Kit Deploy Testing Publication Production Smart City Services Mobile and Web Apps Develop Mobile & Web Applications Exploiting Snap4City Smart City Services
  • 76. Developing Web and Mobile Apps, MicroApps,.. Snap4City (C), November 2019 80 Knowledge Base, Km4City Swagger Advanced Smart City API Web App HTML5, MicroApplications Embed into Web pages Mobile Apps ServiceMap DeveloperKm4City Open Source development tool kit City User Mobile Application Monitoring Administrator
  • 77. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org IOT Directory Back Office Processes IOT Broker IOT Broker IOT Broker IOT Broker ETL Process Data Analytics ETL Process ETL Process ETL Process Data Analytics Data Analytics Data Analytics Knowledge Base, Km4City Smart City API from Knowledge Base and other tools Ontology SPARQL, FLINT LOG.disit.org ServiceMap ServiceMap3D Swagger MicroServices IOT ApplicationsWeb and Mobile AppsDISCES and back office management tools Snap4City (C), November 2019 81 MicroApplications Resource Manager
  • 78. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org How to Add Functions that are not present in the Platform Snap4City (C), November 2019 82 TOP
  • 79. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Connect IOT/IOE Upload context Open Data Connect external Services Advanced Smart City API, MicroServices Manage Apps & Dashboards, User Engagement Monitor City Platform experiments workshops tutorials networking agreements events Start-ups Research groups City Users Large Industries collaborations Licensing, Gold services personal services Case Studies Inhouse companies Resource Operators Tech providers partnerships documentation Help desk Category Associations Corporations Advertisers Community Building subscription to applications Produce City IOT Applications & Dashboards Promote Applications & Dashboards Produce Apps and Dashboards for City Users Data Ingestion and Analytic algorithms Early Adopters Snap4City (C), November 2019 83 Collaborative Platform hackathons GO! City Operators Living Lab Accelerating
  • 80. Development Life Cycle Smart City Services Snap4City (C), November 2019 84 Analysis Design Data Discovery Data Ingestion IOT App Development Dashboard Development Data Analytics Development Deploy TestingPublication Production Special Tool Development Data Analytics Development Deploy Analysis & Design
  • 81. Adding a Function that is not present on the platform • Snap4City allows at the AreaManagers to Develop new functions for: – Interacting with the field: IOT Sensors/Actuators, IOT Devices – Data production by using: IOT Devices, IOT Brokers, .. – Data Ingestion/Transformation by using: ETL, IOT Applications, WebScraper, DataGate – Data Analytics by using: R Studio, Java, C++, Python – Data Rendering on Dashboards: PHP, HTML5/JavaScript, Web App, etc. – User interaction and App: Mobile App, MicroServices, Dashboards Snap4City (C), November 2019 85
  • 82. Snap4City vs Fi-Ware Snap4City (C), November 2019 86 TOP
  • 84. 89
  • 85. • Is open to the Development of Applications leaving large space to developers • Is cantered on the Orion Broker that result central in the architecture: any Broker or data source is sending data to Orion • Security level is not clear, partially demanded to developers • Visual Flexible IOT processing is not clearly provided • Limited API for IOT data access • Knowage BI presents several limitations in showing Smart City Data • Market place on Open Data • Support of Developers via Fi-Ware • Deployed as VM and Dockers • open source, not the application parts Snap4City (C), November 2019 90 • Is open to the Development of Applications leaving large space and providing a large set of ready to use applicative tools and solutions to build their solutions on top or aside. • Is fully distributed, any kind of data source can be ingested, automatically. • Orion Broker is only one of the Brokers that can be used. It can be also protected by Snap4City tech, with Mutual Authentication • Visual Flexible IOT processing is provided as Node-RED and Snap4City MicroServices suites • Advanced Smart City API are provided on top of Knowledge Base • Dashboard Builder has been designed for Smart City Data and automated dashboards’ production • Market Place on Open Data, tools, processes, experiences • Full Support for Living Lab of the city, coworking, tutorials • Deployed as VM and Dockers • 100% open Source vs
  • 86. vs • Snap4City is an official Fi-Ware Solution via – NGSI V1, V2 The IOT Orion Broker – IOT Orion Broker can connect JSON, MQTT, Lightweight M2M, LoraWAN, OPC, SigFOX, etc. see Fi-Ware Https://www.fiware.org • Snap4City is compatible with all the above protocols – via IOT Orion Broker, – via direct connection on ETL processes on their corresponding IOT brokers, and/or – via IOT Applications. • Snap4City is also compatible with many other protocols, see the table reported in page: https://www.snap4city.org/65 Snap4City (C), November 2019 91
  • 87. Standards and Interoperability Snap4City (C), November 2019 92 Compliant with: AMQP, COAP, MQTT, OneM2M, HTTP, HTTPS, TLS, Rest Call, SMTP, TCP, UDP, NGSI, LoraWan, TheThingsNetwork, SigFOX, DATEX II, SOAP, WSDL, Twitter, FaceBook, Telegram, SMS, OLAP, MySQL, Mongo, HBASE, SOLR, SPARQL, EMAIL, FTP, FTPS, WebSocket, WebSocket Secure, ModBUS, OPC, GML, RS485, WFS, WMS, ODBC, JDBC, Elastic Search, Phoenix, XML, JSON, CSV, db, GeoJSON, Enfuser FMI, Android, Raspberry Pi, Local File System, ESP32, Libelium, IBIMET, OBD2, SVG, XLS, XLSX, TXT, HTML, CSS, etc.
  • 88. • In Snap4City you can chose to connect your devices at Snap4City Platform in different manners: – (a) directly to Snap4City, or – (b) via an IOT Orion Broker (external IOT Broker or those provided by Snap4City), or – (c) via any third party IOT Brokers in any protocol you have. • Snap4City has – Improved IOT Orion Broker with the so called Orion Broker Filter (Orion Broker Filter, NGSI Security Wrapper) which is a secure wrapper for NGSI V1 and V2 protocol for enforcing Mutual Authentication, Security, roles, etc. – Produced open hardware and open software NGSI Compliant: as • IOT Devices with mutual authentication and security based for NGSI on: Android, Arduino and ESP32, IOT Button, etc. • IOT Edge devices with mutual authentication and security based for NGSI on: Raspberry PI, Windows, Linux. Snap4City (C), November 2019 93 vs
  • 89. Snap4City vs State of the Art Solutions Snap4City (C), November 2019 94 TOP
  • 90. Market Solutions Snap4City (C), November 2019 95 IOTDiscoveryAbstraction Authentication,Authorization Securityend-2-end,secureon IOTandDashboards OpenHWandOpenSW IntegratedCommunity management DataTypes:IOTDevices,IOT App,Dashboard,Data DataType:Publish/share, Delegation,Consentandchange DataType:Downloadand Delete AuditingonDataTypeAccess OpenSourceend-to-end ScalabilityIOT VisualProgrammingend-to-end applications AdvancedSmartCityAPI, MicroServices MultiDomainSemantic Platform StandardbasedModulesand IOT,OpenDevices ResourceSharing DataAnalyticsintegrated DashboardH24/7,protected connection Multi-protocolonIOT G G G G G Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N Siemens MindSphere [60] Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
  • 91. Snap4City managed to provide a maximum of information, flows, in depth analysis with the data provided. There is no other platform that collects all city actors together. The City officials and ICT officials were impressed with the performance of the Platform when loading the heavy, “resource-” demanding applications and dashboard. The data handling throughout the Platform is considered as one of the strong points in the Platform and of an extremely sophisticated level. The technical level of the Platform and its strong points such as the way real-time data is used, the algorithms, data clean-up possibilities of the Platform, presented data and information is state-of-the-art and impressive. Snap4City (C), November 2019 97
  • 92. Snap4City Services: Consulting & Developing Snap4City (C), November 2019 98 TOP
  • 93. Providing consulting, customization, training, and developments • Snap4City solution can be installed on premise and one cloud, private and public. • Snap4City, Snap4 SRL, provide support, if needed, for design and/or Develop: – Training and tutoring; – Smart city infrastructure and architecture; – data analytics, that could be developed as proprietary solution for the customer; – data ingestion processes, to enable them to have data into the platform; – adaptor for some specific protocol or legacy/third part Tool, that we prefer to release as open source, but if the connection is with some proprietary tool, the buyer could be interested to keep these solutions as private; – IOT devices, full solutions, dashboards, specific dashboard widgets, etc.Snap4City (C), November 2019 99
  • 94. Snap4Industry • Snap4City technology can be exploited on Industy and IOT solutions: – Snap4Industry: Snap4City for Industry 4.0 (SLIDES) – Scenario: 5G Enabled Water Cleaning Control – Scenario: High Level Control of Industrial Plant – Custom Synoptics and Widgets for Dashboards – The integration between data and devices: the Snap4City solution – Snap4City (C), November 2019 100
  • 95. Snap4Industry Snap4City tech. for Industry 4.0 https://www.snap4city.org Snap4City (C), November 2019 101 TOP
  • 96. Snap4Industry as a Lambda Architecture Knowledge base Semantic reasoners Data Sources PULL Data Data Sources, Brokers, External Services Data Driven, Real Time Data Ingestion, aggregation, regularization, reconcile ETL, NIFI, DISCES Big Data Cluster HDFS, Hbase, Phoenix Search and Query, Smart API Facet, semantic searchIndexing and aggregating Elastic search Data Analytics R, Tensor Flow, Python, MapReduce, … Visual Analytics Special dash/ tools Rendering Acting, Widgets, MicroApps User interface, Interactive Dashboard, Drill down, maps, heatmaps Inform, announce, Act!, warning, alarms, What-IF, .. IOT Applications Node-RED + Snap4City MicroSrvs
  • 97. Security Architecture at a glance Data Visualization Virtual Devices Cloud Processor Data Injection Device Management Edge Processor Data Sources IoT Devices (sensors, actuators) Security and Privacy Management Context Brokers IoT Broker IoT App IoT Context Brokers IoT App IoT Edge IoT Directory Registries and storage Dashboards Snap4City (C), November 2019 103
  • 98. Industry 4.0 Application Snap4City (C), November 2019 104 Sensors/ Actuators Sensors/ Actuators Sensors/ Actuators IOT Broker IOT Gateway [IOT Application] MicroServices Ext-Services ……. Executing permanent computation 0,00 200,00 400,00 600,00 800,00 1 3 5 7 9 11 13 15 17 19 21 23 25 Serie2 Serie3 Serie4 Serie5 ETL Process Executing local computation Internet Dashboards -- predictions -- suggestions -- Anomaly detection IOT Edge Sensors/ Actuators Sensors/ Actuators Energy Providers Data Analytics IOT Broker On the field IOT Devices
  • 99. GIDA set up ModBus to Snap4City Gateway Edge 5G network devices Smart City data from many sources Dashboards and Apps IOT Applications Big Data Analytics, Artificial Intelligence IOT Data Shadow Snap4City Telemonitoring Telecontrol
  • 102. Green Inpact project Snap4City (C), November 2019 108 Green Impact Capacity (GIC) Altair Control room
  • 103. Green Impact Capacity (GIC) • Improve productivity of chemical plant • Keep GREEN the environmental impact • Exploiting innovative technologies • Diversify the production • Monitoring environmental conditions Snap4City (C), November 2019 109
  • 104. Installing Snap4City Snap4City (C), November 2019 110 TOP https://www.snap4city.org/471 To get an updated version read it!
  • 105. Knowledge base Semantic reasoners Data Sources PULL Data Data Sources, Brokers, External Services Data Driven, Real Time Data Ingestion, aggregation, regularization, reconcile ETL, NIFI, DISCES Big Data Cluster HDFS, Hbase, Phoenix Search and Query, Smart City API Facet, semantic searchIndexing and aggregating Elastic search Data Analytics R, Tensor Flow, Python, MapReduce, … Visual Analytics Special dash/ tools Rendering Acting, Widgets, MicroApps User interface, Interactive Dashboard, Drill down, maps, heatmaps Inform, announce, Act!, warning, alarms, What-IF, .. IOT Applications Node-RED + Snap4City MicroSrvs Snap4City as a Lambda Architecture
  • 106. Overview of Snap4City platform, for Buyers, for all Snap4City (C), November 2019 112 Snap4City MAIN Dashboard, User Management, etc. IOT brokers HeatMaps and Maps Knowledge base, semantic Store Other Services And Servers Mobile Apps, Dashboards Cluster for Smart Processes container: ingestion, analytics, scraping, etc. Cluster for big data storage: static and real time, Indexing and search GIS IOT Networks Social Media CRM ….. Living Lab Portal Legacy Platform Back End Front End
  • 107. Overview of Snap4City platform, for Buyers, for all Snap4City (C), November 2019 113 Snap4City MAIN Dashboard Builder, Data Inspector, User Stats, ExternalSrv, Res Manager, MyKPI, MyPOI, Synoptic, IOT Device Manager, Micro Applications, Authorization/ Authentication, Roles, … IOTBrokers, secure IOTOBSFs GISGeoServer Heatmap Manager, WFS, WMS KBSSM KB/ServiceMap, .. SmartCity API, WFS, ServiceMap3D, LOG/FLINT, SuperServiceMap, Virtuoso Services CKAN DataGate Routing Server Chat Manager R Studio Server ETLSDK Server Engager Server ……. Server Mobile Apps, Dashboards Container Cluster MCLSCont, NCLSCont Marathon, Mesos, Zookeeper, Disces-EM, Executing in Container: IOT App, Web Scraping, R-Studio, Python, Java, ETL, ... Data Shadow Cluster IOTDSES NIFI, Elastic Search, DevDash, Amma [DataCluster] ETL-VM Phoenix HBASE, HDFS DISCES, ETL support IOTBrokers, secure GIS IOT Networks Social Media CRM ….. Living Lab Portal Drupal support CRM, ..Legacy Platform Back End Front End
  • 108. What is included in the Buyers / Full Platform • Snap4City MAIN – Dashboard Builder, Wizard – Dashboard Engine – Data Inspector, Notificator – External Srv manager – Menu Manager – Ownership Manager – Authentication and Authorisation – WS server – Resource Manager – User Stats – JavaScript Web App in a Snap, MicroApplications – Synoptics, custom Widgets – IOT device manager: IOT Directory – Snap4City MicroServices on IOT Applications: basic and advanced – Snap4City GIS Player – What-IF tools Snap4City (C), November 2019 114 • IOTOBSF – IOT Orion Broker Fi-Ware – Secure Filter (Snap4City) • IOT Edge support – Linux Ubuntu – Windows – Raspberry Pi – Android • IOT Devices support – ESP32 – Arduino • IOT Application – Node-RED – Snap4City Library of nodes • KBSSM – Knowledge Base Km4City – ServiceMap – ServiceMap3D – SuperServiceMap – LOG/Flint – OSM2Km4City – Smart City API – WFS API • GIS GeoServer – GeoServer – Heatmap Manager • Living Lab Portal – CRM Drupal – LDAP Snap4City – Registration mng – Etc. etc. • MCLSCount, NCLSCount Container[Cluster] – Containers models – DISCES-EM – Monitoring App • ETL Server Data[Cluster] – DISCES – Phoenix Drivers – Hbase Model – ETL processes • IOTDSES DataShadow[Cluster] – NIFI process, Squid – Elastic Search Model – Kibana – AMMA data flow – DevDash data store • Other Services – Routing Server – DataGate CKAN – Chat as Rocket – ETL SDK VM – Engager .. – ….. • Data Analytics – Heatmap production MS Rstudio – OD production – Predictions MS RStudio – Anomaly detection MS RStudio – ETL Collection – …. All in source code, and most of components in Appliances/VMs
  • 109. Platform Maintenance, K3.14 • Snap4City modules are released on GITHUB/DISIT and can be updated from: – GITHUB/DISIT into VM Appliance or Servers in which they are installed • https://github.com/disit – Node-RED tool, using official Library regarding: Snap4City Libraries – Drupal for the Living-Lab Portal aspects – Other tools from their corresponding providers • Maintenance would not be a problem, all users should be capable to perform the updates autonomously • Updates on new versions will be provided by Snap4City periodically, Notification will be provided on NEWS and GitHUB Snap4City (C), November 2019 115
  • 110. Keeping Platform at the State of The Art • Updates will be provided by Snap4City periodically and released on GITHUB, and other portals. • Snap4City team is involved in contracts since now, so that this will guarantee that the solution will be evolved to anticipate the state of the art as we have done in Iteration 3, in which we released a number of developments. – See also the activity of dissemination and updated roadmap – See the presence of Snap4City on EOSC, BeeSmartCity, EO15, etc. – See the list of concreate developments – See the list of supporters on the Stand with respect to those of the 2018 – See the planned new developments Snap4City (C), November 2019 116
  • 111. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Set-up of Open Source version from Scratch • Guidelines reported on: – https://www.snap4city.org/drupal/node/471 • How to proceed: – We have presented 7 Configurations models from A:Alone to F:FullPlatform+LivingLab, but they can be customized as you like. – By follow the Configurations the Customers are guided to identify the most suitable according to their needs; – Once the most suitable Configuration has been identified, they are supported into the: i. Download and deploy of the Appliances provided as Virtual Machines or Containers, or ii. Download and install modules from GITHUB according to the recipes for VM/Container and the user manual of the single tools, or iii. Mixt of the above (i) and (ii) approaches. – IOT Edge can created by (1) installing Node-RED, (2) adding Snap4City library of Nodes from the Palette Manager of Node-RED, (3) registering on Snap4City: https://nodered.org/docs/user-guide/editor/palette/manager Snap4City (C), November 2019 117 Note that some of the material we are presenting has not been published on the portal yet, for the competition.
  • 112. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City Scalable Platform with your business Snap4City (C), November 2019 118 Configuratio n Kind Min#VM Dockers IOTBrokerint IOTBrokerExt MAP KBSM IOTAPP, MicroServices Storage&IOT DataShadow SSO,Roles IOTSecurity Dash+Wizard GISin GISout Heatmap Mob.App ASCAPI LivingLabSup. Options supported by the configuration A: Alone 1 6 X X 1 mf X X X X (x) ETL, DG, CM B: Basic 2 7 X X X 1 mf Small X X X X (x) ETL, DG, CM C: CityStart 3-4 8-9 X X X X 1 mf Small X X X X X X X X ETL, DG, R, CM, Eng D: DataCity 5-6 Ask X X X X 70 Mediu m X X X X X X X X ETL, DG, R, CM, Eng, LL E: ExtensiveCity 6-8+ Ask X X X X 70 Scalable X X X X X X X X ETL, DG, R, CM, Eng, LL F: FullPlatform 12-14+ Ask X X X X Scalab le Scalable X X X X X X X X ETL, DG, R, RS, CM, Eng, LL F: FullPlatform + LivingLab 13-15+ Ask X X X X Scalab le Scalable X X X X X X X X X ETL, DG, R, RS, CM, Eng mf: multiple flows for each IOT App Read next slide for the other notes and legenda
  • 113. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Legenda and Notes on the previous table • In all configurations you: – have: Multiple IOT Brokers; unlimited number of data sources; interoperability support; KB – may have: HA, DRS, FT, Balancing, cloning and configuring additional VM; – may pass at the next configuration without reinstalling the VM; – may add one or more Optional Services as VM/Containers: ETL, DataGate, Routing, Engager, Data Analytics, etc. etc. • From Config. E:ExtensiveCity and upper the number of VMs depends on the volume of DataStorage and the volume of Data Stream in input. – These two aspects are managed by two independent clusters of VMs and scale independently each other. This allows to satisfy any different combination of volumes in streams and data storage. • We suggest using Living Lab support only on FullPlatform, while it can be installed on Smaller Configurations with some limitations in terms of features Snap4City (C), November 2019 119 Optional Services • DG: DataGate CKAN • R: Routing • ETL: ETL SDK VM • RS: R Studio Server • Eng: Engager • CM: Chat Manager • LL: Living Lab • .. • ..
  • 114. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Computational and Storage Costs • The VM is considered (appliances are provided) as 16/24 cores 2.2 Ghz, 16-24 GB Ram, 500 GB HD in thin provisioning, with 25-40 GB HD used at the start, in most cases Debian. – to estimate the needed CPU, RAM, Storage for each configuration it is easy from the #of VM in the configuration. – Please note that configurations A and B can be executed on 4 cores, 4 GB Ram, .. • In solutions, with a very large number of users on the Front End: Smart City API (mobile App users, Dashboard users, a frontend balancer and more FrontEnd servers would be needed). Please note that the Smart City API are also used by DataAnalytics processes and by MicroServices in the IOT Applications. – Typically a thousand of simultaneous users on the front end can be sustained for each VM – More precise estimations can be performed by knowing the actual workload Snap4City (C), November 2019 120
  • 115. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org A Alone Configuration Snap4City (C), November 2019 121 TOP
  • 116. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org A: Alone) For Small Size Business, for example an industry 4.0 application you can need to install only one VM: Snap4CityMAINVM • For connecting data sources to Dashboards onpremise! – Data driven solution, no large data store – Industry 4.0 – Small Smart City business – Start with Snap4City platform as seeding element Snap4City (C), November 2019 122 • Snap4City MAIN VM – Dashboard Builder, Wizard – Dashboard Engine – Data Inspector, Notificator – External Srv manager – Menu Manager – Ownership Manager – Authentication and Authorisation – WS secure server – Resource Manager – User Stats – JavaScript Web App in a Snap, MicroApplications – Synoptics, custom Widgets – IOT device manager: IOT Directory – Snap4City MicroServices on IOT Applications: basic and advanced – Snap4City WFS player – What-IF tools Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator, WS Server, Data Inspector, User Stats, ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI, … IOT Applications, Synoptic, Snap4City WFS GIS player, IOT Device Manager…. External Data Stores
  • 117. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4CityMAIN: Alone Configuration • As Virtual Machine: – HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City – https://www.snap4city.org/drupal/node/487 • As Container Composer: – HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City – https://github.com/disit/snap4city-docker/tree/master/Alone Snap4City (C), November 2019 123
  • 118. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org B Basic Configuration Snap4City (C), November 2019 124 TOP
  • 119. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org B: Basic) Addressing a relevant number of IOT devices/ IOT brokers in your smart city or industry 4.0, a few smart Applications with multiple flows Snap4City (C), November 2019 125 For managing beyond A:Alone Configuration + large number of IOT devices and brokers, any protocol, format + IOT Devices + IOT Edge + Etc. Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator, WS Server, Data Inspector, User Stats, ExternalServices, Resource Manager, MicroApplications Authorization/Authentic. MyKPI, MyPOI, … IOT Applications, Synoptic, Snap4City GIS WFS player, IOT Device Manager IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOTOBSFs Sensors/ ActuatorsIOT Edge External Data Stores
  • 120. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org B: Basic) Addressing a relevant number of IOT devices/ IOT brokers, a few smart Applications • IOT Edge: local intelligence, processes and Applications located on the field, on premise. Those autonomous smart tools can be implemented by IOT Applications working on IOT Edge: • Linux Ubuntu Appliance VM: https://www.snap4city.org/drupal/node/298 • Android as IOT Edge https://www.snap4city.org/drupal/node/278 and • Raspberry Pi as IOT Edge: https://www.snap4city.org/drupal/node/77 You can install Snap4City library in any Node-RED from the official Node-RED library • IOT Applications = Node-RED + Snap4City libs of Nodes/MicroServices, – Available in Snap4CityMAIN VM, so that in any installation – any installation of Node-RED add Snap4City Library from the official Library of Node-RED, add new Nodes into the Palette as • https://nodered.org/docs/user-guide/editor/palette/manager – Snap4City MicroService collection from the JS foundation with full documentation inside https://flows.nodered.org/?term=snap4city • BASIC: suitable for Final User • ADVANCED: DEVELOPER (you have to install both libraries: basic and advanced) Snap4City (C), November 2019 126 • IOTOBSF VM – IOT Orion Broker Fi-Ware – Secure Filter (Snap4City) • IOT Edge support – Linux Ubuntu – Windows – Raspberry – Android – Snap4City Library of nodes • IOT Devices support – ESP32 – Arduino • IOT Application – Node-RED – Snap4City Library of nodes
  • 121. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org C) CityStart Configuration Snap4City (C), November 2019 127 TOP
  • 122. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org C: CityStart) Addressing a relevant number of GeoLocated Points of Interest, PIN on maps, and eventually heatmaps, a Small Smart City with a few smart IOT applications on cloud and many on edge Snap4City (C), November 2019 128 For managing Beyond B:Basic Configuration + Large number of GEO elements on maps + Heatmaps + Geo shapes + GIS, …. Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator Data Inspector ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI IOT Device Managner SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso SCAPIBalanc. SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso DeveloperBalancer Snap4CityMAIN Dashboard Builder Wizard, Widgets, IOT App1 Notificator, WS Server, Data Inspector, User Stats, ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI, Synoptic IOT Device Manager KBSSM GIS, GeoServer GIS, GeoServer IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOTOBSFs Sensors/ Actuators IOT Edge External Data Stores
  • 123. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org C: CityStart) Addressing a relevant number of GeoLocated Points, heatmaps, ... Snap4City (C), November 2019 129 • KBSSM VM – Knowledge Base Km4City – ServiceMap – ServiceMap3D – SuperServiceMap – LOG/Flint – OSM2Km4City – Smart City API – WFS API • GIS GeoServer VM – GeoServer – Heatmap Manager • KMSSM VM: – knowledge base, KB, positioning elements on maps, enabling geo- spatial and temporal reasoning – Ingestion OSM to load on KB, ServiceMap and reasoning on it – GeoReverse data, from street to data and vice versa – WFS export and import • GIS GeoServer VM: – Producing and providing heatmaps – Managing Heatmaps colour maps. – WMS, WFS export, distribution
  • 124. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org D) DataCity Configuration Snap4City (C), November 2019 130 TOP
  • 125. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org D: DataCity) A medium/large size Smart City with a number of smart applications on cloud and a number of IOT brokers, relevant volume of data entering into the cloud. Snap4City (C), November 2019 131 MCLSCont Mesos Balancing Cnt. Scheduling DISCES-EM: Cloud Elastic Management Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator Data Inspector ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI IOT Device Managner SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso SCAPIBalanc. SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso DeveloperBalancer Snap4CityMAIN Dashboard Builder Wizard, Widgets, IOT App1 Notificator, WS Server, Data Inspector, User Stats, ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI, Synoptic IOT Device Manager IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOTDSES KBSSM IOTOBSFs GIS, GeoServer GIS, GeoServer Services CKAN DataGate Routing Server Chat Manager R Studio Server ETLSDK Server Engager Server ……. Server
  • 126. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org D: DataCity) A medium/large size Smart City with a number of smart applications on cloud and a number of IOT brokers, relevant volume of data entering into the cloud. Snap4City (C), November 2019 132 • MCLSCount Container[Cluster] – Containers models – DISCES-EM – Monitoring App • ETL Server Data[Cluster] – DISCES – Phoenix Drivers – Hbase Model – ETL processes • IOTDSES DataShadow[Cluster] – NIFI process – Elastic Search Model – AMMA and DevDash For Managing Beyond C:CityStart Configuration + A relevant number of IOT Applications for data transformation and Analytics + A small sized Data Shadow for IOT data management Optionally: • Data Analytics in R Studio, Tensor Flow (NVIDIA) • Hbase / Phoenix Big Data Store • Data Gate CKAN for Open Data • Chat Manager on Dashboard • Routing Solutions
  • 127. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org E) ExtensiveCity Configuration Snap4City (C), November 2019 133 TOP
  • 128. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org E: ExtensiveCity) A Large size Smart City with smart applications on cloud and a number of IOT brokers, large volume of data streams entering in the cloud Snap4City (C), November 2019 134 Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator Data Inspector ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI IOT Device Managner SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso SCAPIBalanc. SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso DeveloperBalancer Snap4CityMAIN Dashboard Builder Wizard, Widgets, IOT App1 Notificator, WS Server, Data Inspector, User Stats, ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI, Synoptic IOT Device Manager IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security Balancer IOTDSES KBSSM IOTOBSFs GIS, GeoServer GIS, GeoServer MCLSCont Mesos Balancing Cnt. Scheduling DISCES-EM: Cloud Elastic Management Services CKAN DataGate Routing Server Chat Manager R Studio Server ETLSDK Server Engager Server ……. Server
  • 129. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org E: ExtensiveCity) A Large size Smart City with smart applications on cloud and a number of IOT brokers, large volume of data streams entering in the cloud Snap4City (C), November 2019 135 • IOTDSES DataShadow Cluster – NIFI process – Squid cache on SCAPI – Elastic Search Model extended API – Kibana – AMMA data flow – DevDash data monitor – … For Managing Beyond D:DataCity Configuration + A relevant number of data streams entering in the platform + A scalable Data Shadow for IOT data management Optionally: • Data Analytics in R Studio, Tensor Flow (NVIDIA) • Hbase / Phoenix Big Data Store • Data Gate CKAN for Open Data • Chat Manager on Dashboard • Routing Solutions
  • 130. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F) FullPlatform Configuration Snap4City (C), November 2019 136 TOP
  • 131. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F: FullPlatform) For Huge scale Smart City with a Huge number of smart applications and processes on cloud (thousands), any number of IOT brokers Snap4City (C), November 2019 137 [Phoenix ] [HDFS, Hbase] [Phoenix ] [HDFS, Hbase] [Phoenix ] [HDFS, Hbase] [Phoenix ] [HDFS, Hbase] Pool of Containers NodeRed, Web Scraping [Penthao Kettle, Java, Karma] …….. R-Stutdio Python, C++, C ………….. Nodes/VM on Cloud in HA, elastic computing, .. . . . …… …….. ………. . Masters in HA: Marathon/Mesos, Zookeeper, …. Master Master Master containersDistribSrv. MCLSCont Mesos Balancing Cnt. Scheduling DISCES-EM: Cloud Elastic Management NCLSCont Snap4CityMAIN Dashboard Builder Wizard, Widgets Notificator Data Inspector ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI IOT Device Managner SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso SCAPIBalanc. SmartCityAPI KB/ServiceMap ServiceMap3D LOG/FLINT Virtuoso DeveloperBalancer Snap4CityMAIN Dashboard Builder Wizard, Widgets, IOT App1 Notificator, WS Server, Data Inspector, User Stats, ExternalSrv, Resource Manager, MicroApps Authorization/Authentic. MyKPI, MyPOI, Synoptic IOT Device Manager IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security IOT Broker IOT Context Broker ProxyFilter Security VM on Cloud HA, DRS, [FT] Balancer IOTDSES KBSSM IOTOBSFs GIS, GeoServer GIS, GeoServer Services CKAN DataGate Routing Server Chat Manager R Studio Server ETLSDK Server Engager Server ……. Server
  • 132. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F: FullPlatform) For Huge scale Smart City with a Huge number of smart applications and processes on cloud (thousands), any number of IOT brokers Snap4City (C), November 2019 138 • MCLSCount • NCLSCount Container[Cluster] – Marathon, Mesos Cluster – Containers models • IOT App • Web Scraping • Data Analytics • ETL • Python • .. – DISCES-EM • Elastic management of containers – Monitoring App For Managing Beyond E:ExtendedCity Configuration + A scalable huge number of Containers including different kind of processes: IOT, ETL, data analytics, scraping + A scalable Data Shadow for IOT data management Optionally: • Data Analytics in R Studio, Tensor Flow (NVIDIA) • Hbase / Phoenix Big Data Store • Data Gate CKAN for Open Data • Chat Manager on Dashboard • Routing Solutions
  • 133. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F+) LivingLab Configuration Snap4City (C), November 2019 139 TOP
  • 134. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as Full Platform plus Living lab Support Snap4City (C), November 2019 140 Snap4City MAIN Dashboard Builder, Data Inspector, User Stats, ExternalSrv, Res Manager, MyKPI, MyPOI, Synoptic, IOT Device Manager, Micro Applications, Authorization/ Authentication, Roles IOTBrokers, secure IOTOBSFs GISGeoServer Heatmap Manager, WFS, WMS KBSSM KB/ServiceMap, SmartCity API, WFS, ServiceMap3D, LOG/FLINT, SuperServiceMap, Virtuoso Services CKAN DataGate Routing Server Chat Manager R Studio Server ETLSDK Server Engager Server ……. Server Mobile Apps, Dashboards Container Cluster MCLSCont, NCLSCont Marathon, Mesos, Zookeeper, Disces-EM, Executing in Container: IOT App, Web Scraping, R-Studio, Python, Java, ETL, ... Data Shadow Cluster IOTDSES NIFI, Elastic Search, DevDash, Amma [DataCluster] ETL-VM Phoenix HBASE, HDFS DISCES, ETL support IOTBrokers, secure GIS IOT Networks Social Media CRM ….. Living Lab Portal Drupal support CRMLegacy Platforms
  • 135. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as Full Platform plus Living lab Support Snap4City (C), November 2019 141 • Living Lab Portal – CRM Drupal – LDAP Snap4City – Blogs, Articles, comments, etc. – Multiple Organization – Multiple Groups – SSO with Snap4City tools – Automated Registration management – SOLR indexing on content – Monitoring activity – Reporting and statistics For Managing Beyond F:FullPlatform Configuration + Providing Living Lab Support, a full CRM for stakeholder, co-working, collaborative work, discussion environment, chart, forum, etc. Strongly suggested: • Data Analytics in R Studio, Tensor Flow (NVIDIA) • Hbase / Phoenix Big Data Store • Data Gate CKAN for Open Data • Chat Manager on Dashboard • ETL SDK servers • User Engagement Server • Routing Solutions
  • 136. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org THE VIEW OF THE ADMINISTRATOR Snap4City (C), November 2019 142 TOP
  • 137. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Knowledge base Semantic reasoners Data Sources PULL Data Data Sources, Brokers, External Services Data Driven, Real Time Data Ingestion, aggregation, regularization, reconcile ETL, NIFI, DISCES Big Data Cluster HDFS, Hbase, Phoenix Search and Query, Smart City API Facet, semantic searchIndexing and aggregating Elastic search Data Analytics R, Tensor Flow, Python, MapReduce, … Visual Analytics Special dash/ tools Rendering Acting, Widgets, MicroApps User interface, Interactive Dashboard, Drill down, maps, heatmaps Inform, announce, Act!, warning, alarms, What-IF, .. IOT Applications Node-RED + Snap4City MicroSrvs Snap4City as a Lambda Architecture
  • 138. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Management by Organization • Organizations may have – name, ID, GPS center, a number of Groups on Snap4City.org – users of different kinds and may impose early bounds on the resourced used by users (IOT Dev, IOT App, Dash) – on cloud users kinds up to level of Tool Administrator • ToolAdmin users (requested by Organizations) may – control processes, consumption of resources, healthiness, etc. – manage tools exploited in your configuration • 24H/7D Help Desk and Assistance Snap4City (C), November 2019 144
  • 139. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Help Desk and SLA • https://www.snap4city.org/drupal/contact • Bug Reporting – https://docs.google.com/forms/d/e/1FAIpQLSfD QtKqgLIIyycNXiazeYEh1SsRG1YL8Ze4ThD8nZoA5 jsoXw/viewform • For Service Level Agreement see: – Service Level Agreement • Help Desk and Contact: – https://www.snap4city.org/drupal/node/3 • Availability rates: – https://www.snap4city.org/388 Snap4City (C), November 2019 145
  • 140. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring Resource Consumption and Traffic Snap4City (C), November 2019 149 TOP
  • 141. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 150
  • 142. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Monitoring Infrastructure Status Snap4City (C), November 2019 151 http://www.disit.org/dashboardSmartCity/view/index.php?i ddasboard=MTQ4 http://www.disit.org/dashboardSmartCity/view/index.ph p?iddasboard=MTkw http://www.disit.org/dashboardSmartCity /view/index.php?iddasboard=MjQ5
  • 143. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 152
  • 144. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Computational Capabilities of Snap4City • Managing: – Periodic Processes → ETL, IOT App (Node-RED) – Asynchronous processes, data driven, real time → Node-RED, IOT Applications • Scalability – Horizontal: Increasing processes performing activities, demand on new processes for new users, for new applications, for new IOT applications: VM, Hosts, clusters, Storage SAN – Vertical: Increasing resources on processes: CPU, MEM, Storage, Network Snap4City (C), November 2019 153
  • 145. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DevDash Case Study (2) • Detect potential anomalies or disfunctions by inspecting the DevDash tool time trend Snap4City (C), November 2019 154
  • 146. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Managing and Monitoring Data-Traffic in the BackOffice Snap4City (C), November 2019 155 TOP
  • 147. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org AMMA (1) Monitor data traffic flows among IoT devices, services, applications etc. and detect potential anomalies Snap4City (C), November 2019 156 Unexpected behaviors can be revealed by inspecting the data flow time trend: a) detecting peaks or valleys in the trend b) drill-down on data to identify single/more malfunctioning devices and/or services quantitatively monitoring data/message traffic and flows
  • 148. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org AMMA (2) Make drill-down activities on data related to a single Process ID and check for unexpected behavior in the Time Trend panel: c) Filtering data by the Process ID (e.g. for example those related to a SmartWaste container) d) Detect a peak with more data traffic than expected during its scheduled activity, by properly filtering on time, the single data portion corresponding to the unexpected data flow can be viewed e) Locate on map the single involved device or service Snap4City (C), November 2019 157
  • 149. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org AMMA by Elastic Search + Kibana Snap4City (C), November 2019 158
  • 150. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org DevDash by Elastic Search + Kibana Snap4City (C), November 2019 159
  • 151. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Auditing Activities Snap4City (C), November 2019 160 TOP
  • 152. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 161
  • 153. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Managing Back Office Processes via Containers Snap4City (C), November 2019 162 TOP
  • 154. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Hosts with VM on Cloud HA, DRS, FT Snap4City (C), November 2019 Event Log Zeppelin Phoenix HDFS, Hbase debian Event Log Zeppelin Phoenix HDFS, Hbase debian Event Log Zeppelin Phoenix HDFS, Hbase debian Event Log Zeppelin Phoenix HDFS, Hbase debian Hosts/VM on Cloud in HA, elastic computing, .. . . . …… …….. ………. . containersDistribSrv. slave slave slave slave Pool of Containers NodeRed, Java Penthao Kettle, Java, Karma …….. R-Stat, R-Parallel Hadoop/Spark/SparkML Python, C++, C, java ………….. Master Master Master Masters in HA: Marathon/Mesos, Zookeeper, …. VmWare VSphere Cnt. Scheduling Cloud Elastic Management Cnt. Scheduling Cloud Elastic Management DISCES + Planner + Elastic Management Containers Control API Cloud Management System & Resource Monitoring Cloud Management System & Resource Monitoring Resource Monitoring & Cloud Resource Management Monitoring API VM Control API 163
  • 155. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Container Snap4City (C), November 2019 164
  • 156. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Elastic Scaling: allocating / deallocating • Allocation/ deallocation, Rebalancing vs compacting – Vertical of resources: Docker and/or VM: CPU, Mem • NodeJS multi-flow for each Docker, the user request data flows and IOT App, Snap4City allocates them dynamically on demand and perform workload optimization • VM: management of Mem, CPU; transparent and automatic in DRS VMware – Horizontal of resources of Dockers and/or VM and/or [Host]: • Docker: addition of containers, migrations/moving, balancing (per moving) of IOT App • VM: on/off • Monitoring resources: – VM via VMware API, Docker via Marathon and Mesos APIs • Algorithm in Python for scaling, actions via APIs: VMware, Marathon,.. Snap4City (C), November 2019 165
  • 157. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 166
  • 158. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Snap4City (C), November 2019 167
  • 159. Snap4City (C), November 2019 168 Monitoring on Cloud
  • 160. DISIT Lab, Distributed Data Intelligence and Technologies Distributed Systems and Internet Technologies Department of Information Engineering (DINFO) http://www.disit.dinfo.unifi.it http://www.disit.org Distributed SCE Scheduler Snap4City (C), November 2019 169
  • 162. Snap4City managed to provide a maximum of information, flows, in depth analysis with the data provided. There is no other platform that collects all city actors together. The City officials and ICT officials were impressed with the performance of the Platform when loading the heavy, “resource-” demanding applications and dashboard. The data handling throughout the Platform is considered as one of the strong points in the Platform and of an extremely sophisticated level. The technical level of the Platform and its strong points such as the way real-time data is used, the algorithms, data clean-up possibilities of the Platform, presented data and information is state-of-the-art and impressive. Snap4City (C), November 2019 171
  • 163. Snap4City (C), November 2019 172 Acknowledgements • Thanks to the European Commission for founding. All slides reporting logo of Snap4City https://www.snap4city.org of Select4Cities H2020 are representing tools and research founded by European Commission for the Select4Cities project. Select4Cities has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n° 688196) • TRAFAIR is a CEF project. All slides reporting logo of TRAFAIR project are representing tools and research founded by the EC on CEF programme http://trafair.eu/ • Thanks to the European Commission for founding. All slides reporting logo of REPLICATE H2020 are representing tools and research founded by European Commission for the REPLICATE project. REPLICATE has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n° 691735). • Thanks to the European Commission for founding. All slides reporting logo of RESOLUTE H2020 are representing tools and research founded by European Commission for the RESOLUTE project. RESOLUTE has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation Programme (grant agreement n° 653460). • Thanks to the MIUR for co-founding and to the University of Florence and companies involved. All slides reporting logo of Sii- Mobility are representing tools and research founded by MIUR for the Sii-Mobility SCN MIUR project. • Km4City is an open technology and research line of DISIT Lab exploited by a number of projects. Some of the innovative solutions and research issues developed into projects are also compliant and contributing to the Km4City approach and thus are released as open sources and are interoperable, scalable, modular, standard compliant, etc. INEA CEF-TELECOM Project funded by European Union
  • 164. Roadmap - Twitter Vigilance - Social Media Analytics, Sentiment Analysis - User engagement - Bike Sharing - Data Analytics ++ - Social Predictions - OBD2 - Sardinia Region Smart City Strategies and plan GHOST SIR (2016-19) - Smart Energy - Sustainable Mobility - Control Room - Dashboard - Resilience Decision Support - Smart First Aid - User Behaviour Analysis, predictions - Risk Analysis - Origin-Destination and trajectories - Traffic Reconstruction - Offer Analysis - OBU, smart devices - Weather Forecast - Real Time Wi-Fi - Entertainment - Events - LOD - Tuscany Road Graph - Mobility - culture, tourism - Events - Parking - Services - Linked open graph 2013 2014 2017 2018 2015 2016 Km4City 1.4 - Infomobility - Mobile App - Routing - Multimodality Km4City Ontology 1.1 Km4City 1.6.2 - IOT/IOE, IOT App - Living Lab - Maker Support - IOT Edge - Smart City IOT - GDPR - Privacy & Security CEF DISIT lab roadmap vs model and tools’ usage Km4City 1.6.6 5G tech Energy Industry 4.0 Synoptics …2022 IOT/IOE SII-MOBILITY SCN (2016-19) - Smart Waste (2017-20) (2015-18) H2020 (2016-21) H2020 (2018-21) - Mobility Demand / Offer Analytics and Strategy (2018-20) Km4City 1.6.4 H2020 (2017-19) GREEN IMPACT - Industry 4.0 - Critical Plant - Monitoring POR FESR 2014-2020 Km4City 1.5 2019 173 - Traffic and Mobility Impact on Pollution - NOX predictions(2018-21) 2020
  • 165. Market Solutions Snap4City (C), November 2019 174 IOTDiscoveryAbstraction Authentication,Authorization Securityend-2-end,secureon IOTandDashboards OpenHWandOpenSW IntegratedCommunity management DataTypes:IOTDevices,IOT App,Dashboard,Data DataType:Publish/share, Delegation,Consentandchange DataType:Downloadand Delete AuditingonDataTypeAccess OpenSourceend-to-end ScalabilityIOT VisualProgrammingend-to-end applications AdvancedSmartCityAPI, MicroServices MultiDomainSemantic Platform StandardbasedModulesand IOT,OpenDevices ResourceSharing DataAnalyticsintegrated DashboardH24/7,protected connection Multi-protocolonIOT G G G G G Snap4City Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y KAA [53] Y Y Y Y Y Y N Y Y Y Y N Y N (Y) N N Y Y Thingsboard [55] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT,coap, http IOT eclipse.org [56] N N N (Y) N Y N N N Y Y N N N Y N N N Y IOT IGNITE [57] N Y N Y N Y N Y Y Y Y Y N N N N N Y MQTT FIWARE [47] N Y N Y N N N Y N Y (Y) (N) Y N Y N N Y Y ARM mbed IoT [48] Y Y Y Y Y N (N) N Y Y Y N N N Y N N Y Limited Airvantage [51] Y Y Y Y N Y N Y Y Y Y N N N N N N Y MQTT, HTTP AWS [43] Y Y Y Y N Y (N) Y Y N Y N N N Y Y (Y) Y Limited Azure IOT [44] Y Y Y Y Y Y Y Y Y N Y N N N Y Y (Y) Y Limited PTC ThingWorkx [59] N Y Y Y Y Y N N Y N Y Y N N Y N N Y Y Bosch IoT Suite [58] Y Y Y Y Y (Y) (N) Y Y N Y Y Y N Y N Y Y Y CISCO Jasper [55] Y Y Y Y N (Y) (N) N Y N Y N N N N -- (Y) Y N Siemens MindSphere [60] Y Y Y (Y) N Y (N) Y Y N Y Y N N Y N Y Y Y Carriots [54] Y Y Y (Y) N Y N N Y N Y N N N -- N N Y MQTT Google IOT [45] Y Y Y Y Y Y N Y Y N Y N N N N N (Y) (Y) MQTT, HTTP Homekit Apple [50] Y Y Y Y N Y N N Y N (Y) N N N N Y N Y Limited Smarthing Samsung [52] Y Y Y Y Y Y (Y) Y Y N (Y) N N N N N N Y Limited
  • 166. Requirements vs Modules Snap4City (C), November 2019 175 Requirement.description SecurityandPrivacyManagement Ownership&Delegation PersonalData IoIContextbrokers IoTDirectory IoTFirewall IoTApps,andmanagement DashboardBui8lderand management MicroSerivces SmartCityAPI IoTDevices IoTEdge,IoTApponpremise DataShadowandStorage KnowledgeBaseKm4CIty Snap4CityPlatformSupportLiving Lab IV.A IV.D IV.C VI IV.C VI.A IV.C IV.C IV.A V.B IV.C V.C IV.B IV.C V.A V.C IV.B V.B IV.B IV.C IV.B IV.A IV.B R1. IOT Brokers X X X R2. IOT Discovery Abstraction X X X R3. Authentication, Authorization X X X X R4. Inform User of Security Level X R5. Developing Secure Applications X X X X X X R6. Secure Communications X X R7. Open HW and Open SW X X X X X R8. Signed Content Vs Data Types X X X R9. Managing IoT Data Types X X X X X X X X X X X X X X R10. Managing Ownership and Delegation X X X X R11. Support Roles, Org. and Groups X X X X X X X R12. Encrypted Personal Data X X X R13. User Profile Management, forgotten X X R14. Auditing Data Types X X X X X X X X X X X X X X X R15. Data Breach Detection X X X X X X X X X X X X X X X R16. Accounting X X X X X X X X X X X X X X X R17. Data Protection by Design X X X X X X X X X X X X X X X R18. Evidence of Level of Security X X X
  • 167. GDPR vs Requirements GDPR Compliance Verification Feature Verif. Reqs. Signed consent UI R8 User profile management and control UI R13 Data Type private as default UI R8 Rights to access per element UI R9 Rights to transfer per element UI R10 Rights to erase per element and total UI R13 Rights to revoke/change per Data Type UI R10 An interface for Right management for Data Type UI R9 Clear Terms of Use and Privacy Policy UI -- Auditing Tools for Data Type UI R14 Publish as Anonymous UI R9 Encrypt personal users’ data Code R12 Secure Authentication and Authorization Code R3 Data protection by Design Code R17 Secure connection Code R6 Security Control, data breach control, anonymization, etc. PEN Test R15, R16, R18 Snap4City (C), November 2019 176
  • 168. Snap4City (C), November 2019 177 Be smart in a SNAP! TOP