Distributed Development of IoT Middleware with Microservices

© 2020 – The symbIoTe Consortium
Distributed Development of
IoT Middleware with Microservices
Mario Kušek, University of Zagreb, FER
16 Jan 2020, FER, Zagreb
Grant Agreement No 688156

© 2020 – The symbIoTe Consortium2
• Introduction to Internet of Things (IoT)
• Project: symbIoTe
– Goal
– Results
– Architecture/Design/Implementation
– Process and Experiences
• Conclusion
Outline

Who am I?
@MarioKusek
mario.kusek@fer.hr
University of Zagreb
Faculty of Electrical Engineering and Computing
Department of Telecommunications
IoT Laboratory

• What is a ”thing”?
– Object from physical world (physical object with
build in sensors and/or actuators) or virtual object
– Internet Connected Object (ICO)
• Has unique identifier and is connected to the Internet
• Communicates and generate data (reading from
environment)
• Can receive data/commands from network
• Can execute commands – actuate (electrical or
mechanical)
• Can receive data from other ICO, process them and
send for processing to cloud
Internet of Things - IoT

• IoT platforms integrate „things” and continuously
acquire data
– Large distributed system
– Processing large amount of data (often in real time)
– Integrates and saves data from different sources
– For application developers offer:
• Searching for things (sensors/actuators)
• Access to data
• There are more then 400 platforms
– Mostly specialised for one area
How to integrate „things” and provide user applications?

Architecture of IoT System
User
IoT
Platform
Gateway
IoT
Application
Smart spaceCloudUser

• Can not easily create cross domain applications!
• Interoperability:
– Data structures are different
– Different measurement units
– Vertical domain specific platforms
• Project: symbIoTe
– Symbiosis of smart objects across IoT environments
– H2020 project: 2016-2018 (3 years)
– 15 EU partners (universities, institutes, SMEs)
Some Challenges

• Middleware that enables IoT interoperability
– SW adaptors for existing IoT platforms
– Client libraries for applications
• Holistic approach
– from Cloud to gateway and smart device
• Customized solution
– 4 “modes” of operation ® compliance levels
• Market positioning
– B2B: middleware for platform owners, gateway/device
manufacturers
– B2C: innovative cross-platform apps and Domain
Enablers
What is symbIoTe?

The symbIoTe approach
Core Services
IoT
Platform A
IoT
Platform B
AdaptorAdaptor
Domain
Enabler
Application
Adaptor
IoT
Platform A
IoT
Platform C
Adaptor
Adaptor
IoT
Platform D
Adaptor
IoT
Platform B
Adaptor
Application
Agent
IoT
Gateway A
IoT
Gateway B Smart
Device
Smart Space Gateway
Agent Agent
L1
L2
L3L4
roaming
Smart Space A
Core Services
Smart Device
Smart Space B
Smart Space GatewaySmart Space Gateway
Smart Device

The symbIoTe ecosystem (I)
1
4
2
2
1
2
1
1
1
symbIoTe
consortium
members
3 OC1 (L1)
partners
2
1
2 OC2 (L1)
partners
1
1
4 OC2 (L2)
partners
1
1
1
1
4 OC2 (L3/4)
partners
1
2
1
3 OC2 (Apps)
partners
1
1
2 OC2 (Trials)
partners
1
1
15
1
Evaluate B2B aspects,
increase use of
middleware, identify
sustainability paths

The symbIoTe ecosystem (II)
1
1
1
1
2
6
symbIoTe
L1
platforms
3 OC1 (L1)
platforms
2
1
2 OC2 (L1)
platforms
1
1

General Overview – Trials
Smart Residence (L1/L2/L3)
Smart Home: Pisa (IT)
AAL: Vienna (AT)
Indoor Air Quality: Barcelona (ES)
EduCampus (L2)
Karlsruhe (D)
Smart Stadium (L1)
Barcelona (ES)
Smart Mobility and Ecological Routing (L1/L2)
Zagreb (HR), Vienna (AT), Porto (PR)
Smart Yachting (L1/L3/L4)
Smart Mooring: Viareggio (IT)
Automated Supply Chain: Viareggio & Marina Cala De’ Medici (IT)

SMEUR in action!

Smart Mobility & Ecological Routing
Platforms Involved (CL) 3 (L1)
Gateways / devices 56 wearable + 20 stationary devices
Apps 2
Enabler 1
Test Sites 3 (Zagreb, Vienna, Porto)
No. Of Users 60
Use Case Numbers

• Goals:
– Provide users with ecological routes to their destinations
– Integrate environmental data from different cities and IoT
platforms (L1)
– Create enabler to provide applications easy access to
interpolated pollution data and routing information
Overview
symbIoTe
SM&ER

• 3 IoT platforms involved (OpenIoT, openUWEDAT, MoBaaS)
– Integrate measurements from
stationary and mobile air quality sensors
– Level 1 integration
• symbIoTe enabler for interpolation, route calculation &
POI search
• Two mobile apps for end-users
– route calculation (Green Routing App)
– mobile data gathering (CUPUS)
Technical Concept
symbIoTe
SM&ER

• 20 prototype devices built
by UNIZG-FER
(used in Zagreb)
• 20 sensor boxes produced by
DunavNet in Vienna
6 by OC2 trial partner
Sindikat Biciklista in Zagreb
• 20 fixed stations from
Ubiwhere plus 10 (fixed
and wearable) sensors
from MONITAR for Porto
Air Quality Sensors
Powered by
OpenIoT
Powered by
MoBaaS

Fixed Air Quality Stations (Vienna &
Zagreb)
powered by
openUwedat

symbIoTe SMEUR Green Routing App
symbIoTe-enabled App intended to be used by a large number of users
Green Routes are calculated for pedestrians and cyclist
POI filter “bike_rental” Two pedestrian routesInitial screen with POIs
from OpenStreetMap
One route (cycling)
/ symbIoTe SMEUR

CUPUS Crowdsensing App
“Pure” OpenIoT App!
Used by a small number of volunteers carrying wearable air quality sensors in
Vienna and Zagreb
/ CUPUS Crowdsensing
Current sensor readings Crowd sensor shown on
the map
Overview on active
sensors
Subscribe to crowd
sensors

• Trial Preparation
– System has been setup for Zagreb, Vienna and Porto
– Validation of apps has been carried out by students in Zagreb and
Vienna
– Testcards, questioners and information material have been prepared
• Trial Execution
– Vienna: 26 users (AIT+UNIVIE) have been tested the sensors and the
CUPUS routing app during August and September 2018
– Zagreb: further evaluation from September till November 2018 in
collaboration with Sindikat Biciklista, in total 40 active users were
using CUPUS Crowdsensing App, 53 users
downloaded the CUPUS Crowdsensing App
and 189 users downloaded the routing app
– Porto: Trials involved 20 users plus 10 users
from OC2 extension winners MONITAR, and
ran through November and December
Trial Preparation & Execution

Zagreb Trial: Promotional Events
Promo event at UNIZG-
FER Unska 3 on Oct
11, 2018
target audience:
students
UNIZG-FER Open Day on
Nov 24, 2018
target audience: general
public
„Brave New World”
at HUB385 on Oct
20, 2018
target audience:
general public

Collected measurements (Fixed Stations)
No.ofmeasurements
Month
0
5000
10000
15000
20000
25000
30000
35000
40000
Jan Feb Mrz Apr Mai Jun Jul Aug Sep Okt Nov Dez
Velika Gorica
Vienna
Zagreb
Overall 1.628.694 measurements
(including HR)

Collected measurements (Zagreb)

Collected measurements (Porto)

Collected measurements (Vienna)
0
1000
2000
3000
4000
5000
6000
7000
8000
7 8 9 11
co
humidity
no2
noise
pressure
temperature
No.ofmeasurements
Month

• User Satisfaction
– Overall end-user satisfaction (SUS > 68)
• Technical Parameters
– Route response time < 2 seconds
– PoI response time < 2 seconds
– RAP plug-in response time < 1 second
– Frequency of requests sent to Core services > 6/hour
– Frequency of requests to platform RAPs > 30/hour /per platform
SMEUR – Target KPIs

High-level Architecture

Level 1 Highlights
CoreServices
Platform
Cloud
Resource
Access
Proxy (RAP)
Registration
Handler
(RH)
Authentication &
Authorization
Manager (AAM)
Monitoring
Interworking Interface
Registry
Search
Engine
Core Authentication &
Authorization
Manager (Core AAM)
Semantic
Manager
Core Resource
Monitor
Core Resource
Access
Monitor
Administration
Core Interface
Cloud-Core Interface
Offerings:
• IoT Portal and semantic
search engine
• Unified access to IoT resources on the platform side
(RESTful OData-like interface)
• Semantic interoperability (platforms can define and use their own
information model which extends the symbIoTe Core Information
Model)
• Flexible access control (attribute-based) to resources and search
results
• Domain Enablers
Main objective: simplify
development of IoT
applications (cross-domain
and cross-platform)
RAP PluginMyPlatform

Semantic Interoperability
Core Services
2. search
3. resource access
(OData)
1. register
selected
resources
(JSON/RDF)
Platform A
Interworking
Interface
Platform B
Interworking
Interface
Application
3. resource
access
(OData)
CoreServicesPlatformCloud
Core Information Model (CIM) with
Extensions
• supports symbIoTs’s CIM defining
the core concepts (Sensors,
Actuator, Services, and Location),
re-uses concepts defined by the
Semantic Sensor Network (SSN)
ontology, Sensor-Observation-
Sample-Actuator (SOSA) ontology,
Actuation-Actuator-Effect ontology
pattern and the SensorThings API
• predefined Best Practice Information
Model (BIM)
• validates Platform-specific Info
Models (PIM) extending CIM
symbIoTe’s Interoperability
Components
• RESTful OData-like interface for
secure access to platform resources
• use JSON if BIM is sufficient for
resource description
• otherwise define PIM using RDF
Centralized search
Decentralized access to resources; their
management and access control remains on
the platform side

L1 Security Aspects
Core Services
get GUEST
token
Use PA HOME token and the
right attributes to get
FOREIGN token for PB
Platform A
Interworking
Interface
Platform B
Interworking
Interface
Application
register to
PA and get
HOME token
Attribute-Based Access Control (ABAC)
• PKI certificates trust chaining architecture
• Actor’s attributes are distributed in standardized trusted data
structures: JSON Web Tokens (JWT)
• Administration of platforms
• Platform and user authentication
• Management of symbIoTe
components and platforms’ CA
X.509 certificates
• Management of local actor
attributes and attribute mapping
functions
CoreServices
Resources protected using the
Attribute-Based Access Control
(ABAC) paradigm
• Access Policy assigned to each
resource
• Management of local certificates
• Management of local actor
attributes and attribute mapping
functionsPlatformCloud

Domain-Specific Enabler (L1)
Core Services
Platform Cloud Platform Cloud
Mobile/Web App
Enabler
service
access
register my services
search for resources
resource
access
search services
and resources
Enabler
Domain-specific
services on top of
interoperable platforms
• added value on top of
resources exposed by
a set of platforms, e.g.
data analytics
• an application sees it
as another (virtual)
platform
• acts as an application
to Core Services and
Platform Cloud

SMEUR Architecture
OpenIoT openUwedat MoBaaS
Interpolator
Air Quality Readings
Green Route
Controller
Interpolated Values:
• Street Id
• Measurement
• AQI
AIT Routing
Engine
(external)
MoBaaS
Routing
Engine
SMEUR Enabler
Platform Proxy
Interpolated Values
PoI Service
OpenStreet
Map
(external)
Mobile App
symbIoTe
SMEUR
Domain
Specific
Interface
Mobile App
CUPUS
Crowdsensing

• Development
– Agile process (iterations) – planned 4 week releases
– Weekly meetings of all developers (video conference)
– One or two weeks for internal component releases
– Programming language is Java
– Unit testing: >70% code coverage
• Tools:
– Attlasian Confluence document management and
collaboration
– Attlasian Jira as the feature planning tool
– github.com as code repository tool
– Git as versioning and code revisioning tool
– Travis CI as Continuous Integration server
Project organisation

Software Releases (1/3)
Feb 2017
March
2017
April 2017 May 2017 June 2017 July 2017
August
2017
September
2017
L1: support for actuators, basic security-related
features for access control (token validation),
platform-side monitoring
L1: PIM definition & verification, ranking and filtering of search
results; new security-related features: challenge-response
procedure, access policies supported and checked on platform;
L2: creating platform federations
L1: sensors exposed over OData-like REST-based
interface on the platform side, basic registration and
search features in the Core
R1
R2
v1.0.0
(R3)
L1: bug fix
for memory
leakage
R2.1
Open source from day 1
https://github.com/symbiote-h2020

Nov 2017 Dec 2017 Jan 2018 Feb 2018
March
2018
April 2018 May 2018 June 2018
L1: token caching, composite access policies,
improved support for PIM in RAP, improved search
capabilities, improved software and project
documentation as well as code coverage
Integrated release of the symbIoTe
middleware
L1: support for single-attribute access policies,
filtering of search results according to policies
specified by platforms.
v1.1.0
(R3.1)
v1.2.0
(R3.2)
v3.0.0
(R5)
First official
Release of
L2/L3/L4
v2.0.0
(R4)

symbIoTe Cloud dockerization
Bug fixing
July 2018 Aug 2018 Sept 2018 Oct 2018 Nov 2018 Dec 2018 Jan 2019 Feb 2019
v3.0.1
Review
v3.1.0
symbIoTe Core dockerization
PIM validation, semantics in Core
support for ngrok, custom
configuration for docker
RabbitMQ overflow problem

• Each component is one microservice
– Has a component owner
– One microservice (mostly implemented by one
organisation)
– One repository
• Frameworks:
– SpringBoot, Spring Cloud
– Communication between microservices:
RabbitMQ
Microservices – Decisions

• Spring Cloud Config – configuration
• Zipkin – distributed tracing system
• Spring Cloud Gateway – did not exist then
– We used NGINX
• Spring Cloud Netflix:
– Eureka – service registration
– Ribbon – REST client that uses Eureka
– Feign – declarative HTTP client
• For accessing core services
Spring Cloud

Technologies
microservices
microservices
Eureka
CoreServicesPlatformCloud
SmartSpace
RPi

symbIoTe Middleware
https://middleware.symbiote-h2020.eu/

Implementation Process
Code statistics
16 Components + 3 Libraries + 6 Enabler Components
Over 133,000 lines of code
Code Coverage: 59%
Documentation:
symbIoTe Core and Cloud installation guides available in GitHub
Each component has its own javadoc hosted in GitHub
e.g. https://symbiote-h2020.github.io/Registry/doxygen
Issues reported in Jira
Features in JIRA
DONE: 634
In Progress: 11
To Do: 146

• Microservices were not divided by domains
• Frequent changing of messages
– Hard integration testing, lots of communication
between teams
• RabbitMQ
– Most of communication is request/response
• Design not appropriate for messaging
– No message (API) versioning
– Late arrival of Spring Cloud Contract (v1.0.0 Sep
2016)
• Not so good support for messages (documentation and
examples)
Problems

• Put all libraries in Jitpack (https://jitpack.io)
• Lots of microservices need to implement the
same data objects:
– Put data objects in library: SymbIoTeLibraries
• Problems with versions of library:
– Start using semantic versioning (https://semver.org)
• All components need to contact AAM
(Authentication Authorization Manager)
component
– Created SymbIoTeSecurity library which has
dependency to SymbIoTeLibraries
Problems/decisions

• Each IoT platform needs to deploy cloud
components
– No automatic deployment
• 3 stages/ways of deployment:
– 1. From sources:
• Support web page (github wiki)
• Problems:
– Complex à automatization
– Slow à automatization
– Hard to configure à configuration generator (web page
similar to https://start.spring.io)
Deployment (1)

• 2. From jars:
– Created a script for downloading, configuring,
starting and stopping
– Problems:
• Microservices have startup dependencies
– Solved by putting small class in main method to wait for some
service (host:port)
• If it doesn’t work first time you need to dive into details
– Similar to SpringBoot
• Bash script - problem on windows if someone just want
to try it
Deployment (2)

• 3. From dockers
– Easy startup
– Starting with docker-compose or docker swarm
– Config git repo is in docker image
• Problem of custom configuration
– Putting config in volume
– Different documentation for using docker directly
(linux) or in docker machine
• Problems with port mapping
Deployment (3)

• For testing, hackathons you run components
on machine that is in local network
• Cloud components need to have public IP,
DNS, and certificate (for https)
– Workaround is ngrok tunnelling
• Different NGINX configuration
Cloud components on dev machine

Memory Consumption – Problem
• 1.5GB on startup
• 2-3GB when working
• Solution:
– Java 11: new GC, String internals, netty, …
– Problem: late release of SpringBoot 2.1 (Oct 2018)
and Spring Cloud (Greenwich)(Jan 2019)

• Distributed teams require lots of
communication
• Be careful in choosing technologies
• Divide features to microservices that are
independent and scalable (up/down)
• Versioning of API is important
– Start with monolith and extract microservices
• Dedicate time for building/testing automation
– Do it in the beginning of the project and maintain
Conclusions

© 2020 – The symbIoTe Consortium
www.symbiote-h2020.eu
middleware.symbiote-h2020.eu
info@symbiote-h2020.eu
@symbiote_h2020
H2020 symbIoTe
github.com/symbiote-h2020
Member of
Thank you!
Questions?
@MarioKusek
mario.kusek@fer.hr

Distributed Development of IoT Middleware with Microservices

More Related Content

Similar to Distributed Development of IoT Middleware with Microservices

Recently uploaded

Distributed Development of IoT Middleware with Microservices