A brief discussion on various open standards and open APIs and their potential use and weaknesses in an IoT implementation

1. 1
Copyright Solace
Confidential
Devising your Data
Movement Strategy for
IoT
Dave McAllister
September, 2016

2. 2
Copyright Solace
Confidential
Data Movement Between Elements of IoT
Top level system considerations
Architectures for different types of
data movement
Which standards to use;
where and why

3. 3
Copyright Solace
Confidential
Relevant System Requirements
Scale of Devices
Capabilities of
Devices
Message Exchange Patterns
Messaging QoS
Traffic
Characteristics
Security
IT Application
Architecture &
Technologies

4. 4
Copyright Solace
Confidential
Problem
Statement
Firewall
Firewall
Load
Balancer
? ? ? ?

5. 5
Copyright Solace
Confidential

6. 6
Copyright Solace
Confidential

7. 7
Copyright Solace
Confidential
Message Bus

8. 8
Copyright Solace
Confidential
IoT Interlude
http://imgs.xkcd.com/comics/driving.png

9. 9
Copyright Solace
Confidential
Firewall
Connection Tier
Load
Balancer
EDGE
FOG
Typical IoT
Cloud
Deployment
Architecture
Enterprise Bus
Firewall
CORE

10. 10
Copyright Solace
Confidential
Firewall
Connection Tier
Load
Balancer
EDGE
FOG
System Considerations
• Connections, queues, subscription
• MEPs
• Device initiated pub/sub
• Device initiated request/reply
• Cloud app initiated pub/sub
• Cloud app initiated request/reply
• QoS – MQTT QoS0, QoS1
• Single point of contact for apps
Enterprise Bus
Firewall
CORE

11. 11
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
System Considerations
Load Balancer
o Connections, queues, subscrptns
o MEPs
‐ Device-initiated pub/sub
‐ Device-initiated request/reply
‐ Cloud app-initiated pub/sub
‐ Cloud app-initiated request/reply
o QoS – MQTT QoS0, QoS1
o Single point of contact for apps

12. 12
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Load Balancer
System Considerations
o Connections, queues, topics
o MEPs
‐ Device-initiated pub/sub
‐ Device-initiated request/reply
‐ Cloud app-initiated pub/sub
‐ Cloud app-initiated request/reply
o QoS – MQTT QoS0, QoS1
o Single point of contact for apps

13. 13
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Load Balancer
Challenges with
Stateless Connect Layer
o Device Queues
‐ Where is the queue for each device for
QoS1 subscriptions?
‐ How does Connection layer find its
queue/state? (among the millions….)
o App to device MEP
‐ How does Core layer find Connection layer
server for given MQTT device for QoS0?
‐ Where is the Q for a given device for
QoS1?
‐ Need massive subscription scalability
updated in real time

14. 15
Copyright Solace
Confidential
MQTT Paho
Proprietary
Proprietary/
JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I
Relevant
Protocols
& APIs

15. 16
Copyright Solace
Confidential
MQTT versus CoAP
MQTT CoAP
Many to many (but Pub-sub) One to one
Event based State based
No discovery (Pub-sub) Supports content negotiation
OASIS / ISO IETF
3 levels QoS 2 levels QoS

16. 17
confidential
o Device Constraints
o Message Exchange Patterns
o Messaging QoS
o Traffic Characteristics
o Security
Fog

17. 18
confidential
Fog
MQTT Paho
Proprietary
Proprietary/
JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I

18. 19
Copyright Solace
Confidential
Edge
o # Concurrent Connections
o “Always on” Connections?
o Burstiness
o Congestion / Prioritization
o Multi-protocol?
o QoS
o Security

19. 20
Copyright Solace
Confidential
MQTT Paho
Proprietary
Proprietary/
JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I
Edge

20. 21
Copyright Solace
Confidential
o Less scale, more
concentration
o Need enterprise features
o Enterprise grade, popular
APIs
o Support streaming &
microservices
Core

21. 22
Copyright Solace
Confidential
Core
MQTT Paho
Proprietary
Proprietary/
JMS/JCA
HTTP/REST In Runtime
AMQP 1.0 Qpid / JMS
CoAP Several OSS
P R O T O C O L A P I

22. 23
Copyright Solace
Confidential
IoT is all about the data
Capacity &
Availability Limits
Big Data RIVER
Shock Absorber
Network
Processing
Storage
Outages
Upgrades
Inconsistent
Aggregate
Input
Stream

23. 24
Copyright Solace
Confidential
Public
Clouds
Public
CloudAp
p
Ap
p
Ap
p
Big Data River
Big Data
Private Cloud
App App App
App App App
PaaS
IaaS
App App App
Public
Cloud
On Premise

24. 25
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Addressing Scalability
o Command & Control sending
message to a specific device

25. 26
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Addressing Scalability
o Command & Control sending
message to a specific device
o Device-initiated request/reply
o Among millions of devices
o Across 2 middleware tiers
o Need Integrated addressing
between Enterprise Bus &
Connection Tier at massive scale

26. 27
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Addressing Scalability
Load Balancer

27. 28
Copyright Solace
Confidential
Enterprise Bus
Connection Tier
Addressing Scalability
o Connection Tier Addressing
‐ MQTT Topics
o Enterprise Bus Addressing
‐ Other topics & queues
o Need to scale
‐ In an integrated manner
‐ Update in real-time
‐ Handling bursts of changes

28. 29
Copyright Solace
Confidential
Singapore Land Transport Authority – ERP2
http://www.straitstimes.com/singapore/transport/ncs-mhi-to-build-islandwide-satellite-based-erp-for-556m
Routing
Server
Payments
Analytics
0 0 0
• 200K Concurrent
Connections
• MQTT Termination
• Request/reply for
firmware upgrades
• Push Notifications
• Persistent Messaging
• Shock Absorber
• High Throughput
• Fanout
• Slow Consumer Handling
MQTT
JMS
MHI
OBU
Next-Gen Electronic
Road Pricing
o Connect 1.5M vehicles
(200K at a time) in real-time
for pay per use road tolls
o Suggest better routes, traffic
management – real time and
long term and value-added
services/analytics

29. 30
Copyright Solace
Confidential
Bus Monitoring
o Collect real-time location,
loading information and
more thousands of buses
o Optimize bus movement,
fleet management, avoid
bunching, forecast arrival
time and more
REST/MQTT
Regulator’s
Dashboard
Commuter’s
Apps and RIAs
Analytics DWH
ESB (if needed)
CEP
JMS
Legacy REST
Legacy
Gateway
REST/MQTT

30. 31
Copyright Solace
Confidential
Overall Architecture Requirements
o Scalability: connections, queues,
subscriptions
o “single bus” connectivity for both
apps & devices – solve the whole
problem!
‐ Unified addressing between cloud
apps & devices
‐ Single point of application connectivity
o Support necessary QoS
o Support all MEPs(?)
‐ Device-initiated pub/sub
‐ Device-initiated request/reply
‐ Cloud app-initiated pub/sub
‐ Cloud app-initiated request/reply
IoT Connectivity Concerns

31. 32
Copyright Solace
Confidential
At the Edge
o Scalability: connections, queues,
subscriptions
o Protocols: MQTT, REST, CoAP(?)
o Security:
‐ Authentication & Encryption
‐ Per-device authorization
‐ deviceID in topics
‐ /in/<devID>/<system>  ACL on
/in/<devID>/>
In the Core
o Single, simple point of contact to
talk to any device
o APIs & protocols to cloud apps
‐ JMS, AMQP, Node.js, REST
‐ App-to-device and app-to-app
o Message filtering
o Application horizontal scaling with
non-exclusive queues
o Integration with Hadoop
IoT Connectivity Concerns(cont)

32. 33
Copyright Solace
Confidential
Can’t predict coming evolution of IoT,
and changing infrastructure isn’t cheap.
Need flexible data movement platform

33. 34
Copyright Solace
Confidential
Strong Architecture
• Decoupling of concerns
• Scales to meet
needs of clients
• Right features
at each layer
Smart Use of Standards
• Right standards
• Features & exchange patterns
• Optimized for needs of users
• Security
• Supported by key products and
open source projects

34. 35
Copyright Solace
Confidential
Thanks!
Dwmcallister dave.mcallister@solace.com
Download the free VMR Community Edition at
http://dev.solace.com