Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape? Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Internet of Things (IoT) and Big Data
30.9.2016 – DOAG 2016 Big Data Days
Guido Schmutz

2. Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Internet of Things (IoT) and Big Data2

3. Agenda
1. Introduction
2. Towards an IoT Architecture
3. IoT Refererence Architecture
4. Summary
Internet of Things (IoT) and Big Data3

4. Introduction
Internet of Things (IoT) and Big Data4

5. Internet of Things (IoT) Wave
Internet of Things (IoT): Enabling
communication between devices,
people & processes to exchange
useful information & knowledge
that create value for humans
Term was first proposed by Kevin
Ashton in 1999
Source: The Economist
Source: Ericsson, June 2016
Internet of Things (IoT) and Big Data5

6. Reasons why IoT opportunity is occurring now ?
Affordable hardware
• Costs of actuators & sensors have been
cut in half over last 10 years
Smaller, more powerful hardware
• Form factors of hardware have shrunk to
millimeter or even nanometer levels
Ubiquitous & cheap mobility
• Cost for mobile devices, bandwidth and
data processing has declined over last
10 years
Availability of supporting tools
• Big data tools & cloud based infrastructure
have become widely available
Mass market awareness
• IoT has surpassed a critical tipping point
• Vision of a connected world has reached
such a followership that companies have
initiated IoT developments
• Commitment is irreversible
Internet of Things (IoT) and Big Data6

7. The Sensing-as-a-Service Model
Internet of Things (IoT) and Big Data7

8. Towards an IoT Architecture
Internet of Things (IoT) and Big Data15

9. Key Challenges for building an IoT application
1. Connect: How to collect data from intelligent devices?
• Abstract complexity associated with device connectivity
• Standardize integration of devices with enterprise
2. Analyze: How to analyze IoT data?
• Reduce noise and detect business event at real-time
• Enable historical big-data analysis
3. Integrate: How to integrate IoT data & events with enterprise infrastructure?
• Make enterprise processes IoT friendly
• Allow enterprise & mobile applications to control devices
Internet of Things (IoT) and Big Data18

10. Today) Existing Service-/API Architecture as a base
19
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
API Gateway
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST / SOAP
REST / SOAP
REST / SOAP
SOAP
Various
SQL
SOAP
REST
Service BusOracle Data Integrator
API Gateway
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data19 = one way = request/response

11. REST / SOAP
REST / SOAP
IoT 1a) Reuse exiting Service-/API-based Architecture
IoT Smart
Devices
20
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST / SOAP
REST
REST
JMS / REST
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
Service BusOracle Data Integrator
API Gateway
API Gateway
JMS
JMS
WeblogicJMS
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data20 = one way = request/response

12. IoT 1a) Challenges
Internet of Things (IoT) and Big Data21
• Do IoT devices contain enough resources to communicate directly over the internet
(HTTP or JMS) ?
• Should the device only collect data (sense) or is there also the way back necessary
(actuator) ?
• Can JMS be used from external devices (firewalls allow traffic over JMS) ?
• How many IoT devices are planned short and long term? How often do they send to
the backend? Are the JMS server as well as the ESB capable to deal with the
resulting message volume ?
• What are the operations on IoT messages / events, only simple transformations, filter
and routing operations?

13. REST / SOAP
REST / SOAP
IoT 1b) Reuse existing Service-/API-based Architecture
IoT Smart
Devices
22
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
API Gateway
Enterprise Service Bus (ESB) / Data Integration
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST
HTTP
REST
REST HTTP
JMS API Gateway
Service BusOracle Data Integrator
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMS
JMS
REST / SOAP
WeblogicJMS
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data22 = one way = request/response

14. REST / SOAP
REST / SOAP
IoT 2) Adding Event Hub and optional IoT Gateway
23
Mobile
Apps
D
B
Rich (Web)
Client Apps
D
B
ESB / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
REST
Kafka / MQTT / REST
REST
Kafka / MQTT / REST
MQTT
Kura
Event Hub
Kafka
Service Bus
Oracle Data Integrator
REST
REST
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMS
API Gateway
API Gateway
REST
Kafka
REST / SOAP
REST
REST
SOA Suite
BPM Suite
Business Activity
Monitoring
MQTT
Internet of Things (IoT) and Big Data23
MQTT
= one way = request/response

15. How to implement an Event Hub?
Apache Kafka to the rescue
Distributed publish-subscribe messaging system
Designed for processing of high-volume, real
time activity stream data (logs, metrics
collections, social media streams, …)
Topic Semantic
does not implement JMS standard
Initially developed at LinkedIn, now part of
Apache
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Internet of Things (IoT) and Big Data24

16. Oracle’s Service Bus as a consumer of Kafka
Service Bus 12c
Cloud
Apps
Business
Service
Cloud
Proxy
Service
Kafka
Cloud
API
Mobile
Apps Pipeline
Routing
Kafka
Sensor / IoT
Web Apps
Business
Service
REST
Business
Service
WSDL
Backend
Apps
REST
Backend
Apps
WSDL
Proxy
Service
Kafka
Pipeline
Routing
Database
DB CDC
Stream
Processing
Internet of Things (IoT) and Big Data25

17. IoT 2) Solutions & Challenges
Internet of Things (IoT) and Big Data26
Solutions
• Event Hub solves the potential scalability issue of JMS
• IoT Gateway makes sure that lightweight sensors can connect to the internet / send their
data
Challenges
• Where to do complex analytics on the events? Is it scalable?
• Can we really send all data down to backend? Network bandwidth?

18. REST / SOAP
REST / SOAP
IoT 3) Adding Stream Processing / Analytics in Backend
27
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
MQTT
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
REST
MQTT
REST
REST
Kafka
Kafka / MQTT / REST
Kafka /
MQTT /
REST
REST
Kafka
REST / SOAP
REST
REST
Kura
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data27 = one way = request/response
KafkaMQTT

19. IoT 3) Solutions & Challenges
Internet of Things (IoT) and Big Data28
Solutions
• Stream Processing handles complex analytics on events in a scalable manner before
sending events to ESB / backend systems
Challenges
• Can we really send all data down to backend? Network bandwidth?

20. Oracle’s Stream Analytics as consumer of Kafka/MQTT
Oracle Stream Analytics
Stream Analytics
Kafka
Kafka
Mobile Apps
Kafka
Sensor / IoT
Web Apps
Machine Data
DB CDC
Kafka
MQTT
Internet of Things (IoT) and Big Data29

21. REST / SOAP
REST / SOAP
IoT 4) Adding Industry 4.0 Data Sources (machine data)
30
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
MQTT
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
REST
MQTT
REST
REST
Kafka
Kafka / MQTT / REST
Kafka /
MQTT /
REST
REST
Kafka
REST / SOAP
REST
REST
Kura
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data30 = one way = request/response
I 4.0
Machine
DB CDC GoldenGate
Kafka / MQTT / REST
KafkaMQTT

22. GoldenGate Gateway
Oracle’s GoldenGate for Change Data Capture of
existing database
Internet of Things (IoT) and Big Data31
Machine Data
DB
Kafka
Oracle GoldenGate
Delivery
Capture
Pump
JMS
Machine Data
DB Oracle GoldenGate
Delivery
Capture
Pump HBase
HDFS

23. REST / SOAP
REST / SOAP
IoT 5) Adding Stream Processing / Analytics at Edge
32
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Stream Processing
ESP / CEP
DB
DB
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Stream Analytics
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
Kafka
JMS
JMSAPI GatewayAPI Gateway
Kafka /
MQTT /
REST
REST REST
Kafka
Kafka
REST
RESTREST
REST / SOAP
REST
MQTT
SOA Suite
BPM Suite
Business Activity
Monitoring
ESP/CEP
Edge Analytics
MQTT
Internet of Things (IoT) and Big Data32
Kafka / MQTT / REST
MQTT
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
Kafka / MQTT / REST
Kafka

24. IoT 5) Solutions & Challenges
Internet of Things (IoT) and Big Data33
Solutions
• Stream Processing at the edge / gateway allows to reduce the amount of messages send
to the backend (cloud / on premises) if necessary
Challenges
• What if we need the raw events persisted? Where do to that?

25. REST / SOAP
REST / SOAP
IoT 6) Adding Raw Data Storage and Batch Analytics
34
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMSAPI GatewayAPI Gateway
REST REST
Kafka
Kafka
SQL
REST
REST
REST / SOAP
Stream Processing
ESP/CEP
DB
DB
Big Data Processing
HDFS
Batch
Processing
DB
Kafka
Kafka
HDFS
ESP/CEP
Edge Analytics
MQTT
MQTT
Stream Analytics
Hadoop / Spark
Oracle Big Data Appliance
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data34
Kafka / MQTT / REST
Kafka /
MQTT /
REST
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
MQTT
Kafka / MQTT / REST
Kafka

26. IoT 6) Solutions & Challenges
Internet of Things (IoT) and Big Data35
Solutions
• Adding Big Data platform allows to store all raw data in the distributed file system in a
scalable and reliable manner
Challenges
• How can we leverage the Big Data platform for more than just storing raw data? How does
it combine with the stream processing?

27. Continuous Ingestion / Fan-In from the Edge
DB Source
Big Data
Log
Stream
Processing
IoT Sensor
Event Hub
Topic
Topic
REST
Topic
IoT GW
CDC GW
Connect
CDC
DB Source
Log CDC
Native
IoT Sensor
IoT Sensor
37
Dataflow GW
Topic
Topic
Queue
MQTT GW
Topic
Dataflow GW
Dataflow
TopicREST
37
File Source
Log
Log
Log
Social
Native
Internet of Things (IoT) and Big Data37
Topic
Topic

28. Challenges for Ingesting Sensor Data
Internet of Things (IoT) and Big Data
• Multitude of sensors
• Multiple Firmware
versions
• Bad Data from
damaged sensors
• Data Quality
38

29. REST / SOAP
REST / SOAP
IoT 6a) Adding Data Mining / Machine Learning and
Model execution
40
Mobile Apps
D
B
Rich (Web)
Client Apps
D
B
(ESB) / Data Integration
IoT
Devices
IoT
Gateways
IoT Smart
Devices
Event Hub
Event Hub
Enterprise Apps
WS
External Cloud
Service
Providers
BPM and SOA
Platform
Event
Business
Logic/Rules
Business
Intelligence
Services
WS
Event
Processes
Visualization
Analytics
DB
Service Bus
Oracle Data IntegratorKafka
Kafka
Kafka
SOAP
Various
SQL
SOAP
REST
WebSocket
JMS
JMSAPI GatewayAPI Gateway
REST REST
Kafka
Kafka
SQL
REST
REST
REST / SOAP
Stream Processing
ESP/CEP
DB
DB
Big Data Processing
HDFS
Batch
Processing
DB
Kafka
Kafka
HDFS
ESP/CEP
Edge Analytics
MQTT
MQTT
Stream Analytics
Hadoop / Spark
Oracle Big Data Appliance
SOA Suite
BPM Suite
Business Activity
Monitoring
Internet of Things (IoT) and Big Data40
Kafka / MQTT / REST
Kafka /
MQTT /
REST
= one way = request/response
I 4.0
Machine
DB CDC GoldenGate
MQTT
Kafka / MQTT / REST
Kafka

30. IoT Reference Architecture
Internet of Things (IoT) and Big Data42

31. IoT Services
IoT Logical Reference Architecture
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Internet of Things (IoT) and Big Data43

32. IoT Services
IoT Logical Reference Architecture – Oracle
on premises
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Edge Analytics
Business Activity
Monitoring
SOA Suite
BPM Suite
Service Bus
Oracle Data Integrator
Stream Analytics
Big Data
Appliance
Stream Analytics
Service Bus
API Gateway
Internet of Things (IoT) and Big Data44
Oracle IoT CS
Gateway
Oracle IoT CS
Client Library

33. IoT Services
IoT Logical Reference Architecture – Oracle
Cloud Services
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Edge Analytics
Oracle BI CS
Oracle Big Data CS
Oracle SOA CS
Oracle Integration CS
Oracle IoT CS
Oracle Streaming
Analytics CS
Oracle Messaging CS
Oracle Big Data
Discovery CS
Oracle Mobile CS
Internet of Things (IoT) and Big Data45
Oracle IoT CS
Gateway
Oracle IoT CS
Client Library
Oracle Process CS
Oracle DataFlow ML CS
Big Data Preparation CS
Application Container CS
Container CS

34. IoT Services
IoT Logical Reference Architecture – Microsoft
Azure
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
Azure IoTHub
Azure Streaming
Analytics
Azure Service Bus
Azure Power BI
HD Insight
Azure Storage
Azure IoT Gateway
SDK
Azure Event Hub
Azure Storage
Azure Machine
Learning
Internet of Things (IoT) and Big Data46

35. IoT Services
IoT
Device
Sensor
Actuator
IoT Gateway
Storage
UIApp
Streaming
Analytics
Enterprise
Applications
BPM and SOA
PlatformStreaming
Analytics
Storage
Endpoint
Management
Event
Hub
Service
Bus
Event
Hub Event
Hub
Service
Bus
Big Data / BI
Storage
Services Processes
UIApp
Storage
Bulk Analytics UI
Bulk
Analytics
UI
Storage
Streaming
Analytics
Service
Bus
API
REST
SOAP
HTTP
KAFKA
MQTT
CoAP
XMPP
DDS
AMQP
KAFKA
WIFI
BLE
ZigBee
WIFI
Wired
IoT Logical Architecture – Open Source
Internet of Things (IoT) and Big Data47

36. Summary
Internet of Things (IoT) and Big Data48

37. Summary
Treat events as events! Infrastructures for handling lots of events are available!
IoT tends to make Big Data / Fast Data infrastructures necessary
Know your use case/requirements to choose the right architecture!
• Can my existing backend landscape handle the new IoT load?
• Do I have to handle huge amount of events in “real-time”?
• Do I need to filter/aggregate data before invoking existing backend systems?
• Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary?
• What is the network bandwidth between device/gateway and cloud/backend?
• Centralized or Decentralized IoT solution?
Internet of Things (IoT) and Big Data49

38. J
Internet of Things (IoT) and Big Data50

39. Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com
Internet of Things (IoT) and Big Data51