SlideShare a Scribd company logo

Internet of Things (IoT) and Big Data

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Dependent on the size and quantity of such events, this can quickly be in the range of Big Data. How can we efficiently collect and transmit these events? How can we make sure that we can always report over historical events? How can these new events be integrated into traditional infrastructure and application landscape? Starting with a product and technology neutral reference architecture, we will then present different solutions using Open Source frameworks and the Oracle Stack both for on premises as well as the cloud.

1 of 39
Download to read offline
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Internet of Things (IoT) and Big Data
30.9.2016 – DOAG 2016 Big Data Days
Guido Schmutz
Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Internet of Things (IoT) and Big Data2
Agenda
1. Introduction
2. Towards an IoT Architecture
3. IoT Refererence Architecture
4. Summary
Internet of Things (IoT) and Big Data3
Introduction
Internet of Things (IoT) and Big Data4
Internet of Things (IoT) Wave
Internet of Things (IoT): Enabling
communication between devices,
people & processes to exchange
useful information & knowledge
that create value for humans
Term was first proposed by Kevin
Ashton in 1999
Source:	The	Economist
Source:	Ericsson,	June	2016
Internet of Things (IoT) and Big Data5
Reasons why IoT opportunity is occurring now ?
Affordable hardware
• Costs of actuators & sensors have been
cut in half over last 10 years
Smaller, more powerful hardware
• Form factors of hardware have shrunk to
millimeter or even nanometer levels
Ubiquitous & cheap mobility
• Cost for mobile devices, bandwidth and
data processing has declined over last
10 years
Availability of supporting tools
• Big data tools & cloud based infrastructure
have become widely available
Mass market awareness
• IoT has surpassed a critical tipping point
• Vision of a connected world has reached
such a followership that companies have
initiated IoT developments
• Commitment is irreversible
Internet of Things (IoT) and Big Data6

Recommended

IOT PROTOCOLS.pptx
IOT PROTOCOLS.pptxIOT PROTOCOLS.pptx
IOT PROTOCOLS.pptxDRREC
 
Internet of things - challenges scopes and solutions
Internet of things - challenges scopes and solutionsInternet of things - challenges scopes and solutions
Internet of things - challenges scopes and solutionsShivam Kumar
 
Introduction to IoT Architectures and Protocols
Introduction to IoT Architectures and ProtocolsIntroduction to IoT Architectures and Protocols
Introduction to IoT Architectures and ProtocolsAbdullah Alfadhly
 
Iot for smart city
Iot for smart cityIot for smart city
Iot for smart citysanalkumar k
 
IoT Platforms and Architecture
IoT Platforms and ArchitectureIoT Platforms and Architecture
IoT Platforms and ArchitectureLee House
 
Iot and cloud computing
Iot and cloud computingIot and cloud computing
Iot and cloud computingeteshagarwal1
 
Security and Privacy considerations in Internet of Things
Security and Privacy considerations in Internet of ThingsSecurity and Privacy considerations in Internet of Things
Security and Privacy considerations in Internet of ThingsSomasundaram Jambunathan
 

More Related Content

What's hot

Internet of things startup basic
Internet of things  startup basicInternet of things  startup basic
Internet of things startup basicMathan kumar
 
Cloud of things (IoT + Cloud Computing)
Cloud of things (IoT + Cloud Computing)Cloud of things (IoT + Cloud Computing)
Cloud of things (IoT + Cloud Computing)Zakaria Hossain
 
Introduction to IoT Security
Introduction to IoT SecurityIntroduction to IoT Security
Introduction to IoT SecurityCAS
 
Introduction to Internet of Things (IoT)
Introduction to Internet of Things (IoT)Introduction to Internet of Things (IoT)
Introduction to Internet of Things (IoT)Amarjeetsingh Thakur
 
Overview of IoT (JNTUK - UNIT 1)
Overview of IoT (JNTUK - UNIT 1)Overview of IoT (JNTUK - UNIT 1)
Overview of IoT (JNTUK - UNIT 1)FabMinds
 
Internet of Things(IOT)_Seminar_Dr.G.Rajeshkumar
Internet of Things(IOT)_Seminar_Dr.G.RajeshkumarInternet of Things(IOT)_Seminar_Dr.G.Rajeshkumar
Internet of Things(IOT)_Seminar_Dr.G.RajeshkumarRAJESHKUMARG12
 
THE INTERNET OF THINGS
THE INTERNET OF THINGSTHE INTERNET OF THINGS
THE INTERNET OF THINGSRamana Reddy
 
Iot architecture
Iot architectureIot architecture
Iot architectureAnam Iqbal
 
Internet of things (IoT)- Introduction, Utilities, Applications
Internet of things (IoT)- Introduction, Utilities, ApplicationsInternet of things (IoT)- Introduction, Utilities, Applications
Internet of things (IoT)- Introduction, Utilities, ApplicationsTarika Verma
 
Business models for business processes on IoT
Business models for business processes on IoTBusiness models for business processes on IoT
Business models for business processes on IoTFabMinds
 
IoT and Big Data
IoT and Big DataIoT and Big Data
IoT and Big Datasabnees
 
IoT Standardization and Implementation Challenges
IoT Standardization and Implementation ChallengesIoT Standardization and Implementation Challenges
IoT Standardization and Implementation ChallengesAhmed Banafa
 

What's hot (20)

IoT Networking
IoT NetworkingIoT Networking
IoT Networking
 
Internet of Things
Internet of ThingsInternet of Things
Internet of Things
 
Architectural reference model
Architectural reference modelArchitectural reference model
Architectural reference model
 
Internet of things startup basic
Internet of things  startup basicInternet of things  startup basic
Internet of things startup basic
 
Cloud of things (IoT + Cloud Computing)
Cloud of things (IoT + Cloud Computing)Cloud of things (IoT + Cloud Computing)
Cloud of things (IoT + Cloud Computing)
 
Introduction to IoT Security
Introduction to IoT SecurityIntroduction to IoT Security
Introduction to IoT Security
 
security and privacy-Internet of things
security and privacy-Internet of thingssecurity and privacy-Internet of things
security and privacy-Internet of things
 
Internet of Things (IoT) - IK
Internet of Things (IoT) - IKInternet of Things (IoT) - IK
Internet of Things (IoT) - IK
 
Introduction to Internet of Things (IoT)
Introduction to Internet of Things (IoT)Introduction to Internet of Things (IoT)
Introduction to Internet of Things (IoT)
 
Overview of IoT (JNTUK - UNIT 1)
Overview of IoT (JNTUK - UNIT 1)Overview of IoT (JNTUK - UNIT 1)
Overview of IoT (JNTUK - UNIT 1)
 
Internet of Things(IOT)_Seminar_Dr.G.Rajeshkumar
Internet of Things(IOT)_Seminar_Dr.G.RajeshkumarInternet of Things(IOT)_Seminar_Dr.G.Rajeshkumar
Internet of Things(IOT)_Seminar_Dr.G.Rajeshkumar
 
THE INTERNET OF THINGS
THE INTERNET OF THINGSTHE INTERNET OF THINGS
THE INTERNET OF THINGS
 
Importance of Big Data Analytics
Importance of Big Data AnalyticsImportance of Big Data Analytics
Importance of Big Data Analytics
 
Iot architecture
Iot architectureIot architecture
Iot architecture
 
Understanding IoT
Understanding IoTUnderstanding IoT
Understanding IoT
 
Internet of things (IoT)- Introduction, Utilities, Applications
Internet of things (IoT)- Introduction, Utilities, ApplicationsInternet of things (IoT)- Introduction, Utilities, Applications
Internet of things (IoT)- Introduction, Utilities, Applications
 
Iot ppt
Iot pptIot ppt
Iot ppt
 
Business models for business processes on IoT
Business models for business processes on IoTBusiness models for business processes on IoT
Business models for business processes on IoT
 
IoT and Big Data
IoT and Big DataIoT and Big Data
IoT and Big Data
 
IoT Standardization and Implementation Challenges
IoT Standardization and Implementation ChallengesIoT Standardization and Implementation Challenges
IoT Standardization and Implementation Challenges
 

Similar to Internet of Things (IoT) and Big Data

IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...Guido Schmutz
 
Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Guido Schmutz
 
IoT Architecture - are traditional architectures good enough?
IoT Architecture - are traditional architectures good enough?IoT Architecture - are traditional architectures good enough?
IoT Architecture - are traditional architectures good enough?Guido Schmutz
 
Building Large-Scale Applications for the Internet of Things at Bosch
Building Large-Scale Applications for the Internet of Things at BoschBuilding Large-Scale Applications for the Internet of Things at Bosch
Building Large-Scale Applications for the Internet of Things at BoschMongoDB
 
IoT Masterclass ESGT Santarem - Connecting The Dots
IoT Masterclass ESGT Santarem -  Connecting The DotsIoT Masterclass ESGT Santarem -  Connecting The Dots
IoT Masterclass ESGT Santarem - Connecting The DotsMarco Silva
 
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQBuilding a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQDominik Obermaier
 
Pragmatic approach to Microservice Architecture: Role of Middleware
Pragmatic approach to Microservice Architecture: Role of MiddlewarePragmatic approach to Microservice Architecture: Role of Middleware
Pragmatic approach to Microservice Architecture: Role of MiddlewareAsanka Abeysinghe
 
Iot cloud-or-onprem-170709204236
Iot cloud-or-onprem-170709204236Iot cloud-or-onprem-170709204236
Iot cloud-or-onprem-170709204236Aravindharamanan S
 
Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Guido Schmutz
 
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17Phoenix Data Conference - Big Data Analytics for IoT 11/4/17
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17Mark Goldstein
 
Internet of Things and Big Data: Vision and Concrete Use Cases
Internet of Things and Big Data: Vision and Concrete Use CasesInternet of Things and Big Data: Vision and Concrete Use Cases
Internet of Things and Big Data: Vision and Concrete Use CasesMongoDB
 
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System      IOT Based Smart City: Weather, Traffic and Pollution Monitoring System
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System IRJET Journal
 
NashTech - Azure IoT Solutions on Microsoft Azure
NashTech - Azure IoT Solutions on Microsoft AzureNashTech - Azure IoT Solutions on Microsoft Azure
NashTech - Azure IoT Solutions on Microsoft AzurePhi Huynh
 
Windows Core IoT for Makers - MVPDays
Windows Core IoT for Makers - MVPDaysWindows Core IoT for Makers - MVPDays
Windows Core IoT for Makers - MVPDaysGuy Barrette
 
Data Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platformsData Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platformsGuido Schmutz
 
Internet of Things IoT Guido Schmutz
Internet of Things IoT Guido SchmutzInternet of Things IoT Guido Schmutz
Internet of Things IoT Guido SchmutzDésirée Pfister
 
Internet of Things - Are traditional architectures good enough?
Internet of Things - Are traditional architectures good enough?Internet of Things - Are traditional architectures good enough?
Internet of Things - Are traditional architectures good enough?Guido Schmutz
 
Internet of Things (IoT)
Internet of Things (IoT)Internet of Things (IoT)
Internet of Things (IoT)Trivadis
 

Similar to Internet of Things (IoT) and Big Data (20)

Internet of Things and Big Data
Internet of Things and Big DataInternet of Things and Big Data
Internet of Things and Big Data
 
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...
IoT Architecture - Are Traditional Architectures Good Enough or do we Need Ne...
 
Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?
 
IoT Architecture - are traditional architectures good enough?
IoT Architecture - are traditional architectures good enough?IoT Architecture - are traditional architectures good enough?
IoT Architecture - are traditional architectures good enough?
 
Building Large-Scale Applications for the Internet of Things at Bosch
Building Large-Scale Applications for the Internet of Things at BoschBuilding Large-Scale Applications for the Internet of Things at Bosch
Building Large-Scale Applications for the Internet of Things at Bosch
 
IoT Masterclass ESGT Santarem - Connecting The Dots
IoT Masterclass ESGT Santarem -  Connecting The DotsIoT Masterclass ESGT Santarem -  Connecting The Dots
IoT Masterclass ESGT Santarem - Connecting The Dots
 
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQBuilding a reliable and scalable IoT platform with MongoDB and HiveMQ
Building a reliable and scalable IoT platform with MongoDB and HiveMQ
 
Pragmatic approach to Microservice Architecture: Role of Middleware
Pragmatic approach to Microservice Architecture: Role of MiddlewarePragmatic approach to Microservice Architecture: Role of Middleware
Pragmatic approach to Microservice Architecture: Role of Middleware
 
Iot cloud-or-onprem-170709204236
Iot cloud-or-onprem-170709204236Iot cloud-or-onprem-170709204236
Iot cloud-or-onprem-170709204236
 
Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?Internet of Things (IoT) - in the cloud or rather on-premises?
Internet of Things (IoT) - in the cloud or rather on-premises?
 
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17Phoenix Data Conference - Big Data Analytics for IoT 11/4/17
Phoenix Data Conference - Big Data Analytics for IoT 11/4/17
 
Internet of Things and Big Data: Vision and Concrete Use Cases
Internet of Things and Big Data: Vision and Concrete Use CasesInternet of Things and Big Data: Vision and Concrete Use Cases
Internet of Things and Big Data: Vision and Concrete Use Cases
 
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System      IOT Based Smart City: Weather, Traffic and Pollution Monitoring System
IOT Based Smart City: Weather, Traffic and Pollution Monitoring System
 
NashTech - Azure IoT Solutions on Microsoft Azure
NashTech - Azure IoT Solutions on Microsoft AzureNashTech - Azure IoT Solutions on Microsoft Azure
NashTech - Azure IoT Solutions on Microsoft Azure
 
iot
iotiot
iot
 
Windows Core IoT for Makers - MVPDays
Windows Core IoT for Makers - MVPDaysWindows Core IoT for Makers - MVPDays
Windows Core IoT for Makers - MVPDays
 
Data Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platformsData Ingestion in Big Data and IoT platforms
Data Ingestion in Big Data and IoT platforms
 
Internet of Things IoT Guido Schmutz
Internet of Things IoT Guido SchmutzInternet of Things IoT Guido Schmutz
Internet of Things IoT Guido Schmutz
 
Internet of Things - Are traditional architectures good enough?
Internet of Things - Are traditional architectures good enough?Internet of Things - Are traditional architectures good enough?
Internet of Things - Are traditional architectures good enough?
 
Internet of Things (IoT)
Internet of Things (IoT)Internet of Things (IoT)
Internet of Things (IoT)
 

More from Guido Schmutz

30 Minutes to the Analytics Platform with Infrastructure as Code
30 Minutes to the Analytics Platform with Infrastructure as Code30 Minutes to the Analytics Platform with Infrastructure as Code
30 Minutes to the Analytics Platform with Infrastructure as CodeGuido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
Event Broker (Kafka) in a Modern Data ArchitectureEvent Broker (Kafka) in a Modern Data Architecture
Event Broker (Kafka) in a Modern Data ArchitectureGuido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsBig Data, Data Lake, Fast Data - Dataserialiation-Formats
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsGuido Schmutz
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!Guido Schmutz
 
Kafka as your Data Lake - is it Feasible?
Kafka as your Data Lake - is it Feasible?Kafka as your Data Lake - is it Feasible?
Kafka as your Data Lake - is it Feasible?Guido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data ArchitectureEvent Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data ArchitectureGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaSolutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaGuido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureEvent Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureGuido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
Location Analytics - Real-Time Geofencing using Apache KafkaLocation Analytics - Real-Time Geofencing using Apache Kafka
Location Analytics - Real-Time Geofencing using Apache KafkaGuido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS and Apache KafkaSolutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS and Apache KafkaGuido Schmutz
 
What is Apache Kafka? Why is it so popular? Should I use it?
What is Apache Kafka? Why is it so popular? Should I use it?What is Apache Kafka? Why is it so popular? Should I use it?
What is Apache Kafka? Why is it so popular? Should I use it?Guido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaSolutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaGuido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
Location Analytics Real-Time Geofencing using KafkaLocation Analytics Real-Time Geofencing using Kafka
Location Analytics Real-Time Geofencing using KafkaGuido Schmutz
 
Streaming Visualisation
Streaming VisualisationStreaming Visualisation
Streaming VisualisationGuido Schmutz
 
Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?Guido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Solutions for bi-directional Integration between Oracle RDMBS & Apache KafkaSolutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Solutions for bi-directional Integration between Oracle RDMBS & Apache KafkaGuido Schmutz
 
Fundamentals Big Data and AI Architecture
Fundamentals Big Data and AI ArchitectureFundamentals Big Data and AI Architecture
Fundamentals Big Data and AI ArchitectureGuido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
Location Analytics - Real-Time Geofencing using Kafka Location Analytics - Real-Time Geofencing using Kafka
Location Analytics - Real-Time Geofencing using Kafka Guido Schmutz
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming VisualizationGuido Schmutz
 

More from Guido Schmutz (20)

30 Minutes to the Analytics Platform with Infrastructure as Code
30 Minutes to the Analytics Platform with Infrastructure as Code30 Minutes to the Analytics Platform with Infrastructure as Code
30 Minutes to the Analytics Platform with Infrastructure as Code
 
Event Broker (Kafka) in a Modern Data Architecture
Event Broker (Kafka) in a Modern Data ArchitectureEvent Broker (Kafka) in a Modern Data Architecture
Event Broker (Kafka) in a Modern Data Architecture
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
Big Data, Data Lake, Fast Data - Dataserialiation-FormatsBig Data, Data Lake, Fast Data - Dataserialiation-Formats
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
 
Kafka as your Data Lake - is it Feasible?
Kafka as your Data Lake - is it Feasible?Kafka as your Data Lake - is it Feasible?
Kafka as your Data Lake - is it Feasible?
 
Event Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data ArchitectureEvent Hub (i.e. Kafka) in Modern Data Architecture
Event Hub (i.e. Kafka) in Modern Data Architecture
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaSolutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Event Hub (i.e. Kafka) in Modern Data (Analytics) ArchitectureEvent Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
 
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache KafkaBuilding Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
 
Location Analytics - Real-Time Geofencing using Apache Kafka
Location Analytics - Real-Time Geofencing using Apache KafkaLocation Analytics - Real-Time Geofencing using Apache Kafka
Location Analytics - Real-Time Geofencing using Apache Kafka
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS and Apache KafkaSolutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
 
What is Apache Kafka? Why is it so popular? Should I use it?
What is Apache Kafka? Why is it so popular? Should I use it?What is Apache Kafka? Why is it so popular? Should I use it?
What is Apache Kafka? Why is it so popular? Should I use it?
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache KafkaSolutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
 
Location Analytics Real-Time Geofencing using Kafka
Location Analytics Real-Time Geofencing using KafkaLocation Analytics Real-Time Geofencing using Kafka
Location Analytics Real-Time Geofencing using Kafka
 
Streaming Visualisation
Streaming VisualisationStreaming Visualisation
Streaming Visualisation
 
Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?Kafka as an event store - is it good enough?
Kafka as an event store - is it good enough?
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Solutions for bi-directional Integration between Oracle RDMBS & Apache KafkaSolutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
 
Fundamentals Big Data and AI Architecture
Fundamentals Big Data and AI ArchitectureFundamentals Big Data and AI Architecture
Fundamentals Big Data and AI Architecture
 
Location Analytics - Real-Time Geofencing using Kafka
Location Analytics - Real-Time Geofencing using Kafka Location Analytics - Real-Time Geofencing using Kafka
Location Analytics - Real-Time Geofencing using Kafka
 
Streaming Visualization
Streaming VisualizationStreaming Visualization
Streaming Visualization
 

Recently uploaded

ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...Neo4j
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoProduct School
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Product School
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxNeo4j
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?MENGSAYLOEM1
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1Inbay UK
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotelPhilippines
 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Product School
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stackSummit
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys VasylievFwdays
 
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions..."How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...Fwdays
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...DianaGray10
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...DianaGray10
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsInflectra
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxInfosec
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 

Recently uploaded (20)

ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
ASTRAZENECA. Knowledge Graphs Powering a Fast-moving Global Life Sciences Org...
 
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, MonzoRevolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
Revolutionizing The Banking Industry: The Monzo Way by CPO, Monzo
 
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
Cultivating Entrepreneurial Mindset in Product Management: Strategies for Suc...
 
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptxThe Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
The Art of the Possible with Graph by Dr Jim Webber Neo4j.pptx
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?Are Human-generated Demonstrations Necessary for In-context Learning?
Are Human-generated Demonstrations Necessary for In-context Learning?
 
IT Nation Evolve event 2024 - Quarter 1
IT Nation Evolve event 2024  - Quarter 1IT Nation Evolve event 2024  - Quarter 1
IT Nation Evolve event 2024 - Quarter 1
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company Profile
 
Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...Launching New Products In Companies Where It Matters Most by Product Director...
Launching New Products In Companies Where It Matters Most by Product Director...
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stack
 
"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev"AIRe - AI Reliability Engineering", Denys Vasyliev
"AIRe - AI Reliability Engineering", Denys Vasyliev
 
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions..."How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
"How we created an SRE team in Temabit as a part of FOZZY Group in conditions...
 
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
Automation Ops Series: Session 1 - Introduction and setup DevOps for UiPath p...
 
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...Building Bridges:  Merging RPA Processes, UiPath Apps, and Data Service to bu...
Building Bridges: Merging RPA Processes, UiPath Apps, and Data Service to bu...
 
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+PluginsFrom Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
From Challenger to Champion: How SpiraPlan Outperforms JIRA+Plugins
 
How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptx
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
 
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 

Internet of Things (IoT) and Big Data

  • 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Internet of Things (IoT) and Big Data 30.9.2016 – DOAG 2016 Big Data Days Guido Schmutz
  • 2. Guido Schmutz Working for Trivadis for more than 19 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data Member of Trivadis Architecture Board Technology Manager @ Trivadis More than 25 years of software development experience Contact: guido.schmutz@trivadis.com Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz Internet of Things (IoT) and Big Data2
  • 3. Agenda 1. Introduction 2. Towards an IoT Architecture 3. IoT Refererence Architecture 4. Summary Internet of Things (IoT) and Big Data3
  • 4. Introduction Internet of Things (IoT) and Big Data4
  • 5. Internet of Things (IoT) Wave Internet of Things (IoT): Enabling communication between devices, people & processes to exchange useful information & knowledge that create value for humans Term was first proposed by Kevin Ashton in 1999 Source: The Economist Source: Ericsson, June 2016 Internet of Things (IoT) and Big Data5
  • 6. Reasons why IoT opportunity is occurring now ? Affordable hardware • Costs of actuators & sensors have been cut in half over last 10 years Smaller, more powerful hardware • Form factors of hardware have shrunk to millimeter or even nanometer levels Ubiquitous & cheap mobility • Cost for mobile devices, bandwidth and data processing has declined over last 10 years Availability of supporting tools • Big data tools & cloud based infrastructure have become widely available Mass market awareness • IoT has surpassed a critical tipping point • Vision of a connected world has reached such a followership that companies have initiated IoT developments • Commitment is irreversible Internet of Things (IoT) and Big Data6
  • 7. The Sensing-as-a-Service Model Internet of Things (IoT) and Big Data7
  • 8. Towards an IoT Architecture Internet of Things (IoT) and Big Data15
  • 9. Key Challenges for building an IoT application 1. Connect: How to collect data from intelligent devices? • Abstract complexity associated with device connectivity • Standardize integration of devices with enterprise 2. Analyze: How to analyze IoT data? • Reduce noise and detect business event at real-time • Enable historical big-data analysis 3. Integrate: How to integrate IoT data & events with enterprise infrastructure? • Make enterprise processes IoT friendly • Allow enterprise & mobile applications to control devices Internet of Things (IoT) and Big Data18
  • 10. Today) Existing Service-/API Architecture as a base 19 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST / SOAP REST / SOAP SOAP Various SQL SOAP REST Service BusOracle Data Integrator API Gateway SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data19 = one way = request/response
  • 11. REST / SOAP REST / SOAP IoT 1a) Reuse exiting Service-/API-based Architecture IoT Smart Devices 20 Mobile Apps D B Rich (Web) Client Apps D B Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST / SOAP REST REST JMS / REST SOAP Various SQL SOAP REST WebSocket JMS Service BusOracle Data Integrator API Gateway API Gateway JMS JMS WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data20 = one way = request/response
  • 12. IoT 1a) Challenges Internet of Things (IoT) and Big Data21 • Do IoT devices contain enough resources to communicate directly over the internet (HTTP or JMS) ? • Should the device only collect data (sense) or is there also the way back necessary (actuator) ? • Can JMS be used from external devices (firewalls allow traffic over JMS) ? • How many IoT devices are planned short and long term? How often do they send to the backend? Are the JMS server as well as the ESB capable to deal with the resulting message volume ? • What are the operations on IoT messages / events, only simple transformations, filter and routing operations?
  • 13. REST / SOAP REST / SOAP IoT 1b) Reuse existing Service-/API-based Architecture IoT Smart Devices 22 Mobile Apps D B Rich (Web) Client Apps D B API Gateway Enterprise Service Bus (ESB) / Data Integration Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST HTTP REST REST HTTP JMS API Gateway Service BusOracle Data Integrator SOAP Various SQL SOAP REST WebSocket JMS JMS JMS REST / SOAP WeblogicJMS SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data22 = one way = request/response
  • 14. REST / SOAP REST / SOAP IoT 2) Adding Event Hub and optional IoT Gateway 23 Mobile Apps D B Rich (Web) Client Apps D B ESB / Data Integration IoT Devices IoT Gateways IoT Smart Devices Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB REST Kafka / MQTT / REST REST Kafka / MQTT / REST MQTT Kura Event Hub Kafka Service Bus Oracle Data Integrator REST REST SOAP Various SQL SOAP REST WebSocket Kafka JMS JMS API Gateway API Gateway REST Kafka REST / SOAP REST REST SOA Suite BPM Suite Business Activity Monitoring MQTT Internet of Things (IoT) and Big Data23 MQTT = one way = request/response
  • 15. How to implement an Event Hub? Apache Kafka to the rescue Distributed publish-subscribe messaging system Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …) Topic Semantic does not implement JMS standard Initially developed at LinkedIn, now part of Apache Kafka Cluster Consumer Consumer Consumer Producer Producer Producer Internet of Things (IoT) and Big Data24
  • 16. Oracle’s Service Bus as a consumer of Kafka Service Bus 12c Cloud Apps Business Service Cloud Proxy Service Kafka Cloud API Mobile Apps Pipeline Routing Kafka Sensor / IoT Web Apps Business Service REST Business Service WSDL Backend Apps REST Backend Apps WSDL Proxy Service Kafka Pipeline Routing Database DB CDC Stream Processing Internet of Things (IoT) and Big Data25
  • 17. IoT 2) Solutions & Challenges Internet of Things (IoT) and Big Data26 Solutions • Event Hub solves the potential scalability issue of JMS • IoT Gateway makes sure that lightweight sensors can connect to the internet / send their data Challenges • Where to do complex analytics on the events? Is it scalable? • Can we really send all data down to backend? Network bandwidth?
  • 18. REST / SOAP REST / SOAP IoT 3) Adding Stream Processing / Analytics in Backend 27 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data27 = one way = request/response KafkaMQTT
  • 19. IoT 3) Solutions & Challenges Internet of Things (IoT) and Big Data28 Solutions • Stream Processing handles complex analytics on events in a scalable manner before sending events to ESB / backend systems Challenges • Can we really send all data down to backend? Network bandwidth?
  • 20. Oracle’s Stream Analytics as consumer of Kafka/MQTT Oracle Stream Analytics Stream Analytics Kafka Kafka Mobile Apps Kafka Sensor / IoT Web Apps Machine Data DB CDC Kafka MQTT Internet of Things (IoT) and Big Data29
  • 21. REST / SOAP REST / SOAP IoT 4) Adding Industry 4.0 Data Sources (machine data) 30 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka MQTT Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway REST MQTT REST REST Kafka Kafka / MQTT / REST Kafka / MQTT / REST REST Kafka REST / SOAP REST REST Kura SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data30 = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST KafkaMQTT
  • 22. GoldenGate Gateway Oracle’s GoldenGate for Change Data Capture of existing database Internet of Things (IoT) and Big Data31 Machine Data DB Kafka Oracle GoldenGate Delivery Capture Pump JMS Machine Data DB Oracle GoldenGate Delivery Capture Pump HBase HDFS
  • 23. REST / SOAP REST / SOAP IoT 5) Adding Stream Processing / Analytics at Edge 32 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Stream Processing ESP / CEP DB DB Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Stream Analytics Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket Kafka JMS JMSAPI GatewayAPI Gateway Kafka / MQTT / REST REST REST Kafka Kafka REST RESTREST REST / SOAP REST MQTT SOA Suite BPM Suite Business Activity Monitoring ESP/CEP Edge Analytics MQTT Internet of Things (IoT) and Big Data32 Kafka / MQTT / REST MQTT = one way = request/response I 4.0 Machine DB CDC GoldenGate Kafka / MQTT / REST Kafka
  • 24. IoT 5) Solutions & Challenges Internet of Things (IoT) and Big Data33 Solutions • Stream Processing at the edge / gateway allows to reduce the amount of messages send to the backend (cloud / on premises) if necessary Challenges • What if we need the raw events persisted? Where do to that?
  • 25. REST / SOAP REST / SOAP IoT 6) Adding Raw Data Storage and Batch Analytics 34 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data34 Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka
  • 26. IoT 6) Solutions & Challenges Internet of Things (IoT) and Big Data35 Solutions • Adding Big Data platform allows to store all raw data in the distributed file system in a scalable and reliable manner Challenges • How can we leverage the Big Data platform for more than just storing raw data? How does it combine with the stream processing?
  • 27. Continuous Ingestion / Fan-In from the Edge DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 37 Dataflow GW Topic Topic Queue MQTT GW Topic Dataflow GW Dataflow TopicREST 37 File Source Log Log Log Social Native Internet of Things (IoT) and Big Data37 Topic Topic
  • 28. Challenges for Ingesting Sensor Data Internet of Things (IoT) and Big Data • Multitude of sensors • Multiple Firmware versions • Bad Data from damaged sensors • Data Quality 38
  • 29. REST / SOAP REST / SOAP IoT 6a) Adding Data Mining / Machine Learning and Model execution 40 Mobile Apps D B Rich (Web) Client Apps D B (ESB) / Data Integration IoT Devices IoT Gateways IoT Smart Devices Event Hub Event Hub Enterprise Apps WS External Cloud Service Providers BPM and SOA Platform Event Business Logic/Rules Business Intelligence Services WS Event Processes Visualization Analytics DB Service Bus Oracle Data IntegratorKafka Kafka Kafka SOAP Various SQL SOAP REST WebSocket JMS JMSAPI GatewayAPI Gateway REST REST Kafka Kafka SQL REST REST REST / SOAP Stream Processing ESP/CEP DB DB Big Data Processing HDFS Batch Processing DB Kafka Kafka HDFS ESP/CEP Edge Analytics MQTT MQTT Stream Analytics Hadoop / Spark Oracle Big Data Appliance SOA Suite BPM Suite Business Activity Monitoring Internet of Things (IoT) and Big Data40 Kafka / MQTT / REST Kafka / MQTT / REST = one way = request/response I 4.0 Machine DB CDC GoldenGate MQTT Kafka / MQTT / REST Kafka
  • 30. IoT Reference Architecture Internet of Things (IoT) and Big Data42
  • 31. IoT Services IoT Logical Reference Architecture IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Internet of Things (IoT) and Big Data43
  • 32. IoT Services IoT Logical Reference Architecture – Oracle on premises IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Business Activity Monitoring SOA Suite BPM Suite Service Bus Oracle Data Integrator Stream Analytics Big Data Appliance Stream Analytics Service Bus API Gateway Internet of Things (IoT) and Big Data44 Oracle IoT CS Gateway Oracle IoT CS Client Library
  • 33. IoT Services IoT Logical Reference Architecture – Oracle Cloud Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Edge Analytics Oracle BI CS Oracle Big Data CS Oracle SOA CS Oracle Integration CS Oracle IoT CS Oracle Streaming Analytics CS Oracle Messaging CS Oracle Big Data Discovery CS Oracle Mobile CS Internet of Things (IoT) and Big Data45 Oracle IoT CS Gateway Oracle IoT CS Client Library Oracle Process CS Oracle DataFlow ML CS Big Data Preparation CS Application Container CS Container CS
  • 34. IoT Services IoT Logical Reference Architecture – Microsoft Azure IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired Azure IoTHub Azure Streaming Analytics Azure Service Bus Azure Power BI HD Insight Azure Storage Azure IoT Gateway SDK Azure Event Hub Azure Storage Azure Machine Learning Internet of Things (IoT) and Big Data46
  • 35. IoT Services IoT Device Sensor Actuator IoT Gateway Storage UIApp Streaming Analytics Enterprise Applications BPM and SOA PlatformStreaming Analytics Storage Endpoint Management Event Hub Service Bus Event Hub Event Hub Service Bus Big Data / BI Storage Services Processes UIApp Storage Bulk Analytics UI Bulk Analytics UI Storage Streaming Analytics Service Bus API REST SOAP HTTP KAFKA MQTT CoAP XMPP DDS AMQP KAFKA WIFI BLE ZigBee WIFI Wired IoT Logical Architecture – Open Source Internet of Things (IoT) and Big Data47
  • 36. Summary Internet of Things (IoT) and Big Data48
  • 37. Summary Treat events as events! Infrastructures for handling lots of events are available! IoT tends to make Big Data / Fast Data infrastructures necessary Know your use case/requirements to choose the right architecture! • Can my existing backend landscape handle the new IoT load? • Do I have to handle huge amount of events in “real-time”? • Do I need to filter/aggregate data before invoking existing backend systems? • Do I want to do Advanced Analytics (predictive analytics) where historical information is necessary? • What is the network bandwidth between device/gateway and cloud/backend? • Centralized or Decentralized IoT solution? Internet of Things (IoT) and Big Data49
  • 38. J Internet of Things (IoT) and Big Data50