SlideShare a Scribd company logo
1 of 35
How to Deploy a Kafka Cluster in
Production onto Your Desk or Anywhere!
IoT Use Cases, Architectures, and Live Demo for Data in Motion powered by Apache Kafka
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
confluent.io
kai-waehner.de
@KaiWaehner
Dominik Pilat
VP of Customer Success & Field CTO
d.pilat@hivecell.com
linkedin.com/in/dpilat
hivecell.com
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Real-time Data beats Slow Data.
Manufacturing
Sensor diagnostics
MES/ERP Integration
Reporting
Edge Computing
Condition Monitoring
Predictive Maintenance
Quality Assurance
Logistics
Supply Chain
Inventory management
Track & Trace
Context-specific routing
Cybersecurity
Threat detection
Intrusion detection
Incident response
Fraud detection
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
AWS Cloud Outage hit Disney World Visitors…
https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
MQTT
Data
Historian
STREAM
PROCESSING
CONNECTORS
Example Architecture for Data in Motion with Event
Streaming
ksqlDB
KStreams
Bi-directional real-time integration between the OT and IT worlds
APP
Oracle
DB
ORACLE CDC
SOURCE
CONNECTOR
APP
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
8
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate with
other kids in the train, …
Always on (even “offline”)
Replayability
Cost-efficiency
Low latency
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Shipping Industry
Marine, Oil Transport, Vessel Fleet, Shipping Line, Drones
Real-time Operations, Logistics, Predictive Maintenance, Security
Customer Data
Crew, Cargo
Vessel Data
Fuel Consumption, Speed,
Planned Maintenance
Automatic Identification System (AIS)
Unique Identification,
Position, Course, Weather, Draft
Drone Data
Deliveries,
Survey/Inspection
of Assets such as Oil Rigs,
Pipelines, Offshore Turbines
Edge Analytics
Bidirectional Edge to Cloud Integration
Data Ingestion
Stream
Processing
Data
Integration
Logistics
Track&Trace
Routing
Monitoring
Alerting
Command&Control
Batch Analytics
Reporting
Machine Learning
Backend Systems
Oracle, SAP,
OSIsoft PI, etc.
X = Event Streaming
X = Other Technologies
Bi-Directional Hybrid Cloud
Replication
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Royal Caribbean - Offline Edge for Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Devon Energy
Oil & Gas Industry
Improve drilling and well completion operations
Edge stream processing/analytics + closed-loop control ready
Vendor agnostic (pumping, wireline, coil, offset wells, drilling
operations, producing wells)
Replication to the cloud in real-time at scale
Cloud agnostic (AWS, GCP, Azure)
Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Event Streaming in Hybrid Industrial
IoT
Wavelength
Public Cloud VPC
Campus 5G Telco Carrier 5G
Smart Factory Edge Computing
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Analytics
Database
Data Lake
CRM
Kafka as Data Historian
Continuous real time data ingestion, processing and monitoring 24/7 at scale
MQTT Modbus Firewall
Siemens S7
Kafka Connect
Integration | Storage | Processing | Access | Cloud | Security
Storage
Kafka Streams
/
ksqlDB
OPC-UA
Stateless + Stateful
Kafka
Connect
Kafka Clients
Java, C++. Python, Go, …
REST
Proxy
SCADA
DCS
ERP
MES
Syslog
File
X
Y
Z
A
B
C
Cloud
Factory
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
3 4 5 6 7
2
1
Time
Sensor Events
1
6
Condition Monitoring
(Temperature Spikes)
Stateless Filter Above-Threshold Events
Streams
builder
.stream(”temperature-sensor")
.filter((key, sensor-data) ->
sensor-data.temperature > 100)
.to(”temperature-spikes");
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
Predictive Maintenance
(Continuous Anomaly Detection)
Stateful Correlation of Events
CREATE TABLE anomaly_detection AS
SELECT temperature_spike_id, COUNT(*) AS total_spikes,
AVG(temperature) AS avg_temperature
FROM sensor-data
WINDOW TUMBLING (SIZE 1 HOUR)
GROUP BY temperature_spike_id
EMIT CHANGES;
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
Predictive Maintenance
(Continuous Anomaly Detection)
Real-time Machine Learning
CREATE STREAM anomaly_detection AS
SELECT sensor_id, detect_anomaly(sensor_values)
FROM machine;
TensorFlow model embedded in User Defined
Function (UDF)
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Edge
Cloud
Texas Bavaria
Hybrid Data Streaming: Edge and Cloud
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Texas
Texas data
Streams of real time events
Texas data
Streams of real time events
Bavaria data
Bavaria
Hybrid Data Streaming: Edge and Cloud
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Conditions of edge environments
• No data closet
• Or data closet is full, overheating and under powered
• Limited or unreliable power supply
• Limited number of ports
• No IT staff on site
• Hundreds or thousands of locations
• Limited network bandwidth
• Demand for compute power always increasing
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Requirements for an edge solution
No special equipment for power, air conditioning or networking
No technicians required onsite to install and configure
Start with smallest footprint possible to demonstrate ROI
Easily add more compute power as workload expands
Run distributed software for containers, messaging and machine learning
Monitor, manage and upgrade centrally, even when behind a firewall
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Hivecell
✓ No special equipment
✓ No technicians required
✓ Start with smallest footprint
✓ Easily add more compute power
✓ Run distributed software
✓ Monitor, manage and update remotely
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
The Rise of Data in Motion
for Edge and Hybrid Scenarios
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
29
2017
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
30
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Container orchestration at the
edge
Enterprise streaming at the
edge
HIVECELL = PAAS AT THE EDGE
Hyperconverged
infrastructure at the edge
HCI
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
EDGE AS A SERVICE.
EVERYTHING
YOU NEED IS
INCLUDED.
• Kubernetes and Confluent as a service
• Framework licenses/support/updates
• Hive Control - Provision/monitor system
• Operating system (OS) support/updates
• Hardware support/replacements
• Evergreen Hardware: upgrades every 3 years
User just uses Kafka. At the edge.
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
34
Hivecell and Confluent enable Kafka at the Edge.
Reliable. Scalable. Simple.
Conclusion
Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Stay in touch!
35
Confluent Cloud
cnfl.io/confluent-cloud
1 hour free edge consulting
session with Dominik Pilat
https://hivecell.com/contact/
Kai Waehner
Field CTO
Confluent
kai.waehner@confluent.io
@KaiWaehner
confluent.io
kai-waehner.de
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!
Dominik Pilat
VP of Customer Success
Field CTO
Hivecell
d.pilat@hivecell.com
linkedin.com/in/dpilat
hivecell.com

More Related Content

What's hot

Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using KafkaKnoldus Inc.
 
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaTop 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaKai Wähner
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafkaconfluent
 
Running Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsRunning Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann
 
Apache kafka performance(throughput) - without data loss and guaranteeing dat...
Apache kafka performance(throughput) - without data loss and guaranteeing dat...Apache kafka performance(throughput) - without data loss and guaranteeing dat...
Apache kafka performance(throughput) - without data loss and guaranteeing dat...SANG WON PARK
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
 
Building a Virtual Data Lake with Apache Arrow
Building a Virtual Data Lake with Apache ArrowBuilding a Virtual Data Lake with Apache Arrow
Building a Virtual Data Lake with Apache ArrowDremio Corporation
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®confluent
 
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Timothy Spann
 
Large Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingLarge Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingDatabricks
 
Benefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use CasesBenefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use Casesconfluent
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafkaconfluent
 
Building Advanced Analytics Pipelines with Azure Databricks
Building Advanced Analytics Pipelines with Azure DatabricksBuilding Advanced Analytics Pipelines with Azure Databricks
Building Advanced Analytics Pipelines with Azure DatabricksLace Lofranco
 
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsSpark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsGuido Schmutz
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka confluent
 

What's hot (20)

Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache KafkaTop 5 Event Streaming Use Cases for 2021 with Apache Kafka
Top 5 Event Streaming Use Cases for 2021 with Apache Kafka
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafka
 
Running Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration OptionsRunning Apache NiFi with Apache Spark : Integration Options
Running Apache NiFi with Apache Spark : Integration Options
 
Apache kafka performance(throughput) - without data loss and guaranteeing dat...
Apache kafka performance(throughput) - without data loss and guaranteeing dat...Apache kafka performance(throughput) - without data loss and guaranteeing dat...
Apache kafka performance(throughput) - without data loss and guaranteeing dat...
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
 
Building a Virtual Data Lake with Apache Arrow
Building a Virtual Data Lake with Apache ArrowBuilding a Virtual Data Lake with Apache Arrow
Building a Virtual Data Lake with Apache Arrow
 
Kafka presentation
Kafka presentationKafka presentation
Kafka presentation
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
 
Apache Kafka Best Practices
Apache Kafka Best PracticesApache Kafka Best Practices
Apache Kafka Best Practices
 
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
 
Large Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured StreamingLarge Scale Lakehouse Implementation Using Structured Streaming
Large Scale Lakehouse Implementation Using Structured Streaming
 
Benefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use CasesBenefits of Stream Processing and Apache Kafka Use Cases
Benefits of Stream Processing and Apache Kafka Use Cases
 
Disaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache KafkaDisaster Recovery Plans for Apache Kafka
Disaster Recovery Plans for Apache Kafka
 
Building Advanced Analytics Pipelines with Azure Databricks
Building Advanced Analytics Pipelines with Azure DatabricksBuilding Advanced Analytics Pipelines with Azure Databricks
Building Advanced Analytics Pipelines with Azure Databricks
 
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsSpark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka Streams
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
 

Similar to Kafka for Real-Time Replication between Edge and Hybrid Cloud

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021confluent
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafkaconfluent
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...Kai Wähner
 
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022HostedbyConfluent
 
App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudKai Wähner
 
Introduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matterIntroduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matterPaolo Castagna
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionKai Wähner
 
Streaming Data Ingest and Processing with Apache Kafka
Streaming Data Ingest and Processing with Apache KafkaStreaming Data Ingest and Processing with Apache Kafka
Streaming Data Ingest and Processing with Apache KafkaAttunity
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023confluent
 
Event Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBEvent Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBScyllaDB
 
Bridge Your Kafka Streams to Azure Webinar
Bridge Your Kafka Streams to Azure WebinarBridge Your Kafka Streams to Azure Webinar
Bridge Your Kafka Streams to Azure Webinarconfluent
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent
 
BBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.comBBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.comCedric Vidal
 
Unlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxUnlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxAhmed791434
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsight
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsightIngestion in data pipelines with Managed Kafka Clusters in Azure HDInsight
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsightMicrosoft Tech Community
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingTimothy Spann
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 

Similar to Kafka for Real-Time Replication between Edge and Hybrid Cloud (20)

Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021The Top 5 Event Streaming Use Cases & Architectures in 2021
The Top 5 Event Streaming Use Cases & Architectures in 2021
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafka
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
 
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
 
App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent Cloud
 
Introduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matterIntroduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matter
 
Apache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy ProductionApache Kafka for Smart Grid, Utilities and Energy Production
Apache Kafka for Smart Grid, Utilities and Energy Production
 
Streaming Data Ingest and Processing with Apache Kafka
Streaming Data Ingest and Processing with Apache KafkaStreaming Data Ingest and Processing with Apache Kafka
Streaming Data Ingest and Processing with Apache Kafka
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Event Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDBEvent Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDB
 
Bridge Your Kafka Streams to Azure Webinar
Bridge Your Kafka Streams to Azure WebinarBridge Your Kafka Streams to Azure Webinar
Bridge Your Kafka Streams to Azure Webinar
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
BBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.comBBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.com
 
Unlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptxUnlock value with Confluent and AWS.pptx
Unlock value with Confluent and AWS.pptx
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsight
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsightIngestion in data pipelines with Managed Kafka Clusters in Azure HDInsight
Ingestion in data pipelines with Managed Kafka Clusters in Azure HDInsight
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
 
The Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and StreamingThe Never Landing Stream with HTAP and Streaming
The Never Landing Stream with HTAP and Streaming
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 

More from Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryKai Wähner
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaKai Wähner
 

More from Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 
Apache Kafka in the Insurance Industry
Apache Kafka in the Insurance IndustryApache Kafka in the Insurance Industry
Apache Kafka in the Insurance Industry
 
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesApache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
 
Connected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache KafkaConnected Vehicles and V2X with Apache Kafka
Connected Vehicles and V2X with Apache Kafka
 

Recently uploaded

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsUniversity of Antwerp
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdfOffsiteNOC
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houstonjennysmithusa549
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurPriyadarshini T
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJpolinaucc
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Inc
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energyjeyasrig
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptxAGATSoftware
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.Ritesh Kanjee
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tipsmichealwillson701
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevpmgdscunsri
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...MyFAA
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...jackiepotts6
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfCloudMetic
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...telebusocialmarketin
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easymichealwillson701
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridMathew Thomas
 

Recently uploaded (20)

MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf8 key point on optimizing web hosting services in your business.pdf
8 key point on optimizing web hosting services in your business.pdf
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houston
 
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJ
 
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
Splashtop Enterprise Brochure - Remote Computer Access and Remote Support Sof...
 
renewable energy renewable energy renewable energy renewable energy
renewable energy renewable energy renewable energy  renewable energyrenewable energy renewable energy renewable energy  renewable energy
renewable energy renewable energy renewable energy renewable energy
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
 
8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.8 Steps to Build a LangChain RAG Chatbot.
8 Steps to Build a LangChain RAG Chatbot.
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
 
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
Take Advantage of Mx Tracking Flight Scheduling Solutions to Streamline Your ...
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
 
Boost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made EasyBoost Efficiency: Sabre API Integration Made Easy
Boost Efficiency: Sabre API Integration Made Easy
 
Revolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM GridRevolutionize Your Field Service Management with FSM Grid
Revolutionize Your Field Service Management with FSM Grid
 

Kafka for Real-Time Replication between Edge and Hybrid Cloud

  • 1. How to Deploy a Kafka Cluster in Production onto Your Desk or Anywhere! IoT Use Cases, Architectures, and Live Demo for Data in Motion powered by Apache Kafka Kai Waehner Field CTO kai.waehner@confluent.io linkedin.com/in/kaiwaehner confluent.io kai-waehner.de @KaiWaehner Dominik Pilat VP of Customer Success & Field CTO d.pilat@hivecell.com linkedin.com/in/dpilat hivecell.com
  • 2. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
  • 3. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
  • 4. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Real-time Data beats Slow Data. Manufacturing Sensor diagnostics MES/ERP Integration Reporting Edge Computing Condition Monitoring Predictive Maintenance Quality Assurance Logistics Supply Chain Inventory management Track & Trace Context-specific routing Cybersecurity Threat detection Intrusion detection Incident response Fraud detection
  • 5. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures AWS Cloud Outage hit Disney World Visitors… https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/
  • 6. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Apache Kafka is the Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 6
  • 7. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures MQTT Data Historian STREAM PROCESSING CONNECTORS Example Architecture for Data in Motion with Event Streaming ksqlDB KStreams Bi-directional real-time integration between the OT and IT worlds APP Oracle DB ORACLE CDC SOURCE CONNECTOR APP
  • 8. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Global Event Streaming Streaming Replication between Kafka Clusters Bridge to Databases, Data Lakes, Apps, APIs, SaaS Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters for RPO=0 and RTO~0 Stream Data Globally with Replication and Cluster Linking 8
  • 9. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures What is the “Edge” for Kafka? • Edge is NOT a data center • Kafka clients AND the Kafka broker(s) • Offline business continuity • Often 100+ locations • Low-footprint and low-touch • Hybrid integration Example: Single broker, 1 GB Ram 100 MB/sec
  • 10. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Cost-efficiency Low latency
  • 11. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Shipping Industry Marine, Oil Transport, Vessel Fleet, Shipping Line, Drones Real-time Operations, Logistics, Predictive Maintenance, Security Customer Data Crew, Cargo Vessel Data Fuel Consumption, Speed, Planned Maintenance Automatic Identification System (AIS) Unique Identification, Position, Course, Weather, Draft Drone Data Deliveries, Survey/Inspection of Assets such as Oil Rigs, Pipelines, Offshore Turbines Edge Analytics Bidirectional Edge to Cloud Integration Data Ingestion Stream Processing Data Integration Logistics Track&Trace Routing Monitoring Alerting Command&Control Batch Analytics Reporting Machine Learning Backend Systems Oracle, SAP, OSIsoft PI, etc. X = Event Streaming X = Other Technologies Bi-Directional Hybrid Cloud Replication
  • 12. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Royal Caribbean - Offline Edge for Swimming Retail Stores https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
  • 13. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Devon Energy Oil & Gas Industry Improve drilling and well completion operations Edge stream processing/analytics + closed-loop control ready Vendor agnostic (pumping, wireline, coil, offset wells, drilling operations, producing wells) Replication to the cloud in real-time at scale Cloud agnostic (AWS, GCP, Azure) Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
  • 14. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures CRM Real-Time Location System (RTLS) for Asset Tracking Customer data Advanced Planning and Scheduling (APS) Manager Get report API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Event Streaming in Hybrid Industrial IoT Wavelength Public Cloud VPC Campus 5G Telco Carrier 5G Smart Factory Edge Computing
  • 15. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Analytics Database Data Lake CRM Kafka as Data Historian Continuous real time data ingestion, processing and monitoring 24/7 at scale MQTT Modbus Firewall Siemens S7 Kafka Connect Integration | Storage | Processing | Access | Cloud | Security Storage Kafka Streams / ksqlDB OPC-UA Stateless + Stateful Kafka Connect Kafka Clients Java, C++. Python, Go, … REST Proxy SCADA DCS ERP MES Syslog File X Y Z A B C Cloud Factory
  • 16. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events
  • 17. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 3 4 5 6 7 2 1 Time Sensor Events 1 6 Condition Monitoring (Temperature Spikes) Stateless Filter Above-Threshold Events Streams builder .stream(”temperature-sensor") .filter((key, sensor-data) -> sensor-data.temperature > 100) .to(”temperature-spikes");
  • 18. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Stateful Correlation of Events CREATE TABLE anomaly_detection AS SELECT temperature_spike_id, COUNT(*) AS total_spikes, AVG(temperature) AS avg_temperature FROM sensor-data WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY temperature_spike_id EMIT CHANGES;
  • 19. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Real-time Machine Learning CREATE STREAM anomaly_detection AS SELECT sensor_id, detect_anomaly(sensor_values) FROM machine; TensorFlow model embedded in User Defined Function (UDF)
  • 20. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
  • 21. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Edge Cloud Texas Bavaria Hybrid Data Streaming: Edge and Cloud
  • 22. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Texas Texas data Streams of real time events Texas data Streams of real time events Bavaria data Bavaria Hybrid Data Streaming: Edge and Cloud
  • 23. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
  • 24. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Conditions of edge environments • No data closet • Or data closet is full, overheating and under powered • Limited or unreliable power supply • Limited number of ports • No IT staff on site • Hundreds or thousands of locations • Limited network bandwidth • Demand for compute power always increasing
  • 25. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Requirements for an edge solution No special equipment for power, air conditioning or networking No technicians required onsite to install and configure Start with smallest footprint possible to demonstrate ROI Easily add more compute power as workload expands Run distributed software for containers, messaging and machine learning Monitor, manage and upgrade centrally, even when behind a firewall
  • 26. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Hivecell ✓ No special equipment ✓ No technicians required ✓ Start with smallest footprint ✓ Easily add more compute power ✓ Run distributed software ✓ Monitor, manage and update remotely
  • 27. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Agenda • Data in Motion at the Edge and in Hybrid Architectures • Live Demo – Hybrid Data Streaming with Confluent on Hivecell and Confluent Cloud • Designing an Edge Solution • Next steps for Kafka at the Edge
  • 28. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures The Rise of Data in Motion for Edge and Hybrid Scenarios 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka 29 2017
  • 29. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures I N V E S T M E N T & T I M E V A L U E 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management... 30
  • 30. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Car Engine Car Self-driving Car Confluent completes Apache Kafka. Cloud-native. Everywhere.
  • 31. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Container orchestration at the edge Enterprise streaming at the edge HIVECELL = PAAS AT THE EDGE Hyperconverged infrastructure at the edge HCI
  • 32. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures EDGE AS A SERVICE. EVERYTHING YOU NEED IS INCLUDED. • Kubernetes and Confluent as a service • Framework licenses/support/updates • Hive Control - Provision/monitor system • Operating system (OS) support/updates • Hardware support/replacements • Evergreen Hardware: upgrades every 3 years User just uses Kafka. At the edge.
  • 33. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures 34 Hivecell and Confluent enable Kafka at the Edge. Reliable. Scalable. Simple. Conclusion
  • 34. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures Stay in touch! 35 Confluent Cloud cnfl.io/confluent-cloud 1 hour free edge consulting session with Dominik Pilat https://hivecell.com/contact/
  • 35. Kai Waehner Field CTO Confluent kai.waehner@confluent.io @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect! Dominik Pilat VP of Customer Success Field CTO Hivecell d.pilat@hivecell.com linkedin.com/in/dpilat hivecell.com