SlideShare a Scribd company logo
1 of 77
AWS Immersion Day
Mapfre - Confluent
Elena Molina
Partner Technical Trainer
Salvatore Alessandro
Enterprise Solutions Engineer
Introduction
09:30 - 09:45
01 Who are we?
What is Confluent cloud?
Introduction to Cloud Features
09:45 - 10:30
02 Fully Manage Connectors, KsqlDB, Cluster Linking.
Break
10:30 - 10:45
03 15 Min - Coffee Break
Hybrid Workshop Introduction
10:45 - 11:00
04 Infrastructure, architecture and use case introduction.
Hybrid Workshop Hands On - Part 1
11:00 - 12:15
05
Hands-on: Build a bridge between on prem and the cloud using
Cluster linking.
2
Agenda
Break
12:15 - 12:30
06 15 Min - Coffee Break
Hybrid Workshop Hands On - Part 2
12:30 - 13:45
07
Hands-on: Create a streaming app using KsqlDB and build a bridge
between the cloud and on prem using Cluster Linking.
Multi-regional Disaster Recovery on AWS
13:45 - 14:15
08
Multi-regional Disaster Recovery with Confluent Cluster Linking on AWS
Lunch and Networking
14:15 - 16:00
09
3
Confluent Mapfre Team
Much more than a platform
Enterprise
Account
Manager
Marcos Yanez
Enterprise
Solutions
Engineer
Salvo
Alessandro
PS &
Education
Gonzalo Garcia
Customer
Success
Manager
Asier
Fernández
Partner
Technical
Trainer
Elena Molina
01. Introduction
4
Loyalty Rewards
Curbside Pickup
Trending Now
Popular on Netflix
Top Picks for Joshua
Created by the founders of
Confluent while at LinkedIn
Apache Kafka has ushered in
the data streaming era…
>70%
of the Fortune 500
>100,000+
Organizations
>41,000
Kafka Meetup Attendees
>200
Global Meetup Groups
>750
Kafka Improvement Proposals (KIPs)
>12,000
Jiras for Apache Kafka
>32,000
Stack Overflow Questions
Real-time Trades
Ride ETA
Personalized Recommendations
The need for a cloud-native, data streaming platform
Connecting all your apps, systems and data into a central nervous system
PUTTING KAFKA IN THE CLOUD…
ISN’T JUST PUTTING KAFKA IN THE CLOUD.
Managing
infrastructure
Development
Resources
Security &
Governance
Global Availability
Trying to get here on your own with Open Source Kafka has
significant challenges…
Why Confluent is the world’s most trusted data streaming
platform
Focus & Expertise
Only company focused on data in motion:
● Founded in 2014 by the Original Creators of
Apache Kafka
● Over 80% of Kafka commits are by
Confluent employees
● Advised on thousands of real-world Kafka
deployments across a wide range of
patterns & industries
Building and supporting a world class product:
● >9 million engineering hours spent
building building our product
● We internally manage >15,000 clusters
(and counting) in Confluent Cloud
● Over 1 million cumulative hours of Kafka
expertise within Confluent support &
services
Execution at Scale
Confluent Cloud
COMPLETE EVERYWHERE
CLOUD-NATIVE
Confluent makes data streaming easy
Open Source
Real-time
Data
Integration
Stream
Processing
Enterprise
Security &
Governance
…100s more
features
Kora Engine
Multi-cloud SaaS & Private Cloud
Open Source
Apache Kafka
Kafka completely re-
architected to
be Cloud-native
A Complete,
enterprise-grade
Data-in-Motion
Platform
Fully managed
service and software,
available Everywhere
Cloud-Native
Elastic, resilient
and performant,
powered by the
Kora Engine
Kora Architecture
NETWORK
COMPUTE
AZ AZ AZ
Cells
Cells
Cells
OBJECT
STORAGE
CUSTOMERS
Multi-Cloud Networking & Routing Tier
Metadata
Durability Audits
METRICS & OBSERVABILITY
CONNECT
PROCESSING
GOVERNANCE
Data Balancing
Health Checks
Real-time
feedback
data
Other Confluent Cloud Services
GLOBAL CONTROL PLANE
Kora: the Cloud-Native Engine for Apache Kafka
Serverless
● Elastic scaling up & down from 0
to GBps
● Auto capacity mgmt, load
balancing, and upgrades
Infinite Storage
● Store data cost- effectively at any
scale without growing compute
Resilience
● Multi-AZ and multi-region
replication
● Durability self-validation
High Availability
● 99.99% SLA
● Multi-region / AZ availability across
cloud providers
● Patches deployed in Confluent
Cloud before Apache Kafka
Network Flexibility
● Public, VPC, and Private Link
● Seamlessly link across clouds
and on-prem with Cluster
Linking
Confluent is so much more than Apache Kafka
Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform
Enterprise-grade Security
RBAC | Audit Logs | Encryption |
BYOK | Private Networking
Stream Governance
Schema Registry & Validation |
Stream Lineage | Stream Catalog |
Stream Sharing
Complete Engagement Model
Data in Motion Blueprint
Business Case Justification
TCO | ROI | Risk
Management & Monitoring
Cloud UI | Metrics API |
Control Center | Health+
Flexible DevOps Automation
Admin REST APIs | Terraform APIs |
Confluent for K8s | Ansible Playbooks
Efficient
Operations at Scale
Production-stage
Prerequisites
Partnership for
Business Success
Multi-language Development
Non-Java Clients |
REST Proxy | MQTT Proxy
Stream Processing & Integration
Connectors | Flink | ksqlDB |
Stream Designer
Unrestricted Developer
Productivity
High Availability
99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters
Infinite Storage
Infinite Storage | Tiered Storage
Elastic Scalability
Expand | Shrink | Self-Balancing Clusters
Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine
Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
Hybrid and Multicloud
Cluster Linking | Replicator
Self-managed software
Kubernetes | VMs | Bare Metal
Fully managed cloud service
AWS | Azure | GCP
Committer-driven
Expertise
Training Partners
Professional
Services
Enterprise
Support
OPERATOR
DEVELOPER ARCHITECT EXECUTIVE
Complete
Go above &
beyond Kafka
with all the
essential tools for
a complete data
streaming
platform
Process
Stream
Connect
Govern
Share
Secure
Confluent is so much more than Apache Kafka
Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform
Enterprise-grade Security
RBAC | Audit Logs | Encryption |
BYOK | Private Networking
Stream Governance
Schema Registry & Validation |
Stream Lineage | Stream Catalog |
Stream Sharing
Complete Engagement Model
Data in Motion Blueprint
Business Case Justification
TCO | ROI | Risk
Management & Monitoring
Cloud UI | Metrics API |
Control Center | Health+
Flexible DevOps Automation
Admin REST APIs | Terraform APIs |
Confluent for K8s | Ansible Playbooks
Efficient
Operations at Scale
Production-stage
Prerequisites
Partnership for
Business Success
Multi-language Development
Non-Java Clients |
REST Proxy | MQTT Proxy
Stream Processing & Integration
Connectors | Flink | ksqlDB |
Stream Designer
Unrestricted Developer
Productivity
High Availability
99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters
Infinite Storage
Infinite Storage | Tiered Storage
Elastic Scalability
Expand | Shrink | Self-Balancing Clusters
Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine
Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
Hybrid and Multicloud
Cluster Linking | Replicator
Self-managed software
Kubernetes | VMs | Bare Metal
Fully managed cloud service
AWS | Azure | GCP
Committer-driven
Expertise
Training Partners
Professional
Services
Enterprise
Support
OPERATOR
DEVELOPER ARCHITECT EXECUTIVE
Everywhere
Connect your
data in real time
with a platform
that spans from
on-prem to cloud
and across clouds
Any Cloud. Any Region.
Everywhere You Want To Be
Build the right data architecture for your business
SELF-MANAGED SOFTWARE
Confluent Platform
The Enterprise Distribution of Apache Kafka
Deploy on-premises or in your private
cloud
VM
FULLY MANAGED CLOUD SERVICE
Confluent Cloud
Cloud-native Data Streaming Platform built by
the founders of Apache Kafka
Available on the leading public clouds
Confluent is so much more than Apache Kafka
Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform
Enterprise-grade Security
RBAC | Audit Logs | Encryption |
BYOK | Private Networking
Stream Governance
Schema Registry & Validation |
Stream Lineage | Stream Catalog |
Stream Sharing
Complete Engagement Model
Data in Motion Blueprint
Business Case Justification
TCO | ROI | Risk
Management & Monitoring
Cloud UI | Metrics API |
Control Center | Health+
Flexible DevOps Automation
Admin REST APIs | Terraform APIs |
Confluent for K8s | Ansible Playbooks
Efficient
Operations at Scale
Production-stage
Prerequisites
Partnership for
Business Success
Multi-language Development
Non-Java Clients |
REST Proxy | MQTT Proxy
Stream Processing & Integration
Connectors | Flink | ksqlDB |
Stream Designer
Unrestricted Developer
Productivity
High Availability
99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters
Infinite Storage
Infinite Storage | Tiered Storage
Elastic Scalability
Expand | Shrink | Self-Balancing Clusters
Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine
Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
Hybrid and Multicloud
Cluster Linking | Replicator
Self-managed software
Kubernetes | VMs | Bare Metal
Fully managed cloud service
AWS | Azure | GCP
Committer-driven
Expertise
Training Partners
Professional
Services
Enterprise
Support
OPERATOR
DEVELOPER ARCHITECT EXECUTIVE
02. Introduction to Cloud Features
24
Fully Managed Connectors
25
Confluent Loves Your Existing Systems
… 200+
connectors
Other
Systems
Other
Systems
Kafka
Connect
Confluent
Kafka
Connect
https://www.confluent.io/hub/
Using fully managed connectors is the fastest, most
efficient way to break data silos
27
Custom-built connector
● Costly to allocate resources to
design, build, test, and maintain
non-differentiated data
integration components
● Delays time-to-value, taking
up to 3-6+ engineering months
to develop
● Perpetual management and
maintenance increases tech
debt and risk of downtime
● Pre-built but requires manual
installation / config efforts to
set-up and deploy connectors
● Perpetual management and
maintenance of connectors that
leads to ongoing tech debt
● Risk of downtime and business
disruption due to connector /
Connect cluster related issues
● Streamlined configurations and
on-demand provisioning of your
connectors
● Eliminates operational
overhead and management
complexity with seamless
scaling and load balancing
● Reduced risk of downtime with
Confluent Cloud’s 99.95% SLA for
all your mission critical use cases
Accelerated time-to-value • Increased developer productivity • Reduced operational burden
Self-managed connector Fully managed connector
Fully Managed
Connectors
28
Confluent Cloud’s portfolio of 70+
fully managed connectors
enables you to boost developer
productivity, eliminate
operational burden, and
accelerate time to value on your
data in motion journey.
Eliminate operational burdens of
self-managing connectors and
reduce total cost of ownership
Operate your business in real time
by modernizing your data systems
Accelerate your entire pipeline
development process with Stream
Designer, SMTs, and data preview
Only Confluent offers 70+ expert-built, fully managed
connectors across the entire stack
29
Connector configurations Connector configurations
Connector development Connector development Connector development
Connector testing Connector testing Connector testing
Connector updates Connector updates Connector updates
Connector support Connector support Connector support
Connect cluster scaling Connect cluster scaling Connect cluster scaling
Connect worker configs Connect worker configs Connect worker configs
Connect internal topics Connect internal topics Connect internal topics
Schema registry Schema registry Schema registry
Monitoring and security Monitoring and security Monitoring and security
Load balancing Load balancing Load balancing
Connect plugin installation Connect plugin installation Connect plugin installation
Other Kafka hosted
services
Apache Kafka -
Kafka Connect
Confluent Fully
Managed Connectors
Ease of use
You Manage
Provider Managed
Connectors
Connect
Workers
*Streamlined configurations
with ability for granular
controls if needed
Provider
managed
features
Connector configurations*
Connect your entire business with just a few clicks
30
70+
fully
managed
connectors
Amazon S3
Amazon Redshift
Amazon DynamoDB
Google Cloud
Spanner
AWS Lambda
Amazon SQS
Amazon Kinesis
Azure Service Bus
Azure Event Hubs
Azure Synapse
Analytics
Azure Blob
Storage
Azure Functions Azure Data Lake
Google
BigTable
200+
pre-built
connectors
Confluent HUB
31
Easily browse connectors by:
• Source vs Sinks
• Confluent vs Partner supported
• Commercial vs Free
• Available in Confluent Cloud
confluent.io/hub
Confluent Hub -
Connector Page
32
• Source or Sink ?
• Free or Commercial ?
• Supported by Confluent or partners
• Can download plugin
• Link to documentation
• License type
• Link to source code (if open source)
confluent.io/hub
Confluent Cloud
Fully Managed Connectors
33
Easily browse connectors by:
• Source vs Sinks
• Confluent vs Partner supported
• Commercial vs Free
• Available in Confluent Cloud
Custom connectors
34
Every organization has their unique data
architecture which requires additional flexibility
Home-grown systems
and custom applications
need custom-built
connectors to break
data silos
Pre-built connectors in
the Kafka ecosystem
need additional
modifications to fit your
specific work context
Lack of managed
connector options for the
long tail of less popular
data systems and apps
Custom Connectors
36
Break any data silo without
needing to manage Kafka
Connect infrastructure by
bringing your own connector
plugins to Confluent Cloud
Quickly connect to any data system
using your own Kafka Connect plugins
without code changes
Ensure high availability & performance
using logs and metrics to monitor the
health of your connectors and workers
Eliminate operational burden of
provisioning and perpetually managing
low-level connector infrastructure
Bring your own connectors and let Confluent
provision and manage connector infrastructure
Responsible for
connectors
Connector infrastructure resources
Connect workers Connect clusters Connect logs
Worker health
metrics
Connector plugins (BYO)
Custom-built
(original) connectors
Connector management
(i.e. upgrades, patching, support)
Modified connectors
(i.e. custom SMTs)
Partner-built
connectors
Community-built
connectors
Users
Confluent Cloud
Responsible for
Connect infrastructure
Infrastructure management & support
KsqlDB
39
What is Stream Processing?
40
Connect to a driver
immediately after
rideshare request
Instant notification
upon package
delivery
Real-time notification
once a new patch is
available
Automatic alert once
fraudulent activity has
been detected
Transportation Retail Technology Banking
Customers demand immediacy in every aspect of their
lives through real-time applications
These applications require reacting to events that happen
in your business immediately
The same charge
was rapidly made
multiple times
A valuable
customer is
beginning the
checkout process
A new patch for
bug fixes was
released
Events occur everywhere across an organization
A rider searches
for an available
rideshare
It requires filtering, joining, and aggregating streams
of events into something more useful
Customer Data
Transaction Data
Payment
Data
Geo Location Data
Security
Services Data
Stream data using Confluent’s connector portfolio
and Easily build real-time data pipelines with KsqlDB
Modern, cloud-based data systems
Legacy data systems
Oracle
Database
ksqlDB
Mainframes
Applications
Cloud-native / SaaS apps
Azure Synapse
Analytics
Expensive,
custom-built
integrations
Expensive,
custom-built
integrations
Expensive,
custom-built
integrations
Source
Connectors
Expensive,
custom-built
integrations
Expensive,
custom-built
integrations
Sink
Connectors
What is KsqlDB?
Why KsqlDB?
45
DB
CONNECTOR
CONNECTOR
APP
APP
DB
STREAM
PROCESSING
CONNECTOR APP
DB
2
3
4
1
With all these moving parts, stream processing
architectures can become quite complex
ksqlDB simplifies the underlying architecture to make it ea
applications
DB
APP
APP
DB
PULL
PUSH
CONNECTORS
STREAM PROCESSING
MATERIALIZED VIEWS
ksqlDB
1
2
APP
Compute Storage
CREATE TABLE activePromotions AS
SELECT rideId,
qualifyPromotion(distanceToDst) AS promotion
FROM locations
GROUP BY rideId
EMIT CHANGES
ksqlDB Kafka
Build a complete real-time application
with just a few SQL statements
Easily Build Real-
Time Applications
ksqlDB is a streaming database
purpose-built for developing real-
time applications that leverage
stream processing, enabling you to
build a complete real-time
application with just a few SQL
statements
Aggregate Joins
Filters
User-Defined
Functions
Push & Pull
Query Support
Embedded
Connectors
49
Filters
CREATE STREAM high_readings AS
SELECT sensor,
reading,
FROM readings
WHERE reading > 41
EMIT CHANGES;
Joins
CREATE STREAM enriched_readings AS
SELECT reading, area, brand_name,
FROM readings
INNER JOIN brands b
ON b.sensor = readings.sensor
EMIT CHANGES;
Aggregate CREATE TABLE avg_readings AS
SELECT sensor,
AVG(reading) AS location
FROM readings
GROUP BY sensor
EMIT CHANGES;
Cluster Linking
52
Building a seamless bridge between on-prem and cloud
has become mission-critical...
On-prem
Datacenters &
Private Clouds
Public Clouds
across AWS, Azure,
& GCP
As companies adopt
hybrid architectures,
building a bridge
between on-prem and
cloud environments
has become critical to
reliably connecting
and sharing data
across the entire
business
5
4
Cloud
Provider 1
Cloud
Provider 2
● More brittle
interconnections to
individually set up and
manage
● Complex new cloud
networking and security
considerations
● New compliance and data
sovereignty challenges
On-premises
...but hybrid cloud today is challenging
Companies need a simpler way to link their hybrid
environments and share data in real-time
Cloud
Provider 1
Cloud
Provider 2
On-premises
55
Hybrid architectures
require a highly
available, consistent,
and secure real-time
bridge between on-
prem and cloud
environments to provide
teams with real-time,
self-service access to
data wherever it resides
Accelerate the enterprise journey to cloud with Cluster
Linking
Cluster Linking accelerates the
enterprise journey to cloud by
securely, reliably, and effortlessly
creating a real-time bridge between
cloud and on-prem environments
• Bridge to cloud: Seamlessly build
hybrid & multi-cloud architectures
that are secure and reliable
without add’l systems to manage
• Cluster migrations: Facilitate
smooth migrations with no data
loss and minimal downtime
• Source-initiated links: Securely
share data between Confluent
Platform and Confluent Cloud
without opening on-prem firewalls
to the cloud
Cluster Linking enables self-service
access to data in real-time across the
business with globally connected
clusters that provide perfectly and
reliably mirrored data
• Self-service access: Provide
access to real-time data across
different regions, clouds,
environments, and teams
• Offset preservation: Keep your
data in sync with native offset
preservation without additional
tooling
• Ease of use: Create a link from one
cluster to another with a single
command or API call
ETL/Batch
process
Data
request
Data
warehouses
Data
Stewards
DBs
SaaS
DB
Apps
Data consumer
Data consumer
Data consumer
Data consumer
Slow
Provide self-service access to real-time data across
all environments
Real-time
Self-
service
Fast
...
...
...
Cluster Linking reduces TCO and
operational burdens with seamless
and cost-effective data replication
across Kafka clusters everywhere they
need to reside
• Decreased costs: Eliminate the
need to manage and maintain
additional infrastructure required
by Connect-based solutions (e.g.,
MM2)
• Simplified Architecture: Remove
architectural complexity and
technical debt
• Reduced manual processes:
Minimize operational burdens and
risks with a single, globally-
consistent connection that’s easier
to monitor and secure
From: Connect-based solutions like MirrorMaker 2
To: Built-in geo-replication with Cluster Linking
Reduce TCO and operational burdens for data replication
across Kafka clusters
Complex architecture
Offsets are not preserved
Cluster 2
Cluster 1
MirrorMaker 2
MirrorMaker 2
Simplified architecture
Globally consistent offsets
Cluster 2
Cluster 1
Effortlessly mirror from one cluster to another
Mirror Topics
byte-for-byte copy
same offsets
same configs
same name
read-only
Source
Topics
Consumer Group
Offsets
Filter by name
ACLs
Filter by name
Destination
Mirror Topics
Consumer Group
Offsets
(optional)
ACLs
(optional)
Start in Minutes
one CLI or API call
to link or mirror
Reliable and
Performant
using Kafka’s inter-
broker protocol
AK 2.4+ CP 5.4+ and CC CP 7.0+ and CC
Cluster Link
What strategies benefit from multiple clusters?
60
Hybrid Cloud & Multi-
Cloud
Stream data between your datacenter
and cloud. Each cluster serves clients in
its environment.
Team Sandboxing
Give each team its own cluster to
separate concerns. Every team controls
its own destiny.
Disaster Recovery
Create a failover cluster in a separate
location that is ready to go when
disaster strikes.
Edge Computing
Put clusters at the edge in order to
minimize latency, lower network cost,
or buffer data when network
connection is unreliable.
Global Kafka
Footprint
Stream events between continents for
a global data strategy.
Cluster Migration
To modernize a cluster, or to
move to cloud.
03. 15 Min Break
61
04. Hybrid Workshop Introduction
62
Why this workshop?
What are we trying to achieve?
Business Use Case & Technical Architecture
63
Stream Processing
Joins streams in real-time
to control the stock
Stream of shipments
that arrive
Stream of purchases from online and/or
physical stores (Example RDBMS or
Mainframe)
Real-time
Inventory
KSQL
High Level Business Use Case: Real-time Retail
Private Cloud
Deploy on-premises in your
datacenter with Confluent
Enterprise
Public Cloud
Migrate to or adopt cloud
at your own pace with fully-
managed Confluent Cloud
Hybrid Cloud
Build a persistent bridge b/n
datacenter and cloud with
Confluent Cluster Linking
Private Cloud
Industry’s Only Solution for Hybrid Kafka
Workshop Architecture
Confluent is so much more than Apache Kafka
Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform
Enterprise-grade Security
RBAC | Audit Logs | Encryption |
BYOK | Private Networking
Stream Governance
Schema Registry & Validation |
Stream Lineage | Stream Catalog |
Stream Sharing
Complete Engagement Model
Data in Motion Blueprint
Business Case Justification
TCO | ROI | Risk
Management & Monitoring
Cloud UI | Metrics API |
Control Center | Health+
Flexible DevOps Automation
Admin REST APIs | Terraform APIs |
Confluent for K8s | Ansible Playbooks
Efficient
Operations at Scale
Production-stage
Prerequisites
Partnership for
Business Success
Multi-language Development
Non-Java Clients |
REST Proxy | MQTT Proxy
Stream Processing & Integration
Connectors | Flink | ksqlDB |
Stream Designer
Unrestricted Developer
Productivity
High Availability
99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters
Infinite Storage
Infinite Storage | Tiered Storage
Elastic Scalability
Expand | Shrink | Self-Balancing Clusters
Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine
Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
Hybrid and Multicloud
Cluster Linking | Replicator
Self-managed software
Kubernetes | VMs | Bare Metal
Fully managed cloud service
AWS | Azure | GCP
Committer-driven
Expertise
Training Partners
Professional
Services
Enterprise
Support
OPERATOR
DEVELOPER ARCHITECT EXECUTIVE
Your Personalized Environment
68
On Premise x N
dc1
Confluent Cloud
dc2
dc3 dc4
HQ
Aggregate
Cluster
Workshop Architecture
One shared Confluent Cloud
Dedicated Cluster
One shared Confluent Platform
On- Prem (name: HQ -
HeadQuarter)
N x On-prem Confluent
Platform data center (name:
dcN)
Each attendee has his or her
own environment.
Let’s Begin
71
Please select one of the environments that have been
shared with you
Password:
Feb9pass123!
05. Hybrid Workshop Hands On - Part 1
73
06. 15 Min Break
74
07. Hybrid Workshop Hands On - Part 2
75
Get started for FREE
No credit card needed
$400 free credits
confluent.io/get-started
AWS Immersion Day Mapfre   -   Confluent

More Related Content

What's hot

Cloud 101 - What is the Cloud?
Cloud 101 - What is the Cloud?Cloud 101 - What is the Cloud?
Cloud 101 - What is the Cloud?
RapidScale
 
Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio
Improving Data Locality for Spark Jobs on Kubernetes Using AlluxioImproving Data Locality for Spark Jobs on Kubernetes Using Alluxio
Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio
Alluxio, Inc.
 
Data Center Migration to the AWS Cloud
Data Center Migration to the AWS CloudData Center Migration to the AWS Cloud
Data Center Migration to the AWS Cloud
Tom Laszewski
 

What's hot (20)

Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)Unified Big Data Processing with Apache Spark (QCON 2014)
Unified Big Data Processing with Apache Spark (QCON 2014)
 
Cloud Architecture - Multi Cloud, Edge, On-Premise
Cloud Architecture - Multi Cloud, Edge, On-PremiseCloud Architecture - Multi Cloud, Edge, On-Premise
Cloud Architecture - Multi Cloud, Edge, On-Premise
 
Cloud 101 - What is the Cloud?
Cloud 101 - What is the Cloud?Cloud 101 - What is the Cloud?
Cloud 101 - What is the Cloud?
 
Building an Effective Data Warehouse Architecture
Building an Effective Data Warehouse ArchitectureBuilding an Effective Data Warehouse Architecture
Building an Effective Data Warehouse Architecture
 
Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio
Improving Data Locality for Spark Jobs on Kubernetes Using AlluxioImproving Data Locality for Spark Jobs on Kubernetes Using Alluxio
Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio
 
Cloud Testing
Cloud TestingCloud Testing
Cloud Testing
 
Scalable Filesystem Metadata Services with RocksDB
Scalable Filesystem Metadata Services with RocksDBScalable Filesystem Metadata Services with RocksDB
Scalable Filesystem Metadata Services with RocksDB
 
Building-a-Data-Lake-on-AWS
Building-a-Data-Lake-on-AWSBuilding-a-Data-Lake-on-AWS
Building-a-Data-Lake-on-AWS
 
Data Center Migration to the AWS Cloud
Data Center Migration to the AWS CloudData Center Migration to the AWS Cloud
Data Center Migration to the AWS Cloud
 
ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱ksqlDB로 시작하는 스트림 프로세싱
ksqlDB로 시작하는 스트림 프로세싱
 
Building large scale transactional data lake using apache hudi
Building large scale transactional data lake using apache hudiBuilding large scale transactional data lake using apache hudi
Building large scale transactional data lake using apache hudi
 
Data warehouse : Order Management
Data warehouse : Order ManagementData warehouse : Order Management
Data warehouse : Order Management
 
Big Query - Utilizing Google Data Warehouse for Media Analytics
Big Query - Utilizing Google Data Warehouse for Media AnalyticsBig Query - Utilizing Google Data Warehouse for Media Analytics
Big Query - Utilizing Google Data Warehouse for Media Analytics
 
Cloud Migration Workshop
Cloud Migration WorkshopCloud Migration Workshop
Cloud Migration Workshop
 
Hadoop World 2011: Replacing RDB/DW with Hadoop and Hive for Telco Big Data -...
Hadoop World 2011: Replacing RDB/DW with Hadoop and Hive for Telco Big Data -...Hadoop World 2011: Replacing RDB/DW with Hadoop and Hive for Telco Big Data -...
Hadoop World 2011: Replacing RDB/DW with Hadoop and Hive for Telco Big Data -...
 
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's DataFrom Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data
 
Azure storage
Azure storageAzure storage
Azure storage
 
IBM - Introduction to Cloudant
IBM - Introduction to CloudantIBM - Introduction to Cloudant
IBM - Introduction to Cloudant
 
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
Maximize Greenplum For Any Use Cases Decoupling Compute and Storage - Greenpl...
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 

Similar to AWS Immersion Day Mapfre - Confluent

Similar to AWS Immersion Day Mapfre - Confluent (20)

DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdfDIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for KubernetesConfluent Operator as Cloud-Native Kafka Operator for Kubernetes
Confluent Operator as Cloud-Native Kafka Operator for Kubernetes
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
 
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans JespersenBest Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersen
 
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWSBridge to Cloud: Using Apache Kafka to Migrate to AWS
Bridge to Cloud: Using Apache Kafka to Migrate to AWS
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles
 
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
Kubernetes Connectivity to Cloud Native Kafka | Christina Lin and Evan Shorti...
 
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdfDIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf
 
Reinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun RaoReinventing Kafka in the Data Streaming Era - Jun Rao
Reinventing Kafka in the Data Streaming Era - Jun Rao
 
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6New Features in Confluent Platform 6.0 / Apache Kafka 2.6
New Features in Confluent Platform 6.0 / Apache Kafka 2.6
 
Elastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using ConfluentElastically Scaling Kafka Using Confluent
Elastically Scaling Kafka Using Confluent
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent RamièreAu delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
 
Jelastic for Hosting & MSP
Jelastic for Hosting & MSPJelastic for Hosting & MSP
Jelastic for Hosting & MSP
 
Agile Integration Workshop
Agile Integration WorkshopAgile Integration Workshop
Agile Integration Workshop
 
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
 
Webinar: Dealing with automation tool overload!
Webinar: Dealing with automation tool overload!Webinar: Dealing with automation tool overload!
Webinar: Dealing with automation tool overload!
 
Transform into a Cloud-First Business with SAP on AWS and Capgemini’s Cloud C...
Transform into a Cloud-First Business with SAP on AWS and Capgemini’s Cloud C...Transform into a Cloud-First Business with SAP on AWS and Capgemini’s Cloud C...
Transform into a Cloud-First Business with SAP on AWS and Capgemini’s Cloud C...
 
Hashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public SectorHashicorp Vault - OPEN Public Sector
Hashicorp Vault - OPEN Public Sector
 

More from confluent

More from confluent (20)

Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 
The Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data StreamsThe Playful Bond Between REST And Data Streams
The Playful Bond Between REST And Data Streams
 
The Journey to Data Mesh with Confluent
The Journey to Data Mesh with ConfluentThe Journey to Data Mesh with Confluent
The Journey to Data Mesh with Confluent
 

Recently uploaded

Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Marc Lester
 

Recently uploaded (20)

UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
COMPUTER AND ITS COMPONENTS PPT.by naitik sharma Class 9th A mittal internati...
 
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
Optimizing Operations by Aligning Resources with Strategic Objectives Using O...
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with GraphGraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
GraphSummit Milan & Stockholm - Neo4j: The Art of the Possible with Graph
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
Entropy, Software Quality, and Innovation (presented at Princeton Plasma Phys...
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
Auto Affiliate  AI Earns First Commission in 3 Hours..pdfAuto Affiliate  AI Earns First Commission in 3 Hours..pdf
Auto Affiliate AI Earns First Commission in 3 Hours..pdf
 
Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024Food Delivery Business App Development Guide 2024
Food Delivery Business App Development Guide 2024
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 

AWS Immersion Day Mapfre - Confluent

  • 1. AWS Immersion Day Mapfre - Confluent Elena Molina Partner Technical Trainer Salvatore Alessandro Enterprise Solutions Engineer
  • 2. Introduction 09:30 - 09:45 01 Who are we? What is Confluent cloud? Introduction to Cloud Features 09:45 - 10:30 02 Fully Manage Connectors, KsqlDB, Cluster Linking. Break 10:30 - 10:45 03 15 Min - Coffee Break Hybrid Workshop Introduction 10:45 - 11:00 04 Infrastructure, architecture and use case introduction. Hybrid Workshop Hands On - Part 1 11:00 - 12:15 05 Hands-on: Build a bridge between on prem and the cloud using Cluster linking. 2 Agenda Break 12:15 - 12:30 06 15 Min - Coffee Break Hybrid Workshop Hands On - Part 2 12:30 - 13:45 07 Hands-on: Create a streaming app using KsqlDB and build a bridge between the cloud and on prem using Cluster Linking. Multi-regional Disaster Recovery on AWS 13:45 - 14:15 08 Multi-regional Disaster Recovery with Confluent Cluster Linking on AWS Lunch and Networking 14:15 - 16:00 09
  • 3. 3 Confluent Mapfre Team Much more than a platform Enterprise Account Manager Marcos Yanez Enterprise Solutions Engineer Salvo Alessandro PS & Education Gonzalo Garcia Customer Success Manager Asier Fernández Partner Technical Trainer Elena Molina
  • 5. Loyalty Rewards Curbside Pickup Trending Now Popular on Netflix Top Picks for Joshua Created by the founders of Confluent while at LinkedIn Apache Kafka has ushered in the data streaming era… >70% of the Fortune 500 >100,000+ Organizations >41,000 Kafka Meetup Attendees >200 Global Meetup Groups >750 Kafka Improvement Proposals (KIPs) >12,000 Jiras for Apache Kafka >32,000 Stack Overflow Questions Real-time Trades Ride ETA Personalized Recommendations
  • 6.
  • 7.
  • 8.
  • 9. The need for a cloud-native, data streaming platform Connecting all your apps, systems and data into a central nervous system
  • 10. PUTTING KAFKA IN THE CLOUD… ISN’T JUST PUTTING KAFKA IN THE CLOUD.
  • 11. Managing infrastructure Development Resources Security & Governance Global Availability Trying to get here on your own with Open Source Kafka has significant challenges…
  • 12. Why Confluent is the world’s most trusted data streaming platform Focus & Expertise Only company focused on data in motion: ● Founded in 2014 by the Original Creators of Apache Kafka ● Over 80% of Kafka commits are by Confluent employees ● Advised on thousands of real-world Kafka deployments across a wide range of patterns & industries Building and supporting a world class product: ● >9 million engineering hours spent building building our product ● We internally manage >15,000 clusters (and counting) in Confluent Cloud ● Over 1 million cumulative hours of Kafka expertise within Confluent support & services Execution at Scale
  • 14. Confluent makes data streaming easy Open Source Real-time Data Integration Stream Processing Enterprise Security & Governance …100s more features Kora Engine Multi-cloud SaaS & Private Cloud Open Source Apache Kafka Kafka completely re- architected to be Cloud-native A Complete, enterprise-grade Data-in-Motion Platform Fully managed service and software, available Everywhere
  • 15. Cloud-Native Elastic, resilient and performant, powered by the Kora Engine Kora Architecture NETWORK COMPUTE AZ AZ AZ Cells Cells Cells OBJECT STORAGE CUSTOMERS Multi-Cloud Networking & Routing Tier Metadata Durability Audits METRICS & OBSERVABILITY CONNECT PROCESSING GOVERNANCE Data Balancing Health Checks Real-time feedback data Other Confluent Cloud Services GLOBAL CONTROL PLANE
  • 16. Kora: the Cloud-Native Engine for Apache Kafka Serverless ● Elastic scaling up & down from 0 to GBps ● Auto capacity mgmt, load balancing, and upgrades Infinite Storage ● Store data cost- effectively at any scale without growing compute Resilience ● Multi-AZ and multi-region replication ● Durability self-validation High Availability ● 99.99% SLA ● Multi-region / AZ availability across cloud providers ● Patches deployed in Confluent Cloud before Apache Kafka Network Flexibility ● Public, VPC, and Private Link ● Seamlessly link across clouds and on-prem with Cluster Linking
  • 17. Confluent is so much more than Apache Kafka Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform Enterprise-grade Security RBAC | Audit Logs | Encryption | BYOK | Private Networking Stream Governance Schema Registry & Validation | Stream Lineage | Stream Catalog | Stream Sharing Complete Engagement Model Data in Motion Blueprint Business Case Justification TCO | ROI | Risk Management & Monitoring Cloud UI | Metrics API | Control Center | Health+ Flexible DevOps Automation Admin REST APIs | Terraform APIs | Confluent for K8s | Ansible Playbooks Efficient Operations at Scale Production-stage Prerequisites Partnership for Business Success Multi-language Development Non-Java Clients | REST Proxy | MQTT Proxy Stream Processing & Integration Connectors | Flink | ksqlDB | Stream Designer Unrestricted Developer Productivity High Availability 99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters Infinite Storage Infinite Storage | Tiered Storage Elastic Scalability Expand | Shrink | Self-Balancing Clusters Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Committer-driven Expertise Training Partners Professional Services Enterprise Support OPERATOR DEVELOPER ARCHITECT EXECUTIVE
  • 18. Complete Go above & beyond Kafka with all the essential tools for a complete data streaming platform Process Stream Connect Govern Share Secure
  • 19. Confluent is so much more than Apache Kafka Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform Enterprise-grade Security RBAC | Audit Logs | Encryption | BYOK | Private Networking Stream Governance Schema Registry & Validation | Stream Lineage | Stream Catalog | Stream Sharing Complete Engagement Model Data in Motion Blueprint Business Case Justification TCO | ROI | Risk Management & Monitoring Cloud UI | Metrics API | Control Center | Health+ Flexible DevOps Automation Admin REST APIs | Terraform APIs | Confluent for K8s | Ansible Playbooks Efficient Operations at Scale Production-stage Prerequisites Partnership for Business Success Multi-language Development Non-Java Clients | REST Proxy | MQTT Proxy Stream Processing & Integration Connectors | Flink | ksqlDB | Stream Designer Unrestricted Developer Productivity High Availability 99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters Infinite Storage Infinite Storage | Tiered Storage Elastic Scalability Expand | Shrink | Self-Balancing Clusters Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Committer-driven Expertise Training Partners Professional Services Enterprise Support OPERATOR DEVELOPER ARCHITECT EXECUTIVE
  • 20. Everywhere Connect your data in real time with a platform that spans from on-prem to cloud and across clouds
  • 21. Any Cloud. Any Region. Everywhere You Want To Be
  • 22. Build the right data architecture for your business SELF-MANAGED SOFTWARE Confluent Platform The Enterprise Distribution of Apache Kafka Deploy on-premises or in your private cloud VM FULLY MANAGED CLOUD SERVICE Confluent Cloud Cloud-native Data Streaming Platform built by the founders of Apache Kafka Available on the leading public clouds
  • 23. Confluent is so much more than Apache Kafka Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform Enterprise-grade Security RBAC | Audit Logs | Encryption | BYOK | Private Networking Stream Governance Schema Registry & Validation | Stream Lineage | Stream Catalog | Stream Sharing Complete Engagement Model Data in Motion Blueprint Business Case Justification TCO | ROI | Risk Management & Monitoring Cloud UI | Metrics API | Control Center | Health+ Flexible DevOps Automation Admin REST APIs | Terraform APIs | Confluent for K8s | Ansible Playbooks Efficient Operations at Scale Production-stage Prerequisites Partnership for Business Success Multi-language Development Non-Java Clients | REST Proxy | MQTT Proxy Stream Processing & Integration Connectors | Flink | ksqlDB | Stream Designer Unrestricted Developer Productivity High Availability 99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters Infinite Storage Infinite Storage | Tiered Storage Elastic Scalability Expand | Shrink | Self-Balancing Clusters Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Committer-driven Expertise Training Partners Professional Services Enterprise Support OPERATOR DEVELOPER ARCHITECT EXECUTIVE
  • 24. 02. Introduction to Cloud Features 24
  • 26. Confluent Loves Your Existing Systems … 200+ connectors Other Systems Other Systems Kafka Connect Confluent Kafka Connect https://www.confluent.io/hub/
  • 27. Using fully managed connectors is the fastest, most efficient way to break data silos 27 Custom-built connector ● Costly to allocate resources to design, build, test, and maintain non-differentiated data integration components ● Delays time-to-value, taking up to 3-6+ engineering months to develop ● Perpetual management and maintenance increases tech debt and risk of downtime ● Pre-built but requires manual installation / config efforts to set-up and deploy connectors ● Perpetual management and maintenance of connectors that leads to ongoing tech debt ● Risk of downtime and business disruption due to connector / Connect cluster related issues ● Streamlined configurations and on-demand provisioning of your connectors ● Eliminates operational overhead and management complexity with seamless scaling and load balancing ● Reduced risk of downtime with Confluent Cloud’s 99.95% SLA for all your mission critical use cases Accelerated time-to-value • Increased developer productivity • Reduced operational burden Self-managed connector Fully managed connector
  • 28. Fully Managed Connectors 28 Confluent Cloud’s portfolio of 70+ fully managed connectors enables you to boost developer productivity, eliminate operational burden, and accelerate time to value on your data in motion journey. Eliminate operational burdens of self-managing connectors and reduce total cost of ownership Operate your business in real time by modernizing your data systems Accelerate your entire pipeline development process with Stream Designer, SMTs, and data preview
  • 29. Only Confluent offers 70+ expert-built, fully managed connectors across the entire stack 29 Connector configurations Connector configurations Connector development Connector development Connector development Connector testing Connector testing Connector testing Connector updates Connector updates Connector updates Connector support Connector support Connector support Connect cluster scaling Connect cluster scaling Connect cluster scaling Connect worker configs Connect worker configs Connect worker configs Connect internal topics Connect internal topics Connect internal topics Schema registry Schema registry Schema registry Monitoring and security Monitoring and security Monitoring and security Load balancing Load balancing Load balancing Connect plugin installation Connect plugin installation Connect plugin installation Other Kafka hosted services Apache Kafka - Kafka Connect Confluent Fully Managed Connectors Ease of use You Manage Provider Managed Connectors Connect Workers *Streamlined configurations with ability for granular controls if needed Provider managed features Connector configurations*
  • 30. Connect your entire business with just a few clicks 30 70+ fully managed connectors Amazon S3 Amazon Redshift Amazon DynamoDB Google Cloud Spanner AWS Lambda Amazon SQS Amazon Kinesis Azure Service Bus Azure Event Hubs Azure Synapse Analytics Azure Blob Storage Azure Functions Azure Data Lake Google BigTable 200+ pre-built connectors
  • 31. Confluent HUB 31 Easily browse connectors by: • Source vs Sinks • Confluent vs Partner supported • Commercial vs Free • Available in Confluent Cloud confluent.io/hub
  • 32. Confluent Hub - Connector Page 32 • Source or Sink ? • Free or Commercial ? • Supported by Confluent or partners • Can download plugin • Link to documentation • License type • Link to source code (if open source) confluent.io/hub
  • 33. Confluent Cloud Fully Managed Connectors 33 Easily browse connectors by: • Source vs Sinks • Confluent vs Partner supported • Commercial vs Free • Available in Confluent Cloud
  • 35. Every organization has their unique data architecture which requires additional flexibility Home-grown systems and custom applications need custom-built connectors to break data silos Pre-built connectors in the Kafka ecosystem need additional modifications to fit your specific work context Lack of managed connector options for the long tail of less popular data systems and apps
  • 36. Custom Connectors 36 Break any data silo without needing to manage Kafka Connect infrastructure by bringing your own connector plugins to Confluent Cloud Quickly connect to any data system using your own Kafka Connect plugins without code changes Ensure high availability & performance using logs and metrics to monitor the health of your connectors and workers Eliminate operational burden of provisioning and perpetually managing low-level connector infrastructure
  • 37. Bring your own connectors and let Confluent provision and manage connector infrastructure Responsible for connectors Connector infrastructure resources Connect workers Connect clusters Connect logs Worker health metrics Connector plugins (BYO) Custom-built (original) connectors Connector management (i.e. upgrades, patching, support) Modified connectors (i.e. custom SMTs) Partner-built connectors Community-built connectors Users Confluent Cloud Responsible for Connect infrastructure Infrastructure management & support
  • 38.
  • 40. What is Stream Processing? 40
  • 41. Connect to a driver immediately after rideshare request Instant notification upon package delivery Real-time notification once a new patch is available Automatic alert once fraudulent activity has been detected Transportation Retail Technology Banking Customers demand immediacy in every aspect of their lives through real-time applications
  • 42. These applications require reacting to events that happen in your business immediately The same charge was rapidly made multiple times A valuable customer is beginning the checkout process A new patch for bug fixes was released Events occur everywhere across an organization A rider searches for an available rideshare
  • 43. It requires filtering, joining, and aggregating streams of events into something more useful Customer Data Transaction Data Payment Data Geo Location Data Security Services Data
  • 44. Stream data using Confluent’s connector portfolio and Easily build real-time data pipelines with KsqlDB Modern, cloud-based data systems Legacy data systems Oracle Database ksqlDB Mainframes Applications Cloud-native / SaaS apps Azure Synapse Analytics Expensive, custom-built integrations Expensive, custom-built integrations Expensive, custom-built integrations Source Connectors Expensive, custom-built integrations Expensive, custom-built integrations Sink Connectors
  • 45. What is KsqlDB? Why KsqlDB? 45
  • 46. DB CONNECTOR CONNECTOR APP APP DB STREAM PROCESSING CONNECTOR APP DB 2 3 4 1 With all these moving parts, stream processing architectures can become quite complex
  • 47. ksqlDB simplifies the underlying architecture to make it ea applications DB APP APP DB PULL PUSH CONNECTORS STREAM PROCESSING MATERIALIZED VIEWS ksqlDB 1 2 APP
  • 48. Compute Storage CREATE TABLE activePromotions AS SELECT rideId, qualifyPromotion(distanceToDst) AS promotion FROM locations GROUP BY rideId EMIT CHANGES ksqlDB Kafka Build a complete real-time application with just a few SQL statements Easily Build Real- Time Applications ksqlDB is a streaming database purpose-built for developing real- time applications that leverage stream processing, enabling you to build a complete real-time application with just a few SQL statements Aggregate Joins Filters User-Defined Functions Push & Pull Query Support Embedded Connectors
  • 49. 49 Filters CREATE STREAM high_readings AS SELECT sensor, reading, FROM readings WHERE reading > 41 EMIT CHANGES;
  • 50. Joins CREATE STREAM enriched_readings AS SELECT reading, area, brand_name, FROM readings INNER JOIN brands b ON b.sensor = readings.sensor EMIT CHANGES;
  • 51. Aggregate CREATE TABLE avg_readings AS SELECT sensor, AVG(reading) AS location FROM readings GROUP BY sensor EMIT CHANGES;
  • 53. Building a seamless bridge between on-prem and cloud has become mission-critical... On-prem Datacenters & Private Clouds Public Clouds across AWS, Azure, & GCP As companies adopt hybrid architectures, building a bridge between on-prem and cloud environments has become critical to reliably connecting and sharing data across the entire business
  • 54. 5 4 Cloud Provider 1 Cloud Provider 2 ● More brittle interconnections to individually set up and manage ● Complex new cloud networking and security considerations ● New compliance and data sovereignty challenges On-premises ...but hybrid cloud today is challenging
  • 55. Companies need a simpler way to link their hybrid environments and share data in real-time Cloud Provider 1 Cloud Provider 2 On-premises 55 Hybrid architectures require a highly available, consistent, and secure real-time bridge between on- prem and cloud environments to provide teams with real-time, self-service access to data wherever it resides
  • 56. Accelerate the enterprise journey to cloud with Cluster Linking Cluster Linking accelerates the enterprise journey to cloud by securely, reliably, and effortlessly creating a real-time bridge between cloud and on-prem environments • Bridge to cloud: Seamlessly build hybrid & multi-cloud architectures that are secure and reliable without add’l systems to manage • Cluster migrations: Facilitate smooth migrations with no data loss and minimal downtime • Source-initiated links: Securely share data between Confluent Platform and Confluent Cloud without opening on-prem firewalls to the cloud
  • 57. Cluster Linking enables self-service access to data in real-time across the business with globally connected clusters that provide perfectly and reliably mirrored data • Self-service access: Provide access to real-time data across different regions, clouds, environments, and teams • Offset preservation: Keep your data in sync with native offset preservation without additional tooling • Ease of use: Create a link from one cluster to another with a single command or API call ETL/Batch process Data request Data warehouses Data Stewards DBs SaaS DB Apps Data consumer Data consumer Data consumer Data consumer Slow Provide self-service access to real-time data across all environments Real-time Self- service Fast ... ... ...
  • 58. Cluster Linking reduces TCO and operational burdens with seamless and cost-effective data replication across Kafka clusters everywhere they need to reside • Decreased costs: Eliminate the need to manage and maintain additional infrastructure required by Connect-based solutions (e.g., MM2) • Simplified Architecture: Remove architectural complexity and technical debt • Reduced manual processes: Minimize operational burdens and risks with a single, globally- consistent connection that’s easier to monitor and secure From: Connect-based solutions like MirrorMaker 2 To: Built-in geo-replication with Cluster Linking Reduce TCO and operational burdens for data replication across Kafka clusters Complex architecture Offsets are not preserved Cluster 2 Cluster 1 MirrorMaker 2 MirrorMaker 2 Simplified architecture Globally consistent offsets Cluster 2 Cluster 1
  • 59. Effortlessly mirror from one cluster to another Mirror Topics byte-for-byte copy same offsets same configs same name read-only Source Topics Consumer Group Offsets Filter by name ACLs Filter by name Destination Mirror Topics Consumer Group Offsets (optional) ACLs (optional) Start in Minutes one CLI or API call to link or mirror Reliable and Performant using Kafka’s inter- broker protocol AK 2.4+ CP 5.4+ and CC CP 7.0+ and CC Cluster Link
  • 60. What strategies benefit from multiple clusters? 60 Hybrid Cloud & Multi- Cloud Stream data between your datacenter and cloud. Each cluster serves clients in its environment. Team Sandboxing Give each team its own cluster to separate concerns. Every team controls its own destiny. Disaster Recovery Create a failover cluster in a separate location that is ready to go when disaster strikes. Edge Computing Put clusters at the edge in order to minimize latency, lower network cost, or buffer data when network connection is unreliable. Global Kafka Footprint Stream events between continents for a global data strategy. Cluster Migration To modernize a cluster, or to move to cloud.
  • 61. 03. 15 Min Break 61
  • 62. 04. Hybrid Workshop Introduction 62
  • 63. Why this workshop? What are we trying to achieve? Business Use Case & Technical Architecture 63
  • 64. Stream Processing Joins streams in real-time to control the stock Stream of shipments that arrive Stream of purchases from online and/or physical stores (Example RDBMS or Mainframe) Real-time Inventory KSQL High Level Business Use Case: Real-time Retail
  • 65. Private Cloud Deploy on-premises in your datacenter with Confluent Enterprise Public Cloud Migrate to or adopt cloud at your own pace with fully- managed Confluent Cloud Hybrid Cloud Build a persistent bridge b/n datacenter and cloud with Confluent Cluster Linking Private Cloud Industry’s Only Solution for Hybrid Kafka
  • 67. Confluent is so much more than Apache Kafka Complete: Go beyond Kafka with all the essential tools for a complete data streaming platform Enterprise-grade Security RBAC | Audit Logs | Encryption | BYOK | Private Networking Stream Governance Schema Registry & Validation | Stream Lineage | Stream Catalog | Stream Sharing Complete Engagement Model Data in Motion Blueprint Business Case Justification TCO | ROI | Risk Management & Monitoring Cloud UI | Metrics API | Control Center | Health+ Flexible DevOps Automation Admin REST APIs | Terraform APIs | Confluent for K8s | Ansible Playbooks Efficient Operations at Scale Production-stage Prerequisites Partnership for Business Success Multi-language Development Non-Java Clients | REST Proxy | MQTT Proxy Stream Processing & Integration Connectors | Flink | ksqlDB | Stream Designer Unrestricted Developer Productivity High Availability 99.99% SLA | Multi-AZ Clusters | Multi-Region Clusters Infinite Storage Infinite Storage | Tiered Storage Elastic Scalability Expand | Shrink | Self-Balancing Clusters Cloud Native: The 10x Apache Kafka® service: elastic, resilient and performant, powered by the Kora Engine Everywhere: Connect your data in real time with a platform that spans from on-prem to cloud and across clouds Hybrid and Multicloud Cluster Linking | Replicator Self-managed software Kubernetes | VMs | Bare Metal Fully managed cloud service AWS | Azure | GCP Committer-driven Expertise Training Partners Professional Services Enterprise Support OPERATOR DEVELOPER ARCHITECT EXECUTIVE
  • 69. On Premise x N dc1 Confluent Cloud dc2 dc3 dc4 HQ Aggregate Cluster
  • 70. Workshop Architecture One shared Confluent Cloud Dedicated Cluster One shared Confluent Platform On- Prem (name: HQ - HeadQuarter) N x On-prem Confluent Platform data center (name: dcN) Each attendee has his or her own environment.
  • 72. Please select one of the environments that have been shared with you Password: Feb9pass123!
  • 73. 05. Hybrid Workshop Hands On - Part 1 73
  • 74. 06. 15 Min Break 74
  • 75. 07. Hybrid Workshop Hands On - Part 2 75
  • 76. Get started for FREE No credit card needed $400 free credits confluent.io/get-started