Building realtime data applications that can seamlessly run and integrate data across On Prem, and multiple public cloud vendors. How Hybrid Cloud can help tackle regulatory requirements for Data Sovereignty, Stressed Exit, and operational resilience.

1. CitiBank Tech Talk - Hybrid Cloud
Solution Engineering, Confluent UK
Sept 2023

2. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Chun-Sing, Chan Certified for
2
More about me
10y+ Data Integration & Data warehousing
• Driving behavior Analysis
• Customer return prediction
• Retail promotion / churn analysis
• AIoT edge projects
UK : Superdrug, TalkTalk, New Look etc.
HK : HSBC, OOCL, HKTVmail etc.

3. Table of Contents
3
1. Hybrid Cloud & Multi Cloud
The initiative
2. Confluent kafka in 2023
What is it in a 5-min recap
3. Confluent - Simply Anywhere
Data Sovereignty at your hands
4. Cluster Linking & Schema
Linking
An asynchronous, multi-cloud and
multi-region solution
5. What do we prevent?
Lock-in VS SaaS
6. Summary / Q&A

4. Hybrid Cloud & Multi Cloud

5. Cloud Outages happen
5
AWS Azure GCP
Dec 2021: An unexplained
AWS outage created
business disruptions all
day
(CNBC)
Nov 2020: A Kinesis
outage brought down
over a dozen AWS
services for 17 hours in
us-east-1
(CRN, AWS)
Apr 1 2021: Some critical
Azure services were
unavailable for an hour
(Coralogix)
Sept 2018: South Central
US region was
unavailable for over a day
(The Register)
Nov 2021: An outage that
affected Home Depot,
Snap, Spotify, and Etsy
(Bloomberg)

6. Outages hurt business
performance
6
A region may be down
for multiple hours–up to
a day–based on
historical experience
Cloud region
has an outage
The applications in that
region that run your
business go offline
Mission-critical
applications fail
Customers are unable to
place orders, discover
products, receive
service, etc.
Customer
Impact
Revenue is lost directly
from the inability to do
business during
downtime, and
indirectly by damaging
brand image and
customer trust
Financial
Impact

7. Failure Types
7
Transient Failures Permanent Failures (Data Loss)
Transient failures in data-centers
or clusters are common and
worth protecting against for
business continuity purposes.
Regional outages are rare but
still worth protecting against for
mission critical systems.
Outages are typically transient
but occasionally permanent.
Users accidentally delete topics,
human error occurs.
If your data is unrecoverable and
mission critical, you need an
additional complementary
solution.

8. Failure Scenarios
Data-Center /
Regional Outages
Platform Failures Human Error
Data-Centers have single
points of failure associated
with hardware resulting in
associated outages.
Regional Outages arise
from failures in the
underlying cloud provider.
People delete topics,
clusters and worse.
Unexpected behaviour arise
from standard operations
and within the CI/CD
pipeline.
Load is applied unevenly or
in short bursts by batch
processing systems.
Performance limitations
arise unexpectedly.
Bugs occur in Kafka,
Zookeeper and associated
systems.

9. As you add more cloud services these problems
get exponentially worse
9
Cloud Cloud
● More brittle
interconnections to
individually set up and
manage
● Complex new cloud
networking and security
considerations
● New compliance and data
sovereignty challenges
On-premises

10. Confluent Kafka in 5min

11. Real-time
Data
A Sale
A shipment
A Trade
A Customer
Experience
Confluent: A New Paradigm for Data in Motion
“We need to shift our thinking from everything
at rest, to everything in motion.” —
Real-Time Stream Processing
Rich Front-End
Customer Experiences
Real-Time Backend
Operations

12. From a Giant Mess to a Connected
Real-time Enterprise
12

13. Confluent Unifies All of your Environments
into a Single, Real-time Data Plane
13
● Eliminate brittle
interconnections with
Confluent’s platform for
data in motion
● Synchronize all of your
environments in real-time
to build innovative
applications faster
● Address networking and
security challenges once
instead of every time a
new connection is made
Cloud Cloud
On-premises

14. Build innovative real-time applications
with global consistency
1
4
Datastores
Web / Mobile
PoS Systems
SaaS
Applications
IoT Sensors
Legacy Apps
and Systems
Machine data Common schemas and a lightweight SQL
syntax for stream processing simplify
real-time application development
Join, enrich, transform
and analyze data in
motion using SQL
Ensure data is
consistent and
real-time across all
global systems
ML Engines
BI Tools
SIEM and
Observability tools
Data lakes and
warehouses
Real-time alerts
and dashboards
Applications
Accelerate your cloud journey
and build new real-time
applications faster
Eliminate batch jobs that result in
stale information being used to
run your business
Easily integrate existing
systems with 160+
out-of-the-box connectors

15. What is Apache Flink used for?
15
Transactions
Logs
IoT
Interactions
Events
…
Messaging
Systems
Files
Databases
Key/Value Stores
Analytics
Event-driven
Applications
ETL
Data
Integration
Messaging
Systems
Files
Databases
Key/Value Stores
Applications

16. Confluent - Simply Anywhere

17. ● Public Cloud
Leverage a fully managed
service with Confluent Cloud
● Private Cloud & On-Prem
Deploy on premises with
Confluent Platform
● Hybrid Cloud & Multicloud
Seamlessly build a persistent
bridge from datacenter to cloud
and across clouds with Cluster
Linking
17
Confluent provides true deployment flexibility
to span all of your environments
Seamlessly connect your data and apps Everywhere they reside
Confluent offers true deployment flexibility to support hybrid and multi-cloud architectures

18. Connectors are a key
For our approach with CDWs, and more broadly
120+
PRE-BUILT
CONNECTORS
Legacy data
infrastructure
Modern, cloud-based
technologies
Azure cloud data warehouse
Synapse

19. Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams

20. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka is Much More than Messaging
Stream
Processing
Pub/Sub
Messaging
ETL
Connectors
Spark
Flink
Beam
TIBCO
IBM MQ
RabbitMQ
Mulesoft
Talend
Informatica
+ Distributed clustered
storage
+ Streaming platform
+ Exactly Once
+ Designed for the Cloud
+ Inter DC
replication
+ Schema
evolution
20

21. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Everywhere: Confluent provides deployment
flexibility to span all of your environments
SELF-MANAGED SOFTWARE
Confluent Platform
The Enterprise Distribution of Apache Kafka
Deploy on-premises or in your private cloud
VM
FULLY MANAGED SERVICE
Confluent Cloud
Cloud-native service for Apache Kafka
Available on the leading public clouds

22. Rapid Pace of Innovation to Enable Enterprises
November 2022
April 2022
CP 7.0 (based on AK 3.0)
Resilience
● Cluster Linking (GA)
○ Source Initiated Links
Flexible DevOps Automation
● Confluent for Kubernetes 2.2
○ Expanded API operations
○ Enhanced scalability with
Shrink API
Management & Monitoring
● Control Center
○ Reduced Infrastructure
Mode
Streaming Database
● ksqlDB 0.21
○ Foreign key table joins
○ DATE & TIME types
November 2021 July 2022
CP 7.1 (based on AK 3.1)
Resilience
● Schema Linking
Flexible DevOps Automation
● Confluent for Kubernetes 2.3
○ Multi-Region Clusters
support
○ Enhanced API operations
Performance & Elasticity
● Expanded options for Tiered
Storage
Management & Monitoring
● New Health+ intelligent alerts
○ Broker Latency (preview),
Connectors, & ksqlDB
Streaming Database
● ksqlDB 0.23
○ Pull queres on streams
○ Custom schema selection 22
April 2023
CP 7.3 (based on AK 3.3)
Resilience
● Multi-Region Clusters
○ Replica Rack Mixing
Flexible DevOps Automation
● Confluent for Kubernetes 2.5
○ Overlays for Pod resources
Integration
● IBM MQ Premium Connectors
for z/OS
Streaming Database
● ksqlDB 0.28.2
○ Pause and resume
persistent queries
○ Wildcard Struct references
○ PROTOBUF_NOSR
serialization format
CP 7.2 (based on AK 3.2)
Resilience
● Cluster Linking
○ Flexible Topic Naming
Flexible DevOps Automation
● Confluent for Kubernetes 2.4
○ Source Initiated Cluster
Links
○ Auto-rotation of certs
○ Pod deletion protection
Streaming Database
● ksqlDB 0.26
○ Complex types for
aggregate functions
○ RIGHT joins
○ New JSON functions
CP 7.4 (based on AK 3.4)
Resilience
● Production-ready KRaft for new
clusters
○ Removes dependency on
zookeeper for metadata
management
Flexible DevOps Automation
● Confluent for Kubernetes 2.6
○ Declarative API driven
control plane
Management and Monitoring
● Data Quality Rules with schema
registry
○ Domain Validation Rules
○ Schema migration rules

23. Cluster Linking & Schema
Linking

24. C O N F I D E N T I A L
Global Data Mesh
Bridge to Cloud
Cluster Linking
Architectures + Use Cases
multi cloud
hybrid cloud
multi region
High-er Availability &
Disaster Recovery
edge
Data Sharing between
Teams, LOBs, Orgs
Edge Aggregation
Cluster Migration

25. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Cluster link
Source topic
Name
Configs
Messages
Consumer Offsets
ACLs
Mirror topic
Name
same name
Configs
synced per Confluent
best practices
Messages
mirrored
Identical partitions &
offsets
Consumer Offsets
Synced (optional)
Filterable by:
* consumer group
name/prefix
ACLs
Synced (optional)
Filterable by:
* topic name/prefix
* principal name .
Source Cluster Destination Cluster
Consumers
Consumers
Producers

26. 26
Cluster Linking
Cluster Linking, built into Confluent Platform
and Confluent Cloud allows you to directly
connect clusters together mirroring topics from
one cluster to another.
Cluster Linking makes it easier to build
multi-cluster, multi-cloud, and hybrid cloud
deployments.
Active cluster
Consumers
Producers
clicks
clicks
Topics
DR cluster
clicks
clicks
Mirror Topics
Cluster Link
Primary Region DR Region

27. 27
Schema Linking
Schema Linking, built into Schema Registry
allows you to directly connect Schema Registry
clusters together mirroring subjects or entire
contexts.
Contexts, introduced alongside Schema Linking
allows you to create namespaces within Schema
Registry which ensures mirrored subjects don’t
run into schema naming clashes.
Active cluster
Consumers
Producers
clicks
clicks
Schemas
DR cluster
clicks
clicks
Mirror Schemas
Schema Link
Primary Region DR Region
Consumers
Producers

28. 28
Prefixing
Prefixing allows you to add a prefix to a topic
and if desired the associated consumer group to
avoid topic and consumer group naming
clashes between the primary and Disaster
Recovery cluster.
This is important when used in an active-active
setup and required to use a two way Cluster Link
strategy which is the recommended approach.
Active cluster
Consumer-Group
clicks
clicks
Topic
DR cluster
clicks
clicks
DR-topic
Cluster Link
Primary Region DR Region
DR-Consumer-Group

29. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Data Mesh in Cloud Hyperscale
29

30. Principle 1: Domain-driven Decentralization
Domain
Domain
Domain
Domain
Microservices and their data belong to domains

31. Domain
Domain
Domain
Domain
Principle 2: Data as a First-Class Product
Objective: Make shared data meaningful, up to date, discoverable,
addressable, trustworthy, secure, so other teams can make good use of it.
• Data is treated
as a true
product, not a
by-product.

32. What we prevented?

33. Reduce TCO by minimizing engineering
time spent on data pipeline projects
3
3
Reduce operational overhead
Free development teams up to build
things related to the core business instead
of work on complex data pipelines.
Minimize vendor lock-in
Leverage geo-replication to mobilize your
data, maintain optionality, and future proof
your technology stack.
Increase efficiency
Consolidate disparate tools and practices
into a single platform, set of APIs, and
trusted vendor.
Maximize the value of cloud
Reduce surprise network and cloud costs
by writing data once and reading it as
many times as necessary.

34. Copyright 2020, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 34
Schema Registry
Connect
REST Proxy
ksqlDB
Control Center
Kafka Brokers
Health+ in parallel with other alerting/monitoring tools