Best Practices for building Hybrid-Cloud Architectures - Hans Jespersen Afternoon opening presentation during Confluent’s streaming event in Paris, presented by Hans Jespersen, VP WW Systems Engineering at Confluent.

1. 1
Best Practices for Building Hybrid-Cloud
Architectures
Hans Jespersen
VP, WW Systems Engineering
Confluent Inc
PARIS - 11 OCTOBER 2018

2. 2
About Me
Description
PHOTO HERE

3. 3
Agenda
The Great “Cloud Shift”
What is a Bridge to Cloud?
What is Confluent Cloud?
Customer Stories
Q&A

4. 4
The Great “Cloud Shift”
A cloud product is not complete
without a cloud migration story.

5. 5
Cloud Migration: A one time thing?

6. 6
In reality, we keep running
Our apps are complex, intricately
tied together.
Some, built to run on legacy,
immovable systems

7. 7
We don’t want to just move.
We want to build for the cloud.

8. 8
DC-1
App
App
App
App App
App
Elasticsearch
MySQL
Oracle Teradata NoSQL Database
Relational Database
Data Warehouse
Blob Storage

9. 9
DC-1
App
App
App
App App
App
Elasticsearch
MySQL
Oracle Teradata NoSQL Database
Relational Database
Data Warehouse
Blob Storage
DC-NY
DC-LON amer-northeast
emea-central

10. 10
App
Oracle
NoSQL Database
Blob Storage

11. 11
What is our bridge?
Confluent Replicator

12. 12
Establish Your Foundation
App
Oracle
NoSQL Database
Blob Storage
1
Deploy Kafka on-premise and your cloud provider of
choice,
Teradata
Confluent Replicator

13. 13
Establish Your Foundation
App
Oracle
NoSQL Database
Blob Storage
2
Create your pipeline and replicate your topics to your cloud
cluster
Teradata
Confluent Replicator

14. 14
Let the Traffic Flow
App
Oracle
NoSQL Database
Blob Storage
3
Migrate app by app, database by
database
App
Topic: dc1users
Topic: dc1clicks

15. 15
DC-1
App
App
App
App App
App
Elasticsearch
MySQL
Oracle Teradata NoSQL Database
Relational Database
Data Warehouse
Blob Storage
DC-NY
DC-LON amer-northeast
emea-central

16. 16
Confluent Replicator vs MirrorMaker
Feature Benefit MirrorMaker Replicator
Data replication
Real-time event streaming between Kafka clusters
and data centers
Yes Yes
Schema replication
Integrate with Confluent Schema Registry for multi-
dc data quality and governance
Yes Yes
Connect replication
Manage data integration across multiple data
centers
Yes Yes
Flexible topic selection
Select topics with white-lists, black-lists and regular
expressions
Partial* Yes
Auto-create topics
New topics are automatically detected and
replicated
Partial* Yes
Add new partitions
New partitions are automatically detected and
replicated
No Yes
Configuration replication
Identical topic configuration between the two
clusters
No Yes
Single Message Transformations Filter, modify and route events on the fly No Yes
Auto-scale
Scale replication processes as Kafka traffic
increases with a single configuration.
No Yes
Active-active replication
Redirect events to avoid infinite replication loops in
active-active configurations
No Yes
Aggregate cluster
Replicate from multiple clusters into one
(aggregator / hub-and-spoke pattern). One
management point for replicating more than a
single cluster.
No Yes
End-to-end monitoring and management
Manage and monitor replication via Control Center
UI
No Yes
License Apache Proprietary
https://docs.confluent.io/current/multi-dc/mirrormaker.html

17. 17
Multi-Cloud, DR and Bridge to Cloud
Enables multi-DC deployments of Apache Kafka
● On-premises: Datacenter to Datacenter
● Hybrid: On-Premise to Cloud
● Cloud: Cloud Region to Cloud Region
● Multi-Cloud: Cloud Provider to Cloud Provider
Confluent Replicator

18. 18
Confluent Platform as a service

19. 19
The most popular platform
Publish and subscribe, store and
process streams of data based on
Kafka’s latest release
Develop faster with the leading streaming platform
19
Fine tune for scale
Custom partitioning
Log compaction
Highest throughput
Best performance of any streaming
data service
Up to 1GB/s write, 2GB/s read
throughput
Retain and replay more data
Up to infinite retention

20. 20

21. 21
Introducing Confluent CloudTM
A Way to Liberate Developers.
Confidential

22. 22
01
Freedom to deploy
Kafka in any public
cloud
22
02
Access the rich
Kafka ecosystem
03
Develop faster using
the leading streaming
platform
04
Liberate yourself from
the operations burden
Introducing Confluent Cloud™

23. 23
● Built on open source Apache Kafka
Confluent Cloud™ Gives You
Vendor Independence.
* support for AWS and GCP available today
● Port your existing pipelines regardless of
initial starting point
● No proprietary technology so not tied to
a single provider*

24. 24
Leading open source
streaming platform
Access the Rich Kafka Ecosystem
Integrates with leading data
systems and apps
Multi-Language Support
24
Includes Kafka Core + Streams and
Connect API’s
Vibrant and growing open source
community
Support for leading languages
Bring more applications to the
platform
200+ pre-built connectors

25. 25
Fastest Way to Develop Code is to not code at all

26. 26
4 ways Confluent
Cloud minimizes your
operations burden
No Infrastructure
Setup
No Cluster
Tuning
No Managing
Upgrades
Deploy Kafka
in Minutes
Liberate yourself from the operations burden
26

27. 27
Feature Benefit Apache Kafka
Confluent Open
Source
Confluent
Enterprise
Confluent
Cloud™
Apache Kafka
High throughput, low latency, high availability, secure
distributed streaming platform
Kafka Connect API
Advanced API for connecting external sources/destinations
into Kafka
Kafka Streams API
Simple library that enables streaming application
development within the Kafka framework
Additional Clients
Supports non-Java clients; C, C++, Python, .NET and several
others
REST Proxy
Provides universal access to Kafka from any network
connected device via HTTP
Schema Registry
Central registry for the format of Kafka data – guarantees
all data is always consumable
Pre-Built Connectors
HDFS, JDBC, Elasticsearch, Amazon S3 and other
connectors fully certified and supported by Confluent
JMS Client
Support for legacy Java Message Service (JMS)
applications consuming and producing directly from Kafka
Confluent Control Center
Enables easy connector management, monitoring and
alerting for a Kafka cluster
Cloud Management
Console
Simply and effectively manage a Confluent Cloud cluster N/A
Auto Data Balancer Rebalancing data across cluster to remove bottlenecks Automatic
Replicator
Multi-datacenter replication simplifies and automates MDC
Kafka clusters
Support
Enterprise class support to keep your Kafka environment
running at top performance
Community Community 24x7x365 24x7x365
Confluent Cloud: Part of the Confluent Family

28. 28
● New project
● Low budget
● Not business-critical
● Wants fast /
frictionless access
● Production apps
● Budget available
● Business-critical
● Wants reliable
service levels
Developer VP Engineering
Cloud Professional & Cloud Enterprise
Cloud Professional Cloud Enterprise
Unleash developer velocity Streaming service for all
your enterprise applications

29. 29
Use Cases: from Born Digital to Brick and Mortar Companies
Retail (Click Stream Analytics, Recommendations, Fraud Detection, Loyalty Apps)
AdTech (Mobile In App Advertising)
Online Gaming (Betting Engines, Regulatory Compliance)
FinTech (Insurance, Peer to Peer Lending, Loan Processing)
Log Aggregation (Application Logging, Syslog, CyberSecurity/SIEM)
IoT (Sensor data ingestion, Connected Factory, Connected Car, Predictive Maintenance)
Micro-Services (State Persistence, Streaming Micro Services)
Legacy Middleware Modernization (Real-time ETL, MQ/JMS replacement, Mainframe Offload)
Anything you can do with Kafka you can do (quicker/easier) with Confluent Cloud!

30. 30
Q&A

31. 31
● Hosted service that reduces the burden on your
operations team
● No Vendor Lock In - take all your streaming
applications to another Kafka provider
● Get Visibility - Operations Dashboard gives you full
transparency into your Kafka pipelines - by topic, by
cluster, by app
● Supported by the team who created Kafka and has the
most experience working in large scale streaming
Confluent Cloud™ for the Enterprise: Scale without the Staff
31