The document outlines the features and enhancements of the Confluent Platform 5.4, emphasizing managed and self-managed solutions, security improvements, and global resilience in disaster recovery. It discusses enhancements in roles and permissions, monitoring capabilities, and data compatibility through schema validation. Additionally, it highlights the importance of stream processing and tiered storage for cost-effective data retention and efficiency.

1. 1
Webinar
Product Release Overview
- Confluent Platform 5.4
February 2020

2. 22Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE
COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-STAGE
PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance & Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed

3. 33
Rapid Pace of Innovation to Enable Enterprises
January 2020
CP 5.4 (based on AK 2.4)
Security
● Role-Based Access Control
● Structured Audit Logs
Resilience
● Multi-Region Clusters
Data Compatibility
● Schema Validation
Management & Monitoring
● Control Center
○ RBAC management
○ Replicator monitoring
Performance & Elasticity
● Tiered Storage (preview)
Stream Processing
● ksqlDB features (preview)
April 2019
CP 5.2 (based on AK 2.2)
Developers
● Free single-broker developer
license
● librdkafka and clients 1.0
KSQL
● New query expressions
● GUI enhancements
Replicator
● Schema migration to CCloud
Control Center
● Dynamix broker configuration
● Schema Registry management
● Multi-cluster Connect & KSQL
● Enhanced scalability
July 2018
CP 5.0 (based on AK 2.0)
Security
● AD/LDAP Authorizer
Replicator
● Automatic offset translation
Control Center
● Consumer lag
● View broker configuration
● View topics
● KSQL editor
Ecosystem
● MQTT Proxy
July 2019
CP 5.3 (based on AK 2.3)
Security
● Role-Based Access Control
(preview)
● Secret Protection
DevOps automation
● Kubernetes Operator
● Ansible Playbooks
Management & Monitoring
● Control Center redesigned user
interface
● New CLI

4. 4C O N F I D E N T I A L 4C O N F I D E N T I A L
Enterprise-grade Security

5. 55
Why you need better
authorization?
● Architecting with security is a design priority
● Avoiding unnecessary complexity is key
● As usage of event streaming spreads, native
tools (e.g. Kafka Access Control Lists) for
managing authorization can become complex
● Problem is exacerbated when failing to
standardize security across the platform
Enterprise
Grade Security

6. 66
Role-Based Access
Control
Provides platform-wide security with fine-tuned
granularity
● Granular control of access
permissions, including:
○ Clusters, topics, consumer groups,
connectors
● Efficient management at large scale
○ Delegate authorization management to
true resource owners
● Platform-wide standardization
○ Enforced via GUI, CLI and APIs
○ Enforced across all CP components:
Connect, KSQL, Schema Registry, REST
Proxy, Control Center and MQTT Proxy
Users/
Groups
Roles Resource
scoping
CLI GUI API
Role
Binding
RBAC
authorization

7. 77
Why you need better
visibility?
● Lack of visibility into actions taken by
users/applications
● Difficult to perform forensics to detect
anomalies and identify bad actors
● Failure to comply with regulatory requirements
Enterprise
Grade Security

8. 88
Structured Audit Logs
Enable security traceability and regulatory
compliance
● Detection of abnormal behavior
and potential security threats
○ Capture authorization logs in a set of
dedicated Kafka topics
○ Process and analyze with KSQL, or
offload to external systems (e.g.
Splunk, S3)
● Industry Standardization
○ Uses CloudEvents specification to
define the syntax of the logs
Event Description Category Capture
Default
Authorize An RBAC authorization is
being requested.
MANAGEMENT Yes
CreateTopics A topic is being created. MANAGEMENT Yes
Produce A Kafka producer is
writing a batch of records
to a topic.
PRODUCE No
FetchConsumer A Kafka consumer is
reading a batch of
records from a topic.
CONSUME No
LeaderAndIsr Controller is sending
leader and ISR state to a
broker.
INTERBROKER No
Sample Audit Logs

9. 9C O N F I D E N T I A L 9C O N F I D E N T I A L
Global Resilience

10. 1010
Why you need better disaster
recovery?
● Modern companies have high expectations for
durability, availability, and latency
● Replication based on Kafka Connect (e.g.
Replicator or MirrorMaker 2) come with
operational complexity and require downtime
● Stretch cluster architectures historically came
with a tradeoff: availability vs. performance
Global
Resilience

11. 1111
Multi-Region Clusters
Change the game for disaster recovery for Kafka
● Minimal downtime
○ Automated client failover
● Streamlined DR operations
○ Leverages Kafka’s internal replication
○ No separate Connect clusters
● Single multi-region cluster with
high write throughput
○ Asynchronous replication using
“Observer” replicas
● Low bandwidth costs and high
read throughput
○ Remote consumers read data locally,
directly from Observers
Broker
1
Broker
2
Broker
3
ZK1
Broker
4
Broker
5
Broker
6
Broker
1
Broker
2
ZK2
Client D Client F Client G
Failover site
ZK3
Broker
3
Broker
4
Broker
5
Broker
6
Client A Client B
us-central-1
Client A Client B
automated
client failover
Observer
replicas
us-west-1 us-east-1
Site failure!
“tie-breaker”
datacenter
Single Kafka Cluster

12. 12C O N F I D E N T I A L 12C O N F I D E N T I A L
Data Compatibility

13. 1313
Why you need enhanced
validation of data quality?
● Confluent Schema Registry increase data
compatibility through client-level
“agreements”, but Kafka is unaware
● No programmatic way of enforcing that
producers talk to Schema Registry before
publishing messages to Kafka
● Leads to risk and uncertainty regarding data
quality for large organizations
Data
Compatibility

14. 1414
Schema Validation
Provides a centralized way of controlling data
compatibility
● Certainty and piece of mind at
scale regarding data quality
○ Automated broker-side schema
validation and enforcement
○ Direct interface from the broker to
Confluent Schema Registry
● Granular control over schema
validation
○ Enabled at the topic level
Producer Broker
Schema
Registry
1. Invalid
schema
2. Error
message
confluent.value.schema.validation=true

15. 15C O N F I D E N T I A L 15C O N F I D E N T I A L
GUI-driven Management and
Monitoring

16. 1616
Why these improvements to
Control Center?
● Control Center is rapidly becoming the de
facto user interface for many Confluent
Platform users
● We must ensure that Control Center can
manage and monitor Confluent Platform
comprehensively
● Need for a wide variety of use cases and
supported scale
GUI-driven
Management
and
Monitoring

17. 1717
GUI-driven mgmt for
new CP 5.4 features
● Role-Based Access Control
○ View own permissions, and manage
subordinate role bindings
● Multi-Region Clusters
○ Track Observer replica placement in
each topic view
● Schema Validation
○ Enable at the topic level when
creating or editing topics
RBAC management

18. 1818
Confluent Replicator
integration
● Simplified monitoring for
multi-site replication with the GUI
○ Track key metrics such as throughput
and lag
Replicator monitoring

19. 1919
New aggregate views
● Simplified monitoring and
troubleshooting for Kafka
clusters
○ Cluster Overview: shows overall
status of the Kafka cluster, including
brokers, replicas, partitions and topics
○ Metrics Dashboard: aggregates all
Kafka cluster metrics into a single
page
Cluster Overview
Metrics Dashboard

20. 20C O N F I D E N T I A L 20C O N F I D E N T I A L
Dynamic Performance and Elasticity

21. 2121
Why you need enhanced
scalability and data
efficiency?
● As event streaming spreads, the platform is
required to store larger amounts of data for
longer periods of time
● Kafka’s tight coupling between compute and
storage leads to difficulty to scale the platform
● Longer data retention leads to high storage
costs
Dynamic
Performance
and Elasticity

22. 2222
Tiered Storage (preview)
Enable Kafka with infinite retention cost-effectively
● Infinite retention
○ Older data is offloaded to inexpensive
object storage, accessible at any time
● Reduced storage costs
○ Storage limitations, like capacity and
duration, are effectively uncapped
● Elastic scalability
○ “Lighter” Kafka brokers enable
instantaneous load balancing when
scaling up
Broker
Compute Storage
Clients
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Object Storage

23. 2323
Confluent Server
Enables enterprise features
Required to enable:
● Operator
● RBAC
● Structured Audit Logs
● Multi-Region Clusters
● Schema Validation
● Tiered Storage (preview)
Optional software package
● Deploy CP with Confluent Server
or Apache Kafka
● In-place migration between
Confluent Server and Kafka
Apache Kafka
enterprise capabilities
Confluent Server
Confluent Platform
KSQL
Schema
Registry
REST Proxy
Control Center Replicator MQTT Proxy

24. 24C O N F I D E N T I A L 24C O N F I D E N T I A L
SQL-based Stream Processing

25. 25C O N F I D E N T I A L
KSQL: Simplify the stream processing architecture
One solution for capturing events, stream processing, and
serving both push and pull queries
DB
APP
APP
DB
APP
PULL
PUSH
CONNECTORS
STREAM PROCESSING
STATE STORES
ksqlDB
1 2

26. 2626
Apache Kafka 2.4
If you want to learn what’s included in Apache
Kafka 2.4, we have resources available for you:
● Technical Blog:
https://www.confluent.io/blog/apache-kafka-
2-4-latest-version-updates
● Overview Video: https://youtu.be/Ipzc--mbvzg
Always Built
on the Latest
Version of
Apache Kafka

27. 2727
Next Steps

28. 28
Register to attend: Kafka-Summit.org -- 25% off Standard Rate: use code
KSL20PartnerPro
Some sponsorship spots still open! Email sponsorships@kafka-summit.org

29. 29C O N F I D E N T I A L