"Confluent Cloud is a cloud-native service based on Apache Kafka. We run tens of thousands of clusters across all major cloud service providers (AWS, GCP and Azure). In this talk, we will go over our journey to make Confluent Cloud 10x faster than Apache Kafka. We will talk about how we designed our various workloads, the complexities involved in our cloud-native service, the challenges we faced, and the various pitfalls we ran into. We will also cover the interesting learnings, which in hindsight, are first principles from this multi-year journey. By attending this talk, attendees will be able to take our learnings from making Confluent Cloud latencies 10x better and possibly apply similar principles to their cloud native data streaming systems."

1. Our Multi-Year
Performance Journey in
Confluent Cloud
Shriram Sridharan - Sr. Manager, Kafka Data Infrastructure
Marc Selwan - Sr. Product Manager, Kafka Data Infrastructure

2. Who are we?
Database background working on
storage and indexing.
Building faster and cheaper Kafka in
Confluent. Built relational databases.
We are simply representing a team of incredibly talented and hard working engineers

3. It’s not just Kafka in the cloud, in reality…

4. There’s a ton that goes into running our cloud service
NETWORK
COMPUTE
AZ AZ AZ
Cells
Cells
Cells
OBJECT
STORAGE
CUSTOMERS
Multi-Cloud Networking & Routing Tier
Metadata
Durability Audits
METRICS & OBSERVABILITY
CONNECT
PROCESSING
GOVERNANCE
Data Balancing
Health Checks
Real-time
feedback
data
Other Confluent Cloud Services
GLOBAL CONTROL PLANE

5. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
● Takeaways

6. Cloud deployments can
be notoriously complex
Each arrow represents an
interaction that has an
associated cost, performance,
and throughput limit. NETWORK
BROKERS
AZ AZ AZ
OBJECT
STORAGE
Multi-Cloud Networking & Routing Tier
PRODUCERS
SSDs SSDs SSDs
CONSUMERS
LOCAL STORAGE

7. Cloud deployments can
be notoriously complex
Each arrow represents an
interaction that has an
associated cost, performance,
and throughput limit.
These aspects—cost,
performance, and throughput
limits—don’t always change
proportionally for different
hardware options.
Available bw
(Gbps)
Instance name

8. Cloud deployments can
be notoriously complex
Each arrow represents an
interaction that has an
associated cost, performance,
and throughput limit.
These aspects—cost,
performance, and throughput
limits—don’t always change
proportionally for different
hardware options.
Pricing model, instance
performance varies across
cloud providers Many operators punt this complexity to customers.

9. Abstractions for a
unified multi-cloud
experience
Logical Kafka Cluster (LKC) as
the unit of access control
Confluent Kafka Unit (CKU) as
unit of cluster capacity in terms
of customer-visible metrics e.g.
50 MB/s ingress, 150MB/s egress
bandwidth per CKU
Cluster load exposes how
loaded a cluster is and provides
a signal when customers need
to scale up/down
NETWORK
BROKERS
AZ AZ AZ
OBJECT
STORAGE
Networking & Routing
PRODUCERS
SSDs SSDs SSDs
CONSUMERS
LOCAL STORAGE
LKC
AZ AZ AZ
PRODUCERS
CONSUMERS
1. Broker instance type?
2. Number of brokers?
3. Block storage type?
4. Block storage throughput?
5. Block storage IOPS?
6. Associated kernel, filesystem, and
Kafka knobs?
7. Which resource is bottlenecked?
1. How many CKUs do I need?
2. Is my cluster overloaded wrt
my latency requirements?
VS

10. • Managing 30K+ clusters
• Adapt and accommodate various
workload profiles
• Adding new features
• Run auxiliary software needed to
run our services
• Handle cloud provider variability
• Operated by machines not people
Ensure consistent
performance behind the
abstractions while…
NETWORK
BROKERS
AZ AZ AZ
OBJECT
STORAGE
Multi-Cloud Networking & Routing Tier
Metadata
Durability Audits
Data Balancing
Health Checks
Real-time
feedback
data
PRODUCERS
SSDs SSDs SSDs
CONSUMERS
LOCAL STORAGE
End
to
End
Latency
The Challenge

11. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
● Takeaways

12. Factors that determine the Latency SLO for Kafka
● Aggregate Broker/Cluster level ?
● Aggregate per week/day/hour/min ?
● E2E or Produce latencies ?
● p50/avg/p95/p99/p9999 ?
Challenges with running a cloud service
● No client visibility (KIP-714)
● Each customer has their own usage pattern
and expectation from the service
Defining a Latency SLO
- Challenges
What doesn’t get
measured, doesn’t get
improved

13. ● External health check probes every broker
● Measure E2E (Produce + Consume)
● Aggregate max per broker per min
● Monitor p99 over a week per cluster
Defining a Latency SLO
- First Attempt

14. ● Did NOT capture latency anomalies during
degradations
● Up-to 100 mins of degraded latency but cluster
still under SLO
Defining a Latency SLO
- Issues

15. Added a new SLO
● External health check probes every broker
● Measure E2E (Produce + Consume)
● Aggregate max per broker per min
● Monitor p99 per window (in mins) per cluster
Latency SLOs
● Steady State Latency SLO
● Degraded State Latency SLO
Defining a Latency SLO
- Current State

16. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
● Takeaways

17. Monitoring
Infrastructure
● Per cluster monitoring
● Alerts
● Operated by machines
● Nightly regression
tests
NETWORK
BROKERS
AZ AZ AZ
OBJECT
STORAGE
Multi-Cloud Networking & Routing Tier
Health check
producer
SSDs SSDs SSDs
Health check
consumer
LOCAL STORAGE
End
to
End
Latency
Internal
Latency
HC agent produce to consume measuring E2E latency

18. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
● Takeaways

19. Steady State Latency - Workload Patterns
Partitions 100s - 100s of thousands
Fanout 1:1 - 1:30
Throughput 10MB - 20GB
Clients 10s - 10s of thousands
Additional Variables Connection Rate, Requests per sec, Keyed vs non-keyed

20. Workloads
Proof of Concept
Benchmark
Tracing
What we got right
● Built distributed tracing
● Encode workloads into Open Messaging
Benchmark
● Fail fast with dirty proof of concepts
Steady State
Latency - Peeling
the Onion

21. What took us some time to figure out
● Hyper-focused on Confluent Kafka
● Kora services had significant impact
Split the investigation into
● Confluent Kafka runtime running on bare EC2
● Kora Services
Workloads
Proof of Concept
Benchmark
Tracing
Steady State
Latency - Peeling
the Onion

22. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
○ Cloud Infrastructure degradation
○ Workload degradation
● Takeaways

23. Kora Optimizations - Kafka Specific Optimizations
Disclaimer: YMMV depending on hardware/workload/configs

24. Replication
Optimizations
Observation
● Replication layer had a lot of CPU overhead and
inefficient allocation patterns
● Predominantly visible in workloads with a lot of
partitions
Improvement
● Kora has a completely rewritten efficient
replication protocol shipping in the next few
weeks aimed at minimizing CPU usage/
allocations.

25. Network Optimizations
Observation
● E2E latency much higher than broker side
latencies.
● Predominantly visible with less number of
clients.
Improvement
● Kora has increased parallelism in Kafka.
● Increases CPU consumption on the broker side
but provides overall better E2E latency

26. Storage
Optimizations
Observation : Background operations interfering with
foreground real-time operations.
Improvement
● Tiered Storage (Compute Storage Separation)
● Catchup consumption happens from the object
storage instead of local storage
● Heavily tuned filesystem and page cache
parameters

27. Incremental
Improvements
Infinite number of
Infinitesimal
improvements
Observation : Death by thousand cuts
Improvement
● Minimize work per request
● Move work out of the critical path
● Tuning (GC, Kernel parameters)

28. Kora Optimizations - Kora Specific Optimizations

29. Improvements to other
Kora Services
Observation
● Some of these services bin-packed with Kafka thus
using the same hardware resources.
Improvement
● Impact minimized by either
○ Complete re-architecture
○ Enforced QoS
● Monitoring agent - Example
Learning : Build with performance as first class citizen

30. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
○ Cloud Infrastructure degradation
○ Workload degradation
○ Multi-cloud vagaries
● Takeaways

31. Up-to 10X Faster
Results - Confluent Cloud vs Apache Kafka
*Blog post with benchmark details/ numbers expected in the new few weeks. Also, as more improvements
come in, these numbers will change

32. Up-to 10X Faster
Results - Confluent Cloud vs Apache Kafka
*Blog post with benchmark details/ numbers expected in the new few weeks. Also, as more improvements
come in, these numbers will change

33. Agenda
● Abstractions for a unified multi-cloud experience
● Latency SLO
● Monitoring
● Steady state latency
○ Workload patterns
○ Kora optimizations
○ Results
● Degraded state latency
● Takeaways

34. Degraded State Latency
Recap of the Degraded State Latency SLO
● p99 E2E latency per window (in mins) per cluster
Investigations broadly reveal the following issues
● Degraded cloud hardware/services
● Workload induced degradation (imbalance in distribution)
● Multi-cloud vagaries

35. Degraded cloud
hardware/services
Observation : Degradation in the cloud is real!
Over a recent 1 week interval we observed
● A few incidents with complete block storage outages
● 10s of incidents with external connectivity loss
● 100s of incidents of storage and network degradation
Improvement
● Built proprietary APIs to transfer leadership to
non-degraded broker (or AZ)
● No compromise on durability and availability
guarantees for predictable performance

36. Degraded cloud
hardware/services -
Automation
Monitor
Aggregate
Mitigate
Improvement : Proprietary APIs enable automated
mitigations
● Monitor, Aggregate and Mitigation pipeline
● Has triggered > 500 times in the last 30 days!!

37. Degraded cloud
hardware/services - An
example

38. Workload Induced
Degradation
Observation : Workload changes causes imbalance in
distribution of load and hence degrades the latency.
Improvement:
● Kora includes a component called Self
Balancing Cluster (SBC) which continuously
rebalances the cluster.
● Rebalancing was heavy-weight/ slow by
computing all required changes up-front and
making required rebalancing.
● Re-architected to be more real time

39. One customer saw ~25% reduction in their load
when rebalancing was enabled.
Another customer saw significant improvement
in latency with rebalancing.
Workload Induced
Degradation

40. Multi-Cloud Vagaries
Observation
● Same Instance type had different CPU generations
● Throughput/ IOPS scaled differently. Eg: GP2 vs GP3.
Improvement
● Abstractions enable us to continuously optimize
latencies as new hardware becomes available
● Tuning of IOPS/throughput per cloud provider

41. Abstractions enabled
latency improvements
while “flying the
plane”
Abstractions enabled continuous optimizations
● Switched from GP2 to GP3
● Moved 20000+ instances to Graviton
● Moved between memory and compute
optimized instance types seamlessly
Customer filed a support ticket asking why their “cluster load”
decreased significantly? They were able to downsize their cluster to
save money!
Customer Example

42. Our Learnings through this multi-year journey
● Primitives vary widely across cloud providers - Abstractions are required to
provide a unified multi-cloud experience
● You cannot improve what you cannot observe
● Build with performance as first class citizen across the stack
● Steady state latency optimizations are necessary but not sufficient
● Cloud hardware/services degradation is real & frequent - Resiliency needs to be
build in for the cloud

43. Thank you! Questions ?