Presented by Gwen Shapira as part 1 of 5 in the Best Practices for Apache Kafka in Production, Confluent Online Talk Series

1. 1
Deploying Confluent
Platform in Production
Gwen Shapira, Product Manager

2. 2
Welcome to
Best Practices for Apache
Kafka in Production
Online Talk Series

3. 3
About me
• Product Manager
• Apache Kafka Committer
• Tweets a lot @gwenshap

4. 4
About you
• Know Linux
• Installed Confluent Platform
• Ran an example application
• Need to productionize Kafka
• Ready to Learn

5. 5
Components Resources
Planning Scaling
Deployment

6. 6
Components of
Confluent Platform

7. 7
ZooKeeper ZooKeeper ZooKeeper

8. 8
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka

9. 9
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy

10. 10
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams
apps

11. 11
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams
apps
Schema
Registry

12. 12
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams
apps
Schema
Registry
Replicator
ADB
Control
Center

13. 13
State in Kafka Stateless – No Storage
Planning for storage and containers
State on local disk
ZooKeeperKafka
Streams
apps
Control
Center
Java app
C# app Go app
Python app
REST ProxyConnect
Connectors
Schema
Registry
Replicator
ADB

14. 14
Required Resources

15. 15
The Storage Questions
How Much?
• Kafka:
Throughput * Retention
• Zookeeper:
Very little
• Control Center:
Lots
• Streams:
It is complicated
Hardware
• Are SSDs
worth it?
• Is shared
storage ok?
Configuration
• RAID vs JBOD
• XFS or EXT4
• Zookeeper log
Partitions
• Per topic
• Per broker
• Total

16. 16
Memory
Page Cache JVM Heap Off Heap Memory
ZooKeeper 1-4 GB
Kafka The more the
merrier
# partitions * max fetch
+ compaction buffer +
~10%
Kafka Connect # tasks * # partitions *
memory buffer per
partition
Kafka Streams 10GB Buffer Cache + 1MB
per partition or 50MB per
broker =~
32GB J
RocksDB
Rest Proxy ~ 1 GB
Schema Registry ~ 1 GB
Clients Java client – batching and retries Other clients – batching and retries
Control Center It is a streams app… 32GB It is a streams app…

17. 17
• CPU
• Keep an eye on Zookeeper and Kafka
• High CPU usually caused by misconfig
• Or by… Compression, encryption, high request rate
• Network
• 1GbE = 100MB/s rate (including replication!)
• Leave room for catching up
• Compress!

18. 18
Special Considerations for Clouds
• Virtual cores are relatively weak
• Network is typically weak
• Shared storage is typically awesome

19. 19
Planning a Deployment
of Confluent Platform

20. 20
1. Which components do I need?
2. Small or Large cluster?
Kafka just for one team and one app?
Centralized cluster for larger organization?
3. Other requirements?
Availability, retention, latency, throughput.
4. How many clusters?
Ask yourself:

21. 21
Small Cluster:
Place many related
components
on each node

22. 22
Large Cluster:
Separate nodes
Scale each component as
needed

23. 23
Scaling Confluent Platform
as you grow

24. 24
Some Stuff you can change easily:
• Additional producers
• Additional schema registries
• Additional REST Proxies
• Additional connectors and tasks
• Additional replicas

25. 25
Some Stuff requires “enough partitions”:
• Additional consumers
• Additional streams instances
• Additional sink connectors

26. 26
Some Stuff takes time/resources to change:
• Additional partitions
• Additional brokers
• Additional Zookeeper instances
• Additional Control Centers (impossible right now)

27. 27
Monitor your cluster closely to know when to add resources

28. 28
1. Key components in Confluent Platform and their requirements
2. Key resources and how to use them effectively
3. Planning your deployment
4. Monitoring in order to scale
5. Want to learn more? There is a paper!
https://www.confluent.io/whitepaper/confluent-enterprise-reference-
architecture/
What did we learn?

29. 29
Thank You!