Make Your Kafka Cluster Production-Ready - Jakub Scholz, Red Hat Kubernetes became the de-facto standard for running cloud-native applications. And more and more users turn to it also to run stateful applications such as Apache Kafka. While there are different tools such as Helm charts or operators which can get you quickly up and running, there is often still a long way to make sure the Kafka cluster is production-ready. This talk will take you through the main aspects you should consider for your Kafka cluster and will cover things such as resource management, storage, scheduling, rolling updates, or reliability. It will show you how to do it using the Strimzi operator, but the lessons learned will apply also to any other Kafka cluster. If you are interested in production-ready Apache Kafka on Kubernetes, this is a talk for you.

1. https://strimzi.io
Make Your Kafka Cluster
Production-ready
Jakub Scholz
DOK Day @ KubeCon NA 2023

2. Make Your Kafka Cluster Production-ready
About me
● Senior Principal Software Engineer @ Red Hat
● Maintainer of Strimzi project (https://strimzi.io)
● Occasional Apache Kafka contributor
@scholzj
https://github.com/scholzj
https://www.linkedin.com/in/scholzj/
2

3. 3
Make Your Kafka Cluster Production-ready

4. Infrastructure
4
Make Your Kafka Cluster Production-ready

5. Make Your Kafka Cluster Production-ready
Infrastructure
● Networking
● Storage
○ Supports JBOD and Local storage
● CPU & Memory
● Resource Limits & Requests
5

6. Make Your Kafka Cluster Production-ready
6
Cache eeeee
JVM
Heap
Limit
Request

7. Security
7
Make Your Kafka Cluster Production-ready

8. Make Your Kafka Cluster Production-ready
Security
● TLS Encryption
● Authentication (SCRAM-SHA, mTLS, OAuth)
● Authorization (Kafka authorizer, Keycloak, Open Policy Agent)
● Network Policies
● RBAC
● Security Context
8

9. Monitoring
9
Make Your Kafka Cluster Production-ready

10. Make Your Kafka Cluster Production-ready
Monitoring
● Understanding the state of the Kafka cluster
● Logs, Metrics, Tracing, Dashboards, Alerts, …
● The usual suspects: Prometheus, OpenTelemetry, …
● Consumer Lag
● Kafka Exporter
10

11. Availability
11
Make Your Kafka Cluster Production-ready

12. Make Your Kafka Cluster Production-ready
Availability
● Topic and client configurations
● Rolling Updates
● Rack Awareness
● Topology Spread Constraints & Affinity
● Voluntary Disruptions
● Pod Disruption Budgets, Readiness probes
● Disaster Recover / Backups
12

13. Rolling Updates
13
Make Your Kafka Cluster Production-ready

14. 14
Make Your Kafka Cluster Production-ready
Controller
Controller
Controller
Controller
Controller

15. 15
Make Your Kafka Cluster Production-ready

16. 15
Make Your Kafka Cluster Production-ready
P1 P1 P1
P2 P2
P3 P3
P2
P3

17. 15
Make Your Kafka Cluster Production-ready
P1 P1
P2 P2
P3 P3
P2
P3
P1

18. 15
Make Your Kafka Cluster Production-ready
P1 P1
P2 P2
P3 P3
P1
P2
P3

19. Rack Awareness
16
Make Your Kafka Cluster Production-ready

20. A
ZONE
C
ZONE
B
ZONE

21. A
ZONE
C
ZONE
B
ZONE

22. A
ZONE
C
ZONE
B
ZONE
P2
P1
P2
P1
P2
P1
P3 P3
P3

23. A
ZONE
C
ZONE
B
ZONE
P2
P1
P2
P1
P2
P1
P3 P3
P3

24. A
ZONE
C
ZONE
B
ZONE
P2
P1
P2
P1
P2
P1
P3 P3
P3

25. Make Your Kafka Cluster Production-ready
Availability
● Topic and client configurations
● Rolling Updates
● Rack Awareness
● Topology Spread Constraints & Affinity
● Voluntary Disruptions
● Pod Disruption Budgets, Readiness probes
● Disaster Recover / Backups
18

26. Performance
19
Make Your Kafka Cluster Production-ready

27. Make Your Kafka Cluster Production-ready
Performance
● Noisy neighbours
● Pod anti-affinity
● Dedicated nodes (Taints & Tolerations)
● Tune the JVM (Garbage collection, …)
● Cluster Balancing
20

28. Cluster Balancing
21
Make Your Kafka Cluster Production-ready

29. 2
1
0

30. 2
1
0

31. 0 2
1

32. 0 2
1

33. Networking, Storage, Memory, CPU, Resource Requests & Limits
Encryption, Authentication, Authorization,
Network Policies, Security Context, RBAC, …
Logs, Metrics, Alerts, Dashboards,
Consumer Lag, Traces, Audits, …
Topology Spread Constraints,
Rolling Updates, Rack awareness,
Node Affinity, Configuration, Disaster
Recovery, Backups, Disruptions, …
Cluster
Balancing,
Pod anti-affinity,
Dedicated nodes,
Tune the JVM, …
Infrastructure
Security
Monitoring
High
Availability
(Reliability)
Performance

34. Why do I need to care about this?
Why not have production-ready Kafka out-of-the box?
25
Make Your Kafka Cluster Production-ready

35. Make Your Kafka Cluster Production-ready
Mind the gap!
● Different environments
● Development vs. CIs vs. Production
● Different requirements
● How much [Security|Monitoring|Availability|Performance] do I really need?
● Different infrastructure
● Labels, versions, topologies, tools, …
● No one-size-fits-all
26

36. Make Your Kafka Cluster Production-ready
Further resources
● Documentation
● Strimzi: https://strimzi.io/documentation/
● Apache Kafka: https://kafka.apache.org/documentation/
● Examples: https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples
● Blog posts: https://strimzi.io/blog/
● “Make your Kafka cluster production-ready”: https://youtube.com/c/Strimzi
27

37. Thank you
Website: https://strimzi.io
GitHub: https://github.com/strimzi
Twitter: @strimziio
YouTube: https://youtube.com/c/Strimzi
28
Make
Your
Kafka
Cluster
Production-ready