This document discusses running Apache Kafka on Kubernetes using the Confluent Operator. It provides an overview of Kafka's architecture including sharding, replication, and scaling. It then covers how the Operator allows automated provisioning, operations, resiliency and monitoring of Kafka and Confluent Platform on Kubernetes through custom resource definitions and controllers. The Operator helps with tasks like rolling upgrades, scaling, and meeting SLAs. Resources are provided to learn more about running Kafka on Kubernetes.

1. Cloud-Native
Streaming Platform:
Apache Kafka
Meets Kubernetes
@gamussa #BOSDataDay @confluentinc

2. @@gamussa #BOSDataDay @confluentinc
#devkafkaops

3. @gamussa #BOSDataDay @confluentinc

4. @@gamussa #BOSDataDay @confluentinc
https://twitter.com/kelseyhightower/status/963413508300812295

5. @@gamussa #BOSDataDay @confluentinc
https://twitter.com/kelseyhightower/status/963414038603427840

6. @@gamussa #BOSDataDay @confluentinc
Don’t despair…
“… not even over the fact that you don't
despair. Just when everything seems over
with, new forces come marching up, and
precisely that means that you are alive”
Franz Kafka

7. Kafka Streaming Architecture
Fundamentals

8. @@gamussa #BOSDataDay @confluentinc

9. @@gamussa #BOSDataDay @confluentinc

10. @gamussa #BOSDataDay @confluentinc
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster of
machines
Partitions live on
different machines
Messages are sent to
different partitions

11. @gamussa #BOSDataDay @confluentinc
Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!
Producers
Consumers

12. @gamussa #BOSDataDay @confluentinc
Replicate to get fault tolerance
replicate
msg
msg
leader
Machine A
Machine B

13. @gamussa #BOSDataDay @confluentinc
Replication provides resiliency
A ‘replica’ takes over on machine failure

14. @@gamussa #BOSDataDay @confluentinc
High-level Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

15. @@gamussa #BOSDataDay @confluentinc
Bare minimum
Zookeeper NodesKafka Brokers

16. @gamussa #BOSDataDay @confluentinc
Workloads Deployment

17. @@gamussa #BOSDataDay @confluentinc
Helm Charts

18. @@gamussa #BOSDataDay @confluentinc
Helm Charts

19. @@gamussa #BOSDataDay @confluentinc
Helm Charts

20. @gamussa #BOSDataDay @confluentinc
https://cnfl.io/helm_video

21. @@gamussa #BOSDataDay @confluentinc
Kafka deployment checklist
PVC for Storage
StatefulSet for 3-node zk
Optional Pod Anti-Affinity to spread the
ZK ensemble across nodes
Headless Service
ConfigMap for Prometheus JMX exporter
Uses ZK Headless Svc
PVC for Storage
StatefulSet for n-node Kafka
A group of NodePort Services for
external traffic
ConfigMap for Prometheus JMX exporter

22. @@gamussa #BOSDataDay @confluentinc
Meet Kubernetes Operator

23. @@gamussa #BOSDataDay @confluentinc
Kubernetes Operator
Embedded with operational knowledge
of both data software and
Kubernetes
Backup/restore
Scale up/down
Rebalance data
Regular health checks

24. @@gamussa #BOSDataDay @confluentinc
Controller
Brain behind Kubernetes
resources
e.g. replication
controller, namespace
controller etc.

25. @@gamussa #BOSDataDay @confluentinc
Custom Resource Definition(CRD)
Extend existing
Kubernetes API StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

26. @@gamussa #BOSDataDay @confluentinc
Custom Resource Definition(CRD)
Usually works together
Custom Controller StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

27. @@gamussa #BOSDataDay @confluentinc
Custom Resource Definition(CRD)
Users can create and
access Customer
Resources with
kubectl, just as they
do for built-in
resources like pods.
StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

28. @@gamussa #BOSDataDay @confluentinc
Operator
Deploy and Manage your production
streaming platform with Confluent
Operator.
Automated Provisioning
Platform Operations
Resiliency
Monitoring

29. @@gamussa #BOSDataDay @confluentinc
Confluent Platform Reference Architecture
Each Confluent Platform
component has specific
characteristics:
Security (SSL certificates)
DNS names and zones
Host selection
Fault tolerance
Scaling
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

30. @@gamussa #BOSDataDay @confluentinc
Confluent Operator: Automated Provisioning
Kafka Pod Kafka Pod Kafka Pod
Storage
Load Balancer

31. @@gamussa #BOSDataDay @confluentinc
Confluent Operator: Scale Horizontally
Automate scaling:
Spin up new broker pod(s)
Distribute partitions to the new
broker(s)
Determine balancing plan
Execute balancing plan
Monitor resources

32. @@gamussa #BOSDataDay @confluentinc
Confluent Operator: Rolling Upgrade
Automated rolling upgrade with no
downtime for Kafka.
Stop broker
Wait for leader election to
complete
Start broker with new version
Wait for zero under-replicated-
partitions
Repeat

33. @gamussa #BOSDataDay @confluentinc
Will it fly? Let’s see

34. @@gamussa #BOSDataDay @confluentinc
Confluent Operator
Automate provisioning
Monitor SLAs through
Confluent Control Center
or Prometheus
Scale your Kafkas and
CP clusters elastically
Operate at scale with
enterprise support
from Confluent

35. @gamussa #BOSDataDay @confluentinc
Advanced use cases
vs.

36. @@gamussa #BOSDataDay @confluentinc
Don’t despair!

37. @gamussa #BOSDataDay @confluentinc
Resources and Next Steps
https://cnfl.io/cp-helm
https://cnfl.io/k8s
https://slackpass.io/confluentcommunity
#kubernetes
https://cnfl.io/helm_video

38. @@gamussa #BOSDataDay @confluentinc
Thanks!
@gamussa
viktor@confluent.io