Running Kafka in Kubernetes: A Practical Guide (Katherine Stanley, IBM United Kingdom Ltd.) Kafka Summit London 2019
•
4 likes•4,025 views
The rise of Apache Kafka as the de facto standard for event streaming has coincided with the rise of Kubernetes for cloud-native applications. While Kubernetes is a great choice for any distributed system, that doesn’t mean it is easy to deploy and maintain a Kafka cluster running on it. At IBM we have hands-on experience with running Kafka in Kubernetes and in this session I will share our top tips for a smooth ride. I will show an example deployment of Kafka on Kubernetes and step through the system to explain the common pitfalls and how to avoid them. This will include the Kubernetes objects to use, resource considerations and connecting applications to the cluster. Finally, I will discuss useful Kafka metrics to include in Kubernetes liveness and readiness probes.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Running Kafka in Kubernetes: A Practical Guide (Katherine Stanley, IBM United Kingdom Ltd.) Kafka Summit London 2019
Similar to Running Kafka in Kubernetes: A Practical Guide (Katherine Stanley, IBM United Kingdom Ltd.) Kafka Summit London 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Running Kafka in Kubernetes: A Practical Guide (Katherine Stanley, IBM United Kingdom Ltd.) Kafka Summit London 2019
- 1. IBM Event StreamsApache Kafka © 2019 IBM Corporation Running Kafka in Kubernetes A practical guide Katherine Stanley Kafka Summit London 2019
- 2. © 2019 IBM Corporation
- 3. © 2019 IBM Corporation Monolith API Monolithic application
- 4. © 2019 IBM Corporation Microservice Microservice Microservice Microservice Microservice Service Registry API API API Microservices application
- 5. © 2019 IBM Corporation Microservice Microservice Microservice Microservice Microservice Service Registry API API API Microservices application REST request
- 6. © 2019 IBM Corporation Microservice Microservice Microservice Microservice Microservice Service Registry API API API Microservices application REST request
- 7. © 2019 IBM Corporation Rise of containers
- 8. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies
- 9. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies Isolation
- 10. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies Isolation Dev-prod parity
- 11. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies Isolation Dev-prod parity Routing
- 12. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies Isolation Dev-prod parity Routing Handling failure
- 13. © 2019 IBM Corporation Rise of containers Encapsulation of dependencies Isolation Dev-prod parity Routing Handling failure
- 14. © 2019 IBM Corporation Kubernetes “Helmsman” in ancient Greek
- 15. © 2019 IBM Corporation Kubernetes “Helmsman” in ancient Greek Declarative —> you describe desired state, Kubernetes makes it happen
- 16. © 2019 IBM Corporation Kubernetes CONTAINER A CONTAINER B POD A
- 17. © 2019 IBM Corporation Kubernetes CONTAINER A CONTAINER B POD A CONTAINER A CONTAINER B POD B CONTAINER A CONTAINER B POD C
- 18. © 2019 IBM Corporation Kubernetes CONTAINER A CONTAINER B POD A CONTAINER A CONTAINER B POD B CONTAINER A CONTAINER B POD C DEPLOYMENT
- 19. © 2019 IBM Corporation Kubernetes CONTAINER A CONTAINER B POD A CONTAINER A CONTAINER B POD B CONTAINER A CONTAINER B POD C SERVICE
- 20. © 2019 IBM Corporation
- 21. © 2019 IBM Corporation
- 22. © 2019 IBM Corporation Components of an event-streaming application App Event Backbone 1 2 App 3 4 Building Blocks 1 :: Event Sources 2 :: Stream Processing 3 :: Event Archive 4 :: Notifications
- 23. © 2019 IBM Corporation
- 24. © 2019 IBM Corporation Topics 0 1 2 3 4 5 Offset
- 25. © 2019 IBM Corporation
- 26. © 2019 IBM Corporation
- 27. © 2019 IBM Corporation
- 28. © 2019 IBM Corporation
- 29. © 2019 IBM Corporation
- 30. © 2019 IBM Corporation
- 31. © 2019 IBM Corporation Kafka Pods
- 32. © 2019 IBM Corporation $ kubectl get pods -w NAME READY STATUS RESTARTS AGE my-pod-123 1/1 Running 0 30s
- 33. © 2019 IBM Corporation $ kubectl get pods -w NAME READY STATUS RESTARTS AGE my-pod-123 1/1 Running 0 30s
- 34. © 2019 IBM Corporation Kubernetes health checks Liveness Readiness
- 35. © 2019 IBM Corporation Kubernetes health checks Liveness Is container running? Failure ! restart container Readiness
- 36. © 2019 IBM Corporation Kubernetes health checks Liveness Is container running? Failure ! restart container Readiness Ready to receive requests Failure ! remove from service list
- 37. © 2019 IBM Corporation Kafka health checks
- 38. © 2019 IBM Corporation Kafka health checks /liveness Port open Responds to basic Kafka protocol
- 39. © 2019 IBM Corporation Kafka health checks /liveness Port open Responds to basic Kafka protocol /readiness e.g Check in-sync replicas
- 40. © 2019 IBM Corporation min.insync.replicas “The minimum number of replicas that must be in-sync for a partition of the topic to be considered available”
- 41. © 2019 IBM Corporation min.insync.replicas “The minimum number of replicas that must be in-sync for a partition of the topic to be considered available” min.insync.replicas = 2
- 42. © 2019 IBM Corporation min.insync.replicas “The minimum number of replicas that must be in-sync for a partition of the topic to be considered available” min.insync.replicas = 2 in-syncNOT in-syncNOT in-sync
- 43. © 2019 IBM Corporation min.insync.replicas “The minimum number of replicas that must be in-sync for a partition of the topic to be considered available” min.insync.replicas = 2 in-syncNOT in-sync NOT in-sync consumerproducer
- 44. © 2019 IBM Corporation Managing Kafka Pods
- 45. © 2019 IBM Corporation CONTAINER A CONTAINER B POD A CONTAINER A CONTAINER B POD B CONTAINER A CONTAINER B POD C DEPLOYMENT
- 46. © 2019 IBM Corporation StatefulSets
- 47. © 2019 IBM Corporation StatefulSets Maintains sticky identity for each Pod Persistence via PersistentVolume
- 48. © 2019 IBM Corporation Persistent Volumes /mydata
- 49. © 2019 IBM Corporation Persistent Volumes /mydata Persistent Volume
- 50. © 2019 IBM Corporation POD Persistent Volumes /mydata Persistent Volume Persistent Volume Claim
- 51. © 2019 IBM Corporation POD Persistent Volumes /mydata Persistent Volume Persistent Volume Claim Binds
- 52. © 2019 IBM Corporation Persistent Volumes /mydata Persistent Volume Persistent Volume Claim Binds
- 53. © 2019 IBM Corporation POD Persistent Volumes /mydata Persistent Volume Persistent Volume Claim Binds
- 54. © 2019 IBM Corporation kafka.common.InconsistentBrokerIdException: Configured brokerId 1 doesn't match stored brokerId 0 in meta.properties
- 55. © 2019 IBM Corporation kafka.common.InconsistentBrokerIdException: Configured brokerId 1 doesn't match stored brokerId 0 in meta.properties
- 56. © 2019 IBM Corporation kafka.common.InconsistentBrokerIdException: Configured brokerId 1 doesn't match stored brokerId 0 in meta.properties
- 57. © 2019 IBM Corporation kafka.common.InconsistentBrokerIdException: Configured brokerId 1 doesn't match stored brokerId 0 in meta.properties
- 58. © 2019 IBM Corporation Deploying Kafka Pods
- 59. © 2019 IBM Corporation
- 60. © 2019 IBM Corporation Affinity/anti-affinity
- 61. © 2019 IBM Corporation Affinity/anti-affinity
- 62. © 2019 IBM Corporation $ kubectl get nodes NAME STATUS ROLES AGE VERSION 172.16.160.168 Ready etcd,management,master,proxy 1d v1.12.4 172.16.163.246 Ready worker 1d v1.12.4 172.16.177.181 Ready worker 1d v1.12.4 172.16.177.193 Ready worker 1d v1.12.4
- 63. © 2019 IBM Corporation $ kubectl get nodes NAME STATUS ROLES AGE VERSION 172.16.160.168 Ready etcd,management,master,proxy 1d v1.12.4 172.16.163.246 Ready worker 1d v1.12.4 172.16.177.181 Ready worker 1d v1.12.4 172.16.177.193 Ready worker 1d v1.12.4 $ kubectl describe node 172.16.163.246 Non-terminated Pods: Namespace Name --------- ---- es myrelease-ibm-es-kafka-sts-1 es myrelease-ibm-es-zookeeper-sts-0
- 64. © 2019 IBM Corporation Accessing Kafka
- 65. © 2019 IBM Corporation
- 66. © 2019 IBM Corporation Headless services apiVersion: v1 kind: Service metadata: labels: app: ibm-es name: myrelease-ibm-es-kafka-broker-svc-0 namespace: es spec: clusterIP: None ports: - name: kafka port: 9092 protocol: TCP targetPort: 9092 selector: kafka: "0"
- 67. © 2019 IBM Corporation $ kubectl get service NAME TYPE PORT(S) AGE myrelease-kafka-0 ClusterIP 9092/TCP 1d myrelease-kafka-1 ClusterIP 9092/TCP 1d myrelease-kafka-2 ClusterIP 9092/TCP 1d
- 68. © 2019 IBM Corporation Listeners and Advertised Listeners listeners advertised.listeners
- 69. © 2019 IBM Corporation Listeners and Advertised Listeners listeners the address the socket server listens on advertised.listeners
- 70. © 2019 IBM Corporation Listeners and Advertised Listeners listeners the address the socket server listens on advertised.listeners hostname and port the broker will advertise to producers and consumers
- 71. © 2019 IBM Corporation $ cloudctl es broker-config 0 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30100 $ cloudctl es broker-config 1 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30101 $ cloudctl es broker-config 2 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30102
- 72. © 2019 IBM Corporation $ cloudctl es broker-config 0 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30100 $ cloudctl es broker-config 1 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30101 $ cloudctl es broker-config 2 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30102
- 73. © 2019 IBM Corporation $ cloudctl es broker-config 0 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30100 $ cloudctl es broker-config 1 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30101 $ cloudctl es broker-config 2 | grep listeners listeners EXTERNAL://:9092 advertised.listeners EXTERNAL://9.30.249.46:30102
- 74. © 2019 IBM Corporation Automation
- 75. © 2019 IBM Corporation Handling automation Helm $ helm install $ helm upgrade
- 76. © 2019 IBM Corporation Handling automation Helm Custom Kubernetes resource $ kubectl get kafka $ kubectl get zookeeper $ helm install $ helm upgrade
- 77. © 2019 IBM Corporation Handling automation Helm Custom Kubernetes resource Kubernetes Operators $ kubectl get kafka $ kubectl get zookeeper $ kubectl scale kafka ?? $ helm install $ helm upgrade
- 78. © 2019 IBM Corporation github.com/ibm/charts/tree/master/stable/ibm-eventstreams-dev
- 79. © 2019 IBM Corporation Thank you IBM Event Streams: ibm.com/cloud/event-streams https://slack-invite-ibm-cloud-tech.mybluemix.net/ Katherine Stanley Software Engineer