"To provide exceptional customer experiences at scale, the data pipelines that can move data reliably across the systems and applications in real-time should be seamlessly scalable. For the past several years, we relied on Message Queue based data pipelines to facilitate the transfer of data across the applications. However, as the number of use cases that require real-time data transfer increased rapidly, it became difficult to scale the messaging platform. Moving to Kafka helped us to resolve the data pipeline scaling issues and reduce the Publisher/Subscriber on-boarding time from several weeks to a few days. To support the on-demand scaling of Kafka clusters, we run them on RedHat OpenShift, an Enterprise Kubernetes. While managing Kafka that handles critical financial events, we have learned some lessons and developed efficient strategies to manage production-grade Kafka clusters on OpenShift. In this talk, we will present: 1. Some of the challenges that we faced with Kafka on OpenShift and how we evolved our infrastructure to overcome them. 2. Share our experiences from operating Kafka clusters at Scale in Production. 3. Our strategy for performing automated Kafka deployment and rollback in OpenShift. 4. Explain our fail-over strategy using Confluent’s Replicator to ensure service availability during cluster failures."

1. Discover Kafka on OpenShift:
Processing Real-Time Financial Events at Scale
The opinions expressed in this presentation are those of the presenter, in their individual capacity, and not necessarily those of Discover.
Anvesh Samineni
Senior Software Engineer
Ehfaj Khan
Principal Software Engineer

2. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Kafka Deployment and Rollback Strategy in OpenShift
Multi-Cluster Replication Design and Failover Strategy
Agenda

3. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Page Cache
• Page Cache is shared
• Performance might vary
• Pods are dynamically provisioned
• Larger node, many pods
• Page Cache is dedicated
• Less performance variations
• Only Kafka pod is provisioned
• Smaller node, only Kafka pod

4. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Assigning Pods to Nodes (1/4)
Pod
Node Affinity
Kafka pod should go to Kafka node only
Node

5. Pod Anti-Affinity
How Kafka pods should be placed relative to one another
(Required: Do not schedule if a Kafka pod already exists in the node.)
Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Pod
Assigning Pods to Nodes (2/4)

6. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Pod Anti-Affinity
How Kafka pods should be placed relative to one another
(Preferred: Try to schedule Kafka pods across AZs.)
Assigning Pods to Nodes (3/4)
Pod

7. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Node
Pod
Assigning Pods to Nodes (4/4)
Taints and Tolerations
Other pod should NOT go to Kafka node

8. Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Disruptions:
• Draining nodes accidentally
• Deleting many pods at a time
Handling Disruptions
PodDisruptionBudget
Limit number of concurrent disruptions
(Example: minAvailable = 2)
Since 2 concurrent disruptions
lead to 1 Available Kafka Pod

9. Dedicated nodes for Kafka pods
Node Affinity
Kafka pod should go to Kafka node only
Pod Anti-Affinity
How Kafka pods should be placed relative to one another
• Required: Do not schedule if a Kafka pod already exists in the node.
• Preferred: Try to schedule Kafka pods across AZs.
Taints and Tolerations
Other pod should NOT go to Kafka node
PodDisruptionBudget
Limit number of concurrent disruptions
Provisioning Kafka Infrastructure on OpenShift for High Availability and Performance
Summary

10. Kafka Deployment and Rollback Strategy in OpenShift
Repeat for every pod except Active Controller
(Starting from last one)
1. Delete the pod
2. Wait till URP=0
3. Once URP=0, delete next pod
1. Delete Active Controller Pod
2. Wait till URP=0
Identify the Active Controller Pod
(Upgrade last)
Deployment Strategy: onDelete
Deployment Strategy

11. Kafka Deployment and Rollback Strategy in OpenShift
Repeat below for every pod except
Active Controller:
(Starting from last one)
1. Delete the pod
2. Wait till URP=0
3. Once URP=0, delete next pod
One of the Pod fails to restart
Revert StatefulSet to the previous
version
Deployment Strategy: onDelete
Identify the Active Controller Pod
(Upgrade last)
Deployment Strategy: onDelete Repeat below for all upgraded pods
in the reverse order of upgrade:
(Start with pod that failed to restart)
1. Delete the pod
2. Wait till URP=0
3. Once URP=0, delete previous pod
Rollback Strategy

12. Multi-Cluster Replication Design and Failover Strategy
Replicator replicates:
• Topics
• Messages
• Consumer groups

13. Multi-Cluster Replication Design and Failover Strategy
During Failover:
• Flip to the bootstrap URL of secondary cluster
• Stop the Replicator

14. Multi-Cluster Replication Design and Failover Strategy
Important to Enable Failover
Monitor Replicator
• Connectors are running
• Provision sufficient tasks
• No replication lag
Centralized Schema Registry
Enable Timestamp Interceptors
• Allows subscription to continue in the secondary cluster where it left off in the primary cluster
• Consumer groups in the secondary cluster are created by the Replicator
Provision ACLs for producers and consumers in the secondary cluster

15. Thank You