Resilient Kafka: How DNS Traffic Management and Client Wrappers Ensure Availability
Report
Share
•0 likes•124 views
Resilient Kafka: How DNS Traffic Management and Client Wrappers Ensure Availability
•0 likes•124 views
Report
Share
Kafka clients sometimes have problems producing messages because of an incident or maintenance. In most cases these issues are short lived, but clients get paged and while long incidents are rare, they are memorable. In this talk, we'll run through our Kafka infrastructure at Shopify and how clients connect to it. Next, we'll describe our solution for performing failovers using DNS. Afterwards, we'll look at some real world scenarios where this system saved us from major outages. In conclusion, Kafka clusters are more resilient to infrastructure hiccups with a DNS traffic manager and opinionated client wrappers.
More Related Content
Similar to Resilient Kafka: How DNS Traffic Management and Client Wrappers Ensure Availability
Similar to Resilient Kafka: How DNS Traffic Management and Client Wrappers Ensure Availability(20)
Recently uploaded
Recently uploaded(20)
Resilient Kafka: How DNS Traffic Management and Client Wrappers Ensure Availability
- 1. Vanessa Vuibert Sta ff Production Engineer Resilient Ka f ka: How DNS Tra ff ic Management and Client Wrappers Ensure Availability @V3_XD
- 2. 862 14 Scale Ka f ka brokers Ka f ka clusters 14M 9 Messages per sec GCP Regions @V3_XD
- 3. • Maintenance • Incidents • Regionalize tra ff ic Tra ff ic management use cases
- 4. Kubernetes (K8s) out of the box 🔓open source Kafka broker
- 5. K8s out of the box dig +short service.namespace.svc.cluster.local IP0 IP1 IP2
- 6. K8s out of the box bootstrap.servers= service.namespace.svc.cluster.local:9092
- 7. K8s out of the box dig +short pod2.service.namespace.svc.cluster.local IP2
- 8. K8s out of the box advertised.listeners= pod2.service.namespace.svc.cluster.local:9092
- 9. • Readiness • Startup • Liveness K8s StatefulSet: probes
- 10. dig +short service.namespace.svc.cluster.local IP0 IP2 K8s readiness probe
- 11. dig +short service.namespace.svc.cluster.local IP0 IP2 IP3 K8s readiness probe not ready publishNotReadyAddresses: true
- 12. Regional pairs
- 13. External tra ff ic: load balancers
- 14. External tra ff ic: load balancers bootstrap.servers
- 15. External tra ff ic: load balancers advertised.listeners
- 16. • Issues scaling • Manual broker DNS records • Limited tra ff ic control
- 17. Built automation with k8s controllers.
- 18. Stateful buddy: load balancers 🔒closed source
- 19. Name buddy: DNS records 🔒closed source
- 20. Ka f ka access buddy: endpoints 🔒closed source
- 21. Ka f ka Access Buddy: consumer
- 22. Ka f ka Access Buddy: producer failover east
- 23. - Elasticsearch on call “Let me failover real quick.”
- 24. Faster failovers with a DNS tra ff ic manager.
- 25. DNS tra ff ic manager 🔒closed source
- 26. DNS tra ff ic manager: normal dig +short us-east1.somedomain.com US-East1-IP
- 27. DNS tra ff ic manager: failover dig +short us-east1.somedomain.com US-Central1-IP
- 28. - A Ka f ka client “DNS trickery.”
- 29. used to take 40 Minutes now only takes 1 Minutes Failover time savings @V3_XD
- 31. Failover during fl ashsale US Central1 -> US East1
- 32. Reduced toil with client wrappers.
- 33. • Failover reconnection • Everything needed for connection • Ruby, go and python Client wrappers
- 34. K8s Deployment template: bootstrap.servers
- 35. K8s Deployment template: client ID
- 37. Improved availability with local consumers.
- 38. • More availability • Reduced latency • Reduced storage costs • Reduced network costs Local consumers
- 40. Local consumers
- 41. Local consumers: DNS records
- 43. Connect directly through private IPs.
- 44. • More secure • Reduced network costs • Fetch from closest replica: KIP - 392 Public to private tra ff ic
- 45. Tra ff ic manager: pod IPs
- 46. Reduction -6% bill Network represents 29% bill Network cost reduction @V3_XD
- 47. • GKE 1.24 -> 1.25 incident • Apply f irewall rules • LB more secure for public tra ff ic
- 49. Single stop shop with Multi- Cluster Services (MCS).
- 52. Regional pairs: uneven distribution
- 53. Regionalize tra ff ic: Ka f ka access buddy east
- 55. 40 18 MCS time savings Minutes to regionalize tra ff ic Minutes to deploy 1 13 Minutes after migration Minutes after migration @V3_XD
- 57. • Resiliency: DNS tra ff ic management • Toil: client wrappers • Availability: local consumption
- 58. Thanks! @V3_XD