Kafka clients sometimes have problems producing messages because of an incident or maintenance. In most cases these issues are short lived, but clients get paged and while long incidents are rare, they are memorable. In this talk, we'll run through our Kafka infrastructure at Shopify and how clients connect to it. Next, we'll describe our solution for performing failovers using DNS. Afterwards, we'll look at some real world scenarios where this system saved us from major outages. In conclusion, Kafka clusters are more resilient to infrastructure hiccups with a DNS traffic manager and opinionated client wrappers.

1. Vanessa Vuibert
Sta
ff
Production Engineer
Resilient Ka
f
ka: How DNS Tra
ff
ic Management
and Client Wrappers Ensure Availability
@V3_XD

2. 862 14
Scale
Ka
f
ka brokers Ka
f
ka clusters
14M 9
Messages per sec GCP Regions
@V3_XD

3. • Maintenance
• Incidents
• Regionalize tra
ff
ic
Tra
ff
ic management use cases

4. Kubernetes (K8s) out of the box
🔓open source
Kafka broker

5. K8s out of the box
dig +short service.namespace.svc.cluster.local
IP0
IP1
IP2

6. K8s out of the box
bootstrap.servers=
service.namespace.svc.cluster.local:9092

7. K8s out of the box
dig +short pod2.service.namespace.svc.cluster.local
IP2

8. K8s out of the box
advertised.listeners=
pod2.service.namespace.svc.cluster.local:9092

9. • Readiness
• Startup
• Liveness
K8s StatefulSet: probes

10. dig +short service.namespace.svc.cluster.local
IP0
IP2
K8s readiness probe

11. dig +short service.namespace.svc.cluster.local
IP0
IP1
IP2
K8s readiness probe
not ready
publishNotReadyAddresses: true

12. Regional pairs

13. External tra
ff
ic: load balancers

14. External tra
ff
ic: load balancers
bootstrap.servers

15. External tra
ff
ic: load balancers
advertised.listeners

16. • Issues scaling
• Manual broker DNS
records
• Limited tra
ff
ic
control

17. Built automation with
k8s controllers.

18. Stateful buddy: load balancers
🔒closed source

19. Name buddy: DNS records
🔒closed source

20. Ka
f
ka access buddy: endpoints
🔒closed source

21. Ka
f
ka Access Buddy: consumer

22. Ka
f
ka Access Buddy: producer failover
east

23. - Elasticsearch on call
“Let me failover real quick.”

24. Faster failovers with a
DNS tra
ff
ic manager.

25. DNS tra
ff
ic manager
🔒closed source

26. DNS tra
ff
ic manager: normal
dig +short us-east1.somedomain.com
US-East1-IP

27. DNS tra
ff
ic manager: failover
dig +short us-east1.somedomain.com
US-Central1-IP

28. - A Ka
f
ka client
“DNS trickery.”

29. used to take
40
Minutes
now only takes
1
Minutes
Failover time savings
@V3_XD

30. Incident during
fl
ashsale

31. Failover during
fl
ashsale
US Central1 -> US East1

32. Reduced toil with
client wrappers.

33. • Failover reconnection
• Everything needed for connection
• Ruby, go and python
Client wrappers

34. K8s Deployment template: bootstrap.servers

35. K8s Deployment template: client ID

36. K8s Deployment template

37. Improved availability
with local consumers.

38. • More availability
• Reduced latency
• Reduced storage costs
• Reduced network costs
Local consumers

39. Aggregate consumer

40. Local consumers

41. Local consumers: DNS records

42. Aggregate
500
ms
Regional
20
ms
Latency 99th
@V3_XD

43. Connect directly
through private IPs.

44. • More secure
• Reduced network costs
• Fetch from closest replica: KIP
-
392
Public to private tra
ff
ic

45. Tra
ff
ic manager: pod IPs

46. Reduction
-6%
bill
Network represents
29%
bill
Network cost reduction
@V3_XD

47. • GKE 1.24 -> 1.25
incident
• Apply
f
irewall rules
• LB more secure for
public tra
ff
ic

48. Failover: pod IPs

49. Single stop shop with Multi-
Cluster Services (MCS).

50. MCS endpoints
🔒closed source

51. Tra
ff
ic sources

52. Regional pairs: uneven distribution

53. Regionalize tra
ff
ic: Ka
f
ka access buddy
east

54. Regionalize tra
ff
ic: MCS

55. 40 18
MCS time savings
Minutes to regionalize tra
ff
ic Minutes to deploy
1 13
Minutes after migration Minutes after migration
@V3_XD

57. • Resiliency: DNS
tra
ff
ic management
• Toil: client wrappers
• Availability: local
consumption

58. Thanks!
@V3_XD