Upgrading Kubernetes can be difficult enough without the additional challenge of the petabytes of data associated with your Kafka cluster. At Shopify, we reduced our live Kafka cluster migration time from six months to two weeks by reusing persistent disks. In this talk I’ll explain why we chose this approach, how we did it, and how you can too. When upgrading Kubernetes it can be preferable to create a new Kubernetes cluster and migrate your workloads instead of performing an in-place upgrade. However, with this approach migrating Kafka to a new Kubernetes cluster over the network can be time consuming. I will go over how to stretch a Kafka cluster across the old and new Kubernetes clusters without adding any extra brokers. Finally, I will discuss how the Kafka brokers in the new Kubernetes cluster get scaled up while the old one gets decommissioned.

1. Vanessa Vuibert
Sta
ff
Production Engineer
Speed Up Your Kubernetes Upgrades For
Your Ka
f
ka Clusters

2. • Bug
f
ixes
• New features
• Mandatory at some point
Kubernetes (k8s) upgrades
are important!

3. 710 14
Ka
f
ka at Shopify
Brokers Ka
f
ka clusters
2 1PB
k8s upgrades per year Average storage usage

4. • Upgrade control plane
• Upgrade node pools
• Drain node pools
In-place k8s upgrades

5. • Cannot rollback
control plane
upgrade
• Leftover resources
• Cannot skip
versions

6. Create a new k8s cluster
and migrate your workloads.

7. Stateless app migration

8. Stateless app migration

9. Stateless app migration

10. Stateless app migration

11. Migrating a stateful
app is more complex.

12. Stateful app migration

13. Stateful app migration

14. Stateful app migration

15. Stateful app migration

16. Stretch the Ka
f
ka cluster between
old and new k8s clusters.

17. 1. Copy the data over
the network
2. Reuse the persistent
disks

18. • Deploy StatefulSet in new k8s cluster with
di
ff
erent broker IDs
• Use LinkedIn Cruise Control to remove brokers
• Remove source StatefulSet
Option 1: Copy the data over the network

19. Option 1: Copy the data over the network

20. Option 1: Copy the data over the network

21. Option 1: Copy the data over the network

22. Option 1: Copy the data over the network

23. 60% 20%
Top expense items
GCP bill in storage GCP bill in network
1PB 3GB/s
Average storage usage Average incoming tra
ff
ic

24. • Create persistent volumes (PVs)
• Create persistent volume claims (PVCs)
• Deploy StatefulSet in new k8s cluster
• Gradually scale down source StatefulSet
Option 2: Reuse the disks

25. Option 2: Reuse the disks

26. Option 2: Reuse the disks

27. Option 2: Reuse the disks

28. Option 2: Reuse the disks

29. Option 2: Reuse the disks

30. Option 2: Reuse the disks

31. Option 2: Reuse the disks

32. Option 2: Reuse the disks

33. Ka
f
ka k8s migrations
used to take
6
months copying data
over the network
now only takes
2
weeks reusing disks
Time savings

34. How to reuse the
persistent disks.

35. • New k8s cluster creation
• Create PVs and PVCs
• Rollback strategy
Preparation

36. Create new k8s cluster

37. Ka
f
ka StatefulSet: cross-cluster

38. Ka
f
ka StatefulSet: persistent storage

39. Ka
f
ka StatefulSet: PV and PVC

40. Create PVs

41. Create PVs and PVCs

42. Source preparation: rollback strategy

43. • Migrate half of Zookeeper servers
• Migrate Ka
f
ka
• Finish Zookeeper migration
Zookeeper and Ka
f
ka migration

44. Zookeeper StatefulSet: single-cluster

45. Zookeeper StatefulSet: cross-cluster

46. Zookeeper and Ka
f
ka migration

47. Zookeeper and Ka
f
ka migration

48. Zookeeper and Ka
f
ka migration

49. Zookeeper and Ka
f
ka migration

50. Zookeeper and Ka
f
ka migration

54. • Recreate k8s
clusters instead of
in-place upgrades
• Reuse disks

55. Thanks!