How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime
•
0 likes•222 views
Have you ever thought about migrating your Kubernetes clusters to Google Cloud to get your services closer to your customers? Yes? Us too! Join us on an interactive journey to discover the main challenges of live migration at scale of etcd’s, traffic routing and application workloads from your on-premise platform to GCP. The talk will discuss the current state of the technical concept, known problems and insides of the already proven migration steps for stateless workloads. As part of the journey, we'll see - The differences between migrating one or one hundred clusters with productive workloads - What parts can be automated? - What steps may need to be done manually?
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime
Similar to How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime (20)
More from loodse
Recently uploaded
Recently uploaded (20)
How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime
- 1. A Future Journey: How to Migrate 100 Clusters to GCP without Downtime?
- 2. Tobias Schneck Head of Professional Service toschneck @toschneck tobi@kubermatic.com Manuel Stößel Systems Architect / Tech Lead @ManuStoessel @Manuel_Stoessel manuel@kubermatic.com What Else? • Part of Professional Services @ Kubermatic • Supporting customers on their cloud-native journey • Geeking out over Kubernetes and adjacent technologies
- 4. Reasons for Cluster Migration Scenarios ● Business Reasons ● Better contract/conditions at another cloud provider ⇒ cost saving ● Data center migration to/from (cloud) providers ● Multi cloud strategy ⇒ decrease dependency to existing provider ● Technical Reasons ● Location migration of data centers ● Migrate to other network segments ● Adaption of on-prem / cloud improvements at new data center provider ● Data location of cloud offered service e.g. machine learning data
- 5. What Are the Main Challenges?
- 6. Kubernetes Abstracts Infrastructure, But: ● Consummation of infrastructure resources ○ (Virtual) Machines ○ Network: ■ Network IP Address Spaces ■ Routing, Firewall ■ Ingress / Egress Traffic ○ DNS ○ External Storage Systems ● Cloud dependent Kubernetes components ○ Cloud Controller Manager ■ Node controller - responsible for updating kubernetes nodes ■ Service controller - responsible for services of type LoadBalancer ■ Route controller - responsible for setting up network routes ○ Storage Classes ○ (sometimes) Overlay Networking
- 7. K8s Master API Server Dashboard Scheduler kubelet kubelet kubelet kubelet Container Registry etcd Controllers web browsers kubectl web browsers Config file Image CNI - Plugin Network (e.g. Flannel, Calico) Developer Developer
- 8. ⇒ Application workload has the highest priority! ● Ensure fundamental networking rules at any time ○ All containers within a pod can communicate (L4) with each other unimpeded. ○ All pods can communicate with all other pods without NAT. ○ All nodes can communicate with all pods (and vice-versa) without NAT. ○ The IP that a pod sees itself as is the same IP that others see it as. ● External dependencies need to be reachable ○ External routed IPs for Load Balancers / Node Port Service ○ DNS Names need to be reachable ● Storage ○ State needs to migrated without data loss Migration Without Downtime
- 9. Scale Level of 100 Clusters ● Larger organizations running a lot of clusters ⇒ different locations, org units, time zones ● Cluster users are only consumers ⇒ following the cluster as a service approach ● Cluster connection and secrets needs to be stable => no change of interface
- 11. Status Quo ● Multi Cloud Setup with Kubermatic Kubernetes Platform (KKP) ○ Seed cluster hold containerized control plane of user clusters ○ Worker nodes provisioned by Cluster API conform Kubermatic machine-controller ○ Canal as default overlay network ● Target ○ Migrate user and seed cluster control planes and worker to different cloud ○ Keep external Cluster Endpoints stable ■ Control Plan: Kubernetes API Server endpoints ■ Application: DNS, Ingress ○ Out-of-Scope (for now): Storage replication ■ Assumption: Application Layer manages storage replication, e.g. etcd
- 12. Kubermatic Kubermatic Master Cluster KubeOne Seed Cluster - Region EU User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd
- 13. Recommended Prerequisites ● Announce maintenance window and block cluster updates ● Ensure backups and recovery procedure for ○ Seed and user clusters ○ Application workload ● Create target cloud cluster as reference ● Ensure control of DNS entries
- 14. Migrate User Clusters Solution Approach
- 15. 1) Migrate User Cluster Workers ● Create new worker nodes in target cloud ⇒ Machine controller with new Machine Deployment at target cloud ● User worker nodes and Pods need to talk to each other at any time ⇒ Strap a VPN overlay by DaemonSets across current and target cloud ⇒ Route overlay CNI traffic through VPN network ● Ensure reachability => Keep old and create new cluster Ingress endpoints => Transfer workload to new cloud => Delete after workload / connectivity is ensured
- 16. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker VPN Server Machine Controller Application User k8s Worker Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (eth0) MetalLB Migrate User Cluster Worker Nodes:
- 17. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker VPN Server Machine Controller User k8s Worker Application Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment VPN Client VPN Client MetalLB
- 18. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker Machine Controller VPN Server User k8s Worker Application Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment VPN Client VPN Client User k8s Worker VPN Client MetalLB User k8s Worker VPN Client GCP LB
- 19. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker Machine Controller User k8s Worker *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment 7. Test new cluster ingress entrypoint 8. Migrate Workload and update DNS VPN Client VPN Client User k8s Worker VPN Client MetalLB GCP LB User k8s Worker VPN Client Application Application
- 20. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd VPN Server Machine Controller *.cluster-1.example.com K8s API Server tunnel Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment 7. Test new cluster ingress entrypoint 8. Migrate Workload and update DNS 9. Cleanup old cloud resource User k8s Worker GCP LB User k8s Worker Application Application Canal Overlay (eth0)
- 22. Migrate Seed Cluster Solution Approach
- 23. Kubermatic KubeOne Master Cluster migrated KubeOne Seed Cluster - Region EU User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd migrated User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd
- 24. 2) Migrate Seed Cluster ● Create new seed master nodes at new cloud => New Kubernetes API Load Balancer => API Endpoint needs to be updated by DNS => Block seed cluster upgrades to ensure worst case recovery ● Migrate user cluster control plane => Handle migration the same way (like user cluster workload) => Ensure etcd quorum and migration by data replication => Block user cluster upgrades to ensure worst case recovery
- 25. User k8s Master API Scheduler Controller etcd KubeOne Seed Cluster - Region EU Seed k8s Master Seed k8s Master Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 26. KubeOne Seed Cluster - Region EU Seed k8s Master Seed k8s Master Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 27. KubeOne Seed Cluster - Region EU seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes 4. Add new LB Service & Update DNS 5. Remove 2 old Master Nodes and move etcd quorum to new cloud Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Master User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 28. KubeOne Seed Cluster - Region EU Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes 4. Add new LB Service & Update DNS 5. Remove 2 old Master Nodes and move etcd quorum to new cloud 6. Create 3rd Master Node at new cloud and remove last old Master Node Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 29. Migrate Seed Cluster Workers Solution Approach
- 30. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 31. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 32. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 33. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 34. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 35. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud 7. Drain missing worker nodes, cleanup old cloud KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 36. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud 7. Drain missing worker nodes, cleanup old cloud 8. Scale down etcd count of user cluster to 3 9. Remove VPN Overlay KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Master Canal Overlay (eth0) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
- 37. Outlook ● Automate clean up procedure ○ Idea: switch back cloud provider / machine controller for clean up ● Manage migration by Operator ○ Health checks ○ Wait conditions for migration steps ● Stabilize VPN connection ○ Multiple VPN servers ○ Soft switchover between VPN / Host network overlay ○ Evaluate Wireguard usage ● Automate Load Balancer and DNS management
- 38. Any Questions?
- 39. Join Us For Our Next Webinar! February 23 at 8 AM CET / 6 PM AEST Empower Your DevOps Organization With Kubernetes
- 40. We’re Hiring! Check Out Our Job Openings: https://careers.smartrecruiters.com/KubermaticGmbH kubermatic/kubermatic @kubermatic marketing@kubermatic.com