SlideShare a Scribd company logo
1 of 40
Download to read offline
A Future Journey:
How to Migrate 100
Clusters to GCP without
Downtime?
Tobias Schneck
Head of Professional Service
toschneck
@toschneck
tobi@kubermatic.com
Manuel Stößel
Systems Architect / Tech Lead
@ManuStoessel
@Manuel_Stoessel
manuel@kubermatic.com
What Else?
• Part of Professional Services @
Kubermatic
• Supporting customers on their
cloud-native journey
• Geeking out over Kubernetes and
adjacent technologies
Why Migrate Clusters?
Reasons for Cluster Migration Scenarios
● Business Reasons
● Better contract/conditions at another cloud provider ⇒ cost saving
● Data center migration to/from (cloud) providers
● Multi cloud strategy ⇒ decrease dependency to existing provider
● Technical Reasons
● Location migration of data centers
● Migrate to other network segments
● Adaption of on-prem / cloud improvements at new data center provider
● Data location of cloud offered service e.g. machine learning data
What Are the Main
Challenges?
Kubernetes Abstracts Infrastructure, But:
● Consummation of infrastructure resources
○ (Virtual) Machines
○ Network:
■ Network IP Address Spaces
■ Routing, Firewall
■ Ingress / Egress Traffic
○ DNS
○ External Storage Systems
● Cloud dependent Kubernetes components
○ Cloud Controller Manager
■ Node controller - responsible for updating kubernetes nodes
■ Service controller - responsible for services of type LoadBalancer
■ Route controller - responsible for setting up network routes
○ Storage Classes
○ (sometimes) Overlay Networking
K8s Master
API Server
Dashboard
Scheduler
kubelet kubelet kubelet kubelet
Container
Registry
etcd
Controllers
web browsers
kubectl
web browsers
Config
file
Image
CNI - Plugin Network (e.g. Flannel, Calico)
Developer
Developer
⇒ Application workload has the highest priority!
● Ensure fundamental networking rules at any time
○ All containers within a pod can communicate (L4) with each other unimpeded.
○ All pods can communicate with all other pods without NAT.
○ All nodes can communicate with all pods (and vice-versa) without NAT.
○ The IP that a pod sees itself as is the same IP that others see it as.
● External dependencies need to be reachable
○ External routed IPs for Load Balancers / Node Port Service
○ DNS Names need to be reachable
● Storage
○ State needs to migrated without data loss
Migration Without Downtime
Scale Level of 100 Clusters
● Larger organizations running a lot of clusters
⇒ different locations, org units, time zones
● Cluster users are only consumers
⇒ following the cluster as a service approach
● Cluster connection and secrets needs to be stable
=> no change of interface
Solution Approach
Status Quo
● Multi Cloud Setup with Kubermatic Kubernetes Platform (KKP)
○ Seed cluster hold containerized control plane of user clusters
○ Worker nodes provisioned by Cluster API conform Kubermatic machine-controller
○ Canal as default overlay network
● Target
○ Migrate user and seed cluster control planes and worker to different cloud
○ Keep external Cluster Endpoints stable
■ Control Plan: Kubernetes API Server endpoints
■ Application: DNS, Ingress
○ Out-of-Scope (for now): Storage replication
■ Assumption: Application Layer manages storage replication, e.g. etcd
Kubermatic
Kubermatic Master Cluster
KubeOne Seed Cluster - Region EU
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
Recommended Prerequisites
● Announce maintenance window and block cluster updates
● Ensure backups and recovery procedure for
○ Seed and user clusters
○ Application workload
● Create target cloud cluster as reference
● Ensure control of DNS entries
Migrate User Clusters
Solution Approach
1) Migrate User Cluster Workers
● Create new worker nodes in target cloud
⇒ Machine controller with new Machine Deployment at target cloud
● User worker nodes and Pods need to talk to each other at any time
⇒ Strap a VPN overlay by DaemonSets across current and target cloud
⇒ Route overlay CNI traffic through VPN network
● Ensure reachability
=> Keep old and create new cluster Ingress endpoints
=> Transfer workload to new cloud
=> Delete after workload / connectivity is ensured
KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
VPN Server Machine
Controller
Application
User k8s
Worker
Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(eth0)
MetalLB
Migrate User Cluster Worker Nodes:
KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
VPN Server Machine
Controller
User k8s
Worker
Application Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
VPN
Client
VPN
Client
MetalLB
KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
Machine
Controller
VPN Server
User k8s
Worker
Application Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
VPN
Client
VPN
Client
User k8s
Worker
VPN
Client
MetalLB
User k8s
Worker
VPN
Client
GCP LB
KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
Machine
Controller
User k8s
Worker
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
7. Test new cluster ingress entrypoint
8. Migrate Workload and update DNS
VPN
Client
VPN
Client
User k8s
Worker
VPN
Client
MetalLB GCP LB
User k8s
Worker
VPN
Client
Application Application
KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
VPN Server Machine
Controller
*.cluster-1.example.com
K8s API Server tunnel
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
7. Test new cluster ingress entrypoint
8. Migrate Workload and update DNS
9. Cleanup old cloud resource
User k8s
Worker
GCP LB
User k8s
Worker
Application Application
Canal Overlay
(eth0)
github.com/kubermatic-labs/cluster-migration
DEMO Time 🎉
Migrate Seed Cluster
Solution Approach
Kubermatic
KubeOne Master Cluster
migrated
KubeOne Seed Cluster - Region EU
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
migrated
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
2) Migrate Seed Cluster
● Create new seed master nodes at new cloud
=> New Kubernetes API Load Balancer
=> API Endpoint needs to be updated by DNS
=> Block seed cluster upgrades to ensure worst case recovery
● Migrate user cluster control plane
=> Handle migration the same way (like user cluster workload)
=> Ensure etcd quorum and migration by data replication
=> Block user cluster upgrades to ensure worst case recovery
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
4. Add new LB Service & Update DNS
5. Remove 2 old Master Nodes and move
etcd quorum to new cloud
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Master
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
4. Add new LB Service & Update DNS
5. Remove 2 old Master Nodes and move
etcd quorum to new cloud
6. Create 3rd Master Node at new cloud
and remove last old Master Node
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Migrate Seed Cluster Workers
Solution Approach
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
7. Drain missing worker nodes, cleanup old cloud
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
7. Drain missing worker nodes, cleanup old cloud
8. Scale down etcd count of user cluster to 3
9. Remove VPN Overlay
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Master
Canal Overlay
(eth0)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
Outlook
● Automate clean up procedure
○ Idea: switch back cloud provider / machine controller for clean up
● Manage migration by Operator
○ Health checks
○ Wait conditions for migration steps
● Stabilize VPN connection
○ Multiple VPN servers
○ Soft switchover between VPN / Host network overlay
○ Evaluate Wireguard usage
● Automate Load Balancer and DNS management
Any Questions?
Join Us For Our Next Webinar!
February 23 at 8 AM CET / 6 PM AEST
Empower Your DevOps
Organization With Kubernetes
We’re Hiring!
Check Out Our Job Openings:
https://careers.smartrecruiters.com/KubermaticGmbH
kubermatic/kubermatic
@kubermatic
marketing@kubermatic.com

More Related Content

What's hot

Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
QAware GmbH
 

What's hot (20)

Enabling ceph-mgr to control Ceph services via Kubernetes
Enabling ceph-mgr to control Ceph services via KubernetesEnabling ceph-mgr to control Ceph services via Kubernetes
Enabling ceph-mgr to control Ceph services via Kubernetes
 
Kubernetes and Istio
Kubernetes and IstioKubernetes and Istio
Kubernetes and Istio
 
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
Kubernetes Concepts And Architecture Powerpoint Presentation SlidesKubernetes Concepts And Architecture Powerpoint Presentation Slides
Kubernetes Concepts And Architecture Powerpoint Presentation Slides
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetes
 
Openstack days sv building highly available services using kubernetes (preso)
Openstack days sv   building highly available services using kubernetes (preso)Openstack days sv   building highly available services using kubernetes (preso)
Openstack days sv building highly available services using kubernetes (preso)
 
Introduction to Kubernetes
Introduction to KubernetesIntroduction to Kubernetes
Introduction to Kubernetes
 
Kubernetes and Cloud Native Update Q4 2018
Kubernetes and Cloud Native Update Q4 2018Kubernetes and Cloud Native Update Q4 2018
Kubernetes and Cloud Native Update Q4 2018
 
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
Kubernetes Architecture | Understanding Kubernetes Components | Kubernetes Tu...
 
Istio canaries and kubernetes
Istio  canaries and kubernetesIstio  canaries and kubernetes
Istio canaries and kubernetes
 
Securing and Automating Kubernetes with Kyverno
Securing and Automating Kubernetes with KyvernoSecuring and Automating Kubernetes with Kyverno
Securing and Automating Kubernetes with Kyverno
 
The Operator Pattern - Managing Stateful Services in Kubernetes
The Operator Pattern - Managing Stateful Services in KubernetesThe Operator Pattern - Managing Stateful Services in Kubernetes
The Operator Pattern - Managing Stateful Services in Kubernetes
 
Social Connections 14 - Kubernetes Basics for Connections Admins
Social Connections 14 - Kubernetes Basics for Connections AdminsSocial Connections 14 - Kubernetes Basics for Connections Admins
Social Connections 14 - Kubernetes Basics for Connections Admins
 
Read ebook Kubernetes Cookbook: Building Cloud Native Applications Full Access
Read ebook Kubernetes Cookbook: Building Cloud Native Applications Full AccessRead ebook Kubernetes Cookbook: Building Cloud Native Applications Full Access
Read ebook Kubernetes Cookbook: Building Cloud Native Applications Full Access
 
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
Kubernetes One-Click Deployment: Hands-on Workshop (Munich)
 
Mattia Gandolfi - Improving utilization and portability with Containers and C...
Mattia Gandolfi - Improving utilization and portability with Containers and C...Mattia Gandolfi - Improving utilization and portability with Containers and C...
Mattia Gandolfi - Improving utilization and portability with Containers and C...
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
Federated mesos clusters for global data center designs
Federated mesos clusters for global data center designsFederated mesos clusters for global data center designs
Federated mesos clusters for global data center designs
 
Salvatore Incandela, Fabio Marinelli - Using Spinnaker to Create a Developmen...
Salvatore Incandela, Fabio Marinelli - Using Spinnaker to Create a Developmen...Salvatore Incandela, Fabio Marinelli - Using Spinnaker to Create a Developmen...
Salvatore Incandela, Fabio Marinelli - Using Spinnaker to Create a Developmen...
 
Kubernetes on Bare Metal at the Kitchener-Waterloo Kubernetes and Cloud Nativ...
Kubernetes on Bare Metal at the Kitchener-Waterloo Kubernetes and Cloud Nativ...Kubernetes on Bare Metal at the Kitchener-Waterloo Kubernetes and Cloud Nativ...
Kubernetes on Bare Metal at the Kitchener-Waterloo Kubernetes and Cloud Nativ...
 
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
 

Similar to Kubermatic How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime?

Kubernetes for Beginners
Kubernetes for BeginnersKubernetes for Beginners
Kubernetes for Beginners
DigitalOcean
 
kubernetesssssssssssssssssssssssssss.pdf
kubernetesssssssssssssssssssssssssss.pdfkubernetesssssssssssssssssssssssssss.pdf
kubernetesssssssssssssssssssssssssss.pdf
bchiriamina2
 

Similar to Kubermatic How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime? (20)

Kubermatic.pdf
Kubermatic.pdfKubermatic.pdf
Kubermatic.pdf
 
Kubermatic CNCF Webinar - start.kubermatic.pdf
Kubermatic CNCF Webinar - start.kubermatic.pdfKubermatic CNCF Webinar - start.kubermatic.pdf
Kubermatic CNCF Webinar - start.kubermatic.pdf
 
Introduction to istio
Introduction to istioIntroduction to istio
Introduction to istio
 
Container orchestration and microservices world
Container orchestration and microservices worldContainer orchestration and microservices world
Container orchestration and microservices world
 
Running Production-Grade Kubernetes on AWS
Running Production-Grade Kubernetes on AWSRunning Production-Grade Kubernetes on AWS
Running Production-Grade Kubernetes on AWS
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the Datacenter
 
Kubernetes for Beginners
Kubernetes for BeginnersKubernetes for Beginners
Kubernetes for Beginners
 
CN Asturias - Stateful application for kubernetes
CN Asturias -  Stateful application for kubernetes CN Asturias -  Stateful application for kubernetes
CN Asturias - Stateful application for kubernetes
 
Scaling Kubernetes to Support 50000 Services.pptx
Scaling Kubernetes to Support 50000 Services.pptxScaling Kubernetes to Support 50000 Services.pptx
Scaling Kubernetes to Support 50000 Services.pptx
 
Workday's Next Generation Private Cloud
Workday's Next Generation Private CloudWorkday's Next Generation Private Cloud
Workday's Next Generation Private Cloud
 
kubernetesssssssssssssssssssssssssss.pdf
kubernetesssssssssssssssssssssssssss.pdfkubernetesssssssssssssssssssssssssss.pdf
kubernetesssssssssssssssssssssssssss.pdf
 
Multi-Cloud Orchestration for Kubernetes with Cloudify
Multi-Cloud Orchestration for Kubernetes with CloudifyMulti-Cloud Orchestration for Kubernetes with Cloudify
Multi-Cloud Orchestration for Kubernetes with Cloudify
 
OSS Japan 2019 service mesh bridging Kubernetes and legacy
OSS Japan 2019 service mesh bridging Kubernetes and legacyOSS Japan 2019 service mesh bridging Kubernetes and legacy
OSS Japan 2019 service mesh bridging Kubernetes and legacy
 
Docker on docker leveraging kubernetes in docker ee
Docker on docker leveraging kubernetes in docker eeDocker on docker leveraging kubernetes in docker ee
Docker on docker leveraging kubernetes in docker ee
 
Webinar- Tea for the Tillerman
Webinar- Tea for the TillermanWebinar- Tea for the Tillerman
Webinar- Tea for the Tillerman
 
Puppet Camp Berlin 2015: Configuration Management @ CERN: Going Agile with Style
Puppet Camp Berlin 2015: Configuration Management @ CERN: Going Agile with StylePuppet Camp Berlin 2015: Configuration Management @ CERN: Going Agile with Style
Puppet Camp Berlin 2015: Configuration Management @ CERN: Going Agile with Style
 
Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: G...
Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: G...Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: G...
Puppet Camp Berlin 2015: Andrea Giardini | Configuration Management @ CERN: G...
 
Multi-Cloud Orchestration for Kubernetes with Cloudify - Webinar Presentation
Multi-Cloud Orchestration for Kubernetes with Cloudify - Webinar PresentationMulti-Cloud Orchestration for Kubernetes with Cloudify - Webinar Presentation
Multi-Cloud Orchestration for Kubernetes with Cloudify - Webinar Presentation
 
Overview of OpenDaylight Container Orchestration Engine Integration
Overview of OpenDaylight Container Orchestration Engine IntegrationOverview of OpenDaylight Container Orchestration Engine Integration
Overview of OpenDaylight Container Orchestration Engine Integration
 
Unleashing k8 s to reduce complexities of an entire middleware platform
Unleashing k8 s to reduce complexities of an entire middleware platformUnleashing k8 s to reduce complexities of an entire middleware platform
Unleashing k8 s to reduce complexities of an entire middleware platform
 

More from Tobias Schneck

Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes MeetupCreating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
Tobias Schneck
 
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
Tobias Schneck
 
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgartOpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
Tobias Schneck
 
OpenShift-Build-Pipelines: Build ► Test ► Run!
OpenShift-Build-Pipelines: Build ► Test ► Run!OpenShift-Build-Pipelines: Build ► Test ► Run!
OpenShift-Build-Pipelines: Build ► Test ► Run!
Tobias Schneck
 
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
Tobias Schneck
 
Testing - Selenium? Rich-Clients? Containers?
Testing - Selenium? Rich-Clients? Containers?Testing - Selenium? Rich-Clients? Containers?
Testing - Selenium? Rich-Clients? Containers?
Tobias Schneck
 
OOP2017: Containerized End-2-End Testing – automate it!
OOP2017: Containerized End-2-End Testing – automate it!OOP2017: Containerized End-2-End Testing – automate it!
OOP2017: Containerized End-2-End Testing – automate it!
Tobias Schneck
 

More from Tobias Schneck (19)

ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes MeetupCreating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
Creating Kubernetes multi clusters with ClusterAPI @ Stuttgart Kubernetes Meetup
 
KubeCI - Cloud Native Continuous Delivery for Kubernetes
KubeCI - Cloud Native Continuous Delivery for KubernetesKubeCI - Cloud Native Continuous Delivery for Kubernetes
KubeCI - Cloud Native Continuous Delivery for Kubernetes
 
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
UI Testing - Selenium? Rich-Clients? Containers? (SwanseaCon 2018)
 
Creating Kubernetes multi clusters with ClusterAPI in the Hetzner Cloud
Creating Kubernetes multi clusters with ClusterAPI in the Hetzner CloudCreating Kubernetes multi clusters with ClusterAPI in the Hetzner Cloud
Creating Kubernetes multi clusters with ClusterAPI in the Hetzner Cloud
 
OpenShift Build Pipelines @ Lightweight Java User Group Meetup
OpenShift Build Pipelines @ Lightweight Java User Group MeetupOpenShift Build Pipelines @ Lightweight Java User Group Meetup
OpenShift Build Pipelines @ Lightweight Java User Group Meetup
 
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgartOpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
OpenShift-Build-Pipelines: Build -> Test -> Run! @JavaForumStuttgart
 
OpenShift-Build-Pipelines: Build ► Test ► Run!
OpenShift-Build-Pipelines: Build ► Test ► Run!OpenShift-Build-Pipelines: Build ► Test ► Run!
OpenShift-Build-Pipelines: Build ► Test ► Run!
 
Kotlin for backend development (Hackaburg 2018 Regensburg)
Kotlin for backend development (Hackaburg 2018 Regensburg)Kotlin for backend development (Hackaburg 2018 Regensburg)
Kotlin for backend development (Hackaburg 2018 Regensburg)
 
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
UI-Testing - Selenium? Rich-Clients? Containers? @APEX connect 2018
 
Continuous Testing: Integration- und UI-Testing mit OpenShift-Build-Pipelines
Continuous Testing: Integration- und UI-Testing mit OpenShift-Build-PipelinesContinuous Testing: Integration- und UI-Testing mit OpenShift-Build-Pipelines
Continuous Testing: Integration- und UI-Testing mit OpenShift-Build-Pipelines
 
Testing - Selenium? Rich-Clients? Containers?
Testing - Selenium? Rich-Clients? Containers?Testing - Selenium? Rich-Clients? Containers?
Testing - Selenium? Rich-Clients? Containers?
 
OOP2017: Containerized End-2-End Testing – automate it!
OOP2017: Containerized End-2-End Testing – automate it!OOP2017: Containerized End-2-End Testing – automate it!
OOP2017: Containerized End-2-End Testing – automate it!
 
Containerized End-2-End Testing - Agile Testing Meetup at Süddeutsche Zeitung...
Containerized End-2-End Testing - Agile Testing Meetup at Süddeutsche Zeitung...Containerized End-2-End Testing - Agile Testing Meetup at Süddeutsche Zeitung...
Containerized End-2-End Testing - Agile Testing Meetup at Süddeutsche Zeitung...
 
Containerized End-2-End-Testing - ContainerConf Mannheim
Containerized End-2-End-Testing - ContainerConf MannheimContainerized End-2-End-Testing - ContainerConf Mannheim
Containerized End-2-End-Testing - ContainerConf Mannheim
 
Containerized End-2-End-Testing - Software-QS-Tag (deutsch)
Containerized End-2-End-Testing - Software-QS-Tag (deutsch)Containerized End-2-End-Testing - Software-QS-Tag (deutsch)
Containerized End-2-End-Testing - Software-QS-Tag (deutsch)
 
Containerized End-2-End Testing - JUG Saxony Day
Containerized End-2-End Testing - JUG Saxony DayContainerized End-2-End Testing - JUG Saxony Day
Containerized End-2-End Testing - JUG Saxony Day
 
Skale your test environment! Containerized End-2-End-Testing @Herbstcampus Nü...
Skale your test environment! Containerized End-2-End-Testing @Herbstcampus Nü...Skale your test environment! Containerized End-2-End-Testing @Herbstcampus Nü...
Skale your test environment! Containerized End-2-End-Testing @Herbstcampus Nü...
 
Containerized End-2-End-Testing - Tobias Schneck
Containerized End-2-End-Testing - Tobias SchneckContainerized End-2-End-Testing - Tobias Schneck
Containerized End-2-End-Testing - Tobias Schneck
 

Recently uploaded

Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 

Kubermatic How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime?

  • 1. A Future Journey: How to Migrate 100 Clusters to GCP without Downtime?
  • 2. Tobias Schneck Head of Professional Service toschneck @toschneck tobi@kubermatic.com Manuel Stößel Systems Architect / Tech Lead @ManuStoessel @Manuel_Stoessel manuel@kubermatic.com What Else? • Part of Professional Services @ Kubermatic • Supporting customers on their cloud-native journey • Geeking out over Kubernetes and adjacent technologies
  • 4. Reasons for Cluster Migration Scenarios ● Business Reasons ● Better contract/conditions at another cloud provider ⇒ cost saving ● Data center migration to/from (cloud) providers ● Multi cloud strategy ⇒ decrease dependency to existing provider ● Technical Reasons ● Location migration of data centers ● Migrate to other network segments ● Adaption of on-prem / cloud improvements at new data center provider ● Data location of cloud offered service e.g. machine learning data
  • 5. What Are the Main Challenges?
  • 6. Kubernetes Abstracts Infrastructure, But: ● Consummation of infrastructure resources ○ (Virtual) Machines ○ Network: ■ Network IP Address Spaces ■ Routing, Firewall ■ Ingress / Egress Traffic ○ DNS ○ External Storage Systems ● Cloud dependent Kubernetes components ○ Cloud Controller Manager ■ Node controller - responsible for updating kubernetes nodes ■ Service controller - responsible for services of type LoadBalancer ■ Route controller - responsible for setting up network routes ○ Storage Classes ○ (sometimes) Overlay Networking
  • 7. K8s Master API Server Dashboard Scheduler kubelet kubelet kubelet kubelet Container Registry etcd Controllers web browsers kubectl web browsers Config file Image CNI - Plugin Network (e.g. Flannel, Calico) Developer Developer
  • 8. ⇒ Application workload has the highest priority! ● Ensure fundamental networking rules at any time ○ All containers within a pod can communicate (L4) with each other unimpeded. ○ All pods can communicate with all other pods without NAT. ○ All nodes can communicate with all pods (and vice-versa) without NAT. ○ The IP that a pod sees itself as is the same IP that others see it as. ● External dependencies need to be reachable ○ External routed IPs for Load Balancers / Node Port Service ○ DNS Names need to be reachable ● Storage ○ State needs to migrated without data loss Migration Without Downtime
  • 9. Scale Level of 100 Clusters ● Larger organizations running a lot of clusters ⇒ different locations, org units, time zones ● Cluster users are only consumers ⇒ following the cluster as a service approach ● Cluster connection and secrets needs to be stable => no change of interface
  • 11. Status Quo ● Multi Cloud Setup with Kubermatic Kubernetes Platform (KKP) ○ Seed cluster hold containerized control plane of user clusters ○ Worker nodes provisioned by Cluster API conform Kubermatic machine-controller ○ Canal as default overlay network ● Target ○ Migrate user and seed cluster control planes and worker to different cloud ○ Keep external Cluster Endpoints stable ■ Control Plan: Kubernetes API Server endpoints ■ Application: DNS, Ingress ○ Out-of-Scope (for now): Storage replication ■ Assumption: Application Layer manages storage replication, e.g. etcd
  • 12. Kubermatic Kubermatic Master Cluster KubeOne Seed Cluster - Region EU User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd
  • 13. Recommended Prerequisites ● Announce maintenance window and block cluster updates ● Ensure backups and recovery procedure for ○ Seed and user clusters ○ Application workload ● Create target cloud cluster as reference ● Ensure control of DNS entries
  • 15. 1) Migrate User Cluster Workers ● Create new worker nodes in target cloud ⇒ Machine controller with new Machine Deployment at target cloud ● User worker nodes and Pods need to talk to each other at any time ⇒ Strap a VPN overlay by DaemonSets across current and target cloud ⇒ Route overlay CNI traffic through VPN network ● Ensure reachability => Keep old and create new cluster Ingress endpoints => Transfer workload to new cloud => Delete after workload / connectivity is ensured
  • 16. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker VPN Server Machine Controller Application User k8s Worker Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (eth0) MetalLB Migrate User Cluster Worker Nodes:
  • 17. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker VPN Server Machine Controller User k8s Worker Application Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment VPN Client VPN Client MetalLB
  • 18. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker Machine Controller VPN Server User k8s Worker Application Application *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment VPN Client VPN Client User k8s Worker VPN Client MetalLB User k8s Worker VPN Client GCP LB
  • 19. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd User k8s Worker Machine Controller User k8s Worker *.cluster-1.example.com K8s API Server tunnel Canal Overlay (kube) Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment 7. Test new cluster ingress entrypoint 8. Migrate Workload and update DNS VPN Client VPN Client User k8s Worker VPN Client MetalLB GCP LB User k8s Worker VPN Client Application Application
  • 20. KubeOne Seed Cluster - Region EU User k8s Master API Controller Scheduler etcd VPN Server Machine Controller *.cluster-1.example.com K8s API Server tunnel Migrate User Cluster Worker Nodes: 1. VPN Daemon Set with client-to-client communication 2. Route Overlay Traffic over VPN interface 3. Pause existing Cluster & Machine Deployment 4. Update Cluster Spec & Cloud Credentials 5. Unpause Cluster with new Cloud Provider 6. Apply new Machine Deployment 7. Test new cluster ingress entrypoint 8. Migrate Workload and update DNS 9. Cleanup old cloud resource User k8s Worker GCP LB User k8s Worker Application Application Canal Overlay (eth0)
  • 23. Kubermatic KubeOne Master Cluster migrated KubeOne Seed Cluster - Region EU User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd migrated User k8s Worker User k8s Worker User k8s Master API Controller Scheduler etcd
  • 24. 2) Migrate Seed Cluster ● Create new seed master nodes at new cloud => New Kubernetes API Load Balancer => API Endpoint needs to be updated by DNS => Block seed cluster upgrades to ensure worst case recovery ● Migrate user cluster control plane => Handle migration the same way (like user cluster workload) => Ensure etcd quorum and migration by data replication => Block user cluster upgrades to ensure worst case recovery
  • 25. User k8s Master API Scheduler Controller etcd KubeOne Seed Cluster - Region EU Seed k8s Master Seed k8s Master Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 26. KubeOne Seed Cluster - Region EU Seed k8s Master Seed k8s Master Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 27. KubeOne Seed Cluster - Region EU seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes 4. Add new LB Service & Update DNS 5. Remove 2 old Master Nodes and move etcd quorum to new cloud Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Master User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 28. KubeOne Seed Cluster - Region EU Seed k8s Master seed-k8s-api.example.com Seed k8s Worker Seed k8s Worker Seed k8s Worker Migrate Seed Master Nodes: 1. Setup VPN Overlay 2. Pause existing Cluster & Machine Deployment 3. Create and join new 2 Master Nodes 4. Add new LB Service & Update DNS 5. Remove 2 old Master Nodes and move etcd quorum to new cloud 6. Create 3rd Master Node at new cloud and remove last old Master Node Seed k8s Master Seed k8s Master VPN Server Canal Overlay (kube) User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 29. Migrate Seed Cluster Workers Solution Approach
  • 30. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 31. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 32. KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 33. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 34. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Worker Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 35. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud 7. Drain missing worker nodes, cleanup old cloud KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Master VPN Server Canal Overlay (kube) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 36. Migrate Seed Worker Nodes: 1. VPN Overlay, Pause existing Cluster, Machine Deployment 2. Create 2 new Workers (migration steps similar to user cluster) 3. Taint existing workers as non-schedule 4. Scale up etcd count of user cluster to 5 ⇒ data replicated by etcd 5. Create new LB for NodePort Proxy and update DNS 6. Add 1 new worker and drain 1 old workers ⇒ etcd quorum migrated to new cloud 7. Drain missing worker nodes, cleanup old cloud 8. Scale down etcd count of user cluster to 3 9. Remove VPN Overlay KubeOne Seed Cluster - Region EU K8s API Server tunnels Seed k8s Master Seed k8s Master *.seed.example.com User k8s Worker User k8s Worker User k8s Worker [cluster-id] NodePort Proxy Service Seed k8s Worker Seed k8s Master Canal Overlay (eth0) Seed k8s Worker Seed k8s Worker User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd User k8s Master API Scheduler Controller etcd
  • 37. Outlook ● Automate clean up procedure ○ Idea: switch back cloud provider / machine controller for clean up ● Manage migration by Operator ○ Health checks ○ Wait conditions for migration steps ● Stabilize VPN connection ○ Multiple VPN servers ○ Soft switchover between VPN / Host network overlay ○ Evaluate Wireguard usage ● Automate Load Balancer and DNS management
  • 39. Join Us For Our Next Webinar! February 23 at 8 AM CET / 6 PM AEST Empower Your DevOps Organization With Kubernetes
  • 40. We’re Hiring! Check Out Our Job Openings: https://careers.smartrecruiters.com/KubermaticGmbH kubermatic/kubermatic @kubermatic marketing@kubermatic.com