Multiple clusters exist in most Kubernetes environments today, and number of clusters will increase overtime. The reasons for having multiple Kubernetes clusters are many, for example, overcoming scale limits, reducing complexity, geo separation, redundancy and having separate production, staging, and development environments. Once you have multiple K8S clusters, it can be useful to have the ability to easily move or duplicate workloads across these different clusters. Kubernetes does not have a native method to allow migration or duplication of workloads across clusters. Fortunately, there are tools that provide this functionality. In this presentation we will explore the different uses cases for cross cluster migration, and what is involved, and how these migration tools work. We'll cover some popular uses cases, such as, Disaster Recovery, Test/Dev, and performance testing. Migration could entail moving the entire cluster, or individual workloads. The components that need to be moved would include configuration and resources stored in etcd, and persistent data residing on PVCs. We'll cover the uses cases and challenges for migration, and run through an example of using one of these migration tools. This talk was given by Ryan Kaw for DoK Day Europe @ KubeCon 2022.

1. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”

2. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q Many reasons for having multiple Kubernetes clusters
q Operations - overcoming scale limits, reducing complexity, geo separation, redundancy
q CI/CD pipeline - separate production, staging, and development environments
q Hybrid environments may have clusters on-prem and cloud, or different cloud vendors.
Production Staging Development QA
Cloud On-Prem
Why Multiple Kubernetes Clusters?
2

3. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q Major platform or infrastructure upgrades (upgrade Kubernetes version, compute and storage hardware)
q Move to different location data center, region, or migrate to cloud
q Migrate data between different cloud vendors, or across cloud accounts
q Clone production workload to staging or QA clusters to test code updates, as part of dev workflow
q Migrate workloads if your production clusters are having hardware issues
q Reduce cluster overload by moving some workloads to new cluster
Production Development
Workload + data
Cross-Cluster Data Management Use Cases
3

4. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Kubernetes clusters are typically isolated from each other. One cluster does not have any knowledge about
another cluster. They are both isolated in terms of storage and networking.
Production Staging Development QA Cloud
Challenges: Kubernetes Cluster are Isolated
4

5. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q Moving persistent workloads between clusters is a challenge.
q Persistent data resides on Persistent Volumes (PVs) within a cluster.
q No native Kubernetes infrastructure to copy, clone, or migrate PVs between clusters.
Production Staging Development QA Cloud
PV PV PV PV PV
Persistent Workloads
5

6. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q No network communications between source and target cluster other then publicly exposed services
q PV data is not typically exposed on the network for Ingress or Egress
q Need a way to transfer data without connectivity between clusters, or exposing PV to public network
q Make sure not to overload network bandwidth, storage bandwidth
q Don’t want to affect production environment during transfers
q Applications should always reference both internal and external services using DNS names, rather than IP
addresses.
Networking Challenges with Moving Persistent Workloads Across Clusters
Workload
Cluster1
PV PV
Cluster2
6

7. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q Entire Cluster
q Re-create an entire cluster
q During migration, change properties on target cluster for DevOps purposes
q Don’t need a full cost production environment in QA, staging, or development.
q Granular selection
q You may only want to move workloads within a particular namespace or label
q Move both etcd resources and PV in a logical unit so that application remains intact
Migration: Entire Cluster or Granular
AWS EKS cluster2
Region: us-east
namespace
AWS EKS Cluster1
Region: us-west
PV PV
Label
7

8. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q PVs on each cluster could be using different storage classes
q In the case below, we want to clone our persistent workload from Cluster1 to Cluster2
q PV on cluster 1 is using the ebs CSI driver, however, cluster2 does not have CSI enabled and is using
storageclass gp2.
Storage Class Differences
Storageclass: ebs-sc
AWS EKS Cluster2
Region: us-east
workload
AWS EKS Cluster1
Region: us-west
PV Storageclass: gp2
PV
8

9. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Velero and Restic:
Tools for Migrating or Cloning Workloads Across Clusters
q Requires manual setup, install on both clusters
q Requires users to setup a common S3 storage endpoint that both clusters can access
q Need to create target cluster manually
9

10. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
CloudCasa.io:
q SaaS service with light weight agent - does not
require dedicated cluster or cluster resources,
and infrastructure is hosted by SaaS service
q Network and disk throttling to prevent
overloading production environment
q Self service web UI and RBAC - let’s users only
access their workloads for cloning and migration.
q Tight integration with EKS and AKS - auto
discovers clusters and re-creates EKS/AKS
clusters on restores.
q No need for standby clusters.
Tools for Migrating or Cloning Workloads Across Clusters
10

11. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
q Migration tools
overcome isolated K8s
networking and storage
via a common storage
such as S3 object
storage
q Tools leverage API
Server to pull etcd
meta-data and use CSI
snapshots to backup
consistent PV data.
q Both clusters should
have access to S3
object storage
How Migration Tools Work – General Workflow
Production QA
PV
PV
snapsho
t
Object
storage
Migration
tool
backup
API server
Create snapshot and
read data from snapshot
Read Etcd
metadata
Write to
object
storage
PV
Migration
tool
restore
Read from
object
storage
Create PV
and write
data
API server
Apply
metadata
1
2
4
5
3
6
11

12. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Cross-Account Restore
Walkthrough
12

13. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Onboard Source and
Target AWS Accounts
via CloudFormation
Stack
13

14. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Both Source and Target
Cluster Registered
14

15. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Install CloudCasa
Agent on Both Source
and Target Clusters
via kubectl Apply
15

16. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Backup Workload
“test-website” on
Source Cluster
16

17. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Restore Workload
”test-website” on
Target Cluster and
Remap Storageclass
to gp2
17

18. Ryan Kaw
DoK Day Europe 2022 @ KubeCon
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Workload “test-
website” with
Resources and PVs
are Migrated to
Cluster2
18

19. DoK Day North America 2021 @ KubeCon
Ryan Kaw
“The Many Uses of Kubernetes Cross-Cluster
Migration of Persistent Data”
Ryan Kaw
rkaw@catalogicsoftware.com
@cloudcasa
cloudcasa.io
Thank you!
19