This presentation provides an overview of how Kubernetes capabilities can be used to simplify use of hybrid infrastructure rather than complicate it. It covers the general challenges posed by hybrid multi-site architectures, including provisioning and operations, ingress traffic management, network connectivity, and distributed data management. The presentation reviews using AWS and Azure as examples how each of these challenges can be addressed with Kubernetes and various Kubernetes controllers used as an infrastructure abstraction layer.

1. Hybrid Architecture Solutions with K8s
Oleg Chunikhin | CTO, Kublr

2. Oleg Chunikhin
CTO, Kublr
• 25 years in software architecture & development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• @olgch
Introductions

3. Automation
Ingress
Custom
Clusters
Infrastructure
Logging Monitoring
Observability
API
Usage
Reporting
RBAC IAM
Air Gap TLS
Certificate
Rotation
Audit
Storage Networking Container
Registry
CI / CD App Mgmt
Infrastructure
Container Runtime Kubernetes
OPERATIONS SECURITY &
GOVERNANCE

4. Application DevOps Automation
Developers
SRE/Ops/DevOps/
SecOps
• Self-service
• Compatible
• Conformant
• Configurable
• Open & flexible
• Governance
• Org multi-tenancy
• Single pane of glass
• Operations
• Monitoring
• Log collection
• Image management
• Identity management
• Security
• Reliability
• Performance
• Portability

5. Hybrid Architecture
● Hybrid ~ Distributed ~ Complex
● Services
○ Connectivity
○ Discovery
● Data
○ Sharding
○ Mirroring / Replication
● BCDR
○ Failure and recovery scenarios
● Architecture
○ Application
○ Middleware
○ Infrastructure
● PoC
● Hardening and Optimization
● Operations

6. Kubernetes as Container Management
K8s is a solution for:
• Uniform application management
• Uniform resource management
• In-cluster failover, load balancing, traffic management (service mesh)
Challenges:
• Heterogeneous middleware ⇒ distributed data is difficult
• Heterogeneous infrastructure ⇒ distributed load balancing and ingress is difficult
Infrastructure
K8S
Applications
Infrastructure
K8S
Applications
Managed Middleware
(e.g. RDS, EFS, ...)
Managed Middleware
(e.g. Azure SQL, FS, ...)
?

7. Kubernetes as Infrastructure and Platform
Adds:
● Homogenous middleware
○ Ceph/Rook, Portworx, Vitess, Strimzi/Kafka
● Open cross-vendor inter-cluster connectivity
○ Submariner
● Uniform BCDR
○ Velero
● Uniform (operator-based) and portable middleware management
● Flexible and portable infrastructure mapping for middleware
Infrastructure
K8S
Applications
VPN
/
WAN
Self-hosted Middleware
(e.g. Ceph/Rook, Vitess, ...)
Infrastructure
K8S
Applications
Managed Middleware
(e.g. RDS, EFS, ...)
Managed Middleware
(e.g. Azure SQL, FS, ...)
VPN
/
WAN
Self-hosted Middleware
(e.g. Ceph/Rook, Vitess, ...)
BCDR
(e.g.
Velero)
BCDR
(e.g.
Velero)
IPSec, Wireguard, ...
Mirroring, Sharding, ...

8. Demo: Stateful App in Hybrid Environment
● AWS and Azure
● Kublr for Infrastructure and K8S provisioning
● Submariner as reliable VPN
● Ceph / Rook as an example of portable storage middleware
● HA PoC: multi-zone, HA storage
● BCDR PoC: mirroring, failover
● Cost control PoC: spot instances
Infrastructure
K8S
Applications
Submariner
Ceph / Rook storage
Infrastructure
K8S
Applications
AZ, EBS Zones, Azure Disks
Submariner
Ceph / Rook storage
IPSec
Mirroring, Snapshots

9. Kubernetes Operators
● Operator Pattern
● CRD
○ Spec: component definition
○ Status: component status
● Operator
○ Links the component and CRD
● Operator in this demo
○ Submariner
○ Rook
○ ~ Kublr

10. VPN: Submariner
Worker Node
Worker Node
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Public Network
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Worker Node
Worker Node
Cluster Node
Route Agent
Gateway Engine
VxLAN Traffic
IPSEC Traffic

11. Storage: Rook / Ceph
Data pool
mon
mon
mon
config
data
raw data
osd
raw data
osd
raw data
mds
osd
Data pool
Image Image
Ceph
Filesystem
Components
Abstractions
Ceph
rgw
S3/Swift
Object Store
mgr
Rook
Operator
CSI plugins
osd
osd
ganesha
NFS
CephCluster
Block Pool
Object Store
Filesystem
NFS
Object Store User
Provisioners
rbd-mirror

12. Stack Definition
SOURCE TYPE DESCRIPTION
Infrastructure Specification Virtual Machines, Networks, Disks, etc
Cloud Formation, ARM Templates, Terraform, Kublr
Kubernetes Specification Cluster topology, masters and workers number, groups, K8S components configuration, versions
System/support Software
Specification
Kubernetes system components: e.g. overlay network, DNS, etc
(Self-)managed application services: Cloud Native Storage (Ceph/Rook), DB (Vitess),
Messaging (Strimzi/Kafka, Nats), API Management (Ambassador, Kong), etc
DevOps tools: CI (Jenkins), CD (Spinnaker), Repositories (Nexus, Artifactory) etc
Provisioning scripts Provisioning procedures and processes: shell, Makefile, Jenkinsfile, CircleCI etc

13. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
apiServerSecurePort: 6443
locations:
- name: aws1
aws:
...
master:
minNodes: 1
...
locations:
- aws:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
...
features:
monitoring: { ... }
packages:
my-package: { ... }
Cluster Specification
Kublr metadata for the cluster - name,
space, labels
Cluster-wide non provider specific
configuration - network, cluster-wide
settings, k8s version, etc
Infrastructure provider specific cluster-wide
configuration - account, access creds, AZs
etc
Infrastructure provider specific group
configuration - AZs, image, group type,
zone locking, etc
Group-specific non provider specific
configuration - k8s options, autoscaling,
etc
Kublr-specific built-in K8S components
Additional custom helm packages

14. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
clusterCIDR: 100.64.0.0/10
dnsDomain: cluster1.local
stubDomains:
- dns: cluster2.local
servers:
- 100.128.0.10
locations:
- name: aws1
aws:
resourcesCloudFormationExtras:
SgDefaultSubmariner500:
Type: AWS::EC2::SecurityGroupIngress
...
...
master:
minNodes: 1
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
pinToZone: pin
availabilityZones:
- us-east-1a
- us-east-1b
- us-east-1c
Infrastructure
Additional ports
for VPN
kind: Cluster
metadata:
name: demo-hybrid-2-azure
spec:
...
network:
clusterCIDR: 100.128.0.0/10
dnsDomain: cluster2.local
stubDomains:
- dns: cluster1.local
servers:
- 100.64.0.10
locations:
- name: azure1
azure:
virtualNetworkSubnetCidrBlock: 172.18.0.0/16
armTemplateExtras:
securityGroup:
...
...
master:
minNodes: 1
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
nodes:
- name: group1
minNodes: 3
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
zones:
- '1'
- '2'
- '3'
pinToZone: pin
Non-intersecting
CIDR
Mutual
discoverability
Mixed instance policy
and spot instances
Multi-zone

15. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
packages:
submariner-broker: { ... }
rook-ceph: { ... }
rook-ceph-additional-configuration: { ... }
rook-ceph-cluster: { ... }
Middleware
Prepare namespace for Submariner
broker
Ceph cluster definition
Rook operator
Auxiliary preconfigured CRD (e.g.
snapshot class etc)

16. Beyond the Demo
● Optimization: Resources, Throughput, Mtu, ...
● Ceph: Filesystem Mirroring, Object Store, NFS, ...
● BCDR: Velero
● Submariner: IPSec/WireGuard, GlobalNet, ...

17. References
@olgch; @kublr
github.com/kublr/hybrid-demo
rook.io/docs/rook/v1.7/
docs.ceph.com/en/pacific/
submariner.io/
docs.kublr.com/
docs.kublr.com/reference/kublr-cluster-spec/

18. Q&A

19. Oleg Chunikhin
CTO
oleg@kublr.com
@olgch
Follow Us
@kublr
Thank You