Submariner enables direct networking between Pods and Services in different Kubernetes clusters, either on-premises or in the cloud. As Kubernetes gains adoption, teams are finding they must deploy and manage multiple clusters to facilitate features like geo-redundancy, scale, and fault isolation for their applications. With Submariner, your applications and services can span multiple cloud providers, data centers, and regions. Submariner is completely open source, and designed to be network plugin (CNI) agnostic. Submariner Provides: cross-cluster L3 connectivity using encrypted VPN tunnels; service Discovery across clusters; subctl, a friendly deployment tool; support for interconnecting clusters with overlapping CIDRs

1. Hybrid K8S Environments with Submariner
Oleg Chunikhin | CTO, Kublr

2. Oleg Chunikhin
CTO, Kublr
• 25 years in software architecture & development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• @olgch
Introductions

3. Automation
Ingress
Custom
Clusters
Infrastructure
Logging Monitoring
Observability
API
Usage
Reporting
RBAC IAM
Air Gap TLS
Certificate
Rotation
Audit
Storage Networking Container
Registry
CI / CD App Mgmt
Infrastructure
Container Runtime Kubernetes
OPERATIONS SECURITY &
GOVERNANCE

4. Service discovery
Overlay network
Physical network
Kubernetes Networking
Node 1
172.16.0.10
Node 2
172.16.0.11
Pod 1
100.64.0.10
Pod 2
100.64.0.11
my-service.my-ns.svc.cluster.local
infrastructure
kubelet, kube-proxy, CNI
K8S, coredns
Service
100.96.0.10

5. Inter-cluster connectivity
Service discovery
Overlay network
Physical network
Node 1
172.16.0.10
Node 2
172.16.0.11
Pod 1
100.64.0.10
Pod 2
100.64.0.11
Service
100.96.0.10
my-service.my-ns.svc.cluster.local
infrastructure
kubelet, kube-proxy, CNI
K8S, coredns
Service discovery
Overlay network
Physical network
Node 1
172.18.0.10
Pod 1
100.128.0.10
Service
100.160.0.10
my-service.my-ns.svc.cluster2.local
kubelet, kube-proxy, CNI
K8S, coredns
1. VPN
2. peering
3. routing
1. VPN
2. CNI
3. routing
1. DNS forwarding
2. SVC export

6. Inter-cluster connectivity
Service discovery
Overlay network
Physical network
Node 1
172.16.0.10
Node 2
172.16.0.11
Pod 1
100.64.0.10
Pod 2
100.64.0.11
Service
100.96.0.10
my-service.my-ns.svc.cluster.local
infrastructure
kubelet, kube-proxy, CNI
K8S, coredns
Service discovery
Overlay network
Physical network
Node 1
172.16.0.10
Pod 1
100.64.0.10
Service
100.96.0.10
my-service.my-ns.svc.cluster.local
kubelet, kube-proxy, CNI
K8S, coredns
Shared
my-service.my-ns.svc.clusterset.local
Service
100.160.0.10
VPN + routing

7. Submariner: VPN
Worker Node
Worker Node
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Public Network
Passive Gateway Node
Active Gateway Node
Gateway Label
Gateway Label
Worker Node
Worker Node
Cluster Node
Route Agent
Gateway Engine
VxLAN Traffic
IPSEC Traffic
Broker
Namespace

8. Submariner: service discovery

9. Submariner Components
● Broker
○ No active components; ns used to exchange connection info
● Gateway Engine (managed by operator)
○ Gateway management, failover, reconnection, LB, routing
○ gateway, route agent
● Service discovery (managed by operator)
○ Services export and import
○ lighthouse agent, lighthouse coredns

10. Submariner Responsibilities
● Broker and operator
○ Share and coordinate connection info between clusters
● Gateway Engine (managed by operator)
○ (Re-)configure routes on each node on changes in
■ peered clusters
■ active gateway
■ imported/exported services
○ Failover gateways
○ (Re-)connect
○ Load balancing
● Service discovery (managed by operator)
○ Share service import/export information
○ Serve DNS requests for clusterset.local domain

11. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
apiServerSecurePort: 6443
locations:
- name: aws1
aws:
...
master:
minNodes: 1
...
locations:
- aws:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
...
features:
monitoring: { ... }
packages:
submariner-broker: { ... }
Cluster Specification
Kublr metadata for the cluster - name,
space, labels
Cluster-wide non provider specific
configuration - network, cluster-wide
settings, k8s version, etc
Infrastructure provider specific cluster-wide
configuration - account, access creds, AZs
etc
Infrastructure provider specific group
configuration - AZs, image, group type,
zone locking, etc
Group-specific non provider specific
configuration - k8s options, autoscaling,
etc
Kublr-specific built-in K8S components
Additional custom helm packages

12. kind: Cluster
metadata:
name: demo-hybrid-1-aws
spec:
...
network:
clusterCIDR: 100.64.0.0/10
dnsDomain: cluster1.local
stubDomains:
- dns: cluster2.local
servers:
- 100.128.0.10
locations:
- name: aws1
aws:
resourcesCloudFormationExtras:
SgDefaultSubmariner500:
Type: AWS::EC2::SecurityGroupIngress
...
...
master:
minNodes: 1
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
nodes:
- name: group1
minNodes: 3
...
locations:
- aws:
groupType: asg-mip
mixedInstancesPolicyCloudFormationExtras:
...
pinToZone: pin
availabilityZones:
- us-east-1a
- us-east-1b
- us-east-1c
Infrastructure
Additional ports
for VPN
kind: Cluster
metadata:
name: demo-hybrid-2-azure
spec:
...
network:
clusterCIDR: 100.128.0.0/10
dnsDomain: cluster2.local
stubDomains:
- dns: cluster1.local
servers:
- 100.64.0.10
locations:
- name: azure1
azure:
virtualNetworkSubnetCidrBlock: 172.18.0.0/16
armTemplateExtras:
securityGroup:
...
...
master:
minNodes: 1
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
nodes:
- name: group1
minNodes: 3
...
locations:
- azure:
armTemplateExtras:
...
priority: Spot
zones:
- '1'
- '2'
- '3'
pinToZone: pin
Non-intersecting
CIDR
Mutual
discoverability
Mixed instance policy
and spot instances
Multi-zone

13. Beyond the Demo
● Optimization: Resources, Throughput, ...
● Intersecting CIDR
● Failover and load balancing
● CNI Support

14. References
@olgch; @kublr
github.com/kublr/submariner-demo
github.com/kublr/hybrid-demo
kublr.com/
docs.kublr.com/
submariner.io/getting-started/architecture/
github.com/submariner-io/
github.com/submariner-io/submariner
github.com/submariner-io/submariner-operator
github.com/submariner-io/submariner-charts
github.com/kubernetes-sigs/mcs-api
K8S multi-cluster EP

15. Q&A

16. Oleg Chunikhin
CTO
oleg@kublr.com
@olgch
Follow Us
@kublr
Thank You