SlideShare a Scribd company logo
Kubernetes Multi-Cluster
Networking
- Are we there yet ?
- Patterns, Projects & Guidelines
Sanjeev Rampal
- Red Hat
Agenda
● Problem Statement (Reminder)
● Multi-Cluster Solution Patterns
● Some relevant CNCF projects, APIs
● Recommendations
● Demo
● Q&A
About the speaker
● Senior Principal Engineer
● Emerging Technologies Group, Office of the CTO, Red Hat
● Many moons of experience building & architecting in Cloud,
Networking, AI platform segments
● Worked up and down the stack (L1 to L7) @ Cisco, Red Hat
● (more than) once stayed at a Holiday Inn Express
Scope of This Talk
● What are we talking about ?
○ Application Networking (primarily containerized applications, also VMs)
○ Spread across multiple Kubernetes clusters (and VMs)
○ Networking aspects only
○ Primarily covering CNCF project based solutions
● What are we not talking about ?
○ Other aspects of Multi-cluster architectures beyond networking
4 Types of questions we try to answer in this talk
● Patterns:
○ What are some key multi-cluster deployment design patterns ?
● Projects:
○ What are some of the relevant open source/ CNCF projects
○ How would we categorize and compare these architecturally ?
● Open source APIs evolution
○ How are the APIs of some of these key projects evolving and possibly aligning ?
● Guidelines and Recommendations
○ What are some deployment design recommendations ?
○ How to plan a multi-level strategy to adopt multi-cluster networking ?
(A few) Projects for other aspects of Multi-Cluster
● Cluster Fleet Management
○ Open Cluster Management
○ Rancher Fleet
○ Bespoke automation via Terraform, GitOps/ Argo, K8s ClusterAPI etc
○ Various Hosted & Public Cloud services
● Application/ Workload Life cycle mgmt across cluster fleet
○ Karmada
○ Open Cluster Management
○ Argo CD/ Workflows/ Rollouts
● Other Multi-cluster Operations (Observability etc)
○ Thanos + Prometheus
○ Clusterpedia
● Multi-Cluster Tenancy & CRD management, virtual clusters
○ KCP
○ vClusters
● And many more …
Use cases reminder: Why Multi-cluster ?
● Application HA & Disaster recovery, reduce unicluster blast radius
● Improved performance when serving clients across geographies
● Data gravity or compliance requirements
● Simplify infrastructure upgrades, rollouts
● Enabler for Multi-Cloud
● Re-use compute capacity from clusters across the enterprise
However
● Several cost-benefit trade offs exist
● Multiple levels of multi-cluster adoption, which do you need ?
Some Key Architecture Pattern Decisions
● North-South (multi-cluster) networking
● East-West (cross-cluster) networking
● Single/ flat Network or Multi-network
● Single Mesh or Multi-mesh
● Single trust domain or multi-domain federation
● Operational needs: Admin vs Dev centric, namespace sameness
● Others:
○ Traffic types: http. tcp, udp, other protocols gRpc, mysql, ipv4, ipv6 address pools
○ Kubernetes endpoints + VMs/ non-K8s endpoints (+ serverless)
○ Multi-cluster Network policy, service discovery
○ Degree of Zero Trust support (Crypto workload identities, encryption options)
○ Commercial and community project health
Network 1 (private) Network 2 (private)
External Network
Cluster-1
Cluster-4
Cluster-2
Cluster-3
Ingress GW
(Multi-Cluster)
E-W GW
Egress
GW
Egress
GW
E-W GW
Patterns: Multi-Cluster N-S/ GSLB ex. DNS based
Cluster-1
Cluster-2
R
B
G
ING
R
B
G
ING
GC1 GC2
Ext. DNS e.g
Rte 53
Client-1
DNS zone
Delegation for
foo.com
foo.com -> 1.1.1.1
Egr
External
statefulset/
db
GC: GSLB Ctrlr
ING: Ingress
Options for “N-S only” GLB/ Multi-cluster Ingress
Solution Architectures Example Analysis notes
External Load Balancer
Appliance
OSS: Traefik, Envoy
Commercial: F5,
HAProxy, API GWs
Appliance is data path performance bottleneck,
SPOF needs its own HA, use if already in use by
enterprise
DNS based GSLB CNCF: k8gb
Kuadrant,Gloo, Kong gw
OSS, k8s native, fine-grained per-app failover
but inconsistent failover performance, 2-clusters
Public Cloud Anycast
Service
AWS Global Accel, GCP
CLB/ Maglev, Cloudflare
Coarse grained per IP failover, expensive, scale
out data path high performance, fast failover, true
global availability
Custom BGP Anycast Custom via BGP, metallb Per IP failover, fast convergence, high perf, scale
out, depends on BGP data center design
Ref: Red Hat Blog: Global Load Balancer Approaches
N-S LB via Cloud Anycast LB service
Cloud Provider Private
Global Backbone Network
App 1
K8s cluster A
App 1
K8s cluster B
Public Cloud Region US-East-1 Public Cloud Region EU-West-1
Cloud Global Load Balancer as a Service
Edge proxy
NYC
Edge proxy
Atlanta
Edge proxy
London
200.1.2.10 200.1.2.10 200.1.2.10
DNS:
app.globaldomain.io ->
200.1.2.10
Multi-Cluster Patterns: Basic E-W Networking
(with single network & namespace sameness)
Cluster-1
Cluster-2
B1
G1
ING ING
G2
B2 B4
G3
B3
Cluster-1
Details: Pod & service discovery across clusters ? Namespace sameness ? How does Policy work ? etc
These depend on which specific project is used to implement this
ns
app-A
Patterns: E-W with Multi-Network
Cluster-1
B1
G1
ING ING
G2
B2 B4
G3
Network-1
GW-a GW-b
Network-2
Additional topics: Load balancing across gateways. Service discovery, CIDRs for pods, services,
hosts, gateways, Policy,
Pattern: Combine E-W + N-S to improve DNS failover
Cluster-1
Cluster-2
B1
G1
ING ING
G2
B2 B4
G3
B3
Cluster-1
Ext. DNS e.g
Rte 53
GC1
GC2
Client-1
Client-2
CNCF OSS projects in Multicluster networking
● Lets categorize them into 4 buckets
a. CNI+ projects
b. Application networking/ service meshes
c. Kubernetes native Multi-cluster networking API
implementations
d. User space application networking frameworks
CNI+ projects App Service
Meshes
K8s native MCS
api
implementation
Virtual Application
Networking
Representative
Projects
Cilium Clustermesh
Calico
Istio, LinkerD,
etc
Submariner Skupper
Single network Yes Yes Yes Yes
Multi network No Yes Yes Yes
K8s seamless Netpol
in multi-cluster
K8s like Cilium
NetPol+Annotations
No (Istio AuthZ) No (exp) No (separate authz)
Traffic mgmt policy L3/ L4 + some L7 L4, L7 L3 L7 (fqdn)
Zero Trust level Medium-High High Medium High
Elevated privileges
needed for some ops
Yes Yes Yes No
(Entirely in user space)
Tied to a CNI Yes No No No
Non-K8S support Yes (WIP) Yes (WIP) No Yes
Services vs Multi-Cluster Services in K8S & Istio
● Kubernetes:
○ A “Service” api instance is always scoped to a single
cluster
○ For Multi-Cluster, need explicit new resources:
Service, ServiceExport, ServiceImport
○ Also new concepts of ClusterSets and ClusterSetIps
○ Apps can’t move unchanged between single
cluster and multi-cluster
● Istio
○ Implicitly treats a Service as a Multi-Cluster Service
across all clusters in the same mesh (subject to
namespace sameness)
● Cilium
○ No new resource but needs an annotation to tag a
service as global/ multi-cluster
apiVersion:
gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example-http-route
namespace: default
spec:
parentRefs:
- name: istio-gateway
namespace: istio-system
rules:
- matches:
- path:
type: PathPrefix
value: /example
backendRefs:
- name: service-A-B
port: 80
So many APIs .. how do they relate/ align ?
K8S Gateway API
Istio APIs
Cilium APIs
“Ingress v2 api”
GEP-1748
Some Multi-cluster support
(k8s mcs api backends)
GEP-1324, 1686 etc
SMI/ GAMMA (Mesh mode)
“Extensive Istio-native mesh & gateway
APIs incl Multi-cluster”
K8s GW api
support
GW api
Gamma support
Ambient mode
K8S CNI, Service, Netpol APIs
Cilium CRDs
Multi-Cluster support
K8s APIs + annotations
K8s GW api
support
GW + mesh evolution
+ align w/ sig-mc,
Sig-net (mcs, netpol)
Evolution + GW api
& k8s alignment
Evolution + GW api
& k8s alignment
Also: K8s Service api, NetPol api, Multi-cluster service api ..
K8s Gateway API - Gateway and Mesh modes
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: example-route
spec:
parentRefs:
- name: example-gateway
hostnames:
- "example.com"
rules:
- backendRefs:
- name: example-svc
port: 80
Gateway application
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: smiley-route
namespace: faces
spec:
parentRefs:
- name: smiley
kind: Service
group: core
port: 80
rules:
- backendRefs:
- name: remote-svc
Mesh application
Sample API - Cilium Multicluster
apiVersion: v1
kind: Service
metadata:
name: blue
annotations:
service.cilium.io/global: "true"
service.cilium.io/shared: "false"
service.cilium.io/affinity: "local"
spec:
type: ClusterIP
ports:
- port: 80
selector:
name: blue
Multi-Cluster Service
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "allow-cross-cluster"
spec:
endpointSelector:
matchLabels:
name: red
io.cilium.k8s.policy.cluster: cluster1
egress:
- toEndpoints:
- matchLabels:
name: blue
io.cilium.k8s.policy.cluster: cluster2
Multi-Cluster Network Policy
Guidelines and Recommendations
● Do start with a Multi-Cluster Kubernetes design plan
● Think holistically about Networking + other DevOps aspects (App
deployment, Observability etc)
● Start simple and hold the line until you need complex
○ “No is temporary, Yes is Permanent” - Solomon Hykes, Docker creator.
○ But if you do need to go complex, don’t be shy, the tools exist
● Consider a multi-stage Multi-Cluster strategy (example on next slide)
Suggested Rules of Thumb
1. Prioritize N-S multi-cluster, you may not need E-W Multi-cluster
○ Many apps can work well via N-S Multi-cluster & namespace sameness only
○ Add E-W multi-cluster deployments only when necessary
2. If you need E-W Multi-Cluster, prioritize single network mode
○ Single/ flat network has fewer issues
○ Better for stateful workloads, policy, load balancing, performance
○ Requires large IP address pools, leverage VPCs and IPv6 if possible
3. For truly advanced and large scale or geo distributed deployments
○ Evaluate full Multi-network, multi-mesh, federated, global anycast etc
4. Evaluate the non-networking aspects at the same time
○ App orchestration, observability etc when moving to more advanced networking
5. Also match with your organizational model, your apps
○ Ex. Consider User-space options if needed
Guidelines - A possible multi-phase strategy
● Minimal inter-cluster dependency
● Basic N-S Load Balancing only
● Namespace sameness
● No E-W Multi-Cluster
● Basic N-S + Basic E-W networking & load balancing
● Single network (flat network) mode only
● Use all IPv6 addressing for pods, services if possible
● Simple Multi-mesh
● Multi-network mode if
needed
● Remote clusters if
needed
● Multi-trust-domain
● Multi-org federation
● Adv. design for high
scale
● KVStore mesh
● Global Anycast
Level 1
Level 2
Level 3
Note: These levels/ paths dont apply if using Skupper which runs in user space
and is less coupled with infra and k8s
Demo 1: Make Level-1 (N-S only) mode more robust
Cluster-1
Cluster-2
R
B
G
ING
R
B
G
ING
GC1 GC2
Ext. DNS e.g
Rte 53
Client-1
Regular DNS
GSLB doesnt
handle all error
scenarios. Lets fix
that
External state/ db
Conclusions
● Plan your phased Multi-Cluster roadmap
○ Start with simple (Level 1, maybe level 2), plan conservatively
for more complex
● Many excellent CNCF projects, OSS, vendor, SaaS
● Understand low level data plane, control plane
details across the available projects
○ There are subtle differences which impact function, performance
● So, are we there yet ? Yes (& it’s a journey, more to come)
Thank You!
Feel free to contact me for any follow ups
DM me at Twitter/ X: @sr2357
Connect/ DM on Linkedin
Longer version of this talk to be uploaded on Youtube
Cloud-Clarity @cloud-clarity4190
Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines

More Related Content

Similar to Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines

cilium-public.pdf
cilium-public.pdfcilium-public.pdf
cilium-public.pdf
Sanjeev Rampal
 
Kubernetes basics and hands on exercise
Kubernetes basics and hands on exerciseKubernetes basics and hands on exercise
Kubernetes basics and hands on exercise
Cloud Technology Experts
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAPCloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Krishna-Kumar
 
Getting started with kubernetes
Getting started with kubernetesGetting started with kubernetes
Getting started with kubernetes
Bob Killen
 
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and IstioAdvanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Animesh Singh
 
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
Ambassador Labs
 
Introduction to istio
Introduction to istioIntroduction to istio
Introduction to istio
Andrea Monacchi
 
AWS Community Day - Amy Negrette - Gateways to Gateways
AWS Community Day - Amy Negrette - Gateways to GatewaysAWS Community Day - Amy Negrette - Gateways to Gateways
AWS Community Day - Amy Negrette - Gateways to Gateways
AWS Chicago
 
Openstack days sv building highly available services using kubernetes (preso)
Openstack days sv   building highly available services using kubernetes (preso)Openstack days sv   building highly available services using kubernetes (preso)
Openstack days sv building highly available services using kubernetes (preso)
Allan Naim
 
Free GitOps Workshop
Free GitOps WorkshopFree GitOps Workshop
Free GitOps Workshop
Weaveworks
 
Evaluating Cloud Native Storage Vendors - DoK Talks #147
Evaluating Cloud Native Storage Vendors - DoK Talks #147Evaluating Cloud Native Storage Vendors - DoK Talks #147
Evaluating Cloud Native Storage Vendors - DoK Talks #147
DoKC
 
Modern VoIP in modern infrastructures
Modern VoIP in modern infrastructuresModern VoIP in modern infrastructures
Modern VoIP in modern infrastructures
Giacomo Vacca
 
.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles
VMware Tanzu
 
DCEU 18: Docker Container Networking
DCEU 18: Docker Container NetworkingDCEU 18: Docker Container Networking
DCEU 18: Docker Container Networking
Docker, Inc.
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetes
Gabriel Carro
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
NETWAYS
 
OS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of MLOS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of ML
Nordic APIs
 
Cloud Native Camel Design Patterns
Cloud Native Camel Design PatternsCloud Native Camel Design Patterns
Cloud Native Camel Design Patterns
Bilgin Ibryam
 
Openshift serverless Solution
Openshift serverless SolutionOpenshift serverless Solution
Openshift serverless Solution
Ryan ZhangCheng
 
Using an API Gateway for Microservices
Using an API Gateway for MicroservicesUsing an API Gateway for Microservices
Using an API Gateway for Microservices
NGINX, Inc.
 

Similar to Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines (20)

cilium-public.pdf
cilium-public.pdfcilium-public.pdf
cilium-public.pdf
 
Kubernetes basics and hands on exercise
Kubernetes basics and hands on exerciseKubernetes basics and hands on exercise
Kubernetes basics and hands on exercise
 
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAPCloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
 
Getting started with kubernetes
Getting started with kubernetesGetting started with kubernetes
Getting started with kubernetes
 
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and IstioAdvanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
Advanced Model Inferencing leveraging Kubeflow Serving, KNative and Istio
 
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
2017 Microservices Practitioner Virtual Summit: Microservices at Squarespace ...
 
Introduction to istio
Introduction to istioIntroduction to istio
Introduction to istio
 
AWS Community Day - Amy Negrette - Gateways to Gateways
AWS Community Day - Amy Negrette - Gateways to GatewaysAWS Community Day - Amy Negrette - Gateways to Gateways
AWS Community Day - Amy Negrette - Gateways to Gateways
 
Openstack days sv building highly available services using kubernetes (preso)
Openstack days sv   building highly available services using kubernetes (preso)Openstack days sv   building highly available services using kubernetes (preso)
Openstack days sv building highly available services using kubernetes (preso)
 
Free GitOps Workshop
Free GitOps WorkshopFree GitOps Workshop
Free GitOps Workshop
 
Evaluating Cloud Native Storage Vendors - DoK Talks #147
Evaluating Cloud Native Storage Vendors - DoK Talks #147Evaluating Cloud Native Storage Vendors - DoK Talks #147
Evaluating Cloud Native Storage Vendors - DoK Talks #147
 
Modern VoIP in modern infrastructures
Modern VoIP in modern infrastructuresModern VoIP in modern infrastructures
Modern VoIP in modern infrastructures
 
.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles.NET Cloud-Native Bootcamp- Los Angeles
.NET Cloud-Native Bootcamp- Los Angeles
 
DCEU 18: Docker Container Networking
DCEU 18: Docker Container NetworkingDCEU 18: Docker Container Networking
DCEU 18: Docker Container Networking
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetes
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
 
OS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of MLOS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of ML
 
Cloud Native Camel Design Patterns
Cloud Native Camel Design PatternsCloud Native Camel Design Patterns
Cloud Native Camel Design Patterns
 
Openshift serverless Solution
Openshift serverless SolutionOpenshift serverless Solution
Openshift serverless Solution
 
Using an API Gateway for Microservices
Using an API Gateway for MicroservicesUsing an API Gateway for Microservices
Using an API Gateway for Microservices
 

More from Sanjeev Rampal

Container security within Cisco Container Platform
Container security within Cisco Container PlatformContainer security within Cisco Container Platform
Container security within Cisco Container Platform
Sanjeev Rampal
 
Kubecon US 2019: Kubernetes Multitenancy WG Deep Dive
Kubecon US 2019: Kubernetes Multitenancy WG Deep DiveKubecon US 2019: Kubernetes Multitenancy WG Deep Dive
Kubecon US 2019: Kubernetes Multitenancy WG Deep Dive
Sanjeev Rampal
 
Kubernetes Multitenancy - KubeSec Enterprise Security Summit
Kubernetes Multitenancy - KubeSec Enterprise Security SummitKubernetes Multitenancy - KubeSec Enterprise Security Summit
Kubernetes Multitenancy - KubeSec Enterprise Security Summit
Sanjeev Rampal
 
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
Sanjeev Rampal
 
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
Sanjeev Rampal
 
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
Sanjeev Rampal
 
Openstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMsOpenstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMs
Sanjeev Rampal
 
NYC Docker Meetup: Contiv networking on Docker
NYC Docker Meetup: Contiv networking on DockerNYC Docker Meetup: Contiv networking on Docker
NYC Docker Meetup: Contiv networking on Docker
Sanjeev Rampal
 

More from Sanjeev Rampal (8)

Container security within Cisco Container Platform
Container security within Cisco Container PlatformContainer security within Cisco Container Platform
Container security within Cisco Container Platform
 
Kubecon US 2019: Kubernetes Multitenancy WG Deep Dive
Kubecon US 2019: Kubernetes Multitenancy WG Deep DiveKubecon US 2019: Kubernetes Multitenancy WG Deep Dive
Kubecon US 2019: Kubernetes Multitenancy WG Deep Dive
 
Kubernetes Multitenancy - KubeSec Enterprise Security Summit
Kubernetes Multitenancy - KubeSec Enterprise Security SummitKubernetes Multitenancy - KubeSec Enterprise Security Summit
Kubernetes Multitenancy - KubeSec Enterprise Security Summit
 
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
Architecture of Cisco Container Platform: A new Enterprise Multi-Cloud Kubern...
 
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
Cisco Live 2017: Container networking deep dive with Docker Enterprise Editio...
 
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
Triangle Kubernetes Meetup: Container cloud networking - Contiv for K8S & Ope...
 
Openstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMsOpenstack Summit: Networking and policies across Containers and VMs
Openstack Summit: Networking and policies across Containers and VMs
 
NYC Docker Meetup: Contiv networking on Docker
NYC Docker Meetup: Contiv networking on DockerNYC Docker Meetup: Contiv networking on Docker
NYC Docker Meetup: Contiv networking on Docker
 

Recently uploaded

Bengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal BrandingBengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal Branding
Tarandeep Singh
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
thezot
 
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENTUnlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
rajesh344555
 
KubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial IntelligentKubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial Intelligent
Emre Gündoğdu
 
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
Web Inspire
 
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENTUnlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
rajesh344555
 
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
APNIC
 
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
dtagbe
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
APNIC
 
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. ITNetwork Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Sarthak Sobti
 
Decentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and EsportsDecentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and Esports
Federico Ast
 
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
Febless Hernane
 
cyber crime.pptx..........................
cyber crime.pptx..........................cyber crime.pptx..........................
cyber crime.pptx..........................
GNAMBIKARAO
 
DocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptxDocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptx
AmitTuteja9
 
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call GirlsBangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
narwatsonia7
 

Recently uploaded (15)

Bengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal BrandingBengaluru Dreamin' 24 - Personal Branding
Bengaluru Dreamin' 24 - Personal Branding
 
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
一比一原版新西兰林肯大学毕业证(Lincoln毕业证书)学历如何办理
 
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENTUnlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
Unlimited Short Call Girls Navi Mumbai ✅ 9967824496 FULL CASH PAYMENT
 
KubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial IntelligentKubeCon & CloudNative Con 2024 Artificial Intelligent
KubeCon & CloudNative Con 2024 Artificial Intelligent
 
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
10 Conversion Rate Optimization (CRO) Techniques to Boost Your Website’s Perf...
 
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENTUnlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
Unlimited Short Call Girls Mumbai ✅ 9833363713 FULL CASH PAYMENT
 
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
Honeypots Unveiled: Proactive Defense Tactics for Cyber Security, Phoenix Sum...
 
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
一比一原版(uc毕业证书)加拿大卡尔加里大学毕业证如何办理
 
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...Securing BGP: Operational Strategies and Best Practices for Network Defenders...
Securing BGP: Operational Strategies and Best Practices for Network Defenders...
 
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. ITNetwork Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
Network Security and Cyber Laws (Complete Notes) for B.Tech/BCA/BSc. IT
 
Decentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and EsportsDecentralized Justice in Gaming and Esports
Decentralized Justice in Gaming and Esports
 
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
EASY TUTORIAL OF HOW TO USE CiCi AI BY: FEBLESS HERNANE
 
cyber crime.pptx..........................
cyber crime.pptx..........................cyber crime.pptx..........................
cyber crime.pptx..........................
 
DocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptxDocSplit Subsequent Implementation Activation.pptx
DocSplit Subsequent Implementation Activation.pptx
 
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call GirlsBangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
Bangalore Call Girls 9079923931 With -Cuties' Hot Call Girls
 

Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines

  • 1.
  • 2. Kubernetes Multi-Cluster Networking - Are we there yet ? - Patterns, Projects & Guidelines Sanjeev Rampal - Red Hat
  • 3. Agenda ● Problem Statement (Reminder) ● Multi-Cluster Solution Patterns ● Some relevant CNCF projects, APIs ● Recommendations ● Demo ● Q&A
  • 4. About the speaker ● Senior Principal Engineer ● Emerging Technologies Group, Office of the CTO, Red Hat ● Many moons of experience building & architecting in Cloud, Networking, AI platform segments ● Worked up and down the stack (L1 to L7) @ Cisco, Red Hat ● (more than) once stayed at a Holiday Inn Express
  • 5. Scope of This Talk ● What are we talking about ? ○ Application Networking (primarily containerized applications, also VMs) ○ Spread across multiple Kubernetes clusters (and VMs) ○ Networking aspects only ○ Primarily covering CNCF project based solutions ● What are we not talking about ? ○ Other aspects of Multi-cluster architectures beyond networking
  • 6. 4 Types of questions we try to answer in this talk ● Patterns: ○ What are some key multi-cluster deployment design patterns ? ● Projects: ○ What are some of the relevant open source/ CNCF projects ○ How would we categorize and compare these architecturally ? ● Open source APIs evolution ○ How are the APIs of some of these key projects evolving and possibly aligning ? ● Guidelines and Recommendations ○ What are some deployment design recommendations ? ○ How to plan a multi-level strategy to adopt multi-cluster networking ?
  • 7. (A few) Projects for other aspects of Multi-Cluster ● Cluster Fleet Management ○ Open Cluster Management ○ Rancher Fleet ○ Bespoke automation via Terraform, GitOps/ Argo, K8s ClusterAPI etc ○ Various Hosted & Public Cloud services ● Application/ Workload Life cycle mgmt across cluster fleet ○ Karmada ○ Open Cluster Management ○ Argo CD/ Workflows/ Rollouts ● Other Multi-cluster Operations (Observability etc) ○ Thanos + Prometheus ○ Clusterpedia ● Multi-Cluster Tenancy & CRD management, virtual clusters ○ KCP ○ vClusters ● And many more …
  • 8. Use cases reminder: Why Multi-cluster ? ● Application HA & Disaster recovery, reduce unicluster blast radius ● Improved performance when serving clients across geographies ● Data gravity or compliance requirements ● Simplify infrastructure upgrades, rollouts ● Enabler for Multi-Cloud ● Re-use compute capacity from clusters across the enterprise However ● Several cost-benefit trade offs exist ● Multiple levels of multi-cluster adoption, which do you need ?
  • 9. Some Key Architecture Pattern Decisions ● North-South (multi-cluster) networking ● East-West (cross-cluster) networking ● Single/ flat Network or Multi-network ● Single Mesh or Multi-mesh ● Single trust domain or multi-domain federation ● Operational needs: Admin vs Dev centric, namespace sameness ● Others: ○ Traffic types: http. tcp, udp, other protocols gRpc, mysql, ipv4, ipv6 address pools ○ Kubernetes endpoints + VMs/ non-K8s endpoints (+ serverless) ○ Multi-cluster Network policy, service discovery ○ Degree of Zero Trust support (Crypto workload identities, encryption options) ○ Commercial and community project health
  • 10. Network 1 (private) Network 2 (private) External Network Cluster-1 Cluster-4 Cluster-2 Cluster-3 Ingress GW (Multi-Cluster) E-W GW Egress GW Egress GW E-W GW
  • 11. Patterns: Multi-Cluster N-S/ GSLB ex. DNS based Cluster-1 Cluster-2 R B G ING R B G ING GC1 GC2 Ext. DNS e.g Rte 53 Client-1 DNS zone Delegation for foo.com foo.com -> 1.1.1.1 Egr External statefulset/ db GC: GSLB Ctrlr ING: Ingress
  • 12. Options for “N-S only” GLB/ Multi-cluster Ingress Solution Architectures Example Analysis notes External Load Balancer Appliance OSS: Traefik, Envoy Commercial: F5, HAProxy, API GWs Appliance is data path performance bottleneck, SPOF needs its own HA, use if already in use by enterprise DNS based GSLB CNCF: k8gb Kuadrant,Gloo, Kong gw OSS, k8s native, fine-grained per-app failover but inconsistent failover performance, 2-clusters Public Cloud Anycast Service AWS Global Accel, GCP CLB/ Maglev, Cloudflare Coarse grained per IP failover, expensive, scale out data path high performance, fast failover, true global availability Custom BGP Anycast Custom via BGP, metallb Per IP failover, fast convergence, high perf, scale out, depends on BGP data center design Ref: Red Hat Blog: Global Load Balancer Approaches
  • 13. N-S LB via Cloud Anycast LB service Cloud Provider Private Global Backbone Network App 1 K8s cluster A App 1 K8s cluster B Public Cloud Region US-East-1 Public Cloud Region EU-West-1 Cloud Global Load Balancer as a Service Edge proxy NYC Edge proxy Atlanta Edge proxy London 200.1.2.10 200.1.2.10 200.1.2.10 DNS: app.globaldomain.io -> 200.1.2.10
  • 14. Multi-Cluster Patterns: Basic E-W Networking (with single network & namespace sameness) Cluster-1 Cluster-2 B1 G1 ING ING G2 B2 B4 G3 B3 Cluster-1 Details: Pod & service discovery across clusters ? Namespace sameness ? How does Policy work ? etc These depend on which specific project is used to implement this ns app-A
  • 15. Patterns: E-W with Multi-Network Cluster-1 B1 G1 ING ING G2 B2 B4 G3 Network-1 GW-a GW-b Network-2 Additional topics: Load balancing across gateways. Service discovery, CIDRs for pods, services, hosts, gateways, Policy,
  • 16. Pattern: Combine E-W + N-S to improve DNS failover Cluster-1 Cluster-2 B1 G1 ING ING G2 B2 B4 G3 B3 Cluster-1 Ext. DNS e.g Rte 53 GC1 GC2 Client-1 Client-2
  • 17. CNCF OSS projects in Multicluster networking ● Lets categorize them into 4 buckets a. CNI+ projects b. Application networking/ service meshes c. Kubernetes native Multi-cluster networking API implementations d. User space application networking frameworks
  • 18. CNI+ projects App Service Meshes K8s native MCS api implementation Virtual Application Networking Representative Projects Cilium Clustermesh Calico Istio, LinkerD, etc Submariner Skupper Single network Yes Yes Yes Yes Multi network No Yes Yes Yes K8s seamless Netpol in multi-cluster K8s like Cilium NetPol+Annotations No (Istio AuthZ) No (exp) No (separate authz) Traffic mgmt policy L3/ L4 + some L7 L4, L7 L3 L7 (fqdn) Zero Trust level Medium-High High Medium High Elevated privileges needed for some ops Yes Yes Yes No (Entirely in user space) Tied to a CNI Yes No No No Non-K8S support Yes (WIP) Yes (WIP) No Yes
  • 19. Services vs Multi-Cluster Services in K8S & Istio ● Kubernetes: ○ A “Service” api instance is always scoped to a single cluster ○ For Multi-Cluster, need explicit new resources: Service, ServiceExport, ServiceImport ○ Also new concepts of ClusterSets and ClusterSetIps ○ Apps can’t move unchanged between single cluster and multi-cluster ● Istio ○ Implicitly treats a Service as a Multi-Cluster Service across all clusters in the same mesh (subject to namespace sameness) ● Cilium ○ No new resource but needs an annotation to tag a service as global/ multi-cluster apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: example-http-route namespace: default spec: parentRefs: - name: istio-gateway namespace: istio-system rules: - matches: - path: type: PathPrefix value: /example backendRefs: - name: service-A-B port: 80
  • 20. So many APIs .. how do they relate/ align ? K8S Gateway API Istio APIs Cilium APIs “Ingress v2 api” GEP-1748 Some Multi-cluster support (k8s mcs api backends) GEP-1324, 1686 etc SMI/ GAMMA (Mesh mode) “Extensive Istio-native mesh & gateway APIs incl Multi-cluster” K8s GW api support GW api Gamma support Ambient mode K8S CNI, Service, Netpol APIs Cilium CRDs Multi-Cluster support K8s APIs + annotations K8s GW api support GW + mesh evolution + align w/ sig-mc, Sig-net (mcs, netpol) Evolution + GW api & k8s alignment Evolution + GW api & k8s alignment Also: K8s Service api, NetPol api, Multi-cluster service api ..
  • 21. K8s Gateway API - Gateway and Mesh modes apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: example-route spec: parentRefs: - name: example-gateway hostnames: - "example.com" rules: - backendRefs: - name: example-svc port: 80 Gateway application apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: smiley-route namespace: faces spec: parentRefs: - name: smiley kind: Service group: core port: 80 rules: - backendRefs: - name: remote-svc Mesh application
  • 22. Sample API - Cilium Multicluster apiVersion: v1 kind: Service metadata: name: blue annotations: service.cilium.io/global: "true" service.cilium.io/shared: "false" service.cilium.io/affinity: "local" spec: type: ClusterIP ports: - port: 80 selector: name: blue Multi-Cluster Service apiVersion: "cilium.io/v2" kind: CiliumNetworkPolicy metadata: name: "allow-cross-cluster" spec: endpointSelector: matchLabels: name: red io.cilium.k8s.policy.cluster: cluster1 egress: - toEndpoints: - matchLabels: name: blue io.cilium.k8s.policy.cluster: cluster2 Multi-Cluster Network Policy
  • 23. Guidelines and Recommendations ● Do start with a Multi-Cluster Kubernetes design plan ● Think holistically about Networking + other DevOps aspects (App deployment, Observability etc) ● Start simple and hold the line until you need complex ○ “No is temporary, Yes is Permanent” - Solomon Hykes, Docker creator. ○ But if you do need to go complex, don’t be shy, the tools exist ● Consider a multi-stage Multi-Cluster strategy (example on next slide)
  • 24. Suggested Rules of Thumb 1. Prioritize N-S multi-cluster, you may not need E-W Multi-cluster ○ Many apps can work well via N-S Multi-cluster & namespace sameness only ○ Add E-W multi-cluster deployments only when necessary 2. If you need E-W Multi-Cluster, prioritize single network mode ○ Single/ flat network has fewer issues ○ Better for stateful workloads, policy, load balancing, performance ○ Requires large IP address pools, leverage VPCs and IPv6 if possible 3. For truly advanced and large scale or geo distributed deployments ○ Evaluate full Multi-network, multi-mesh, federated, global anycast etc 4. Evaluate the non-networking aspects at the same time ○ App orchestration, observability etc when moving to more advanced networking 5. Also match with your organizational model, your apps ○ Ex. Consider User-space options if needed
  • 25. Guidelines - A possible multi-phase strategy ● Minimal inter-cluster dependency ● Basic N-S Load Balancing only ● Namespace sameness ● No E-W Multi-Cluster ● Basic N-S + Basic E-W networking & load balancing ● Single network (flat network) mode only ● Use all IPv6 addressing for pods, services if possible ● Simple Multi-mesh ● Multi-network mode if needed ● Remote clusters if needed ● Multi-trust-domain ● Multi-org federation ● Adv. design for high scale ● KVStore mesh ● Global Anycast Level 1 Level 2 Level 3 Note: These levels/ paths dont apply if using Skupper which runs in user space and is less coupled with infra and k8s
  • 26. Demo 1: Make Level-1 (N-S only) mode more robust Cluster-1 Cluster-2 R B G ING R B G ING GC1 GC2 Ext. DNS e.g Rte 53 Client-1 Regular DNS GSLB doesnt handle all error scenarios. Lets fix that External state/ db
  • 27.
  • 28. Conclusions ● Plan your phased Multi-Cluster roadmap ○ Start with simple (Level 1, maybe level 2), plan conservatively for more complex ● Many excellent CNCF projects, OSS, vendor, SaaS ● Understand low level data plane, control plane details across the available projects ○ There are subtle differences which impact function, performance ● So, are we there yet ? Yes (& it’s a journey, more to come)
  • 29. Thank You! Feel free to contact me for any follow ups DM me at Twitter/ X: @sr2357 Connect/ DM on Linkedin Longer version of this talk to be uploaded on Youtube Cloud-Clarity @cloud-clarity4190