The document discusses leveraging the Kubernetes (k8s) ecosystem for edge automation and computing, highlighting a transformation towards distributed application architectures and multi-cluster orchestration solutions. It emphasizes the benefits of using k8s for managing microservices and VNFs in 5G core applications, as well as the importance of community collaborations like ONAP and Akraino. Key takeaways include the need for careful orchestration of management workloads and the potential advantages of k8s in edge computing scenarios.

1. Harnessing the Kubernetes (K8s)
Ecosystem for Edge Automation
& Computing
Ramki Krishnan – VMware (ramkik@vmware.com)
Sincere Acknowledgements to Srini Addepalli, Intel & Ravi Chunduru,
Verizon for leverage of their presentation on “Kubernetes for Edge/NFV”
at ONS Europe 2019

2. Agenda
• Transformation Journey towards all K8S
• Application Transformation from Centralized to Distributed
• Multi K8S Cluster Orchestration Solution
• Multi K8S Cluster App (L7/HTTP) Coupling
• 5G Core Application
• Community Efforts – ONAP, Akraino etc.

3. Compute nodes
VNF VNF
Non-K8S
VIM
Compute nodes
CNF MS
Kubernetes
Site
• Two different resource orchestrators
across isolated compute nodes
• K8S for CNFs and Micro Services (MS)
• Non-K8S VIM (OpenStack, VMware,
Wind River, Azure stack etc.)
Compute nodes
VNF
Non-K8SVIM
Site
K8S
VMs
CNFs
MS
• Non-K8S VIM for VNFs and VMs.
• K8S on set of VMs.
• Strict VM-based multitenancy - More
K8S clusters for different tenants
Compute nodes
VNF
Kubernetes
Site
CNF
CNF
MS
• Run all on bare-metal, one
resource orchestrator – K8S
• Strict VM-based multitenancy -
More K8S clusters for different
tenants
VMs
VMs
K8S
CNFs
MS
K8S
CNFs
MS
Transformation Journey Towards All Kubernetes (K8S)

4. Application Transformation from Central to Distributed
mS – Microservice
Key
mS4 mS4
mS3
mS2
mS1 mS1
WAN
Public/Private cloud
An App consisting of four Micro-services
ms1 talks to ms2, ms2 to ms3 and ms3 to ms4
ms1” is user facing service
“ms1”, “ms2” are expected to be there together
“ms2” is stateful and hence need to talk to each
other
To
mS2
mS1 mS1
Network (LAN/WAN)
Edge Platform
Edge 1
mS2
mS1 mS1
Edge N
W
A
N
WAN
mS4
mS3
mS4
Public/Private cloud
Figure x: Centralized computing to distributed computing
Edge Platform
Cloud platform
Cloud Platform
Key Drivers:
• Proximity
• Data
sovereignty
• Cost
• Context
Takeaway:
• Transformation Applies to Management components
(Analytics, Closed Loop etc.) besides Managed workloads

5. Multi K8S Cluster Orchestration
mS2
mS1 mS1
Edge Platform
Edge 1
mS2
mS1 mS1
Edge N
W
A
N
mS4
mS3
mS4
Public/Private cloud
Edge Platform Cloud platform
W
A
N
Multi-Cluster Management/Managed
Workload Orchestrator
Deployment Intent
An App consisting of four Micro-services
ms1 talks to ms2, ms2 to ms3 and ms3 to ms4
ms1” is user facing service and need to respond
within 20Micro-seconds
“ms1”, “ms2” are expected to be there together
“ms3”, “ms4” don’t have any latency requirements
Why MC Orchestration?
• Geo replication
• Geo Distribution
New Edges locations -> No
manual intervention
Not only for orchestrating
for apps, but also
VNFs/CNFs.
Takeaway:
• Applies to Management workloads (Analytics, Service Mesh Control Plane etc.) besides Managed workloads
• Typically Management workloads are instantiated before/removed after the Managed workloads
Multi K8S Cluster Orchestration Solution

6. Multi K8S Cluster App (L7/HTTP) Coupling
mS2
mS1 mS1
Edge Platform
Edge 1
mS2
mS1 mS1
Edge N
W
A
N
mS4
mS3
mS4
Public/Private cloud
Edge Platform Cloud platform
W
A
N
Multi-Cluster Workload
Orchestrator
MC Traffic Orchestrator for
L7/HTTP services:
User facing Geo-replicated
services:
- GSLB (K8S aware DNS
Server)
App coupling using Service
Mesh (SM) across Micro-
services (E-W traffic) of
different sites:
- Programming SM (Istio,
Linkerd etc.) egress/ingress
- Auto NAT (in cases sites
having overlapping
addresses + sites having
limited public IP addresses)
Multi-Cluster Traffic
Orchestrator
SM
GSLB
Takeaway:
• Careful consideration show be given to no. of SM instances and dependent components (e.g. Prometheus for metric
collection)
SMSM
Multi K8S Cluster App (L7/HTTP) Coupling

7. 7
5G Core Control Plane (CP)
5G Core Data
Plane (DP)
5G Core Control Plane
(CP)
• Primarily uses Service-
based Interfaces (SBA)
• SBA Highlight - HTTP 2.0
based
5G Core Architecture - Key Highlights
UE (R)AN UPF
AF
AMF SMF
PCF UDM
DNN6
NRFNEF
N3
N2 N4
AUSF
Nausf Namf Nsmf
NpcfNnrfNnef Nudm Naf
NSSF
Nnssf
N9
SCP
Takeaway
• Great opportunity for 5G Core CP to benefit from Cloud Native Service Mesh (primarily HTTP)
Ref: Adapted from 5G System Architecture - 3GPP Spec

8. 5G Core Data
Plane (DP)
5G Core CP5G Core CP
NRFUDM
Npcf
Nnrf
PCF
Nudm
UE (R)AN UPF DNN3 N6
AMF SMF
PCF
Namf Nsmf
Npcf
NRF
Nnrf
N4
AUSF
Nnef
Nausf
NEF
Nnef
NEF
VPLMN HPLMN
AF
NafNnssf
NSSF
vSEPP hSEPP
N9
Ni-nef
I-NEF
Distributed 5G Core Architecture – Key Highlights
Ref: Adapted from 5G Roaming System Architecture - 3GPP Spec
Takeaway:
• Need Multi K8S Cluster App (L7/HTTP) Coupling

9. Community Efforts – ONAP, Akraino etc.
• ONAP Edge Coordinator
• Lead: Ramki Krishnan, VMware
• ONAP4K8S project in ONAP – code development
• ICN Blueprint in Akraino – integration testing
• Looking forward to your participation!