Use case of Kubernetes based NFV infrastructure used in production to run an open source evolved packet core. Presented by Facebook Connectivity and Mirantis at KubeCon + CloudNativeCon Europe 2020.

1. Using Kubernetes to make
cellular data plans cheaper for
50M users
July 28th, 2020

2. 2
Introductions
Amar Padmanabhan
Lead Developer, Magma Project
Facebook Connectivity
Chandra Dodda
Sr. DevOps Engineer
Mirantis

3. The problem of bringing the next 50
million onto a faster internet is a
problem of heterogeneity
Heterogeneity in access, backhaul, scale and business models

4. 1. Edge vs Fabric decomposition of the network
2. State based orchestration of edges
4
Design Principles

5. Edge vs Fabric
Modularization of the network

6. ● Lots of in-network processing appliances:
IDS, Firewall, Proxies, Load balancers
● Chokepoint devices
○ Force a topology
○ Expensive: Fast pipes + rich policies
6
Traditional Datacenter: Hierarchical Networks

7. Modularize the network: Fabric
responsible for moving packets faster.
Distributed edge responsible for rich
policy enforcement
Modern datacenters: Fabric and policy rich edge

8. 8
Notes on edge services
● Fundamentally distributed
○ Good candidate for x86: Does complex things well but slowly.
● Software only policy enforcement allows for rapid iteration.
● Leverage programmable interfaces like openflow, e-BPF etc.
● Distributed services are harder to manage
○ Operationalizing the solution is key

9. X X
XX
SGW/PGW are chokepoint
devices
Today’s GSM/LTE architecture

10. ● Distribute policy enforcement point
○ Let the ideal topology decide the policy enforcement point
● Move policy enforcement to software
○ Leverage rapid iteration and programmability of software
● Keep core network simple
○ Allows for easy scale up/down
○ Cheap: Core network only needs to move packets fast
● Focus on operationalizing the distributed network
○ What used to be a single central node is now distributed services across multiple
edge locations
10
Why Magma?
Summary: Modularize the cellular network

11. 4G LTE / 5G / Wi-Fi
INTERNETACCESS NETWORK MAGMA CONVERGED CORE
REST APIs
S1
Cell Site Private
Cloud
*Core Deployment Options
Converged
Core*
Radio
Backhaul
SGi (User IP traffic)
Public
Cloud
Orchestrator
& NMS
Magma Converged Core
(OTS H/W)
Federation
Gateway
CPE
HTTP2
HTTP2

12. The core problem we built Orchestrator to tackle:
Managing a fleet of heterogeneous,
edge-deployed service meshes for core
convergence.

13. ● Declaratively configure the edge fleet from
centralized management
‒ Push state, not procedures
‒ Declare the desired topology for your fleet and the
service meshes running on each device
● Delegate procedural control to the edge device
‒ The device always has the most up-to-date view of its
operational state, so it’s the best place to perform
procedural reconciliation of declared configuration and
actual state
Anchoring Principles
13
Declare new
configuration
Synchronize
updates
Monitor
aggregated
edge state
Send new
operating state
Reconcile local and
desired states
Edge
Cloud

14. ● 1 instance per managed device
● Applies configuration updates to the local
service mesh based on updates from
management
● Ships local state and operational history to
management
‒ Service states, performance metrics, application
timeseries, events, logs, etc.
● In Magma LTE, this is a custom solution built
on top of systemd, apt, and custom Python
code
‒ Plans are in place to migrate to a k8s-based
control plane at the edge
Edge Control Plane
Orchestrator Components
14
magma cl
updates state,
timeseries
fluent
logs,
events
control state
update
local
config
store
local
state
store
Control: systemd / logs
Config store: disk / Config Map
local
service
mesh
Redis
1
3
2

15. ● Singleton, multi-tenant cloud application built on
k8s
● Aggregates state and history from edge fleet and
sends configuration updates to fleet (GRPC)
● Exposes endpoints to query and set edge fleet
configuration and query edge fleet state (REST)
● Batteries included:
‒ Multi-tenant wrappers around Grafana and
Elasticsearch to expose timeseries dashboards, events,
and logs
‒ ECDSA-based trust negotiation for authenticating edge
devices
‒ REST authorization framework
‒ Integration points for external timeseries and event
datasinks
Central Management Plane
Orchestrator Components
15
REST
Core Orchestrator
SQL
Prom
ES
GRPC
Domain
Plugins
Edge Fleet:
● OCN 5G
● Magma LTE
● Etc.
stateconfig
Field
Cloud

16. 16
Edge Infrastructure Stack
Mirantis Cloud Platform

17. Body Level One
● Body Level Two
‒ Body Level Three
‒ Body Level Four
BODY LEVEL FIVE
Optional subtitle (delete if not used)
Title Text
17
MCP Overview
Lifecycle Management (LCM)
DriveTrain
Version Control
Artifact Store
Code Review
Node Classification
CI/CD
Orchestration
Compute Storage Network
Operations Support
System (OSS)
StackLight
Web NFV AnalyticsMediaIoT
Deployment
Kubernetes
Calico SDN
OpenStack
Tungsten Fabric
Ceph
Flexible Infrastructure
containers Bare Metal VMs
Notifications
Logging
Tungsten Fabric Neutron + OvS
Monitoring
Alerting
Analytics
Trending/Capacity
Enterprise Integrations (e.g. Security & Single Sign-on)

18. 18
MCP Edge Architecture
● Minimum footprint: 4 nodes
○ 3 control plane nodes that run
DriveTrain, StackLight, and K8s
Masters in HA
○ 1 or more worker nodes that run
workload
● Coexistence and seamless networking
with container pods and VM pods
● Virtlet: use to implement VM pods in
k8s environment
○ Open source project

19. 19
Virtlet

20. 20
Kubernetes CRI implementation for running VM workloads
● Targeted at VM workloads that need to behave as containers on
the outside
● Run unmodified VM images using qcow2 format
● Build higher-level Kubernetes objects using VM pods
● Use familiar kubectl pod commands to work with your VMs
● Integrate with cluster networking using normal CNI plugins
● Easy to deploy - only need to install simple CRI Proxy package on
the nodes
What is Virtlet?

21. 21
● Virtlet enables you to run
unmodified QEMU/KVM
virtual machines that do not
include an additional Docker
layer as in similar solutions in
Kubernetes.
● Virtlet supports all standard
Kubernetes objects, such as
ReplicaSets, Deployments,
DaemonSets, and so on, as
well as their operations.
● This diagram describes the
Virtlet components and
interactions between them.
What Does Virtlet Enable?

22. 22
Magma Integration for
MCP Edge

23. 23
Optimizing Carrier-Grade Wi-Fi Offload with Magma
Provide operators with a sustainable and efficient way to
address consumer data demands through Wi-Fi offload
Seamless User
Experience
● Flexible
distributed Core
integration
● Extend user reach
through FB app
footprint
● Steer users to the
best connection
Optimized
Business Platform
● Identify areas best
suited for mobile data
offload
● Analytics and
management
capabilities for
large-scale Wi-Fi
networks and
hotspots
Wi-Fi
Ecosystem
● Vendor Integrations
● Offload ecosystem
leveraging additional
bandwidth and
capacity for hungry
applications

24. 24
Elevate subscriber’s quality of experience on Wi-Fi to match that on mobile data
Mobile Core Integration

25. 25
MCP Edge Architecture
Magma Worker 1
Control Plane Node 1 Control Plane Node 3Control Plane Node 2
DriveTrain
StackLight
K8s
Master
DriveTrain
StackLight
K8s
Master
DriveTrain
StackLight
K8s
Master
Docker
Pod
Orchestrator
Virtlet Pod
containerd
} MCP Edge
Access Gateway
VM
Pod
VM
Pod
Fed. Gateway

26. 26
Covered in this demo :
1. Deploy MySQL pods in HA
2. Deploy Orchestrator in HA
3. Deploy Federation Gateway ( As VM using Virtlet )
4. Deploy Wi-Fi Access Gateway ( As VM using Virtlet )
5. Connect to hotspot and access Internet from mobile
phone
Carrier Wi-Fi Demo
Not covered and out of scope:
1. Deploy and configure Wifi AP
2. Configure user in HSS/PCR and OCS

27. 27
MCP Edge
Private Cloud
Magma Worker Node
User
Gateway
Wi-Fi
Node 3
mysql03
containerd
Carrier Wi-Fi
Access Gateway
Virtlet
CNICNI CNI
Orchestrator Service
Wireless
Controller
Policy Control and
Charging Rules Function
Online Charging System
Home Subscriber
Server
Node 2
mysql02
containerd
Federation
Gateway
Virtlet
CNI CNI
SCTP
EoGRE
DHCP, DNS, NAT
User
MySQL Service
Container VM Pod
Node 1
mysql01
containerd
Orchestrator
Ctl + Proxy
Network
Monitoring

28. ● Flexibility: Modularize the network into a fast fabric and a policy rich edge
to maximize deployment and business models
● Hierarchical orchestration: Distributed edge services require hierarchical
orchestration
● Consistent runtime: k8s based edge clusters offer a consistent runtime
for edge services
● Get involved! https://github.com/magma
28
Summarizing: Solving for heterogeneity

29. ● magma-dev@googlegroups.com : For more information on Magma
● magma-announce@googlegroups.com : Join our mailing list to receive
updates and announcements
● https://connectivity.fb.com/magma : For a broader perspective on how
Magma fits into Facebook’s other connectivity programs
For more information

30. 30
Kontena Lens: Kubernetes IDE simplifies
Kubernetes for application developers
●View demo at Mirantis booth
●Download from: k8slens.dev
Kontena Lens

31. Thank You