For the past 5 years, Canonical has engaged with dozens of communications service providers to design, build and operate virtualization infrastructure for network functions -- for the acronym lovers, delivering NFVI for VNFs. This presentation goes over the approach, challenges and learnings from multiple NFVI projects supporting multiple telco use cases.

1. OpenStack Telco Architecture
Christian "kiko" Reis <kiko+openstack@canonical.com>
OpenStack Summit Boston
May 2017

2. We are the company
behind Ubuntu

3. Public
Cloud
Bulk compute data center
Cloud Edge
Elastic, hybrid cloud Datacenter & IoT
Private
Cloud

4. Public cloud | Private cloud | Containers | Serverless Edge of network /IoT
Cloud Edge
Elastic hybrid cloud Transactional Edge ( IoT )

5. NFVI hardware
VNF vendors
MANO NFVO
(Orchestrators)
VNF Types SD-WAN, vCPE, vEPC, vIMS, vFW, vLB, vCDN, ..
NFVI partners

6. Remember, it's Sunday

7. About the presenter
● Responsible for WW NFVi pre-sales & implementation
DTAG, AT&T, Etisalat, Comcast, Sky, Bell.ca, Tele2, Centurylink and more
● Engineer on Ubuntu platform since 2004
● Interned in 1997 on Brazilian x86-based CO switch
● ♡ Lawrence of Arabia, Sisters of Mercy & Wasteland
● Cat 1 cyclist, rode San Francisco to Panama City
● Have 1 attributed quote
Christian "kiko" Reis, Canonical VP Field Engineering

8. Software never works the first time

9. Software never works the first time
(so make sure you're not stuck using it
when it's the first time)

10. And the audience?
● Telco
● NEP
● SI
● NFVI Distro
● NFV ISV?

11. Part I:
The Trouble with NFVI

13. First, NFV is hard
(Do I even need to make this point?)

14. First, NFV is hard
● Complex, multi-layer distributed system
● Networking: secure, multi-tenant, HA, interconnected
● Use cases are unique to telco
● But technically it's not THAT hard
● Economics and politics
● Vendor-induced confusion
● It's just computers and software

15. Why we care about Telco NFV
● Critical area for technological evolution
● OpenStack for NFVI
● Opportunity to define own technology roadmap
● Multi-vendor
● Commodity hardware
● Software defined
● Open source
● Great economics... If you can pull it off

16. Horizontal NFVI

17. Horizontal NFVI
An NFVI must be able to run multiple use cases
with VNFs from multiple vendors

18. Vertical NFVI
One NFVI, one vendor

19. Vertical NFVI
(bigger, more expensive, PNF)

20. Horizontal Vertical
Design &
Architecture
Customer-
defined
Vendor-
supplied
SLA model Multi-vendor Single-vendor
Software
Multiple VNFs,
multiple vendors
Single VNF
Hardware
Commodity
can source from any
certified vendor
Who knows?

21. Observation #1:
With Horizontal NFVI, you own the design — and the SLA
Horizontal
Design &
Architecture
Customer-
defined
SLA model Multi-vendor

22. Intermission:
Geology of VNFs

23. Yesterday:
Lift and Shift
Tomorrow:
Virtualized
The future:
Cloud Native

24. Yesterday:
Lift and Shift
Tomorrow:
Virtualized
The future:
Cloud Native
We are mostly here

25. Scale-up Partially scale-out Fully scale-out
RTOS,
homegrown Linux
Mostly
commodity Linux
Commodity
Linux
Legacy
automation
Some common
automation
Standardized
automation
Customized
hypervisor
Customized
hypervisor
Standard
hypervisor

26. Scale-up Partially scale-out Fully scale-out
RTOS,
homegrown Linux
Mostly
commodity Linux
Commodity
Linux
Legacy
automation
Some common
automation
Standardized
automation
Customized
hypervisor
Customized
hypervisor
Standard
hypervisor

27. 2 pairs of HA VMs Up to 8 VMs Fully scale-out
VxWorks Linux
Ubuntu
Linux
Virtual SD card
mounts
Vendor-provided VNF
manager
Generic VNFM
Custom hypervisor &
NIC tuning, 24 vCPU
Custom hypervisor
kernel, also tuning
Standard
hypervisor

28. Placeholder:
datasheet examples

29. Traditional vendors not
moving fast enough
Identify & work with challenger vendors
Workarounds for the Pilocene epoch
Specific VNFM Vendor struggling to support a generic VNFM in the short term.
Use a proxy charm as a wrapper around the specific VNFM
Ouch...…. Short term validate against a vendor neutral OpenStack
cloud - Ubuntu or RHEL
No testing automation
Specific VM image
hardware requirements
VM image requires SR-IOV or other accelerations. Do X

30. Flexibility Initially Virtual Machines
Few Cloudified VNFs
Able to modify and scale
as required
You want apps not VM images
vs
SLAs adherence 1:1 hardwired with “some” cloud
characteristics (SR-IOV)
Able to modify and
scale as required
There is no free lunch , there is always
virtualization overhead
Able to modify and
scale as required
VM overhead
Micro services &
network slicing
Can be only accomplished
with cloudification
Very difficult to implement, pseudo
solutions

31. Upshot:
current-era NFVI is upside-down

32. Application
Virtualization Infrastructure
Application Application

33. Infrastructure
Application
● VNFs will demand specific
infrastructure configuration
● Hypervisors will need
custom configuration,
sometimes incompatible
● Host aggregates and/or
dynamic reconfiguration
● Mostly driven by SLA,
performance guarantees

34. Real-world VNFs and Horizontal VNFI
Observation #2:
NFVI design must account for today's VNF requirements
● Pick an initial VNF
● Initial VNF will define initial NFVI architecture
● Expect your architecture to dynamically adapt & evolve
● Change-tracking the architecture is a must

35. Real-world VNFs and Horizontal VNFI
Observation #3:
VNF requirements may force your hard
● May be unable to handle new use cases
● Ensure excess NFVI capacity to avoid being boxed in
● More, bigger machines, more NICs
● May be unable to upgrade
● Careful with kernel, OS, OpenStack dependencies

36. Part I: Summary
● NFV is hard, but necessary
● It is also – at least technically – definitely achievable
● Strive for horizontal NFVI
● Avoid known gotchas
● Be tactically aware
● For instance, in a pinch, vertical NFVI can be handy

37. Part II:
OpenStack NFVI Architecture

38. High-level Architecture
Largely defined by two big decisions:
1. (Software-defined) Networking
2. Placement of storage, compute and
control plane services

39. How NFV informs OpenStack architecture
● VNFs come first
● Latency and bandwidth are key attributes
● Architecture will evolve, must be flexible
● High availability everything
● Including ability to upgrade
● Containerized control plane

40. Canonical OpenStack: Design Goals
Highly Available, Standardized Deployment
● No single point of failure in any domain
● Ensure initial deployment is always sane (and repeatable)
● Reduce gratuitous variation across installations
Architectural Flexibility
● Accommodate variety of environments and use cases
● Iterative approach to changes and additions
Zero-downtime Upgrades & Evolution
● Live upgrades and service additions/placement changes
● Never force a redeployment

41. Canonical OpenStack: Design Goals
Implies automation for 4 critical aspects:
1. Deployment
2. Routine operations
Backup, restore, hardware maintenance ("replace dead disk"),
service maintenance ("rebalance storage across nodes")
3. Architectural evolution
4. OpenStack Upgrades
Because nobody should be stuck on Icehouse

42. Compute
Fabric
Hardware Resources
KVM/QEMU
CINDER
PLUGIN
OVS
OPENSTACK
COMPUTE
SERVICES
(Nova)
OPENSTACK
STORAGE
SERVICES
(Swift, Cinder)
OPENSTACK
IMAGE
SERVICES
(Glance)
OPENSTACK
NETWORK
SERVICES
(Neutron)
Storage
Fabric
Network
Fabric
ML2
OPENSTACK
IDENTITY
SERVICES
(Keystone)
OPENSTACK
PERFORMANCE
SERVICES
(Ceilometer)
MAAS & Juju
Monitoring &
Automation
Horizon
Heat
Compute API
Block
storage
API
Object
storage
API
Image API Networking API Identity API Telemetry API
Canonical OpenStack
Orchestration Automation Juju
VNF
VNFc VNFc
VNF Enterprise IT
Canonical OpenStack: Functional Overview
LXD

43. Canonical Automation Portfolio
Provisioning
+
Dynamic
Re-purposing
Deployment
+
Integration
+
Upgrades &
architectural
evolution
System
Administration
+
Audit
+
Compliance
MAAS

44. Canonical OpenStack: Infrastructure
Role Component Details
Physical Provisioning MAAS
Provides DHCP & PXE
Automated hardware inventory & config
Service Modeling Juju
Installation, config, upgrade and management
of infra & control plane components
Systems Management Landscape Managed upgrades & compliance reporting
Log Aggregation ELK
Centralizes logs, provides structured searching
and dashboards for analysis
Monitoring & Alerting Nagios
Tracks service availability and key metrics into
SPOG with flexible, built-in alert integration
Capacity Planning Prometheus Tracking & forecasting of available capacity

45. MAAS Cluster
Controller
DHCP PXE
Server
Server
Server
Server
Infra node
Juju
Client
MAAS
API
MAAS Region
Controller
Bootstrap
Juju Controller
MAAS Cluster
Controller
DHCP PXE
Server
Server
Server
Server
Infra node
Juju
Client
MAAS
API
MAAS Region
Controller
Juju Controller
Deploy
Service
Infra services

46. Canonical OpenStack: Core Components
Role Component Details
Instance Storage
bcache-backed DAS
& Ceph options
bcache-backed local storage for high-IOPS
service with best economics; Ceph available
for use by legacy workloads
Core Block Storage Ceph with bcache
Always deployed and made available as
default Cinder backend. Additional storage
backends available as options.
Object Storage Swift
For use cases where first-class Object Storage
is required, Swift is always deployed.
Networking OVS or vendor SDN
OpenVSwitch as default SDN, with additional
SDN options for telco requirements
Hypervisor KVM & LXD
Dual hypervisor options ensure maximum
compatibility for legacy applications, and
maximum density for cloud-native

47. High-level Architecture: Networking
3 high-level options:
1. OpenVSwitch
2. Third-party SDN
3. No SDN
Trade-offs would be a workshop in itself:
● Tenant segregation, upstream architecture, performance

48. OpenVSwitch

49. Contrail

50. Nuage

51. Calico

52. High-level Architecture: Service Placement
● Co-location of services
● "Converged vs. Dedicated"
● Key trade-offs
● Density – and resulting economics
● Deterministic performance
● Inventory management

53. High-level Architecture: Service Placement
Hyperconverged
Converged
Storage+Compute
Fully Dedicated
nova-compute
ceph-osd
Control Plane
Collection #1
nova-compute
ceph-osd
nova-compute
ceph-osd
Control Plane
Collection #1 Control Plane
Collection #1

54. Part II: Summary
● Upgrades & Flexibility are tier-1 requirements
● Automating architecture is rocket science
● Deploying an OpenStack NFVI is table stakes
● OpenStack largely delivers on NFVI needs
● Third party networking & storage easily introduced
● Main decision points are SDN & placement
● Wider network architecture & goals drive SDN
● Economics vs. performance drive placement

55. For discussion
● Experiences with NFVI
● Operations
● Upgrades
● Evolution
● Alternative views on default architecture
● VM network termination & the future
● Thanks and see you at the show