This talk is about the real life experiences and challenges of taking a multi-vendor NFV deployment into commercial operation. On the 9th of March NTT DOCOMO went live with a fully virtualized multi-vendor packet core solution running on an Ericsson provided cloud platform, utilizing COTS hardware. This talk will focus on the success and the areas that we see are in need of more attention from the community to achieve full interoperability across vendors. Beside, a few areas of enhancements to OPNFV to achieve a full carrier grade solution will be discussed as well, like Doctor and Promise.

1. Hyperscale Now | Ericsson Internal | © 3/287 01-FGB 101 0261 Uen, Rev B | 2016-02-22 | Page 1
Experiences from
the trench
Bringing DOCOMO Multi Vendor NFV Solution into Commercial Service
Victor Souza
Strategic Product Manager, Ericsson AB, Sweden
Keisuke Suzuki
Manager of Network Virtualization Platform Group, NTT DOCOMO, INC., Japan

2. 9th of march…

3. 9th of march…

4. Public | © Ericsson AB 2016 | 2016-05-31 | Page 4
Project objectives
CAPEX/OPEX Reduction Rapid Service Provisioning
Virtualization layer
Virtualization
management
system
General-purpose
hardware
General-purpose
hardware
Software Software
General-purpose
hardware
Service launch
Software installation,
setup
Testing
Start preparing
services
 Gain benefit of BOTH Strength and Flexibility
Improve Network ReliabilityImprove Connectivity
Virtualization layer
Hardware
Software Software
Hardware
Virtualization
management
system
Add
resources
Congested
Virtualization layer
Hardware
Software Software
Hardware
Virtualization
management
system
Instruct
change
-over
Trouble
Hardware

5. Public | © Ericsson AB 2016 | 2016-05-31 | Page 5
Project objectives
 Increase NFV/SDN benefit by multi-vendor system combination
Virtualization layer
Combination of multiple vendorsSingle vendor
Virtualization layer
Virtualization
management
system
Hardware
Hardware
(SBY)
Software
New service
Hardware
(SBY)
Virtualization layer
Virtualization
management
system
Software
Hardware
Hardware
(SBY)
Virtualization layer
Virtualization
management
system
Software
Vendor A
HardwareCongestion
Vendor B
Vendor Ｘ
Virtualization layer
Virtualization
management
system
HardwareHardware
SoftwareSoftware
Congestion or trouble
Hardware
(SBY)
1 STBY HW for all vendors
Vendor A Vendor B
1 STBY hardware for vendor B
Virtualization
management
system
HardwareHardware
SoftwareSoftware
Vendor A Vendor B
HW/VM/SW installation is needed
Only Software installation is needed
1 STBY hardware for vendor A
Vendor Ｘ
Software
(New service)
Hardware
Add new service
Add new service
Trouble
NFV solution(OPNFV) , Standardization(ETSI NFV), and
system integration are key for a multi-vendor system

6. Public | © Ericsson AB 2016 | 2016-05-31 | Page 6
The building blocks
Hardware
Monitor
S-VNFM
VNF 1
NFVO
NFVI VIM
OSS / BSS
S-VNFM
VNF 1
SDN Controller
G-VNFM

7. Public | © Ericsson AB 2016 | 2016-05-31 | Page 7
Open vendor environment
COTS server Cisco Storage
Vendor
Cisco APIC
Ericsson
Cloud
Manager
Ericsson
Cloud
Execution
Environment
Hardware
Monitor
S-VNFM
VNF 1
NFVI VIM
OSS / BSS
S-VNFM
VNF 1
SDN Controller
NEC VNF Fujitsu VNF
NFVO
G-VNFM
External DCM
systems

8. Public | © Ericsson AB 2016 | 2016-05-31 | Page 8
FUEL used for automated
installation of NFVI and VIM
SR-IOV implemented in
IceHouse using nova-flavors
Cloud auto-healing providing
automatic evacuation of VMs in
case of host failure
OpenStack Icehouse with
enhanced KVM for smart NUMA
allocation, and OVS
Vim and nfvi

9. Public | © Ericsson AB 2016 | 2016-05-31 | Page 9
Solution design
OPENSTACK COMPUTE
SERVICE
ENHANCED
VSWITCH
KVM
OPENSTACK
STORAGE
SERVICES
OPENSTACK
NETWORK
SERVICES
NFVO G-VNFM
CEILOMETER
NOVA API CINDER API GLANCE API SWIFT API
CINDER
PLUGIN
NEUTRON API
OPENSTACK
SECURITY
SERVICE
KEYSTONE API
PERFORMANCE
MANAGEMENT
FAULT
MANAGEMENT
CISCO APIC
DRIVER
FUEL based Deploy
DASHBOARD
HEAT based Orch.
CLOUD EXECUTION ENVIRONMENT
VNF
Catalogue
ERICSSON CLOUD MANAGER
NFVO G-VNFM
VNF
Catalogue
ERICSSON CLOUD MANAGER
COTS Server CiscoStorage
Vendor
Geo-redundant

10. Public | © Ericsson AB 2016 | 2016-05-31 | Page 10
Host networking
Create SR-IOV flavor
and configure it
VM provisioning
NOVA
DB
Nova
CLI/API
Nova
Compute
Nova
Conductor
Nova
Scheduler
SR-IOV Net and
port creation
Neutron
DB
NOVA
DB
Nova
Conductor
Nova
Compute
Nova
CLI/API
Neutron
CLI/API
Nova
Scheduler
VM provisioning

11. Public | © Ericsson AB 2016 | 2016-05-31 | Page 11
› We had a long discussion with each vendors to adjust interface specification.
– Need for detail standardized specification
› Amount of time spent on testing of multiple vendors integration.
– Need for test automation
Technical challenges #1
VNF VNFM
NFVI VIM
NFVO
plug-in
SDNC
OSS
Multiple vendor’s integration points
HW
No detail standardized spec.No detail standardized spec.

12. Public | © Ericsson AB 2016 | 2016-05-31 | Page 12
› SR-IOV driver version (incl. DPDK version) issue
Technical challenges #2
Guest OS
Host OS
PF driver
VF driver
NIC
There was a problem caused by incompatibility
between VF driver version and PF driver version.
This problem sometimes happened, but in other
cases, VF and PF worked correctly. It was quite
difficult to find the root cause.
Incompatibility
SR-IOV is a useful technology to improve performance, but we have to take
tight coupling between Guest OS and Host OS into account.

13. Public | © Ericsson AB 2016 | 2016-05-31 | Page 13
› OpenStack parameter tuning
–OpenStack is open and flexible platform.
–But we need careful parameter tuning at actual integration due to its flexibility.
Technical challenges #3
VNF VNFM
OpenStack
VNF verification @ VNF vendor’s test bed Integration @ NTT DOCOMO’s test bed
Host OS
Reference platform of VNF
Tuned to their OpenStack
Tuned to their VNFM
VNF VNFM
OpenStackHost OS
Unexpected behavior
Unclear VNFM’s use case

14. Public | © Ericsson AB 2016 | 2016-05-31 | Page 14
› More interoperability standards should be defined between VNF and VIM
– Many implementation details have to be agreed with 3rd party
– E.g., SR-IOV, DPDK versions, configuration parameters, …
› Validation of 3rd party VNF is time consuming
– OPNFV is strategic to create a reference implementation
– Test automation to validate the infrastructure before VNF onboarding (e..g., Yardstick)
› APIs and supported features should be discovered automatically
– Easier onboarding of applications onto a carrier-grade cloud
Key learnings

15. Public | © Ericsson AB 2016 | 2016-05-31 | Page 15
› Add quality of service related features
– Proprietary implementation of scheduling needs to be standardized
– Policing of network traffic to bring more determinism to applications
› Move onto a software defined storage solution
– Distributed solution providing a higher level of redundancy
› Develop tools and processes to allow for simpler onboarding of applications
Future steps