This document discusses NFV infrastructure manager Lagopus, a high performance software switch developed by NTT. It begins with an agenda that covers NFV, OpenStack, performance requirements for NFV, and a comparison of Lagopus and Open vSwitch. It then provides details on Lagopus, including that it uses DPDK for high performance packet processing in userspace without kernel involvement. The document aims to demonstrate Lagopus's advantages over Open vSwitch for NFV workloads through its ability to achieve high packet processing speeds needed for telco use cases.

1. Copyright©2015 NTT corp. All Rights Reserved.
NFV Infrastructure Manager with
High Performance Software Switch
Lagopus
Hiorufmi Ichihara
NTT Software Innovation Center
Collaborator: Tomoya Hibi(NTT), Hiroki Kumazaki(NTT)

2. 2Copyright©2015 NTT corp. All Rights Reserved.
• NFV
• NFV Infrastructure Manager
• OpenStack
• Performance requirement for NFV
• OpenVSwtich and Lagopus
• Lagopus vs OVS
• Lagopus advantage
• Demo(video)
Agenda

3. 3Copyright©2015 NTT corp. All Rights Reserved.
Hirofumi Ichihara
• Engineer at NTT Software Innovation Center
• OpenStack developer
• Neutron and Devstack contributor in OpenStack
projects
Who

4. 4Copyright©2015 NTT corp. All Rights Reserved.
• NFV(Network Function Virtialization)
What is NFV?
ref: https://portal.etsi.org/nfv/nfv_̲white_̲paper.pdf
• Rapid development
• Reducing CAPEX
• Reducing OPEX
• Avoiding vendor lock
in

5. 5Copyright©2015 NTT corp. All Rights Reserved.
ETSI NFV
Today
main topic
ref: http://www.etsi.org/deliver/etsi_̲gs/nfv/001_̲099/002/01.01.01_̲60/gs_̲nfv002v010101p.pdf

6. 6Copyright©2015 NTT corp. All Rights Reserved.
Storage
• Manage infrastructure for VNF
• Compute
• Network
• Storage
• IaaS Softwares
• OpenStack
• CloudStack
• Eucalyptus
• VMware
NFV Infrastructure Manager
Hypervisor Network
Physical Switch
Virtual Network
VM VM
LRFW
VM
VM
LR
LV
FW LR Logical Router
Logical Volume
Firewall
VM Virtual Machine
LV
LV
LV

7. 7Copyright©2015 NTT corp. All Rights Reserved.
OpenStack
• Cloud Operating System
• Provides API to users
• Multi Hypervisor support
• KVM, QEMU, Xen, VMware, Hyper-‐‑‒V, LXC and others
• Integrated many network appliance and
storage appliance
Ref: http://www.openstack.org/software/

8. 8Copyright©2015 NTT corp. All Rights Reserved.
OpenStack Conceptual Architecture
Ref: http://docs.openstack.org/admin-‐‑‒guide-‐‑‒cloud/content/conceptual-‐‑‒architecture.html

9. 9Copyright©2015 NTT corp. All Rights Reserved.
OpenStack Release Schedule
2010
2011
2012
2013
2014
Major
release
Provided
functions
▲
Oct 21
Austin
▲
Feb 3
Bexar
▲
Apr 15
Cactus
▲
Sep 22
Diablo
▲
Apr 5
Essex
▲
Sep 27
Folsom
▲
Apr 4
Grizzly
▲
Oct 17
Havana
▲
Apr 1７
Icehouse
Nova
Swift
Glance
Keystone
Horizon
Cinder
Quantum
Nova
Swift
Nova
Swift
Glance Virtual machine image
Object storage
Virtual machine Nova
Swift
Glance
Keystone
Horizon
Cinder
Neutron
Ceilometer
HeatVirtual environment provisioning
Metering/Monitoring
Nova
Swift
Glance
Keystone
Horizon
Cinder
Neutron
Ceilometer
Heat
Trove
▲
Oct 16
Juno
Nova
Swift
Glance
Keystone
Horizon
Cinder
Quantum
Block storage
Database
Nova
Swift
Glance
Keystone
HorizonWeb user interface
Authentication service
Virtual network
Nova
Swift
Glance
Keystone
Horizon
Cinder
Neutron
Ceilometer
Heat
Trove
SaharaHadoop
▲
Apr 30
Kilo
Nova
Swift
Glance
Keystone
Horizon
Cinder
Neutron
・
・
・
・
・
・
• 1 release / half a year
• Apr 30, 2015 Kilo release

10. 10Copyright©2015 NTT corp. All Rights Reserved.
• Provides API enables to define Virtual
Machine
• Supporting a wide variety of virtualization
technologies, including KVM, Xen, Docker, etc
OpenStack Nova
KVM Host Docker Host
VM VM
VM VM
Container
Container
ContainerVM VM

11. 11Copyright©2015 NTT corp. All Rights Reserved.
• Provides API enables to define virtual network
• Neutron core resources
• Network: Virtual netowork （L2 Switch）
• Subnet: Manage IP address assign to network
• Port: Port connected with network
• Service plugin resources
• Router(include Floating IP)
• Load balancer
• VPN
• Firewall
OpenStack Neutron
L2 Switch
Virtual
Machine
Router
VPN
Firewall
Load balancer
Tenant A Tenant B

12. 12Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack has been designed for Data Center
use cases
NFV History in OpenStack
Hong Kong
Summit 2014
Past
Present
Telco and vendor expect to use
OpenStack to build NFV infrastructure
They realized
OpenStack NOT NFV orchestrator for CARRIER
They didnʼ’t realized
Cloud Provider Telco Carrier
What is NFV
requirement?

13. 13Copyright©2015 NTT corp. All Rights Reserved.
• Canʼ’t help failure
• User must HA
• A few network down
• User gets angry
• Guaranteed by
service contract
• Sometimes Fail
Gaps between Cloud and Telco
• Must help failure
• Provider must HA
• Donʼ’t down network
• Government gets
angry too
• Guaranteed by law
• 24/365 on Ready!
Cloud Provider Telco Carrier
What is NFV
requirement?

14. 14Copyright©2015 NTT corp. All Rights Reserved.
• Performance
• Packet processing speed with short packet
• Low latency
• High Availability
• Interface for management
• Stable
• Monitoring
• Fault detection
• Security
• Fault tolerance
NFV Telco Requirements

15. 15Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack has been designed for Data Center
use cases
NFV History in OpenStack
Hong Kong
Summit 2014
Past
Present
Telco and switch vendor expects to use
OpenStack to build NFV infrastructure
They realized
OpenStack NOT NFV orchestrator for CARRIER
They didnʼ’t realized
After Summit Telco Working Group and NFV subteam was organized
Oct. 2014 OPNFV was organized
Juno, Kilo
release Some NFV requirements was merged in OpenStack

16. 16Copyright©2015 NTT corp. All Rights Reserved.
• Proposed in OpenStack
• VLAN aware VM
• VM Scheduler
• High Availability method
• Service chaining API
• Driver for network high performance
• Liberty summit sessions related to NFV
• 41 sessions (my grep)
• OPNFV Days
OpenStack Activity for NFV

17. 17Copyright©2015 NTT corp. All Rights Reserved.
# of packet to be proceeded
for 10Gbps with 1 CPU core
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
0 256 512 768 1024 1280
# of packets per seconds
Packet size (Byte)
Short packet 64Byte
14.88 MPPS, 67.2 ns
• 2Ghz: 134 clocks
• 3Ghz: 201 clocks
Computer packet 1KByte
1.2MPPS, 835 ns
• 2Ghz: 1670 clocks
• 3Ghz: 2505 clocks

18. 18Copyright©2015 NTT corp. All Rights Reserved.
• OpenvSwitch
• Multilayer software switch
• Supports OpenFlow 1.4 protocol
• Including DPDK implement
• Source code: https://github.com/openvswitch/ovs
• Lagopus
• High performance software switch with DPDK
• Supports OpenFlow 1.3 protocol
• Source code: https://github.com/lagopus/lagopus
OpenvSwitch/Lagopus

19. 19Copyright©2015 NTT corp. All Rights Reserved.
OpenFlow
OpenFlow controller
OpenFlow switch
Data-‐‑‒plane
Flow Table
Control plane
(Routing /
swithching)
Data-‐‑‒plane
(ASIC, FPGA)
Control plane
(routing/ switching)
Flow match action
Flow match action counter
OpenFlow Protocol
Flexible flow match pattern
(Port #, VLAN ID, MAC
addr, IP addr, TCP port #)
Action (frame processing)
(output port, drop,
modify/pop/push header)
Flow statistics
(# of packet, byte size,
flow duration,… )
counter
Conventional NW node OpenFlow
Flow
Table
#2
Flow
Table
#3
OpenFlow switch agent

20. 20Copyright©2015 NTT corp. All Rights Reserved.
high-‐‑‒performance user-‐‑‒space packet
processing with Intel DPDK
NIC
sk_̲buf
Ethernet Driver API
Socket API
vswitch
packet
buffer
Dataplane
1. Interrupt
& DMA
2. system call (read)
User
space
Kernel
space
Driver
4. DMA
3. system call (write)
NIC
Ethernet Driver API
Socket API
vswitch
packet
buffer
agent
agent
1. DMA
Write
DPDK
dataplane
Userspace
packet processing
(Event-‐‑‒based)
DPDK apps
(polling-‐‑‒based)
2. DMA
READ

21. 21Copyright©2015 NTT corp. All Rights Reserved.
• Many rx interrupt causes overhead
• Polling-‐‑‒based packet receiving
• Lock with multiple thread causes bottleneck
• Thread local storage and lockless-‐‑‒queue
• High TLB miss rates causes slowing
performance
• Huge DTLB for miss of memory controller
• Context switch and memory copy causes
overhead
• Direct data copy from NIC buffer to CPU
• Kernel stack bypass of network
Issues about high performance
packet processing on x86

22. 22Copyright©2015 NTT corp. All Rights Reserved.
• OpenFlow software switch
• Run on x86 server
• High performance packet processing
• > 10Gbps
• Multiple protocol
• Conform OpenFlow1.3
• Protocol(MPLS, PBB) for WAN
• OSS
About Lagopus
OpenFlow Controller
Control Plane
OpenFlow
Protocol
OpenFlow Switch
Data Plane
Flow Table
Flow Pattern Action
Flow Pattern Action Counter
Counter
Flow
Table
#2
Flow
Table
#3
Flow
Table
#4

23. 23Copyright©2015 NTT corp. All Rights Reserved.
• OVS-‐‑‒DPDK master
• Back for DPDK 1.8.0
• commit 66cabc46ecc09eeae536277a0fc7d5e44836f845
• Lagopus v0.1.2
• DPDK 1.8.0
• System
• CPU Intel(R) Xeon(R) CPU E5-‐‑‒2697 v2 @ 2.70GHz
• Memory 64GB(fugapages: 1GBx8)
• NIC Intel X520 10GNIC
• OpenFlow Controller
• Ryu (https://github.com/osrg/ryu)
• Traffic Generator
• IXIA
Switch Performance Conditions
commit 543342a41cbceffaac30ace2c66b6e489eb359c8
Author: Mark Kavanagh <mark.b.kavanagh@intel.com>
Date: Mon Apr 20 12:37:14 2015 -‐‑‒0700
DPDK: add support for v2.0.0
Update relevant artifacts to add support for DPDK v2.0.0
-‐‑‒ INSTALL.DPDK.md
-‐‑‒ travis build script
-‐‑‒ acinclude.m4: add 'mssse3' flag to OVS_̲CFLAGS
-‐‑‒ netdev-‐‑‒dpdk: fix build with unified offload types in DPDK v2.0.0
Note that this breaks compatibility with DPDK v1.8.0

24. 24Copyright©2015 NTT corp. All Rights Reserved.
diff -‐‑‒-‐‑‒git a/config /common_̲linuxapp b /config /common_̲linuxapp
index 2f9643b..78738dc 100644
-‐‑‒-‐‑‒-‐‑‒ a/config /common_̲linuxapp
+++ b /config /common_̲linuxapp
@@ -‐‑‒81,7 +81,7 @@ CONFIG_̲RTE_̲BUILD_̲SHARED_̲LIB=n
#
# Combine to one single library
#
-‐‑‒CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=n
+CONFIG_̲RTE_̲BUILD_̲COMBINE_̲LIBS=y
CONFIG_̲RTE_̲LIBNAME="intel_̲dpdk"
#
@@ -‐‑‒160,7 +160,7 @@ CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲TX_̲FREE=n
CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲DEBUG_̲DRIVER=n
CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲PF_̲DISABLE_̲STRIP_̲CRC=n
CONFIG_̲RTE_̲LIBRTE_̲IXGBE_̲RX_̲ALLOW_̲BULK_̲ALLOC=y
-‐‑‒CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=y
+CONFIG_̲RTE_̲IXGBE_̲INC_̲VECTOR=n
CONFIG_̲RTE_̲IXGBE_̲RX_̲OLFLAGS_̲ENABLE=y
#
@@ -‐‑‒372,7 +372,7 @@ CONFIG_̲RTE_̲KNI_̲VHOST_̲DEBUG_̲TX=n
# fuse-‐‑‒devel is needed to run vhost.
# fuse-‐‑‒devel enables user space char driver development
#
-‐‑‒CONFIG_̲RTE_̲LIBRTE_̲VHOST=n
+CONFIG_̲RTE_̲LIBRTE_̲VHOST=y
CONFIG_̲RTE_̲LIBRTE_̲VHOST_̲DEBUG=n
#
DPDK config

25. 25Copyright©2015 NTT corp. All Rights Reserved.
• Full route
• Set about 51,000 flow rule (IPv4)
• packets: 1flow, random ipv4_̲dst 100000flow
• Port VLAN
• Set port VLAN to two port (2 flow rule for round trip)
• packets: 1flow, random ipv4_̲dst 10000flow
Test scenario

26. 26Copyright©2015 NTT corp. All Rights Reserved.
Evaluation -‐‑‒ full route
64
512 1024 1500
0
1
2
3
4
5
6
7
8
9
10
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
• Lagopus wins with 1flow packets
• OVS-‐‑‒DPDK wins with 100000flow packets
64
512 1024 1500
0
1
2
3
4
5
6
7
8
9
10
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
Result 1flow packets Result 100000flow packets

27. 27Copyright©2015 NTT corp. All Rights Reserved.
Evaluation – port VLAN
64
512 1024 1500
0
1
2
3
4
5
6
7
8
9
10
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
64
512 1024 1500
0
1
2
3
4
5
6
7
8
9
10
0 500 1000 1500
Gbps
byte/packet
Wire-‐‑‒rate
OVS-‐‑‒DPDK
Lagopus
• Lagopus wins with 1flow packets
• Lagopus wins with 10000flow packets
Result 1flow packets Result 10000flow packets

28. 28Copyright©2015 NTT corp. All Rights Reserved.
• Lagopus has some advantages
• Telco friendly network protocol
• QinQ, PBB, MPLS
• Comfortable more OpenFlow protocols than
OVS
• Important protocol is Meter for NFV
• OVS is used by general services but lagopus
is designed for network services
Lagopus vs OVS-‐‑‒DPDK

29. 29Copyright©2015 NTT corp. All Rights Reserved.
OpenFlow 1.3 Conformance Status
Type Action Set field Match Group Meter Total
# of test scenario
(mandatory,
optional)
56
(3 , 53)
161
(0 , 161)
714
(108 , 606)
15
(3 , 12)
30
(0 , 30)
991
(114 , 877)
Lagopus
2015.3.19
56
(3, 53)
161
(0, 161)
714
(108, 606)
15
(3, 12)
30
(0, 30)
976
(114, 862)
OVS (kernel)
2014.08.08
34
(3, 31)
96
(0, 96)
534
(108, 426)
6
(3, 3)
0
(0, 0)
670
(114, 556)
OVS (netdev)
2014.11.05
34
(3, 31)
102
(0, 102)
467
(93, 374)
8
(3, 5)
0
(0, 0)
611
(99, 556)
IVS
2015.02.11
17
(3, 14)
46
(0, 46)
323
(108, 229)
3
(0, 2)
0
(0, 0)
402
(111, 291)
ofswitch
2015.01.08
50
(3, 47)
100
(0, 100)
708
(108, 600)
15
(3, 12)
30
(0, 30)
962
(114, 848)
LINC
2015.01.29
24
(3, 21)
68
(0, 68)
428
(108, 320)
3
(3, 0)
4
(0, 4)
523
(114, 409)
Trema
2014.11.28
50
(3, 47)
159
(0 , 159)
708
(108, 600)
15
(3, 12)
34
(0, 34)
966
(114, 854)
http://osrg.github.io/ryu/certification.html

30. 30Copyright©2015 NTT corp. All Rights Reserved.
• System
• Intel(R) Core(TM) i7-‐‑‒4790 CPU @ 3.60GHz
• Memory 16GB(fugapages: 1GBx8)
• NIC Intel X520 10GNIC
• Traffic Generator
• Pktgen DPDK(master 2015.4.28)
Meter Demo
Lagopus
Data Plane
Flow Table
Flow Pattern Action
in:port1 Meter 100000kbps out:port2
Pktgen DPDK
port1 port2

31. 31Copyright©2015 NTT corp. All Rights Reserved.
• How do we setup lagopus for NFV?
• Lagopus is designed as network switch not
switch for VM on hypervisour
• Big issue
• Canʼ’t add/delete port
• OpenFlow controller essential
• Canʼ’t try easily
Using Lagopus for NFV

32. 32Copyright©2015 NTT corp. All Rights Reserved.
• Made by me + other developers
• Tomoya Hibi: Lagopus developer
• Hiroki Kumazaki: Lock free master
• What is Gondola?
• Manage KVM and Docker with Hybrid tenants and
support for various tunneling protocols(VLAN, MPLS)
• Try to run Lagopus as Hypervisor switch easily
• Exhibited in SDN/Cloud Program Contest 2014 Okinawa
• Not OSS yet
Gondola

33. 33Copyright©2015 NTT corp. All Rights Reserved.
• Using etcd for DB and
notification
• Master node
• REST/GUI
• Scheduler
• Alive monitoring
• Manage agent nodes
• Agent node
• Manage VMs and containers
Gondola Architecture
HV（KVM+Docker）
HV（KVM+Docker）
gondola
gondola
HV（KVM+Docker）
HV（KVM+Docker）
gondola
master
HV（KVM+Docker）
lagopus
Container
VM
Ryu
Libvirt
API
NW
IF
API
REST
API
VM
management
db
etcd
Docker
API
gondola
agent
DC
Network
Virtual/Phisical
port
Physical
machine
GUI User App
heart
beat
change
notification
write
Network
Gondola
SW
App.

34. 34Copyright©2015 NTT corp. All Rights Reserved.
• Demo1
• Create VM/Container and ping/ssh between them
• Isolation between multi-‐‑‒tenant networks
• Demo2
• OpenStack Integration
Demo video
Host (ubuntu02)
kvm-‐‑‒test
192.168.0.10
Host (ubuntu03)
docker-‐‑‒test
192.168.0.5
lagopus
lagopus
docker-‐‑‒test2
192.168.0.5
Host (ubuntu02)
docker-‐‑‒kvm
Host (ubuntu03)
docker-‐‑‒openstack
lagopus
lagopus
kvm-‐‑‒openstack2OpenStack
API/GUI
mistake
-> kvm-openstack

35. 35Copyright©2015 NTT corp. All Rights Reserved.
• OpenStack as NFV Infrastructure
Manager
• NFV requirements discussion
• DPDK for high performance packet
processing
• Advantage of Lagopus
Summary