Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop

0Copyright©2015 NTT corp. All Rights Reserved.
Lagopus:
SDN software switch
Yoshihiro Nakajima
NTT Labs
Feb 26th 2015

What is Lagopus (雷鳥属)?
 Lagopus is a small genus of birds in
the grouse subfamily, commonly known as ptarmigans.
All living in tundra or cold upland areas.
Reference: http://en.wikipedia.org/wiki/Lagopus

Agenda
1. Background & motivation
2. Lagopus SDN switch
I. vSwitch
II. 40GbE FPGA NIC
III.DPDK extension
IV. OSS community
3. What’s next?

Trend shift in networking
 Closed (Vender lock-in)
 Yearly dev cycle
 Waterfall dev
 Standardization
 Protocol
 Special purpose HW / appliance
 Distributed cntrl
 Custom ASIC / FPGA
 Open (lock-in free)
 Monthly dev cycle
 Agile dev
 DE fact standard
 API
 Commodity HW/ Server
 Logically centralized cntrl
 Merchant Chip

What is Software-Defined Networking?
4
Innovate services and applications
in software development speed.
Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf
Decouple control plane and data plane
Free control plane out of the box
（OpenFlow is one of APIs）
Logically centralized view
Hide and abstract complexity of networks,
provide entire view of the network
Programmability via abstraction layer
Enables flexible and rapid service/application
development
SDN conceptual model

OpenFlow vs conventional NW node
OpenFlow controller
OpenFlow switch
Data-plane
Flow Table
Control plane
(Routing /
swithching)
Data-plane
(ASIC, FPGA)
Control plane
(routing/ switching)
Flow match action
Flow match action counter
OpenFlow Protocol
Flexible flow match pattern
(Port #, VLAN ID, MAC addr,
IP addr, TCP port #)
Action (frame processing)
(output port, drop,
modify/pop/push header)
Flow statistics
(# of packet, byte size,
flow duration,… )
counter
Conventional NW node OpenFlow
Flow
Table
#2
Flow
Table
#3
OpenFlow switch agent

Why SDN?
6
Differentiate services (Innovation)
• Increase user experience
• Provide unique service which is not in the
market
Time-to-Market（Velocity）
•Not to depend on vendors’ feature roadmap
•Develop necessary feature when you need
Cost-efficiency
•Reduce OPEX by automated workflows
•Reduce CAPEX by COTS hardware

7
Evaluate the benefits of SDN
by implementing
our control plane and switch

Lagopus switch
• Lagopus overview
• Xilinx FPGA 40GbE NIC
• DPDK extension
• OSS Community

Goal of Lagopus project
 Provide NFV/SDN-aware switch software stack
 OpenFlow switch agent
 100Gbps-capable software dataplane and I/O libs
 Extensible switch configuration data store
 Expand software-based packet processing to
carrier networks
 Hardware acceleration, processing offload

# of packet to be proceeded for 10Gbps
with 1 CPU core
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
14,000,000
16,000,000
0 256 512 768 1024 1280
#ofpacketsperseconds
Packet size (Byte)
Short packet 64Byte
14.88 MPPS, 67.2 ns
• 2Ghz: 134 clocks
Computer packet 1KByte
1.2MPPS, 835 ns

# of CPU cycle for typical packet
processing (2Ghz CPU)
0 1000 2000 3000 4000 5000 6000
L2 forwarding
IP routing
L2-L4 classification
TCP termination
Simple OF processing
DPI
# of required CPU cycles
10Gbps 1Gbps

Lagopus switch

Lessons learned from existing vswitch
 Not enough performance 
 Packet processing speed < 1Gbps 
 10K flow entry add > two hours 
 Flow management processing is too heavy 
 Develop & extension difficulties 
 Many abstraction layer cause confuse for me 
• Interface abstraction, switch abstraction, protocol
abstraction Packet abstraction… 
 Many packet processing codes exists for the same
processing 
• Userspace, Kernelspace,…
 Invisible flow entries cause chaos debugging 

vSwitch requirement from user side
 Commodity x86 PC server and NIC support
 Gateway function to connect different NW
domains (MPLS, Access NW)
 10Gbps-wire rate with >= 1M flow entries
 Flow-aware control
 Multi flow table configuration
 High performance flow entry operation
 Userland-based dataplane
 Easy software upgrade and deploy
 Strong management plane supports
 Co-operable upper management systems

Approach for vSwitch development
 Simple is better for everything
 Packet processing for speed
 Protocol processing for implement
 Straight forward approach
 Full scratch (No use of existing vSwitch code)
 No OpenFlow 1.0 support
 User land packet processing as much as
possible
 Keep kernel code small
 Every component & algorithm can be replaced
 Flow lookup, library, protocol handling

Lagopus switch design
 Simple modular-based design
 Unified configuration data store
 Multiple agent support
 Support multiple data-planes
 Dataplane abstraction layer
 Multi-threaded dataplane
 Flexible parallel pipeline
• Multi-core CPU-aware design
• Intel DPDK and lock-free libraries
for I/O

high-performance user-space
packet processing with Intel DPDK
NIC
skb_buf
Ethernet Driver API
Socket API
vswitch
packet
buffer
Dataplane
1. Interrupt
& DMA
2. system call
(read)
User
space
Kernel
space
Driver
4. DMA
3. system call (write)
NIC
Ethernet Driver API
Socket API
vswitch
packet
buffer
agentagent
1. DMA
Write
2. DMA
READ
DPDK
dataplane
Userspace
packet processing
(Event-based)
DPDK apps
(polling-based)

Packet processing using multi core CPUs
 Exploit many core CPUs
 Reduce data copy & move (reference access)
 Simple packet classifier for parallel processing in I/O RX
 Decouple I/O processing and flow processing
 Improve D-cache efficiency
 Explicit thread assign to CPU core
NIC 1
RX
NIC 2
RX
I/O RX
CPU0
I/O RX
CPU1
NIC 1
TX
NIC 2
TX
I/O TX
CPU6
I/O TX
CPU7
Flow lookup
packet processing
CPU2
Flow lookup
packet processing
CPU4
Flow lookup
packet processing
CPU3
Flow lookup
packet processing
CPU5
NIC 3
RX
NIC 4
RX
NIC 3
TX
NIC 4
TX
NIC RX buffer
Ring buffer
Ring buffer NIC TX buffer

Implementation techniques for speed
 Reduce overhead
 Polling-based packet receiving
 Reduction of # of access in I/O and memory
 Huge DTLB for $ miss of memory controller
 Bypass in packet processing
 Direct data copy from NIC buffer to CPU $
 Kernel stack bypass of network
 Fast flow lookup and flow $ mechanism
 Reduce memory access and leverage locality
 Packet batching
 Novel flow lookup algorithm
 Thread local storage and lockless-queue, RCU
 Explicit CPU and thread assignment
 Hardware-topology aware resource assignment

Performance Evaluation
 Summary
 Throughput: 10Gbps wire-rate
 Flow entries: > 1M flow entires
> 4000 flow entries mods /sec
 Evaluation models
 WAN-DC gateway
• MPLS-VLAN mapping
 L2 switch
• Mac address switching

WAN-DC Gateway
0
1
2
3
4
5
6
7
8
9
10
0 200 400 600 800 1000 1200 1400 1600
Throughput(Gbps)
Packet size (byte)
10 flow rules
100 flow rules
1k flow rules
10k flow rules
100k flow rules
1M flow rules
Throughput vs packet size, 1 flow, flow-cache
0
1
2
3
4
5
6
7
8
9
10
1 10 100 1000 10000 100000 1000000
Throughput(Gbps)
flows
10k flow rules
100k flow rules
1M flow rules
Throughput vs flows, 1518 bytes packet

L2 switch performance (Mbps)
10GbE x 2 (RFC2889 test)
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
LINC OVS
(netdev)
OVS
(kernel)
Lagopus
(software)
Mbps
72
128
256
512
1024
1280
1518
Intel xeon E5-2660 (8cores, 16threads),
Intel X520-DA2, DDR3-1600 64GB
Packet size

OpenFlow 1.3 Conformance Status
Type Action Set field Match Group Meter Total
# of test scenario
(mandatory,
optional)
56
(3 , 53)
161
(0 , 161)
714
(108 , 606)
15
(3 , 12)
36
(0 , 36)
991
(114 , 877)
Lagopus
2014.11.09
56
(3, 56)
161
(0, 161)
714
(108, 606)
15
(3, 12)
34
(0, 34)
980
(114, 866)
OVS (kernel)
2014.08.08
34
(3, 31)
96
(0, 96)
534
(108, 426)
6
(3, 3)
0
(0, 0)
670
(114, 556)
OVS (netdev)
2014.11.05
34
(3, 31)
102
(0, 102)
467
(93, 374)
8
(3, 5)
0
(0, 0)
611
(99, 556)
IVS
2015.02.11
17
(3, 14)
46
(0, 46)
323
(108, 229)
3
(0, 2)
0
(0, 0)
402
(111, 291)
ofswitch
2015.01.08
50
(3, 47)
100
(0, 100)
708
(108, 600)
15
(3, 12)
30
(0, 30)
962
(114, 848)
LINC
2015.01.29
24
(3, 21)
68
(0, 68)
428
(108, 320)
3
(3, 0)
4
(0, 4)
523
(114, 409)
Trema
2014.11.28
50
(3, 47)
159
(0 , 159)
708
(108, 600)
15
(3, 12)
34
(0, 34)
966
(114, 854)
http://osrg.github.io/ryu/certification.html

40GbE FPGA NIC

Lessoned learned from development
 vSwitch needs hardware acceleration for NFV
 Focus CPU cores for upper-layer packet processing
 Encryption, Packet scheduler are CPU heavy processing
 vSwitch has several improvement points
 vSwitch cannot leverage multicore CPU power
 Software-based packet scheduler is too heavy
 vSwitch cannot protect from ingress DoS attack
Performance
Flexiblity Availability
Existing SDN
hardware switch
+ HW
accelerator
High-performance & scalability with HW
advance features with HW

Lagopus FPGA NIC
HW
SW
x86 (Xeon) + DPDK + Frontend API
VNF1@VM
QSFP+
QSFP+
40G TRX
[LOOK-UP]
40G TRX
vSwitch
VNF2@VM
PCIe Gen3 x8 PCIe Gen3 x8
VNF3@VM
Inter-DC (WAN)
Intra-DC (East-West)
[SR-IOV DEMUX]
[LABELING]
[PARSE]
[TAP-CTRL]
・・・
QoS
[TM]
FPGA
[DISPATCH]
DMA Queues x32 (duplex)
Frontend
Xilinx 40G/80G QoS NIC (XNIC)
IA server@DC
TrafficEscalation
Flow Director function for many core
Packet Mirroring function（NFV tap）

Throughput
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
64 68 256 512 1024 1512
Throughput(Gpbs)
Frame Size (Bytes)
Throughput on Each Frame Size
Xeon(R) E5-2680 v2 @ 2.80GHz, 10C/20T, DPDK-1.7.0
(1) RX1, TX1, W2
(2) RX1, TX1, W6
(3) RX1, TX1, W2
(4) RX1, TX1, W6
(5) RX2, TX1, W5
Frame Size
(Bytes)
Line Rate
40G (Gbps)
Measured Results of
Xilinx NIC and Lagopus Interwork
RX --> TX Throughput (Gbps)
Flow Director = OFF Flow Director = ON
Core assignment (1) RX1, TX1, W2 (2) RX1, TX1, W6 (3) RX1, TX1, W2 (4) RX1, TX1, W6 (5) RX2, TX1, W5
64 30.47 2.05 4.96 4.20 5.00 8.15
68 30.90 2.17 5.62 4.41 5.69 8.53
256 37.10 8.18 21.68 16.70 21.88 30.32
512 38.49 16.26 38.46 33.50 38.46 38.47
1024 39.23 32.28 39.23 39.23 39.23 39.23
1512 39.47 39.46 39.47 39.46 39.47 39.46

High performance virtual NIC
for DPDK-enabled NFV apps

Concept
 DPDK is good for performance, but not for operation and
management
 Performance acceleration for DPDK apps and legacy apps
 Simultaneous restart support of DPDK apps and DPDK
vswitch
 Isolation for security and trust for guest VM and vswitch
 Trusted guest VM: zero-copy
 Non-trusted guest VM: one-copy
Guest1
QEMU
App
DPDK
Guest2
QEMU
App
DPDK
Virtio-net PMD Virtio-net PMD
Lagopus
vswitch
DPDK
PMD
Map memory in guest VM
to lagopus memory
Map memory in guest VM
to lagopus memory
virtio virtio
virtio
queue
virtio
queue
PMD
vNIC
PMD
vNIC

DPDK VNIC extension (virtq-pmd)
 4.8 times faster than existing vhost pmd
 Support of simultaneous restart of VM gust and
host DPDK apps
0
500
1000
1500
2000
2500
3000
3500
0 500 1000 1500 2000
Required#ofCPUClocks
Packet size (bytes)
Transmission overhead per
packet
copy overhead hole PMD
virtq PMD vhost app
Kernel driver
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
0 500 1000 1500 2000
Gbps
COPY OFF COPY ON
Released as OSS on github

Expand SDN OSS community

Activity
 Hands-on in Tokyo & Taiwan
 Total over 400 engineers enjoyed
 United states?
Tokyo hands-on Taiwan hands-on

PoC in SDN Japan 2014
 Location-aware
bandwidth-controlled
Internet WiFi with Ryu
and Lagopus
 Good audience can
connect to the Internet
with guaranteed BW
 Front area sheet, good
connection.
 Back area sheet, poor
connection

What’s next?

M-plane needs innovation!
 No sufficient abstraction in M-plane
 Proprietary CLI, script
 Invisibility leads to inefficiency
 Still legacy (SNMP is neither SIMPLE nor realtime)
plan
Provision
& control
monitor
Analyze
&
visualize
Plan NW based on
NW abstraction model
Provisioning on
Feedback to
Think NW based-on

Y2013 Y2014 Y2015
Control plane
Management
plane
Data plane
Lagopus development
OF1.3 agent
High-performance
software dataplane
with DPDK
(static pipeline
configuration)
Switch config/mngmt
Db prototype
OF-CONFIG,
CLI, SNMP
prototypes
Flexible reconfigurable
pipeline in dataplane
Tunnel protocol
(VxLAN, GRE,
MPLS pseudo-wire)
40Gbps-capable
Soft dataplane
Extensible switch resource
Management/configuration
datastore
(CLI, OF-CONFIG, SNMP)
OF1.5 agent
Legacy protocol ctrl support
M-plane extension
(Openconfig)

Thanks!
Web: http://lagopus.github.io/
Twitter: lagopusvswitch

Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop

Similar to Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop (20)

Recently uploaded

Recently uploaded (20)

Lagopus presentation on 14th Annual ON*VECTOR International Photonics Workshop