SlideShare a Scribd company logo
1 of 35
Download to read offline
0 
Copyright©2014 NTT corp. All Rights Reserved. 
High-performance vSwitch of the user, by the user, for the user 
A bird in cloud 
Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano 
NTT Labs
1 
Copyright©2014 NTT corp. All Rights Reserved. 
Agenda 
vSwitch of the user 
Background and motivation 
vSwitch by the user 
High-performance software-switch 
•Concept 
•Design 
•Implementation & techniques 
•Performance Evaluation 
vSwitch for the user 
PoC of carrier usecase 
•Segment Routing or Service chaining (Tentative) 
Open source community
2 
Copyright©2014 NTT corp. All Rights Reserved. 
Our mission 
Open innovation with open source 
for networking
3 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch of the user 
•Background 
•Motivation
4 
Copyright©2014 NTT corp. All Rights Reserved. 
Trend shift in networking 
Closed (Vender lock-in) 
Yearly dev cycle 
Waterfall dev 
Standardization 
Protocol 
Special purpose HW / appliance 
Distributed cntrl 
Custom ASIC / FPGA 
Open (lock-in free) 
Monthly dev cycle 
Agile dev 
DE fact standard 
API 
Commodity HW/ Server 
Logically centralized cntrl 
Merchant Chip
5 
Copyright©2014 NTT corp. All Rights Reserved. 
What is Software-Defined Networking? 
5 
Innovate services and applications in software development speed. 
Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf 
Decouple control plane and data plane →Free control plane out of the box (OpenFlow is one of APIs) 
Logically centralized view →Hide and abstract complexity of networks, provide entire view of the network 
Programmability via abstraction layer 
→Enables flexible and rapid service/application development 
SDN conceptual model
6 
Copyright©2014 NTT corp. All Rights Reserved. 
OpenFlow vs conventional NW node 
OpenFlow controller 
OpenFlow switch 
Data-plane 
Flow Table 
Control plane (Routing / swithching) 
Data-plane 
(ASIC, FPGA) 
Control plane (routing/ switching) 
Flow match 
action 
Flow match 
action 
counter 
OpenFlow Protocol 
Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) 
Action (frame processing) 
(output port, drop, modify/pop/push header) 
Flow statistics (# of packet, byte size, flow duration,… ) 
counter 
Conventional NW node 
OpenFlow 
Flow Table 
#2 
Flow Table 
#3
7 
Copyright©2014 NTT corp. All Rights Reserved. 
7 
Evaluate the benefits of SDN by implementing our control plane and switch
8 
Copyright©2014 NTT corp. All Rights Reserved. 
Motivation of vSwitch 
Agile and flexible networking 
Full automation in provisioning, operation and management 
Seamless networking for customers 
Server virtualization and NFV needs a high-performance software switch 
Short latency 
Wire-rate in case of short packet (64B) 
No High-performance OpenFlow 1.3 software switch for wide-area networks 
1M flow rules 
10Gbps-wire-rate 
Multiple table supports
9 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch requirement from user side 
1.Run on the commodity PC server and NIC 
2.Provide a gateway function to allow connect different various network domains 
Support of packet frame type in DC, IP-VPN, MPLS and access NW 
3.Achieve 10Gbps-wire rate with >= 1M flow rules 
low-latency packet processing 
flexible flow lookup using multiple-tables 
High performance flow rule setup/delete 
4.Run in userland and decrease tight-dependency to OS kernel 
easy software upgrade and deployment 
5.Support various management and configuration protocols.
10 
Copyright©2014 NTT corp. All Rights Reserved. 
Why Intel x86 server for networking? 
Strong processing power 
Many CPU cores 
•8 cores/CPU (2012), 12 cores/CPU (2013),… 
Rapid development cycle 
Available everywhere 
Can be purchased in Akihabara! 
3 month lead time in case of special purpose HW 
Reasonable price and flexible configuration 
8 core CPU (ATOM) with 1,000 USD 
Can chose configuration according to your budget 
Many programmers and engineers 
Porting to a NPU needs special skill and knowledge
11 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch by the user 
•Concept 
•Design 
•Implementation & techniques 
•Performance Evaluation
12 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch by the user Concept
13 
Copyright©2014 NTT corp. All Rights Reserved. 
Target of Lagopus vSwitch 
High performance soft-based OpenFlow switch 
10Gbps wire-rate packet processing / port 
1M flow rules 
Expands the SDN apps to wide-area networks 
Not only for data centers 
WAN protocols, e.g. MPLS and PBB 
Various management /configuration interfaces 
Open Innovation 
Community-based development
14 
Copyright©2014 NTT corp. All Rights Reserved. 
Approach for vSwitch development 
Simple is better for everything 
Packet processing 
Protocol handling 
Straight forward approach 
No OpenFlow 1.0 support 
Full scratch (No use of existing vSwitch code) 
User land packet processing as much as possible, keep kernel code small 
Kernel module update is hard for operation 
Every component & algorithm can be replaced 
Flow lookup, library, protocol handling
15 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch design 
Simple modular-based design 
Switch control agent 
Data-plane 
Switch control agent 
Unified configuration data store 
Multiple management IF 
Data-plane control with HAL 
Data-plane 
High performance I/O library (Intel DPDK) 
Raw socket for test
16 
Copyright©2014 NTT corp. All Rights Reserved. 
Implementation strategy for vSwitch 
Massive RX interrupts handling for NIC device 
=> Polling-based packet receiving 
Heavy overhead of task switch 
=> Thread assignment (one thread/one physical CPU) 
Lower performance of PCI-Express I/O and memory bandwidth compared with CPU 
=> Reduction of # of access in I/O and memory 
Shared data access is bottleneck between threads 
=> Lockless-queue, RCU, batch processing
17 
Copyright©2014 NTT corp. All Rights Reserved. 
Processing bypass for speed 
NIC 
skb_buf 
Ethernet Driver API 
Socket API 
vswitch 
packet buffer 
packet buffer memory 
Standard linux application 
1. Interrupt & DMA 
2. system call (read) 
User space 
Kernel space 
Driver 
4. DMA 
3. system call (write) 
NIC 
Ethernet Driver API 
User-mode I/O & HAL 
vswitch 
packet buffer 
Application with intel DPDK 
1. DMA Write 
2. DMA READ 
DPDK Library 
Polling-base 
packet handling 
Event-base packet handling
18 
Copyright©2014 NTT corp. All Rights Reserved. 
Packet processing using multi core CPUs 
Exploit many core CPUs 
Reduce data copy & move (reference access) 
Simple packet classifier for parallel processing in I/O RX 
Decouple I/O processing and flow processing 
Improve D-cache efficiency 
Explicit thread assign to CPU core 
NIC 1 
RX 
NIC 2 RX 
I/O RX 
CPU0 
I/O RX CPU1 
NIC 1 
TX 
NIC 2 TX 
I/O TX 
CPU6 
I/O TX 
CPU7 
Flow lookup packet processing 
CPU2 
Flow lookup packet processing 
CPU4 
Flow lookup packet processing 
CPU3 
Flow lookup packet processing 
CPU5 
NIC 3 
RX 
NIC 4 RX 
NIC 3 
TX 
NIC 4 TX 
NIC RX buffer 
Ring buffer 
Ring buffer 
NIC TX buffer
19 
Copyright©2014 NTT corp. All Rights Reserved. 
Packet batching 
Reduce # of lock in flow lookup table 
Frequent locks are required 
•Switch OpenFlow agent and counter retrieve for SNMP 
•Packet processing 
Input packet 
Lookup table 
Input packet 
Lookup table 
Lock 
Unlock 
ロック 
Unlock 
●Naïve implementation 
●Packet batching implementation 
Lock is required for each packet 
Lock can be reduced due to packet batching
20 
Copyright©2014 NTT corp. All Rights Reserved. 
Bypass pipeline with flow $ 
20 
●Naïve implementation 
table1 
table2 
table3 
Input packet 
table1 
table2 
table3 
Input packet 
Flow cache 
1. New flow 
2. Success flow 
Output packet 
Multiple flow $ generation & mngmt 
Write flow $ 
●Lagopus implementation 
Output packet 
Packet 
Reduce # of flow lookup in multiple table 
Exploit flow $ for
21 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch by the user Performance Evaluation
22 
Copyright©2014 NTT corp. All Rights Reserved. 
Performance Evaluation 
Summary 
Throughput: 10Gbps wire-rate 
Flow rules: 1M flow rules 
Evaluation models 
WAN-DC gateway 
•MPLS-VLAN mapping 
L2 switch 
•Mac address switching
23 
Copyright©2014 NTT corp. All Rights Reserved. 
Performance Evaluation 
Evaluation setup 
Server spec. 
CPU: Dual Intel Xeon E5-2660 
•8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge 
Memory: DDR3-1600 ECC 64GB 
•Quad-channel 8x8GB 
Chipset: Intel C602 
NIC: Intel Ethernet Converged Network Adapter X520-DA2 
•Intel 82599ES, PCIe v2.0 
Server 
Lagopus 
Flow table 
tester 
Flows 
Throughput (bps/pps/%) 
Flow 
rules 
Packet size 
Flow cache (on/off)
24 Copyright©2014 NTT corp. All Rights Reserved. 
WAN-DC Gateway 
0	 
1	 
2	 
3	 
4	 
5	 
6	 
7	 
8	 
9	 
10	 
0	 200	 400	 600	 800	 1000	 1200	 1400	 1600	 
Throughput	(Gbps)	
 
Packet	size	(byte)	
 
10	flow	rules	 
100	flow	rules	 
1k	flow	rules	 
10k	flow	rules	 
100k	flow	rules	 
1M	flow	rules	 
Throughput vs packet size, 1 flow, flow-cache 
0	 
1	 
2	 
3	 
4	 
5	 
6	 
7	 
8	 
9	 
10	 
1	 10	 100	 1000	 10000	 100000	 1000000	 
Throughput	(Gbps)	
 
flows	
 
10k	flow	rules	 
100k	flow	rules	 
1M	flow	rules	 
Throughput vs flows, 1518 bytes packet
25 
Copyright©2014 NTT corp. All Rights Reserved. 
L2 switch performance (Mbps) 10GbE x 2 (RFC2889 test) 
0 
1000 
2000 
3000 
4000 
5000 
6000 
7000 
8000 
9000 
10000 
LINC 
OVS 
(netdev) 
OVS 
(kernel) 
Lagopus 
(software) 
Mbps 
72 
128 
256 
512 
1024 
1280 
1518 
Intel xeon E5-2660 (8cores, 16threads), Intel X520-DA2, DDR3-1600 64GB 
Packet size
26 
Copyright©2014 NTT corp. All Rights Reserved. 
vSwitch for the user 
PoC of carrier usecase 
Open source community
27 
Copyright©2014 NTT corp. All Rights Reserved. 
SPRING 
alternative way of IP forwarding 
use “labels” to forward packets 
much simpler architecture than MPLS network 
can utilize traditional MPLS routers 
good chemistry with NFV 
For what ? 
policy-based routing 
•explicit routing instead of shortest path routing 
service-chaining (NFV) 
•apply services to specific traffic pattern
28 Copyright©2014 NTT corp. All Rights Reserved. 
Career Network with SPRING 
Source 
Destination 
NFV server 
 use lagopus as NFV server with career grade 
routers 
 all nodes include NFV server are labeled
29 Copyright©2014 NTT corp. All Rights Reserved. 
Career Network with SPRING 
Source 
Destination 
NFV server 
 at the ingress router, label stack is pushed 
 label stack represents the policy 
IP
30 
Copyright©2014 NTT corp. All Rights Reserved. 
PoC Implementation
31 
Copyright©2014 NTT corp. All Rights Reserved. 
Demo
32 
Copyright©2014 NTT corp. All Rights Reserved. 
Open Source Activity for the user 
Lagopus goes wild since this summer 
Your contribution are very welcome! 
http://lagopus.github.io/ 
Twitter: lagopusvswitch 
Ryu: component-based SDN framework 
OpenFlow 1.3, 1.4, BGP, BMP, etc.. 
http://osrg.github.io/ryu/
33 
Copyright©2014 NTT corp. All Rights Reserved. 
Summary 
We contribute vSwitch for user 
Improve performance and scalability 
Enhance the functionality and capability 
DPDK is great library for user 
Easy to develop & implement 
Easy to deploy like UNIX apps
34 
Copyright©2014 NTT corp. All Rights Reserved. 
Thank you! 
This research is a part of the project for “Research and Development of Network Virtualization Technology” supported by the Ministry of Internal Affairs and Communications.

More Related Content

What's hot

What's hot (20)

DPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet ProcessingDPDK & Layer 4 Packet Processing
DPDK & Layer 4 Packet Processing
 
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun  to do the impossible with DPDKDPDK summit 2015: It's kind of fun  to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
 
Intel DPDK Step by Step instructions
Intel DPDK Step by Step instructionsIntel DPDK Step by Step instructions
Intel DPDK Step by Step instructions
 
Intel dpdk Tutorial
Intel dpdk TutorialIntel dpdk Tutorial
Intel dpdk Tutorial
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles Shiflett
 
DPDK: Multi Architecture High Performance Packet Processing
DPDK: Multi Architecture High Performance Packet ProcessingDPDK: Multi Architecture High Performance Packet Processing
DPDK: Multi Architecture High Performance Packet Processing
 
Dpdk applications
Dpdk applicationsDpdk applications
Dpdk applications
 
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
DPDK Summit - 08 Sept 2014 - 6WIND - High Perf Networking Leveraging the DPDK...
 
Disruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on LinuxDisruptive IP Networking with Intel DPDK on Linux
Disruptive IP Networking with Intel DPDK on Linux
 
High Performance Networking Leveraging the DPDK and Growing Community
High Performance Networking Leveraging the DPDK and Growing CommunityHigh Performance Networking Leveraging the DPDK and Growing Community
High Performance Networking Leveraging the DPDK and Growing Community
 
Netsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfvNetsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfv
 
DPDK Summit 2015 - HP - Al Sanders
DPDK Summit 2015 - HP - Al SandersDPDK Summit 2015 - HP - Al Sanders
DPDK Summit 2015 - HP - Al Sanders
 
1 intro to_dpdk_and_hw
1 intro to_dpdk_and_hw1 intro to_dpdk_and_hw
1 intro to_dpdk_and_hw
 
DPDK In Depth
DPDK In DepthDPDK In Depth
DPDK In Depth
 
100 M pps on PC.
100 M pps on PC.100 M pps on PC.
100 M pps on PC.
 
Performance challenges in software networking
Performance challenges in software networkingPerformance challenges in software networking
Performance challenges in software networking
 
Devconf2017 - Can VMs networking benefit from DPDK
Devconf2017 - Can VMs networking benefit from DPDKDevconf2017 - Can VMs networking benefit from DPDK
Devconf2017 - Can VMs networking benefit from DPDK
 
Accelerate Service Function Chaining Vertical Solution with DPDK
Accelerate Service Function Chaining Vertical Solution with DPDKAccelerate Service Function Chaining Vertical Solution with DPDK
Accelerate Service Function Chaining Vertical Solution with DPDK
 
DPDK in Containers Hands-on Lab
DPDK in Containers Hands-on LabDPDK in Containers Hands-on Lab
DPDK in Containers Hands-on Lab
 
Dpdk performance
Dpdk performanceDpdk performance
Dpdk performance
 

Viewers also liked

Huawei Analyst 4Q07(TBR)
Huawei Analyst 4Q07(TBR)Huawei Analyst 4Q07(TBR)
Huawei Analyst 4Q07(TBR)
jiugedmu
 

Viewers also liked (17)

Qinetiq Business Strategy Proposal
Qinetiq Business Strategy ProposalQinetiq Business Strategy Proposal
Qinetiq Business Strategy Proposal
 
Strategy Proposal: IKEA
Strategy Proposal: IKEAStrategy Proposal: IKEA
Strategy Proposal: IKEA
 
Business leaders are engaging labor differently - Is your IT ready?
Business leaders are engaging labor differently - Is your IT ready?Business leaders are engaging labor differently - Is your IT ready?
Business leaders are engaging labor differently - Is your IT ready?
 
An open management plane (2015 Open Networking Summit)
An open management plane (2015 Open Networking Summit)An open management plane (2015 Open Networking Summit)
An open management plane (2015 Open Networking Summit)
 
Packet Framework - Cristian Dumitrescu
Packet Framework - Cristian DumitrescuPacket Framework - Cristian Dumitrescu
Packet Framework - Cristian Dumitrescu
 
9 creating cent_os 7_mages_for_dpdk_training
9 creating cent_os 7_mages_for_dpdk_training9 creating cent_os 7_mages_for_dpdk_training
9 creating cent_os 7_mages_for_dpdk_training
 
APMP Foundation: Developing Proposal Strategy
APMP Foundation: Developing Proposal StrategyAPMP Foundation: Developing Proposal Strategy
APMP Foundation: Developing Proposal Strategy
 
5 pipeline arch_rationale
5 pipeline arch_rationale5 pipeline arch_rationale
5 pipeline arch_rationale
 
NTT DOCOMO Deployment Case Study: Your Security, More Simple.
NTT DOCOMO Deployment Case Study: Your Security, More Simple.NTT DOCOMO Deployment Case Study: Your Security, More Simple.
NTT DOCOMO Deployment Case Study: Your Security, More Simple.
 
An Open Platform Strategy for NTT DoCoMo
An Open Platform Strategy for NTT DoCoMoAn Open Platform Strategy for NTT DoCoMo
An Open Platform Strategy for NTT DoCoMo
 
Huawei Analyst 4Q07(TBR)
Huawei Analyst 4Q07(TBR)Huawei Analyst 4Q07(TBR)
Huawei Analyst 4Q07(TBR)
 
Consulting proposal
Consulting proposalConsulting proposal
Consulting proposal
 
Lagopusで試すFW
Lagopusで試すFWLagopusで試すFW
Lagopusで試すFW
 
Tutorial-DeepLearning-PCSJ-IMPS2016
Tutorial-DeepLearning-PCSJ-IMPS2016Tutorial-DeepLearning-PCSJ-IMPS2016
Tutorial-DeepLearning-PCSJ-IMPS2016
 
Strategy Management of Ford Motor Company.
Strategy Management of Ford Motor Company.Strategy Management of Ford Motor Company.
Strategy Management of Ford Motor Company.
 
⼤企業で実現するイマドキの内製開発
⼤企業で実現するイマドキの内製開発⼤企業で実現するイマドキの内製開発
⼤企業で実現するイマドキの内製開発
 
Understanding DPDK
Understanding DPDKUnderstanding DPDK
Understanding DPDK
 

Similar to DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
 
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
Ceph Community
 
FlowER Erlang Openflow Controller
FlowER Erlang Openflow ControllerFlowER Erlang Openflow Controller
FlowER Erlang Openflow Controller
Holger Winkelmann
 

Similar to DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch (20)

Software Stacks to enable SDN and NFV
Software Stacks to enable SDN and NFVSoftware Stacks to enable SDN and NFV
Software Stacks to enable SDN and NFV
 
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
 
Approaching hyperconvergedopenstack
Approaching hyperconvergedopenstackApproaching hyperconvergedopenstack
Approaching hyperconvergedopenstack
 
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
Ceph Day Berlin: Deploying Flash Storage for Ceph without Compromising Perfor...
 
OpenFlow Tutorial
OpenFlow TutorialOpenFlow Tutorial
OpenFlow Tutorial
 
FD.IO Vector Packet Processing
FD.IO Vector Packet ProcessingFD.IO Vector Packet Processing
FD.IO Vector Packet Processing
 
Inside Microsoft's FPGA-Based Configurable Cloud
Inside Microsoft's FPGA-Based Configurable CloudInside Microsoft's FPGA-Based Configurable Cloud
Inside Microsoft's FPGA-Based Configurable Cloud
 
OpenPOWER Acceleration of HPCC Systems
OpenPOWER Acceleration of HPCC SystemsOpenPOWER Acceleration of HPCC Systems
OpenPOWER Acceleration of HPCC Systems
 
Ceph Day Seoul - AFCeph: SKT Scale Out Storage Ceph
Ceph Day Seoul - AFCeph: SKT Scale Out Storage Ceph Ceph Day Seoul - AFCeph: SKT Scale Out Storage Ceph
Ceph Day Seoul - AFCeph: SKT Scale Out Storage Ceph
 
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)
 
2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL
 
FlowER Erlang Openflow Controller
FlowER Erlang Openflow ControllerFlowER Erlang Openflow Controller
FlowER Erlang Openflow Controller
 
FIWARE Global Summit - Real-time Media Stream Processing Using Kurento
FIWARE Global Summit - Real-time Media Stream Processing Using KurentoFIWARE Global Summit - Real-time Media Stream Processing Using Kurento
FIWARE Global Summit - Real-time Media Stream Processing Using Kurento
 
Inside Microsoft's FPGA-Based Configurable Cloud
Inside Microsoft's FPGA-Based Configurable CloudInside Microsoft's FPGA-Based Configurable Cloud
Inside Microsoft's FPGA-Based Configurable Cloud
 
SDN/OpenFlow #lspe
SDN/OpenFlow #lspeSDN/OpenFlow #lspe
SDN/OpenFlow #lspe
 
100G Networking Berlin.pdf
100G Networking Berlin.pdf100G Networking Berlin.pdf
100G Networking Berlin.pdf
 
Summit 16: How to Compose a New OPNFV Solution Stack?
Summit 16: How to Compose a New OPNFV Solution Stack?Summit 16: How to Compose a New OPNFV Solution Stack?
Summit 16: How to Compose a New OPNFV Solution Stack?
 
Introduction to DPDK
Introduction to DPDKIntroduction to DPDK
Introduction to DPDK
 
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux DeviceAdding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
Adding IEEE 802.15.4 and 6LoWPAN to an Embedded Linux Device
 
Mellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDNMellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDN
 

More from Jim St. Leger

Transforming Communications Networks
Transforming Communications NetworksTransforming Communications Networks
Transforming Communications Networks
Jim St. Leger
 

More from Jim St. Leger (8)

DPDK Summit 2015 - Intro - Tim O'Driscoll
DPDK Summit 2015 - Intro - Tim O'DriscollDPDK Summit 2015 - Intro - Tim O'Driscoll
DPDK Summit 2015 - Intro - Tim O'Driscoll
 
DPDK Summit 2015 - Sprint - Arun Rajagopal
DPDK Summit 2015 - Sprint - Arun RajagopalDPDK Summit 2015 - Sprint - Arun Rajagopal
DPDK Summit 2015 - Sprint - Arun Rajagopal
 
DPDK Summit - 08 Sept 2014 - Intel - Closing Remarks
DPDK Summit - 08 Sept 2014 - Intel - Closing RemarksDPDK Summit - 08 Sept 2014 - Intel - Closing Remarks
DPDK Summit - 08 Sept 2014 - Intel - Closing Remarks
 
DPDK Summit - 08 Sept 2014 - Microsoft- PacketDirect
DPDK Summit - 08 Sept 2014 - Microsoft- PacketDirectDPDK Summit - 08 Sept 2014 - Microsoft- PacketDirect
DPDK Summit - 08 Sept 2014 - Microsoft- PacketDirect
 
DPDK Summit - 08 Sept 2014 - Ericsson - A Multi-Socket Ferrari for NFV
DPDK Summit - 08 Sept 2014 - Ericsson - A Multi-Socket Ferrari for NFVDPDK Summit - 08 Sept 2014 - Ericsson - A Multi-Socket Ferrari for NFV
DPDK Summit - 08 Sept 2014 - Ericsson - A Multi-Socket Ferrari for NFV
 
DPDK Summit - 08 Sept 2014 - Introduction - St Leger
DPDK Summit - 08 Sept 2014 - Introduction - St LegerDPDK Summit - 08 Sept 2014 - Introduction - St Leger
DPDK Summit - 08 Sept 2014 - Introduction - St Leger
 
Transforming Communications Networks
Transforming Communications NetworksTransforming Communications Networks
Transforming Communications Networks
 
3D Printing Overview
3D Printing Overview3D Printing Overview
3D Printing Overview
 

Recently uploaded

Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Marc Lester
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Lisi Hocke
 

Recently uploaded (20)

Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024OpenChain @ LF Japan Executive Briefing - May 2024
OpenChain @ LF Japan Executive Briefing - May 2024
 
how-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdfhow-to-download-files-safely-from-the-internet.pdf
how-to-download-files-safely-from-the-internet.pdf
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
^Clinic ^%[+27788225528*Abortion Pills For Sale In witbank
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024Modern binary build systems - PyCon 2024
Modern binary build systems - PyCon 2024
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
What is a Recruitment Management Software?
What is a Recruitment Management Software?What is a Recruitment Management Software?
What is a Recruitment Management Software?
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
^Clinic ^%[+27788225528*Abortion Pills For Sale In birch acres
^Clinic ^%[+27788225528*Abortion Pills For Sale In birch acres^Clinic ^%[+27788225528*Abortion Pills For Sale In birch acres
^Clinic ^%[+27788225528*Abortion Pills For Sale In birch acres
 
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
Salesforce Introduced Zero Copy Partner Network to Simplify the Process of In...
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdfStrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi.pdf
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 

DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

  • 1. 0 Copyright©2014 NTT corp. All Rights Reserved. High-performance vSwitch of the user, by the user, for the user A bird in cloud Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano NTT Labs
  • 2. 1 Copyright©2014 NTT corp. All Rights Reserved. Agenda vSwitch of the user Background and motivation vSwitch by the user High-performance software-switch •Concept •Design •Implementation & techniques •Performance Evaluation vSwitch for the user PoC of carrier usecase •Segment Routing or Service chaining (Tentative) Open source community
  • 3. 2 Copyright©2014 NTT corp. All Rights Reserved. Our mission Open innovation with open source for networking
  • 4. 3 Copyright©2014 NTT corp. All Rights Reserved. vSwitch of the user •Background •Motivation
  • 5. 4 Copyright©2014 NTT corp. All Rights Reserved. Trend shift in networking Closed (Vender lock-in) Yearly dev cycle Waterfall dev Standardization Protocol Special purpose HW / appliance Distributed cntrl Custom ASIC / FPGA Open (lock-in free) Monthly dev cycle Agile dev DE fact standard API Commodity HW/ Server Logically centralized cntrl Merchant Chip
  • 6. 5 Copyright©2014 NTT corp. All Rights Reserved. What is Software-Defined Networking? 5 Innovate services and applications in software development speed. Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf Decouple control plane and data plane →Free control plane out of the box (OpenFlow is one of APIs) Logically centralized view →Hide and abstract complexity of networks, provide entire view of the network Programmability via abstraction layer →Enables flexible and rapid service/application development SDN conceptual model
  • 7. 6 Copyright©2014 NTT corp. All Rights Reserved. OpenFlow vs conventional NW node OpenFlow controller OpenFlow switch Data-plane Flow Table Control plane (Routing / swithching) Data-plane (ASIC, FPGA) Control plane (routing/ switching) Flow match action Flow match action counter OpenFlow Protocol Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) Action (frame processing) (output port, drop, modify/pop/push header) Flow statistics (# of packet, byte size, flow duration,… ) counter Conventional NW node OpenFlow Flow Table #2 Flow Table #3
  • 8. 7 Copyright©2014 NTT corp. All Rights Reserved. 7 Evaluate the benefits of SDN by implementing our control plane and switch
  • 9. 8 Copyright©2014 NTT corp. All Rights Reserved. Motivation of vSwitch Agile and flexible networking Full automation in provisioning, operation and management Seamless networking for customers Server virtualization and NFV needs a high-performance software switch Short latency Wire-rate in case of short packet (64B) No High-performance OpenFlow 1.3 software switch for wide-area networks 1M flow rules 10Gbps-wire-rate Multiple table supports
  • 10. 9 Copyright©2014 NTT corp. All Rights Reserved. vSwitch requirement from user side 1.Run on the commodity PC server and NIC 2.Provide a gateway function to allow connect different various network domains Support of packet frame type in DC, IP-VPN, MPLS and access NW 3.Achieve 10Gbps-wire rate with >= 1M flow rules low-latency packet processing flexible flow lookup using multiple-tables High performance flow rule setup/delete 4.Run in userland and decrease tight-dependency to OS kernel easy software upgrade and deployment 5.Support various management and configuration protocols.
  • 11. 10 Copyright©2014 NTT corp. All Rights Reserved. Why Intel x86 server for networking? Strong processing power Many CPU cores •8 cores/CPU (2012), 12 cores/CPU (2013),… Rapid development cycle Available everywhere Can be purchased in Akihabara! 3 month lead time in case of special purpose HW Reasonable price and flexible configuration 8 core CPU (ATOM) with 1,000 USD Can chose configuration according to your budget Many programmers and engineers Porting to a NPU needs special skill and knowledge
  • 12. 11 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user •Concept •Design •Implementation & techniques •Performance Evaluation
  • 13. 12 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user Concept
  • 14. 13 Copyright©2014 NTT corp. All Rights Reserved. Target of Lagopus vSwitch High performance soft-based OpenFlow switch 10Gbps wire-rate packet processing / port 1M flow rules Expands the SDN apps to wide-area networks Not only for data centers WAN protocols, e.g. MPLS and PBB Various management /configuration interfaces Open Innovation Community-based development
  • 15. 14 Copyright©2014 NTT corp. All Rights Reserved. Approach for vSwitch development Simple is better for everything Packet processing Protocol handling Straight forward approach No OpenFlow 1.0 support Full scratch (No use of existing vSwitch code) User land packet processing as much as possible, keep kernel code small Kernel module update is hard for operation Every component & algorithm can be replaced Flow lookup, library, protocol handling
  • 16. 15 Copyright©2014 NTT corp. All Rights Reserved. vSwitch design Simple modular-based design Switch control agent Data-plane Switch control agent Unified configuration data store Multiple management IF Data-plane control with HAL Data-plane High performance I/O library (Intel DPDK) Raw socket for test
  • 17. 16 Copyright©2014 NTT corp. All Rights Reserved. Implementation strategy for vSwitch Massive RX interrupts handling for NIC device => Polling-based packet receiving Heavy overhead of task switch => Thread assignment (one thread/one physical CPU) Lower performance of PCI-Express I/O and memory bandwidth compared with CPU => Reduction of # of access in I/O and memory Shared data access is bottleneck between threads => Lockless-queue, RCU, batch processing
  • 18. 17 Copyright©2014 NTT corp. All Rights Reserved. Processing bypass for speed NIC skb_buf Ethernet Driver API Socket API vswitch packet buffer packet buffer memory Standard linux application 1. Interrupt & DMA 2. system call (read) User space Kernel space Driver 4. DMA 3. system call (write) NIC Ethernet Driver API User-mode I/O & HAL vswitch packet buffer Application with intel DPDK 1. DMA Write 2. DMA READ DPDK Library Polling-base packet handling Event-base packet handling
  • 19. 18 Copyright©2014 NTT corp. All Rights Reserved. Packet processing using multi core CPUs Exploit many core CPUs Reduce data copy & move (reference access) Simple packet classifier for parallel processing in I/O RX Decouple I/O processing and flow processing Improve D-cache efficiency Explicit thread assign to CPU core NIC 1 RX NIC 2 RX I/O RX CPU0 I/O RX CPU1 NIC 1 TX NIC 2 TX I/O TX CPU6 I/O TX CPU7 Flow lookup packet processing CPU2 Flow lookup packet processing CPU4 Flow lookup packet processing CPU3 Flow lookup packet processing CPU5 NIC 3 RX NIC 4 RX NIC 3 TX NIC 4 TX NIC RX buffer Ring buffer Ring buffer NIC TX buffer
  • 20. 19 Copyright©2014 NTT corp. All Rights Reserved. Packet batching Reduce # of lock in flow lookup table Frequent locks are required •Switch OpenFlow agent and counter retrieve for SNMP •Packet processing Input packet Lookup table Input packet Lookup table Lock Unlock ロック Unlock ●Naïve implementation ●Packet batching implementation Lock is required for each packet Lock can be reduced due to packet batching
  • 21. 20 Copyright©2014 NTT corp. All Rights Reserved. Bypass pipeline with flow $ 20 ●Naïve implementation table1 table2 table3 Input packet table1 table2 table3 Input packet Flow cache 1. New flow 2. Success flow Output packet Multiple flow $ generation & mngmt Write flow $ ●Lagopus implementation Output packet Packet Reduce # of flow lookup in multiple table Exploit flow $ for
  • 22. 21 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user Performance Evaluation
  • 23. 22 Copyright©2014 NTT corp. All Rights Reserved. Performance Evaluation Summary Throughput: 10Gbps wire-rate Flow rules: 1M flow rules Evaluation models WAN-DC gateway •MPLS-VLAN mapping L2 switch •Mac address switching
  • 24. 23 Copyright©2014 NTT corp. All Rights Reserved. Performance Evaluation Evaluation setup Server spec. CPU: Dual Intel Xeon E5-2660 •8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge Memory: DDR3-1600 ECC 64GB •Quad-channel 8x8GB Chipset: Intel C602 NIC: Intel Ethernet Converged Network Adapter X520-DA2 •Intel 82599ES, PCIe v2.0 Server Lagopus Flow table tester Flows Throughput (bps/pps/%) Flow rules Packet size Flow cache (on/off)
  • 25. 24 Copyright©2014 NTT corp. All Rights Reserved. WAN-DC Gateway 0 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 1000 1200 1400 1600 Throughput (Gbps) Packet size (byte) 10 flow rules 100 flow rules 1k flow rules 10k flow rules 100k flow rules 1M flow rules Throughput vs packet size, 1 flow, flow-cache 0 1 2 3 4 5 6 7 8 9 10 1 10 100 1000 10000 100000 1000000 Throughput (Gbps) flows 10k flow rules 100k flow rules 1M flow rules Throughput vs flows, 1518 bytes packet
  • 26. 25 Copyright©2014 NTT corp. All Rights Reserved. L2 switch performance (Mbps) 10GbE x 2 (RFC2889 test) 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 LINC OVS (netdev) OVS (kernel) Lagopus (software) Mbps 72 128 256 512 1024 1280 1518 Intel xeon E5-2660 (8cores, 16threads), Intel X520-DA2, DDR3-1600 64GB Packet size
  • 27. 26 Copyright©2014 NTT corp. All Rights Reserved. vSwitch for the user PoC of carrier usecase Open source community
  • 28. 27 Copyright©2014 NTT corp. All Rights Reserved. SPRING alternative way of IP forwarding use “labels” to forward packets much simpler architecture than MPLS network can utilize traditional MPLS routers good chemistry with NFV For what ? policy-based routing •explicit routing instead of shortest path routing service-chaining (NFV) •apply services to specific traffic pattern
  • 29. 28 Copyright©2014 NTT corp. All Rights Reserved. Career Network with SPRING Source Destination NFV server  use lagopus as NFV server with career grade routers  all nodes include NFV server are labeled
  • 30. 29 Copyright©2014 NTT corp. All Rights Reserved. Career Network with SPRING Source Destination NFV server  at the ingress router, label stack is pushed  label stack represents the policy IP
  • 31. 30 Copyright©2014 NTT corp. All Rights Reserved. PoC Implementation
  • 32. 31 Copyright©2014 NTT corp. All Rights Reserved. Demo
  • 33. 32 Copyright©2014 NTT corp. All Rights Reserved. Open Source Activity for the user Lagopus goes wild since this summer Your contribution are very welcome! http://lagopus.github.io/ Twitter: lagopusvswitch Ryu: component-based SDN framework OpenFlow 1.3, 1.4, BGP, BMP, etc.. http://osrg.github.io/ryu/
  • 34. 33 Copyright©2014 NTT corp. All Rights Reserved. Summary We contribute vSwitch for user Improve performance and scalability Enhance the functionality and capability DPDK is great library for user Easy to develop & implement Easy to deploy like UNIX apps
  • 35. 34 Copyright©2014 NTT corp. All Rights Reserved. Thank you! This research is a part of the project for “Research and Development of Network Virtualization Technology” supported by the Ministry of Internal Affairs and Communications.