Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
0Copyright©2015 NTT corp. All Rights Reserved.
It's kind of fun
to do the impossible with DPDK
Yoshihiro Nakajima, Hirokaz...
1Copyright©2015 NTT corp. All Rights Reserved.
Agenda
 Motivation for fun 
 Fun with Lagopus SDN switch 
 Fun with sp...
2Copyright©2015 NTT corp. All Rights Reserved.
Motivation for fun 
Trend
3Copyright©2015 NTT corp. All Rights Reserved.
Trend shift in networking
 Closed (Vender lock-in)
 Yearly dev cycle
 Wa...
4Copyright©2015 NTT corp. All Rights Reserved.
4
Evaluate the benefits of SDN
by implementing control plane and switch
for...
5Copyright©2015 NTT corp. All Rights Reserved.
Lagopus SDN switch project
6Copyright©2015 NTT corp. All Rights Reserved.
Goal of Lagopus project
 Provide NFV/SDN-aware switch framework
 SDN swit...
7Copyright©2015 NTT corp. All Rights Reserved.
 High-performance SDN/OF software switch
 40-Gbps packet processing throu...
8Copyright©2015 NTT corp. All Rights Reserved.
Lagopus vSwitch
switch configuration datastore
(config/stats API, SW DSL)
F...
9Copyright©2015 NTT corp. All Rights Reserved.
Fun with speed
• Performance Improvement
• Smart FPGA NIC for software data...
10Copyright©2015 NTT corp. All Rights Reserved.
Performance Improvement
11Copyright©2015 NTT corp. All Rights Reserved.
Basic packet processing
Network I/O
RX
packet
Frame
processing
Flow lookup...
12Copyright©2015 NTT corp. All Rights Reserved.
What we did for performance
Network I/O
RX
packet
Frame
processing
Flow lo...
13Copyright©2015 NTT corp. All Rights Reserved.
Road to 10Gbps packet processing
with 1M OpenFlow flow entries
0
2
4
6
8
1...
14Copyright©2015 NTT corp. All Rights Reserved.
Big change from Y2013
NowBefore project
 
10Gpbs by
software
dataplane?
...
15Copyright©2015 NTT corp. All Rights Reserved.
Smart FPGA NIC for
software dataplane
16Copyright©2015 NTT corp. All Rights Reserved.
Motivation
 Network I/O is not optimized for multi-core CPU
 Std NIC doe...
17Copyright©2015 NTT corp. All Rights Reserved.
 Leverage hardware offload processing of
smart FPGA NIC
 Flexible hardwa...
18Copyright©2015 NTT corp. All Rights Reserved.
FPGA Flow Classification & Dispatch
FPGA
・・・
DMA transfter
DST DST DST
RAM...
19Copyright©2015 NTT corp. All Rights Reserved.
Lagopus software data-plane
• Hash, cache search and match are omitted
• I...
20Copyright©2015 NTT corp. All Rights Reserved.
Improved performance with Smart FPGA NIC
ETH/VLAN/IPV4, 1 Flow, Intel Xeon...
21Copyright©2015 NTT corp. All Rights Reserved.
Summery
 Small extension of NIC give great benefit for software
dataplane...
22Copyright©2015 NTT corp. All Rights Reserved.
Fun with experience
SDN IX
23Copyright©2015 NTT corp. All Rights Reserved.
SDN IX
@ Interop Tokyo 2015 ShowNet
Interop Tokyo is
the biggest Internet-...
24Copyright©2015 NTT corp. All Rights Reserved.
Motivation
 IX (Internet eXchange)
 Packet exchange point between ISP an...
25Copyright©2015 NTT corp. All Rights Reserved.
What is SDN IX?
 Next generation IX with SDN technology
 Web portal-base...
26Copyright©2015 NTT corp. All Rights Reserved.
Lagopus @ ShowNet 2015
 Two Lagopus (soft switch) are deployed for
SDN-IX...
27Copyright©2015 NTT corp. All Rights Reserved.
Lagopus @ ShowNet rack
28Copyright©2015 NTT corp. All Rights Reserved.
Path provisioning
Router Router
AS290AS131154
DIX-IEJPIXISP
Router
10G-LR
...
29Copyright©2015 NTT corp. All Rights Reserved.
 Average 2Gbps throughput
 No packet drop
 No reboot & no trouble for 1...
30Copyright©2015 NTT corp. All Rights Reserved.
Big change happened
Before After
 
vSwitch has
lots of issues on
perform...
31Copyright©2015 NTT corp. All Rights Reserved.
 It's kind of fun to do the impossible with DPDK
 Enjoy hacking with DPD...
32Copyright©2015 NTT corp. All Rights Reserved.
 Lagopus demonstration
 vSwitch performance benchmark
• Haswell-EP and F...
Upcoming SlideShare
Loading in …5
×

DPDK summit 2015: It's kind of fun to do the impossible with DPDK

3,523 views

Published on

It's kind of fun to do the impossible with DPDK and Lagopus SDN/OpenFlow vSwitch.

Published in: Software

DPDK summit 2015: It's kind of fun to do the impossible with DPDK

  1. 1. 0Copyright©2015 NTT corp. All Rights Reserved. It's kind of fun to do the impossible with DPDK Yoshihiro Nakajima, Hirokazu Takahashi, Kunihiro Ishiguro, Koji Yamazaki NTT Labs
  2. 2. 1Copyright©2015 NTT corp. All Rights Reserved. Agenda  Motivation for fun   Fun with Lagopus SDN switch   Fun with speed   Smart FPGA for software dataplane  Fun with experience   SDN IX @ Interop Tokyo 2015 ShowNet
  3. 3. 2Copyright©2015 NTT corp. All Rights Reserved. Motivation for fun  Trend
  4. 4. 3Copyright©2015 NTT corp. All Rights Reserved. Trend shift in networking  Closed (Vender lock-in)  Yearly dev cycle  Waterfall dev  Standardization  Protocol  Special purpose HW / appliance  Distributed cntrl  Custom ASIC / FPGA  Wired logic dataplane  Open (lock-in free)  Monthly dev cycle  Agile dev  DE fact standard  API  Commodity HW/ Server  Logically centralized cntrl  Merchant Chip  Software dataplane
  5. 5. 4Copyright©2015 NTT corp. All Rights Reserved. 4 Evaluate the benefits of SDN by implementing control plane and switch for fun 
  6. 6. 5Copyright©2015 NTT corp. All Rights Reserved. Lagopus SDN switch project
  7. 7. 6Copyright©2015 NTT corp. All Rights Reserved. Goal of Lagopus project  Provide NFV/SDN-aware switch framework  SDN switch agent (OpenFlow, REST)  100Gbps high-performance soft dataplane  Flexible/extensible switch configuration datastore  DPDK extension (library, FPGA NIC, vNIC)  Cloud middleware integration  Expand software-based packet processing to carrier networks  Hardware acceleration and processing offload for scalable software dataplane
  8. 8. 7Copyright©2015 NTT corp. All Rights Reserved.  High-performance SDN/OF software switch  40-Gbps packet processing throughput / port  Mega-class flow entries support  Low-latency and wire-rate speed in smaller packet size  Expands SDN to WAN, GW and NFV  Multiple frame format support • MPLS, PBB, MACinMAC, IPv4, IPv6  Hybrid SDN support • REST API, OpenFlow 1.3, OVSDB, NETCONF • Legacy protocol support  vSwtich for hypervisor, container virtualization Target
  9. 9. 8Copyright©2015 NTT corp. All Rights Reserved. Lagopus vSwitch switch configuration datastore (config/stats API, SW DSL) FPGA NIC DPDK NIC DPDK libs/PMD driver Lagopus soft dataplane flow lookup flow cache OpenFlow pipeline queue/ policer Flow table Flow table flow table Flow table Flow tableGroup table Flow table Flow tablemeter table switch HAL OpenFlow 1.3agent JSON IF SNMP CLI OVSDB NET CONF L2bridge L3 (IPv4ICMP) SDN switch Agent • Full OpenFlow 1.3.4 support • Controller-less basic L2 and L3 support SDN-aware management API • OVSDB, REST • Ansible support DPDK-enabled soft dataplane • Over-10-Gbps performance • Low latency packet processing • high performance multi-layer flow lookup DPDK-enabled vNIC for NFV (virtq-pmd) Switch configuration datastore • Pub/sub mechanism • Switch config DSL • JSON IF support Soft-dataplane aware FPGA NIC
  10. 10. 9Copyright©2015 NTT corp. All Rights Reserved. Fun with speed • Performance Improvement • Smart FPGA NIC for software dataplane (collaboration with Xilinx)
  11. 11. 10Copyright©2015 NTT corp. All Rights Reserved. Performance Improvement
  12. 12. 11Copyright©2015 NTT corp. All Rights Reserved. Basic packet processing Network I/O RX packet Frame processing Flow lookup & Action QoS・Queue Network I/O TX Packet classification & packet distribution to buffers Packet parsing lookup, Header rewrite Encap/decap Policer, Shaper Marking packet
  13. 13. 12Copyright©2015 NTT corp. All Rights Reserved. What we did for performance Network I/O RX packet Frame processing Flow lookup & Action QoS・Queue Network I/O TX packet • Delayed packet frame evaluation • Delayed action (processing) evaluation • Packet batching to improve CPU $ efficiency • Delayed flow stats evaluation • Smart flow classification • Thread assignment optimization • Parallel flow lookup • Lookup tree compaction • High-performance lookup algorithm for OpenFlow (multi-layer, mask, priority-aware flow lookup) • Flow $ mechanism • Batch size tuning
  14. 14. 13Copyright©2015 NTT corp. All Rights Reserved. Road to 10Gbps packet processing with 1M OpenFlow flow entries 0 2 4 6 8 10 12 14 2013 Q2 2013 Q3 2013 Q4 2014 Q1 2014 Q2 MPPS DPDK-powered I/O Lookup performance Project start point OF lookup improved Lookup optimization Flow $ and Lookup optimization with 12 CPU cores Packet processing optimization with 4 CPU cores 
  15. 15. 14Copyright©2015 NTT corp. All Rights Reserved. Big change from Y2013 NowBefore project   10Gpbs by software dataplane? Impossible!! Software dataplane becomes great performance. We try vSwitch for our usecases.
  16. 16. 15Copyright©2015 NTT corp. All Rights Reserved. Smart FPGA NIC for software dataplane
  17. 17. 16Copyright©2015 NTT corp. All Rights Reserved. Motivation  Network I/O is not optimized for multi-core CPU  Std NIC does not support RSS for WAN protocol  Software-based processing are heavy  Packet classifier  Packet dispatcher are heavy  QoS and needs lots of CPU cycles NIC 1 RX NIC 2 RX I/O RX I/O RX NIC 1 TX NIC 2 TX I/O TX I/O TX Flow lookup packet processing Flow lookup packet processing Flow lookup packet processing Flow lookup packet processing NIC 3 RX NIC 4 RX NIC 3 TX NIC 4 TX NIC RX buffer Ring buffer Ring buffer NIC TX buffer Bottleneck for Multi-core CPU QoS Compact flow lookup tree Multiple network I/O for multi CORE CPU
  18. 18. 17Copyright©2015 NTT corp. All Rights Reserved.  Leverage hardware offload processing of smart FPGA NIC  Flexible hardware-based packet classifier & dispatcher  Hardware-based packet marking for post-packet-processing  Optimized to multi-core CPU  Efficient packet processing for multi-thread Co-design approach for performance Performance Flexiblity Availability SDN hardware switch +HW offload Advance features leveraging HW High-relibility, High-performance with HW
  19. 19. 18Copyright©2015 NTT corp. All Rights Reserved. FPGA Flow Classification & Dispatch FPGA ・・・ DMA transfter DST DST DST RAM3 RAM2 RAM1 RAMN RAM5 RAM4 #1 #2 #3 #4 #5 #N Cores Workload #1 #2 #3 #4 #5 #N Cores Workload Ingress traffic (RX) Classifier (SDNet) Multi-core Processor (Xeon®E5) Processor RX workload w/o smart FPGA NIC w/ smart FPGA NIC 32 channels RX DMA Flow Dispatcher Dispatcher Classifier Xilinx Virtex®-7 (XC7VX690T)
  20. 20. 19Copyright©2015 NTT corp. All Rights Reserved. Lagopus software data-plane • Hash, cache search and match are omitted • Increase network I/O performance  Front-end hardware-based packet classification and packet dispatcher  Reconfigurable packet filter for dynamic load balance for worker thread  DPDK flow director API compatible FPGA NIC and soft dataplane RX Disp TX TX threadRX thread Header parsing Hash calc Cache search Match Action Init OpenFlow worker thread For 32 queues Hardware classification & dispatch on FPGA NIC • Reprogrammable filters in look-up table • Configured by DPDK flow director API VLAN 1 VLAN 2 VLAN 3 ・ ・ ・ ・ ・ VLAN 32 Q1 Q2 Q3 ・ ・ ・ ・ ・ Q32 Result (QID)Perfect match filter (ex. VLAN-ID) Filterentries Filters For 32 queues
  21. 21. 20Copyright©2015 NTT corp. All Rights Reserved. Improved performance with Smart FPGA NIC ETH/VLAN/IPV4, 1 Flow, Intel Xeon® E5-2680 v2 (Ivy Bridge-EP)@ 2.8GHz, 10C/20T, Xilinx FPGA NIC, DPDK-1.7.1 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 40.00 0 500 1000 1500 2000 Throughput(Gpbs) Frame Size (Bytes) 40G Line Rate 4 cores only, Flow Director=OFF 4 cores + FPGA, Flow Director=ON 8 cores + FPGA, Flow Director=ON 4 cores: RX(1C), Worker(2C), TX(1C) 8 cores: RX(2C), Worker(5C), TX(1C) 40Gbps wire-speed from 384 Bytes
  22. 22. 21Copyright©2015 NTT corp. All Rights Reserved. Summery  Small extension of NIC give great benefit for software dataplane  Performance improvement  Save CPU cycle for important processing  Save CPU cores for the same processing with standard NIC
  23. 23. 22Copyright©2015 NTT corp. All Rights Reserved. Fun with experience SDN IX
  24. 24. 23Copyright©2015 NTT corp. All Rights Reserved. SDN IX @ Interop Tokyo 2015 ShowNet Interop Tokyo is the biggest Internet-related technology show in Japan. This trial was collaboration with NECOMA project (NAIST & University of Tokyo)
  25. 25. 24Copyright©2015 NTT corp. All Rights Reserved. Motivation  IX (Internet eXchange)  Packet exchange point between ISP and DC-SP  Boarder router of ISP exchanges route information  Issue  Enhance automation in provisioning and configuration  DDoS attack is one of the most critical issues • ISP wants to reduce DDoS-related traffic in origin • DDoS traffic occupies link bandwidth IX ISP-CISP A ISP-DISP B SW SWSW SW ISP-EISP F IX ISP-CISP A ISP-DISP B SW SWSW SW ISP-EISP F
  26. 26. 25Copyright©2015 NTT corp. All Rights Reserved. What is SDN IX?  Next generation IX with SDN technology  Web portal-based path provisioning between ISPs • Inter-AS L2 connectivity – VLAN-based path provisioning – Private peer provisioning  Protect network from DDoS attack • On-demand 5-tuple-baesd packet filtering  SDN IX controller and distributed SDN/OpenFlow IX core switch Developed by NECOMA project (NAIST and University of Tokyo) ISP-CISP A ISP-DISP B ISP-EISP F ISP-CISP A ISP-DISP B SW SWSW SW ISP-EISP F
  27. 27. 26Copyright©2015 NTT corp. All Rights Reserved. Lagopus @ ShowNet 2015  Two Lagopus (soft switch) are deployed for SDN-IX core switch  Multiple 10Gbps links  Dual Xeon E5 8core CPUs
  28. 28. 27Copyright©2015 NTT corp. All Rights Reserved. Lagopus @ ShowNet rack
  29. 29. 28Copyright©2015 NTT corp. All Rights Reserved. Path provisioning Router Router AS290AS131154 DIX-IEJPIXISP Router 10G-LR lagopus-1 (DPID:2) OFS (DPID:1) IXJGNX lagopus-2 (DPID:4) OFS (DPID:3) xg-89:0.1 (port 4) xg-83:0.0 (port 1) xg-89:00.0 (port 3) xg-83:00.1 (port 2) xg-83:0.0 (port 1) xg-83:0.1 (port 2) xg-89:0.0 (port 3) xg-1-0-49 (port 49) xg-1-0-51 (port 51) xg-1-0-52 (port 52) xg-1-0-50 (port 50) xg-1-0-49 (port 49) xg-1-0-50 (port 50) xg-1-0-51 (port 51) Otemachi Makuhari (Interop Veneue)
  30. 30. 29Copyright©2015 NTT corp. All Rights Reserved.  Average 2Gbps throughput  No packet drop  No reboot & no trouble for 1 week during Interop Tokyo  Sometimes 10Gbps burst traffic Traffic on Lagopus @Makuhari
  31. 31. 30Copyright©2015 NTT corp. All Rights Reserved. Big change happened Before After   vSwitch has lots of issues on performance, scalability, stability, ….. vSwitch works well without any trouble! Good performance, Good stability.
  32. 32. 31Copyright©2015 NTT corp. All Rights Reserved.  It's kind of fun to do the impossible with DPDK  Enjoy hacking with DPDK for your networking!  Performance optimization is fun   Lagopus project commit to high-performance vswitch development for fun   We still have lots of issues for fun  • Lookup optimization, performance improvement, ….  Changing one’s mind is great fun   Real experience change their mind  Conclusion
  33. 33. 32Copyright©2015 NTT corp. All Rights Reserved.  Lagopus demonstration  vSwitch performance benchmark • Haswell-EP and Fortville • Carrier usecase  MPLS-based segment routing (source routing) and NFV integration https://github.com/lagopus/ vSwitch, DPDK extension, and more… Visit our booth #172 in IDF15SFO

×