Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

5,308 views

Published on

Wataru Ishida and Yoshihiro Nakajima of NTT present a high performance vSwitch design that uses DPDK for acceleration of the vSwitch.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

DPDK Summit - 08 Sept 2014 - NTT - High Performance vSwitch

  1. 1. 0 Copyright©2014 NTT corp. All Rights Reserved. High-performance vSwitch of the user, by the user, for the user A bird in cloud Yoshihiro Nakajima, Wataru Ishida, Tomonori Fujita, Takahashi Hirokazu, Tomoya Hibi, Hitoshi Matsutahi, Katsuhiro Shimano NTT Labs
  2. 2. 1 Copyright©2014 NTT corp. All Rights Reserved. Agenda vSwitch of the user Background and motivation vSwitch by the user High-performance software-switch •Concept •Design •Implementation & techniques •Performance Evaluation vSwitch for the user PoC of carrier usecase •Segment Routing or Service chaining (Tentative) Open source community
  3. 3. 2 Copyright©2014 NTT corp. All Rights Reserved. Our mission Open innovation with open source for networking
  4. 4. 3 Copyright©2014 NTT corp. All Rights Reserved. vSwitch of the user •Background •Motivation
  5. 5. 4 Copyright©2014 NTT corp. All Rights Reserved. Trend shift in networking Closed (Vender lock-in) Yearly dev cycle Waterfall dev Standardization Protocol Special purpose HW / appliance Distributed cntrl Custom ASIC / FPGA Open (lock-in free) Monthly dev cycle Agile dev DE fact standard API Commodity HW/ Server Logically centralized cntrl Merchant Chip
  6. 6. 5 Copyright©2014 NTT corp. All Rights Reserved. What is Software-Defined Networking? 5 Innovate services and applications in software development speed. Reference: http://opennetsummit.org/talks/ONS2012/pitt-mon-ons.pdf Decouple control plane and data plane →Free control plane out of the box (OpenFlow is one of APIs) Logically centralized view →Hide and abstract complexity of networks, provide entire view of the network Programmability via abstraction layer →Enables flexible and rapid service/application development SDN conceptual model
  7. 7. 6 Copyright©2014 NTT corp. All Rights Reserved. OpenFlow vs conventional NW node OpenFlow controller OpenFlow switch Data-plane Flow Table Control plane (Routing / swithching) Data-plane (ASIC, FPGA) Control plane (routing/ switching) Flow match action Flow match action counter OpenFlow Protocol Flexible flow match pattern (Port #, VLAN ID, MAC addr, IP addr, TCP port #) Action (frame processing) (output port, drop, modify/pop/push header) Flow statistics (# of packet, byte size, flow duration,… ) counter Conventional NW node OpenFlow Flow Table #2 Flow Table #3
  8. 8. 7 Copyright©2014 NTT corp. All Rights Reserved. 7 Evaluate the benefits of SDN by implementing our control plane and switch
  9. 9. 8 Copyright©2014 NTT corp. All Rights Reserved. Motivation of vSwitch Agile and flexible networking Full automation in provisioning, operation and management Seamless networking for customers Server virtualization and NFV needs a high-performance software switch Short latency Wire-rate in case of short packet (64B) No High-performance OpenFlow 1.3 software switch for wide-area networks 1M flow rules 10Gbps-wire-rate Multiple table supports
  10. 10. 9 Copyright©2014 NTT corp. All Rights Reserved. vSwitch requirement from user side 1.Run on the commodity PC server and NIC 2.Provide a gateway function to allow connect different various network domains Support of packet frame type in DC, IP-VPN, MPLS and access NW 3.Achieve 10Gbps-wire rate with >= 1M flow rules low-latency packet processing flexible flow lookup using multiple-tables High performance flow rule setup/delete 4.Run in userland and decrease tight-dependency to OS kernel easy software upgrade and deployment 5.Support various management and configuration protocols.
  11. 11. 10 Copyright©2014 NTT corp. All Rights Reserved. Why Intel x86 server for networking? Strong processing power Many CPU cores •8 cores/CPU (2012), 12 cores/CPU (2013),… Rapid development cycle Available everywhere Can be purchased in Akihabara! 3 month lead time in case of special purpose HW Reasonable price and flexible configuration 8 core CPU (ATOM) with 1,000 USD Can chose configuration according to your budget Many programmers and engineers Porting to a NPU needs special skill and knowledge
  12. 12. 11 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user •Concept •Design •Implementation & techniques •Performance Evaluation
  13. 13. 12 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user Concept
  14. 14. 13 Copyright©2014 NTT corp. All Rights Reserved. Target of Lagopus vSwitch High performance soft-based OpenFlow switch 10Gbps wire-rate packet processing / port 1M flow rules Expands the SDN apps to wide-area networks Not only for data centers WAN protocols, e.g. MPLS and PBB Various management /configuration interfaces Open Innovation Community-based development
  15. 15. 14 Copyright©2014 NTT corp. All Rights Reserved. Approach for vSwitch development Simple is better for everything Packet processing Protocol handling Straight forward approach No OpenFlow 1.0 support Full scratch (No use of existing vSwitch code) User land packet processing as much as possible, keep kernel code small Kernel module update is hard for operation Every component & algorithm can be replaced Flow lookup, library, protocol handling
  16. 16. 15 Copyright©2014 NTT corp. All Rights Reserved. vSwitch design Simple modular-based design Switch control agent Data-plane Switch control agent Unified configuration data store Multiple management IF Data-plane control with HAL Data-plane High performance I/O library (Intel DPDK) Raw socket for test
  17. 17. 16 Copyright©2014 NTT corp. All Rights Reserved. Implementation strategy for vSwitch Massive RX interrupts handling for NIC device => Polling-based packet receiving Heavy overhead of task switch => Thread assignment (one thread/one physical CPU) Lower performance of PCI-Express I/O and memory bandwidth compared with CPU => Reduction of # of access in I/O and memory Shared data access is bottleneck between threads => Lockless-queue, RCU, batch processing
  18. 18. 17 Copyright©2014 NTT corp. All Rights Reserved. Processing bypass for speed NIC skb_buf Ethernet Driver API Socket API vswitch packet buffer packet buffer memory Standard linux application 1. Interrupt & DMA 2. system call (read) User space Kernel space Driver 4. DMA 3. system call (write) NIC Ethernet Driver API User-mode I/O & HAL vswitch packet buffer Application with intel DPDK 1. DMA Write 2. DMA READ DPDK Library Polling-base packet handling Event-base packet handling
  19. 19. 18 Copyright©2014 NTT corp. All Rights Reserved. Packet processing using multi core CPUs Exploit many core CPUs Reduce data copy & move (reference access) Simple packet classifier for parallel processing in I/O RX Decouple I/O processing and flow processing Improve D-cache efficiency Explicit thread assign to CPU core NIC 1 RX NIC 2 RX I/O RX CPU0 I/O RX CPU1 NIC 1 TX NIC 2 TX I/O TX CPU6 I/O TX CPU7 Flow lookup packet processing CPU2 Flow lookup packet processing CPU4 Flow lookup packet processing CPU3 Flow lookup packet processing CPU5 NIC 3 RX NIC 4 RX NIC 3 TX NIC 4 TX NIC RX buffer Ring buffer Ring buffer NIC TX buffer
  20. 20. 19 Copyright©2014 NTT corp. All Rights Reserved. Packet batching Reduce # of lock in flow lookup table Frequent locks are required •Switch OpenFlow agent and counter retrieve for SNMP •Packet processing Input packet Lookup table Input packet Lookup table Lock Unlock ロック Unlock ●Naïve implementation ●Packet batching implementation Lock is required for each packet Lock can be reduced due to packet batching
  21. 21. 20 Copyright©2014 NTT corp. All Rights Reserved. Bypass pipeline with flow $ 20 ●Naïve implementation table1 table2 table3 Input packet table1 table2 table3 Input packet Flow cache 1. New flow 2. Success flow Output packet Multiple flow $ generation & mngmt Write flow $ ●Lagopus implementation Output packet Packet Reduce # of flow lookup in multiple table Exploit flow $ for
  22. 22. 21 Copyright©2014 NTT corp. All Rights Reserved. vSwitch by the user Performance Evaluation
  23. 23. 22 Copyright©2014 NTT corp. All Rights Reserved. Performance Evaluation Summary Throughput: 10Gbps wire-rate Flow rules: 1M flow rules Evaluation models WAN-DC gateway •MPLS-VLAN mapping L2 switch •Mac address switching
  24. 24. 23 Copyright©2014 NTT corp. All Rights Reserved. Performance Evaluation Evaluation setup Server spec. CPU: Dual Intel Xeon E5-2660 •8 core(16 thread), 20M Cache, 2.2 GHz, 8.00GT/s QPI, Sandy bridge Memory: DDR3-1600 ECC 64GB •Quad-channel 8x8GB Chipset: Intel C602 NIC: Intel Ethernet Converged Network Adapter X520-DA2 •Intel 82599ES, PCIe v2.0 Server Lagopus Flow table tester Flows Throughput (bps/pps/%) Flow rules Packet size Flow cache (on/off)
  25. 25. 24 Copyright©2014 NTT corp. All Rights Reserved. WAN-DC Gateway 0 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 1000 1200 1400 1600 Throughput (Gbps) Packet size (byte) 10 flow rules 100 flow rules 1k flow rules 10k flow rules 100k flow rules 1M flow rules Throughput vs packet size, 1 flow, flow-cache 0 1 2 3 4 5 6 7 8 9 10 1 10 100 1000 10000 100000 1000000 Throughput (Gbps) flows 10k flow rules 100k flow rules 1M flow rules Throughput vs flows, 1518 bytes packet
  26. 26. 25 Copyright©2014 NTT corp. All Rights Reserved. L2 switch performance (Mbps) 10GbE x 2 (RFC2889 test) 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 LINC OVS (netdev) OVS (kernel) Lagopus (software) Mbps 72 128 256 512 1024 1280 1518 Intel xeon E5-2660 (8cores, 16threads), Intel X520-DA2, DDR3-1600 64GB Packet size
  27. 27. 26 Copyright©2014 NTT corp. All Rights Reserved. vSwitch for the user PoC of carrier usecase Open source community
  28. 28. 27 Copyright©2014 NTT corp. All Rights Reserved. SPRING alternative way of IP forwarding use “labels” to forward packets much simpler architecture than MPLS network can utilize traditional MPLS routers good chemistry with NFV For what ? policy-based routing •explicit routing instead of shortest path routing service-chaining (NFV) •apply services to specific traffic pattern
  29. 29. 28 Copyright©2014 NTT corp. All Rights Reserved. Career Network with SPRING Source Destination NFV server  use lagopus as NFV server with career grade routers  all nodes include NFV server are labeled
  30. 30. 29 Copyright©2014 NTT corp. All Rights Reserved. Career Network with SPRING Source Destination NFV server  at the ingress router, label stack is pushed  label stack represents the policy IP
  31. 31. 30 Copyright©2014 NTT corp. All Rights Reserved. PoC Implementation
  32. 32. 31 Copyright©2014 NTT corp. All Rights Reserved. Demo
  33. 33. 32 Copyright©2014 NTT corp. All Rights Reserved. Open Source Activity for the user Lagopus goes wild since this summer Your contribution are very welcome! http://lagopus.github.io/ Twitter: lagopusvswitch Ryu: component-based SDN framework OpenFlow 1.3, 1.4, BGP, BMP, etc.. http://osrg.github.io/ryu/
  34. 34. 33 Copyright©2014 NTT corp. All Rights Reserved. Summary We contribute vSwitch for user Improve performance and scalability Enhance the functionality and capability DPDK is great library for user Easy to develop & implement Easy to deploy like UNIX apps
  35. 35. 34 Copyright©2014 NTT corp. All Rights Reserved. Thank you! This research is a part of the project for “Research and Development of Network Virtualization Technology” supported by the Ministry of Internal Affairs and Communications.

×