Dpdk accelerated Ostinato

165,651 views

Published on

Solo Prize Winner - 6WIND Speed Matters: The Challenge Contest

Ostinato is a network packet and traffic generator and analyzer with a friendly GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark. It is useful for both functional and performance testing. (GPL, Linux/BSD/OSX/Win32)

Accompanying code: https://github.com/pstavirs/dpdk-ostinato

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
165,651
On SlideShare
0
From Embeds
0
Number of Embeds
35,811
Actions
Shares
0
Downloads
152
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide

Dpdk accelerated Ostinato

  1. 1. http://ostinato.org/ Srivats P. 6WIND SPEED MATTERS The Challenge 2014 DPDK Design Contest OSTINATO DPDKa elerated
  2. 2. http://ostinato.org/ What is Ostinato? Open Source Cross Platform Traffic Generator
  3. 3. http://ostinato.org/ What is DPDK? Application Libraries and user-space NIC drivers to boost packet processing performance
  4. 4. http://ostinato.org/ Together DPDK Packet Processing Power OSTINATO Features and Flexibility 1/10G line rate feature rich traffic generator on commodity hardware
  5. 5. http://ostinato.org/ Multi-core Processor DPDK Programming Model Core Packet Processing Software Engine Core Packet Processing Software Engine Core Launch and Control of Engines Core Packet Processing Software Engine DPDK Libraries + Environment Abstraction Layer (EAL) SlavesMaster Run to completion model for engines Alternatively, they can collaborate in a pipeline model
  6. 6. http://ostinato.org/ Ostinato Controller -Agent Architecture Agent (Drone) Controller (Ostinato) ►Packet Generation ►Packet Capture ►Statistics GUI/Python-Script ►Configuration ►Control ►Results ProtoBuf based RPC Protocols, Packet Length, Rates etc.
  7. 7. http://ostinato.org/ Drone Classes and Interfaces PortPortPortPort RpcServer OstService RPC interface Port Class Interface Ostinato (Controller) Drone (Agent)
  8. 8. http://ostinato.org/ RPC Interface service OstService { rpc checkVersion(VersionInfo) returns (VersionCompatibility); rpc getPortIdList(Void) returns (PortIdList); rpc getPortConfig(PortIdList) returns (PortConfigList); rpc modifyPort(PortConfigList) returns (Ack); rpc getStreamIdList(PortId) returns (StreamIdList); rpc getStreamConfig(StreamIdList) returns (StreamConfigList); rpc addStream(StreamIdList) returns (Ack); rpc deleteStream(StreamIdList) returns (Ack); rpc modifyStream(StreamConfigList) returns (Ack); rpc startTransmit(PortIdList) returns (Ack); rpc stopTransmit(PortIdList) returns (Ack); rpc startCapture(PortIdList) returns (Ack); rpc stopCapture(PortIdList) returns (Ack); rpc getCaptureBuffer(PortId) returns (CaptureBuffer); rpc getStats(PortIdList) returns (PortStatsList); rpc clearStats(PortIdList) returns (Ack); }
  9. 9. http://ostinato.org/ Port Class Interface (1/2) class AbstractPort { public: // Common functionality for all ports implemented by AbstractPort int id(); const char* name(); bool modify(const OstProto::Port &port); int streamCount(); StreamBase* streamAtIndex(int index); StreamBase* stream(int streamId); bool addStream(StreamBase *stream); bool deleteStream(int streamId); void updatePacketList(); (more)
  10. 10. http://ostinato.org/ Port Class Interface (2/2) (contd.) // Pure virtual functions implemented by platform specific code virtual OstProto::LinkState linkState() = 0; virtual bool hasExclusiveControl() = 0; virtual bool setExclusiveControl(bool exclusive) = 0; virtual void clearPacketList() = 0; virtual void setPacketListSize(quint64 size) = 0 virtual void loopNextPacketSet(qint64 size, qint64 repeats, long repeatDelaySec, long repeatDelayNsec) = 0; virtual bool appendToPacketList(long sec, long nsec, const uchar *packet, int length) = 0; virtual void setPacketListLoopMode(bool loop, quint64 secDelay, quint64 nsecDelay) = 0; virtual void startTransmit() = 0; virtual void stopTransmit() = 0; virtual bool isTransmitOn() = 0; virtual void startCapture() = 0; virtual void stopCapture() = 0; virtual bool isCaptureOn() = 0; virtual QIODevice* captureData() = 0; void stats(PortStats *stats) = 0; }
  11. 11. http://ostinato.org/ AbstractPort PcapPort LinuxPort BsdPort WinPcapPort Port Class Diagram (UML) DpdkPort Uses libpcap for packet transmit and capture
  12. 12. http://ostinato.org/ Key Point for transmitted packets AbstractPort computes the minimum set of unique packets required to be transmitted pre-builds this minimum unique packet set during Tx, this packet set is looped over and over No packet buffers are allocated or built during transmit
  13. 13. http://ostinato.org/ DpdkPort Engines (1/2) Port Rx Ring Polling while (!stopRxPolling) { rte_eth_rx_burst(rxPkts) foreach pkt in rxPkts rte_pktmbuf_free(pkt) } Note: Ostinato, being a traffic generator, does not need to process Rx packets Port Tx while (!stopTransmit) { rte_eth_tx_burst(txPkts) // txPkts is pre-built rte_delay_us(timeTillNextPktOrBurst) } Note: ensure rte_eth_tx_burst() does not free mbufs by bumping their refcnt via rte_pktmbuf_clone() during packet list pre-building time so we can keep transmitting the same mbuf again and again without realloc/rebuild
  14. 14. http://ostinato.org/ DpdkPort Engines (2/2) Port Rx Capture while (!stopCapture) { rte_eth_rx_burst(rxPkts) foreach pkt in rxPkts rte_memcpy(capBuf++, pkt) rte_pktmbuf_free(pkt) } Port Statistics Rx/Tx while (!stopStatsPolling) { rte_eth_stats_get() sleep(1) }
  15. 15. http://ostinato.org/ Engine - Core assignment strategies DpdkPort engines to be assigned to cores Always On (core assignment at init) Drone core functionality RPC, Protocol Builders, PacketList Builder etc. Port Rx Ring Polling Port Statistics Rx/Tx On-Demand (core reserved at init or assigned on-demand) Port Tx (of pre-built packets) Port Rx Capture
  16. 16. http://ostinato.org/ Core assignment example strategy #1 Core Assignment Master: Drone core functionality + DPDK port stats (one thread polls all ports) Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports Slave 1: Port 1 Tx Slave 2: Port 2 Tx … Slave n: Port n Tx Pros Dedicated core to a port Tx Optimal for port with only 1 Tx queue Cons 2 cores are potentially under-utilized (Master, Slave 0) Port Rx + Capture Rx may exceed core capacity leading to packet drops Slave 1 – n cores may be under-utilized depending on Tx Rate (or if Tx off) #ports = (#cores – 2) Quad-core processor can support only 2 ports
  17. 17. http://ostinato.org/ Core assignment example strategy #2 Core Assignment Master: Drone core functionality + DPDK port stats (one thread polls all ports) Slave 0: Poll Rx Rings for all ports Slave 1: Port 1 Tx + Port 1 Capture Rx Slave 2: Port 2 Tx + Port 2 Capture Rx … Slave n: Port n Tx + Port n Capture Rx Pros More Rx Capture Bandwidth for simultaneous capture on multiple ports Optimal when packets transmitted from one port are captured on another Cons 2 cores are potentially under-utilized (Master, Slave 0) Port Tx rate may be affected by the multiplexed capture functionality #ports = (#cores – 2) Quad-core processor can support only 2 ports
  18. 18. http://ostinato.org/ Core assignment example strategy #3 Core Assignment Master: Drone core functionality + DPDK port stats (one thread polls all ports) Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports Slave 1: Port Tx Q1 for all ports Slave 2: Port Tx Q2 for all ports … Slave n: Port Tx Qn for all ports Pros Multiple cores (equal to no. of supported Tx queues) dedicated to a port Tx Optimal for ports with multiple Tx queues All ports supported Cons 2 cores are potentially under-utilized (Master, Slave 0) Port Rx + Capture Rx may exceed core capacity leading to packet drops Slave 1 – n may be under/over-utilized depending on supported Tx queues and Tx rates
  19. 19. http://ostinato.org/ The “core” problem Engine - Core assignment strategy will determine Max number of supported ports Max packet transmit/capture rate on a port No one strategy fits all use cases Optimal solution requires knowledge of system configuration and capacity use case Implement multiple strategies and allow end-user to override the default strategy
  20. 20. http://ostinato.org/ DpdkPort Performance Unfortunately, we don’t have access to DPDK supported hardware NICs and a server-grade multi- core processor Current development is being done on emulated 82545EM NICs on a VirtualBox VM running on a quad-core laptop On this development platform, for 64-byte packets, DpdkPort generates a max of ~190Kpps against PcapPort’s ~15Kpps
  21. 21. http://ostinato.org/ DPDK Potential Improvements DPDK Build Environment Not straight-forward to integrate into existing applications having their own build environment A traffic generator doesn’t need active Rx Ring polling If capture is not enabled on port, can PMD/NIC automatically empty rx ring and free mbuf? Note: Rx stats still need to be maintained and collected Nano-second resolution Delay API rte_delay_ns()
  22. 22. http://ostinato.org/ Source Code Repo URL: https://code.google.com/r/pstavirs-dpdk/ Caveats Code is work in progress (aka alpha quality) Not everything discussed here is implemented as of today (Aug 1, 2014)
  23. 23. http://ostinato.org/ Build Instructions DPDK Download DPDK - http://dpdk.org/download Follow build and verification steps from http://dpdk.org/doc/quick-start Drone hg clone https://code.google.com/r/pstavirs-dpdk/ ostinato Export environment variables RTE_SDK and RTE_TARGET Follow build steps from http://ostinato.org/wiki/BuildingFromSource
  24. 24. http://ostinato.org/ Ostinato DpdkPort Screenshot
  25. 25. http://ostinato.org/ Support For programmers who know their “stuff” Contact the author Srivats P. <pstavirs@gmail.com> Everyone else Please wait till the code is ready for “General Availability” (aka beta quality)
  26. 26. http://ostinato.org/ That's all folks! Want to help with DPDK-Ostinato?

×