http://ostinato.org/
Srivats P.
6WIND SPEED MATTERS
The Challenge 2014 DPDK Design Contest
OSTINATO
DPDKa elerated

http://ostinato.org/
What is Ostinato?
Open Source Cross Platform
Traffic Generator

http://ostinato.org/
What is DPDK?
Application
Libraries and
user-space NIC
drivers to boost
packet
processing
performance

http://ostinato.org/
Together
DPDK
Packet Processing
Power
OSTINATO
Features and
Flexibility
1/10G line rate
feature rich
traffic generator
on commodity
hardware

http://ostinato.org/
Multi-core Processor
DPDK Programming Model
Core
Packet
Processing
Software
Engine
Core
Packet
Processing
Software
Engine
Core
Launch
and
Control
of Engines
Core
Packet
Processing
Software
Engine
DPDK Libraries + Environment Abstraction Layer (EAL)
SlavesMaster
Run to completion
model for engines
Alternatively,
they can collaborate
in a pipeline model

http://ostinato.org/
Ostinato Controller -Agent Architecture
Agent
(Drone)
Controller
(Ostinato)
►Packet Generation
►Packet Capture
►Statistics
GUI/Python-Script
►Configuration
►Control
►Results
ProtoBuf based RPC
Protocols, Packet Length,
Rates etc.

http://ostinato.org/
Drone Classes and Interfaces
PortPortPortPort
RpcServer
OstService
RPC interface
Port Class
Interface
Ostinato
(Controller)
Drone
(Agent)

http://ostinato.org/
RPC Interface
service OstService {
rpc checkVersion(VersionInfo) returns (VersionCompatibility);
rpc getPortIdList(Void) returns (PortIdList);
rpc getPortConfig(PortIdList) returns (PortConfigList);
rpc modifyPort(PortConfigList) returns (Ack);
rpc getStreamIdList(PortId) returns (StreamIdList);
rpc getStreamConfig(StreamIdList) returns (StreamConfigList);
rpc addStream(StreamIdList) returns (Ack);
rpc deleteStream(StreamIdList) returns (Ack);
rpc modifyStream(StreamConfigList) returns (Ack);
rpc startTransmit(PortIdList) returns (Ack);
rpc stopTransmit(PortIdList) returns (Ack);
rpc startCapture(PortIdList) returns (Ack);
rpc stopCapture(PortIdList) returns (Ack);
rpc getCaptureBuffer(PortId) returns (CaptureBuffer);
rpc getStats(PortIdList) returns (PortStatsList);
rpc clearStats(PortIdList) returns (Ack);
}

http://ostinato.org/
Port Class Interface (1/2)
class AbstractPort {
public:
// Common functionality for all ports implemented by AbstractPort
int id();
const char* name();
bool modify(const OstProto::Port &port);
int streamCount();
StreamBase* streamAtIndex(int index);
StreamBase* stream(int streamId);
bool addStream(StreamBase *stream);
bool deleteStream(int streamId);
void updatePacketList();
(more)

http://ostinato.org/
Port Class Interface (2/2)
(contd.)
// Pure virtual functions implemented by platform specific code
virtual OstProto::LinkState linkState() = 0;
virtual bool hasExclusiveControl() = 0;
virtual bool setExclusiveControl(bool exclusive) = 0;
virtual void clearPacketList() = 0;
virtual void setPacketListSize(quint64 size) = 0
virtual void loopNextPacketSet(qint64 size, qint64 repeats,
long repeatDelaySec, long repeatDelayNsec) = 0;
virtual bool appendToPacketList(long sec, long nsec,
const uchar *packet, int length) = 0;
virtual void setPacketListLoopMode(bool loop, quint64 secDelay,
quint64 nsecDelay) = 0;
virtual void startTransmit() = 0;
virtual void stopTransmit() = 0;
virtual bool isTransmitOn() = 0;
virtual void startCapture() = 0;
virtual void stopCapture() = 0;
virtual bool isCaptureOn() = 0;
virtual QIODevice* captureData() = 0;
void stats(PortStats *stats) = 0;
}

http://ostinato.org/
AbstractPort
PcapPort
LinuxPort BsdPort WinPcapPort
Port Class Diagram (UML)
DpdkPort
Uses libpcap for
packet transmit
and capture

http://ostinato.org/
Key Point for transmitted packets
AbstractPort
computes the minimum set of unique packets
required to be transmitted
pre-builds this minimum unique packet set
during Tx, this packet set is looped over and over
No packet buffers are allocated or built
during transmit

http://ostinato.org/
DpdkPort Engines (1/2)
Port Rx Ring Polling
while (!stopRxPolling) {
rte_eth_rx_burst(rxPkts)
foreach pkt in rxPkts rte_pktmbuf_free(pkt)
}
Note: Ostinato, being a traffic generator, does not need to process Rx
packets
Port Tx
while (!stopTransmit) {
rte_eth_tx_burst(txPkts) // txPkts is pre-built
rte_delay_us(timeTillNextPktOrBurst)
}
Note: ensure rte_eth_tx_burst() does not free mbufs by bumping their
refcnt via rte_pktmbuf_clone() during packet list pre-building time so we
can keep transmitting the same mbuf again and again without
realloc/rebuild

http://ostinato.org/
DpdkPort Engines (2/2)
Port Rx Capture
while (!stopCapture) {
rte_eth_rx_burst(rxPkts)
foreach pkt in rxPkts
rte_memcpy(capBuf++, pkt)
rte_pktmbuf_free(pkt)
}
Port Statistics Rx/Tx
while (!stopStatsPolling) {
rte_eth_stats_get()
sleep(1)
}

http://ostinato.org/
Engine - Core assignment strategies
DpdkPort engines to be assigned to cores
Always On (core assignment at init)
Drone core functionality
RPC, Protocol Builders, PacketList Builder etc.
Port Rx Ring Polling
Port Statistics Rx/Tx
On-Demand (core reserved at init or assigned on-demand)
Port Tx (of pre-built packets)
Port Rx Capture

http://ostinato.org/
Core assignment example strategy #1
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports
Slave 1: Port 1 Tx
Slave 2: Port 2 Tx
…
Slave n: Port n Tx
Pros
Dedicated core to a port Tx
Optimal for port with only 1 Tx queue
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Rx + Capture Rx may exceed core capacity leading to packet drops
Slave 1 – n cores may be under-utilized depending on Tx Rate (or if Tx off)
#ports = (#cores – 2)
Quad-core processor can support only 2 ports

http://ostinato.org/
Core assignment example strategy #2
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports
Slave 1: Port 1 Tx + Port 1 Capture Rx
Slave 2: Port 2 Tx + Port 2 Capture Rx
…
Slave n: Port n Tx + Port n Capture Rx
Pros
More Rx Capture Bandwidth for simultaneous capture on multiple ports
Optimal when packets transmitted from one port are captured on another
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Tx rate may be affected by the multiplexed capture functionality
#ports = (#cores – 2)
Quad-core processor can support only 2 ports

http://ostinato.org/
Core assignment example strategy #3
Core Assignment
Master: Drone core functionality + DPDK port stats (one thread polls all ports)
Slave 0: Poll Rx Rings for all ports, Capture Rx for requested ports
Slave 1: Port Tx Q1 for all ports
Slave 2: Port Tx Q2 for all ports
…
Slave n: Port Tx Qn for all ports
Pros
Multiple cores (equal to no. of supported Tx queues) dedicated to a port Tx
Optimal for ports with multiple Tx queues
All ports supported
Cons
2 cores are potentially under-utilized (Master, Slave 0)
Port Rx + Capture Rx may exceed core capacity leading to packet drops
Slave 1 – n may be under/over-utilized depending on supported Tx queues
and Tx rates

http://ostinato.org/
The “core” problem
Engine - Core assignment strategy will determine
Max number of supported ports
Max packet transmit/capture rate on a port
No one strategy fits all use cases
Optimal solution requires knowledge of
system configuration and capacity
use case
Implement multiple strategies and allow
end-user to override the default strategy

http://ostinato.org/
DpdkPort Performance
Unfortunately, we don’t have access to DPDK
supported hardware NICs and a server-grade multi-
core processor
Current development is being done on emulated
82545EM NICs on a VirtualBox VM running on a
quad-core laptop
On this development platform, for 64-byte packets,
DpdkPort generates a max of ~190Kpps against
PcapPort’s ~15Kpps

http://ostinato.org/
DPDK Potential Improvements
DPDK Build Environment
Not straight-forward to integrate into existing applications
having their own build environment
A traffic generator doesn’t need active Rx Ring polling
If capture is not enabled on port, can PMD/NIC
automatically empty rx ring and free mbuf?
Note: Rx stats still need to be maintained and collected
Nano-second resolution Delay API
rte_delay_ns()

http://ostinato.org/
Source Code
Repo URL: https://code.google.com/r/pstavirs-dpdk/
Caveats
Code is work in progress (aka alpha quality)
Not everything discussed here is implemented as of today
(Aug 1, 2014)

http://ostinato.org/
Build Instructions
DPDK
Download DPDK - http://dpdk.org/download
Follow build and verification steps from
http://dpdk.org/doc/quick-start
Drone
hg clone https://code.google.com/r/pstavirs-dpdk/ ostinato
Export environment variables RTE_SDK and RTE_TARGET
Follow build steps from
http://ostinato.org/wiki/BuildingFromSource

http://ostinato.org/
Ostinato DpdkPort Screenshot

http://ostinato.org/
Support
For programmers who know their “stuff”
Contact the author Srivats P. <pstavirs@gmail.com>
Everyone else
Please wait till the code is ready for “General Availability”
(aka beta quality)

http://ostinato.org/
That's all folks!
Want to help with DPDK-Ostinato?